Data Discovery

Learn how to explore and search for datasets in the DestinE Data Portfolio using the HDA API.

Overview

The HDA API provides two main approaches for data discovery:

1. Service Discovery (/services) - Find cloud-native edge services (no auth required)

2. Dataset Discovery (/stac/v2) - Explore data collections and search for items

Reference info

For more information about the tools and libraries used in this guide see:

• EODAG Documentation

• pySTAC Client Documentation

Services Discovery

Explore available DEDL edge services and their capabilities (no authentication required):

# Get all services
curl "https://hda.data.destination-earth.eu/services"

# Get the dedl-fdp service
curl "https://hda.data.destination-earth.eu/services/dedl-fdp"

# Filter services by freetext search (title, description and keywords)
curl "https://hda.data.destination-earth.eu/services?q=processing"

Datasets Discovery

Explore the available data collections to find the ones that match your needs.

EODAG

# Get all DEDL product types
product_types = dag.list_product_types(provider='dedl')

# Get all the product types related to climae change
product_types = dag.guess_product_type(free_text="climate change")

product_types = dag.guess_product_type(free_text="temperature OR land")

# Get all product types related to temperature and land
product_types = dag.guess_product_type(free_text="temperature AND land")

# Search for product types containing temperature, land cover, or soil moisture data
product_types = dag.guess_product_type(free_text="temperature OR (land AND cover) OR (soil AND moisture)")

PySTAC

import pystac_client

catalog = pystac_client.Client.open("https://hda.data.destination-earth.eu/stac/v2")

# List all collections
collections = list(catalog.get_collections())

# Get specific collection details
collection = catalog.get_collection("EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION")

# Search collections with freetext
collections = catalog.collection_search(q="climate change").collection_list()

# Search collections with multiple terms
collections = catalog.collection_search(q="temperature,land").collection_list()

# Search collections with datetime filter
collections = catalog.collection_search(
    datetime="1981-10-01T00:00:00Z/1981-10-31T00:00:00Z"
).collection_list()

# Search collections with bbox filter
collections = catalog.collection_search(bbox=[2.2, 48.8, 2.4, 49.0]).collection_list()

# Complex collection search: bbox + datetime + limit
collections = catalog.collection_search(
    bbox=[2.2, 48.8, 2.4, 49.0],
    datetime="1981-10-01T00:00:00Z/1981-10-31T00:00:00Z",
    limit=10,
).collection_list()

# Search collections by federation backend using CQL2 filter
collections = catalog.collection_search(
    query={"federation:backends": {"eq": "dedt_lumi"}}
).collection_list()

cURL

# Get all collections
curl "https://hda.data.destination-earth.eu/stac/v2/collections"

# Get collection EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.DT_CLIMATE_ADAPTATION"

# Search collections related to climate change using freetext-search
curl "https://hda.data.destination-earth.eu/stac/v2/collections?q=%22climate%20change%22"

# Search collections related to either temperature or land
curl "https://hda.data.destination-earth.eu/stac/v2/collections?q=temperature,land"

# Get collections with datetime filter
curl "https://hda.data.destination-earth.eu/stac/v2/collections?datetime=1981-10-01T00:00:00Z/1981-10-31T00:00:00Z"

# Get collections with geometry filter (bounding box)
curl "https://hda.data.destination-earth.eu/stac/v2/collections?bbox=2.2,48.8,2.4,49.0"

# Complex search: bbox + datetime + limit
curl "https://hda.data.destination-earth.eu/stac/v2/collections?bbox=2.2,48.8,2.4,49.0&datetime=1981-10-01T00:00:00Z/1981-10-31T00:00:00Z&limit=10"

# Search collections by federation backend
curl "https://hda.data.destination-earth.eu/stac/v2/collections?query={%22federation:backends%22:%20{%22eq%22:%20%22dedt_lumi%22}}"

Searching for Data

Search for specific data items using collection-specific endpoints and cross-collection search.

Basic Search

Use basic search filters with collection items and search endpoints:

EODAG

# Define area of interest
extent = {
    'lonmin': 2.2,   # west
    'lonmax': 2.4,   # east
    'latmin': 48.8,  # south
    'latmax': 49.0   # north
}

# Search within a specific product type
search_results = dag.search(
    productType="MSG_AMVR02",
    geom=extent,
    start="2024-07-01",
    end="2024-07-31"
)

# Search with specific ID. EODAG accepts only one id
search_results = dag.search(
    productType="MSG_AMVR02",
    id="W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,MET2+MVIRI_C_EUMS_19811003040000Z_19811003060000Z_1_OR_FES_E0000_0200"
)

# Search with sorting (newest first)
search_results = dag.search(
    productType="MSG_AMVR02",
    start="2024-07-01",
    end="2024-07-31",
    sort_by=[("startTimeFromAscendingNode", "DESC")]  # Sort by datetime descending
)

PySTAC

# Search within a specific collection
search = catalog.search(
    collections=["EO.EUM.DAT.AMVR02"],
    bbox=[2.2, 48.8, 2.4, 49.0],
    datetime="1981-10-01/1981-10-31",
    limit=10
)

# Search with EUMETSAT IDs (PySTAC handles commas automatically)
search = catalog.search(
    collections=["EO.EUM.DAT.MSG.AMVR02"],
    ids=[
        "W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,MET2+MVIRI_C_EUMS_19811003040000Z_19811003060000Z_1_OR_FES_E0000_0200",
        "W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,MET2+MVIRI_C_EUMS_19810906100000Z_19810906120000Z_1_OR_FES_E0000_0200"
    ]
)

# Search with reverse sorting (newest first)
search = catalog.search(
    collections=["EO.EUM.DAT.AMVR02"],
    sortby="-datetime",
    limit=10
)

# Get items from search results
items = list(search.items())

cURL

# Search within a specific collection using /collections/{collection_id}/items
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.EUM.DAT.AMVR02/items?bbox=2.2,48.8,2.4,49.0&datetime=1981-10-01T00:00:00Z/1981-10-31T00:00:00Z&limit=10"

# Cross-collection search (MUST specify exactly 1 collection)
curl "https://hda.data.destination-earth.eu/stac/v2/search?collections=EO.EUM.DAT.AMVR02&bbox=2.2,48.8,2.4,49.0&datetime=1981-10-01T00:00:00Z/1981-10-31T00:00:00Z&limit=10"

# Search with sorting
curl "https://hda.data.destination-earth.eu/stac/v2/search?collections=EO.EUM.DAT.AMVR02&sortby=datetime&limit=10"

# Search with reverse sorting (newest first)
curl "https://hda.data.destination-earth.eu/stac/v2/search?collections=EO.EUM.DAT.AMVR02&sortby=-datetime&limit=10"

# Search with specific item IDs (simple IDs without commas)
curl "https://hda.data.destination-earth.eu/stac/v2/search?collections=EO.ESA.DAT.SENTINEL-1.L1_GRD&ids=S1A_IW_GRDH_1SSV_20141004T010233_20141004T010258_002673_002FAB_19F2,S1A_IW_GRDH_1SSV_20141004T010208_20141004T010233_002673_002FAB_58A8"

# Search with EUMETSAT IDs (POST required due to commas in IDs)
curl -X POST "https://hda.data.destination-earth.eu/stac/v2/search" \
  -H "Content-Type: application/json" \
  -d '{
    "collections": ["EO.EUM.DAT.MSG.AMVR02"],
    "ids": [
      "W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,MET2+MVIRI_C_EUMS_19811003040000Z_19811003060000Z_1_OR_FES_E0000_0200",
      "W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,MET2+MVIRI_C_EUMS_19810906100000Z_19810906120000Z_1_OR_FES_E0000_0200"
    ]
  }'

Note

EUMETSAT Item IDs with Commas

EUMETSAT datasets use item IDs with embedded commas (e.g., W_XX-EUMETSAT-Darmstadt,IR+ATMOSPHERE+AMV,...). These commas conflict with URL parameter parsing, so you must use POST requests for EUMETSAT ID searches.

Note

Sorting Options

Common sort fields include: datetime, eo:cloud_cover

• Use + or no prefix for ascending order (oldest first)

• Use - prefix for descending order (newest first)

• Multiple sort fields: sortby=datetime,-eo:cloud_cover (datetime asc, cloud cover desc)

Advanced Search with Queryables

Use queryables to discover available filters and perform advanced searches.

EODAG

# Get queryables for a specific collection
queryables = dag.get_queryables("S2_MSI_L2A")

# Search with cloud cover filter and specific tile
search_results = dag.search(
    productType="S2_MSI_L2A",
    start="2024-07-01",
    end="2024-07-31",
    cloudCover=20,  # Max 20% cloud cover
    tileIdentifier="31TFK" # Specific tile ID
)

PySTAC

# First, get queryables for the collection
queryables = catalog.get_merged_queryables(collections=["EO.ESA.DAT.SENTINEL-2.MSI.L2A"])

# Advanced search using queryables
search = catalog.search(
    collections=["EO.ESA.DAT.SENTINEL-2.MSI.L2A"],
    datetime="1981-10-01T00:00:00Z/1981-10-31T00:00:00Z",
    query={
        "eo:cloud_cover": {"lte": 20},
        "tileIdentifier": {"eq": "T31UDQ"}
    }
)

items = search.items()

cURL

# First, discover available queryables for a collection
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ESA.DAT.SENTINEL-2.MSI.L2A/queryables"

# Searching data for a specific tile with cloud cover filter
curl -X POST "https://hda.data.destination-earth.eu/stac/v2/search" \
  -H "Content-Type: application/json" \
  -d '{
       "collections": ["EO.ESA.DAT.SENTINEL-2.MSI.L2A"],
       "datetime": "2020-01-01T00:00:00Z/..",
       "query": {
           "eo:cloud_cover": {"lte": 20},
           "tileIdentifier": {"eq": "31TFK"}
       }
   }'

Search data from ECMWF datasets

ECMWF collections have dynamic queryables that change based on available data.

For ECMWF datasets, you need to:

1. Discover available queryables for the collection

2. Use the queryables to identify the combination of parameters you want to use

3. Search data using the selected parameters

You can also directly order the STAC item using the /order endpoint. It is described in the Ordering and downloading ECMWF data section.

EODAG

params = {"ecmwf:variable": ["carbon_dioxide"]}
queryables = dag.list_queryables(
    productType="CAMS_GREENHOUSE_EGG4_MONTHLY",
    fetch_providers=False,
    **params,
)

# We validate the queryables against the parameters
# This ensures that the parameters are valid for the selected product type
# Here it will raise an error because pressure_level is required for carbon_dioxide
invalid_queryables = queryables.get_model().model_validate(params)

# Now we try with a valid combination of parameters
params = {
    "ecmwf:variable": ["carbon_dioxide"],
    "ecmwf:pressure_level": ["100"],
    "ecmwf:time": ["0000"],
    "ecmwf:step": ["1"]
}
search_results = dag.list_queryables(
    productType="CAMS_GREENHOUSE_EGG4_MONTHLY",
    **params
)
valid_queryables = queryables.get_model().model_validate(params)

# Now that we have the valid queryables, we can search for data
search_results = dag.search(
    productType="CAMS_GREENHOUSE_EGG4_MONTHLY",
    **params
)

cURL

# Get all queryables for dataset EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS
# Properties are sorted in the order they need to be used in search
# This is important for ECMWF datasets as they have dynamic queryables
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS/queryables"

# ECMWF queryables are dynamic. For instance, if you pick the variable sea_ice_cover, you will see that pressure levels have no available values
# This is because sea_ice_cover is not available at pressure levels
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS/queryables?ecmwf:variable=sea_ice_cover"

# If I pick the variable carbon_dioxide, I will see that pressure levels are available
# This is because carbon_dioxide is available at pressure levels
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS/queryables?ecmwf:variable=carbon_dioxide"

# This request will return an error because carbon_dioxide is not available at pressure level 3000
# The error message will indicate that the combination of variable and pressure level is invalid and show the available pressure levels
curl "https://hda.data.destination-earth.eu/stac/v2/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS/queryables?ecmwf:variable=carbon_dioxide&ecmwf:pressure_level=3000"

# Once you have the combination of parameters you want to use, you can search for data
# Notice the year/month/day parameters instead of using datetime. This is the preferred way to search ECMWF data
# datetime is still available but offer you less flexibility
curl -X POST "https://hda.data.destination-earth.eu/stac/v2/search" \
  -H "Content-Type: application/json" \
  -d '{
    "collections": ["EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS"],
    "query": {
      "ecmwf:variable": {"eq": "carbon_dioxide"},
      "ecmwf:pressure_level": {"eq": 30},
      "ecmwf:year": {"eq": 2020},
      "ecmwf:month": {"eq": 1},
      "ecmwf:day": {"eq": 1}
    }
  }'

The response of the search will be 1 item matching the search criteria. The item does not have any asset at the moment and is defind as order:status: orderable and storage:tier: offline.

The item contains a link with relation retrieve. Executing the link will trigger the order process and return the STAC item ordered.

// Example of a STAC item returned by the search for ECMWF datasets
{
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "assets": {},
            "id": "CAMS_GREENHOUSE_EGG4_MONTHLY_ORDERABLE_a00cdc677748b95ee46dbce24066e43e6a5bd429",
            "geometry": {...},
            "bbox": [
                -180.0,
                -90.0,
                180.0,
                90.0
            ],
            "collection": "EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS",
            "stac_version": "1.0.0",
            "properties": {
                "order:status": "orderable",
                "storage:tier": "offline",
                "title": "CAMS_GREENHOUSE_EGG4_MONTHLY_ORDERABLE_a00cdc677748b95ee46dbce24066e43e6a5bd429",
                "datetime": "2020-01-01T00:00:00.000000Z",
                "start_datetime": "2020-01-01T00:00:00.000000Z",
                "end_datetime": "2020-01-01T00:00:00.000000Z",
                "license": "proprietary",
                "collection": "CAMS_GREENHOUSE_EGG4_MONTHLY",
                "alias": "EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS",
                "stacCollection": "/eodag/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS.json",
                "scope": {
                    "discover": null,
                    "search": "hda-public-data-access",
                    "download": "hda-public-data-access"
                },
                "ecmwf:year": 2020,
                "ecmwf:data_format": "grib",
                "ecmwf:dataset": "cams-global-ghg-reanalysis-egg4-monthly",
                "ecmwf:day": 1,
                "ecmwf:product_type": "monthly_mean",
                "ecmwf:variable": "carbon_dioxide",
                "ecmwf:pressure_level": 30,
                "ecmwf:month": 1,
                "federation:backends": [
                    "cop_ads"
                ]
            },
            "stac_extensions": [
                "https://stac-extensions.github.io/storage/v1.0.0/schema.json",
                "https://stac-extensions.github.io/order/v1.1.0/schema.json"
            ],
            "links": [
                {
                    "rel": "retrieve",
                    "type": "application/geo+json",
                    "href": "https://hda.dedl.csgroup.space/stac/v2/collections/EO.ECMWF.DAT.CAMS_GLOBAL_GREENHOUSE_GAS_REANALYSIS_MONTHLY_AV_FIELDS/order",
                    "method": "POST",
                    "title": "Retrieve",
                    "body": {
                        "data_format": "grib",
                        "day": 1,
                        "month": 1,
                        "pressure_level": 30,
                        "product_type": "monthly_mean",
                        "variable": "carbon_dioxide",
                        "year": 2020
                    }
                },
                ...
            ]
        }
    ],
    ...
}

ECMWF Dynamic Queryables Features:

• Real-time updates: Available variables and parameters reflect current data availability

• Forecast-specific: Parameters like forecast steps, pressure levels, and forecast types

• Time-dependent: Some parameters may only be available for specific time ranges

• Validation: Invalid parameter combinations are rejected with helpful error messages

Search Best Practices

Efficient Search Tips

1. Always specify exactly one collection for /search endpoint

2. Start with queryables to understand available filters

3. Use small limits for initial testing, then increase

4. Combine spatial and temporal filters to reduce result size

Search Limitations

• Cross-collection search is not available. Each search request requires exactly one collection to be specified

• ECMWF queryables change dynamically - always check current availability

• Complex queries may have slower response times

Next Steps

• Data Access - Download and work with discovered data

• FAQ & Troubleshooting - Troubleshooting search issues

• Tutorials - Practical search examples