Harmonised Data Access (HDA)
The Destination Earth Data Lake Harmonised Data Access (HDA) API provides seamless access to “DestinE Data” and “Federated Data” as defined in the DestinE Data Portfolio.
The API is based on the Spatio Temporal Asset Catalog specification (STAC) allowing users to discover datasets and services. Datasets contain data which can be accessed or downloaded for further processing. HDA exposes a set of RESTful APIs for programmatic access to data, along with a STAC 1.0.0-compliant interface.
Datasets (Collections) can be discovered without authentication, the data access requires instead a DestinE account.
It is possible to create a DestinE account following the instruction at this link: How to create a DestinE Platform account.
Default HDA quota provided for users includes
maximum 4 requests per second,
bandwidth up to 20 Mbps per connection and
monthly transfer up to 6 TB per month.
How to use HDA
Here we list some of the tools that can be configured with the HDA endpoint (STAC API endpoint ) and with which one can gain access to the vast amount of diverse data exposed by Destination Earth - Data Lake.
- PyStac
Python library for working with SpatioTemporal Asset Catalogs (STACs). In general, the STAC interface allows integration with STAC-compatible tools, such as the STAC BROWSER, QGIS with STAC extension.
- Any REST client pointing to the HDA STAC API endpoint
REST clients are available in various programming languages and also provided by tools like Postman, Swagger UI, SoapUI, Talend API tester and others.
- cURL
Command line tool.
STACK Service provided by Destination Earth Data Lake, can also be used in conjunction with the HDA component to facilitate access and processing of data. The hosted service JupyterLab Environment will allow users to execute their own Python code. Jupyter Notebook examples found at Destination Earth on Github are pre-loaded in the DEDL JupyterLab environment and can be executed using the dedicated ‘DEDL Python’ kernel.
Jupyter Notebook examples at Destination Earth on Github will help to dive into the HDA. These are the links to the appropriate folders in that repository:
- REST
How to use the HDA APIs by sending a few HTTP requests, using Python code.
Includes authentication using DESP accounts, discovery search access and visualization of data.
- PySTAC
How to use HDA with pystac-client.
Includes iterating through Collections and Items as well as how to perform simple spatio-temporal searches.
- DestinE Digital Twins
How to access and visualize DestinE Digital Twins data through HDA.
- EUM_data
How to access and visualize EUMETSAT data.
- CDS_data
How to access and visualize Climate Data Store data.
How to perform common operations using HDA
This section provides a guidance on accessing and utilizing the HDA. It includes detailed step-by-step instructions for accessing specific endpoints, discovering available services along with their corresponding descriptions. It also covers the process of listing and searching for available STAC collections as well as gaining further insights and information about them.
To run the examples below, you can either use the DestinE Platform provided Insula service or the DEDL provided STACK Service. Insula service is an integrated code environment based on JupyterHub, available through the DestinE Platform. The ‘datalake-lab’ folder in Insula’s navigation panel offers useful HDA tutorials.
Open API documentation
The HDA STAC API documentation is available here: OpenAPI documentation
The HDA STAC API documentation allows to give a try with the different exposed API. Some of them are available without authentication, some other instead require to be authenticated.
However, if you are authenticated / registered, you will be able to discover authorised resources.
DestinE credential allows to authenticate directly in the OpenAPI documentation page, as described in the following section, and give a try with all the exposed API.
Authentication
To authenticate use the ‘Authorize’ button.
This will open a window where to put DestinE credential. The window needs also other parameters to be set as described in the Open API documentation, ‘client_id’ must be set to hda-public and the checkbox ‘openid default scope’ must be marked. Once done, press the button ‘Authorize’ in the opened window to have access to all the exposed API.
Discovery services
The discovery services API allows to find the services provided by the DestinE Data Lake.
This API does not require to be authenticated / registered.
The Open API documentation: https://hda.data.destination-earth.eu/docs/#/DEDL%20API%20-%20Services allows to discover DEDL services:
You can also get a description of a specific discovered service:
