Provided Hooks

Table of Contents

Overview

On this page, we give more details on the ‘Provided Hooks’ i.e. those workflows/functions that have been pre-developed and pre-deployed by Destination Earth Data Lake.

The Hook service provides ready-to-use high level serverless workflows and functions preconfigured to efficiently access and manipulate Destination Earth Data Lake (DEDL) data. A growing number of workflows and functions will provide on-demand capabilities for the diverse satellite data analysis needs.

The full list of available Hooks are seen in the Hook Descriptions section further below.

Note

The main processor that will be of use to Destination Earth Data Lake users is the data-harvest processor; with this you will be able to download data of interest to your S3 Object Storage.

The collection of Jupyter Notebooks examples on how to use the DestinE Data Lake services can be found at Destination Earth on Github.

Getting Started

  1. Jupyter Notebook available here

  1. Authenticate to get your token

    ../../../_images/provided_hook1.png

  2. Discover available Workflows using API using https://odp.data.destination-earth.eu/odata/v1/Workflows

    ../../../_images/provided_hook2.png
  3. Get details of just one workflow.
    • Narrow results down to one workflow using the filter option.

      • e.g. https://odp.data.destination-earth.eu/odata/v1/Workflows?$expand=WorkflowOptions&$filter=(Name eq 'card_bs_private')

    ../../../_images/provided_hook3.png

  4. Setting up input and output parameters

    ../../../_images/provided_hook4.png

  5. Make an order (i.e. Trigger a Provided Hook - card_bs_private) - Private Bucket Output Storage

    ../../../_images/provided_hook5.png

  6. Make an order (i.e. Trigger a Provided Hook - card_bs_private) - Temporary Storage

    ../../../_images/provided_hook6.png
  7. Checking status of an order

    • Here we can see that the 2 Orders are queued awaiting treatment. Execute this until you see completed status

    ../../../_images/provided_hook7.png
  8. Checking for completed status of an order

    • Here we can see that the 2 Orders are in the completed status and the resulting files can be found either in the configured private bucket or in temporary storage

    ../../../_images/provided_hook8.png
  9. Check results in private bucket

    • We can check the results in the output bucket using python ‘boto’ for example

    ../../../_images/provided_hook9.png
  10. Check results in temporary storage

    • Here we can see that the resulting files are stored in temporary storage and can be obtained through the provided link

    ../../../_images/provided_hook10.png

Hook Descriptions

In the table below we can see a list of pre-developed and pre-deployed Hooks made available by Destination Earth Data Lake:

Workflow Name
Display Name
Version
Description
Input Product Type(s)
Output Product Type(s)
Compatible DEDL HDA
data-harvest
Data Harvest
0.0.1
The data harvest processor is considered the main processor of interest for Destination Earth Data Lake users. It is able to download datasets of interest to S3 object storage configured by the user.
Data-harvest is a workflow that allows users to download data from external sources. It requires a URL to the external catalogue, credentials and data to download. The workflow is mainly used to download data from HDA (https://hda.data.destination-earth.eu/) using STAC.
Yes
card_bs
Sentinel-1: Terrain-corrected backscatter
3.6.2
Sentinel-1 CARD BS (Copernicus Analysis Ready Data Backscatter) processor generates terrain-corrected geocoded Sentinel-1 Level 2 backscattering by removing the radiometric effect imposed by relief (provided by DEM). This allows comparability of images, e.g. for analysis of changes in land cover. This processor provided by the Joint Research Centre is based on a GPT graph that can be run with ESA SNAP.
GRD, IW_GRDH_1S, IW_GRDM_1S, EW_GRDH_1S, EW_GRDM_1S, WV_GRDM_1S, GRD-COG, IW_GRDH_1S-COG, IW_GRDM_1S-COG, EW_GRDH_1S-COG, EW_GRDM_1S-COG, WV_GRDM_1S-COG
CARD-BS
Yes
card_cohinf
Sentinel-1: Coherence/Interferometry
1.0.0
The Sentinel-1 CARD COH-INF (Copernicus Analysis Ready Data Coherence/Interferometry) processor generates a Sentinel-1 product form SLC product type. Concurrently, a terrain-correction (using DEM) is performed. This processor provided by the Joint Research Centre is based on a GPT graph that can be run with ESA SNAP.
SLC, S1_SLC__1S, S2_SLC__1S, S3_SLC__1S, S4_SLC__1S, S5_SLC__1S, S6_SLC__1S, IW_SLC__1S, EW_SLC__1S, WV_SLC__1S
Yes
c2rcc
Sentinel-2: C2RCC
1.1.1
The C2RCC (Case 2 Regional Coast Colour) processor allows water constituents in coastal or inland water bodies to be derived from optical satellite data. The processor uses a database of radiative transfer simulations inverted by neural networks. It can be applied to all past and current ocean colour sensors as well as Sentinel-2 imagery. It has been validated in various studies and is also implemented in SNAP. It is also used in the Sentinel 3 OLCI ground segment processor for the generation of the Case 2 water products.
S2MSI1C, OL_2_LFR___, OL_2_LRR___, OL_2_WFR___, OL_2_WRR___
Yes
lai
Sentinel-2: SNAP-Biophysical
1.1.1
The SNAP-BIOPHYSICAL processor derives vegetation biophysical variables based on top-of-canopy spectral reflectances from Sentinel-2 data. Following the approach described by Weiss et al. (2000, DOI: 10.1051/agro:2000105), the processor estimates: LAI (Leaf Area Index), FAPAR (Fraction of Absorbed Photosynthetically Active Radiation) and FVC (Fractional Vegetation Cover), all recognized as Essential Climate Variables (ECVs) by international organizations such as the Global Climate Observing System (GCOS)
S2MSI2A
Yes
maja
Sentinel-2: MAJA Atmospheric Correction
3.0.1
The Sentinel-2 L2A MAJA (The MACCS-ATCOR Joint Algorithm) is a processor for generation of Bottom-of-Atmoshere reflectances of Sentinel-2 imagery. It builds on MACCS (Multi-sensor Atmospheric Correction and Cloud Screening) developed by CNES and CESBIO, and features from ATCOR (Atmospheric and Topographic Correction) developed by the German Aerospace Centre (DLR). It performs atmospheric correction, cloud and cloud shadow masking, and topographic correction. For atmospheric correction, MAJA retrieves aerosol optical thickness from multi-temporal data assuming the relative stability of surface reflectances with time (within a few days), compared to the high variability of AOT (which can change hourly). That is why MAJA operates on a series of Sentinel-2 images rather than a single image. MAJA can also use information on AOT from Copernicus Atmosphere Monitoring Service (CAMS).
S2MSI1C
No
sen2cor
Sentinel-2: Sen2Cor
2.10.0
The Sen2Cor processor generates Sentinel-2 Level 2A product (Bottom-Of-Atmosphere reflectances) by performing the atmospheric, terrain and cirrus correction of Top-Of-Atmosphere Level 1C input data. In addition, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities can be generated. The Sen2Cor products are in the equivalent format to the Level 1C User Product: JPEG 2000 images, preserving the original band spatial resolutions, i.e. 10, 20 and 60 meters.
S2MSI2A
Yes
copdem
Copernicus DEM Mosaic
1.0
The CopDEM workflow first searches the catalogue for products that intersect the input product geometry and are of product type DGE_30. The processor then executes SNAP/GDAL methods and generates input products into the output dem.tif file.
*
No