Using AWS CLI to access Fresh Data Pool (EODATA) on Destination Earth

In this article, you will learn how to access Fresh Data Pool (EODATA) on Destination Earth Data Lake, using AWS CLI.

DEDL Fresh Data Pool includes satellite imagery and environmental data. The former is also known as Fresh Data Pool (EODATA) (Earth Observation Data) and with AWS CLI, you can interact with data in AWS S3 format to retrieve and manage these large datasets directly from the command line.

What we are going to cover

Fresh Data Pool (EODATA) Storage endpoint

S3 is an object storage service with which you can retrieve data over HTTPS using REST API. The default S3 endpoint address to work with Fresh Data Pool (EODATA) on Destination Earth is:

https://eodata.data.destination-earth.eu

Prerequisites

No. 1 Access to My DataLake Services

You need access to My DataLake Services. See How to create profile on My DataLake Services.

No. 2 Credentials for S3 EODATA access from My DataLake Services

See article How to obtain eodata S3 keys through My DataLake Services

No. 3 AWS CLI installed on your local computer or virtual machine

AWS CLI is a free and open source software which can manage different clouds, not only those hosted by Amazon Web Services. In this article, you will use it to control your resources hosted on Destination Earth cloud.

This article was written for Ubuntu 22.04. The commands may work on other operating systems, but might require adjusting.

Here is how to install AWS CLI on Ubuntu 22.04:

sudo apt install awscli

Before using the AWS CLI, ensure that:

  • The AWS CLI is installed (version 2 is recommended).

  • Your credentials are properly configured.

To configure the CLI, run:

aws configure

Enter the following when prompted:

AWS Access Key ID [None]: YOUR_ACCESS_KEY
AWS Secret Access Key [None]: YOUR_SECRET_KEY
Default region name [None]: eu-central-1
Default output format [None]: json

Exploring Fresh Data Pool (EODATA) with aws s3 ls

Use the aws s3 ls command to explore directories. The example below lists the contents of a specific directory:

aws s3 ls s3://DIAS/Envisat-ASAR/ASAR/ASA_WSS_1P/2012/04/08/ \
  --endpoint-url https://eodata.data.destination-earth.eu

This command will display a list of subdirectories or files:

PRE ASA_WSS_1PNESA20120408_110329_000000603113_00267_52867_0000.N1/
PRE ASA_WSS_1PNESA20120408_110428_000000603113_00267_52867_0000.N1/
PRE ASA_WSS_1PNESA20120408_110446_000000603113_00267_52867_0000.N1/

Listing the root directory

To explore the root of the Fresh Data Pool (EODATA) repository:

aws s3 ls s3://DIAS/ --endpoint-url https://eodata.data.destination-earth.eu

This is the result:

../../../../_images/entire-repository.png

Path format guidelines

When using aws s3 ls, follow these rules for specifying paths:

  • Use / as a separator between directories.

  • Do not start the path with a leading slash.

  • To list the contents of a folder, always end the path with a slash.

  • The first part of the path must be a dataset available at the root level (e.g. Sentinel-1, Sentinel-2, Envisat-ASAR).

Optional: Filtering output in Bash

You can extract just the directory names from the listing using:

aws s3 ls s3://DIAS/Envisat-ASAR/ASAR/ASA_WSS_1P/2012/04/08/ \
  --endpoint-url https://eodata.data.destination-earth.eu \
  | awk '{print $NF}'

This command returns:

ASA_WSS_1PNESA20120408_110329_000000603113_00267_52867_0000.N1/
ASA_WSS_1PNESA20120408_110428_000000603113_00267_52867_0000.N1/
ASA_WSS_1PNESA20120408_110446_000000603113_00267_52867_0000.N1/

PNG files from a Fresh Data Pool (EODATA) repository

Basic info on PNG files

PNG stands for Portable Network Graphics. It is a raster image format known for:

  • Storing pixel-based image data,

  • Supporting transparency,

  • Providing lossless compression.

In Earth Observation (EO) data, .png files are often used to represent processed imagery, such as false-color composites or classification maps. These files are typically not geo-referenced like .tif or .SAFE files, but they provide quick, visual access to EO content.

How to download a PNG file with AWS CLI

To download a .PNG file using the AWS CLI, use the command:

aws s3 cp s3://DIAS/Landsat-5/TM/L1T/2011/11/16/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1.BP.PNG \
  . \
  --endpoint-url https://eodata.data.destination-earth.eu

This command downloads the file to your current working directory. If the file already exists locally, it will be overwritten without prompting for confirmation.

Replace the file path above with the full key to the PNG file you want to download.

Explanation of the command

  • aws s3 cp: Copies a file from an S3 bucket to your local system.

  • s3://DIAS/…: The full key to the PNG file in the Fresh Data Pool (EODATA) repository.

  • .: Target location (current directory).

  • --endpoint-url: Specifies the Fresh Data Pool (EODATA) endpoint for AWS S3-compatible access.

After running the command, you will see a download progress bar if the terminal supports it. Once completed, the file will be available in your current directory.

Expected output

The downloaded PNG file should appear like this in your terminal (on Linux):

../../../../_images/some_other_image_for_output.png

Here is what the downloaded image may look like:

../../../../_images/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1.BP1.PNG

Example of a downloaded PNG visualization from Fresh Data Pool (EODATA)

EOF files from an Fresh Data Pool (EODATA) repository

What is a .EOF file

eof is short for Earth Observation Format. It is a custom format used to store and share Fresh Data Pool (EODATA), though it’s less common than formats like .tiff or .SAFE. Commonly used in environmental and Earth sciences, both in research and commercial EODATA systems alike. May require specialized software for reading and processing.

Internally, it is a binary or structured file format that typically includes both image data and metadata about the observation; it usually stores both raw and processed data for specific types of EO sensors or projects.

Downloading .EOF file

To download a .EOF file using AWS CLI, execute the following command:

aws s3 cp s3://eodata/Sentinel-1/AUX/AUX_RESORB/2023/10/03/S1A_OPER_AUX_RESORB_OPOD_20231003T050549_V20231003T010842_20231003T042612.EOF \
 . \
 --endpoint-url https://eodata.data.destination-earth.eu

This command will download the .EOF file to the directory from which the command has been executed. If you want it directed to another directory, replace the dot with the concrete path such as /your/path/to/python_eodata/. For example, to download it to Desktop folder, use

aws s3 cp s3://eodata/Sentinel-1/AUX/AUX_RESORB/2023/10/03/S1A_OPER_AUX_RESORB_OPOD_20231003T050549_V20231003T010842_20231003T042612.EOF \
 ~/Desktop/ \
 --endpoint-url https://eodata.data.destination-earth.eu

The output of the command will confirm whether the download is successful:

../../../../_images/download_image_to_folder.png

The internal format for .EOF files is XML:

../../../../_images/eof_downloaded1.png

.TIFF files in Fresh Data Pool (EODATA) repositories

What is a .TIFF file

tiff or TIFF stands for Tagged Image File Format. It is commonly used in Earth Observation for storing satellite images, especially where high precision is required.

A .tiff file consists of pixel data (images), often with associated geo-referencing data in the file header. The format can store both single-band (grayscale) and multi-band (color) raster data. It is used in a variety of remote sensing applications, including vegetation mapping, land cover classification, and terrain analysis.

Downloading a .TIFF file from Fresh Data Pool (EODATA) repositories

To download a .TIFF file using AWS CLI, execute the following command:

aws s3 cp s3://eodata/Sentinel-1/SAR/SLC/2019/10/13/S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE/measurement/s1b-iw1-slc-vh-20191013t155949-20191013t160014-018459-022c6b-001.tiff . --endpoint-url https://eodata.data.destination-earth.eu

This command will download the .TIFF file to your local directory, as denoted by using the dot as folder designation. Replace that dot with /your/path/to/download/ if you want it in another folder.

The output will display the download status, including the file size in MB:

../../../../_images/download_tiff_image.png

The downloaded grayscale .TIFF file looks like:

../../../../_images/downloaded-tiff-file.png

.SAFE objects in Fresh Data Pool (EODATA) repository

What is a .SAFE format

The .SAFE file format is a directory-based structure typically used by the European Space Agency (ESA) for its Sentinel satellite missions (e.g., Sentinel-1, Sentinel-2). It is not a single file but a directory structure containing metadata, image bands, calibration data, and other auxiliary files necessary for processing the satellite data. Because of that complexity, usually it requires specialized software like the ESA Sentinel Application Platform (SNAP) for proper viewing and processing.

Common files inside a .SAFE directory are:

Manifest file (manifest.safe)

Contains high-level metadata about the product.

Measurement files (e.g., .tiff, .png)

Represent the EO image data.

Metadata files (.xml)

Contain detailed information about the acquisition, including satellite orbit data, processing level, and time stamps.

How to download a .SAFE directory

To download an entire .SAFE directory using AWS CLI, execute the following command:

aws s3 cp --recursive s3://eodata/Sentinel-1/SAR/SLC/2019/10/13/S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE/ \
. --endpoint-url https://eodata.data.destination-earth.eu

This command will recursively download all files within the .SAFE directory to your local directory specified by the dot. To download to another folder, replace the dot with /your/path/to/download/. Replace the path with your own directory.

../../../../_images/download_safe_files.png

Files downloaded from this .SAFE project

Here are the directories downloaded for this .SAFE project:

../../../../_images/ubuntu-success-safe-downloaded.png

File /preview/quick-look.png shows what part of Earth surface has been observed and recorded in .SAFE format:

../../../../_images/quick-look-safe-download1.png

Under the measurement subdirectory, we see the downloaded .TIFF files:

../../../../_images/oct-downloaded-ubuntu-tiff-files.png

Note that .TIFF files are all longer than 1 GB of data.

There is also a PDF file for the .SAFE product:

../../../../_images/pdf-file-downloaded1.png

Eventual problems when downloading .SAFE files

.SAFE format consists of a large number of files, and the main problem is that they can become corrupted during the download. Other issues include incomplete downloads, slow speeds, or API throttling.

To mitigate these issues, ensure you have a stable internet connection and enough storage space. Additionally, the AWS S3 service limits the number of objects per request (1000), so large directories may require multiple requests.

Other common problems include:

  • Firewall or network configuration issues

  • Timeouts

  • Invalid or missing metadata

  • Unrecognized or unsupported file formats

  • Nested directory structure handling

What To Do Next

To access Fresh Data Pool (EODATA) on Destination Earth, you can also use boto3 Python library and s3cmd:

Using s3cmd to obtain Fresh Data Pool (EODATA) on Destination Earth

Using boto3 to access Fresh Data Pool (EODATA) on Destination Earth