Using AWS CLI to access Fresh Data Pool on Destination Earth

In this article, you will learn how to access Fresh Data Pool on Destination Earth Data Lake, using AWS CLI.

Fresh Data Pool, also known as EODATA (Earth Observation Data), includes satellite imagery and environmental data. With AWS CLI, you can interact with AWS S3 to retrieve and manage these large datasets directly from the command line.

What we are going to cover

EODATA Storage endpoint

S3 is an object storage service with which you can retrieve data over HTTPS using REST API. The default S3 endpoint address to work with EODATA on Destination Earth is:

https://eodata.data.destination-earth.eu

Prerequisites

No. 1 Access to My DataLake Services

You need access to My DataLake Services. See How to create profile on My DataLake Services.

No. 2 Credentials for S3 EODATA access from My DataLake Services

See article How to obtain eodata S3 keys through My DataLake Services

No. 3 AWS CLI installed on your local computer or virtual machine

AWS CLI is a free and open source software which can manage different clouds, not only those hosted by Amazon Web Services. In this article, you will use it to control your resources hosted on Destination Earth cloud.

This article was written for Ubuntu 22.04. The commands may work on other operating systems, but might require adjusting.

Here is how to install AWS CLI on Ubuntu 22.04:

sudo apt install awscli

You can browse the EODATA repository not only using Python’s boto3, but also with the AWS Command Line Interface (CLI), which is convenient for quick checks.

Before using the AWS CLI, ensure that:

  • The AWS CLI is installed (version 2 is recommended).

  • Your credentials are properly configured.

To configure the CLI, run:

aws configure

Enter the following when prompted:

AWS Access Key ID [None]: YOUR_ACCESS_KEY
AWS Secret Access Key [None]: YOUR_SECRET_KEY
Default region name [None]: eu-central-1
Default output format [None]: json

Exploring EODATA with aws s3 ls

Use the aws s3 ls command to explore directories. The example below lists the contents of a specific directory:

aws s3 ls s3://DIAS/Envisat-ASAR/ASAR/ASA_WSS_1P/2012/04/08/ \
  --endpoint-url https://eodata.data.destination-earth.eu

This command will display a list of subdirectories or files:

PRE ASA_WSS_1PNESA20120408_110329_000000603113_00267_52867_0000.N1/
PRE ASA_WSS_1PNESA20120408_110428_000000603113_00267_52867_0000.N1/
PRE ASA_WSS_1PNESA20120408_110446_000000603113_00267_52867_0000.N1/

Listing the root directory

To explore the root of the EODATA repository:

aws s3 ls s3://DIAS/ --endpoint-url https://eodata.data.destination-earth.eu

This is the result:

../../../../_images/entire-repository.png

Path format guidelines

When using aws s3 ls, follow these rules for specifying paths:

  • Use / as a separator between directories.

  • Do not start the path with a leading slash.

  • To list the contents of a folder, always end the path with a slash.

  • The first part of the path must be a dataset available at the root level (e.g. Sentinel-1, Sentinel-2, Envisat-ASAR).

Optional: Filtering output in Bash

You can extract just the directory names from the listing using:

aws s3 ls s3://DIAS/Envisat-ASAR/ASAR/ASA_WSS_1P/2012/04/08/ \
  --endpoint-url https://eodata.data.destination-earth.eu \
  | awk '{print $NF}'

This command returns:

ASA_WSS_1PNESA20120408_110329_000000603113_00267_52867_0000.N1/
ASA_WSS_1PNESA20120408_110428_000000603113_00267_52867_0000.N1/
ASA_WSS_1PNESA20120408_110446_000000603113_00267_52867_0000.N1/

PNG files from an EODATA repository

Basic info on PNG files

PNG stands for Portable Network Graphics. It is a raster image format known for:

  • Storing pixel-based image data,

  • Supporting transparency,

  • Providing lossless compression.

In Earth Observation (EO) data, .png files are often used to represent processed imagery, such as false-color composites or classification maps. These files are typically not geo-referenced like .tif or .SAFE files, but they provide quick, visual access to EO content.

How to download a PNG file with AWS CLI

To download a .PNG file using the AWS CLI, use the command:

aws s3 cp s3://DIAS/Landsat-5/TM/L1T/2011/11/16/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1.BP.PNG \
  . \
  --endpoint-url https://eodata.data.destination-earth.eu

This command downloads the file to your current working directory. If the file already exists locally, it will be overwritten without prompting for confirmation.

Replace the file path above with the full key to the PNG file you want to download.

Explanation of the command

  • aws s3 cp: Copies a file from an S3 bucket to your local system.

  • s3://DIAS/…: The full key to the PNG file in the EODATA repository.

  • .: Target location (current directory).

  • --endpoint-url: Specifies the EODATA endpoint for AWS S3-compatible access.

After running the command, you will see a download progress bar if the terminal supports it. Once completed, the file will be available in your current directory.

Expected output

The downloaded PNG file should appear like this in your terminal (on Linux):

../../../../_images/some_other_image_for_output.png

Here is what the downloaded image may look like:

../../../../_images/LS05_RMPS_TM__GTC_1P_20111116T100042_20111116T100111_147386_0194_0035_4BF1.BP1.PNG

Example of a downloaded PNG visualization from EODATA

EOF files from an EODATA repository

What is a .EOF file

eof is short for Earth Observation Format. It is a custom format used to store and share EODATA, though it’s less common than formats like .tiff or .SAFE. Commonly used in environmental and Earth sciences, both in research and commercial EODATA systems alike. May require specialized software for reading and processing.

Internally, it is a binary or structured file format that typically includes both image data and metadata about the observation; it usually stores both raw and processed data for specific types of EO sensors or projects.

Downloading .EOF file

To download a .EOF file using AWS CLI, execute the following command:

aws s3 cp s3://eodata/Sentinel-1/AUX/AUX_RESORB/2023/10/03/S1A_OPER_AUX_RESORB_OPOD_20231003T050549_V20231003T010842_20231003T042612.EOF \
 . \
 --endpoint-url https://eodata.data.destination-earth.eu

This command will download the .EOF file to the directory from which the command has been executed. If you want it directed to another directory, replace the dot with the concrete path such as /your/path/to/python_eodata/. For example, to download it to Desktop folder, use

aws s3 cp s3://eodata/Sentinel-1/AUX/AUX_RESORB/2023/10/03/S1A_OPER_AUX_RESORB_OPOD_20231003T050549_V20231003T010842_20231003T042612.EOF \
 ~/Desktop/ \
 --endpoint-url https://eodata.data.destination-earth.eu

The output of the command will confirm whether the download is successful:

../../../../_images/download_image_to_folder.png

The internal format for .EOF files is XML:

../../../../_images/eof_downloaded1.png

.TIFF files in EODATA repositories

What is a .TIFF file

tiff or TIFF stands for Tagged Image File Format. It is commonly used in Earth Observation for storing satellite images, especially where high precision is required.

A .tiff file consists of pixel data (images), often with associated geo-referencing data in the file header. The format can store both single-band (grayscale) and multi-band (color) raster data. It is used in a variety of remote sensing applications, including vegetation mapping, land cover classification, and terrain analysis.

Downloading a .TIFF file from EODATA repositories

To download a .TIFF file using AWS CLI, execute the following command:

aws s3 cp s3://eodata/Sentinel-1/SAR/SLC/2019/10/13/S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE/measurement/s1b-iw1-slc-vh-20191013t155949-20191013t160014-018459-022c6b-001.tiff . --endpoint-url https://eodata.data.destination-earth.eu

This command will download the .TIFF file to your local directory, as denoted by using the dot as folder designation. Replace that dot with /your/path/to/download/ if you want it in another folder.

The output will display the download status, including the file size in MB:

../../../../_images/download_tiff_image.png

The downloaded grayscale .TIFF file looks like:

../../../../_images/downloaded-tiff-file.png

.SAFE objects in EODATA repository

What is a .SAFE format

The .SAFE file format is a directory-based structure typically used by the European Space Agency (ESA) for its Sentinel satellite missions (e.g., Sentinel-1, Sentinel-2). It is not a single file but a directory structure containing metadata, image bands, calibration data, and other auxiliary files necessary for processing the satellite data. Because of that complexity, usually it requires specialized software like the ESA Sentinel Application Platform (SNAP) for proper viewing and processing.

Common files inside a .SAFE directory are:

Manifest file (manifest.safe)

Contains high-level metadata about the product.

Measurement files (e.g., .tiff, .png)

Represent the EO image data.

Metadata files (.xml)

Contain detailed information about the acquisition, including satellite orbit data, processing level, and time stamps.

How to download a .SAFE directory

To download an entire .SAFE directory using AWS CLI, execute the following command:

aws s3 cp --recursive s3://eodata/Sentinel-1/SAR/SLC/2019/10/13/S1B_IW_SLC__1SDV_20191013T155948_20191013T160015_018459_022C6B_13A2.SAFE/ \
. --endpoint-url https://eodata.data.destination-earth.eu

This command will recursively download all files within the .SAFE directory to your local directory specified by the dot. To download to another folder, replace the dot with /your/path/to/download/. Replace the path with your own directory.

../../../../_images/download_safe_files.png

Files downloaded from this .SAFE project

Here are the directories downloaded for this .SAFE project:

../../../../_images/ubuntu-success-safe-downloaded.png

File /preview/quick-look.png shows what part of Earth surface has been observed and recorded in .SAFE format:

../../../../_images/quick-look-safe-download1.png

Under the measurement subdirectory, we see the downloaded .TIFF files:

../../../../_images/oct-downloaded-ubuntu-tiff-files.png

Note that .TIFF files are all longer than 1 GB of data.

There is also a PDF file for the .SAFE product:

../../../../_images/pdf-file-downloaded1.png

Eventual problems when downloading .SAFE files

.SAFE format consists of a large number of files, and the main problem is that they can become corrupted during the download. Other issues include incomplete downloads, slow speeds, or API throttling.

To mitigate these issues, ensure you have a stable internet connection and enough storage space. Additionally, the AWS S3 service limits the number of objects per request (1000), so large directories may require multiple requests.

Other common problems include:

  • Firewall or network configuration issues

  • Timeouts

  • Invalid or missing metadata

  • Unrecognized or unsupported file formats

  • Nested directory structure handling

What To Do Next

To access EODATA on Destination Earth, you can also use boto3 Python library and s3cmd:

Using s3cmd to access Fresh Data Pool on Destination Earth

Using boto3 to access Fresh Data Pool on Destination Earth