Load data
The DEFAIR function call Dataset.from_source() reads a product from a local disk or from cloud storage and returns a Dataset. This section covers how the reader is chosen, the options readers take, reading from S3 (cloud object storage), reading many files as one dataset, and streaming a list of files too long to hold at once.
Three kinds of plugin take part. A plugin is a piece of code DEFAIR loads by name.
A source plugin finds and opens the path (
local,s3,s3botoorhda).A reader plugin understands the product and returns xarray arrays that are read only when needed. xarray is the standard Python container for labelled arrays.
For a long list of files, a streaming plan reads a fixed number of scene groups at a time and appends each group to Zarr. A scene group is one file, or the several files that make up one scene.
Choosing an S3 source does not turn on streaming, and streaming=True
does not change which S3 source is used.
Prerequisites
No. 1 Installed packages and products you can read
Install DEFAIR as described in Install DEFAIR, and use the reader catalogue to find the reader and download location for your product. Replace the example paths with local files or remote objects you can read.
No. 2 Credentials for remote reads, when used
For private S3 data, obtain the storage service’s endpoint URL and credentials with permission to read the required objects. If you use an example that lists files, those credentials must also allow object listing. For DEDL standalone S3 storage, follow How to obtain S3 keys through My DataLake Services. Follow Credentials and endpoints to configure S3 access or one of the HDA authentication options described below.
For Harmonized Data Access (HDA), DEFAIR accepts the username and password of a DestinE Platform (DESP) account, or a service-account client ID and secret. DEFAIR uses these credentials to obtain an access token: a temporary credential for authenticating requests to HDA. You can instead supply a token you have already obtained. If you supply only a token, DEFAIR cannot renew it, so it must remain valid until all required reads are complete.
For an HDA read, you also need a product URL that your account is authorized to access. For restricted collections, follow How to request HDA roles on My DataLake Services to obtain the required access and quota roles. Allow enough free space in the system’s temporary folder for the complete product, because DEFAIR downloads it before reading its contents.
Choose a data-loading task
What to do next
Start with Read individual products to open a file and choose reader options. For products in object storage, follow Read from S3 to select a source plugin and configure access.