What is DEFAIR?
DEFAIR is a Python library that reads satellite data and turns it into files machine learning tools can use.
EUMETSAT publishes its data as products. A product is one satellite data file, or one package of files, for one instrument and one time. A Meteosat scan, a Metop orbit and a Sentinel-3 package are stored differently, and each needs its own code to open. DEFAIR opens them all the same way.
One call reads a product and gives you an xarray dataset, the standard Python container for labelled arrays. From there you can crop it, move it onto a different map grid, combine it with data from another satellite, and save it in a format built for large arrays.
Use DEFAIR from Python, or write the job in a settings file and run it from the command line.
DEFAIR is a Python library that you install with pip from PyPI and run in your own Python environment. It is not a managed DEDL or EUMETSAT service.
How the pieces fit
Product file → source plugin → reader plugin → Dataset → transformations → writer
A source fetches the bytes, from a local disk or from cloud storage. A reader understands the product and builds the dataset. Transformations change the dataset. A writer saves it. Each of these is a plugin, a piece of code DEFAIR loads by name, so you pick one by naming it.
What DEFAIR reads and writes
DEFAIR includes 50 readers, covering MSG, MTG, Metop and Sentinel-3 products from EUMETSAT. Writers save to Zarr (a storage format that keeps a large array as many small files, so a program can read one piece without loading all of it), NetCDF, HDF5, Parquet, CSV, GeoTIFF, PNG and JPEG.
Data is not read until you ask for it. DEFAIR uses Dask, the Python library that lets you work on arrays too large for memory, so a full-disk image or a month of scans can go through a pipeline on a laptop.
What to do next
Continue with Key concepts to learn how readers, transformations, writers and workflows fit together. For the supported products and where to download them, consult the reader catalogue.