Edge Services - Big Data Processing Services
Note
Edge Services are reserved for policy and decision makers. Access is granted by a governance board using eligibility and approval processes.
Access requests can be made by following the instructions in Access DestinE Edge Services
The DestinE Data Lake (DEDL) offers Big Data Processing Services (also called Edge Services) such as:
Islet Compute Service (IaaS / PaaS)
The Destination Earth Data Lake Islet Compute Service (IaaS/PaaS) enables near data processing activities in vicinity of the data from the “DestinE Data Portfolio”.
The Islet Service is a cloud computing service based on OpenStack that provides users with Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS) solutions, allowing users to manage and deploy virtualised workloads on a scalable and flexible architecture. Virtual resources (e.g. Virtual Machines or Object Storage) can be managed by users using the Horizon GUI, API or CLI. In addition to Virtual Machines (VMs), Kubernetes can be used to deploy containerised workloads.
Islet Storage Service (S3)
The Destination Earth Data Lake Islet Storage Service (S3) enables near data processing activities in the vicinity of the data from the “DestinE Data Portfolio”.
The Service provides Object Storage (S3) to store user’s data and processing results from other services of DestinE Data Lake (Hook, Stack, HDA).
Stack Service (SaaS – Jupyterlab / DASK / OpenDataCube)
The Destination Earth Data Lake Stack Service (SaaS) enables near data processing activities in vicinity of the data from the “DestinE Data Portfolio”. The Stack Service of the Destination Earth Data Lake provides solutions based on JupyterHub, Dask/Dask Gateway and Open Data Cube to facilitate processing in vicinity of the data from the “DestinE Data Portfolio”. This service can be combined with the Destination Earth Data Lake Islet Storage service for convenient saving of processing results in user’s S3 buckets.
Hook Service (FaaS - Workflows)
The Destination Earth Data Lake Hook Service (FaaS) enables near data processing activities in vicinity of the data from the “DestinE Data Portfolio”. The Hook service provides capabilities to execute high level pre-defined functions (Function-as-a-Service) that users can invoke in vicinity of the data from the “DestinE Data Portfolio”. The data harvest workflow will allow users to harvest data from a federated data provider a priori of planned data processing or analysis. The harvested data will be stored inside the DestinE Data Lake for fast and efficient processing. This service can be combined with the Destination Earth Data Lake Islet Storage service for convenient saving of processing results in user’s S3 buckets.