How to obtain object storage S3 keys through My DataLake Services

S3 is an object storage service with which you can retrieve data over HTTPS using REST API. To use it, one must have a pair of credentials called secret and access keys. In this article, you will be able to generate these keys through My DataLake Services.

Prerequisites

No. 1 Account

You need a profile on My DataLake Services: How to create a profile on My DataLake Services.

No. 2 Administrative privileges within a project

You need to be a member of a project and have administrative privileges in it. That project must be accepted by an operator of My DataLake Services.

To learn how to create a project, check How to create a project on My DataLake Services.

Alternatively, you can join an existing project and its admin can grant you admin privileges.

Available S3 keys in My DataLake Services

Generally speaking, S3 keys can be used to access

EO data

from Copernicus or other satellites

object storage data

from containers in the cloud, for instance, in an OpenStack environment.

In My DataLake Services, there are services for both of these data types – they are called eodata and s3-object-storage, respectively.

You can use only one or the other, or you can use both at the same time. In this article, we present working with s3-object-storage service only.

Get access and quota for S3 object storage

Assuming this is the very first time you are trying to get access and quota roles for S3 object storage, the general situation will be like this:

Here is how to request access and quota roles for s3-object-storage-access. First click on button Request access in row with s3-object-storage label.

Access roles

For s3-object-storage, there is only one access role:

s3-object-storage-access

Access to S3 object storage.

Although there is only one, you still have to click on Select role circle.

It is mandatory to enter text into field Description of planned activities.

Quota roles

Then select among three quota roles – low, medium and high:

s3-object-storage-low

Low quota for S3 object storage. Maximum 216 buckets and 432,000 bucket objects, maximum user quota 115.96 GB.

s3-object-storage-medium

Medium quota for S3 object storage. Maximum 540 buckets and 1,080,000 bucket objects, maximum user quota 289.91 GB.

s3-object-storage-high

High quota for S3 object storage. Maximum 900 buckets and 1,800,000 bucket objects, maximum user quota 483.18 GB.

Select the quota you would like to have.

It is mandatory to enter text into field Description of planned activities.

Finish with clicking on Request role and you will see a list of your requests.

You can initiate role requests for other services – hda, hook etc. Once you finish requesting them, they will all be shown together as PENDING.

Operator approval

The next phase is waiting for the DEDL operator to approve (or disapprove) your requests. When approved, you will see it in the list of Active roles.

Important

In order to have better control over the available capacity, the operator will autonomously decide on which of the available servers will the object store be installed.

You now have active role for S3 object storage:

List active roles

With option Access -> Active roles from the left side menu, you can see the existing active roles. Here is the situation in which there are active roles for both eodata, hda and s3-object-storage:

List role requests

Another option, Role requests shows the history of requested roles. Here is a situation in which all access and quota roles have been approved by the operator, for eodata and s3-object-storage:

Click on Details button to see the exact details of, say, s3-object-storage-low role:

Getting S3 keys for object storage

Creating S3 keys for object storage is a two step process:

1. open a new account for object storage and

2. obtain S3 credentials for access.

Activate option Account & keys from the main menu on the left side.

Use button Generate account to create a specialized account for access to data in object storage.

Click on Confirm. You will get a chance to copy secret and access keys required for access to the EO data repository.

Access key will be available to you at all times within My DataLake Services but secret key need be copied and stored safely as this will be the only time you will see it.

Deleting the entire object storage account

After click on Delete account, you will have to confirm:

The account will be deleted and you will be given opportunity to create a new one:

The keys associated with the deleted account will be deleted as well. If you create a new account afterwards, the existing key pairs will have to be replaced with a newly generated ones.

How to refresh a secret key

Click on button Refresh to reset a secret key – resetting here means generating another secret key instead of the existing one. You would do that, for example, to stop a compromised code from accessing the S3 object data.

Access key remains the same and only the secret key is generated anew.

How to delete the existing S3 object storage key pair

Click on Account & keys in the left side menu and see the current object storage account and corresponding object storage keys.

You can have only one active S3 key for the object storage. Delete it with a click on Delete button, which leads to a modal screen for confirmation:

Controlling usage of object storage

Option Usage shows the state of resources:

How to change roles

It is possible to change the role you already have for another one that is available for the service. If there was only one role to start with, it is not possible to change it for something else. Concretely, there is only one access role for s3 object storage service, so you cannot change it. However, there are three quota roles and there you can request for a change.

Let’s say that you have decided to start small with s3-object-storage-low and that this is the state you see in option Active roles:

To try to change current quota from s3-object-storage-low to s3-object-storage-medium or s3-object-storage-high, click on Role requests to see all active services and click on Edit access for row s3-object-storage:

You will now be able to choose from the other available quota roles for s3-object-storage service:

There is a total of three quota roles and you already have the low one, so now you can choose from the other two, for medium and high access.

Click on the Select role column and enter some text into field Description of planned activities. The Request role button will become active so click on it. A message will appear in the bottom right corner:

In the list of role requests, a new request to the operator will appear as PENDING:

There are two options now available for that request:

Details

This is the standard option for all requests.

Delete

This is the new option, with which you can delete the request before the operator sees it.

You will have a chance to cancel or confirm:

If you confirm, a message will appear in the bottom right corner of the browser window:

The request will become REJECTED and if you now click on Details, you will see that the reason for rejection is labeled as deleted directly.

If not deleted by the user, the request will in due time appear before the operator, who will approve it or reject it.

In case of approval, the role request will become APPROVED:

Click on Active roles to verify that the quota has changed to high:

If rejected, you will, under Details, see the message that the operator sent you. In this case it is “Not available” but it can be anything else the operator wants you know about the rejection:

How to remove access to a service on My DataLake Services

Once you are approved for access to a service, there is no option within My DataLake Services application to stop being attached to that service. You will have to ask Support for changes of that type.

Similarly, if you want to stop being an owner of a My DataLake Services account, ask the same Support to delete it for you.