S3 data-sharing and backup scenarios

Large S3 datasets can be shared in two ways: keep the data in its current bucket and grant access to it, or copy the objects to another bucket or storage service. The correct choice depends on where the data should remain, who needs access, and whether a separate copy is required.

Use a bucket policy when data remains in OpenStack Object Storage and another OpenStack project needs controlled access. Copy the objects when transferring data between OpenStack Object Storage and standalone S3 object , delivering data to another project, or creating an independent backup.

This article presents complete scenarios for project-to-project sharing, temporary delivery, cross-service transfers, upload-only collection, shared working areas, backup, and recovery. Each scenario identifies who performs the work, which credentials are used, and which detailed procedure to follow.

Choosing the correct workflow

Choose the workflow according to where the data should remain and what the recipient needs to do.

  • Keep the data in its current OpenStack bucket – Grant another OpenStack project access through a bucket policy. The recipient works with the original objects without creating another copy.

  • Give temporary access to one private object – Create a signed URL. The recipient can download the object until the link expires, without receiving access to the complete bucket.

  • Place a separate copy in another bucket or S3 service – Read the data from the source with one credential pair and upload it to the destination with another.

  • Collect files without exposing previous submissions – Grant upload-only access to a separate prefix assigned to each contributor project.

  • Create a shared working area – Grant read/write access only to a selected prefix while keeping the rest of the bucket private.

  • Protect against accidental overwrite or deletion – Enable versioning on the bucket that must preserve earlier object states.

  • Maintain an independent backup – Copy the objects to another bucket, OpenStack project, or storage service and repeat the transfer according to the required schedule.

Sharing and copying are not the same operation. Sharing keeps one authoritative copy and grants another project access to it. Copying creates another set of objects under separate ownership, credentials, policies, and storage settings.

Prerequisites

No. 1 Access to the required storage services

You need access to every storage service involved in the selected workflow.

  • OpenStack Object Storage – The buckets and EC2 credentials belong to an OpenStack project in an Islet Compute environment.

  • standalone S3 Object Service – The buckets and S3 credentials are managed through My DataLake Services and do not use the OpenStack EC2 credential pair.

For a transfer between the two services, the person performing the complete transfer must be authorized to read from the source and write to the destination. Do not exchange permanent access keys between colleagues merely to complete a transfer. When one person does not have access to both services, divide the work so that the source owner downloads or exports the data and the recipient uploads it with their own credentials.

No. 2 S3 credentials for each connection

Each connection requires an endpoint, access key, and secret key belonging to the same storage service and site.

Do not place credentials from both services in one configuration file. Keep separate files such as:

~/.s3cfg-openstack
~/.s3cfg-storage-only

No. 3 Configured S3 client

Use Cyberduck for manual graphical operations or s3cmd for repeatable transfers, policy management, and command-line verification.

For graphical access, see Using Cyberduck with S3-compatible object storage on Destination Earth.

For s3cmd configuration, see Configuration files for s3cmd command.

For general s3cmd access, see How to access object storage using s3cmd.

Test each configuration before starting a transfer:

s3cmd -c ~/.s3cfg-openstack ls
s3cmd -c ~/.s3cfg-storage-only ls

A successful connection to one service does not confirm that the other endpoint or credential pair is correct.

No. 4 Temporary disk space for cross-service transfers

The procedures in this article use a local staging folder when objects must be copied between separate S3 services. The computer or virtual machine performing the transfer must have enough free disk space for the selected objects.

For very large datasets, transfer and verify smaller prefixes or batches instead of downloading the entire bucket at once. Do not remove the source data until the destination copy has been verified.

Scenario 1: Share an OpenStack bucket with another project

Use this scenario when the data should remain in its existing OpenStack bucket and another OpenStack project in the same cloud needs continuing access.

The main participants are:

Bucket owner

Controls the bucket and applies, updates, tests, and removes the bucket policy.

Recipient project

Receives the permissions defined in the policy.

Recipient

Uses EC2 credentials belonging to the authorized recipient project.

Tenant administrator

Creates projects and controls their membership when access must be limited to a selected group of users.

The recipient first provides the immutable OpenStack project ID to the bucket owner. The owner then selects the required access model: read-only, read/write, upload-only, access to a selected prefix, or another pattern supported by the bucket-policy procedure.

Follow Sharing S3 buckets between OpenStack projects using bucket policies on Destination Earth to create, validate, apply, and test the policy.

After applying the policy, the owner confirms that normal bucket administration still works. The recipient then tests every operation that should succeed and at least one operation that should remain forbidden.

A shared bucket may not appear automatically in the recipient’s root bucket list because the recipient does not own it. With s3cmd, address it explicitly:

s3cmd -c ~/.s3cfg-recipient ls s3://MY_SHARED_BUCKET

In Cyberduck, enter the bucket name in the Path field or save a bookmark that opens the shared bucket directly.

When the policy grants write, overwrite, or delete access, enable versioning before placing important objects in the shared area. Versioning does not restrict an authorized project, but it can preserve an earlier object state after an accidental change.

Scenario 2: Give temporary access to one private object

Use a signed URL when a recipient needs one private object for a limited period but does not need continuing access to the bucket.

The bucket owner opens the object in Cyberduck, selects Copy URL, chooses Signed URL, and sets the required validity period. Test the complete link in a private browser window before sending it.

Anyone who obtains the signed URL can use it until it expires. Send it only through an appropriate communication channel and avoid unnecessarily long validity periods.

A signed URL allows the recipient to download the object. It does not automatically copy the object into the recipient’s S3 storage. To store it in another bucket, the recipient downloads the object and then uploads it with credentials belonging to the destination service.

Use a bucket policy instead when the recipient needs several objects, recurring access, bucket listings, or upload permission.

Scenario 3: Copy data from standalone S3 object service to OpenStack Object Storage

Use this scenario when data stored through My DataLake Services must be copied into an OpenStack project’s S3 bucket.

The transfer uses two separate connections:

Source:
  Standalone S3 object service endpoint
  Storage ONLY access key
  Storage ONLY secret key

Destination:
  OpenStack S3 endpoint
  OpenStack EC2 access key
  OpenStack EC2 secret key

First verify that the source configuration can list the required bucket and that the destination configuration can list or create objects in the target bucket:

s3cmd -c ~/.s3cfg-storage-only ls \
   s3://SOURCE_BUCKET

s3cmd -c ~/.s3cfg-openstack ls \
   s3://DESTINATION_BUCKET

Create a local staging folder:

mkdir -p ~/s3-transfer/storage-only-to-openstack

Download the required source prefix:

s3cmd -c ~/.s3cfg-storage-only sync \
   s3://SOURCE_BUCKET/SOURCE_PREFIX/ \
   ~/s3-transfer/storage-only-to-openstack/

Replace SOURCE_BUCKET and SOURCE_PREFIX with the actual source location. To copy the complete bucket, omit SOURCE_PREFIX but retain the trailing slash after the bucket name.

After the download finishes, inspect the transfer result and confirm that the expected files are present locally. Then upload them to the OpenStack destination:

s3cmd -c ~/.s3cfg-openstack sync \
   ~/s3-transfer/storage-only-to-openstack/ \
   s3://DESTINATION_BUCKET/DESTINATION_PREFIX/

The command creates a separate copy. It does not transfer the source bucket policy, project membership, signed URLs, versioning configuration, or lifecycle rules. Configure the destination bucket separately according to its purpose.

Verify the destination before removing the local staging files or changing the source data.

Scenario 4: Copy data from OpenStack Object Storage to Standalone S3 Object Service

Use this scenario when results produced in an OpenStack project must be delivered to a Storage ONLY bucket or retained outside the Islet Compute project.

The source is read with OpenStack EC2 credentials, while the destination is written with the S3 keys generated through My DataLake Services.

Test both connections:

s3cmd -c ~/.s3cfg-openstack ls \
   s3://SOURCE_BUCKET

s3cmd -c ~/.s3cfg-storage-only ls \
   s3://DESTINATION_BUCKET

Create a separate staging folder:

mkdir -p ~/s3-transfer/openstack-to-storage-only

Download the source objects:

s3cmd -c ~/.s3cfg-openstack sync \
   s3://SOURCE_BUCKET/SOURCE_PREFIX/ \
   ~/s3-transfer/openstack-to-storage-only/

Upload them to Storage ONLY:

s3cmd -c ~/.s3cfg-storage-only sync \
   ~/s3-transfer/openstack-to-storage-only/ \
   s3://DESTINATION_BUCKET/DESTINATION_PREFIX/

Verify the destination copy before deleting either the local staging files or the OpenStack source.

Standalone S3 object service can also be used as a private delivery or review location for user-generated data proposed for inclusion in the DestinE Data Lake.

See Promote user data to become DestinE data for the corresponding data-promotion procedure.

Scenario 5: Collect submissions through upload-only prefixes

Use this scenario when several projects must submit data without browsing or downloading files already delivered by other contributors.

Create a separate prefix for each recipient project:

incoming/PROJECT_A/
incoming/PROJECT_B/
incoming/PROJECT_C/
reviewed/
published/

The bucket owner applies an upload-only policy to each project and limits its s3:PutObject permission to the assigned prefix. Use the upload-only policy pattern from Sharing S3 buckets between OpenStack projects using bucket policies on Destination Earth.

A contributor uploads to a known destination key:

s3cmd -c ~/.s3cfg-contributor put \
   ./DELIVERY_FILE \
   s3://MY_SHARED_BUCKET/incoming/PROJECT_A/DELIVERY_FILE

The contributor cannot list the bucket, download submitted objects, or delete them unless the policy grants those actions separately. The bucket owner verifies and processes the delivery with the owner’s credentials.

Upload-only does not mean write-once. A contributor who knows an existing object key may upload another object to the same key. Use unique file names and enable versioning when earlier submissions must remain recoverable.

When the submission period ends, remove the contributor projects from the bucket policy and confirm that further uploads fail with AccessDenied.

Scenario 6: Create a shared collaboration prefix

Use this scenario when collaborators need read/write access to one part of a bucket while the owner keeps other data private.

A typical bucket structure is:

shared/
private/
archive/

Grant the recipient project permission to list, read, upload, overwrite, and delete objects only under shared/. The bucket-listing statement should use an s3:prefix condition, while the object statement should refer to:

arn:aws:s3:::MY_SHARED_BUCKET/shared/*

Use the prefix-restricted read/write pattern from Sharing S3 buckets between OpenStack projects using bucket policies on Destination Earth.

After applying the policy, the recipient tests the permitted prefix:

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET/shared/

The recipient should also test a forbidden prefix:

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET/private/

The second command should fail or return no authorized listing. The owner should also confirm that the recipient cannot retrieve a known object outside shared/.

Enable versioning when collaborators can overwrite or delete important objects. Review recipient-project membership regularly because every user whose credentials belong to that project may receive the access granted by the policy.

Scenario 7: Maintain a second S3 copy as a backup

Versioning and backup protect against different risks. Versioning preserves earlier object states inside the same bucket. A backup creates another copy in a different bucket, OpenStack project, or storage service.

A second bucket in the same project can protect against accidental changes to the working prefix, but it does not provide strong separation from project-wide credential misuse, project deletion, or a storage-service failure. A bucket in another project or another storage service provides a more independent copy.

Choose the destination according to the required level of separation:

  • Another bucket in the same OpenStack project – Simplest to manage, but uses the same project credentials and service.

  • A bucket in another OpenStack project – Separates ownership and credentials but requires controlled access to both projects during the transfer.

  • Standalone S3 Object Service – Uses separate credentials managed through My DataLake Services and can provide a copy outside the OpenStack project.

Perform the initial full copy with a local staging folder or another tested transfer method. For example:

mkdir -p ~/s3-backup

s3cmd -c ~/.s3cfg-source sync \
   s3://SOURCE_BUCKET/SOURCE_PREFIX/ \
   ~/s3-backup/

s3cmd -c ~/.s3cfg-backup sync \
   ~/s3-backup/ \
   s3://BACKUP_BUCKET/BACKUP_PREFIX/

Run the same transfer again when the backup must be updated. By default, s3cmd sync copies new and changed objects but does not remove destination objects merely because they disappeared from the source.

Do not add --delete-removed unless the destination is intentionally maintained as an exact mirror and deletions in the source must also be propagated. A mirror is not a historical backup when every accidental source deletion is immediately repeated at the destination.

Record the transfer result and investigate incomplete or failed objects. Periodically restore a representative object from the backup and confirm that it is readable.

Scenario 8: Recover after an overwrite or deletion

Use this scenario when versioning was enabled before an object was replaced or deleted.

The bucket owner performs the recovery:

  1. List the available versions or delete markers.

  2. Identify the version that should become available again.

  3. Download or restore the selected version.

  4. Confirm that the current object contains the expected data.

  5. Review whether another project’s write or delete permission should remain active.

Follow S3 bucket object versioning for the complete recovery procedure.

Versioning cannot recover data that was deleted before versioning was enabled, and it does not protect against deletion of every stored version by a principal that has permission to remove versions. Maintain an independent backup when the data requires protection beyond the current bucket.

Verifying a sharing or transfer workflow

Do not consider a workflow complete merely because a command finished without an error. Verify both the expected access and the expected restrictions.

For bucket sharing:

  • Confirm that the owner can still administer the bucket.

  • Test every permitted operation with recipient credentials.

  • Test at least one operation that should remain forbidden.

  • Confirm that the recipient cannot access prefixes outside the policy scope.

  • Test every recipient project separately.

For copied data:

  • Compare the expected object names and relative paths.

  • Compare object counts and sizes.

  • Download and open at least one representative destination object.

  • Compare checksums when both services expose values that can be compared reliably.

  • Confirm that no required object was skipped because of an interrupted transfer.

  • Keep the source and local staging copy until the destination has been verified.

An S3 ETag is not always the same as a local MD5 checksum, especially for multipart uploads. Do not treat a differing ETag as proof of corruption without first determining how the object was uploaded.

Revoking access and cleaning up

The cleanup procedure depends on the workflow.

  • OpenStack bucket sharing – Remove the recipient project from every relevant statement, apply the complete updated policy, and confirm that the former recipient receives AccessDenied.

  • Signed URL – Allow the link to expire. If access must stop immediately, replace or remove the object according to the service’s supported behavior.

  • Upload-only delivery – Remove contributor projects from the policy after the submission period closes.

  • Cross-service transfer – Remove temporary local files only after the destination copy has been verified.

  • Backup – Retain or delete old copies according to the agreed retention policy. Do not remove historical copies merely to make the backup match the current source unless an exact mirror is the intended design.

  • Credentials – Rotate any access key that was exposed, shared improperly, or stored on a computer that should no longer perform the workflow.

What To Do Next

Continue with the article that provides the detailed procedure for the next task.