Sharing S3 buckets between OpenStack projects using bucket policies on Destination Earth

When a dataset is too large to copy for every collaborator, share the bucket instead. An S3 bucket policy keeps the data private while giving selected OpenStack projects exactly the access they need. The bucket owner can allow recipients to list, download, upload, overwrite, or delete objects, either across the whole bucket or only under a selected prefix.

This article shows how the bucket owner creates, applies, tests, updates, and removes policies with s3cmd. It includes ready-to-adapt patterns for read-only access, read/write collaboration, upload-only delivery, several recipient projects, prefix-restricted access, and access limited to a source IP range. Use these patterns as the technical building blocks for controlled sharing between projects.

Prerequisites

No. 1 OpenStack projects in the same cloud

The bucket owner and every recipient must have access to OpenStack projects in the same Destination Earth cloud and use the same S3 endpoint. A bucket policy created for one cloud or site does not grant access through another endpoint.

Before preparing the policy, confirm that the bucket, the owner’s EC2 credentials, the recipient project IDs, and the recipients’ EC2 credentials all belong to the same cloud.

No. 2 Existing private S3 bucket

The bucket owner needs an existing S3 bucket and credentials that provide administrative access to it. The policy is attached to this bucket and controls access granted to projects other than the owning project. Before sharing it, verify that the owner can list the bucket normally:

s3cmd ls s3://MY_SHARED_BUCKET

Replace MY_SHARED_BUCKET with the actual bucket name throughout this article.

No. 3 Project IDs

Obtain the project ID of the bucket-owning project and the project ID of every project that should receive access. A project name is not sufficient because the policy principal is constructed from the immutable project ID. The recipient should provide the ID directly rather than asking the bucket owner to identify a project from its display name.

When the OpenStack CLI is authenticated for the required project, its ID can normally be displayed with:

openstack token issue -f value -c project_id

A tenant administrator can also obtain project IDs through the OpenStack management interface or the OpenStack CLI. Record the values carefully because a policy containing the wrong project ID can be accepted syntactically while still granting no useful access.

No. 4 Configured s3cmd

Configure s3cmd for the S3 endpoint on which the bucket is stored. See Configuration files for s3cmd command.

The bucket owner and each recipient should use their own EC2 access and secret keys. When you need to test the owner and recipient accounts from one computer, keep their settings in separate configuration files, for example ~/.s3cfg-owner and ~/.s3cfg-recipient, instead of repeatedly replacing the credentials in ~/.s3cfg.

You can select a particular configuration file with the -c option:

s3cmd -c ~/.s3cfg-owner ls
s3cmd -c ~/.s3cfg-recipient ls

Limitations and security considerations

The sharing method described here grants access at the project level. It does not create an independent S3 user, group, or role policy for each person, and it does not provide a graphical group-management system inside Cyberduck. The membership of every authorized project must therefore be controlled by the relevant tenant administrator.

A bucket policy is attached to one bucket and the complete policy is maintained as one JSON document. Applying an edited file replaces the previous bucket-policy definition, so statements that must remain active need to be preserved in the new file. Keep the current JSON file in a controlled location, review it before every update, and retain a backup when changing a policy used by several projects.

Avoid broad principals and resources unless public or cloud-wide access is genuinely intended. In particular, combining "Principal": "*" with "arn:aws:s3:::*" can grant far more access than the name of the policy suggests. The examples in this article identify specific projects and actual bucket resources so that the result remains limited and auditable.

An IP-address condition is an additional restriction, not a replacement for identity. Several users may share the same public egress address when they connect through a company network, VPN, proxy, or network address translation, while one user’s public address may also change. For that reason, the IP-restricted pattern in this article combines a project principal with an approved CIDR range instead of granting access to every project connecting from that address.

Ceph supports a subset of the Amazon S3 policy language, and the available actions can depend on the Ceph version and service configuration deployed by the cloud operator. The patterns below deliberately use only actions needed for the described operations. Do not assume that every action documented for Amazon S3 is available or behaves identically on this service.

Understanding a bucket policy

Basic policy structure

A bucket policy is stored in a JSON file. You can create it with a plain-text editor such as nano, vim, or a code editor that can validate JSON syntax. JSON does not support comments, so explanatory text must remain outside the policy file.

The following template shows the main sections:

{
  "Version": "2012-10-17",
  "Id": "POLICY_ID",
  "Statement": [
    {
      "Sid": "STATEMENT_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ACTION"
      ],
      "Resource": [
        "arn:aws:s3:::RESOURCE"
      ],
      "Condition": {
        "CONDITION_OPERATOR": {
          "CONDITION_KEY": "CONDITION_VALUE"
        }
      }
    }
  ]
}

The Condition section is optional and should be omitted when no additional restriction is required. A policy may contain several statements, and this is usually necessary because bucket-level actions and object-level actions use different resource ARNs. Keeping these statements separate also makes the resulting permissions easier to review.

Version: The policy-language version. Use 2012-10-17 for the policies in this article.
Id: A descriptive identifier for the complete policy. It is useful for recognizing the purpose of a stored policy file but does not control access by itself.
Statement: A list of one or more permission statements. Each statement combines a principal, effect, actions, resources, and optional conditions.
Sid: A descriptive identifier for one statement. Choose a name that explains the access being granted, such as recipient-read-objects or contributors-upload.
Effect: Either Allow or Deny. The patterns in this article use Allow and limit its scope through specific principals, actions, resources, and conditions.
Principal: The project or projects to which the statement applies.
Action: The S3 operation or operations permitted by the statement.
Resource: The bucket, objects, or object prefix to which the permitted actions apply.
Condition: An optional additional test, such as requiring the request to come from a specified source IP range.

Project principals

Use the recipient project’s OpenStack project ID in the principal ARN:

"Principal": {
  "AWS": [
    "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
  ]
}

To name several projects, place several project principals in the same array:

"Principal": {
  "AWS": [
    "arn:aws:iam::BUCKET_USER_PROJECT_ID_1:root",
    "arn:aws:iam::BUCKET_USER_PROJECT_ID_2:root",
    "arn:aws:iam::BUCKET_USER_PROJECT_ID_3:root"
  ]
}

Several policy patterns below include BUCKET_OWNER_PROJECT_ID alongside the recipient project IDs, following the convention used in the existing Destination Earth procedure. The owner already controls the bucket, so this entry is not required to preserve owner access. The upload-only and source-IP patterns therefore identify only the recipient project.

Actions used in this article

The patterns use a limited collection of S3 actions rather than presenting a long action catalogue that may not match the deployed Ceph release.

s3:ListBucket: Allows the principal to list object keys in the bucket. This is a bucket-level action and must use the bucket ARN without /*.
s3:GetObject: Allows the principal to download or read objects. This is an object-level action and normally uses BUCKET_NAME/* or a more restricted prefix.
s3:PutObject: Allows the principal to upload a new object or replace an object at an existing key.
s3:DeleteObject: Allows the principal to delete an object. Grant it only when recipients genuinely need deletion rights.
s3:ListBucketMultipartUploads: Allows listing multipart uploads associated with the bucket. It can be needed by clients transferring large objects.
s3:ListMultipartUploadParts: Allows the client to inspect parts belonging to a multipart upload.
s3:AbortMultipartUpload: Allows an incomplete multipart upload to be cancelled and its uploaded parts to be removed.

When recipients must work with historical versions in a version-enabled bucket, additional actions such as s3:GetObjectVersion, s3:ListBucketVersions, or s3:DeleteObjectVersion may be required. Granting access to previous versions is a separate decision from granting access to the current object and should not be added automatically to every sharing policy.

Bucket and object resources

A bucket-level action and an object-level action do not use the same resource ARN. To refer to the bucket itself, use:

arn:aws:s3:::MY_SHARED_BUCKET

To refer to all objects in that bucket, use:

arn:aws:s3:::MY_SHARED_BUCKET/*

To refer only to objects under the published/ prefix, use:

arn:aws:s3:::MY_SHARED_BUCKET/published/*

S3 does not store directories in the same way as a conventional file system. What Cyberduck displays as a folder is normally a shared prefix at the beginning of one or more object keys. A policy that grants access to published/* therefore applies to objects whose keys begin with published/, not to an independent directory object with its own permissions.

Do not use the following broad resource in a normal project-sharing policy:

"Resource": "arn:aws:s3:::*"

It refers to all S3 resources to which the policy interpretation can apply and is far broader than the bucket being shared. Use the actual bucket ARN and object ARN instead.

IP-address conditions

Use the aws:SourceIp condition key with the IpAddress operator to allow requests only from a particular public address or CIDR range:

"Condition": {
  "IpAddress": {
    "aws:SourceIp": "PERMITTED_IP_RANGE"
  }
}

For one address, use a /32 value, for example 198.51.100.24/32. For a network range, use the corresponding CIDR notation, such as 198.51.100.0/24. Determine which public egress address the S3 service sees before relying on this condition, especially when users connect through a VPN or shared organizational network.

Applying and maintaining policies

Validate the JSON file

A missing comma, unmatched bracket, or invalid quotation mark prevents the policy from being parsed. Before applying it, validate the file locally with Python:

python3 -m json.tool POLICY_FILE.json > /dev/null

An empty result means that the JSON syntax is valid. It does not prove that every project ID, action, resource, or condition is correct, so review the actual values separately.

Apply a policy

Apply the complete policy file to the bucket with:

s3cmd setpolicy POLICY_FILE.json s3://MY_SHARED_BUCKET

If the command finishes without an error, inspect the bucket information:

s3cmd info s3://MY_SHARED_BUCKET

Keep the JSON file after applying it because it is the clearest record of what the bucket owner intended to configure. The decisive verification is not the absence of an error from setpolicy, but a successful test using credentials from every authorized recipient project and a failed test for operations that should remain forbidden.

Update a policy

To add a project, remove a project, or change its permissions, edit the existing JSON file and apply the complete file again:

python3 -m json.tool POLICY_FILE.json > /dev/null
s3cmd setpolicy POLICY_FILE.json s3://MY_SHARED_BUCKET

Do not create several unrelated policy files and assume that s3cmd will merge them. Keep all statements that must remain effective in one current policy document, store previous revisions separately, and record why each project was added or removed.

Remove a policy

To remove the bucket policy completely, run:

s3cmd delpolicy s3://MY_SHARED_BUCKET

Removing the policy removes access granted by that policy, but it does not delete the bucket or its objects. After removal, test the bucket using recipient credentials and confirm that the former recipient receives AccessDenied for the previously permitted operations.

Sample bucket policy patterns

The following patterns are technical building blocks rather than complete collaboration or backup workflows. Each one shows the JSON policy, the command used by the bucket owner, and a minimal recipient-side test. Replace every placeholder before applying the file, and use a separate test bucket before introducing a new pattern on production data.

1. Grant read-only access to one recipient project

Use this pattern when another project should be able to list and download the contents of a bucket but must not upload, overwrite, or delete objects. The policy separates the bucket-level ListBucket action from the object-level GetObject action, because the two actions apply to different resource ARNs. This is the usual starting point for distributing a prepared dataset to another project.

Create read-only-policy.json:

{
  "Version": "2012-10-17",
  "Id": "project-read-only",
  "Statement": [
    {
      "Sid": "list-shared-bucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ]
    },
    {
      "Sid": "read-shared-objects",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/*"
      ]
    }
  ]
}

Apply it as the bucket owner:

s3cmd setpolicy read-only-policy.json s3://MY_SHARED_BUCKET

The recipient should then list the bucket and download a known object using credentials belonging to the authorized project:

s3cmd -c ~/.s3cfg-recipient ls s3://MY_SHARED_BUCKET
s3cmd -c ~/.s3cfg-recipient get \
   s3://MY_SHARED_BUCKET/SHARED_OBJECT \
   ./SHARED_OBJECT

An attempt to upload or delete an object should fail with AccessDenied. Test the restriction with an unimportant temporary file rather than an object containing valuable data.

2. Grant read/write access to one recipient project

Use this pattern when the recipient project must participate in a shared working area and can therefore list, download, upload, replace, and delete objects. Because deletion is included, every user whose credentials belong to the authorized project may be able to remove shared data. Apply this policy only when the recipient project is managed as a trusted collaboration boundary and consider enabling versioning before granting delete access.

Create read-write-policy.json:

{
  "Version": "2012-10-17",
  "Id": "project-read-write",
  "Statement": [
    {
      "Sid": "list-shared-bucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ]
    },
    {
      "Sid": "read-write-shared-objects",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/*"
      ]
    }
  ]
}

Apply the policy:

s3cmd setpolicy read-write-policy.json s3://MY_SHARED_BUCKET

The recipient can test the permissions with a temporary object created specifically for this purpose:

printf 'bucket policy test\n' > /tmp/bucket-policy-test.txt

s3cmd -c ~/.s3cfg-recipient put \
   /tmp/bucket-policy-test.txt \
   s3://MY_SHARED_BUCKET/bucket-policy-test.txt

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET/bucket-policy-test.txt

s3cmd -c ~/.s3cfg-recipient get \
   s3://MY_SHARED_BUCKET/bucket-policy-test.txt \
   /tmp/bucket-policy-test-downloaded.txt

s3cmd -c ~/.s3cfg-recipient del \
   s3://MY_SHARED_BUCKET/bucket-policy-test.txt

Do not use an existing production object for the delete test. A successful test demonstrates that the recipient has substantial control over the shared contents, so review project membership before placing important data under this policy.

3. Grant upload-only access to a selected prefix

Use this pattern as an S3 drop box when a recipient should upload files to an assigned prefix without listing, downloading, or deleting objects already stored there. Give each recipient project a separate prefix, such as incoming/BUCKET_USER_PROJECT_ID/, to keep submissions from different projects apart.

This policy is upload-only, but it is not write-once. A recipient who knows an existing object key can upload another object to the same key. Enable versioning when earlier copies must remain recoverable, and use unique file names when submitted objects must not replace one another accidentally.

The bucket owner can inspect and process the submitted objects using the owner’s own credentials.

Create upload-only-policy.json:

{
  "Version": "2012-10-17",
  "Id": "project-upload-only",
  "Statement": [
    {
      "Sid": "upload-to-recipient-prefix",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:PutObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/incoming/BUCKET_USER_PROJECT_ID/*"
      ]
    },
    {
      "Sid": "manage-multipart-upload",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ]
    }
  ]
}

Apply the policy:

s3cmd setpolicy upload-only-policy.json s3://MY_SHARED_BUCKET

The recipient uploads to the assigned prefix using a known destination key:

s3cmd -c ~/.s3cfg-recipient put \
   ./DELIVERY_FILE \
   s3://MY_SHARED_BUCKET/incoming/BUCKET_USER_PROJECT_ID/DELIVERY_FILE

The recipient cannot list the bucket, download submitted objects, or delete them because the policy grants neither s3:ListBucket, s3:GetObject, nor s3:DeleteObject. The recipient can, however, replace an object by uploading again to a known key.

The bucket owner verifies the delivery with:

s3cmd -c ~/.s3cfg-owner ls \
   s3://MY_SHARED_BUCKET/incoming/BUCKET_USER_PROJECT_ID/

Some synchronization workflows list the destination before uploading and will therefore not work with a strict upload-only policy. Use a direct upload command with a known destination key, or design a less restrictive contributor policy when the client must compare remote and local contents.

4. Grant read-only access to several recipient projects

Use this pattern when one owner distributes the same dataset to several collaborating projects. The projects are listed as separate principals in the policy, and all of them receive the same read-only permissions. Adding or removing a recipient requires editing the principal arrays and applying the complete policy again.

Create multi-project-read-only-policy.json:

{
  "Version": "2012-10-17",
  "Id": "multi-project-read-only",
  "Statement": [
    {
      "Sid": "projects-list-shared-bucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_1:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_2:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_3:root"
        ]
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ]
    },
    {
      "Sid": "projects-read-shared-objects",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_1:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_2:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID_3:root"
        ]
      },
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/*"
      ]
    }
  ]
}

Apply the policy:

s3cmd setpolicy \
   multi-project-read-only-policy.json \
   s3://MY_SHARED_BUCKET

Test access separately with credentials from every recipient project. Do not assume that success from the first project proves that all project IDs were entered correctly, because one mistyped principal affects only that recipient.

5. Grant read/write access only to a selected prefix

Use this pattern when recipients should collaborate in one part of a bucket while the rest remains visible only to the owner. The example grants access to keys beginning with shared/ and limits bucket listings to that prefix through the s3:prefix condition. Objects stored under prefixes such as private/ or archive/ are not included in the granted resources.

Create prefix-read-write-policy.json:

{
  "Version": "2012-10-17",
  "Id": "prefix-read-write",
  "Statement": [
    {
      "Sid": "list-shared-prefix",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ],
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "shared",
            "shared/*"
          ]
        }
      }
    },
    {
      "Sid": "read-write-shared-prefix",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_OWNER_PROJECT_ID:root",
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/shared/*"
      ]
    }
  ]
}

Apply the policy:

s3cmd setpolicy \
   prefix-read-write-policy.json \
   s3://MY_SHARED_BUCKET

The recipient should address the permitted prefix explicitly:

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET/shared/

s3cmd -c ~/.s3cfg-recipient put \
   ./COLLABORATION_FILE \
   s3://MY_SHARED_BUCKET/shared/COLLABORATION_FILE

Test a forbidden prefix as well:

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET/private/

The last command should fail or return no authorized listing, depending on how the client and service report the denied request. The bucket owner should also confirm that the recipient cannot retrieve a known object stored outside shared/.

6. Restrict project access to a source IP range

Use this pattern when a recipient project should have read/write access only while requests come through an approved network. The project principal still identifies who is authorized, while the source-IP condition determines where an accepted request may originate. This is safer than setting "Principal": "*" because an unrelated project connecting through the same network does not become authorized by the policy.

Create project-read-write-ip-policy.json:

{
  "Version": "2012-10-17",
  "Id": "project-read-write-from-approved-network",
  "Statement": [
    {
      "Sid": "list-bucket-from-approved-network",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "PERMITTED_IP_RANGE"
        }
      }
    },
    {
      "Sid": "read-write-from-approved-network",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::BUCKET_USER_PROJECT_ID:root"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::MY_SHARED_BUCKET/*"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "PERMITTED_IP_RANGE"
        }
      }
    }
  ]
}

Replace PERMITTED_IP_RANGE with a public IP address in CIDR notation, such as 198.51.100.24/32, or an approved network range. Validate the JSON and apply the policy:

python3 -m json.tool \
   project-read-write-ip-policy.json \
   > /dev/null

s3cmd setpolicy \
   project-read-write-ip-policy.json \
   s3://MY_SHARED_BUCKET

Test the policy from the permitted network and, when possible, from a network outside the permitted range. A test performed only from the approved address proves that the positive path works but does not prove that the condition rejects other source addresses.

Combining policy patterns

A real bucket policy can contain several statements derived from different patterns. For example, one project may receive upload-only access to incoming/, a curator project may receive read/write access to the entire bucket, and several consumer projects may receive read-only access to published/. These permissions should be assembled into one reviewed policy document rather than applied as unrelated files one after another.

Keep every statement narrowly scoped to the principal, actions, and resources required by its participants. Reusing one broad read/write statement for all projects is easier to write, but it removes the separation between contributors, curators, consumers, and backup processes. The separate S3 data-sharing and backup scenarios article should explain how these patterns can be combined into complete operational designs.

Testing access from both sides

Every policy change should be tested with the owner’s credentials and with credentials from each recipient project. The owner should first confirm that normal bucket administration still works, after which the recipient should test all operations that are expected to succeed and at least one operation that is expected to fail. A policy is not fully verified when it merely allows the desired action; it must also continue to deny actions that were intentionally withheld.

A useful test sequence for a read-only policy is:

s3cmd -c ~/.s3cfg-recipient ls \
   s3://MY_SHARED_BUCKET

s3cmd -c ~/.s3cfg-recipient get \
   s3://MY_SHARED_BUCKET/SHARED_OBJECT \
   /tmp/SHARED_OBJECT

s3cmd -c ~/.s3cfg-recipient put \
   /tmp/SHARED_OBJECT \
   s3://MY_SHARED_BUCKET/recipient-should-not-upload

The first two commands should succeed and the upload should fail with AccessDenied. For read/write or upload-only policies, create a disposable object with an unmistakable name, perform only the actions included in the policy, and remove it with the owner’s credentials when the recipient has no delete permission.

Revoking access

When one project should no longer have access but other recipients must remain authorized, remove only that project’s principal from every relevant statement and apply the updated complete policy file. Search the file carefully because the project may appear in several statements controlling listing, reading, writing, multipart uploads, or selected prefixes. Test the removed project after applying the update and confirm that access is denied.

When no external project should retain policy-based access, delete the entire bucket policy:

s3cmd delpolicy s3://MY_SHARED_BUCKET

Revoking a policy does not remove files previously downloaded by recipients, nor does it invalidate copies they created elsewhere. Access revocation controls future requests to the bucket and should therefore be combined with appropriate project-membership management, credential rotation, data-handling agreements, and retention procedures where required.

What To Do Next

After the bucket policy has been applied and tested, continue with the article that matches the next task.

The bucket owner needs to upload or organize the shared data – See Using Cyberduck with S3-compatible object storage on Destination Earth. The owner can use Cyberduck to create folders, upload files, inspect objects, and confirm that the shared bucket contains the expected data.
The recipient needs to open the shared bucket – Use S3 credentials belonging to the authorized recipient project. The shared bucket may not appear in the recipient’s root bucket list because the recipient does not own it. In Cyberduck, enter MY_SHARED_BUCKET in the Path field or create a bookmark that opens the bucket directly. With s3cmd, address the bucket explicitly as s3://MY_SHARED_BUCKET.

See Using Cyberduck with S3-compatible object storage on Destination Earth for graphical access, or How to access object storage using s3cmd for command-line access.

The bucket owner has granted write or delete access – See S3 bucket object versioning before placing important data in the shared area. Versioning can preserve earlier copies after an object is overwritten or deleted, although it does not restrict what an authorized recipient may do.
Several projects need different roles in one workflow – See S3 data-sharing and backup scenarios. It shows how to combine the policy patterns from this article for contributors, curators, read-only consumers, upload-only delivery areas, and backup or recovery access.

When access is no longer required, the bucket owner should return to this article, remove the recipient project from the policy, apply the updated JSON file, and test that the former recipient now receives AccessDenied.

Sharing S3 buckets between OpenStack projects using bucket policies on Destination Earth

Prerequisites

No. 1 OpenStack projects in the same cloud

No. 2 Existing private S3 bucket

No. 3 Project IDs

No. 4 Configured s3cmd

How project-level sharing works

Limitations and security considerations

Understanding a bucket policy

Basic policy structure

Project principals

Actions used in this article

Bucket and object resources

IP-address conditions

Applying and maintaining policies

Validate the JSON file

Apply a policy

Update a policy

Remove a policy

Sample bucket policy patterns

1. Grant read-only access to one recipient project

2. Grant read/write access to one recipient project

3. Grant upload-only access to a selected prefix

4. Grant read-only access to several recipient projects

5. Grant read/write access only to a selected prefix

6. Restrict project access to a source IP range

Combining policy patterns

Testing access from both sides

Revoking access

What To Do Next