Configuration files for s3cmd command

s3cmd can access remote data using S3 protocol. This includes the EODATA repository and object storage cloud.

To connect to the S3 storage, s3cmd uses several parameters, such as access key, secret key, S3 endpoint and so on. During configuration, you can enter these data interactively and the command saves them into a file. The file can then be passed to s3cmd to issue commands on the connection described in it.

If you want to use multiple connections from a single virtual machine (such as the EODATA repository and object storage from Destination Earth cloud), you can store them in multiple files, one file per connection.

This article gives examples of how to create and save these configuration files under various circumstances and some potential problems associated with it. The examples provided are not intended to be executed one after another as part of a workflow. Instead, they serve as examples of different ways in which s3cmd operates.

Prerequisites

No. 1 s3cmd installed

To use s3cmd, it has to be installed first. Here is the necessary information:

How to install s3cmd on Linux

What We Are Going To Cover

  • Initializing the configuration process - creating a configuration file

  • Using –configure on an existing file

  • Executing S3 commands

Initializing the configuration process - creating a configuration file

Saving an s3cmd configuration file is really a two-part process:

  • first, answering questions interactively and

  • second, saving them to the configuration file.

Answering the questions will look similarly to this:

../_images/How-does-s3cmd-store-its-connection-data-01_creodias1.png

The result of this process will be saved to a file.

If you started this configuration process accidentally but with no desire of finishing it at that moment, cancel it by using the key combination CTRL+C.

After having answered all of the configuration questions, you will once again be prompted whether you wish to test the connection and save the data.

In this article, we will only explain different ways of initializing the configuration process, depending on your choice of the location of resulting configuration file.

Details regarding the configuration process can be found in What To Do Next.

Explanation of parameters

Here are the s3cmd parameters used in this article.

-c

Specifies the name and/or location of the configuration file. Note that there is only one dash at the start.

--config

Is used for achieving the same purpose as -c. Note two dashes at the start.

--configure

Creates configuration files. Initiates on-screen procedure and saves data to the file.

You can use either the -c or --config parameters together with --configure.

Make sure that the name and/or location of the file is passed to the shell correctly. Be mindful of any spaces, escape characters, quotation marks etc.

Destination: default file

The default location for s3cmd configuration file is a hidden file named .s3cfg located in your Home directory.

To check what is your Home directory, execute this command:

echo $HOME

For Linux, it will most likely be /home/<USER-NAME> where <USER-NAME> is your user name. On a VM cloud, the home directory will usually be /home/eouser. In this case, the default location of the configuration file will be /home/eouser/.s3cfg

Let’s assume that you want the resulting configuration file to be saved in the default location. In this case, you can initialize the configuration process with this command:

s3cmd --configure

Destination: custom file

If your destination of choice is a custom file, pass its name and/or location to the command under -c parameter. Finish the command with --configure to let s3cmd know that you want to create such a file. Here are the example commands:

File named object-storage-access in your current working directory

s3cmd -c object-storage-access --configure

File named eodata-access in /home/eouser/ directory

s3cmd -c /home/eouser/eodata-access --configure

File named object-storage-access located in the parent directory of your current working directory

s3cmd -c ../object-storage-access --configure

Using –configure on an existing file

The --configure command will always use a file of some kind. If -c or -config parameters are omitted, --configure will use the default location as the target. And vice versa, if you did use -c or -config parameters to designate a file, --configure will try to use it.

Using s3cmd on and existing and valid s3cmd configuration file

If you

  • pass an existing and valid s3cmd configuration file,

  • use --configure and

  • approve of saving the file after the questioning process is finished,

it will change the configuration found in that file.

If you stop the process of answering the questions or do not approve of saving the data, your old answers will be kept intact.

Existing file but not a valid s3cmd configuration file

If you

  • pass an existing file which is not an s3cmd configuration file and

  • use --configure at the same time,

that might lead to undesired results. Therefore, make sure that you are using the correct file name and/or path.

Executing S3 commands

You can execute S3 commands on the connection found in your configuration file. In this article, we will limit ourselves only to ls command, which can, among others, list buckets available to a connection. More details, including other commands, can be found in articles listed in What To Do Next.

Existing and valid s3cmd configuration file - non-default location

Using -c without --configure allows you to execute S3 commands on the connection described in that file. For example, to list buckets available to the configuration described in file eodata-now which is in your current working directory, execute this command:

s3cmd -c eodata-now ls

The output should list buckets available to you, like so:

2017-11-15 10:40  s3://DIAS
2017-11-15 10:40  s3://EODATA

Existing and valid s3cmd configuration file - default location

If you chose the default location for your s3cmd configuration file, you can execute S3 commands on connection described in it without passing its name. For example:

s3cmd ls

Non-existent file

If

  • the file specified does not exist on that location

  • and you do not use --configure,

an error will appear.

As an example, let us assume that file on location /home/eouser/this-file does not exist, but that the following command was executed nevertheless:

s3cmd -c /home/eouser/this-file ls

Instead of a list of buckets, the result will be a series of errors like this:

ERROR: /home/eouser/this-file: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

Existing file, not an s3cmd configuration file

If you have submitted a

  • file that exists but

  • is not a valid s3cmd configuration file and you

  • do not use --configure,

that might also lead to undesired results. Again, make sure that you are using the correct file name and/or path.

This also applies if you did not use -c or --config to specify a file and the file saved under the default location is not an s3cmd configuration file.

What To Do Next

You can use s3cmd to access object storage in the cloud: How to access private object storage using S3cmd or boto3