Configuration files for s3cmd command
s3cmd can access remote data using S3 protocol. This includes the EODATA repository and object storage cloud.
To connect to the S3 storage, s3cmd uses several parameters, such as access key, secret key, S3 endpoint and so on. During configuration, you can enter these data interactively and the command saves them into a file. The file can then be passed to s3cmd to issue commands on the connection described in it.
If you want to use multiple connections from a single virtual machine (such as the EODATA repository and object storage from Destination Earth cloud), you can store them in multiple files, one file per connection.
This article gives examples of how to create and save these configuration files under various circumstances and some potential problems associated with it. The examples provided are not intended to be executed one after another as part of a workflow. Instead, they serve as examples of different ways in which s3cmd operates.
Prerequisites
No. 1 s3cmd installed
To use s3cmd, it has to be installed first. Here is the necessary information:
What We Are Going To Cover
Initializing the configuration process - creating a configuration file
Using –configure on an existing file
Executing S3 commands
Initializing the configuration process - creating a configuration file
Saving an s3cmd configuration file is really a two-part process:
first, answering questions interactively and
second, saving them to the configuration file.
Answering the questions will look similarly to this:
The result of this process will be saved to a file.
If you started this configuration process accidentally but with no desire of finishing it at that moment, cancel it by using the key combination CTRL+C.
After having answered all of the configuration questions, you will once again be prompted whether you wish to test the connection and save the data.
In this article, we will only explain different ways of initializing the configuration process, depending on your choice of the location of resulting configuration file.
Details regarding the configuration process can be found in What To Do Next.
Explanation of parameters
Here are the s3cmd parameters used in this article.
- -c
Specifies the name and/or location of the configuration file. Note that there is only one dash at the start.
- --config
Is used for achieving the same purpose as -c. Note two dashes at the start.
- --configure
Creates configuration files. Initiates on-screen procedure and saves data to the file.
You can use either the -c or --config parameters together with --configure.
Make sure that the name and/or location of the file is passed to the shell correctly. Be mindful of any spaces, escape characters, quotation marks etc.
Destination: default file
The default location for s3cmd configuration file is a hidden file named .s3cfg located in your Home directory.
To check what is your Home directory, execute this command:
echo $HOME
For Linux, it will most likely be /home/<USER-NAME> where <USER-NAME> is your user name. On a VM cloud, the home directory will usually be /home/eouser. In this case, the default location of the configuration file will be /home/eouser/.s3cfg
Let’s assume that you want the resulting configuration file to be saved in the default location. In this case, you can initialize the configuration process with this command:
s3cmd --configure
Destination: custom file
If your destination of choice is a custom file, pass its name and/or location to the command under -c parameter. Finish the command with --configure to let s3cmd know that you want to create such a file. Here are the example commands:
File named object-storage-access in your current working directory
s3cmd -c object-storage-access --configure
File named eodata-access in /home/eouser/ directory
s3cmd -c /home/eouser/eodata-access --configure
File named object-storage-access located in the parent directory of your current working directory
s3cmd -c ../object-storage-access --configure
Using –configure on an existing file
The --configure command will always use a file of some kind. If -c or -config parameters are omitted, --configure will use the default location as the target. And vice versa, if you did use -c or -config parameters to designate a file, --configure will try to use it.
Using s3cmd on and existing and valid s3cmd configuration file
If you
pass an existing and valid s3cmd configuration file,
use --configure and
approve of saving the file after the questioning process is finished,
it will change the configuration found in that file.
If you stop the process of answering the questions or do not approve of saving the data, your old answers will be kept intact.
Existing file but not a valid s3cmd configuration file
If you
pass an existing file which is not an s3cmd configuration file and
use --configure at the same time,
that might lead to undesired results. Therefore, make sure that you are using the correct file name and/or path.
Executing S3 commands
You can execute S3 commands on the connection found in your configuration file. In this article, we will limit ourselves only to ls command, which can, among others, list buckets available to a connection. More details, including other commands, can be found in articles listed in What To Do Next.
Existing and valid s3cmd configuration file - non-default location
Using -c without --configure allows you to execute S3 commands on the connection described in that file. For example, to list buckets available to the configuration described in file eodata-now which is in your current working directory, execute this command:
s3cmd -c eodata-now ls
The output should list buckets available to you, like so:
2017-11-15 10:40 s3://DIAS
2017-11-15 10:40 s3://EODATA
Existing and valid s3cmd configuration file - default location
If you chose the default location for your s3cmd configuration file, you can execute S3 commands on connection described in it without passing its name. For example:
s3cmd ls
Non-existent file
If
the file specified does not exist on that location
and you do not use --configure,
an error will appear.
As an example, let us assume that file on location /home/eouser/this-file does not exist, but that the following command was executed nevertheless:
s3cmd -c /home/eouser/this-file ls
Instead of a list of buckets, the result will be a series of errors like this:
ERROR: /home/eouser/this-file: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.
Existing file, not an s3cmd configuration file
If you have submitted a
file that exists but
is not a valid s3cmd configuration file and you
do not use --configure,
that might also lead to undesired results. Again, make sure that you are using the correct file name and/or path.
This also applies if you did not use -c or --config to specify a file and the file saved under the default location is not an s3cmd configuration file.
What To Do Next
You can use s3cmd to access object storage in the cloud: How to access private object storage using S3cmd or boto3