S3cmd: FAQ and Knowledge Base

Tips, Tricks and More

About the s3cmd configuration file  CloudFront support in s3cmd  Enforcing server-side encryption for all objects in a bucket  How can I remove a bucket that is not empty?  How to configure s3cmd for alternative S3 compatible services  How to restrict access to a bucket to specific IP addresses  How to throttle bandwidth in s3cmd  Why doesn't 's3cmd sync' support PGP / GPG encryption for files?

The s3cmd configuration file is named .s3cfg and it is located in the user's home directory, e.g. /home/username/ ($HOME).

On Windows the configuration file is called s3cmd.ini and it is located in %USERPROFILE% -> Application Data , usually that is c:\users\username\AppData\Roaming\s3cmd.ini


The s3cmd configuration file contains all s3cmd settings. This includes the Amazon access key and secret key for s3cmd to use to connect to Amazon S3.

A basic configuration file is created automatically when you first issue the s3cmd --configure command after installation. You will be asked a few questions about your Amazon access key and secret key and other settings you wish to use, and then s3cmd will save that information in a new config file.

Other advanced settings can be changed (if needed) by editing the config file manually. Some of the settings contain the default values for s3cmd to use. For instance, you could change the multipart_chunk_size_mb default value from 15 to 5, and that would become the new default value for the s3cmd option --multipart-chunk-size-mb.


The following is an example of a s3cmd config file:

CloudFront is Amazon content delivery network (CDN) — a ton of webservers distributed in multiple datacentres across the globe that should provide a fast access to public files stored in your buckets. The idea of a CDN is to bring some content as close to the user as possible, for instance when an european user is browsing your website he should be served by Europe-based webservers, if the same content is being accessed by clients from Japan they should be served by the CDN servers in Asia. In this case the web content is stored in Amazon S3 and the CDN in use is Amazon CloudFront. See more details at Amazon’s CloudFront page.

Since these two services are very closely related, it makes sense to have CloudFront support directly in s3cmd. CloudFront support has been added since version 0.9.9.

How is CloudFront related to Amazon S3

• About buckets — As you know the files uploaded to Amazon S3 are organised in buckets. A bucket can have a name of your choice, but it pays off to name it in a DNS-compatible way. That in general means lower case only characters of the following groups: a-z, 0-9, - (dash) and . (dot). DNS-incompatible named buckets are not usable with CloudFront. A DNS compatible bucket name is for instance s3tools-test, with s3cmd URI syntax it is s3://s3tools-test

• About publicly accessible files — A file uploaded to S3 with a Public ACL is accessible to anyone over a standard HTTP protocol. For example upload a file logo.png to the above named bucket:

  s3cmd put --acl-public logo.jpg s3://s3tools-test/example/logo.png

  
  The HTTP host name is always http://bucketname.s3.amazonaws.com so in our case the file would be accessible as http://s3tools-test.s3.amazonaws.com/example/logo.png

• About virtual-hosts — If you don’t like the public URL above check out Amazon S3 Virtual Hosts: if your bucket name is a fully qualified domain name and your DNS is set properly you can refer to the bucket directly with its name. For instance let’s have a bucket called s3://public.s3tools.org and upload the above mentioned logo.png in there:

  s3cmd put --acl-public logo.jpg s3://public.s3tools.org/example/logo.png

  Create a DNS record for public.s3tools.org to have a CNAME of public.s3tools.org.s3.amazonaws.com:

  public.s3tools.org.   IN   CNAME   public.s3tools.org.s3.amazonaws.com.

  From now on everybody can access the logo as http://public.s3tools.org/example/logo.png – this way you can offload all the static images, pdf documents, etc from your web server to Amazon S3.

• About CloudFront on the scene — The disadvantage in the above is that your content is in a data centre either in the US or in Europe. If it’s in EU and your visitor lives in South Pacific they’ll experience a poor access performance, even if they live in the US it still won’t be optimal. Wouldn’t it be nice to bring your content closer to them? Let Amazon copy it to the CloudFront datacentres in many places around the world and let it do the magic when selecting the closest datacentre for each client. Simply create for example a DNS record cdn.s3tools.org pointing to a special CNAME that we’ll find out in a later example and have all your static content at http://cdn.s3tools.org/.... This cdn.s3tools.org name will resolve to different IP addresses in different parts of the world, always pointing to the closest CloudFront datacentre available. The before mentioned logo.png accessed through CDN now has a URL: http://cdn.s3tools.org/example/logo.png

How manage CloudFront using s3cmd

• CloudFront is set up at a bucket level — you can publish one or more of your buckets through CloudFront, creating a CloudFront distribution (CFD) for each bucket in question. To publish our public.s3tools.org bucket let’s do:

  s3cmd cfcreate s3://public.s3tools.org

• Each CFD has a unique Distribution ID (DistId) in a form of a URI: cf://123456ABCDEF It’s printed in the output of s3cmd cfcreate:
  

  Distribution created:
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  CNAMEs:
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Status:         InProgress
  Enabled:        True
  Etag:           E3JGOIONPT9834

• Each CFD has a unique “canonical” hostname automatically assigned by Amazon at the time the CFD is created. This could be for instance d11jv2ffak0j4h.nrt4.cloudfront.net.. It can also be found in the cfcreate output, or later on with cfinfo:
  

  ~$ s3cmd cfinfo
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         Deployed
  Enabled:        True

• Apart from the canonical name you can assign up to 10 DNS aliases for each CFD. For example the above canonical name can have an alias of cdn.s3tools.org. Either add the CNAMEs at the time of CFD creation or later with cfmodify command:
  

  ~$ s3cmd cfmodify cf://E3RPA4Z4ALGTGO --cf-add-cname cdn.s3tools.org
  Distribution modified:
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         InProgress
  CNAMEs:         cdn.s3tools.org
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Enabled:        True
  Etag:           E19WWJ5059E2W3

  
  At this moment you should update your DNS again:

  cdn.s3tools.org.   IN   CNAME   d11jv2ffak0j4h.cloudfront.net.

• Run cfinfo to confirm that your change has been deployed. Look for the Status: and Enabled: fields:
  

  ~$ s3cmd cfinfo cf://E3RPA4Z4ALGTGO
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         Deployed
  CNAMEs:         cdn.s3tools.org
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Enabled:        True
  Etag:           E19WWJ5059E2W3

• Congratulation, you’re set up. Now you should be able to access CloudFront using the host name of your choice: http://cdn.s3tools.org/example/logo.png and serve your visitors faster then ever ;-)
• Oh, you may want to remove your CloudFront Distributions later, indeed. Simply run s3cmd cfremove cf://E3RPA4Z4ALGTGO to achieve that. Be aware that it will take a couple of minutes to finish because the CFD must be disabled first and that change must be propagated (“deployed”) before actually removing the the distribution. It’s perhaps easier to disable it manually using s3cmd cfmodify --disable cf://E3RPA4Z4ALGTGO, go get a coffee and once you’re back check that cfinfo says Enabled: False and Status: Deployed. At that moment s3cmd cfremove should succeed immediately.

Amazon S3 supports bucket policy that you can use if you require server-side encryption for all objects that are stored in your bucket. For example, the following bucket policy denies upload object (s3:PutObject) permission to everyone if the request does not include the x-amz-server-side-encryption header requesting server-side encryption.

{
   "Version":"2012-10-17",
   "Id":"PutObjPolicy",
   "Statement":[{
         "Sid":"DenyUnEncryptedObjectUploads",
         "Effect":"Deny",
         "Principal":{
            "AWS":"*"
         },
         "Action":"s3:PutObject",
         "Resource":"arn:aws:s3:::YourBucket/*",
         "Condition":{
            "StringNotEquals":{
               "s3:x-amz-server-side-encryption":"AES256"
            }
         }
      }
   ]
}

In S3cmd, the --server-side-encryption option adds the x-amz-server-side-encryption header to uploaded objects.

You have to empty it first, sorry :-) There are two ways:

1. The convenient one is available in s3cmd 0.9.9 and newer and is as simple as s3cmd del --recursive s3://bucket-to-delete
2. The less convenient one available prior to s3cmd 0.9.9 involves creating an empty directory, say /tmp/empty and synchronizing its content (i.e. nothing) to the bucket: s3cmd sync --delete /tmp/empty s3://bucket-to-delete

Once the bucket is empty it can then be removed with s3cmd rb s3://bucket-to-delete

Please see: wiki on GitHub

To secure our files on Amazon S3, we can restrict access to a S3 bucket to specific IP addresses.

The following bucket policy grants permissions to any user to perform any S3 action on objects in the specified bucket. However, the request must originate from the range of IP addresses specified in the condition. The condition in this statement identifies 192.168.143.* range of allowed IP addresses with one exception, 192.168.143.188.

{
    "Version": "2012-10-17",
    "Id": "S3PolicyIPRestrict",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::bucket/*",
            "Condition" : {
                "IpAddress" : {
                    "aws:SourceIp": "192.168.143.0/24"
                },
                "NotIpAddress" : {
                    "aws:SourceIp": "192.168.143.188/32"
                }
            }
        }
    ]
}

The IPAddress and NotIpAddress values specified in the condition uses CIDR notation described in RFC 2632. For more information, go to http://www.rfc-editor.org/rfc/rfc4632.txt

On Linux you can throttle bandwidth using the throttle command line program, e.g.

cat mybigfile | throttle -k 512 | s3cmd put - s3://mybucket/mybigfile

which would limit reads from mybigfile to be 512kbps.

Throttle is available by apt-get or yum in all the major Linux distributions.

What the s3cmd sync command does is:

1. Walk the filesystem to generate a list of local files
2. Retrieve a list of remote files uploaded to Amazon S3
3. Compare these two lists to find which local files need to be uploaded and which remote files should be deleted

The information about remote files we get from Amazon S3 is limited to names, sizes and md5 of the stored files. If the stored file is GPG encrypted we only get size and md5 of the encrypted file, not the original one and therefore we can't compare the local and remote lists against each other.