FAQ

Frequently asked questions about Amazon S3 tools including s3cmd

s3cmd complains about invalid signature after upgrade from older s3cmd

S3cmd 0.9.5 added support for buckets created in European Amazon’s datacentre. Unfortunately the change has broken access to existing buckets with upper-case characters in their names. This regression has long stayed unnoticed and has only recently been fixed in s3cmd 0.9.8.4.

Therefore if you’re suddenly getting errors like:

ERROR: S3 error: 403 (Forbidden): SignatureDoesNotMatch

after upgrade from s3cmd 0.9.4 or older to 0.9.5 or newer you’re advised to upgrade even further to s3cmd 0.9.8.4 or newer to regain access to your upper-case named buckets. [Download s3cmd]

Why ‘s3cmd sync’ doesn’t support PGP / GPG encryption for files?

What ‘s3cmd sync’ does is:

  1. walk the filesystem to generate a list of local files
  2. retrieve a list of remote files uploaded to Amazon S3
  3. compare these two lists to find which local files need to be uploaded and which remote files should eventually be deleted

The information about remote files we get from Amazon S3 is limited to names, sizes and md5 sums of the stored files. If the stored file is GPG encrypted we only get size and md5sum of the encrypted file, not the original one and therefore we can’t compare the local and remote lists against each other.

There are several solutions to this problem:

  1. Use unconditional recursive “s3cmd put —recursive” instead of “s3cmd sync”. This way s3cmd will always transfer all your files. There is no comparison taking place in this case therefore the files could be encrypted before transfer.
  2. Fetch the original plaintext size / md5sum from every remote file’s headers. This can be very time consuming and also quite expensive since s3cmd would have to perform a query to Amazon S3 for every single file to be compared. This is on my TODO list but is not yet implemented.
  3. To ease some of the problems from previous point s3cmd could have a local cache mapping sizes and md5sums of the encrypted files to the original plaintext sizes and checksums. This is on my TODO list as well. Stay tuned :-)

KeyError: ‘etag’

Please upgrade to s3cmd 0.9.8.4 or later. [Download s3cmd]

AttributeError: ‘module’ object has no attribute ‘format_exc’

You are probably using Python 2.3 or older, however s3cmd only supports Python 2.4, Python 2.5 and Python 2.6 at the moment.

How can I remove a bucket that is not empty?

You have to empty it first, sorry :-) There are two ways:

  1. The convenient one is available in s3cmd 0.9.9 and newer and is as simple as s3cmd del —recursive s3://bucket-to-delete
  2. The less convenient one available prior to s3cmd 0.9.9 involves creating an empty directory, say /tmp/empty and synchronising its content (i.e. nothing) to the bucket: s3cmd sync —delete /tmp/empty s3://bucket-to-delete

Once the bucket is empty it can be removed with s3cmd rb s3://bucket-to-delete

Is your question not answered here?

Use the comments form below or contact the developers on s3tools’ mailing list or forum.

By Michal Ludvig on 10 September 2008

Tags:

---

Comments

  1. Neil S wrote:

    Hello there,

    I was wondering what license s3tools is distributed under? Are there plans to release it as GPL v2 or above? I think it will greatly help the community.

    thanks much.

    ( 5 October 2008, 13:15 · #)

  2. Michal wrote:

    Hi, s3tools including s3cmd is and always has been licensed under GPL v2. I should make it clearer here on this web I think ;-)

    (11 November 2008, 01:26 · #)

  3. Stephen wrote:

    The available command set appears to mirror the WebDAV http methods – is that basically driven by (and hence limited to) what Amazon offers?

    Specifically:

    - Is there a way to block-move a set of files from one bucket to another, ideally without routing the data locally?

    - Is there a way to block-rename files within a bucket (I understand this is how you would also change the paths of a set of files, since paths are really just part of the names in S3)?

    Thanks,
    Stephen

    (28 November 2008, 03:08 · #)

  4. Michal wrote:

    Hi Stephen,

    the Amazon S3 interface is pretty basic, essentially limited to PUT, GET and DELETE commands.

    There is however a way to rename remote files and moving them between buckets. Check out s3cmd 0.9.9-pre2 – have a look at ‘cp’ and ‘mv’ commands. These do what you want. See the s3cmd 0.9.9-pre2 release announce for more details.

    (28 November 2008, 16:28 · #)

  5. Sid Stuart wrote:

    Will s3cmd copy files that are larger than the S3 file size limit?

    ( 4 March 2009, 08:55 · #)

  6. Maung wrote:

    Hello,

    Thanks for making this nice and useful tool. I thought of providing back here for FYI.

    When I execute s3cmd —help, the help message does not mention about COMMANDs such as put, get and sync. I also do not see any mention about commands against buckets: mb, xb and del. Why?

    I am using version 0.9.9.

    Thanks,
    Maung

    (22 April 2009, 06:42 · #)

  7. Josh wrote:

    I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.

    I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!

    Let me know! :)

    Thank you!

    ( 7 August 2009, 14:01 · #)

  8. Josh wrote:

    I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.

    I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!

    Let me know! :)

    Thank you!

    ( 7 August 2009, 14:01 · #)

  9. Josh wrote:

    I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.

    I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!

    Let me know! :)

    Thank you!

    ( 7 August 2009, 14:02 · #)

  10. Reinaldo wrote:

    I was wondering if there’s any way to set headers in the objects uploaded to S3, particulary the “expires” header as described in this article:

    http://www.drunkenfist.com/304/2007/12/26/setting-far-future-expires-headers-for-images-in-amazon-s3/

    Thanks for the info,

    Reinaldo

    ( 9 September 2009, 16:06 · #)

  11. Reinaldo wrote:

    Another question here,

    We are looking to use CloudFront as our CDN for user uploaded images (250,000+ in total).

    Because of OS limitations we had to split the files in several directories of the form:

    /00/11/32/{id}/fileN.jpg

    The thing is that if we try to copy the whole directory recursively from the root up, s3cmd is taking hours sitting there not doing anthig, and then it dies with the following error:

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the following lines to: s3tools-bugs@lists.sourceforge.net
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Problem: MemoryErr:
    S3cmd: 0.9.9

    Traceback (most recent call last): File “/usr/bin/s3cmd”, line 1736, in ? main() File “/usr/bin/s3cmd”, line 1681, in main cmd_func(args) File “/usr/bin/s3cmd”, line 314, in cmd_object_put local_list[key][‘remote_uri’] = unicodise(destination_base + key)
    MemoryError

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the above lines to: s3tools-bugs@lists.sourceforge.net
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Can anybody please help us?

    Thanks

    R

    (11 September 2009, 14:39 · #)

  12. rtconner wrote:

    For future folks that may have this problem…

    On my Ubuntu server I was getting the error “ERROR: Test failed: 403 (RequestTimeTooSkewed): The difference between the request time and the current time is too large.”

    I simply ran `dpkg-reconfigure tzdata` to fix the time on my server and that fixed the problem for me.

    (26 December 2009, 02:15 · #)

 
---