s3cmd complains about invalid signature after upgrade from older s3cmd
S3cmd 0.9.5 added support for buckets created in European Amazon’s datacentre. Unfortunately the change has broken access to existing buckets with upper-case characters in their names. This regression has long stayed unnoticed and has only recently been fixed in s3cmd 0.9.8.4.
Therefore if you’re suddenly getting errors like:
ERROR: S3 error: 403 (Forbidden): SignatureDoesNotMatch
after upgrade from s3cmd 0.9.4 or older to 0.9.5 or newer you’re advised to upgrade even further to s3cmd 0.9.8.4 or newer to regain access to your upper-case named buckets. [Download s3cmd]
Why ‘s3cmd sync’ doesn’t support PGP / GPG encryption for files?
What ‘s3cmd sync’ does is:
- walk the filesystem to generate a list of local files
- retrieve a list of remote files uploaded to Amazon S3
- compare these two lists to find which local files need to be uploaded and which remote files should eventually be deleted
The information about remote files we get from Amazon S3 is limited to names, sizes and md5 sums of the stored files. If the stored file is GPG encrypted we only get size and md5sum of the encrypted file, not the original one and therefore we can’t compare the local and remote lists against each other.
There are several solutions to this problem:
- Use unconditional recursive “s3cmd put —recursive” instead of “s3cmd sync”. This way s3cmd will always transfer all your files. There is no comparison taking place in this case therefore the files could be encrypted before transfer.
- Fetch the original plaintext size / md5sum from every remote file’s headers. This can be very time consuming and also quite expensive since s3cmd would have to perform a query to Amazon S3 for every single file to be compared. This is on my TODO list but is not yet implemented.
- To ease some of the problems from previous point s3cmd could have a local cache mapping sizes and md5sums of the encrypted files to the original plaintext sizes and checksums. This is on my TODO list as well. Stay tuned :-)
Please upgrade to s3cmd 0.9.8.4 or later. [Download s3cmd]
AttributeError: ‘module’ object has no attribute ‘format_exc’
You are probably using Python 2.3 or older, however s3cmd only supports Python 2.4, Python 2.5 and Python 2.6 at the moment.
How can I remove a bucket that is not empty?
You have to empty it first, sorry :-) There are two ways:
- The convenient one is available in s3cmd 0.9.9 and newer and is as simple as s3cmd del —recursive s3://bucket-to-delete
- The less convenient one available prior to s3cmd 0.9.9 involves creating an empty directory, say /tmp/empty and synchronising its content (i.e. nothing) to the bucket: s3cmd sync —delete /tmp/empty s3://bucket-to-delete
Once the bucket is empty it can be removed with s3cmd rb s3://bucket-to-delete