Frequently asked questions about Amazon S3 tools including s3cmd
s3cmd complains about invalid signature after upgrade from older s3cmd
S3cmd 0.9.5 added support for buckets created in European Amazon’s datacentre. Unfortunately the change has broken access to existing buckets with upper-case characters in their names. This regression has long stayed unnoticed and has only recently been fixed in s3cmd 0.9.8.4.
Therefore if you’re suddenly getting errors like:
ERROR: S3 error: 403 (Forbidden): SignatureDoesNotMatch
after upgrade from s3cmd 0.9.4 or older to 0.9.5 or newer you’re advised to upgrade even further to s3cmd 0.9.8.4 or newer to regain access to your upper-case named buckets. [Download s3cmd]
Why ‘s3cmd sync’ doesn’t support PGP / GPG encryption for files?
What ‘s3cmd sync’ does is:
- walk the filesystem to generate a list of local files
- retrieve a list of remote files uploaded to Amazon S3
- compare these two lists to find which local files need to be uploaded and which remote files should eventually be deleted
The information about remote files we get from Amazon S3 is limited to names, sizes and md5 sums of the stored files. If the stored file is GPG encrypted we only get size and md5sum of the encrypted file, not the original one and therefore we can’t compare the local and remote lists against each other.
There are several solutions to this problem:
- Use unconditional recursive “s3cmd put —recursive” instead of “s3cmd sync”. This way s3cmd will always transfer all your files. There is no comparison taking place in this case therefore the files could be encrypted before transfer.
- Fetch the original plaintext size / md5sum from every remote file’s headers. This can be very time consuming and also quite expensive since s3cmd would have to perform a query to Amazon S3 for every single file to be compared. This is on my TODO list but is not yet implemented.
- To ease some of the problems from previous point s3cmd could have a local cache mapping sizes and md5sums of the encrypted files to the original plaintext sizes and checksums. This is on my TODO list as well. Stay tuned :-)
KeyError: ‘etag’
Please upgrade to s3cmd 0.9.8.4 or later. [Download s3cmd]
AttributeError: ‘module’ object has no attribute ‘format_exc’
You are probably using Python 2.3 or older, however s3cmd only supports Python 2.4, Python 2.5 and Python 2.6 at the moment.
How can I remove a bucket that is not empty?
You have to empty it first, sorry :-) There are two ways:
- The convenient one is available in s3cmd 0.9.9 and newer and is as simple as s3cmd del —recursive s3://bucket-to-delete
- The less convenient one available prior to s3cmd 0.9.9 involves creating an empty directory, say /tmp/empty and synchronising its content (i.e. nothing) to the bucket: s3cmd sync —delete /tmp/empty s3://bucket-to-delete
Once the bucket is empty it can be removed with s3cmd rb s3://bucket-to-delete
Is your question not answered here?
Use the comments form below or contact the developers on s3tools’ mailing list or forum.
Neil S wrote:
Hello there,
I was wondering what license s3tools is distributed under? Are there plans to release it as GPL v2 or above? I think it will greatly help the community.
thanks much.
( 5 October 2008, 14:15 · #)
Michal wrote:
Hi, s3tools including s3cmd is and always has been licensed under GPL v2. I should make it clearer here on this web I think ;-)
(11 November 2008, 02:26 · #)
Stephen wrote:
The available command set appears to mirror the WebDAV http methods – is that basically driven by (and hence limited to) what Amazon offers?
Specifically:
- Is there a way to block-move a set of files from one bucket to another, ideally without routing the data locally?
- Is there a way to block-rename files within a bucket (I understand this is how you would also change the paths of a set of files, since paths are really just part of the names in S3)?
Thanks,
Stephen
(28 November 2008, 04:08 · #)
Michal wrote:
Hi Stephen,
the Amazon S3 interface is pretty basic, essentially limited to PUT, GET and DELETE commands.
There is however a way to rename remote files and moving them between buckets. Check out s3cmd 0.9.9-pre2 – have a look at ‘cp’ and ‘mv’ commands. These do what you want. See the s3cmd 0.9.9-pre2 release announce for more details.
(28 November 2008, 17:28 · #)
Sid Stuart wrote:
Will s3cmd copy files that are larger than the S3 file size limit?
( 4 March 2009, 09:55 · #)
Maung wrote:
Hello,
Thanks for making this nice and useful tool. I thought of providing back here for FYI.
When I execute s3cmd —help, the help message does not mention about COMMANDs such as put, get and sync. I also do not see any mention about commands against buckets: mb, xb and del. Why?
I am using version 0.9.9.
Thanks,
Maung
(22 April 2009, 07:42 · #)
Josh wrote:
I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.
I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!
Let me know! :)
Thank you!
( 7 August 2009, 15:01 · #)
Josh wrote:
I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.
I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!
Let me know! :)
Thank you!
( 7 August 2009, 15:01 · #)
Josh wrote:
I can’t seem to find the exact location, but somewhere on your site you mentioned that you might include a caching database function in the future which would allow the sync process to be faster as you could determine if the local file has been modified “locally” instead of having to query S3 for the information.
I was wondering if I could donate to this cause! I haven’t used s3cmd yet because this feature is missing, although with the feature I would be using it in a heartbeat!
Let me know! :)
Thank you!
( 7 August 2009, 15:02 · #)
Reinaldo wrote:
I was wondering if there’s any way to set headers in the objects uploaded to S3, particulary the “expires” header as described in this article:
http://www.drunkenfist.com/304/2007/12/26/setting-far-future-expires-headers-for-images-in-amazon-s3/
Thanks for the info,
Reinaldo
( 9 September 2009, 17:06 · #)
Reinaldo wrote:
Another question here,
We are looking to use CloudFront as our CDN for user uploaded images (250,000+ in total).
Because of OS limitations we had to split the files in several directories of the form:
/00/11/32/{id}/fileN.jpg
The thing is that if we try to copy the whole directory recursively from the root up, s3cmd is taking hours sitting there not doing anthig, and then it dies with the following error:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the following lines to: s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Problem: MemoryErr:
S3cmd: 0.9.9
Traceback (most recent call last): File “/usr/bin/s3cmd”, line 1736, in ? main() File “/usr/bin/s3cmd”, line 1681, in main cmd_func(args) File “/usr/bin/s3cmd”, line 314, in cmd_object_put local_list[key][‘remote_uri’] = unicodise(destination_base + key)
MemoryError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the above lines to: s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Can anybody please help us?
Thanks
R
(11 September 2009, 15:39 · #)
rtconner wrote:
For future folks that may have this problem…
On my Ubuntu server I was getting the error “ERROR: Test failed: 403 (RequestTimeTooSkewed): The difference between the request time and the current time is too large.”
I simply ran `dpkg-reconfigure tzdata` to fix the time on my server and that fixed the problem for me.
(26 December 2009, 03:15 · #)
kettle wrote:
Will newer versions of s3cmd support the new asian datacenter? And will there be support for enabling versioning in s3cmd?
I’d find both of these really useful.
(30 April 2010, 23:43 · #)
yoshikawa wrote:
Where can I find “s3cmd encryption HowTo?”
The URL http://s3tools.org/s3cmd-encryption says “404 Not Found.”
( 9 June 2010, 21:03 · #)
Andrew Answer wrote:
Hello,
I check s3cmd together with s3fs and found what directories created by s3cmd don’t visible by s3fs. Also, when I create directories with s3fs (it allow to create empty dirs like filesystem) s3cmd can show info for these dirs with “s3cmd info s3://[bucket]/dir” command (no end slash). I see MIMe type “application/x-directory”, MD5 hash and other useful info. But, if I upload file with prefix by s3cmd “s3cmd put s3://[bucket]/dir/file.txt” and try to get info for dir with “s3cmd info s3://[bucket]/dir” I see NOTHING. s3cmd hang. Why?
(22 August 2010, 06:52 · #)
Ron Dyck wrote:
I’m considering upgrading python to v 3.1.2
Will this cause incompatibility problems with s3cmd-1.0.0-rc1?
Thanks.
ron
(31 October 2010, 03:57 · #)
Michal Ludvig wrote:
Ron:
yes very likely it will break most Python programs on your system, not only s3cmd. I wouldn’t replace the current Python 2.x if I were you. If you need Py3 for something install it into a separate directory alongside Py2 or you risk some major problems.
(31 October 2010, 08:20 · #)
Mike Shaver wrote:
Is it possible to have multiple S3 accounts and use s3tools? It doesn’t seem like you can pass an argument for the S3 account to use?
I know you can just copy the .s3cfg file to modify the keys for another account, but can you switch to another .s3cfg file for use?
(11 November 2010, 13:41 · #)
Miller Peterson wrote:
I’m trying to use the ‘sync’ command to sync a local and remote directory, but when I do so and there are a large number of files in the directories being compared, it never seems to complete. When I run the command with the -v option, it seems stuck at “INFO: Verifying attributes…” There are about 350 files in the remote directory, and slightly more in the local directory. I’ve never seen it finish (even after waiting upwards of 20 minutes) – is it reasonable that it would take a very long time to do this, or is something else wrong?
(18 November 2010, 12:45 · #)
pratfall wrote:
<blockquote>
yoshikawa wrote:
Where can I find “s3cmd encryption HowTo?”
The URL http://s3tools.org/s3cmd-encryption says “404 Not Found.”
( 9 June 2010, 20:03 · #)
</blockquote>
Fix this please. Also, what’s the status of the Fuse subproject? I’m using s3fs and it does all kinds of stupid stuff with metadata that makes browsing the actual filesystem suck.
(14 January 2011, 08:09 · #)
Corin wrote:
Hi, I’ve come across a bug in your otherwise great software.
When putting or syncing a file and specifying a Content-Type header, the process fails.
s3cmd -r -P —add-header ‘Content-Type: text/plain’ etc…
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.
Problem does not exist when using other headsers (such as Cache-Control).
Thanks!
(29 January 2011, 17:37 · #)
Dave wrote:
Corin, just use -m or —mime-type=
these set the content-type correctly.
( 2 March 2011, 10:21 · #)
Rob wrote:
Is there any recommended way of also encrypting the name of files before they are sent up?
(10 May 2011, 00:34 · #)
Jason wrote:
I am able to list the contents of my bucket, however when I attempt to download, I get this error:
ERROR: S3 error: 403 (Forbidden):
no other information is given about the error. Is there anything further I can do to debug this?
Thanks!
Jason
(21 May 2011, 08:01 · #)
qliq wrote:
Hi, I used to love s3cmd for the smooth file transfer but recently it just fails to put any file on the bucket, after a lot of efforts like this:
…
WARNING: Upload failed: /bk-file.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec…
….
This happens even for files as small as 100MB. I am using s3cmd version 1.0.1 on Ubuntu (the very same server on which s3cmd used to work like a charm).
Your fix or suggestions is much appreciated.
(10 November 2011, 17:15 · #)
Kevin JAmes wrote:
How do I add a username & password when using a proxy?
I dont see the option anywhere.
Thanks.
(24 February 2012, 13:03 · #)
Jeremy wrote:
I found beta3 on the sourceforge site, but don’t see any mention of it here or other news updates. Any chance of a package release to the repos?
I peeked at github and can see that it’s still being actively developed. :-)
The sync + invalidation is great!
(26 April 2012, 04:32 · #)