S3cmd is alive! We are pulling patches together for the upcoming s3cmd 1.0.0 release. The current development snapshot has just been released as 1.0.0-rc1.
New features
Support for Reduced Redundancy Storage
s3cmd put, sync, cp and mv commands now accept --reduced-redundancy (or --rr for short) parameter to that tells Amazon to store the files in a little cheaper but slightly less reliable way. See the Amazon S3 RRS page for more details
Access logging for S3 Buckets and for CloudFront
Turn access logging on and off for S3 buckets with s3cmd accesslog command or in case of turning access logs on/off for a CloudFront distribution with s3cmd cfmodify command. In either case use --access-logging-target-prefix=s3://some-other-bucket/logs/blah/ to enable access logging and --no-access-logging to disable them again. The access logs would be stored under s3://some-other-bucket/logs/blah/… and could be listed with s3cmd ls and downloaded for further processing with s3cmd get or s3cmd sync.
New setacl --acl-grant and --acl-revoke supporting email addresses and other kinds of grantees
Earlier releases of s3cmd recognise only two kinds of ACL (access control list): public and private. Amazon S3 can, however, do much more – it can grant access based on email address, username and even url in some special cases. From now on s3cmd setacl can set these permissions too. For example to give someone from example.com a read-only access to all files in the bucket do: --acl-grant=read:someone@example.com
Note that someone@example.com must have an Amazon AWS account registered with this address. This feature has been contributed by Timothee Groleau
Support for creating buckets in locations outside the original US and EU.
It is now possible to create buckets in any S3 location with s3cmd mb --bucket-location=.... As of now the bucket locations are US (default if no explicit bucket location is set), EU, us-west-1 and ap-southeast-1. These names are no longer pre-set in the code which means that as soon as Amazon opens a new datacentre somewhere s3cmd will be able to create buckets in there.
Follow local symlinks with --follow-symlinks parameter
s3cmd put and sync will now follow symbolic links. i.e. upload the file the symlink points to instead of ignoring it, when --follow-symlinks is used. This feature has been contributed by Aaron Maxwell
Support for Default Root Object in CloudFront
CloudFront distributions now support specifying a default root object which is returned when the user requests a URL without an explicit object name, for instance http://cdn.example.com/
Using s3cmd cfmodify —default-root-object=index.html s3://cdn.example.com the above URL will behave exactly the same as if it was http://cdn.example.com/index.html
This feature was contributed by Luke Andrew
CloudFront commands now accept bucket name (s3://bucket) as well as CloudFront distribution id (cf://A1B2C3D4E5) for convenience
This is a minor convenience improvement. Most s3cmd cf*-commands used to require the cryptic CloudFront distribution ID (e.g. cf://A1B2C3D4E5) that users usually don’t remember (at least I don’t). From now on it’s possible to use the S3 URI with most cf*-commands, e.g. s3cmd cfinfo s3://my.bucket.com
Bugs fixed
Obviously we’ve had some annoying bugs fixed too. The most important ones are:
- Don’t crash when files disappear during upload
- Don’t crash on the infamous “Error 21 – Is a directory” condition
- Don’t crash when listing buckets with 1000+ directories. Contributed by Timothee Groleau.
- Don’t give up easily on failed requests, retry upload in more cases than before.
- Improved python 2.7 compatibility
- Lots of other minor bugfixes
Documentation
One major thing holding us back from releasing the final 1.0.0 is indeed the documentation. Any volunteers to bring the manpage up to date with the current code? You, there! Will you? No? Oh well… :(
Download
Pretty please test test test the current s3cmd 1.0.0-rc1 and let me know if you experience any problems:
http://sourceforge.net/projects/s3tools/files/s3cmd/1.0.0-rc1/s3cmd-1.0.0-rc1.tar.gz/download
Any questions or problems? Please send an email to the mailing list: s3tools-general@lists.sourceforge.net
Looking forward for your feedback!
Enjoy!
Don wrote:
Problem running the setup script in Windows 7 64-bit. This is the error:
File “C:\s3cmd-1.0.0-rc1\setup.py”, line 14 print “Using xml.etree.ElementTree for XML processing” ^ SyntaxError: invalid syntax( 3 November 2010, 12:00 · #)
Michal Ludvig wrote:
Hi Don,
it looks like you’re using Python 3 – that’s not supported yet. Please install Python 2.7 and try again.
Michal
( 3 November 2010, 13:31 · #)
emsa wrote:
Are —acl-grant/—acl-revoke really available in rc1? I cannot get them to to work an —debug does not show that the information added are used in any way … ?
/Magnus
( 4 November 2010, 22:46 · #)
Michal Ludvig wrote:
Hi Magnus,
yes these parameters are available, however they only work with setacl command. Not with put/sync as you might have expected. Hopefully in the final 1.0.0…
Michal
( 4 November 2010, 23:01 · #)
Pavel Heinrich wrote:
Would you be so kind and add it to the CentOS repository? ;o)
( 8 December 2010, 23:06 · #)
diego wrote:
I did try the rc1 to bypass the “Err21 – is a directory” crash but even when it does not crash it does not work.
Either with sync or get it fails in some cases.
I did a s3cmd get s3://mybucket/files/backup/* /mnt/backup/ -v -d —force but even when it does not crash it does not download the files.
It normally works when I pull the files from
an already live instance. But when the instance is starting up it sometimes retrieves the files and some times it doesn´t.
This is the debug output:
: ConfigParser: access_key->AK…17_chars…Q
DEBUG: ConfigParser: acl_public->False
DEBUG: ConfigParser: bucket_location->USsoc
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: cloudfront_resource->/2008-06-30/distribution
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d —verbose —no-use-agent —batch —yes —passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c —verbose —no-use-agent —batch —yes —passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->so…8_chars…4
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: secret_key->CH…37_chars…d
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: Updating Config.Config encoding -> ANSI_X3.4-1968
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config force -> True
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising ‘get’ using ANSI_X3.4-1968
DEBUG: Unicodising ‘s3://mybucket/files/backups/*’ using ANSI_X3.4-1968
DEBUG: Unicodising ‘/mnt/backups/’ using ANSI_X3.4-1968
DEBUG: Command: get
DEBUG: Unicodising ‘files/backups/’ using ANSI_X3.4-1968
DEBUG: Unicodising ‘files/backups/’ using ANSI_X3.4-1968
INFO: Retrieving list of remote files for s3://mybucket/files/backups/ …
DEBUG: String ‘files/backups/’ encoded to ‘files/backups/’
DEBUG: SignHeaders: ‘GET\n\n\n\nx-amz-date:Tue, 14 Dec 2010 01:04:55 +0000\n/mybucket/’
DEBUG: CreateRequest: resource[uri]=/
DEBUG: SignHeaders: ‘GET\n\n\n\nx-amz-date:Tue, 14 Dec 2010 01:04:55 +0000\n/mybucket/’
DEBUG: Processing request, please wait…
DEBUG: get_hostname(mybucket): mybucket.s3.amazonaws.com
DEBUG: format_uri(): /?prefix=files/backups/&delimiter=/
Generating public/private rsa key pair.
DEBUG: Response: {‘status’: 200, ‘headers’: {‘x-amz-id-2’: ‘s/vI0iRGQdk9/iM6xzui7rDxVlmRp/oR7NwH/we3GHxwa0vmEwL0jOMsmgI1BzFP’, ‘server’: ‘AmazonS3’, ‘transfer-encoding’: ‘chunked’, ‘x-amz-request-id’: ‘13551896900FB621’, ‘date’: ‘Tue, 14 Dec 2010 01:04:56 GMT’, ‘content-type’: ‘application/xml’}, ‘reason’: ‘OK’, ‘data’: ‘<?xml version=“1.0” encoding=“UTF-8”?>\n<ListBucketResult xmlns=“http://s3.amazonaws.com/doc/2006-03-01/”><Name>mybucket</Name><Prefix>files/backups/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>files/backups/restoreme.tar.gz</Key><LastModified>2010-12-13T17:57:50.000Z</LastModified><ETag>"f078001e5cad26a44ebaef747023764c"</ETag><Size>307093997</Size><Owner><ID>44277703242175099626d7bbec3a245495249670020aed236ce88b704b066100</ID><DisplayName>diegofbarajas</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>’}
INFO: Applying —exclude/—include
DEBUG: CHECK: restoreme.tar.gz
DEBUG: PASS: restoreme.tar.gz
INFO: Summary: 1 remote files to download
DEBUG: DeUnicodising u’/mnt/backups/’ using ANSI_X3.4-1968
DEBUG: Unicodising ‘/mnt/backups/’ using ANSI_X3.4-1968
ERROR: Skipping /mnt/backups/: Is a directory
(14 December 2010, 15:25 · #)
paulwintech wrote:
Hi,
Thanks for the s3cmd-100rc1-released to support RR. But there any release or patch to support Server Side encryption please…. Since i found a patch in for sse support http://code.google.com/p/s3sync-s3cmd/issues/detail?id=8, but no luck Kindly let me know options for sever side encryption(sse)Thanks
Paul
( 7 December 2011, 22:57 · #)
Michael wrote:
I’m getting this error when trying to compile on windows7 using python 2.7.3
Using xml.etree.ElementTree for XML processing
Traceback (most recent call last): File “setup.py”, line 38, in <module> if sys.argv1 == “sdist”:
IndexError: list index out of range
(31 July 2012, 13:01 · #)
hreinn wrote:
Michael, run python setup.py install
( 5 December 2012, 00:04 · #)
Bruno Dias wrote:
To use s3cmd with a bucket in Sao Paulo region, change this lines in .s3cfg:
host_base = s3-sa-east-1.amazonaws.com
host_bucket = %(bucket)s.s3-sa-east-1.amazonaws.com
( 4 April 2013, 10:37 · #)