s3cmd 0.9.9-pre2 is ready for testing

s3cmd 0.9.9-pre2 has been uploaded to SourceForge just a few minutes ago. This pre-release contains many new features that you may find handy:

Progress meter

The old useless info about of every block sent/received is now gone. Instead I have implemented user friendly progress meter for both upload and download operations:

 big_document.pdf     126976 of 2304411     5% in 9s  12.51 kB/s

This is on by default on interactive terminals (TTYs) and off when running non-interactively, for instance as a cron job. However on / off can be forced with —progress / —no-progress parameters.

Continue getting partially downloaded files

Should a download of a large file from Amazon S3 bucket fatally fail, for instance a laptop went offline, it is now possible to restart get from where it stopped using —continue parameter.

The other alternative is to use —force and start download again from the beginning. That’s good enough for small files but in case of, say, ISO images you may prefer to continue the download.

Improved resistance to errors

Although s3cmd 0.9.8.4 improved resistance to some upload problems there were still places where s3cmd would fail, especially during the initial (connect) phase. Furthermore errors in download, that is get command and sync from S3 to local filesystem, were not handled at all.

All these code paths were much improved and every failed request, be it file upload, file download or a generic command like getting a bucket listing, is now reissued up to 5 times. I have spent a lot of
time with ‘tcpkill’ tool and hopefully there should be no more “Connection reset by peer”, “Name or service not known”, etc errors breaking out.

Removing of non-empty buckets

“How do I remove non-empty bucket?” FAQ now has a simple answer: Use “s3cmd rb —force “

Recursively remove objects from buckets

Related task, i.e. removing all objects with a given prefix, is now possible with —recursive (-r) parameter.

Copying, renaming and moving objects

Amazon S3 now supports Copy operation on existing objects. That allows s3cmd to have cp command for making a copy of an object and mv command for renaming / moving objects. Destination can be the same bucket or a different one. It’s even possible to copy / move objects from US buckets to EU buckets and back, however note that Amazon will charge you a data transfer rate for such a move. Thanks Andrew Ryan for initial patch.

Note that you can change ACLs on copy / move with —acl-public and —acl-private options.

Although s3cmd 0.9.9-pre2 is a “works for me” release, all the above features deserve a good testing. I may have missed some corner cases.

I’m pretty confident this is a stable enough release and the major reason why it isn’t called “final” is a list of features that I’d like to implement for 0.9.9. Here it goes:

TODO list for final 0.9.9 (in the order of importance):

• Support —recursive for most commands. As you know s3cmd ls on a large bucket is useless. Also ‘put’, ‘get’, ‘cp’ and ‘mv’ are good candidates for —recursive support. They should also process —exclude lists.
• Add setacl command for changing access rights for existing objects. At least for making private objects public and vice versa.
• Add —include/—include-from/—rinclude*, similar to what —exclude does. This in fact would be a good exercise for a wannabe python developer who’d like to contribute to s3cmd project ;-)
• Update documentation and manual page ;-)

That’s it. Give s3cmd 0.9.9-pre2 some bashing if you can. Download the source either from ‘testing’ package on SourceForge:
http://sourceforge.net/project/showfiles.php?group_id=178907
or stay up to date with a SVN checkout of s3cmd “trunk”:
http://s3tools.logix.cz/download

I’m ready to hear what problems you experience. Emails of “it works great, no problems at all” nature are welcome even more ;-)

Enjoy