S3cmd: FAQ and Knowledge Base

FAQ

Does s3cmd support Amazon S3 server-side encryption?  Does s3cmd support multipart uploads?  Does s3cmd work on Windows?  Does s3cmd work with CloudFront in Frankfurt (eu-central-1)?  Does S3cmd work with Frankfurt region (eu-central-1)?  Is there a mailing list for s3cmd?

Yes, file encryption can optionally be used to make a backup/upload to S3 more secure. Files can be stored on the Amazon S3 servers encrypted (i.e. at rest).

Server-side encryption is only available starting with s3cmd 1.5.0-beta1.

S3cmd provides two types of file encryption: server-side encryption and client-side encryption.

Server-Side encryption is about data encryption at rest, that is, Amazon S3 encrypts your data as it writes it to disks in its data centers and decrypts it for you when you access it. As long as you authenticate your request and you have access permissions, there is no difference in the way you access encrypted or unencrypted objects. Amazon S3 manages encryption and decryption for you. For example, if you share your objects using a pre-signed URL, the pre-signed URL works the same way for both encrypted and unencrypted objects.

Amazon S3 Server Side Encryption employs strong multi-factor encryption. Amazon S3 encrypts each object with a unique key. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates. Amazon S3 Server Side Encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.

When you upload one or more objects with S3cmd, you can specify in your request if you want Amazon S3 to save your object data encrypted. To specify that you want Amazon S3 to save your object data encrypted use the flag --server-side-encryption. Server-side encryption is optional. Your bucket might contain both encrypted and unencrypted objects. Encrypted objects are marked automatically with the metadata header x-amz-server-side-encryption set to AES256.

With Client-Side encryption, you add an extra layer of security by encrypting data locally BEFORE uploading the files to Amazon S3. Client-side encryption and server-side encryption can be combined and used together. In S3cmd, client-side encryption is applied by specifying the flag -e or --encrypt.

Yes, the latest version of s3cmd supports Amazon S3 multipart uploads.

Multipart uploads are automatically used when a file to upload is larger than 15MB.
In that case the file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is then uploaded separately and then reconstructed at destination when the transfer is completed.

With this new feature, if an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded.

There are two options related to multipart uploads in s3cmd. They are:

--disable-multipart

Yes, however, being written in Python, s3cmd requires Python 2.4+ for Windows to be installed and also it requires the Python library date-util.

Alternatively, you can try S3Express. S3Express is a "native" Windows command line program, that works out of the box and does not require any additional library or software to run. S3Express is a commercial program. It's very compact and has very small footprint: the entire program is less than 5MB. S3Express is perfect for uploading, querying, listing Amazon S3 objects via the command line on Windows.

No. CloudFront in Frankfurt currently requires newer signing code than 1.5.0 includes. This issue will be addressed in a future release. Other S3 operations do work for Frankfurt.

Yes. S3cmd supports the new Frankfurt S3 region since version 1.5

Yes, there are 4 s3cmd mailing lists available. Old e-mails are archived and searchable and can be accessed for reference.

The mailing lists are available here:

http://sourceforge.net/p/s3tools/mailman/