Amazon S3 Tools: Command Line S3 Client Software and S3 Backup

AWS S3 Command Line Clients for Windows, Linux, Mac. Backup to S3, upload, retrieve, query data on Amazon S3.


Amazon S3 Overview

Amazon S3 (Simple Storage Service) provides a managed internet-accessible storage service where anyone can store any amount of data and retrieve it later again. Maximum amount of data in one “object” is 5GB, maximum number of objects is not limited.

Use our command-line tool S3cmd to work with Amazon S3 and upload/retrieve your data.

What is Amazon S3

S3 is a paid service operated by the well known Amazon.com internet book shop. Before storing anything into S3 you must sign up for an “AWS” account (where AWS = Amazon Web Services) to obtain a pair of identifiers: Access Key and Secret Key. You will need to give these keys to S3cmd. Think of them as if they were a username and password for your S3 account.

Pricing explained

At the time of this writing the costs of using S3 are:

  1. US$0.15 per GB-Month of storage used.
  2. US$0.20 per GB of data transferred.

If for instance on 1st of January you upload 2GB of photos in JPEG from your holiday in New Zealand, at the end of January you will be charged $0.30 for using 2GB of storage space for a month and $0.40 for transferring 2GB of data. That comes to $0.70 for a complete backup of your precious holiday pictures.

In February you don’t touch it. Your data are still on S3 servers so you pay $0.30 for those two gigabytes, but not a single cent will be charged for any transfer. That comes to $0.30 as an ongoing cost of your backup. Not too bad.

In March you allow anonymous read access to some of your pictures and your friends download, say, 500MB of them. As the files are owned by you, you are responsible for the costs incurred. That means at the end of March you’ll be charged $0.30 for storage plus $0.07 for the traffic generated by your friends.

There is no minimum monthly contract or a setup fee. What you use is what you pay for. At the beginning my bill used to be like US$0.03 or even nil.

That’s the pricing model of Amazon S3 in a nutshell. Check Amazon S3 homepage at http://aws.amazon.com/s3 for more details.

Needless to say that all these dollars are charged by Amazon, there is obviously no payment for using S3cmd :-)

Amazon S3 basics

Files stored in S3 are called objects and their names are officially called keys. Each object belongs to exactly one bucket. Buckets are kind of directories or folders with some restrictions:

  1. each user can only have 100 buckets at the most,
  2. bucket names must be unique amongst all users of S3,
  3. buckets can not be nested into a deeper hierarchy and
  4. a name of a bucket can only consist of basic alphanumeric characters plus dot (.) and dash (-). No spaces, no accented or UTF-8 letters, etc.

On the other hand there are almost no restrictions on object names – keys. These can be any UTF-8 strings of up to 1024 bytes long. Interestingly enough the object name can contain forward slash character (/) thus a my/funny/picture.jpg is a valid object name. Note that there are not directories nor buckets called “my” and “funny” – it is really a single object name called “my/funny/picture.jpg” and S3 does not care at all that it looks like a directory structure.

To describe objects in S3 storage we invented a URI-like schema in the following form:

s3://BUCKET/OBJECT

See the S3cmd HowTo for example usages of this S3-URI schema.


Top   Home

Copyright© 2009-2014