S3cmd: FAQ and Knowledge Base

CloudFront is Amazon content delivery network (CDN) — a ton of webservers distributed in multiple datacentres across the globe that should provide a fast access to public files stored in your buckets. The idea of a CDN is to bring some content as close to the user as possible, for instance when an european user is browsing your website he should be served by Europe-based webservers, if the same content is being accessed by clients from Japan they should be served by the CDN servers in Asia. In this case the web content is stored in Amazon S3 and the CDN in use is Amazon CloudFront. See more details at Amazon’s CloudFront page.

Since these two services are very closely related, it makes sense to have CloudFront support directly in s3cmd. CloudFront support has been added since version 0.9.9.

How is CloudFront related to Amazon S3

• About buckets — As you know the files uploaded to Amazon S3 are organised in buckets. A bucket can have a name of your choice, but it pays off to name it in a DNS-compatible way. That in general means lower case only characters of the following groups: a-z, 0-9, - (dash) and . (dot). DNS-incompatible named buckets are not usable with CloudFront. A DNS compatible bucket name is for instance s3tools-test, with s3cmd URI syntax it is s3://s3tools-test

• About publicly accessible files — A file uploaded to S3 with a Public ACL is accessible to anyone over a standard HTTP protocol. For example upload a file logo.png to the above named bucket:

  s3cmd put --acl-public logo.jpg s3://s3tools-test/example/logo.png

  
  The HTTP host name is always http://bucketname.s3.amazonaws.com so in our case the file would be accessible as http://s3tools-test.s3.amazonaws.com/example/logo.png

• About virtual-hosts — If you don’t like the public URL above check out Amazon S3 Virtual Hosts: if your bucket name is a fully qualified domain name and your DNS is set properly you can refer to the bucket directly with its name. For instance let’s have a bucket called s3://public.s3tools.org and upload the above mentioned logo.png in there:

  s3cmd put --acl-public logo.jpg s3://public.s3tools.org/example/logo.png

  Create a DNS record for public.s3tools.org to have a CNAME of public.s3tools.org.s3.amazonaws.com:

  public.s3tools.org.   IN   CNAME   public.s3tools.org.s3.amazonaws.com.

  From now on everybody can access the logo as http://public.s3tools.org/example/logo.png – this way you can offload all the static images, pdf documents, etc from your web server to Amazon S3.

• About CloudFront on the scene — The disadvantage in the above is that your content is in a data centre either in the US or in Europe. If it’s in EU and your visitor lives in South Pacific they’ll experience a poor access performance, even if they live in the US it still won’t be optimal. Wouldn’t it be nice to bring your content closer to them? Let Amazon copy it to the CloudFront datacentres in many places around the world and let it do the magic when selecting the closest datacentre for each client. Simply create for example a DNS record cdn.s3tools.org pointing to a special CNAME that we’ll find out in a later example and have all your static content at http://cdn.s3tools.org/.... This cdn.s3tools.org name will resolve to different IP addresses in different parts of the world, always pointing to the closest CloudFront datacentre available. The before mentioned logo.png accessed through CDN now has a URL: http://cdn.s3tools.org/example/logo.png

How manage CloudFront using s3cmd

• CloudFront is set up at a bucket level — you can publish one or more of your buckets through CloudFront, creating a CloudFront distribution (CFD) for each bucket in question. To publish our public.s3tools.org bucket let’s do:

  s3cmd cfcreate s3://public.s3tools.org

• Each CFD has a unique Distribution ID (DistId) in a form of a URI: cf://123456ABCDEF It’s printed in the output of s3cmd cfcreate:
  

  Distribution created:
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  CNAMEs:
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Status:         InProgress
  Enabled:        True
  Etag:           E3JGOIONPT9834

• Each CFD has a unique “canonical” hostname automatically assigned by Amazon at the time the CFD is created. This could be for instance d11jv2ffak0j4h.nrt4.cloudfront.net.. It can also be found in the cfcreate output, or later on with cfinfo:
  

  ~$ s3cmd cfinfo
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         Deployed
  Enabled:        True

• Apart from the canonical name you can assign up to 10 DNS aliases for each CFD. For example the above canonical name can have an alias of cdn.s3tools.org. Either add the CNAMEs at the time of CFD creation or later with cfmodify command:
  

  ~$ s3cmd cfmodify cf://E3RPA4Z4ALGTGO --cf-add-cname cdn.s3tools.org
  Distribution modified:
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         InProgress
  CNAMEs:         cdn.s3tools.org
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Enabled:        True
  Etag:           E19WWJ5059E2W3

  
  At this moment you should update your DNS again:

  cdn.s3tools.org.   IN   CNAME   d11jv2ffak0j4h.cloudfront.net.

• Run cfinfo to confirm that your change has been deployed. Look for the Status: and Enabled: fields:
  

  ~$ s3cmd cfinfo cf://E3RPA4Z4ALGTGO
  Origin:         s3://public.s3tools.org/
  DistId:         cf://E3RPA4Z4ALGTGO
  DomainName:     d11jv2ffak0j4h.cloudfront.net
  Status:         Deployed
  CNAMEs:         cdn.s3tools.org
  Comment:        http://public.s3tools.org.s3.amazonaws.com/
  Enabled:        True
  Etag:           E19WWJ5059E2W3

• Congratulation, you’re set up. Now you should be able to access CloudFront using the host name of your choice: http://cdn.s3tools.org/example/logo.png and serve your visitors faster then ever ;-)
• Oh, you may want to remove your CloudFront Distributions later, indeed. Simply run s3cmd cfremove cf://E3RPA4Z4ALGTGO to achieve that. Be aware that it will take a couple of minutes to finish because the CFD must be disabled first and that change must be propagated (“deployed”) before actually removing the the distribution. It’s perhaps easier to disable it manually using s3cmd cfmodify --disable cf://E3RPA4Z4ALGTGO, go get a coffee and once you’re back check that cfinfo says Enabled: False and Status: Deployed. At that moment s3cmd cfremove should succeed immediately.