CloudFront is Amazon’s content delivery network (CDN) — a ton of webservers distributed in multiple datacentres across the globe that should provide a fast access to public files stored in your buckets. The idea of a CDN is to bring some content as close to the user as possible, for instance when an european user is browsing your website he should be served by Europe-based webservers, if the same content is being accessed by clients from Japan they should be served by the CDN servers in Asia. In this case the web content is stored in Amazon S3 and the CDN in use is Amazon CloudFront. See more details at Amazon’s CloudFront page.
Since these two services are that closely related it makes sense to have CloudFront support directly in s3cmd. It has been added in s3cmd 0.9.9.
How is CloudFront related to Amazon S3
- About buckets — As you know the files uploaded to Amazon S3 are organised in buckets. A bucket can have a name of your choice, but it pays off to name it in a DNS-compatible way. That in general means lower case only characters of the following groups: a-z, 0-9, - (dash) and . (dot). DNS-incompatible named buckets are not usable with CloudFront. A DNS compatible bucket name is for instance
s3tools-test, with s3cmd URI syntax it is
- About publicly accessible files — A file uploaded to S3 with a Public ACL is accessible to anyone over a standard HTTP protocol. For example upload a file logo.png to the above named bucket:
s3cmd put --acl-public logo.jpg s3://s3tools-test/example/logo.png
The HTTP host name is always http://bucketname.s3.amazonaws.com so in our case the file would be accessible as
- About virtual-hosts — If you don’t like the public URL above check out Amazon S3 Virtual Hosts: if your bucket name is a fully qualified domain name and your DNS is set properly you can refer to the bucket directly with its name. For instance let’s have a bucket called
s3://public.s3tools.organd upload the above mentioned logo.png in there:
s3cmd put --acl-public logo.jpg s3://public.s3tools.org/example/logo.pngCreate a DNS record for public.s3tools.org to have a CNAME of public.s3tools.org.s3.amazonaws.com:
public.s3tools.org. IN CNAME public.s3tools.org.s3.amazonaws.com.From now on everybody can access the logo as http://public.s3tools.org/example/logo.png – this way you can offload all the static images, pdf documents, etc from your web server to Amazon S3.
- About CloudFront on the scene — The disadvantage in the above is that your content is in a data centre either in the US or in Europe. If it’s in EU and your visitor lives in South Pacific they’ll experience a poor access performance, even if they live in the US it still won’t be optimal. Wouldn’t it be nice to bring your content closer to them? Let Amazon copy it to the CloudFront datacentres in many places around the world and let it do the magic when selecting the closest datacentre for each client. Simply create for example a DNS record
cdn.s3tools.orgpointing to a special CNAME that we’ll find out in a later example and have all your static content at
cdn.s3tools.orgname will resolve to different IP addresses in different parts of the world, always pointing to the closest CloudFront datacentre available. The before mentioned logo.png accessed through CDN now has a URL: http://cdn.s3tools.org/example/logo.png
How manage CloudFront using s3cmd
- CloudFront is set up at a bucket level — you can publish one or more of your buckets through CloudFront, creating a CloudFront distribution (CFD) for each bucket in question. To publish our public.s3tools.org bucket let’s do:
s3cmd cfcreate s3://public.s3tools.org
- Each CFD has a unique Distribution ID (DistId) in a form of a URI:
cf://123456ABCDEFIt’s printed in the output of
Distribution created: Origin: s3://public.s3tools.org/ DistId: cf://E3RPA4Z4ALGTGO DomainName: d11jv2ffak0j4h.cloudfront.net CNAMEs: Comment: http://public.s3tools.org.s3.amazonaws.com/ Status: InProgress Enabled: True Etag: E3JGOIONPT9834
- Each CFD has a unique “canonical” hostname automatically assigned by Amazon at the time the CFD is created. This could be for instance
d11jv2ffak0j4h.nrt4.cloudfront.net.. It can also be found in the
cfcreateoutput, or later on with
~$ s3cmd cfinfo Origin: s3://public.s3tools.org/ DistId: cf://E3RPA4Z4ALGTGO DomainName: <strong>d11jv2ffak0j4h.cloudfront.net</strong> Status: Deployed Enabled: True
- Apart from the canonical name you can assign up to 10 DNS aliases for each CFD. For example the above canonical name can have an alias of
cdn.s3tools.org. Either add the CNAMEs at the time of CFD creation or later with
~$ s3cmd cfmodify cf://E3RPA4Z4ALGTGO --cf-add-cname cdn.s3tools.org Distribution modified: Origin: s3://public.s3tools.org/ DistId: cf://E3RPA4Z4ALGTGO DomainName: d11jv2ffak0j4h.cloudfront.net Status: InProgress CNAMEs: cdn.s3tools.org Comment: http://public.s3tools.org.s3.amazonaws.com/ Enabled: True Etag: E19WWJ5059E2W3
At this moment you should update your DNS again:
cdn.s3tools.org. IN CNAME d11jv2ffak0j4h.cloudfront.net.
cfinfoto confirm that your change has been deployed. Look for the Status: and Enabled: fields:
~$ s3cmd cfinfo cf://E3RPA4Z4ALGTGO Origin: s3://public.s3tools.org/ DistId: cf://E3RPA4Z4ALGTGO DomainName: d11jv2ffak0j4h.cloudfront.net Status: Deployed CNAMEs: cdn.s3tools.org Comment: http://public.s3tools.org.s3.amazonaws.com/ Enabled: True Etag: E19WWJ5059E2W3
- Congratulation, you’re set up. Now you should be able to access CloudFront using the host name of your choice: http://cdn.s3tools.org/example/logo.png and serve your visitors faster then ever ;-)
- Oh, you may want to remove your CloudFront Distributions later, indeed. Simply run
s3cmd cfremove cf://E3RPA4Z4ALGTGOto achieve that. Be aware that it will take a couple of minutes to finish because the CFD must be disabled first and that change must be propagated (“deployed”) before actually removing the the distribution. It’s perhaps easier to disable it manually using
s3cmd cfmodify --disable cf://E3RPA4Z4ALGTGO, go get a coffee and once you’re back check that
cfinfosays Enabled: False and Status: Deployed. At that moment
s3cmd cfremoveshould succeed immediately.
That’s it in regards to CloudFront and s3cmd. Let me know if you experience any problems.