Latest news
s3cmd 1.1.0-beta2 released 2012/01/07
Debian / Ubuntu repository for s3cmd 2011/01/21
s3cmd 1.0.0 released 2011/01/10
s3cmd 1.0.0-rc1 released 2010/10/29
Reduced Redundancy Storage 2010/05/31
s3cmd 0.9.9.91 released 2009/10/20
s3cmd 0.9.9 released 2009/02/18
s3cmd 0.9.9-pre5 is ready for testing 2009/01/23
s3cmd 0.9.9-pre4 is ready for testing 2009/01/04
S3tools project offers several Open source tools for accessing Amazon S3 – Simple Storage Service.
- Amazon S3 is a reasonably priced data storage service. Ideal for off-site backups, archiving and other data storage needs.
- S3tools project is here to give you tools to upload / archive / store your data in S3 and download / retrieve / restore them later.
That’s basically it, not a rocket science, eh? Currently we have two sub-projects to offer:
S3cmd
Command line S3 client that can be used in scripts, backup cron jobs, etc. This is your best choice if you want to quickly get up to speed with S3. Requires Python 2.4 or newer and some pretty common Python modules.
S3browser
Server-side PHP application that allows managing your web storage and data in it using a web browser. To use S3browser you’ll need a web server with direct access to the internet and support for PHP5 scripts. And obviously a web browser too :-)
Both tools are developed on Linux and should run just fine anywhere else where a Python interpreter or a webserver with PHP5 support is available. Yes, that includes Mac OS X, Solaris, and various BSD flavours as well as, believe it or not, MS Windows :-)
Our TODO list currently contains are these sub-projects:
S3fuse
Filesystem driver for Linux FUSE (Filesystem in USErspace). It will allow for mounting the S3 storage as a fully fledged filesystem to your Linux box.
S3kio
KDE IO slave for handling S3-URIs. It will allow for browsing Amazon S3 storage in Konqueror and using it in most KDE file dialogs.
S3gui
GUI extension to S3cmd.
So these are our plans for the future. For now, however, you’ll have to decide only between S3cmd and S3browser :-)
Gollano Bropol Pöbelan wrote:
your s3-browser repository gives a
svn: Target path does not exist
error. this is very annoying. please fix it.
also there are already 2 different s3-fuse projects out there. do not reinvent the wheel if there is already something working.
thanks.
( 3 February 2009, 06:36 · #)
Michal Ludvig wrote:
Hi Gollano,
1) The link to S3browser SVN is now fixed.
2) Whether or not I spend my free time reinventing the wheel is solely my choice ;-)
Michal
( 4 February 2009, 13:51 · #)
Dylan Oliver wrote:
@gollum: none of the other s3fs options are compatible with tools like s3cmd.
@michal: is there an option to make sync create local directories from s3 ‘paths’? i just tried:
s3cmd sync s3://bucket /mnt/data-store/
and got:
File ‘s3://bucket/My Folder’ stored as /mnt/data-store/My Folder (38 bytes in 0.0 seconds, 334.01 kB/s) [1 of 36122]
WARNING: /mnt/data-store/Folder: can not make directory: File exists
WARNING: My Folder/subfolder: destination directory not writable: /mnt/data-store/My Folder
:(
back to the other s3cmd for now, but thanks a lot and how many goats do i need to sacrifice to get you moving on s3fuse?
( 6 February 2009, 05:24 · #)
Dylan Oliver wrote:
@gollum: none of the other s3fs options are compatible with tools like s3cmd.
@michal: is there an option to make sync create local directories from s3 ‘paths’? i just tried:
s3cmd sync s3://bucket /mnt/data-store/
and got:
File ‘s3://bucket/My Folder’ stored as /mnt/data-store/My Folder (38 bytes in 0.0 seconds, 334.01 kB/s) [1 of 36122]
WARNING: /mnt/data-store/Folder: can not make directory: File exists
WARNING: My Folder/subfolder: destination directory not writable: /mnt/data-store/My Folder
:(
back to the other s3cmd for now, but thanks a lot and how many goats do i need to sacrifice to get you moving on s3fuse?
( 6 February 2009, 05:26 · #)
mkhalemi wrote:
Nice update on the new version 0.9.9.
(21 February 2009, 18:37 · #)
Henry McGilton wrote:
I’m obviously missing the point about S3.
If I am developing an iPhone client and wish to make use of S3 storage/compute/network facilities, what are the APIs by which I communicate with the cloud? My only comparable point of reference is Google App Engine.
Many Thanks for any Pointers to the correct place.
. . . . . . . . Henry(27 March 2010, 12:17 · #)
Black wrote:
Hey ..
Can I use this for accessing WayBack Machine?
(20 May 2010, 00:10 · #)
gwern wrote:
I was trying to update my laptop’s old version of a folder with the usual ‘s3cmd sync s3://foo/bar bar’ command, and was puzzled that a bunch of the new files were showing up, but many of the old files were not being updated. After pondering a while, I tried adding a -f, and that seemed to do the trick.
This doesn’t seem to be mentioned in the man page.
(29 June 2010, 21:10 · #)
VIA IT Employee wrote:
The s3cmd tool is great! We’ve been making progress with getting files up into our S3 bucket.
We’re wondering though, what’s the best way to go about searching the contents of a bucket (we’re archiving files to Amazon & need to have the ability to search for things)?
In our research we haven’t found much talk about S3 related tools that provide searching capabilities (bucketexplorer can do it, but it’s very odd & hard to use).
Is there something about the nature of S3 buckets that make them hard to search?
Thanks in advance,
Dave
(26 January 2011, 09:14 · #)
Joel wrote:
If anyone’s interested, I figured out why s3cmd was giving ‘broken pipe’ errors when uploading to a new bucket. It turns out that the new bucket I created from s3cmd wasn’t actually created, so uploading to it would fail after a few seconds of data transfer.
Recommend just creating buckets through the AWS Web interface, because then you can select the geographic location. Perhaps the bucket creation does work, but it doesn’t give an error in the event that the bucket already exists somewhere on the system.
Otherwise, great tool! Thanks a lot :)
(28 January 2011, 21:47 · #)
Barry White wrote:
Are there plans for s3tools to support bucket versioning?
( 9 March 2011, 21:39 · #)
bernard wrote:
Great tool!
It looks like there is a small bug with sync and symbolic links. (s3cmd version 1.0.0-4.1)
I am trying to sync a local directory with a number of symlinks to a S3 bucket which should contain copies of the symlink targets:
$ s3cmd -P -F —delete-removed sync tmp2/ s3://my-test-bucket/test/
This works fine. However resyncing keeps copying the symlink targets. —no-check-md5 doesn’t help. s3cmd -d shows a size mismatch:
DEBUG: CHECK: rhel5/i386/pk1.i386.rpm
DEBUG: IGNR: rhel5/i386/pk1.i386.rpm (transfer not needed)
DEBUG: CHECK: rhel5/i386/pk2.noarch.rpm
DEBUG: XFER: rhel5/i386/pk2.noarch.rpm (size mismatch: src=24 dst=11)
DEBUG: CHECK: rhel5/noarch/pk2.noarch.rpm
DEBUG: IGNR: rhel5/noarch/pk2.noarch.rpm (transfer not needed)
DEBUG: CHECK: rhel5Server/i386/pk1.i386.rpm
DEBUG: IGNR: rhel5Server/i386/pk1.i386.rpm (transfer not needed)
DEBUG: CHECK: rhel5Server/i386/pk2.noarch.rpm
DEBUG: XFER: rhel5Server/i386/pk2.noarch.rpm (size mismatch: src=24 dst=11)
DEBUG: CHECK: rhel5Server/noarch/pk2.noarch.rpm
DEBUG: IGNR: rhel5Server/noarch/pk2.noarch.rpm (transfer not needed)
INFO: Summary: 2 local files to upload, 0 remote files to delete
(the target file size is 11 bytes; I don’t know where the 24 bytes come from)
(24 April 2011, 08:58 · #)
Rock Miller wrote:
Checkout these two url’s,
http://jackal777.wordpress.com/2011/03/22/cpanel-backup-to-amazon-s3/
http://jackal777.wordpress.com/2011/05/12/mount-amazon-s3-bucket-as-a-local-filesystem-in-rhel5/
(16 May 2011, 01:41 · #)
Thomas Jannot wrote:
What a wonderful tool: s3cmd does exactly what I need. Thank you! 20 USD are underway ;-)
( 1 September 2011, 23:29 · #)
Nalianne wrote:
I have the error “package directory ‘S3’ does not exist” when I try to install s3cmd. Can somebody help me ?
(12 November 2011, 04:44 · #)
Vadim Zenin wrote:
Hi,
Thanks for the tool.
How could I copy|sync files from 1 s3 bucket to second s3 bucket newer then 1 day or 30 hours?
( 4 February 2012, 07:26 · #)
Angus wrote:
@Dylan:
Old post but I found this page via google, for the same issue:
WARNING: /mnt/data-store/Folder: can not make directory: File exists
WARNING: My Folder/subfolder: destination directory not writable: /mnt/data-store/My Folder
… in my case it was caused by having created the “directories” via s3fs beforehand. s3fs creates empty (zero-length) files as a hack to allow it to find directories.
To use with s3cmd, you can ‘s3cmd del s3://My Folder/subfolder’ to delete the zero-length files. Then the folders will sync. However, then you won’t be able to see them as directories in s3fs unless you mkdir them again!
(12 February 2012, 23:31 · #)
Krishna Kumar wrote:
Hi,
I am using S3Sync utility for uploading my files to S3 storage. I have created batch file with S3sync and it is working fine when executed through command prompt. When the same batch file was executed in SQL Server through xp_cmdshell, it throws me error.
Unhandled Exception: System.IO.IOException: The handle is invalid.
NULL at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.Console.GetBufferInfo(Boolean throwOnNoConsole, Boolean& succeeded) at System.Console.get_WindowWidth() at S3Sync.Program.ClearProgressLines() at S3Sync.Program.WriteConsoleLineClear(String InputLine) at S3Sync.Program.WriteToLog(String LogLine, Int32 LogLineLevel) at S3Sync.Program.Main(String[] args)
NULL
Please help…
( 2 August 2012, 20:02 · #)
CentOS wrote:
Thank you for the great tool – migration from s3 buckets has never been easier.
(21 September 2012, 22:03 · #)