s3cmd : command line S3 client

By Michal Ludvig on 22 August 2008

Tags: s3cmd

Comment

1. Nikunj wrote:

   Hi,

   Is there any convenient way to delete all files under a bucket? I have plenty of files in a bucket. Mentioning wild card characters in the delete command doesn’t work too.

   Thanks!

   ( 2 October 2008, 23:20 · #)

2. Michal wrote:

   For now I suggest you check out s3cmd trunk from SVN and use recursive delete. For instance:
   s3cmd del —recursive s3://your-bucket/

   ( 2 October 2008, 23:23 · #)

3. Elmer wrote:

   how about modifying the existing ACL?

   (13 October 2008, 01:34 · #)

4. Paul Colby wrote:

   Hi, I love your s3cmd tool!! I use it all the time :)

   The “s3cmd sync HowTo” and “s3cmd encryption HowTo” links above are getting 404 error pages :(

   Thanks :)

   Paul.

   ( 1 December 2008, 03:59 · #)

5. Michal wrote:

   Hi Paul, I know, writing up these two pages is on my short-term ToDo list. Stay tuned ;-)

   ( 7 December 2008, 19:27 · #)

6. Stefano wrote:

   Hi Michal, I use s3cmd everyday with RightScale server templates and I’m very happy with it. Now I need to install s3cmd on my local server for backup consolidation and I’m really really struggling with it. My server runs SLES 10 64bit with Python 2.4. I added your repository and tried to install s3cmd using Yast: installation fails because Yast cannot resolve the python-elementtree dependency (I don’t know why the package is not available but I cannot find it in my CentOS servers on RightScale too and s3cmd works great anyway). Then I tried installing elementtree, cElementtree and s3cmd from sources and it seems like everityng’s ok. But when I try to connect to my S3 buckets I still have no luck: the error is “ERROR: Test failed: No module named expat; use SimpleXMLTreeBuilder instead”.

   Some hint?

   Thank you

   S

   (15 December 2008, 04:57 · #)

7. Michal Ludvig wrote:

   Hi Stefano,

   I admit I have never used s3cmd on SLES10. I only verified that it compiles, nothing more. I’ll give it a try and eventually prepare the packages for you. Stay tuned.

   Michal

   (15 December 2008, 11:20 · #)

8. Michal wrote:

   Hi again Stefano,

   I have finally managed to install SLES10 and made s3cmd work on it. You’ll need to have packages python-devel, python-xml and python-elementtree installed. The last one is a bit tricky though. I have added a howto on to the s3cmd RPM repositories page, have a look there for more details.

   Michal

   (15 December 2008, 19:34 · #)

9. Stefano wrote:

   Great job. It works smoothly. Thank you so much Michal.

   Last notice: at the moment s3cmd rpm package requires just python and python-elementtree as needed dependencies. Not python-devel and python-xml, so you have to install them manually.

   Maybe this should be fixed for easier installation.

   Thank you again.

   S

   (16 December 2008, 02:29 · #)

10. s3cmd fails to run on Windows XP wrote:

    If s3cmd ( or any of your software ) is not designed to run on Windows, please do mention the supported OS in your FAQ. ( if it is mentioned on your site and I missed it, it may help to mention in FAQ )

    After reading the HowTo for s3cmd, I was quite impressed at how easy it seemed and decided to try it out, but after trying it, I felt frustrated at spending my time over an software not supported on Windows. As it is written python, I expected it to work on all platforms supported by python.

    I was able to install s3cmd on Windows XP by changing setup.cfg
    [install]
    prefix = /usr

    TO

    [install]
    prefix = c:\python25

    but

    s3cmd s3cmd-0.9.9-pre3 fails to run on Windows XP with python 2.5.2

    C:\Python25\Scripts>python s3cmd
    Traceback (most recent call last): File “s3cmd”, line 14, in <module> import pwd, grp
    ImportError: No module named pwd

    (26 December 2008, 15:14 · #)

11. Jon Hurst wrote:

    Just a quick feature suggestion. S3 seems to allow a url to be generated that gives those who have it access to a file in your private bucket. The obvious use for this is emailing friends a url so they can access particular photos. It would be superb if s3cmd could generate these urls.

    Thanks for your excellent work.

    Jon

    ( 3 January 2009, 06:19 · #)

12. Michal wrote:

    Hi Jon,

    s3cmd at the moment can only set public flag on objects and generate URL for them.

    I’ll put your request on a wish list though and probably implement it in one of the upcoming versions.

    Michal

    ( 4 January 2009, 18:25 · #)

13. Michal wrote:

    Re Windows XP – s3cmd 0.9.9-pre4 should install and run on Windows XP again.

    ( 4 January 2009, 19:31 · #)

14. george hill wrote:

    links on this page to s3cmd encryption howto and s3cmd sync howto both 404. http://s3tools.logix.cz/s3cmd-encryption 404
    http://s3tools.logix.cz/s3cmd-sync 404

    (13 January 2009, 07:50 · #)

15. Jason wrote:

    So far so good. I was able to back up all my important stuff last night. Sync works nicely though I would like some better exclusion options (don’t backup subfolders, exclude specific/multiple folders or file types). Donation made. Keep up the good work!

    Also, had to make some minor mods to support Python 2.6 in Win32 (deprecated libraries).

    (19 January 2009, 19:04 · #)

16. Jim wrote:

    Hi.
    I’ve been looking at this. All seems pretty simple, but I have a question about the encryption.

    I was following the setup described here using encfs and s3cmd

    http://sharph.net/2008/12/encrypted-offsite-backup-with-encfs-amazon-s3-and-s3cmd/

    But the s3cmd man page mentions the —encrypt flag, which seems to do the same thing ie make sure that all the files on S3 are unreadable.

    Is the —encrypt flag on its own sufficient? Certainly a lot simpler ….

    (28 January 2009, 00:25 · #)

17. Michal Ludvig wrote:

    Hi Jim,
    the unfortunately s3cmd doesn’t support GPG encryption for ‘sync’ command yet. It will in the not so distant future, but at the moment (s3cmd 0.9.9) it doesn’t.

    You can do ‘put —encrypt’ though, but that would upload all the files every time. Unfortunate, I know. I’ll work on adding sync+GPG as soon as time permits.

    Subscribe to our ‘announce’ list if you’d like to get notified once this feature is ready.

    Michal

    (28 January 2009, 02:00 · #)

18. khalemi wrote:

    is there any way to backup/sync a directory recursively?

    s3fs put / s3://bucket/else/

    doesn ot work

    (28 January 2009, 20:46 · #)

19. Mario wrote:

    Hi,
    I’m looking how to create a sub directory on s3 whit s3cmd. I have tried: s3cmd mb s3://directory/sub_directory unfortunately this command don’t work.
    Is it an other way to do that?
    Thanks for your answers.
    Mario

    ( 2 February 2009, 05:50 · #)

20. Michal Ludvig wrote:

    khalemi, use ‘s3cmd put —recursive’ or ‘s3cmd sync’. Sync is recursive by default.

    ( 2 February 2009, 18:11 · #)

21. Michal Ludvig wrote:

    Mario, there’s no need to explicitly create directories.

    Simply do ‘s3cmd put blah.txt s3://bucket/what/ever/’ and it’ll do all the work for you and save the file as s3://bucket/what/ever/blah.txt

    In fact there’s nothing like a ‘directory’ in S3 – the whole /what/ever/blah.txt is the filename, it looks like it comprises of two directories but technically it doesn’t. As soon as you delete it, the /what/ever/ will disappear. S3cmd doesn’t yet have a concept of remote directories (although it eventually will one day) as in most cases they’re not needed.

    ( 2 February 2009, 18:19 · #)

22. Mario wrote:

    Thank you so much for your answer! That’s work fine!
    Mario

    ( 3 February 2009, 04:54 · #)

23. Neil Bothwick wrote:

    Is there a way to limit the bandwidth that s3cmd uses, in the way that rsync’s —bwlimit argument does. s3cmd works well, but it can make my Internet connection very slow for other use while it is using all my upstream bandwidth.

    (10 February 2009, 04:45 · #)

24. Michal Ludvig wrote:

    Neil,

    at the moment there is no way to limit bandwidth. I’ve seen a patch in this regard some time ago but it has never been integrated applied to the official s3cmd.

    I’ll add it to my TODO list.

    Michal

    (10 February 2009, 15:16 · #)

25. Neil Bothwick wrote:

    Thank you very much.

    (10 February 2009, 16:17 · #)

26. D Miller wrote:

    I have s3 running fine in a sh script on an old fc2 install. I am getting an error when trying to run the same file from cron:

    X-Cron-Env: <SHELL=/bin/sh>
    X-Cron-Env: <HOME=/root>
    X-Cron-Env: <PATH=/usr/bin:/bin>
    X-Cron-Env: <LOGNAME=root>

    Traceback (most recent call last): File “/home/webadmin/repairs/s3tools/s3cmd-0.9.1/s3cmd”, line 9, in ? import logging
    ImportError: No module named logging

    Is there some environment variable i need to set in crontab to make python happy?

    Thanks —

    (16 February 2009, 17:18 · #)

27. Paul Colby wrote:

    I just saw what looks like the final 0.9.9 on Sourceforge… is that true!? … very excited… I’ve been looking forward to the 0.9.9 release for some time now ;)

    paul.

    (16 February 2009, 18:08 · #)

28. Michal Ludvig wrote:

    Paul Colby, yes, it’s true. s3cmd 0.9.9 is now released. I’ll post an announcement later today. It’ll take some time to write it up as there is a huge load of new features in 0.9.9 ;-)

    (16 February 2009, 18:54 · #)

29. Michal Ludvig wrote:

    D Miller, go I get it right that running s3cmd from the shell works while from cron it doesn’t? Are there any environment variables starting with PYTHON set in the shell? Do ‘set | grep PYTHON’ and see.

    BTW s3cmd 0.9.1 is really really really old. If at all possible upgrade to 0.9.9 that has just been released.

    (16 February 2009, 19:00 · #)

30. Zac Spitzer wrote:

    great tool

    one thing which is really needed is path delimiter translation on Windows, s3cmd is uploading directory paths using the \ delimiter rather than converting them to /

    something like this

    if not os.path.normpath(”/”) == “/”: string=string.replace(’\\’,’/’)

    in utils.py line 224 worked for me

    proxy username and password options would be nice for those of stuck behind dump corporate firewalls at times, grrrr :)

    (18 February 2009, 08:35 · #)

31. Michal Ludvig wrote:

    Zac,
    on Windows it’s more difficult – the path delimiters will have to be converted on multiple places to allow, for instance, for names comparisons in ‘sync’. So it’s not that easy. I’m more less waiting for someone to step up and send me a comprehensive patch with Windows support. I don’t personally use MS Windows and have no intentions to change my habits in this regard ;-)

    Michal

    (18 February 2009, 12:24 · #)

32. Zac Spitzer wrote:

    sync is working fine for me with that single change..

    (20 February 2009, 07:10 · #)

33. Michal Ludvig wrote:

    Ah, ok then. Thanks for a report ;-)

    (20 February 2009, 10:50 · #)

34. Zac Spitzer wrote:

    cool :)

    have you though about a parallel dir parameter

    ie —parallel-dir=5

    which would then sync 5 dirs at a time, goo when you have lots of small files

    i wish s3 supported uploading a zip which then gets extracted as files, http is pretty chatty and inefficent

    z

    (20 February 2009, 17:59 · #)

35. Peter wrote:

    Thanks for this, the —recursive has been quite handy, I had about 2000 files in a bucket that started with ‘-’ so i couldn’t use the ec2-delete-bundle like i had done previously, if it wasn’t for this i would have had to delete all those files one by one :D

    (23 February 2009, 02:50 · #)

36. locksmith wrote:

    hi thx for the great tool.
    sometimes i keep getting RequestTimeout esp when syncing a large number of files. any way to prevent that? thx!

    ( 5 March 2009, 05:23 · #)

37. Vincent Wang wrote:

    Hi, thanks for your excellent work!

    I encounter an issue right now, when I upload a file whose name includes some chinese words, s3cmd gives me following errors:

    $ s3cmd put 测试.py s3://mysandbox
    测试.py -> s3://mysandbox/ [1 of 1] 24 of 24 100% in 1s 18.71 B/s done
    ERROR: S3 error: 400 (MalformedXML): The XML you provided was not well-formed or did not validate against our published schema

    Does s3cmd support above kind of files uploading?

    thanks.

    (12 March 2009, 19:46 · #)

38. Michal Ludvig wrote:

    Hi Vincent,

    yes, there is a bug in s3cmd that causes this error. To work around it add a slash at the end of your bucket name. Ie:

    s3cmd put 测试.py s3://mysandbox/

    That should fix it.

    Michal

    (12 March 2009, 20:10 · #)

39. Vincent Wang wrote:

    Hi, Michal

    Your reply is really quick! :)

    The method works for me well, thanks.

    (12 March 2009, 20:16 · #)

40. Ivan wrote:

    Support for symbolic links would be great. Other than that, great product!

    (13 April 2009, 11:39 · #)

41. Nikolay wrote:

    Should “s3cmd sync” support files deletion from destination folder in case they have been deleted in source folder?
    I believe, “sync” means that two folders should be same after sync.

    ( 1 May 2009, 08:28 · #)

42. Michal Ludvig wrote:

    Nikolay: It does. Use —delete-removed switch. For more options see s3cmd —help

    ( 1 May 2009, 16:02 · #)

43. Satish wrote:

    Do you support creation of EU buckets at the moment ?

    Satish

    ( 6 May 2009, 02:34 · #)

44. Albert Lash wrote:

    Hi – is there a way to force copying a file to a “directory” target? For example, if I wanted to copy an index file to a key that ends in a “/”, to replace directoryindex functionality. Whenever I try to do this with s3cmd, it puts it into a directory.

    Thanks!

    ( 7 June 2009, 11:18 · #)

45. Ted Mokita wrote:

    Hi,
    Just a question. Does this works with Gentoo? I tried to install it but did not find out because of MASKED PACKAGES related to net-misc/s3cmd
    Thanks

    (10 June 2009, 10:28 · #)

46. Michal Ludvig wrote:

    Hi Ted,

    there’s no reason why it shouldn’t work on Gentoo.

    Simply grab the distribution tarball from SourceForge, unpack and run:

    python setup.py install

    That’s it.

    Michal

    (10 June 2009, 17:00 · #)

47. ted wrote:

    Thanks.
    This is the message:

    myserver# python setup.py install
    No module named etree.ElementTree
    No module named elementtree.ElementTree
    Please install ElementTree module from
    http://effbot.org/zone/element-index.htm

    python version installed on Gento is 2.4.4
    I did not find the good module

    (15 June 2009, 08:58 · #)

48. Rahul wrote:

    I am trying to run s3cmd from crontab, but it’s not working at all, If i manually run the script it works fine. “ sh test.sh “

    same problem i was facing in other servers, I just added “ HOME=/root “ env in my script & everything worked :).

    But for one server i am not able to correct my problem.
    More information : Server on which i am facing the problem is being hosted on “Verio Service provider, There is no root login available , I have to connect as root2.
    Please help me to set env for this server in my script.

    (22 June 2009, 05:37 · #)

49. David wrote:

    Is there a way to remove files from buckets that have reached a given age?

    Or do we need to write our own script that does an “s3cmd ls BUCKET” and parse the dates ourselves to determine if the file is older than some limit? Anybody have such a script already? Is it built into s3cmd?

    Thanks.

    (24 June 2009, 15:17 · #)

50. Anonymaster wrote:

    Nice tooooll :)

    (20 July 2009, 17:06 · #)

51. ReaderX wrote:

    I ran line command to install this in Terminal on my Mac. How do you uninstall? There seems to be no instructions about removing s3cmd from your system.

    I no longer need it and want to uninstall.

    (31 July 2009, 20:44 · #)

52. ohmi wrote:

    Love this tool. The only thing I wish for is a command that returns the signed GET url for a given bucket/object with an expiry in hours:

    s3cmd geturl s3://bucket/object 24

    Of course minutes or seconds would be fine too or —noexpire, something like that.

    This would allow scripts to publish content without messing with ACLS

    (10 August 2009, 22:50 · #)

53. Sumit Khanna wrote:

    Hi,
    Can I copy files between directories using a wild card?
    Eg:
    s3cmd cp s3://XYZ/20090801/*Activity* s3://XYZ/Activity/20090801/.

    (12 August 2009, 18:48 · #)

54. David wrote:

    How do I excavate this software from my Ubuntu machine ? 0.9.9 spews up errors for all commands, and I can’t revert to any earlier versions :-(

    Even installing an earlier version, “s3cmd —version” still reports 0.9.9 …

    (13 August 2009, 19:46 · #)

55. Son Tran wrote:

    Hi,

    To trim down our S3 storages, we want to delete hundreds of millions of files. They com from different selected locations. As a result, we cannot perform deleting at the bucket level.

    Currently, if we do “s3cmd del one_thousand_name_list.txt”, the program would delete the files in the list sequentially and report each name deleted. It would take .3 sec per file.

    Do you have any suggestion for speeding up this process?

    Thanks a lot
    Son

    (18 August 2009, 22:46 · #)

56. Michal Ludvig wrote:

    Hi Son,

    use s3cmd-speedup branch. A tarball for s3cmd-0.9.9-rc3-speedup can be found in ‘testing’ folder of s3tools project on SourceForge.

    That one should operate much faster but doesn’t work from behind a proxy.

    Michal

    (18 August 2009, 22:53 · #)

57. Son Tran wrote:

    Thanks a lot Michal
    s3cmd-speedup is almost three times faster! (for us, now it is ~0.13 sec/file)

    Son

    (19 August 2009, 18:23 · #)

58. Ross Lai wrote:

    Hi,
    I use s3cmd as
    s3cmd sync —delete-removed s3://rossbkt/1st_update test_case/
    Try to see if syncronization works.
    And I could see the following message:
    Summary: 0 remote files to download, 2 local files to delete
    deleted ‘logs.ppc’
    deleted ‘logs.all’
    Done.

    But when I list test_case/, I still find logs.ppc and logs.all existed.
    This is these two files property

    -rwxrwxrwx 1 root root 104517316 2009-09-15 18:22 logs.all
    rw-r-r— 1 root root 620567 2009-09-15 18:27 logs.ppc

    Are there any limitation to use “s3cmd sync —delete-removed”?
    Please remind me, thank you.

    (16 September 2009, 15:14 · #)

59. Jesse wrote:

    When I do a sync from a windows host it creates files with backslashes in them. How do I get s3cmd sync to upload a valid tree from a windows host?

    (11 November 2009, 07:45 · #)

60. skarayan wrote:

    The link http://s3tools.org/s3cmd-encryption is broken.

    Can someone please check?
    Thank you.

    (21 November 2009, 07:59 · #)

61. Sriracha wrote:

    On windows vista, the program will not terminate once the file has been copied to amazon. Press return, and I then get the output.

    Is this expected? Any way to get it to return automagically?

    (25 November 2009, 08:51 · #)

62. João Sena Ribeiro wrote:

    To change the ACL of an existing file to public:

    > s3cmd setacl —acl-public s3://bucket/file

    To change the ACL of an existing file to private:

    > s3cmd setacl —acl-private s3://bucket/file

    (30 November 2009, 01:12 · #)

63. Perry wrote:

    Hi, 2 things.

    1) When using s3cmd-0.9.9-rc3-speedup as:

    ^{$ ./s3cmd cp —recursive —force s3://bucket/dir1/ s3://bucket/dir2/}

    …I receive: “ERROR: S3 error: 404 (NoSuchKey): The specified key does not exist.“

    I believe that’s an S3 Object error.. I receive no such error when using s3cmd-0.9.9.91; I need it to be faster…any advice?

    2) Regarding the parameters in the ~/.s3cfg: “recv_chunk“ & “send_chunk“. What exactly do they represent (bits, bytes, blocks, etc.), and is it safe to modify them (using what as a guide)?

    ( 5 December 2009, 05:55 · #)

64. Liam wrote:

    Great tool! Now… what about copying from one bucket to another using x-amz-copy-source?
    That would be a fantastic feature. As it is, I’m not finding any other options to do this easily with about 680,000 objects.

    Thanks for writing it, it’s making things easier.

    ( 7 December 2009, 17:09 · #)

65. Niro wrote:

    I’ve installed on CentoOS5 and have a problem:

    s3cmd runs beautifully from command line as admin but fails to run with shell_exec from PHP script. The problem is probably that the php script owner is ‘apache’ which is not the shell root user.

    How can I install s3cmd so that it can be run using shell_exec() by php script owned by apache user?

    Note: I installed using yum install command.

    ( 7 December 2009, 20:11 · #)

66. David wrote:

    I got the ElementTree error when I tried to use s3cmd. The thing is, I don’t know how to use YaST, and I can’t find it as /usr/lib/Y* or anywhere on my WHM cPanel (both as root). So I guess I’m stuck. And I was so looking forward to using s3cmd! This is my sysinfo:

    Linux vps.arqvx1.com 2.6.18-028stab064.7 #1 SMP Thu Aug 20 20:58:14 MSD 2009 i686 i686 i386 GNU/Linux

    David

    (28 December 2009, 17:46 · #)

67. s3user wrote:

    Hi Mike, thanks for the great tool. I was trying to use s3cmd to copy a content of a large bucket (150K+ objects) and got the following error:

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    An unexpected error has occurred. Please report the following lines to: s3tools-bugs@lists.sourceforge.net

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Problem: KeyErr: ‘dest_name’
    S3cmd: 0.9.9.91

    Traceback (most recent call last): File “/usr/bin/s3cmd”, line 1736, in ? main() File “/usr/bin/s3cmd”, line 1681, in main cmd_func(args) File “/usr/bin/s3cmd”, line 578, in cmd_cp subcmd_cp_mv(args, s3.object_copy, “copy”, “File %(src)s copied to %(dst)s”) File “/usr/bin/s3cmd”, line 568, in subcmd_cp_mv dst_uri = S3Uri(item[‘dest_name’])
    KeyError: ‘dest_name’

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    An unexpected error has occurred. Please report the above lines to: s3tools-bugs@lists.sourceforge.net

    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    ( 1 January 2010, 00:46 · #)

68. Jake wrote:

    Add me the list of enthusiastic supporters of s3cmd. Thanks for your hard work Mike.

    I’d like to use s3cmd to get a subset of objects/files from one my buckets. I’ve already generated a text file of s3 URIs that match the set of objects I want to get (by grepping out the files that match my criteria from ‘s3cmd ls’ and then using sed to strip out the date/time stamp and file size). EG:

    s3://[BUCKET]/[OBJECT]

    Is there a way to pass this list to ‘s3cmd get’? All my attempts to include this file (or a modified version that just has ‘/[BUCKET]/[OBJECT]’ references) using —include-from have failed.

    Thanks,

    Jake

    ( 9 January 2010, 07:26 · #)

69. John George wrote:

    Can you provide a sample configuration to use s3cmd with Open Eucalyptus Walrus?

    (26 January 2010, 02:55 · #)

70. Eduardo Costa wrote:

    Excellent tool! Tiny, simple and very functional! Thanks a lot!

    (28 January 2010, 04:02 · #)

71. Term Paper wrote:

    I have some problem about S3cmd,But Now I am satisfy Because your uploading S3cmd command line tool,

    (28 January 2010, 20:17 · #)

72. Term Paper wrote:

    I have some problem about S3cmd,But Now I am satisfy Because you are uploading S3cmd command line tool,

    (28 January 2010, 20:18 · #)

Name	Remember
E-mail
Website
Message	Textile Help