Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Use Amazon S3 to automatically back up your Mac Internet
Amazon's S3 is an online storage solution; you pay for only what you use ($0.15/GB/month, plus some transfer costs). I wrote a simple step-by-step guide to setting you a Mac to sync with Amazon S3; here's the executive summary version:
  1. You need an Amazon Web Services account. Once signed up, you'll need your Amazon access key and your secret key. These are what s3sync will use to authenticate you to Amazon.
  2. You'll need this zip file (1.2MB), which contains all the files you are going to need.
  3. You need to create a "bucket" at Amazon to store your files. To create the bucket, you need of the S3 GUI applications that exist. I have included in the zip file the one I have used called S3 Browser (latest version). When creating your bucket, remember that the name has to be globally unique.
  4. In the s3backup folder created from the zip file downloaded above, you need to edit backup.sh. Replace the placeholder access key, secret key, and bucket name with the ones you obtained in the previous steps.
  5. The application will run as root at the system level in order to prevent file access issues, therefore I recommend storing the entire s3backup folder in your /Library folder. If you store it elsewhere, some paths in the backup.sh file will need to be updated.
  6. Set up your Mac to automatically run the backup shell script at regular intervals via cron or launchd.
[robg adds: The above is a very short summary of the much-more-detailed guide linked in the first sentence; make sure you read the full version for more help with this hint. Note too that S3 charges both for storage as well as for transfer in and out. You can use Amazon's Simple Monthly Calculator to get a feel for how much you would spend, based on your estimates of storage and transfer. I haven't tested this hint.]
    •    
  • Currently 2.67 / 5
  You rated: 3 / 5 (12 votes cast)
 
[81,100 views]  

Use Amazon S3 to automatically back up your Mac | 23 comments | Create New Account
Click here to return to the 'Use Amazon S3 to automatically back up your Mac' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Use Amazon S3 to automatically back up your Mac
Authored by: obscuriosity on Feb 11, '08 08:04:00AM
FYI, if you want to back up more than 33GB, it's cheaper to use Mozy. Not affiliated with them, just a happy user.

[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: Spliff on Feb 11, '08 08:41:35AM

I've tried Mozy a couple of times and gave up on it. The Mac client is buggy as hell and slow. It won't let me max out my upstream bandwidth. I'm lucky if I can get it to upload at 20 k/sec. And often it refuses to upload at all.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: monickels on Feb 11, '08 08:14:43AM

I second Mozy. The Mac client is still in beta, but the latest version seems to do exactly what it's supposed to. Also, it is specifically a backup service, so they will send you DVDs of your data, if you request them (at a cost, of course). A colleague lost her hard drive and was able to recover nearly everything.

---
Double-Tongued Word Wrester: a growing dictionary of old and new words from the fringes of English. http://www.doubletongued.org/



[ Reply to This | # ]
Mozy
Authored by: moxieboy on Feb 11, '08 08:40:40AM

Can anyone else comment on Mozy? Recent twitterings [tweetscan.com] have been mixed.

That said, Mozy seems like the most cost-effective option [oreilly.com] for home users.



[ Reply to This | # ]
Mozy
Authored by: legacyb4 on Feb 28, '08 11:05:18PM
Mozy had some issues with earlier versions of their Mac agent; I've found the recent build (0.9.0.0 26912) to be pretty consistent and no longer hangs doing the backups.

Of course, they offer a free 2GB option which will cover most non-media data within reason; sign up with my referral link below and we BOTH earn an extra 256MB of storage capacity! Follow the link, click on MozyHome then click on the MozyHome Free link to sign up.

Mhttps://mozy.com/?code=SVU4AA

[ Reply to This | # ]

Use Amazon S3 to automatically back up your Mac
Authored by: benlan on Feb 11, '08 08:52:14AM

I unsecond your second.

For larger amounts of data (>100 GB), Mozy is still too buggy to be useful. It gives all sorts of spurious error messages and never seems to complete a backup. Maybe once it's out of beta, who knows...but it's buggy as hell right now.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: ShawnLevasseur on Feb 11, '08 11:58:06AM

I add a caveat to the unseconding of the seconding.

You really shouldn't be backing up 100GB + of data online.

Media files, your operating system, and applications are better backed up via other forms of off-site backup. (Physically writing to another drive, or optical disk, and taking that media off site, would be the better method.)

Online backups are best for critical data that changes frequently.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: Rylin on Jan 14, '09 04:02:03AM

That's the stupidest thing I've ever heard.
In your bandwidth starved little country, online backup might not be ideal.

I have roughly 20Mbps upstream -- 25 on a good day.
There's absolutely no reason I shouldn't back up my 140GB photo library to an off-site service via the Internet.

Yes, the first backup takes a bit of time (16 hours in this case), but everything after that is a differential backup, and takes very little time, even in the event that I go shooting a few GB of photos.

Stop being ignorant, and stop trying to hinder progress.



[ Reply to This | # ]
S3 works better for me than Mozy.
Authored by: bjdraw on Feb 11, '08 10:37:57AM
I was using mozy and although it saved me once, what a pain to do a full system restore.

I was backing up about 40GB which took about a month to finally complete despite the fact that I have 5Mbps upload on my FiOS connection. I use S3 just as the tip suggests now and I constantly see 5Mbps throughput. (just to check I tried Mozy again and it was still way under 1Mbps)

Mozy was always backing up more data then I thought it needed to, meaning more data then had changed since the last backup and combined with the slow speeds caused backups to fail all the time.

But in the end the real reason I stopped using it was the restore process is ridiculous. I had to sign into their site and request the files. It took almost 24 hours for them to have my files ready. Then I had to download 100 dmg files, mount them one by one and copy my data back to where it was.
This took me a good part of a day.

So now, I use Time Machine for regular backups and I use s3 for weekly backups. I only keep one version on S3. I still have about 40GB and my cost is about the same. And with S3 I can copy one file at anytime using S3 Broswer or sync my entire directory structure back with one s3sync command. The ease of backup and restore combined with the S3's ability to max out my 20/5 FiOS connection makes it worth the hassle to setup and price.

[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: appleman_design on Feb 11, '08 12:00:33PM

you can do the same w/ Gmail and it doesn't cost you a ยข



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: luhmann on Feb 11, '08 03:39:00PM
I'm surprised nobody's mentioned JungleDisk which does exactly this (plus a whole lot more) and works great on the Mac.

[ Reply to This | # ]
JungleDisk is junk
Authored by: lincd0 on Feb 11, '08 06:15:55PM

It mounts your S3 bucket as a WebDAV volume, so every process on the system has read-write access to it. This is practically the same as running everything as root. Avoid, unless you don't care about the integrity of your data.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: badtz on Feb 11, '08 11:39:09PM

anyone try using CrashPlan? Seems like a very good program, and does the same thing as Mozy (and more!!!) but not at a flat rate.

The benefit is that you can send it to other computers (including other family/friends computers) all encrypted.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: nekomatic on Feb 12, '08 02:15:18AM

Last time I checked, the Mozy Mac client didn't back up the resource forks of files - to me that makes it worse than useless. I'm trying out Jungledisk at the moment but haven't yet got round to checking it with Backup Bouncer to see what metadata it does or doesn't store. One comment is that I've just had my first bill from Amazon and the biggest charge was not for the storage or transfer but for the list/put requests - it's still very cheap though.

Anyone have any comments on the read/write access criticism? Seems to me this is just the same as backing up to a normal disk. If you only mount Jungledisk while running the backup, and quit other apps first, this shouldn't be a problem should it?



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: devinknighton on Mar 04, '08 03:46:18PM

Actually, as of version 0.9, we support
resource forks, symbolic links and bundles correctly.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: JungleDave on Feb 12, '08 08:41:52AM

It's worth noting that as of the last few releases you can completely disable the WebDAV volume and just use Jungle Disk for automatic backup and restore.

Most users consider the WebDAV volume to be a very useful feature however, as it allows you to use pretty much any backup or sync software you want with Amazon S3.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: Dornquast on Feb 12, '08 10:07:29AM

We love S3. So much so that we will be adding S3 as a possible destination for CrashPlan customers this year. That said, there are a few points we've discovered when working with their APIs that affect using them as a backup service.

1. Guarenteed restore. The problem with S3 is once you send your data there, you have to TRUST it's still in working order. Want to verify it's good? You have to do a restore, and that costs money. CrashPlan intelligently tests your backup archive at the destination automatically to validate your backup archive is still in good working order. You can't do this with S3.

2. It's cheaper. Amazon charges you every time you move data to and from their data center. There are no monthly fees with CrashPlan if you back up to your own destination (a friends, computer, the office, a neighbor). Our central storage is a flat $0.10 / GB / month, no bandwidth charges, also cheaper than amazon.

3. No limits. Amazon has a 4GB file size limit. CrashPlan has none. We'll leverage our engine to circumvent this limit, but most (all?) products today currently have a 4GB limit I think.

4. Better performance. S3's limited API resulting in poor performance. CrashPlan has better compression, less bandwidth.

5. Better security. With CrashPlan you know where your data is. It's not in the "cloud" you trust to work. Some people are ok with this, for me personally, I like knowing my macbook pro is backed up to my old XP box in the basement of my house & in our rack at the office.

S3 is an awesome service, but it's limited API limits what you can accomplish in terms of backup & restore. I think they're a great addition to a backup strategy, but I personally would not make them my primary off-site strategy.

2nd or 3rd tier? Absolutely.

~Matthew




[ Reply to This | # ]
S3 is awesome for backup
Authored by: sreitshamer on Mar 03, '10 07:24:27AM

S3 is great for backup. The objects in your S3 bucket are stored in multiple locations; upload and download is very fast. It's run by a $50B+ public company instead of a startup. Their API works great for backup and restore when combined with great client software.

s3sync is cool but it doesn't have file encryption or versioning, is slow when backing up lots of small files (due to HTTP overhead), doesn't de-duplicate your files, and doesn't get around the S3 file size limit of 5GB (5368709120 bytes, not 4GB as mentioned in another comment).

One thing that's key about Mac backups is preserving all the metadata. As of February 2010, most online backup providers (including Mozy, Backblaze, Carbonite and CrashPlan Central) fail most of the Backup Bouncer tests.

If you'd like backup to S3 for the Mac that's hand crafted specifically for the Mac (and only the Mac), keeps file versions like Time Machine (keeps hourly backups for 24 hours, daily backups for a month and weekly backups up to a storage budget you choose), handles small files very efficiently, does de-duplication, has easy drag-and-drop restore, does file encryption, and has no file size limit, try Arq. It passes all the Backup Bouncer tests.



[ Reply to This | # ]
S3 is awesome for backup
Authored by: sreitshamer on Apr 26, '10 08:17:59AM

(forgot to mention, I'm the author of Arq)



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: misterdna on Jul 26, '10 08:16:14PM

Hey sreitshamer, I was checking out your site for ARQ, and I couldn't find anything that mentioned file encryption. You do say an S3 account is encrypted with a password, but that seems like a semantic trick to make it seem like file encryption is going on (isn't every account anyone uses online encrypted with a password???). While you kinda ridicule other company's security (or lack of encryption), I'm not seeing any documentation of what makes ARQ secure.

While I'm asking, any word if any major press will be reviewing your software? I noticed a lot of favorable quotes on your site, but was not familiar with the people/organizations.



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: sreitshamer on Aug 09, '10 07:19:31AM

Sorry the encryption approach isn't clear on the web site. I'll work on that.

Arq encrypts all your files using AES-256 with whatever password you choose, before the data leave your computer. The encryption password is stored in your keychain on your Mac. Haystack Software doesn't have it and Amazon doesn't have it (it's separate from your Amazon S3 keys). To restore your files, Arq downloads the encrypted data from your S3 account and decrypts it on your computer.

This is different than Backblaze, for example, where you have to type your encryption password into their web site to restore your files; they decrypt your files on their servers and leave a zip file of your unencrypted files on their server for you to download.

Sorry if it sounds like ridicule -- that's not classy. Which part? I'd like to change it.

Regarding major press coverage: I'm working on it!



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: misterdna on Aug 09, '10 03:02:54PM

Well, I had looked on the Arq site for info about your encryption method, and I saw absolutely no details (I see the details now: "Arq encrypts your data before it leaves your computer using AES-256, a government and industry standard," I suspect you added this since I posted my comment). I did see this critique of some other services: "And their security felt like a lot of handwaving instead of straight talk. They all said they encrypt my files, but there was no way for me to verify that since I couldn't access my stuff on their servers." So, without clear explanation of Arq's encryption method, while speculating about the encryption used by other companies, it seemed unfair (at best). But now that I see you specify the AES-256 encryption, I have no complaints.

As far as press goes, I can see that as you slowly tweak and upgrade the application, it may be hard to decide when the right time is to try to get Arq reviewed (with a better product always coming with the next update)... Not that I know a thing about getting press. But perhaps letting the application ripen over time, then really pushing to get it reviewed, might be the way to go. I would imagine that one really good review in a well-read Mac or Tech column (Macworld, Gizmodo, etc.) could make your customer base expand far beyond what I imagine it is now.

I guess the tough part is Arq isn't dumbed down, like the Mac version of Carbonite (which I tried and then removed, as it just didn't have the flexibility I was looking for). While just about anyone can use Carbonite with the click of a few buttons, and the price is low and fixed, Arq requires extra effort of signing up for S3, and it's somewhat hard to know what the yearly cost will end up being for the S3 service. So perhaps Arq will never be for the many Mac users who just aren't very adept, and want to keep it as simple as possible...



[ Reply to This | # ]
Use Amazon S3 to automatically back up your Mac
Authored by: tonyp13 on Mar 07, '11 01:29:56PM

There is also a GUI tool for Mac ( multithreaded ) : http://www.dragondisk.com/



[ Reply to This | # ]