I have a large and growing collection of photos, videos, music, reference files, and project files on my home network. I’ve long passed the point where it all fits on any one laptop or desktop PC… it’s probably upwards of 400GB. And now with a home office I have a lot of work-related files that I really really can’t afford to lose. Frankly, all this data has become a gigantic headache. So I’ve recently invested some time organizing everything to be Just So. I thought I would share my nerdy little organizing scheme, in the hope that others might find it useful.
1) Organize your folders. All important files live in folders named as follows:
\Music = music currently in play
\MusicArchive = music archive (not currently in play)
\Photos = archive of photographs (subfolders 2004 ,2005, 2006, etc.)
\Projects = active projects and archived project files (subfolders for each project)
\Reference = miscellaneous folders, reference/archival use only
2) Synch “active” folders using Foldershare. All files within folders that I access and update on a regular basis are synchronized (mirrored) across all home computers using Microsoft Foldershare. This is done with a single Foldershare account, just to keep things simple. Foldershare automatically synchronizes each machine whenever it happens to be on the network, which is most of the time in our house… we have a wired and wireless network. Its pretty good at letting multiple people work independently on files without stepping on each others’ toes. I’m told Groove is also good at this, but I haven’t tried it.
3) Revision-Control Special Files. Our source code and production web site files need to be revision controlled, so that we can roll back to older versions if need be. For this we use a Subversion repository provided by our web host, HostingRails.com. Subversion is nicely integrated into Windows Explorer with TortoiseSVN (so you can manage any file that Explorer can see), and into Eclipse with the Subclipse plugin (so you can manage source code easily). There’s also a command line interface, of course. I’d recommend subversion to anybody working with source code or other versioned files. The server setup is a bit of a pain; I imagine there are hosts that do it all for you, though.
4) Get a Big Backup Box. As I mentioned above, this data doesn’t all fit on one box. So we purchased a 2TB Buffalo Terastation network-attached storage box, which several friends have recommended as a reliable backup and file server box. 2 TB! Sheesh.
The terastation has three jobs: (i) provide a comprehensive mirror/archive of all our files; (ii) act as standalone media server for music and photos, and (iii) act as a standalone print server (you can hook up printers to it directly through USB). So all the folders we care about, regardless of whether they are actively being updated or not, get synchronized regularly to the Terastation. I also map network drives to make it easy to access things… M: for Music, P: for Photos, W: for Work, and so on.
The Terastation is configured by default to use RAID-5, so with the 2 TB model you get about 1.4 TB of usable storage. If one of the four internal hard drives fails you can swap it out without losing any data.
Terastation comes bundled with Memeo Backup software, but from I’ve seen so far I don’t think I can use it as a file synch solution. Memeo doesn’t appear to let you configure any thing but the absolute basics, which doesn’t work for me. It also doesn’t support true mirroring of files, which is what we really need for most of our data. (“Rolling back” to a previous revision doesn’t really make sense for photos or music libraries… we need mirroring, not backup.) So my current solution is to run Robocopy periodically from one of the PCs, mirroring files to the Terastaton box. This can be done lazily, e.g. copy once an hour or once a day, so that you have a “safe window” of time during which you can fetch old copies of files from the NAS box in case you screw up. Or you can make it copy within one minute of any change.
In an ideal world Foldershare would run on the Terastation too, or handle network drives, so that we wouldn’t have to deal with another piece of software like Robocopy or Memeo. It looks like Foldershare does (or did) run on at least one NAS box out there, but it doesn’t on Terastation. Anyone know of a solution? I’d guess Microsoft wants to make money from licensing Foldershare for use on file servers, so I’ll be surprised if it’s easy to make this work.
5) Playback. I plan to hook up a music streaming device to the Terastation. This should be super easy… the Terastation exposes SMB file shares, which devices like Sonos, Audiotron etc. all work with. And I’d love to have a digital photo frame that pulls in photos from the Photos folder on the Terastation, over ethernet or wireless. This could sit in the living room or kitchen or wherever. Anyone know of one?
6) Bonus Points. File encryption and search on the NAS box. I’ll post again if/when I get that figured out.
Cost: Two things in this solution cost money: the Terastation ($800+ USD, $1100+ CAD) and my hosting account at HostingRails.com (~ $100 USD/yr). Everything else is free.
Keep it Simple (and cheap): Don’t bother with network attached storage devices like the Terastation if you don’t need to. Just use Foldershare to mirror files (or particular important folders) between your computers.
Archival versus Active Use: Some folders, like MusicArchive and photos from earlier years, will end up living on the Terastation only for archival/reference purposes. This makes sense because they are rarely used, and anyway would not fit in entirety on any one of our PCs. We can move files into one of the “active” folders by simply moving or copying, e.g. moving a music album from MusicArchive\ to Music\. Or we can just reference them directly on the Terastation box, using it like a big remote hard drive.
Compatibility: in theory this setup should work nicely with both PCs and Macs, since Foldershare and the Terastation work well with Macs.
Foldershare quirks: Foldershare does have a limit on filename length, so you can’t have a really deep directory structure. The free version of Foldershare is also limited to 10,000 files per shared library. I get around this limit by keeping folders small, and only using Foldershare to synch folders that are being actively updated. For instance, in January 2007 I remove my Photos\2006\ folder from Foldershare’s control, since I no longer have any need to add files to that folder, and I add Photos\2007\ to Foldershare so that new photos stay up-to-date during the year.
Versioning everything: I recently came across a post by Joey Hess, who uses Subversion to version ALL of his files and keep them in synch across multiple machines. This is, I think, a little extreme. And scary… personally I wouldn’t really want a copy of every file I’ve ever created to exist. But it’s pretty damned cool too, so if you want to turn the volume up to level 11, check out Joey’s solution.