It has been quite a long time since I have posted anything here—the spam bucket was choked with well over a hundred pages of who-knows-what when I first came back.

One thing I did notice was that several folks have taken advantage of the iTunes-to-Playlist script that I put together last year, and that’s always encouraging. I haven’t really done any work on tweaking or refining the script because there was a small bit of a problem in the homestead: my happy little Shuttle server died an untimely death.

I bought it off of eBay brand new for a little over a hundred bucks last February and was quite pleased with my purchase. It was a Shuttle K45, a pretty old platform with a lightweight processor, but more than enough horsepower to be a home Squeezebox server as well as a Wiki server and a CVS server.

But it was too good to be true. After I had settled in and set up my workflow around my silent little Shuttle box, it simply died one day. I pressed the power button to restart it and was greeted with a blinking light that was anything but encouraging—the server was dead. An autopsy revealed the probable cause of death to be bad capacitors.

If you find yourself dealing with wonky hardware problems, such as a PC that doesn’t boot or frequent BSOD when watching video, pop open the case and give the caps a gander. You might save yourself hours of troubleshooting effort. See below for details…

Shuttle K45 = Bad Caps

I now know why I occasionally find new-in-box K45s on eBay: there was a bad production run of them.

Just do a quick search for “Shuttle K45 bad capacitors” and you will find plenty of complaints. Here is a good example of a post about the problem from Tom’s Hardware.

Here’s the Problem

This is how the motherboard looked. That bit of brown goop that is oozing out of the top of the capacitor on the left is a sure sign of death. You might note that the one on the far right seems to be bulging a bit at the top as well, ready to blow. In fact, that’s why the manufacturer put the little cross hatch in the top of the aluminum can: it’s a sort of safety release when pressure gets to great within.

I checked all of them, and there were a half dozen of them that were oozing brown stuff.

Not the first time

As I had alluded to at the start, I have seen this behavior before: I once wasted many hours of my time trying to track down the demons that were causing random blue screens on my old Windows Media Server box.

I had re-imaged the machine twice with the factory disks and had even gone so far as to run a live CD version of Linux in order to eliminate Windows and the hard drive from the equation. Indeed, watching video in Ubuntu caused similar hard crashes, so it was clear that I had a hardware problem.

I had a distant memory of having read some article talking about bulging electrolytic capacitors, so on a hunch I popped off the cover and looked at the video card. Sure enough, the caps on that old ATI card had little brown dots in their crowns. That episode ended fairly well: I bought a new video card and that particular machine returned to service for several more years.

A Heroic Repair Attempt

I really did like my little Shuttle server and hated to see it go, so I immediately began looking in to how I might bring it back to life. I have wielded a soldering iron before and so I decided to roll up my sleeves and fix it myself.

I didn’t find the right capacitor sizes at Radio Shack—it’s sad to see that their selection has diminished greatly from my childhood memories of going to buy parts for radio kits with my dad.

I ended up going to an outfit called Badcaps.net that sells prepackaged kits for the usual suspects, and they do indeed sell a Shuttle K45 18pc Capacitor Kit for $17 and shipping.

Here they are, fresh out of the plastic baggie:

My soldering exercise took about two hours, after which I understood the following statements to be true:

• Desoldering is much more annoying than soldering.
• Those solder sucker tools just don’t work. Maybe I needed to use that desoldering braid.
• The motherboard of a K45 is a sweet pain in the backside to extract from the case.
• The server was just as dead after my efforts as it was before.

I decided that I had already invested too much time and effort into this exercise and threw in the towel. I lost no data in the process—it was all backed up elsewhere. Life is good!

Last week I set up a new server on an aging Dell PC, but I’m on the hunt for a new server.

I’m serious. You don’t need to know anything at all about Linux to use one of these. Just download the image, install, and you suddenly have a full featured NAS file server, or you might have a database or a source code repository.

Last year I wrote an article on how to set up a NAS device using Ubuntu Linux. I have been a fan of Ubuntu since the start because it is a very easy distribution to install and configure. The down-side of using Linux has always been the fairly steep learning curve. Before you can get around to using the server, you need to get down in the weeds with configuration files and other stuff.

TurnKey Linux changes all of that.

Painless Installation

A few weeks back, I was setting up an aging PC as a standalone wiki server for a small office—this machine was going to provide a place for the office staff to document their procedures, how-tos, and other things.

I was about to set up an Ubuntu server, as I have done before many times, and install MoinMoin, like I did some months back. I remembered that it was a bit of a pain to get everything tweaked just right, so I did a quick check to see what kind of standalone wiki options were available online.

This is how I found TurnKey Linux. This project is all about single-purpose preconfigured Ubuntu server images.

One of those preconfigured images happens to be a MediaWiki appliance—the wiki engine behind Wikipedia—and I was in business.

The installation took about fifteen minutes, with very little user interaction. I answered a few basic questions and the installer took over from there. As soon as the install was done, the machine rebooted and displayed a message on the monitor with the IP addresses where you can browse to from any other machine.

Full Featured

The work that has gone in to these appliances is amazing. In fifteen minutes I had installed a complex configuration that has the Apache, PHP, MySQL, MediaWiki core, as well as maintenance utilities such as a neat tool that provides a Flash-based pure-AJAX-based SSH command line in a remote browser (i.e. your browser becomes a terminal). Even someone with Linux experience would have to spend quite a bit of time fiddling around with different packages and configuration options in other to provide the same functionality that TurnKey gives you out of the box.

As with most open source projects, the documentation is about 80% complete, with deep detail in some areas, but leaving others fairly sparsely documented. But don’t let this deter you: in most cases users know how to use the product they are installing (e.g. MediaWiki) but don’t want the hassle of configuring it on Linux. That’s where TurnKey shines.

Some Examples

In minutes, you can set up a NAS device. If you want to try advanced content management in your office, try Joomla or Drupal.

If you are working on a small project team and want to protect your source code, try Redmine or Trac and do your bug tracking using Bugzilla.

And while you are at it, you can document your organization’s working practices using a wiki such as MoinMoin or MediaWiki.

Don’t forget to back it up!

As with any computer, you should include your new TurnKey appliance in your backup strategy. The nice thing is that you don’t really need to care at all about backing up Linux or the other software; just back up the data. I don’t need to back up my entire MediaWiki machine; I just need to back up the database and image files. If anything goes wrong, you can rebuild the TurnKey appliance from scratch in minutes and then restore your data.

To save yourself some pain, keep notes on any small tweaks you made to the configuration.

One Machine, One Purpose

These disk images share common Ubuntu underpinnings, but they are referred to as Appliances because they turn your PC into a purpose-built appliance.

This means that if you want a content management system and you also want a ticket management system, you will need two old computers—not a rare commodity these days.

Take a look at what they have to offer and give TurnKey a shot—specialized software used in corporate environments is now within reach of small offices at the right price.

Among other things he discusses options for prioritizing and cutting down on the total volume of stuff you plan on keeping, digital or otherwise.

“Backup, backup, backup” made number four on his list.

And finally, he provides a few notes on some helpful free organizing software. I think I’m going to check out that Know Your Stuff application he mentioned.

Five Tips to Paperless Finances (moneytalksnews.com)

Now prudence would lead one to take careful steps and wait until all of the wrinkles are ironed out before starting. I was almost not prudent enough this week.

Mac OS X Snow Leopard

So folks have been talking about the new Snow Leopard operating system for Mac. Over the past year, Apple has been positioning this version as more of a “under the hood” upgrade that tightens things up rather than a glitzy overhaul of the user interface. No matter what they said it was, I figured that it was newer, and therefore better, than the current OS—Leopard–and I had to have it.

I ordered my copy last week on Amazon and sat down with a smile as I awaited its arrival. And then I thought about doing a few quick Googles to see how other people have been making out with Snow Leopard. I immediately happened upon a few upgrade guides like this one, providing sage advice about the upgrade process. They recommended the “slash and burn” method, starting from a clean hard drive, and I felt that was a good idea. Nothing better than a wipe and fresh install to make your machine zip along twice as fast. And therein lies a tale.

The first sign of trouble

As I was reading up on the Snow Leopard upgrade process, I happened upon lists of “unsupported software” and casually glanced at the lists, expecting esoteric tools only used by three über geeks in the audio recording industry or perhaps some exotic ray-tracing software. Much to my surprise, I saw two of my favorite applications, in a very very short list of troublesome apps: Parallels and EyeTV.

I immediately checked the versions and breathed a sigh of relief when I saw that my EyeTV version was safe. But, Parallels was another story… They have no plans for patching Parallels 3 to work with Snow Leopard, and why should they, when they can sell us Parallels 4!

So, I ordered my fresh copy of Parallels 4, from Amazon with a twenty dollar rebate. When it arrived, I spent an evening upgrading Parallels, and I thought I was all set for Snow Leopard.

Preventative Measures

Following the advice of the upgrade websites, and prior experience, I used Carbon Copy Cloner to make a full backup of my hard drive on a spare external drive. On a hunch, I turned on the drive that I use for Time Machine and had it do one final “Time Machine” sweep through the system before bidding adiu to Leopard.

I knew that I had all of my installation media for stuff like iLife and Photoshop Elements, and I had all of my license keys in electronic form. It would be a simple matter of mounting the backup drive, copying over my loads of documents, and peering into them to find keys.

The first attempt

I boldly inserted the Snow Leopard disk and booted from the DVD drive, selecting the “Slash and Burn” method of installation. I reformatted the hard drive and went off for dinner while Snow Leopard installed.

Trouble

When I got home that evening, I started the lengthy process of installing stuff. I suddenly realized that it was not as easy as I had hoped: it’s one thing to reinstall something like Microsoft Office, but there seemed to be more loose ends than I had considered:

• How would I migrate my Mail settings from the old image to the new?
• What was the best way to migrate the Address Book contents?
• iTunes is great, but it has tendrils in everything. Can I simply copy my old library to the new without messing up my iPhone, Address Book, or other linked stuff?
• How about those nice password tools such as 1Password and SplashID that keep your passwords safe and sound? I had no clue how to get their contents from the backup. I wasn’t sure if it was even possible to do so—perhaps I was supposed to have exported the data beforehand.

It was becoming clearer to me that I had not done my homework at all.

More trouble

My initial shock at the depth of the upgrade process led me to start making a list of applications and looking at what I needed for each one. I soon found out that Snow Leopard support is somewhat spotty in many applications. In particular, the FineReader for ScanSnap software that I depend on so much for my scanning work flow is not fully supported. Fujitsu says that they will have an update soon and to keep checking their web site.

My password tool, 1Password, is another problem child. It works only on 32-bit Safari, and Snow Leopard now runs Safari in 64-bit mode. Of course, a new version is coming, and I will probably have to pay for it, but it is still in beta.

There was quite a bit of chatter on the Web about whether Adobe Photoshop Elements would work on Snow Leopard, and the responses seem split fifty-fifty for now.

Three very important tools were in danger of running in limited mode or not running at all, so I had to throw in the towel.

Time Machine saves the day!

As I sat, humbled, before my vanilla install of Snow Leopard, I admitted defeat. I slipped the Snow Leopard DVD back in the drive and rebooted from the DVD. This time, I selected the “Restore from Time Machine” option and turned on my Time Machine drive.

Guess what? It worked perfectly! Unlike many software products, Time Machine does exactly what it promises.

Within a few hours, my machine was fully restored to the way it looked seconds before I made my first attempt at Snow Leopard.

A Final Word

Learn from my mistakes, and my salvation by the full backup. As much as you can’t wait to upgrade, please do the following:

• Inventory all of your applications that you really need.
• Obtain the installation media (download or CD) for every single one.
• Obtain the keys for every single one.
• Investigate whether you need to export data from any of them, and make a checklist for these exports prior to upgrade.
• Check the “Unsupported Software” lists that are out there for any red flags.
• Check the web sites of your most important apps for their official word.
• And finally, do a complete backup!

It’s amazing how many applications and weird little utilities we forget we have. How could I have possibly remembered that I compiled a custom copy of the “rsync” executable for my backup workflow? I would have lost that and had to figure out how to rebuild it on Snow Leopard.

And I haven’t even talked about making sure your documents make it safely onto the new machine. That’s a whole ‘nother story.

In case I forgot to say it, please make a full backup.

[Update: I'm giving Snow Leopard a rest for a few months]

It has been said that Time Machine allows you to do a full restore from bare metal, and I’m living proof: I have done exactly that twice in the past week, with astounding success.

Encouraged by an episode of the Mac Geek Gab where they talked about their experiences upgrading their existing systems to Snow Leopard, I decided I would give the upgrade-in-place option a try. I expected some things to not work well and others to be quirky, but here’s what happened…

The actual install was painless, taking an hour or so to complete. I then began to kick the tires to see what was broken.

It was clear where those 64 bits went: apps like Safari were positively zippy, and I was pleasantly surprised with each new application I launched. All of my special settings seemed to make it through alive, including my password manager, though I did have to re-enter some of my registration keys. All of my mail and contacts made it through well. I was able to sync my iPhone without incident.

I found a few apps that weren’t working correctly and I looked for newer 10.6-compatible versions. I found newer versions of Yep and xGestures.

I did note that there is currently no ad blocker available for Safari that runs in 64-bit mode. This is disappointing because even though I understand that Apple wants us to see their ads, I can’t imagine that they really want us to suffer from the flickering jumping dreck that should have ended with the hated “punch the monkey” banners of years gone by. The fact of the matter is, if I want that 64-bit speed and snap, I guess I have to watch ads.

The Showstopper

I decided to scan a document to see just how difficult it would be to get my workflow going again. Michael F, below, wrote the truth about the situation: the scanner works fine in certain modes, but the OCR software doesn’t.

He pointed out that it was a problem of the FineReader software looking for a specific bit of metadata in the PDF identifying it as a ScanSnap PDF. Sadly, that metadata string changed.

The Finereader software is looking for “Mac OS X 10.5.8 Quartz PDFContext”, but under Snow Leopard, the string is set to “Mac OS X 10.6 Quartz PDFContext” instead.

There are ways to tweak PDF metadata, and one of them is by using pdftk.

I went to the pdftk site, all ready to download it and start OCRing my PDFs. I was greeted with less than optimal news: they have a version compiled for Panther, a version of OS X from several years ago.

I knew it wouldn’t work, but I gave it a try anyway: the app told me it needed Rosetta to run. I could have installed Rosetta at that point, but I figured I wanted a proper compiled version.

From there, I looked into compiling the app on OS X 10.6. I should have remembered my struggles with this several months ago on a Solaris Unix box when I found that pdftk depends on a monster called GCJ that required about forty other software packages to compile—it seemed a gargantuan task that I wasn’t ready to begin.

On a hunch, I inspected the content of a new pdf and an old pdf, the latter still acceptable to FineReader. Though much of the file was raw binary, the metadata was in text at the end. A short sed script was all it took to swap the nice text string for the offending 10.6 one.

In spite of my best efforts, FineReader still rejected my hand-tooled PDF file. It knew that it was a bogus file.

I have looked into Abbyy FineReader several times before, as well as Fujitsu’s ScanSnap support, and was unimpressed. For two vendors that produce products that are at the top of their class—FineReader is arguably the best OCR you can get for Mac, and ScanSnap is the best document scanner for the common man—they sure do have miserable customer support.

It is as if neither company cares a whit about the Macintosh platform or their customers. While most other vendors are busily patching their products and giving hourly updates on their Snow Leopard compatibility progress, Abbyy and Fujitsu just don’t seem to care that their best-of-breed combo suddenly doesn’t work on Mac.

Once they get this sorted out (hopefully in the next few months) I’ll give Snow Leopard another try. In the meantime, I’m sticking with good old Leopard.

My own choice for document format is almost always Portable Document Format (PDF), but rather than just state this, I would like to consider some of the factors involved.

This is the first of a series of articles covering document formats. This article focuses specifically on the distinction between works in progress and finished product.

Two Kinds of Documents

In general, we can consider two broad categories of documents: working documents (works in progress) and archived documents. You can call these by many different names, but the fundamental distinction is still there.

Working Documents

These are documents that you are still writing. They share some characteristics:

• They must be retained in their original format, such as Microsoft Word.
• The formats are often very specialized. Quite often another tool can import such a document, but you usually lose something in the translation.
• You and your colleagues need to have the same editor software to view and modify the documents.
• They are often short-lived. This phase of a document’s life usually doesn’t more than a few months (though a template document might be kept for many years).
• A good backup strategy will need a short window between backups; these documents change often, so they should be backed up frequently.
• You may want to consider a document versioning strategy, so you can see how the document appeared at different stages during its life.

Here are some examples:

• Microsoft Word documents
• Visio diagrams
• Photos that you are still retouching
• Audio files that you are in the process of curating (e.g. applying ID3 tags)

Archived Documents

These are documents that are read-only, meant to be viewed but never modified.

• They often must be rendered in very precise ways, so each viewer sees the document as intended (consider a 1040 form from the IRS)
• They may be around for a long time.
• These documents should be less tightly bound to a particular software product. PDF vs. MS Word; JPG vs. Adobe Photoshop.
• They typically have a wider audience. You may share a work-in-progress with a co-worker or two, but a finished read-only document might be read by hundreds or thousands.
• Any user should be able to read these documents, with little effort.
• Your backup strategy is probably going to be more focused on longevity and less focused on frequency. These documents are in it for the long haul.

Why not start with a simple example?

Here is a screenshot of an application I use in my day job:

Just in case you did not recognize the unmistakable visage of this small molecule, I have labeled it appropriately.

This is an application called ChemDraw from Cambridgesoft, and unless you are a chemist you have probably never heard of it. My molecule is saved as caffeine.cdx in a format that only ChemDraw knows intimately (though there are other similar chemistry tools that can import this file format).

My point is simple: if your friend sent you a copy of caffeine.cdx, how exactly would you open it?

In contrast, here is a more accessible rendition of the same molecule in PDF format. Try it out; you should be able to view the molecule, and zoom in on details.

What if you had to show someone this document five years down the road? Do you want to have to chase down a possibly obsolete version of a very expensive application that might not even run on your operating system?

Obsolescence

Some time back I was sifting through some files on an old server at work that apparently had been written by me. Fifteen years ago I was attending night classes and writing many of my English assignments on a VAX running VMS at work (over my lunch break!). I was using some anemic version of WordPerfect that had been ported to VMS. This arrangement saw me safely through college, but was not conducive to long term document storage.

Do you have any idea what VMS directory structures look like? Maybe, and maybe not. Are these files compatible with the contemporary DOS versions of WordPerfect? Maybe.

Could I open these files on a Windows Vista machine in 2009 using Microsoft Word? With luck. What about using Pages from Apple iWork on my Mac running OS X? Doubtful.

Not only do we need to be concerned with special applications that only a select few (with expensive licenses) have, but we also need to consider that the file format might be obsolete beyond hope.

For an exaggerated example, consider the image of punched paper tape at the top of this article. I would have no clue what to do if I were given a roll of this tape.

Which do you keep?

Look at the characteristics of the document types listed above and see which one fits your document best. Quite often you will find yourself keeping both the original document and a PDF rendition. Indeed, this is what many professional document databases do.

If you can’t easily choose one, keep both. In most cases, I have found that I only need the PDF rendition for the long term and I couldn’t care less about the source document.

Summary

In the world of the paperless home, much of what we do is store digital copies of old documents for searching and possible reprinting some time in the future. Don’t make the mistake of keeping all of your documents only in their original editable format; you might just find yourself with a digital file that cannot be viewed!

I guess there must have been a time when we all lived simpler lives and didn’t care a whit about who was grubbing through our trash. Honestly, as I am shredding the week’s load of pre approved credit card offers and the like, I imagine I could just be bold and tear them all in half and toss them, unshredded (gasp!).

But anyone who is packing serious scanning hardware should also be packing serious shredding hardware. It may not matter if another Capital One offer slips into the trash intact, but there’s no way I’m going to dispose of old tax records or medical records without rendering them completely useless to the enemy.

Here is my own short list of things to look for when you are buying a new shredder.

A Serious Tool for a Serious Job

There are lots of sites out there that have buying recommendations for shredders, and once you wade through all of the commercial spam sites and get to some good articles, they offer pretty even coverage of the different features to look for. This is where my own opinion differs from theirs.

Don’t waste your time with some namby-pamby little shredder that sits on your desk and shreds three thin sheets of rice paper at a time. If you are serious about reclaiming your home from paper, you need something that can wolf down loads of documents, paperclips and all.

This is my most firm recommendation: go for the most solid machine you can afford. Otherwise you will be buying a new one next year.

I don’t know about you, but I have been through three of the darned things. I tried the relatively cheap one and killed it within six months by overheating it. I then bought a heavy duty model that served five very good years before expiring with some mysterious illness. My current one is a somewhat smaller model, selected specifically for its reduced size by my wife.

Good Capacity

Make sure that your shredder is rated at a minimum of 12 sheets at a time. This way, you can insert just about any stapled document and be confident that it is not going to choke the machine.

See if the device can eat both credit cards and CDs. Both of these features are pleasing to have.

Don’t bother with those silly things that have narrow slots that require you to fold the paper like a letter before insertion. Do you really want to have to fold five hundred sheets of paper like that?

Solid Construction

This is a fairly subjective assessment. Look at several of the machines and see which one is built better. Chances are that the el cheapo version will have plastic gearing and other brittle parts. If you can look inside (often from the underside) look at the gears to make sure they are metal.

Emptying the Basket

Imagine yourself emptying the basket. You will be doing this dozens of times.

Many machines have a built-in trash can that flips out from the front, leaving the rest of the machine standing. I like this feature and look for it. You should try removing and replacing the basket a few times to see if it is a fiddly task and if there are little cheap plastic tabs that are going to snap off after two weeks.

One kind of shredder sits on top of its basket—you must lift it off the basket entirely in order to empty the waste. Pick it up a few times. Do you mind lifting the weight? I know that my wife didn’t want to have to lift the shredder every time she emptied it, so we skipped past this style.

Another point to consider is whether those special shredder bags will fit in the basket of your machine. As far as I know, all shredders have some sort of mechanical interlock between the machine and the basket that shuts off the device if it is not on the basket. This is usually achieved by some little tab on the basket that slides in some slot, closing a switch. If you use a shredder bag, make sure you can do so without interfering with this safety mechanism.

Basket Size

A small amount of paper becomes a large volume of confetti. If you don’t mind having a larger unit taking up space in the corner, then go for the one with the bigger basket. Trust me, you are going to be lazy and let it get filled up anyway—why not at least have a longer delay before you are forced to empty it?

Consider a tall basket rather than a short basket. Once the top of the confetti reaches the underside of the shredder mechanism, it is possible for the blades to draw in the already shredded paper and come to a grinding standstill that may not be easy to recover from.

Strip vs. Crosscut

I don’t know just how much to worry about this one. Everyone says to get a crosscut shredder, so that it tears your paper into little tiny diamonds, but they still are manufacturing strip shredders. The fact of the matter is, strip shredders can handle a heavier load since they aren’t chopping the paper to little tiny bits.

Once the paper is shredded into skinny strips, I’m pretty happy with it. Unless you are some high profile person with valuable secrets, why get worked up about it? Do you really think that someone is going to piece together all of those little strips and read about your colonoscopy three years ago?

I’m not saying that it doesn’t matter at all; if you want the added security of crosscut, then go for it. I’m saying that this can be a lower priority than some of the other criteria. Don’t automatically rule out a solid performer because it is a strip shredder.

Other Features

They all come with various other nifty features such as automatically detecting inserted sheets and automatic reversal when there is a jam. One nice feature is an automatic cutoff when the unit has worked too hard. I think all of them do this, but some do it ungracefully by dying forever, while others tout a graceful cutoff.

Think Before You Shred

You don’t get a second chance. Make sure whatever you shred has been electronically captured and backed up. Is there an electronic copy safely in two places?

If you do not have an electronic copy, then make absolutely certain that you intend to destroy the document forever—this is exactly what you are doing.

Summary

I believe in buying the best tools I can afford for the job, and this holds especially true for shredders. It is a serious machine that has migrated from government/military circles, to offices, and finally to the home. Pick a solid performer that can chew through whatever you throw at it.

A final thought: Don’t buy one sight unseen. These devices must be handled in person to be truly appreciated!

“How a lifestyle experiment and a disaster made me realize the value of turning atoms into bits” — Mike Elgin

A couple of months back, Mike Elgin of Computerworld posted an article on his foray into the paperless world: Paperless office? Ha! How about a paperless life?

In this followup article, he considers how lifestyle changes and the raging wildfires closing in on his city have made it clear to him that it is critical to protect whatever can be easily preserved in digital form.

Mike gives a sound strategy, starting off with a fast pass at just getting it digitized with little regard for perfection, followed by more focused efforts.

He pointed out how some objects just aren’t as important as the memories that they represent, illustrating this with a photograph of his son’s martial arts trophies that he has since donated to various organizations.

Mike makes it very clear that backups are mandatory. My kind of guy!

Here’s the whole article: Why you should digitize ‘everything’ (Computerworld.com)

Here are a few thoughts I have to share on these great media library projects. Not only are they never-ending, but the longevity of the task should guide some key decisions you make as you plod along.

Accept that it’s Never Going to End

I currently have ongoing projects to scan in every scrap of paper in my house, rip every CD to MP3, and scan in every pre-digital-era photograph. But the corpus is just too large. And I haven’t even considered ripping video; I have accepted that a full digital video library is a little too ambitious for me these days.

We need to be aware of this from the very beginning; if you feel like you bit off more than you can chew, you are normal.

Never Give Up

Try to keep your goals clear, so that you don’t lose steam and give up. Perhaps you are looking forward to being able to place the plastic boxes all in storage. Perhaps you can’t wait to be able to make those uber party mixes from your rich collection. In any case, every hour you spend doing the drudgery of ripping CDs brings you a little closer to these goals.

Prioritize Your Work

It is a whole lot better to spend your first days ripping your favorite music to MP3 than to be ripping some weird CDs your roommate left behind in college. Today I happily skipped over the Talking Heads CDs that I inherited from my brother—those don’t really need to be in my iTunes library, do they?

Keep it Simple

Make your work flow as simple as possible. You may still be doing this in three years—don’t over complicate things.

I have been working on my own collection for the past three years. When I first started, I was ripping the music on a PC using a tool called cdex to rip the MP3s and another tool called Mp3tag to edit the tags. These days I use iTunes on my Mac to do both of these tasks. If I had some really fancy work flow going, it might not have been so easy to change machines. By keeping things simple, I was able to do exactly the same thing today that I did three years ago.

Choose Good Settings

…for your software. You don’t want to have to do this again, do you? And don’t go cheap on the bit rate, if you have the space.

In addition, you will probably want to decide between the convenience of a universal format versus the power of a proprietary or niche format. In my example, I opted to use MP3 for all of my music instead of some of the other, more powerful, formats available. I was stung once by the switch from PC to Mac where the handful of CDs that I had ripped to .wmf files were no longer usable.

If you are ripping everything to some killer lossless format, be sure you are looking far down the road and considering compatibility issues.

Back it up!

Make sure your backup process is ready for the increased load. If your backups consist of burning DVDs (or worse, CDs) of your data, you will very quickly tire of this, and you may just abandon one of the most important parts of your work flow. Oh, and remember, if there aren’t two copies in separate places, it isn’t a backup.

A couple of months ago I bought two of those nice pocket-size portable hard drives, and I always leave one at the office and I swap them once per week. This works for me: they are 320GB hard drives, providing plenty of room for all of my thousands of scanned documents, tens of thousands of photographs, and many thousands of MP3s, with plenty of elbow room.

When ripping a CD collection, the data risk is lower since you still retain the original CDs, but it still is a lot of work to rip them all. Don’t overlook the backup, or you will cry when the hard drive crashes.

Know Where You Have Been

Come up with a simple “bread crumb” mechanism to know where you have already been. You want to be able to quickly identify the music CDs you already ripped, without having to bring up iTunes every time. Keep the three year horizon in mind, so you choose a technique that will not be mysterious to you after some time passes (Hmmmm…. did I dog ear the corner of the CD insert to mark it? Where did I put that notebook where I wrote them down?)

I chose simplicity: I have one of those nice metallic Sharpie markers, the silvery kind. I put a small dot on the hub of each CD after I ripped it, just above the center hole. This way, several years from now, I should easily be able to spot the CDs that haven’t been introduced to my digital library.

I do this with photographs as well. As I scan them, I put a small marker dot on the back of each photograph, in the corner.

Finally, Enjoy Your Music!

I am quite capable of getting all wrapped up in the process for the sake of the process. Don’t forget to actually listen to the music you ripped!

How can you keep your data from being held hostage?

Have you ever stopped to consider exactly how much information is permanently stored within your favorite applications, locked down to all but the most determined command-line commando?

Perhaps the easiest way to explain what I’m getting at is by way of an example…

My Wife’s Email

Some months back, my wife was moving from a PC to a Mac, and all that remained was to configure her email. She had been using Eudora on the PC for many years and had several thousand emails in her mail folders. It went without saying that there would be dire consequences if I were to lose a single scrap of her electronic correspondence.

My first thought was to simply install Eudora on the Mac and then copy the database from one to the other. But there was bad news waiting for me in Eudoraland… The product is discontinued and no longer supported. They haven’t released a version since 2006, and you can’t even buy a license to turn off the ads anymore.

Eventually, after quite some time in Google, I found that one could simply point Eudora to a Gmail IMAP account, then drag all of the messages from the local folders to the IMAP folders.

Unbelievably, it worked exactly as advertised. I dragged thousands of messages to her Gmail folders and the machine bogged down for a whole day as it uploaded everything to Gmail.

Garbled Messages

Once I finished the job, I inspected the moved messages. Much to my horror, many of them were displayed in XML format, with embedded HTML tags all over the place. A few experiments verified that this was not an error on my part. The same message would show up garbled in the Gmail web interface, the Apple Mail client, but would still appear cleanly in Eudora, served from Gmail.

The message above, shown in Apple Mail as XML, was rendered properly in a fresh install of Eudora on a clean PC:

Clearly, Eudora had mixed in some special sauce of its own when it stored all of those files on Gmail’s IMAP server!

In the end, we accepted the situation: from that point forward she was golden, using Apple Mail and Gmail. If she ever needs an old message, we can fire up Eudora on an old machine and view the message.

What’s the point?

Many applications that we use in our paperless workflows and lives have sinister tendrils working their way throughout your data, slowly adding value to your life by their secret internal data that they will never relinquish. You may never notice just how dependent you are on their “secret sauce” until you are forced to change applications.

Tagging

One of the most harmless appearing kinds of secret data comes in the form of tagging. Just exactly where do you think those tags in your iPhoto or Picasa library are going? If you wanted to switch from one of those fine products to the other, how would you get thousands and thousands of tags saying “Rome, 1998″ and “Joey’s Graduation” moved over?

The problem is that these applications keep their tag information in a separate database from the actual images. Many similar applications suffer from the same problem: if you tag documents in a product such as DEVONthink, your tags are stored separately from your files.

There are standardized ways of embedding certain kinds of tags in many multimedia files, such as the ID3 tags for MP3 files and metadata tags in PDF documents, but often applications do not modify these in an effort to avoid touching your original.

And the day you want to leave them, they will hold your tags hostage!

Mysterious Black Boxes

I blame iTunes for the plethora of black box applications that encourage you to feed them multimedia files, which they promise they will squirrel away in a perfectly sensible and secure location.

When you drop a bunch of MP3 files on iTunes, they get shuttled into the relatively orderly iTunes library. When you drop a set of photos on iPhoto, they go into a pretty odd arrangement of folders that are created based on dates and source folder names. Sometimes iPhoto pays attention to the file timestamp; other times it uses the EXIF data within the picture.

DEVONthink does similar magic with PDF documents; they tell you to dump stuff in there and let DEVONthink’s smarts sort it all out for you.

This file organization is a different kind of “secret sauce” that you would lose if you were to switch horses.

An Unhappy iTunes Story

I smiled when I read this silly little anecdote:

“Eddie finally came in with his portable disk and uploaded his 85GB collection of MP3 files into her computer. Okay, it might be illegal, but who cares – everybody’s doing it. Eddie looked at her importing the MP3 files into iTunes, Angie’s favourite player, and suggested that she switch to Winamp, which is just a way cooler piece of software. Angie wasn’t sure. Eddie explained that he had all those carefully crafted playlists for Winamp, mixing the files he just gave to Angie into really spacey all-night sessions, and Angie should really try it out.

Angie did install Winamp with Eddie and has opened it a few times since then. She still doesn’t know … well, in fact she does. That software just doesn’t feel right to her. What’s worse, Angie has spent years in rating her song files with iTunes, and quite rarely hears any bad pieces anymore. But if she goes for Winamp, all those ratings seem to be lost.

Wouldn’t you know it, just after installing Winamp it started – most embarrassingly – playing her forgotten Backstreet Boys disks. Eddie said nothing but did raise his eyebrows disapprovingly. Well, now those files are gone, but shouldn’t there be a way to export ratings from iTunes to Winamp?

After a few days of fiddling with Winamp, Angie gives up. She will stick with iTunes. She’s got so much content already in that library that she could not be bothered to switch. Maybe Eddie could get an iTunes converter for his precious playlists?” 

— Lehikoinen, Juha, Antti Aaltonen, Pertti Huuskonen, and Ilkka Salminen. “Chapter 4 – Metadata Magic”. Personal Content Experience: Managing Digital Life in the Mobile Age. John Wiley & Sons. © 2007.

What can you do about it?

Not much. Software companies have a financial incentive to make it easy to import data, but there is not much incentive to make it easy to export data. If your chosen product was very popular, as Eudora was, then there is a pretty good chance that a future product will be able to import your metadata.

Keep this in mind as you handle your files.

• If you can put keywords in the files themselves via EXIF or ID3 or similar tags, do so. Once they are there, they won’t be lost.
• Make the file names meaningful. It’s the lowest common denominator, and may be all you have at times.
• Make sure your files are searchable where possible.
• Try not to relinquish control over the folder structure. If possible, use a good folder structure that captures events and related documents.

This is a very deep topic, which is treated with respect in business and industry—many people specialize in managing data migrations from the old to the new. In addition, the diversity of metadata out there is startling; there are loads of products and standards for every industry out there.

At the end of the day, we face the very same problem that they do, with our humble email application.

I never thought that Eudora would go out of business, did you?

I would like to share with you a pair of similar free utilities, one for the PC and one for the Mac, that allow you to easily evaluate exactly where your disk space is going.

(This topic was recently discussed on the Mac Geek Gab podcast, which inspired this post.)

The Old Way

In the past I would often poke around the hard drive, checking out the “usual suspects,” occasionally finding a few plump temporary files or some stale unused applications that could be dumped. When I needed to do a more thorough cleaning, I would run a “find” for files larger than, say, 10MB, and I would sort them in descending order and hack away at the resulting list.

But I have never been satisfied with the results of my efforts—there has always been some lingering feeling that I just wasn’t seeing what the true space hogs were on my system.

Then I found an application that uses the concept of Treemaps to display your hard drive in a colorful visual form.

Treemaps

This is what a treemap of my Mac hard drive looks like:

The application has used color coding and rectangles to represent files on the hard drive in such a way that accentuates clusters of similar items.

Each directory is represented by some rectangular area in this image. The colors represent different file types (e.g. MP3 files), making it easier to spot large sets of similar files, and easier to visualize where your space is going.

For example, the thin yellow rectangle is highlighting the space (in blue) occupied by my iTunes music library. The large red area on the right (seen by the tool as a single file) is my iPhoto library. The area between those two sections is my current video recordings from EyeTV (in green), along with all of my text documents.

You can see quickly from this image that photos, video, and music occupy the lion’s share of my hard drive. Most of the remaining jumble across the bottom is the Mac OS X operating system and all of my program files.

At a glance, I can see that if I deleted all of my EyeTV recordings, I would free up a large percentage of my drive space.

It is through this tool that I realized how impractical tools like Xslimmer are for me. These tools save disk space by removing non-Intel binaries and unused language files (Mac applications often come with Intel and Power PC binaries packaged together). But even if I deleted my entire Applications folder, it would only free up a corner of the map above. I just wouldn’t gain much by slimming down application files and language resource files.

Since I work with both Mac OS X and Windows, I found a freeware version of this utility for each.

Disk Inventory X

This is the Mac utility that I used to make the screenshot above. Here is the application developer’s website.

Besides showing the treemap, Disk Inventory X provides a file browser tree and a sidebar with information about the color coding. You can bring up an Information pane that shows the details of whatever item you have selected in the treemap.

Again, I have selected my iTunes library. You can see in sidebar that blue means “MP3 Audio File.”

WinDirStat

This is pretty much the same application concept in an open-source version for Windows. Here is the developer’s web site. Scroll down to the bottom of their page for the download link.

In this screenshot from Windows, I selected my C:\Program Files directory. You can see the white rectangle on the right side, showing us exactly how much space all of my programs occupy.

Other notable features on this screenshot: my system page file is that fat bluish rectangle in the bottom right, and the collection of red in the upper left is a large quantity of Windows patches.

Summary

Both of these tools provide an excellent means of visualizing exactly where the fat is in your hard drive. The price is right. Try one today!