How can you keep your data from being held hostage?

Have you ever stopped to consider exactly how much information is permanently stored within your favorite applications, locked down to all but the most determined command-line commando?

Perhaps the easiest way to explain what I’m getting at is by way of an example…

My Wife’s Email

Some months back, my wife was moving from a PC to a Mac, and all that remained was to configure her email. She had been using Eudora on the PC for many years and had several thousand emails in her mail folders. It went without saying that there would be dire consequences if I were to lose a single scrap of her electronic correspondence.

My first thought was to simply install Eudora on the Mac and then copy the database from one to the other. But there was bad news waiting for me in Eudoraland… The product is discontinued and no longer supported. They haven’t released a version since 2006, and you can’t even buy a license to turn off the ads anymore.

Eventually, after quite some time in Google, I found that one could simply point Eudora to a Gmail IMAP account, then drag all of the messages from the local folders to the IMAP folders.

Unbelievably, it worked exactly as advertised. I dragged thousands of messages to her Gmail folders and the machine bogged down for a whole day as it uploaded everything to Gmail.

Garbled Messages

Once I finished the job, I inspected the moved messages. Much to my horror, many of them were displayed in XML format, with embedded HTML tags all over the place. A few experiments verified that this was not an error on my part. The same message would show up garbled in the Gmail web interface, the Apple Mail client, but would still appear cleanly in Eudora, served from Gmail.

The message above, shown in Apple Mail as XML, was rendered properly in a fresh install of Eudora on a clean PC:

Clearly, Eudora had mixed in some special sauce of its own when it stored all of those files on Gmail’s IMAP server!

In the end, we accepted the situation: from that point forward she was golden, using Apple Mail and Gmail. If she ever needs an old message, we can fire up Eudora on an old machine and view the message.

What’s the point?

Many applications that we use in our paperless workflows and lives have sinister tendrils working their way throughout your data, slowly adding value to your life by their secret internal data that they will never relinquish. You may never notice just how dependent you are on their “secret sauce” until you are forced to change applications.

Tagging

One of the most harmless appearing kinds of secret data comes in the form of tagging. Just exactly where do you think those tags in your iPhoto or Picasa library are going? If you wanted to switch from one of those fine products to the other, how would you get thousands and thousands of tags saying “Rome, 1998″ and “Joey’s Graduation” moved over?

The problem is that these applications keep their tag information in a separate database from the actual images. Many similar applications suffer from the same problem: if you tag documents in a product such as DEVONthink, your tags are stored separately from your files.

There are standardized ways of embedding certain kinds of tags in many multimedia files, such as the ID3 tags for MP3 files and metadata tags in PDF documents, but often applications do not modify these in an effort to avoid touching your original.

And the day you want to leave them, they will hold your tags hostage!

Mysterious Black Boxes

I blame iTunes for the plethora of black box applications that encourage you to feed them multimedia files, which they promise they will squirrel away in a perfectly sensible and secure location.

When you drop a bunch of MP3 files on iTunes, they get shuttled into the relatively orderly iTunes library. When you drop a set of photos on iPhoto, they go into a pretty odd arrangement of folders that are created based on dates and source folder names. Sometimes iPhoto pays attention to the file timestamp; other times it uses the EXIF data within the picture.

DEVONthink does similar magic with PDF documents; they tell you to dump stuff in there and let DEVONthink’s smarts sort it all out for you.

This file organization is a different kind of “secret sauce” that you would lose if you were to switch horses.

An Unhappy iTunes Story

I smiled when I read this silly little anecdote:

“Eddie finally came in with his portable disk and uploaded his 85GB collection of MP3 files into her computer. Okay, it might be illegal, but who cares – everybody’s doing it. Eddie looked at her importing the MP3 files into iTunes, Angie’s favourite player, and suggested that she switch to Winamp, which is just a way cooler piece of software. Angie wasn’t sure. Eddie explained that he had all those carefully crafted playlists for Winamp, mixing the files he just gave to Angie into really spacey all-night sessions, and Angie should really try it out.

Angie did install Winamp with Eddie and has opened it a few times since then. She still doesn’t know … well, in fact she does. That software just doesn’t feel right to her. What’s worse, Angie has spent years in rating her song files with iTunes, and quite rarely hears any bad pieces anymore. But if she goes for Winamp, all those ratings seem to be lost.

Wouldn’t you know it, just after installing Winamp it started – most embarrassingly – playing her forgotten Backstreet Boys disks. Eddie said nothing but did raise his eyebrows disapprovingly. Well, now those files are gone, but shouldn’t there be a way to export ratings from iTunes to Winamp?

After a few days of fiddling with Winamp, Angie gives up. She will stick with iTunes. She’s got so much content already in that library that she could not be bothered to switch. Maybe Eddie could get an iTunes converter for his precious playlists?” 

— Lehikoinen, Juha, Antti Aaltonen, Pertti Huuskonen, and Ilkka Salminen. “Chapter 4 – Metadata Magic”. Personal Content Experience: Managing Digital Life in the Mobile Age. John Wiley & Sons. © 2007.

What can you do about it?

Not much. Software companies have a financial incentive to make it easy to import data, but there is not much incentive to make it easy to export data. If your chosen product was very popular, as Eudora was, then there is a pretty good chance that a future product will be able to import your metadata.

Keep this in mind as you handle your files.

• If you can put keywords in the files themselves via EXIF or ID3 or similar tags, do so. Once they are there, they won’t be lost.
• Make the file names meaningful. It’s the lowest common denominator, and may be all you have at times.
• Make sure your files are searchable where possible.
• Try not to relinquish control over the folder structure. If possible, use a good folder structure that captures events and related documents.

This is a very deep topic, which is treated with respect in business and industry—many people specialize in managing data migrations from the old to the new. In addition, the diversity of metadata out there is startling; there are loads of products and standards for every industry out there.

At the end of the day, we face the very same problem that they do, with our humble email application.

I never thought that Eudora would go out of business, did you?

On my last birthday, I received one of the coolest cards from the kids. It was one of those cute cards that people give that are “from the cat.” In this case, however, they had chased down each one of our felines and subjected them to a forced paw-printing exercise, which you can see at the right, complete with each cat’s name.

Now that’s just too cute for words. Who can imagine ever throwing out such a neat card?

How about the card that my wife slipped in my luggage last October when I went to India on business? That really warmed my heart to read while I was unpacking my luggage in a Bangalore hotel. That one’s a keeper too.

The problem is not one individual card; as the saying goes, “No single snowflake feels responsible for the avalanche.”

Besides the sentimental value, quite often cards hold important information: names and addresses. In the treeware world, people have been saving the corners of envelopes for years in an effort to keep the important bits while tossing the rest.

You know where this is going, don’t you?

Scan it all in

Once you have accepted the fact that you have never actually sat down with your shoebox to read twenty years of cards, and that no one will be hurt at their loss, you can calmly sit down and begin scanning them in, so that you always remember them.

I found that a sheet-fed scanner works wonders, but I often have to snip on the fold lines and feed in the individual pieces, assembling the whole lot into a multi-page PDF when I’m done.

Some kinds of cards just don’t work in a sheet-fed scanner: lumpy cards covered in glitter with popups and stuff. I use my flatbed scanner for those.

No matter how I do it, I make sure that in the end I have a single PDF that contains the whole card, and no blank pages. I try to make the pretty part of the card be the first page, so in any thumbnail view I will see the card.

If the envelope has useful information, such as a return address, just scan it in as well and attach it as another page to the PDF.

Make sure to pick a good name for the file; I choose to use the date of the event followed by the type of card and maybe some special info about the event. Such as 20081225 Christmas card from Mom.pdf. That’s unambiguous and can be found regardless of what kind of document management system you use (if any).

Not just greeting cards

I’m certain that your shoebox has other flat paper items that aren’t greeting cards. Some are letters that you exchanged with your spouse when you were dating, perhaps others are letters from distant family members. Maybe someone sent you a copy of a photo of the eight of you on that fun rafting trip back in ’92. The point is, you can go through all of this and handle it accordingly:

• Set aside anything that must be kept—things like stray automotive titles and precious letters from deceased family members.
• Scan in all the rest, creating PDF files that contain exactly one letter, in its entirety, with envelope.
• Don’t bother running OCR on it unless it is typewritten or the address is a printed label.
• You may wish to scan in any photos at higher resolution in your photo management software (such as iPhoto or Picasa).

Once you have everything checked and safely backed up, you can recycle all of that paper and use the shoebox for something useful like network cables and power supplies.

When you are all done, you will have plenty of beautiful memories on your computer (and backed up!) that you can.

[Update]

Last weekend, a friend mentioned that her mom taught her to always keep the most recent letter from a family member. This way, if the person dies, you will have the last letter they sent to you.

She still has the last letter her mother wrote before she passed away.

The message is clear: Don’t be a fool. Keep precious letters that have great sentimental value. You probably do want to scan them in, so that even if they are lost in a fire, you still have the scan.

But think very carefully before taking anything to the shredder.