I like the playlist tools that iTunes provides, and I find it very convenient to create Smart Playlists or to create Genius lists. However, I want to keep my Master Library of music elsewhere, far from iTunes. Wouldn’t it be great if I could share these playlists on my home network in a completely nondenominational format? Wouldn’t it be nice if my Squeezebox server would have those same playlists available? And wouldn’t it be spectacular if those playlists would magically appear on a network drive whenever the lists change in iTunes?

Read on for details on how I accomplished this and to download the free utility I wrote to handle this task.

The Problem To Be Solved

In order to understand what this application does, you need to know what I was trying to accomplish.

I have a NAS drive on my network that contains my Master Library of mp3 files. The files on this drive are located under /media/music.

I have a Squeezebox server, running on an Ubuntu machine. This server is used by Squeezebox devices to play music anywhere in my house. It holds a copy of the Master Library, located under /srv/squeezebox/music

What I want: I want my Mac to magically push proper m3u playlists to the Squeezebox server—the Squeezebox server understands m3u playlists.

What I don’t want: Squeezebox supports direct access to iTunes. But I don’t want to do this because I don’t want to have to run Squeezebox on my Mac. Why did I bother setting up a dedicated server if I have to keep my Mac running?

Why is this challenging: Even if I could point Squeezebox to the bare iTunes database file, all of the file paths are wrong. My master library might have a song at …/artist/album_name/disc 1/, while iTunes might put it at …/Artist/Album Name/, omitting the disc 1 folder and reformatting names.

My Solution

I wrote a small command line application in Perl that supports the following:

• Paths to songs in Apple’s mysterious black box are converted to nice paths to the same songs in my golden Master Library.
• Even if a song name changes or a folder path changes, the original master copy is found.
• All playlists are generated in m3u format.
• Smart playlists and Genius playlists are supported.
• The generated playlists are automatically moved to either a local directory or a remote server, using scp.
• Network shares are automatically mounted at beginning of process and unmounted at end of process.
• You can provide a file path to be prepended to all playlist entries.
• Exits early if the iTunes library has not changed since the last run.
• Supports exclusion patterns in order to skip some playlists.
• A local Squeezebox server can be automatically pinged to cause it to refresh playlists.

Here is the full documentation for the Playlist Mapper application.

Basic Installation

Please note that this is a command line program. Some day I might figure out how to make a pretty GUI installer for it, but for now you have to get your hands a little dirty in order to install and configure.

Download the application here: plmapper

This is a Perl script that uses many standard utilities already present in a Snow Leopard OS X installation. You should not need anything other than the plmapper file itself.

Unzip the file and place plmapper in your home directory.

Open a Terminal window and make the script executable:

chmod u+x plmapper

Read the man page for the application by doing this:

./plmapper --man

There are many command line arguments, but you don’t need to worry about most of them. Create a file called plmapper.config in your home directory and put any arguments you need in that file, as name-value pairs.

Example:

itunes=/Users/Bozo/Music/iTunes Music Library.xml
dest=/Users/Bozo/Desktop
library=/Volumes/media/music

Once you have added all of the configuration settings, go ahead and run it:

./plmapper

You should see plenty of status information go by as each playlist is processed.

Making it Run Automatically

There are better ways of doing this, but I’m a command-line commando, so I simply added the following cron entry (see crontab):

0 22 * * * /Users/Me/bin/plmapper >/Users/Me/plmapper.log 2>&1

This causes the plmapper program to run once daily at 10pm, logging output to a file called plmapper.log in my home directory.

Of course, this can be tweaked to taste; it doesn’t matter if you run it hourly since the app skips any heavy lifting once it sees that the iTunes library file hasn’t been touched.

How It Works

The application performs the following basic steps:

• Mounts network shares using mount_smbfs.
• All music files in the external non-iTunes Master Library are listed in an internal data structure, sorted by file size.
  This allows us to quickly find a “short list” of possible candidate files that match a given source file. This works because mp3 files have fairly random sizes.
• The iTunes database XML file is processed using an XSLT transformation to generate a set of basic m3u playlist files. The xsltproc command line utility is used to run the transformation.
• All paths in the playlists are converted from escaped URI syntax (e.g. %20 for space) to regular UTF-8 characters.
• Each path is resolved to a single music file (i.e. an mp3 file) in the Master Library.
  This is done by obtaining the size of the iTunes version of the file and then looking this up in the internal list created in the first step.
• Once a short list of candidates is found, the first few thousand bytes of each file are compared until a match is found.The risks of choosing the wrong file are low: all that will happen is that a song might be switched in the playlist accidentally. As such, it isn’t worth scanning the full file to ensure they are exact matches.This algorithm works quite well, and can detect music files that have been renamed and moved around.

• The playlist files are now converted from utf8mac to utf8 using the iconv command line utility.This process collapses wide UTF-8 characters, consisting of a letter plus an accent add-on, into their single-letter equivalents. This process is known as Unicode Normalization, and this tool uses Normalization form C: Compatibility decompisitiion followed by canonical composition.
• Playlist files are now copied to a local or remote directory, either as direct file copy or via the scp utility.
• Optionally, a special call is made to a specified Squeezebox server to tell the server to refresh all playlists.
• Any mounted shares that were mounted by this process are unmounted via diskutil unmount.

Limitations

• Nested folders of playlists currently act funny. The lists will come over, but you may get two lists. You can use the exclude option to prevent this.
• Songs with non-ASCII characters in the file name might not be recognized by all players. Squeezebox has a problem with many special characters—I tested this by adding songs like Águas de Março.mp3 using Squeezebox’s own playlist editor, and when I reloaded the list the songs were not there. This is a bug in Squeezebox server.
• Occasionally the file matching algorithm might pick the wrong song. In an effort to speed up processing, the program only looks at the first few thousand bytes of files when comparing. It isn’t a big deal if the wrong file is added to a playlist, is it?

Closing Thoughts

As often happens, this program was written because I had an itch that needed to be scratched. Now that I went through all of the hassles, I hope others can benefit.

Through good fortune, I am working in a Macintosh environment. This means that all of the important tools were already present on my machine: perl, xsltproc; mount; iconv; scp; nc; and others. If I were writing this for a PC, I probably would have written the whole thing in Java—a more familiar language to me, but one that comes with its own configuration issues.

I plan on updating plmapper over time, and if anyone has suggestions, please let me know.

And if a Perl guru wants to tell me my Perl skillz are subpar, go right ahead—but be gentle, and let me know how to improve the app.

[Update]

Fixed a couple of bugs related to path and “qx” calls, thanks to Chris. Updated version here.

I have been struggling with making everything digital for years now, and there are stacks of photos and thick albums that remain uncharted territory. Every once in awhile I sit down and scan in an album.

Trouble is, unless you do the whole job at once, it’s pretty easy to lose track of where you have already been. And if that’s not bad enough, we always bought double prints—they didn’t cost much more, and we could give some away. Now how do I know if I have scanned the long-lost twins of some photos that are already on my computer? That’s a problem for another day.

Here’s how I make sure I don’t scan the same single photo twice:

That’s right, I leave breadcrumbs in the form of little dots on the back of each one.

Such a simple thing, but it saves me from a lot of head scratching and wasted time scanning the same stuff over and over.

Scanned photographs are an entirely different brew. They typically represent events from the distant past, and scanner software EXIF data is hit or miss.

I looked for commercial software to handle a few special cases of EXIF data troubles, but came up empty handed. So I wrote a few useful AppleScript droplets that do these tasks quite nicely, and I will share them here.

Warning!

These scripts use jhead to manipulate and rewrite your JPEG files!

Don’t be a fool. Experiment first with a safe set of throwaway JPEGs. And never use these tools on original files; always keep a backup.

Prerequisite

All of these scripts depend on a fine piece of free software called jhead written by Matthias Wandel.

Installation is not difficult, but it does involve the command line.

• Go to the jhead site
• Scroll down to the Releases section and look for Pre-built OS-X Intel executable
• Right click on the jhead link on that row and choose Save Linked File to “Downloads”

At this point, I found that jhead was saved as jhead.txt. Oh well. We needed to do some command-line magic anyway.

• Open a terminal window and enter the following:

These lines do the following:

• Lines 1 and 2 navigate to the Downloads directory and remove the “.txt” from the name
• Line 3 makes the file executable by everyone on your Mac
• Line 4 places the file in a public area where everyone on your Mac can see it (you will be prompted for your password)

Stripping All EXIF Data

Sometimes I receive files that have corrupt EXIF data. I had a large quantity of scanned files in my collection that claimed to be scanned some time in 2038, while others insisted that they had been around since 1901. Neither situation is good, and I found that standard EXIF editing tools may fail to change these corrupt EXIF sections.

The answer is to blow away the EXIF data.

Download: Strip EXIF

This zip file contains a compiled AppleScript application. You can unzip it and place the application on your desktop. Safari will probably unzip it for you when you click the link.

To be safe, open AppleScript Editor and use it to open the Strip EXIF app to see its magic.

Now you can drop any number of JPEG files onto the Strip EXIF app and it will kindly eviscerate each JPEG, removing all traces of EXIF data.

Adding Basic EXIF Data to a vanilla JPEG

Some tools create JPEG files without EXIF date and time information within. This is typically the hallmark of photo manipulation software and dodgy scanner software. And if you happened to use the Strip EXIF app to rip out a bad EXIF block, then you will want to replace it with a proper data block so that you can still use camera date and timestamps.

Download: Add Basic EXIF

Again, unzip the file, place the app on your desktop, and then drop any number of JPEG files onto Add Basic EXIF.

The app will set the EXIF date to the file creation timestamp.

Spreading EXIF Timestamps

This is the real reason why I wrote these scripts. I couldn’t find a satisfactory tool on the market that would allow me to automatically spread out the shooting times for a series of images.

Why would anyone want to do this? Because some processes give you fifty JPEG files all with the exact same creation time and exact same shooting time. I like to use file renaming tools to incorporate the shooting time in the filename, so that files sort by chronological order. This doesn’t work if all of the timestamps are the same.

So I wrote a little app that adjusts the first photo by one minute, the second by two minutes, and so on. If there are fifty photos, then the last one will have its shooting time adjusted by fifty minutes.

The result is a series of photos/scans that have different timestamps.

Download: Spread EXIF Timestamps

Again, please look at the short program before you run it.

Closing Thoughts

I hope that my favorite tool implements these tricks soon (A Better Finder Attributes, I’m looking at you!), but until then, I will be dropping my files onto these three little droplets.

The jhead tool is so versatile that I will probably end up with a whole slew of similar droplets that will do all kinds of spiffy stuff. Nevertheless, I would rather the commercial products already provided these features. Not everyone likes dipping into AppleScript and the command line!

The problem is this: with Outlook we will be limited to 1000 messages in our in-box; sadly, many of us have tens of thousands of emails in our old GroupWise mail. Even after a fairly rigorous slash and burn mission, hacking out all of the low hanging fruit, there will be many thousands remaining and I don’t want to lose that information. It might be useful to search and find how I set up a Zebra bar code printer in 2003, no?

A bundle of different freeware glue tools came to my rescue. Read on to hear about the toolset that has made it so I can keep those messages for years to come.

Possible Solutions

Right out of the gate, I began looking for ways to migrate messages from one mail client to the other. Some apps have this built right in, and if not, there are scripts and utilities out there to do this; but I was hampered by a few key facts:

• I have no control over the email clients and their configuration. Even if there is a menu option for exporting GroupWise messages from version 7.2, I’m stuck at 6.4 and cannot use that option.
• GroupWise is a minor player in the email world. I’m not sure if Outlook would import from GroupWise, but I doubt it.
• They are replacing the client in one shot. There will be no interim period where both GroupWise and Outlook will be available.
• There is no getting around the hard limit of 1000 messages.
• I don’t want to spend money on this.

With these constraints in mind, I immediately thought about PDF documents. I then considered the following questions:

• How do I convert my email to PDF?
• How can I do this automatically with thousands of emails?
• Once I’m done, how do I search these documents?

Here’s what I did:

Conversion to PDF

The first part was easy. I downloaded one of the many free print-to-PDF products available.

I chose PDFCreator, because I am familiar with its use and I know that it does not munge the fonts.

Like many other PDF generation utilities, PDFCreator functions by providing a virtual printer to which any application can print. For example, to make a PDF of a web page, you use the Firefox Print menu and select PDFCreator from the drop-down list of available printers.

You are provided with a list of metadata fields that you can fill in, and these fields are used in the PDF generation.

Here’s what the PDFCreator screen looks like:

A word of caution: PDF Creator is free, but you must be careful to deselect their spammy toolbar options in two different places during the installation process. I don’t like software that comes with preselected toolbars to install (even nice ones like Google’s) because I’m certain that 95% of the folks who actually install the toolbar would never have chosen to do so if it were unchecked by default.

Running Everything Automatically

This was the interesting bit. I work with Windows machines at work, so there was no AppleScript option available. So I did the next best thing: I used AutoIT.

I will warn you that AutoIT is pretty much the Windows analog of AppleScript, without the cutesy pseudo English syntax. In other words, you will need to roll up your sleeves and get your hands a little dirty in order to put together a decent AutoIT script.

The payoff comes when you finish your work and compile it into a tight executable that you can share with your friends, allowing them to automate some complex series of button clicks and copy/paste operations.

I walked through the manual process of exporting an email to PDF and listed each action:

• Get the date, sender, and subject
• Create a filename based on date + sender + subject
• Launch the Print dialog
• Select PDFCreator
• Fill in the Document Title, Creation Date, and Subject in the PDFCreator dialog
• Fill in the full file path in the Save dialog

In addition, I wanted to make the script a little better by adding the following:

• Check that user has PDFCreator installed
• Verify that GroupWise is running and that the user has selected one or more messages
• Prompt the user for a target directory before processing the messages
• Sanitize the filenames by replacing illegal characters with underscores and truncating to meet maximum filename and path length in Windows
• Skip over files that have already been generated, quickly, so that one doesn’t need to worry about accidentally selecting messages that were already printed

There were other adjustments needed, but the process was the same: run the script, hit a problem, tweak the script a little to address the problem, and repeat.

Here’s a little bit of the AutoIT script:

You can see that it is a bit more intense than AppleScript, but remember that the full script wasn’t written in one go. I had a little short ten-line script that I kept tweaking as small problems cropped up until I had adjusted things to my liking.

Note that this is a GUI macro language. The machine starts clicking and typing away right in front of you and you probably shouldn’t interfere until your script finishes.

As of this afternoon, I have generated around 4,000 PDF documents for my email messages.

Searching All of Those Documents

This was the easiest part. These days there is an excellent tool available for searching documents on your desktop: Google Desktop. This product indexes every useful file on your desktop and provides a full Google search with a quick double-tap of the <control> key.

So you can enter a search like “Zebra bar code”

And the results look exactly like a Google web search, but it’s showing your desktop files. And you can see inline previews too.

Macintosh users can install Google Desktop as well, but all of these files should already be indexed and searchable by Spotlight.

Closing Thoughts

Whenever I reach for tools like this I feel a twinge of guilt—it’s outright hackery, isn’t it?

But there is a place for quick and dirty jobs in every workplace. I needed to get my files from one place to another, one time only. It just didn’t make sense to spend money or time on a more elegant solution.

Play around with each of these tools a little. Especially AutoIT—it’s a handy Swiss Army Knife to have at your disposal.

As it turns out, my original script does not work in Snow Leopard, and I promised that I would one day clean up and publish my new and improved version.

Chris posted a comment today as a gentle reminder, so here is the new and improved version without further delay…

The Details

Unfortunately, Snow Leopard came around and caused some indigestion. For starters, the ScanSnap Manager didn’t work correctly and Abbyy Finereader would not process anything made by the ScanSnap. A couple of months later they got everything straightened out and delivered new versions of each product.

The new version of the Abbyy Finereader product does not play well with my original script.

Since I cannot do without this important functionality, I rolled up my sleeves and rewrote most of the script. The new version works in Snow Leopard quite nicely with one small annoyance: you really don’t want to try to use the machine for anything other than scanning or OCR while it is going because the new Finereader version keeps bouncing the darned icon all the time it is running and that is quite annoying to watch.

Fortunately, I really don’t need to use my machine for anything else while it is chewing on the docs; I just wanted to be able to continue scanning at the same time!

Note: Before going forward, note that you will need to upgrade the ScanSnap Manager and Abbyy Finereader to the Snow Leopard versions first! Get the files here.

Here is a link to the new script…

And here’s the code itself:

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action Scripts
  You may have to create the Folder Action Scripts folder.
• Now open a Finder window and navigate to the parent folder of your scanned documents folder.
• Right click (control-click) the scanned documents folder and choose Folder Actions Setup…
• At this point if folder actions are not enabled, you will likely have to enable them and add the script manually.
  • Check Enable Folder Actions
  • Use the “+” buttons on the left and right sides to add the scan folder and then this script.
• Otherwise, a list of scripts will come up. Choose this script from the Choose a Script to Attach dialog.
• Close all windows.

That’s it! The script will be invoked automatically every time a new file appears in your scanned documents folder.

Please let me know if you have any ideas that can improve this script. I’m not an AppleScript guru, so someone might just know how to keep that annoying Finereader icon from jumping.

I had promised that once I cleaned up the embarrassing bits of code I would post a perfect polished version here, but such promises are rarely fulfilled. A reader posted a comment asking for that source code, so I will post it here in its current state. The truth is, I have been using this script for months and, though it has some quirks, it works fine.

So this post is about Macintosh, AppleScript, and the ScanSnap-to-FineReader workflow. If these don’t interest you, better move on.

Update: The script on this page works only with Leopard (10.5). Get the Snow Leopard version here

The Original Problem

The Fujitsu ScanSnap S510m, my workhorse scanner, was designed to scan documents quickly and generate PDF files—this it does flawlessly. In order to provide OCR support, they have shipped a special version of FineReader, called FineReader for ScanSnap. The standard OCR configuration is to chain the output of the scanner to the FineReader program.

The problem is that this forces scanning and OCR to run in lockstep: you scan a document, you wait for OCR, and then you scan another document.

My desire was to write a simple AppleScript that would detach the “Scan a Document” process from the “OCR” process. By using this script, I can scan documents at whatever rate pleases me, and the OCR engine will chunk along at its own pace, consuming my scanned documents and producing OCR documents.

My Approach

I really looked hard at the OCR application, trying to find AppleScript hooks or special command line switches that might allow me to control it better. Sadly, it was not designed to be scriptable. The only thing I could do is call the FineReader application with a source file.

Given this limitation, I considered writing a script that would look at a particular folder, identifying new files as they appear and passing them on to FineReader.

Fortunately, AppleScript provides this kind of functionality with little effort in the form of Folder Actions. Perhaps the best way to see these in action (and try it out) is to see this post on Exploring the power of Folder Actions.

In order to achieve my goals, I did the following:

• Created a folder called “Pending Documents”
• Wrote the script to find the oldest-unprocessed-file and call FineReader with it
• Attached the script to the folder as a Folder Action

The Script

Let’s jump right in to the AppleScript. Download the script here.

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action ScriptsYou may have to create the Folder Action Scripts folder.
• Now open a Finder window, control-click and choose More / Configure Folder Actions…
• Check the Enable Folder Actions checkbox, if not checked
• Click the “+” in the bottom left
• Select a folder and click Open
• Choose the script Run OCR on New Folder Items and click Attach

Picky Details

As you can see in the source code, there were several issues to address:

• I had to make sure the script didn’t step on itself. If FineReader was running, I would wait until it was ready before processing.
• The script needed to determine which files had been processed already. This was handled fairly trivially by looking for a matching file with the processed by FineReader.pdf suffix. In other words, if I was looking at Scan001.pdf, I would see if there was a matching Scan001 processed by FineReader.pdf file.
• Part of checking for a source file’s “buddy” was stripping off the PDF suffix. This was done in a hackish way by using a one-line shell script, at lines 106-107.
• I thought it was important to verify that the source file was, indeed, a ScanSnap file—the FineReader will not process other PDF documents. This was done at lines 117-121 by looking at the Spotlight metadata for the Creator of the source file. That took some more shell scripting (133-154).
• The actual work was done by a single line, line 63.

The real work was fairly simple, while the bulk of the code was needed to polish pesky little details. Isn’t that the way code development often is?

If anyone has any improvements on my script, please let me know!

I have always been a big fan of HowStuffWorks, with their detailed in-depth articles describing such disparate topics as manual transmissions and money laundering.

Anyway, author Diane Dannenfeldt has written a lengthy article on How Paperless Offices Work, giving ample coverage to myriad aspects of the topic:

• Introduction to How Paperless Offices Work
• Benefits of a Paperless Office
• Transitioning to a Paperless Office
• Managing Digital Documents
• Going Paperless at Home
• Paperless Office Solutions

Take a look at the full article here: How Paperless Offices Work (howstuffworks.com)

The basic tool required to bridge the gap between the carbon-intensive paper document present and the greener electronic document future was invented in 1993 as a way to streamline communications between publishers and printers.

Today, the humble PDF file is the file format of choice for “final” electronic documents. Properly created and deployed, PDF alone serves a critical need in reducing both business costs and environmental impacts.

Duff Johnson of Planet PDF wrote an article about two of my favorite topics: PDF Documents and Paperless Offices.

In the article, he discusses several key motivating factors for fitting PDF into your own workflow, with the underlying goal of making your office a little bit greener.

Read it all here: PDF is green tech for your office

Here are a few thoughts I have to share on these great media library projects. Not only are they never-ending, but the longevity of the task should guide some key decisions you make as you plod along.

Accept that it’s Never Going to End

I currently have ongoing projects to scan in every scrap of paper in my house, rip every CD to MP3, and scan in every pre-digital-era photograph. But the corpus is just too large. And I haven’t even considered ripping video; I have accepted that a full digital video library is a little too ambitious for me these days.

We need to be aware of this from the very beginning; if you feel like you bit off more than you can chew, you are normal.

Never Give Up

Try to keep your goals clear, so that you don’t lose steam and give up. Perhaps you are looking forward to being able to place the plastic boxes all in storage. Perhaps you can’t wait to be able to make those uber party mixes from your rich collection. In any case, every hour you spend doing the drudgery of ripping CDs brings you a little closer to these goals.

Prioritize Your Work

It is a whole lot better to spend your first days ripping your favorite music to MP3 than to be ripping some weird CDs your roommate left behind in college. Today I happily skipped over the Talking Heads CDs that I inherited from my brother—those don’t really need to be in my iTunes library, do they?

Keep it Simple

Make your work flow as simple as possible. You may still be doing this in three years—don’t over complicate things.

I have been working on my own collection for the past three years. When I first started, I was ripping the music on a PC using a tool called cdex to rip the MP3s and another tool called Mp3tag to edit the tags. These days I use iTunes on my Mac to do both of these tasks. If I had some really fancy work flow going, it might not have been so easy to change machines. By keeping things simple, I was able to do exactly the same thing today that I did three years ago.

Choose Good Settings

…for your software. You don’t want to have to do this again, do you? And don’t go cheap on the bit rate, if you have the space.

In addition, you will probably want to decide between the convenience of a universal format versus the power of a proprietary or niche format. In my example, I opted to use MP3 for all of my music instead of some of the other, more powerful, formats available. I was stung once by the switch from PC to Mac where the handful of CDs that I had ripped to .wmf files were no longer usable.

If you are ripping everything to some killer lossless format, be sure you are looking far down the road and considering compatibility issues.

Back it up!

Make sure your backup process is ready for the increased load. If your backups consist of burning DVDs (or worse, CDs) of your data, you will very quickly tire of this, and you may just abandon one of the most important parts of your work flow. Oh, and remember, if there aren’t two copies in separate places, it isn’t a backup.

A couple of months ago I bought two of those nice pocket-size portable hard drives, and I always leave one at the office and I swap them once per week. This works for me: they are 320GB hard drives, providing plenty of room for all of my thousands of scanned documents, tens of thousands of photographs, and many thousands of MP3s, with plenty of elbow room.

When ripping a CD collection, the data risk is lower since you still retain the original CDs, but it still is a lot of work to rip them all. Don’t overlook the backup, or you will cry when the hard drive crashes.

Know Where You Have Been

Come up with a simple “bread crumb” mechanism to know where you have already been. You want to be able to quickly identify the music CDs you already ripped, without having to bring up iTunes every time. Keep the three year horizon in mind, so you choose a technique that will not be mysterious to you after some time passes (Hmmmm…. did I dog ear the corner of the CD insert to mark it? Where did I put that notebook where I wrote them down?)

I chose simplicity: I have one of those nice metallic Sharpie markers, the silvery kind. I put a small dot on the hub of each CD after I ripped it, just above the center hole. This way, several years from now, I should easily be able to spot the CDs that haven’t been introduced to my digital library.

I do this with photographs as well. As I scan them, I put a small marker dot on the back of each photograph, in the corner.

Finally, Enjoy Your Music!

I am quite capable of getting all wrapped up in the process for the sake of the process. Don’t forget to actually listen to the music you ripped!

Of course, it is only a matter of days, or even hours, before the new glow has worn off and I begin to grouse about the things I wish they had done and I grumble about the nit-picky little bugs that are always there.

Even when the program works perfectly, we often want to use it in a work flow, using each program for what it does best.

Work flow is inherent in the kind of work that we do when scanning, indexing, searching, filing, tagging, and backing up all of our documents, photos, music, and video. Once you are committed to digital media, you will find that you often need to cobble together different programs in order to do away with some of the tedious manual labor.

Scripting tools can be as simple as macro recorders, as fancy as drag-and-drop pipelining tools, and as complex as full programming languages. Choose one that fits you and use it to bridge the gaps in your work flow.

An Example

I have never had a harsh thought about my sheet-fed scanner, the venerable ScanSnap S510m. But one thing has always annoyed me about the software package they provide: you can’t start scanning a new document until the OCR process is complete on the first.

What happens is this: you scan in a document; ScanSnap Manager receives it as a PDF and immediately passes it to Abbyy FineReader for OCR. All is good at this point.

But, the OCR process can take anywhere from 10 seconds to a few minutes, and if you were to scan in another document, FineReader complains:

This is where a little bit of AppleScript magic smoothed over this minor work flow annoyance.

I found that you can attach a bit of AppleScript as a “folder action” to a folder, and when a file appears in that folder, your AppleScript gets called. All I needed to do was figure out a way to get AppleScript to do the following:

• Wake up when a file appears
• Grab the first unprocessed PDF file
• Tell FineReader to start up, if it isn’t running
• Tell FineReader to open the file
• Wait for the processing to finish
• Go back to step 2 until all files are done.

It wasn’t exactly trivial—FineReader doesn’t actually have any special AppleScript support built in. But after a few experiments (i.e. a full weekend of fussing around) it was working smoothly.

Now I can scan as many documents as I want, knowing that OCR is happening at its own pace in a totally different flow.

Some day, when I have cleaned up the embarrassing bits of the code, I will share it here for fellow ScanSnap users.

What scripting tools are out there

The automation tools you use will depend on what software you are working with and what operating system you are using. On the Macintosh side, AppleScript and Automator will probably cover the bases. Things aren’t so pretty on the Windows side, but if you are willing to get a little dirty, Windows scripting can work miracles.

Whatever tool you use, I recommend finding as many simple examples on the Web as you can and running them and tweaking them slightly until you make them do what you want.

I am not an AppleScript programmer; I simply downloaded other people’s sample scripts that did file manipulations and poked and prodded them into shape.

AppleScript

In the Macintosh world, the old standby has always been AppleScript, with its quirky syntax that attempts to look like English. It is surprisingly easy to do some fairly heavy lifting with a minimal of effort.

In fact, though I find its syntax challenging and condescending at the same time, I heartily recommend AppleScript. The key is that Apple has built in AppleScript support to most of the main Macintosh applications.

For a taste of what AppleScript can do, here’s a whole mess of cool scripts for automating iTunes.

Imagine running a script that builds playlists for you or tags files automatically or finds duplicates and marks them.

This is the main AppleScript site, and here’s a tutorial from Apple.

Automator

Another Macintosh product, that provides a more graphical approach to hooking your apps together, is Automator. Here is an article that gives a good overview of the tool.

This is what a simple work flow looks like in designer mode:

You can see more or less what is going on here: You use a predefined set of actions, on the left side, to build up a pipeline on the right side.

In this case, the new automation action will get the selected documents, sort them, combine them into one PDF document, and then open that document in a PDF viewer.

I have always been a little confused with the utility of this product: it’s great if your application provides Automator actions, but what do you do if your app is like FineReader, with no special hooks built in?

Nevertheless, with Automator, you can do some pretty fancy things with many common tasks such as file management, email, and browsing. And there are plenty of custom Automator actions out there for the downloading.

Both Automator and AppleScript are Macintosh OS X built-ins.

For the serious Mac user, there is Bash Unix shell scripting, but that’s all I’m going to say about that. It’s there if you want it!

Windows Scripting

Things are somewhat different on the Windows side.

On the positive side, I believe that Microsoft’s COM object architecture makes it possible to script some pretty fancy things using off-the-shelf software.

In addition, the Office suite has amazing macro facilities that have been used by many as a full programming language, as the core of many successful commercial products.

But Windows just doesn’t seem to have anything analogous to AppleScript for gluing things together without getting your hands too dirty.

As a trivial example of the power of the Windows COM architecture, this short script will launch Excel, create a fresh spreadsheet, and automatically feed in some data. To try it out, save the script to your desktop and double click it.

This is what I saw on my machine when I ran the script:

Just like AppleScript on the Macintosh side, many Windows applications provide hooks for scripting.

The down side is that the scripting language is much more like a programming language, so it’s more comparable to writing shell scripts than to using a drag-and-drop automation tool like Automator.

But when you do the same thing over and over every day, it may be worth it to roll up your sleeves and get down and dirty with some VBScript.

An added benefit of this approach is that the language shares much of its functionality with Visual Basic for Applications, found in Word and Excel macros. If you are writing standalone Windows scripts, you can also write very complex Word and Excel macros. You also have the ability to do much of the same application automation within Word/Excel macros.

AutoIT

This is a freeware tool that many use for scripting their applications.

From the product website:

• Easy to learn BASIC-like syntax
• Simulate keystrokes and mouse movements
• Manipulate windows and processes
• Interact with all standard windows controls
• Scripts can be compiled into standalone executables
• Create Graphical User Interfaces (GUIs)

What more could you ask for?

Closing Thoughts

There are other tools out there; don’t be afraid to check them out and try some of the examples. If you can automate just one annoying task that you do every day, it may be worth the hassle of spending a weekend writing the script.

Epilogue

Yesterday at work I finally rolled up my sleeves at work to tackle a nagging document problem that had been lingering for months.

We have a large quantity of PowerPoint slide decks that have critical company information embedded within. I work in the pharmaceutical industry, so many of those PowerPoint slides had chemical structures for various drug molecules embedded within as OLE objects.

The problem was that we wanted to extract that knowledge (the molecule files) electronically and text mine the slides, but our tools did not support PowerPoint. They did, however, support Microsoft Word.

How could I convert the slides to Word documents? There are dozens of ways on the Net to go in the other direction, but the only way to do what I wanted resulted in the OLE objects being converted to pictures—no good for my purpose. I did find that a copy/paste of each molecule from PowerPoint to Word worked correctly, but who wants to do that by hand with hundreds of documents?

After weeks of trying to find a real Microsoft developer at work to do the task, I finally gave in and tried my hand using VBScript and the Windows Scripting Host. In four hours I had the basics of a script going.

My simple VBScript file does the following:

• Prompt the user for a PowerPoint file and target directory
• Launch PowerPoint and Word in the background
• Loop through every slide, inspecting any OLE objects found within to see if it was created by ChemDraw or IsisBase Sketch, two common chemistry editor tools
• If any such molecules are found, the following happens:
  • All of the “shapes” on the slide are copied to the clipboard
  • A new Word document is created with “Slide (1..n)” appended to the file name
  • The clipboard contents are pasted into the document

Sure, there are plenty of warts to work out, but I was happy to be able to run my script and watch as it magically consumed huge slide decks and spit out Word documents containing the exact data I was looking for. Windows Scripting Host is a powerful tool, as good as, if not better than AppleScript.

[Update: Added epilogue singing the joys of Windows Scripting Host]