Scanned photographs are an entirely different brew. They typically represent events from the distant past, and scanner software EXIF data is hit or miss.

I looked for commercial software to handle a few special cases of EXIF data troubles, but came up empty handed. So I wrote a few useful AppleScript droplets that do these tasks quite nicely, and I will share them here.

Warning!

These scripts use jhead to manipulate and rewrite your JPEG files!

Don’t be a fool. Experiment first with a safe set of throwaway JPEGs. And never use these tools on original files; always keep a backup.

Prerequisite

All of these scripts depend on a fine piece of free software called jhead written by Matthias Wandel.

Installation is not difficult, but it does involve the command line.

• Go to the jhead site
• Scroll down to the Releases section and look for Pre-built OS-X Intel executable
• Right click on the jhead link on that row and choose Save Linked File to “Downloads”

At this point, I found that jhead was saved as jhead.txt. Oh well. We needed to do some command-line magic anyway.

• Open a terminal window and enter the following:

These lines do the following:

• Lines 1 and 2 navigate to the Downloads directory and remove the “.txt” from the name
• Line 3 makes the file executable by everyone on your Mac
• Line 4 places the file in a public area where everyone on your Mac can see it (you will be prompted for your password)

Stripping All EXIF Data

Sometimes I receive files that have corrupt EXIF data. I had a large quantity of scanned files in my collection that claimed to be scanned some time in 2038, while others insisted that they had been around since 1901. Neither situation is good, and I found that standard EXIF editing tools may fail to change these corrupt EXIF sections.

The answer is to blow away the EXIF data.

Download: Strip EXIF

This zip file contains a compiled AppleScript application. You can unzip it and place the application on your desktop. Safari will probably unzip it for you when you click the link.

To be safe, open AppleScript Editor and use it to open the Strip EXIF app to see its magic.

Now you can drop any number of JPEG files onto the Strip EXIF app and it will kindly eviscerate each JPEG, removing all traces of EXIF data.

Adding Basic EXIF Data to a vanilla JPEG

Some tools create JPEG files without EXIF date and time information within. This is typically the hallmark of photo manipulation software and dodgy scanner software. And if you happened to use the Strip EXIF app to rip out a bad EXIF block, then you will want to replace it with a proper data block so that you can still use camera date and timestamps.

Download: Add Basic EXIF

Again, unzip the file, place the app on your desktop, and then drop any number of JPEG files onto Add Basic EXIF.

The app will set the EXIF date to the file creation timestamp.

Spreading EXIF Timestamps

This is the real reason why I wrote these scripts. I couldn’t find a satisfactory tool on the market that would allow me to automatically spread out the shooting times for a series of images.

Why would anyone want to do this? Because some processes give you fifty JPEG files all with the exact same creation time and exact same shooting time. I like to use file renaming tools to incorporate the shooting time in the filename, so that files sort by chronological order. This doesn’t work if all of the timestamps are the same.

So I wrote a little app that adjusts the first photo by one minute, the second by two minutes, and so on. If there are fifty photos, then the last one will have its shooting time adjusted by fifty minutes.

The result is a series of photos/scans that have different timestamps.

Download: Spread EXIF Timestamps

Again, please look at the short program before you run it.

Closing Thoughts

I hope that my favorite tool implements these tricks soon (A Better Finder Attributes, I’m looking at you!), but until then, I will be dropping my files onto these three little droplets.

The jhead tool is so versatile that I will probably end up with a whole slew of similar droplets that will do all kinds of spiffy stuff. Nevertheless, I would rather the commercial products already provided these features. Not everyone likes dipping into AppleScript and the command line!

Anyway, some weeks ago I was thinking how cool it would be to have a totally fanless and silent little black box that would serve up something useful in my house, such as the wiki I use to keep track of my geeky pursuits.

The other day I managed to obtain a spare Soekris net4801 box, a device best known for embedded applications such as firewalls. Between last night and this morning, I spent about three hours working out the hitches and getting MoinMoin and Ubuntu up and running on this neat little box.

The Device

So this is what I am talking about:

Cute, isn’t it?

It’s a Soekris net4801, a single board computer, a bit smaller than a hardcover book, with a 233MHz chip and 128MB of RAM soldered on the board. It doesn’t even have a hard drive—it runs everything off of a CF card.

The Operating System

After searching around for various combinations, I chose Ubuntu. Many people use FreeBSD on these boxes, because it is such a lightweight Linux distro, but I immediately ran into issues during the FreeBSD install process, and it was clear that I was going to have to learn many details of this different distro if I wanted to make my own custom install.

The approach I followed involved creating an image using an existing Ubuntu machine—a Hardy instance (8.04LTS) that I have running in a VM on my Mac.

The Process

I followed two different pages for getting Ubuntu up and running:

• Installing Ubuntu 7.04 Server via debootstrap
• Ubuntu 9.04 (jaunty) on a soekris net-4801

As it turned out, I had to keep bouncing between the two pages, using a little bit from one and then grabbing something from the other. For example, one page mentioned using parted for partitioning the CF card, while the other used cfdisk. I found cfdisk a little less intimidating, so that’s the one I used.

Both of these pages talk about using a tool called Debootstrap to install a new Debian base system in a subdirectory of another.

By mounting the CF card on a running Ubuntu instance, and then mounting the ISO Ubuntu disk image, you use debootstrap to generate a base Ubuntu install on the CF card.

It ain’t simple, but it ain’t rocket science. There are several tedious details, but for the most part it is a matter of following the directions.

The overall process is as such:

• Format and partition the CF card with Linux filesystems
• Use debootstrap to create a base Ubuntu install on the card
• Use chroot to set the card as the root filesystem and make several tweaks, such as setting up mount points, configuring networking, and adding users
• Get the latest kernel and set up grub

Then you plug it into the Soekris and boot

Other Hardware Issues

• You need a CF card reader, a device which is becoming more rare as each day passes.
• You also need a serial cable and a machine with a serial port—that’s how you get to the console on the Soekris.

Ubuntu Configuration

Once I got Ubuntu up and running and was connected via the console, I did the following:

• Installed ssh
• Installed apache2
• Installed and configured MoinMoin

Once ssh was up and running, I could abandon the serial console and do my work from a terminal window on my Mac.

Final Outcome

I was saddened by the results. Each page takes about seven seconds to render. This is probably a combination of these factors:

• My box uses a CF card, and not a hard drive. It is quite possible that the claimed 15MB/s written on the card is a lie, and I haven’t even checked to see how far that is from hard drive access times.
• MoinMoin is filesystem intensive, and they recommend running on a system with lots of memory, to keep the files in-memory.
• This device has only 128MB of memory, and it isn’t expandable.
• The processor isn’t that fast: 233MHz.
• I have done nothing to tune the Ubuntu install to make it faster.

I looked at top while the pages were being generated, and it was clear that the CGI script for MoinMoin was really taking seven seconds each time.

I looked up MoinMoin tuning and saw that I should be using WSGI instead of CGI, but there wasn’t a ready made package for Hardy, and I didn’t want to spend the time building it, chasing after a possible second or two of performance.

Maybe some day I’ll buy the hard drive adapter for the Soekris and see if I can bump up the speed that way. It’s better to run a system like this on a hard drive anyway.

Another thing I have in mind is to try this on my PC Engines ALIX box that runs my home firewall—that’s a much faster board with more memory.

So, if you landed here because you were trying something similar, let me know. Share your thoughts and successes or failures.

Update

So I had some spare time today and I decided to take another whack at loading some of the other variants of Unix/Linux onto the device, hopefully finding something lightweight with a smaller memory footprint.

The mysteries of PXE were slowly made clear to me—if you read a the Wikipedia page, it looks fairly complex with lots of configuration on the various bits. The reality is that PXE involves a few simple things:

• Local DHCP server tweaks to point the way to the PXE server
• A tftp server—meaning Trivial File Transfer Protocol
• The actual PXE boot payload file (typically a bootstrap file along with a config file)
• Some kind of file server, such as HTTP or NFS, to host the full install directory tree from the ISO of choice (FreeBSD, Ubuntu, etc)

Any decent geek who messes around with Linux boxes probably already has a few of these pieces running somewhere.

The reason it seems so darned complicated is that most online solutions do all of this in one shot on one machine, building a DHCP server as well as an NFS server, talking about this mysterious tftp along with inetd (“Trivial FTP” sounds so much more manageable, doesn’t it?).

Upon realizing that most of the scary stuff was all about getting tftp running, I followed this fine set of instructions on Installing and setting TFTPD in Ubuntu and had my Ubuntu tftp server running in about five minutes.

I was also able to relatively quickly add two single entries in my m0n0wall firewall configuration (my DHCP server) so that it would redirect local PXE requests correctly. This involved two “hidden options” from this m0n0wall page: What about hidden config.xml options?

And the web server? I already had an Apache server installed on my Ubuntu box, so that was no problem whatsoever.

Once you separate it out into its pieces, it is a pretty quick setup.

The hardest part is setting up the PXE payload. I was able to get Soekris to boot a couple of different boot kernels, but at this point I realized that any more progress would require custom boot loaders and such, compiled on a FreeBSD machine, and this was more than I wanted to get into.

So I reimaged the borrowed Soekris as a m0n0wall firewall, as it had been before, and restored the original configuration file. In a few days it will be back in its home, doing a proper job of being a firewall.

I’m serious. You don’t need to know anything at all about Linux to use one of these. Just download the image, install, and you suddenly have a full featured NAS file server, or you might have a database or a source code repository.

Last year I wrote an article on how to set up a NAS device using Ubuntu Linux. I have been a fan of Ubuntu since the start because it is a very easy distribution to install and configure. The down-side of using Linux has always been the fairly steep learning curve. Before you can get around to using the server, you need to get down in the weeds with configuration files and other stuff.

TurnKey Linux changes all of that.

Painless Installation

A few weeks back, I was setting up an aging PC as a standalone wiki server for a small office—this machine was going to provide a place for the office staff to document their procedures, how-tos, and other things.

I was about to set up an Ubuntu server, as I have done before many times, and install MoinMoin, like I did some months back. I remembered that it was a bit of a pain to get everything tweaked just right, so I did a quick check to see what kind of standalone wiki options were available online.

This is how I found TurnKey Linux. This project is all about single-purpose preconfigured Ubuntu server images.

One of those preconfigured images happens to be a MediaWiki appliance—the wiki engine behind Wikipedia—and I was in business.

The installation took about fifteen minutes, with very little user interaction. I answered a few basic questions and the installer took over from there. As soon as the install was done, the machine rebooted and displayed a message on the monitor with the IP addresses where you can browse to from any other machine.

Full Featured

The work that has gone in to these appliances is amazing. In fifteen minutes I had installed a complex configuration that has the Apache, PHP, MySQL, MediaWiki core, as well as maintenance utilities such as a neat tool that provides a Flash-based pure-AJAX-based SSH command line in a remote browser (i.e. your browser becomes a terminal). Even someone with Linux experience would have to spend quite a bit of time fiddling around with different packages and configuration options in other to provide the same functionality that TurnKey gives you out of the box.

As with most open source projects, the documentation is about 80% complete, with deep detail in some areas, but leaving others fairly sparsely documented. But don’t let this deter you: in most cases users know how to use the product they are installing (e.g. MediaWiki) but don’t want the hassle of configuring it on Linux. That’s where TurnKey shines.

Some Examples

In minutes, you can set up a NAS device. If you want to try advanced content management in your office, try Joomla or Drupal.

If you are working on a small project team and want to protect your source code, try Redmine or Trac and do your bug tracking using Bugzilla.

And while you are at it, you can document your organization’s working practices using a wiki such as MoinMoin or MediaWiki.

Don’t forget to back it up!

As with any computer, you should include your new TurnKey appliance in your backup strategy. The nice thing is that you don’t really need to care at all about backing up Linux or the other software; just back up the data. I don’t need to back up my entire MediaWiki machine; I just need to back up the database and image files. If anything goes wrong, you can rebuild the TurnKey appliance from scratch in minutes and then restore your data.

To save yourself some pain, keep notes on any small tweaks you made to the configuration.

One Machine, One Purpose

These disk images share common Ubuntu underpinnings, but they are referred to as Appliances because they turn your PC into a purpose-built appliance.

This means that if you want a content management system and you also want a ticket management system, you will need two old computers—not a rare commodity these days.

Take a look at what they have to offer and give TurnKey a shot—specialized software used in corporate environments is now within reach of small offices at the right price.

The problem is this: with Outlook we will be limited to 1000 messages in our in-box; sadly, many of us have tens of thousands of emails in our old GroupWise mail. Even after a fairly rigorous slash and burn mission, hacking out all of the low hanging fruit, there will be many thousands remaining and I don’t want to lose that information. It might be useful to search and find how I set up a Zebra bar code printer in 2003, no?

A bundle of different freeware glue tools came to my rescue. Read on to hear about the toolset that has made it so I can keep those messages for years to come.

Possible Solutions

Right out of the gate, I began looking for ways to migrate messages from one mail client to the other. Some apps have this built right in, and if not, there are scripts and utilities out there to do this; but I was hampered by a few key facts:

• I have no control over the email clients and their configuration. Even if there is a menu option for exporting GroupWise messages from version 7.2, I’m stuck at 6.4 and cannot use that option.
• GroupWise is a minor player in the email world. I’m not sure if Outlook would import from GroupWise, but I doubt it.
• They are replacing the client in one shot. There will be no interim period where both GroupWise and Outlook will be available.
• There is no getting around the hard limit of 1000 messages.
• I don’t want to spend money on this.

With these constraints in mind, I immediately thought about PDF documents. I then considered the following questions:

• How do I convert my email to PDF?
• How can I do this automatically with thousands of emails?
• Once I’m done, how do I search these documents?

Here’s what I did:

Conversion to PDF

The first part was easy. I downloaded one of the many free print-to-PDF products available.

I chose PDFCreator, because I am familiar with its use and I know that it does not munge the fonts.

Like many other PDF generation utilities, PDFCreator functions by providing a virtual printer to which any application can print. For example, to make a PDF of a web page, you use the Firefox Print menu and select PDFCreator from the drop-down list of available printers.

You are provided with a list of metadata fields that you can fill in, and these fields are used in the PDF generation.

Here’s what the PDFCreator screen looks like:

A word of caution: PDF Creator is free, but you must be careful to deselect their spammy toolbar options in two different places during the installation process. I don’t like software that comes with preselected toolbars to install (even nice ones like Google’s) because I’m certain that 95% of the folks who actually install the toolbar would never have chosen to do so if it were unchecked by default.

Running Everything Automatically

This was the interesting bit. I work with Windows machines at work, so there was no AppleScript option available. So I did the next best thing: I used AutoIT.

I will warn you that AutoIT is pretty much the Windows analog of AppleScript, without the cutesy pseudo English syntax. In other words, you will need to roll up your sleeves and get your hands a little dirty in order to put together a decent AutoIT script.

The payoff comes when you finish your work and compile it into a tight executable that you can share with your friends, allowing them to automate some complex series of button clicks and copy/paste operations.

I walked through the manual process of exporting an email to PDF and listed each action:

• Get the date, sender, and subject
• Create a filename based on date + sender + subject
• Launch the Print dialog
• Select PDFCreator
• Fill in the Document Title, Creation Date, and Subject in the PDFCreator dialog
• Fill in the full file path in the Save dialog

In addition, I wanted to make the script a little better by adding the following:

• Check that user has PDFCreator installed
• Verify that GroupWise is running and that the user has selected one or more messages
• Prompt the user for a target directory before processing the messages
• Sanitize the filenames by replacing illegal characters with underscores and truncating to meet maximum filename and path length in Windows
• Skip over files that have already been generated, quickly, so that one doesn’t need to worry about accidentally selecting messages that were already printed

There were other adjustments needed, but the process was the same: run the script, hit a problem, tweak the script a little to address the problem, and repeat.

Here’s a little bit of the AutoIT script:

You can see that it is a bit more intense than AppleScript, but remember that the full script wasn’t written in one go. I had a little short ten-line script that I kept tweaking as small problems cropped up until I had adjusted things to my liking.

Note that this is a GUI macro language. The machine starts clicking and typing away right in front of you and you probably shouldn’t interfere until your script finishes.

As of this afternoon, I have generated around 4,000 PDF documents for my email messages.

Searching All of Those Documents

This was the easiest part. These days there is an excellent tool available for searching documents on your desktop: Google Desktop. This product indexes every useful file on your desktop and provides a full Google search with a quick double-tap of the <control> key.

So you can enter a search like “Zebra bar code”

And the results look exactly like a Google web search, but it’s showing your desktop files. And you can see inline previews too.

Macintosh users can install Google Desktop as well, but all of these files should already be indexed and searchable by Spotlight.

Closing Thoughts

Whenever I reach for tools like this I feel a twinge of guilt—it’s outright hackery, isn’t it?

But there is a place for quick and dirty jobs in every workplace. I needed to get my files from one place to another, one time only. It just didn’t make sense to spend money or time on a more elegant solution.

Play around with each of these tools a little. Especially AutoIT—it’s a handy Swiss Army Knife to have at your disposal.

As it turns out, my original script does not work in Snow Leopard, and I promised that I would one day clean up and publish my new and improved version.

Chris posted a comment today as a gentle reminder, so here is the new and improved version without further delay…

The Details

Unfortunately, Snow Leopard came around and caused some indigestion. For starters, the ScanSnap Manager didn’t work correctly and Abbyy Finereader would not process anything made by the ScanSnap. A couple of months later they got everything straightened out and delivered new versions of each product.

The new version of the Abbyy Finereader product does not play well with my original script.

Since I cannot do without this important functionality, I rolled up my sleeves and rewrote most of the script. The new version works in Snow Leopard quite nicely with one small annoyance: you really don’t want to try to use the machine for anything other than scanning or OCR while it is going because the new Finereader version keeps bouncing the darned icon all the time it is running and that is quite annoying to watch.

Fortunately, I really don’t need to use my machine for anything else while it is chewing on the docs; I just wanted to be able to continue scanning at the same time!

Note: Before going forward, note that you will need to upgrade the ScanSnap Manager and Abbyy Finereader to the Snow Leopard versions first! Get the files here.

Here is a link to the new script…

And here’s the code itself:

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action Scripts
  You may have to create the Folder Action Scripts folder.
• Now open a Finder window and navigate to the parent folder of your scanned documents folder.
• Right click (control-click) the scanned documents folder and choose Folder Actions Setup…
• At this point if folder actions are not enabled, you will likely have to enable them and add the script manually.
  • Check Enable Folder Actions
  • Use the “+” buttons on the left and right sides to add the scan folder and then this script.
• Otherwise, a list of scripts will come up. Choose this script from the Choose a Script to Attach dialog.
• Close all windows.

That’s it! The script will be invoked automatically every time a new file appears in your scanned documents folder.

Please let me know if you have any ideas that can improve this script. I’m not an AppleScript guru, so someone might just know how to keep that annoying Finereader icon from jumping.

I was reading a post on a board I frequent where a person was describing exactly this kind of activity—removing sensitive information from PDF documents. Several suggestions were made, but one individual suggested opening the file in Acrobat Pro and replacing the sensitive text with good old Lorem Ipsum.

It was at that moment that I recalled a peculiar feature of the PDF file format: it is designed to support nondestructive updates, allowing people to make vast changes to a PDF document while still retaining the original document, fully intact. I did a few experiments and was surprised with the results.

A Brief Note on the PDF File Format

For the geeky types among us, one place to begin is this article:

Portable Document Format: An Introduction for Programmers

The key points to get out of the article is this: A PDF document is comprised of several distinct sections, a Header, a Body, an “xref” Table, and a Trailer. At the very end of the file you will find the character sequence %%EOF

The PDF standard was designed to allow multiple updates to a document, while retaining the original version. This is accomplished by appending anything new to the end of the document, after the original EOF tag. The document will now have two EOF tags: one indicating where the original document ended, and a new EOF tag indicating where the new changes end.

If we wish to revert PDF changes, it should be a simple matter of opening the PDF file in a binary editor, searching for the first EOF tag, and deleting everything following.

A Simple Experiment

Let’s start with a proper secret document containing missile plans…

Suppose we want to obscure some special information in paragraph 37. We can open the file in Acrobat Professional and use its text editing features to swap in the venerable Lorem Ipsum text.

Here’s what it looks like after the switch:

You can see here that the first seven lines of text starting on paragraph 37 have been replaced with appropriate unreadable text.

Now, open the new PDF file in a binary editor (since PDF files contain a mix of text and binary, the editor must be a binary editor).

Note the %%EOF character sequence embedded in the text. This is the first EOF tag, indicating where the original file ended. All we need to do is place the cursor to the right of the EOF and delete everything to the end of the file.

Once we have done so, it’s like magic:

The edits that replaced lines of paragraph 37 with gibberish have neatly been undone!

More Details

From the PDF Intro document linked earlier:

“The trailer, it turns out, plays an important role in the way PDF implements incremental updating. The key concept to understand here is that a PDF file is never overwritten, only added to. That goes for all portions of the PDF file – even the trailer itself, and the end-of-file marker. In other words, a multiply-updated PDF document may contain multiple trailers – and multiple end-of-file markers! (There may be numerous occurrences of %%EOF.) Each time the file is edited, an addendum is written to the tail of the file, consisting of the content objects that have changed, a new xref section, and a new trailer containing all the information that was in the previous trailer, as well as a /Prev key specifying the byte offset (from the beginning of the file) of the previous xref section. The cross-reference info will then be distributed across more than one xref section. To access all of the cross-references, the reader must walk the list of /Prev keys in all the trailers, in reverse order.

Space doesn’t permit a detailed exploration of updates here, but you can find several examples in Appendix A of the PDF 1.3 specification (available at http://partners.adobe.com/asn/developer).”

Summary

It is important to understand that the PDF standard allows for appended updates to files that leave the original document intact, regardless of how drastic the changes are. If you are intent on redacting text from PDF documents, do not depend on simply deleting the secrets using a PDF editor—you must use a proper redaction tool that addresses these issues correctly.

That said, I did some experimenting with a few utilities (Apple Preview, PDFpen, and Adobe Acrobat Pro) and found that some write the file from scratch each time, with no lingering cruft from former versions, while others respect the original intent of the PDF standard. This means that you can’t trust that older revisions are being retained in your file and you can’t trust that they aren’t.

Be conservative: use a redaction tool for secrecy and proper backups for versioning.

As it would turn out, there were several thousand other documents on a file server that shared the same funny behavior. By the time we were done struggling with this problem I had gained new respect for PDF corruption issues and their prevention.

The Problem

We were looking to load a few thousand of these scientific reports into a fancy-schmancy new database, with linguistics searching and other bells and whistles. Much to our chagrin, these documents just weren’t loading, and we couldn’t understand why. They were text documents, with some embedded images, but mostly straightforward text.

Here is an excerpt:

And you can tell that it is right and proper text because when I blow it up all the way, the fonts are nice and smooth—this isn’t just an image of text.

But if I copy and paste that particular paragraph into any handy editor (Notepad, in this case), this is what I see:

And as far as I know, at this point the actual text is beyond the reach of average folks like me. We tried, believe me we tried.

What went wrong?

A quick Google of the subject led us to understand that many PDF generation tools embed subsets of fonts, with nonstandard mappings from the text to the font.

This fellow explains it nicely:

“The PDF file does not contain all the information to extract the text. The problem is that a character in a PDF file may not contain information what “real” character it relates to. Some PDF generators do a pretty bad job when they embed fonts into PDF files. They use a proprietary encoding mechanism (e.g. 1 is A, 2 is B, 3 is C, …) in both the embedded font and when they place glyphs on the page. Without a table that implements the reverse (e.g. character code 1 is ‘A’) you cannot extract text from such a file.

There is nothing you can do (besides to complain to whoever created the PDF file, and the author of the software that created this file).”
— from khkremer on experts-exchange.com

As it would turn out, many of the reports had been generated by printing to Adobe Distiller from Microsoft Word. It would seem that the default settings used for Distiller included the “totally hose my document content” switch.

The Solution

We fretted over this quite a bit. These are important scientific reports, and there is no way to easily ungarble them. We finally ended up contacting the Abbyy Finereader folks and trying out their OCR toolkit for Linux: not only did this product make fast work of running optical character recognition on the sample document, but once we had a script running, we managed to blow through the 10,000 pages the trial license gave us, in a day or two.

Imperfect, at best

I am happy that we were able to salvage the bulk of the electronic knowledge found within those thousands of files, but our work barely scratched the surface.

For example, most of these documents have rich bookmarking of sections and keywording, such as this (content tastefully blurred on purpose).

In addition, scientific documents typically have loads of tables full of numbers. Though it is possible to mine this data with a good OCR tool (the FineReader API provides tools for just this purpose), the tables are far more difficult to extract correctly once the original text information is lost.

Final thoughts

I wrote a few weeks about document formats, mentioning the PDF/A document standard. This is worth investigating, regardless of what your document needs are.

If our thousands of files had been originally generated as PDF/A, it is certain that we would have been able to copy/paste from them without problem: PDF/A prohibits such font shenanigans as were perpetrated on our garbled reports.

In the end, our OCR sledgehammer approach worked like a charm, and is probably sufficient for our needs. Text mining is a pretty slushy business, so no-one will complain if there are a few typos on each page—if they find the doc in a search, they can print it and read it the old fashioned way.

Over the past couple of years I have considered several different approaches for keeping a grip on which computers had which service patch, which router is getting flaky, and which cable connects the library to the classroom at the end of the hall.

I have tried Excel spreadsheets, an Access database, even a spiral-bound notebook—none of them made the job any easier. A few weeks ago I thought about trying a Wiki and this has turned out to be a perfect fit!

If you are looking to keep a loose scrapbook of notes with lots of arbitrary categories and relationships between them, a wiki might do the trick. In this article I’ll cover two simple freeware wikis you can carry around on a thumb drive.

What’s in a Wiki?

All of us have used Wikipedia at one time or another, and though it may be regarded with disdain by high school teachers, when you consider how it works, Wikipedia is an amazing achievement. But what is the nature of a wiki?

One of the key features is that any page can be easily edited at any time (of course this can be limited by permissions). Another attribute is the ability to breathe life into a new page just by calling its name.

Between these two features, you get the essence of wiki-ness.

For example, if I have a page that discusses North American bears, I can type in a list of bears in a special format, often in jammed-together Wiki Words, like this:

• GrizzleyBear
• BlackBear
• BrownBear

As soon as I save the page, those bear names become hyperlinks. Even though I haven’t written any pages about the individual bears, whenever it finally suits me, I can click on BlackBear and accept the invitation to “Create a new page called BlackBear”

Better still, a friend who knows about black bears might click on BlackBear and write a beautiful page about the animals.

That’s what wikis are all about.

Back to the School Computers

In a matter of minutes I was able to make a page that described the building and listed the various rooms in the building. I was able to then click on each room and “auto-vivify” a page for the room.

From that point, it was easy to create custom pages for each computer in the building, with each page listing the machine’s stats. I also created pages for each network switch or router.

In a matter of two or three evenings I had the skeleton of a solid knowledge base populated—it’s a pretty fancy looking web site with dozens of pages that took little effort to put together.

Last night I noticed that one of the machines wasn’t connecting to the Internet, though it connects fine to internal servers. I popped open its page on the wiki and added a simple note at the bottom of the page:

2009-10-11 - This machine isn't able to connect to the Internet. Not sure why. It connects fine to internal servers.

A few weeks ago I replaced a fan in a network switch. An easy annotation on the wiki page for that device.

Personal Wikis

There are many uses for personal wikis, mostly centered around personal knowledge management and personal information management. People use wikis as a replacement for time and task management tools, as a place for gathering thoughts, as a sort of amorphous database, and many other things.

There are many different personal wikis available—here’s a short list of free ones. One nice simple wiki to try is TiddlyWiki. If you are looking for something with a bit more substance, you can try a portable version of MediaWiki—the engine behind Wikipedia—that runs off your thumb drive.

TiddlyWiki

This afternoon I downloaded the flyweight portable wiki called TiddlyWiki. This is an amazingly tight little application—it comes in the form of a single fat web page that you copy to your thumb drive. As you make edits to your TiddlyWiki, the single html page is saved with your changes. Since it’s a single fancy file, backups are dead easy.

Here’s what it looks like when you first launch the “empty.html” file:

After a half hour of twiddling around, I had thrown together this basic set of “Tiddlers”

In this screen shot you can see that there are now links that bring up custom “Tiddlers” for each computer and for each room. I have opened one of the little pages for Computer21.

They describe these pages as being comparable to note cards. All in all, it is tight and easy to use.

Want to give it a try? Download it from the TiddlyWiki site. You really need to play with it to get a feel for what it can do!

MediaWiki

If you are looking for something with a little more meat on it, you can run the Wikipedia engine on your USB drive.

The easiest way to set this up is to let MoWeS do everything for you. MoWeS stands for Modular Webserver System. It’s a free product that you can configure as a self-contained Apache web server with a variety of cool apps like MediaWiki, running off a thumb drive.

Here’s how to set up MediaWiki in five minutes:

• Go to the MoWeS Mixer
• The first time around choose “I do not have a MoWeS Portable II Package and want to obtain a new package” when prompted and click Go.
• On the software lists, check Apache2, MySQL5, PHP5, and MediaWiki
• Click Download Now
• At this point they ask you some kind of question in German, to filter spambots, but it seems to be a simple math problem. Fill in the answer and click Submit Query
  (“Zum Schutz vor Downloadrobotern geben Sie bitte das Ergebnis dieser Aufgabe ein: 5 + 8 = ?“)
• Unzip the downloaded zip file, mowes_portable.zip, and copy the files to your USB drive
• Open your thumb drive and double-click mowes.exe
• Select your language and accept the license
• Click install, and confirm when prompted

The installation process may take several minutes, but rest assured that it isn’t installing anything on your computer.

Note: I received two or three firewall warnings for the Apache web server and the MySQL database. I had to click the “Unblock” button for all of them before my new MediaWiki-on-a-stick would work correctly.

After all of the dust settled, I have this little window on my screen:

In order to shut down and close out, just click the End button.

Once your MediaWiki USB key is running, you can go to this web page:

http://127.0.0.1/mediawiki/index.php/Main_Page

It looks just like Wikipedia, doesn’t it?

What a truly amazing thing: you can carry around your own Wikipedia server on a USB key and plug it in any random machine and start it up.

Different Wiki Features

As you try out different wiki software, you will notice that there are plenty of differences in the features they support:

• Each wiki has a different kind of editor. Some are visual; others are simple text editors.
• The markup syntax you use for pages is different from wiki to wiki.
• Most wikis support features such as “category pages” that find all pages tagged with a category.
• Some support adding images and other content; others don’t. I imagine that TiddlyWiki probably has some means of embedding images, but I couldn’t find it.
• A quick glance at the MediaWiki screenshot above shows extended features such as the Discussion tab and the History tab.
• Some use the filesystem for their pages; others use a database.

Since I wanted a central wiki for the whole school, I chose a different product from the portable wikis I discussed here—I decided to run MoinMoin on a Ubuntu installation on an aging Gateway desktop machine. Nevertheless, the basic idea is still the same.

Once that arrangement becomes a little more stable I’ll write up a howto document, like the Linux NAS one from a few months back.

Other Sources

There are loads of different personal wiki options out there and many people have written how-to documents and tutorials. Here’s a few:

• Run Your Personal Wikipedia from a USB Stick (Lifehacker.com)
• Geek to Live: Set up your personal Wikipedia (Lifehacker.com)
• Wiki On A Stick (PmWiki.org)
• Getting Started with Wiki on a Stick (About.com)
• TiddlyWiki for the rest of us (giffmex)

Now prudence would lead one to take careful steps and wait until all of the wrinkles are ironed out before starting. I was almost not prudent enough this week.

Mac OS X Snow Leopard

So folks have been talking about the new Snow Leopard operating system for Mac. Over the past year, Apple has been positioning this version as more of a “under the hood” upgrade that tightens things up rather than a glitzy overhaul of the user interface. No matter what they said it was, I figured that it was newer, and therefore better, than the current OS—Leopard–and I had to have it.

I ordered my copy last week on Amazon and sat down with a smile as I awaited its arrival. And then I thought about doing a few quick Googles to see how other people have been making out with Snow Leopard. I immediately happened upon a few upgrade guides like this one, providing sage advice about the upgrade process. They recommended the “slash and burn” method, starting from a clean hard drive, and I felt that was a good idea. Nothing better than a wipe and fresh install to make your machine zip along twice as fast. And therein lies a tale.

The first sign of trouble

As I was reading up on the Snow Leopard upgrade process, I happened upon lists of “unsupported software” and casually glanced at the lists, expecting esoteric tools only used by three über geeks in the audio recording industry or perhaps some exotic ray-tracing software. Much to my surprise, I saw two of my favorite applications, in a very very short list of troublesome apps: Parallels and EyeTV.

I immediately checked the versions and breathed a sigh of relief when I saw that my EyeTV version was safe. But, Parallels was another story… They have no plans for patching Parallels 3 to work with Snow Leopard, and why should they, when they can sell us Parallels 4!

So, I ordered my fresh copy of Parallels 4, from Amazon with a twenty dollar rebate. When it arrived, I spent an evening upgrading Parallels, and I thought I was all set for Snow Leopard.

Preventative Measures

Following the advice of the upgrade websites, and prior experience, I used Carbon Copy Cloner to make a full backup of my hard drive on a spare external drive. On a hunch, I turned on the drive that I use for Time Machine and had it do one final “Time Machine” sweep through the system before bidding adiu to Leopard.

I knew that I had all of my installation media for stuff like iLife and Photoshop Elements, and I had all of my license keys in electronic form. It would be a simple matter of mounting the backup drive, copying over my loads of documents, and peering into them to find keys.

The first attempt

I boldly inserted the Snow Leopard disk and booted from the DVD drive, selecting the “Slash and Burn” method of installation. I reformatted the hard drive and went off for dinner while Snow Leopard installed.

Trouble

When I got home that evening, I started the lengthy process of installing stuff. I suddenly realized that it was not as easy as I had hoped: it’s one thing to reinstall something like Microsoft Office, but there seemed to be more loose ends than I had considered:

• How would I migrate my Mail settings from the old image to the new?
• What was the best way to migrate the Address Book contents?
• iTunes is great, but it has tendrils in everything. Can I simply copy my old library to the new without messing up my iPhone, Address Book, or other linked stuff?
• How about those nice password tools such as 1Password and SplashID that keep your passwords safe and sound? I had no clue how to get their contents from the backup. I wasn’t sure if it was even possible to do so—perhaps I was supposed to have exported the data beforehand.

It was becoming clearer to me that I had not done my homework at all.

More trouble

My initial shock at the depth of the upgrade process led me to start making a list of applications and looking at what I needed for each one. I soon found out that Snow Leopard support is somewhat spotty in many applications. In particular, the FineReader for ScanSnap software that I depend on so much for my scanning work flow is not fully supported. Fujitsu says that they will have an update soon and to keep checking their web site.

My password tool, 1Password, is another problem child. It works only on 32-bit Safari, and Snow Leopard now runs Safari in 64-bit mode. Of course, a new version is coming, and I will probably have to pay for it, but it is still in beta.

There was quite a bit of chatter on the Web about whether Adobe Photoshop Elements would work on Snow Leopard, and the responses seem split fifty-fifty for now.

Three very important tools were in danger of running in limited mode or not running at all, so I had to throw in the towel.

Time Machine saves the day!

As I sat, humbled, before my vanilla install of Snow Leopard, I admitted defeat. I slipped the Snow Leopard DVD back in the drive and rebooted from the DVD. This time, I selected the “Restore from Time Machine” option and turned on my Time Machine drive.

Guess what? It worked perfectly! Unlike many software products, Time Machine does exactly what it promises.

Within a few hours, my machine was fully restored to the way it looked seconds before I made my first attempt at Snow Leopard.

A Final Word

Learn from my mistakes, and my salvation by the full backup. As much as you can’t wait to upgrade, please do the following:

• Inventory all of your applications that you really need.
• Obtain the installation media (download or CD) for every single one.
• Obtain the keys for every single one.
• Investigate whether you need to export data from any of them, and make a checklist for these exports prior to upgrade.
• Check the “Unsupported Software” lists that are out there for any red flags.
• Check the web sites of your most important apps for their official word.
• And finally, do a complete backup!

It’s amazing how many applications and weird little utilities we forget we have. How could I have possibly remembered that I compiled a custom copy of the “rsync” executable for my backup workflow? I would have lost that and had to figure out how to rebuild it on Snow Leopard.

And I haven’t even talked about making sure your documents make it safely onto the new machine. That’s a whole ‘nother story.

In case I forgot to say it, please make a full backup.

[Update: I'm giving Snow Leopard a rest for a few months]

It has been said that Time Machine allows you to do a full restore from bare metal, and I’m living proof: I have done exactly that twice in the past week, with astounding success.

Encouraged by an episode of the Mac Geek Gab where they talked about their experiences upgrading their existing systems to Snow Leopard, I decided I would give the upgrade-in-place option a try. I expected some things to not work well and others to be quirky, but here’s what happened…

The actual install was painless, taking an hour or so to complete. I then began to kick the tires to see what was broken.

It was clear where those 64 bits went: apps like Safari were positively zippy, and I was pleasantly surprised with each new application I launched. All of my special settings seemed to make it through alive, including my password manager, though I did have to re-enter some of my registration keys. All of my mail and contacts made it through well. I was able to sync my iPhone without incident.

I found a few apps that weren’t working correctly and I looked for newer 10.6-compatible versions. I found newer versions of Yep and xGestures.

I did note that there is currently no ad blocker available for Safari that runs in 64-bit mode. This is disappointing because even though I understand that Apple wants us to see their ads, I can’t imagine that they really want us to suffer from the flickering jumping dreck that should have ended with the hated “punch the monkey” banners of years gone by. The fact of the matter is, if I want that 64-bit speed and snap, I guess I have to watch ads.

The Showstopper

I decided to scan a document to see just how difficult it would be to get my workflow going again. Michael F, below, wrote the truth about the situation: the scanner works fine in certain modes, but the OCR software doesn’t.

He pointed out that it was a problem of the FineReader software looking for a specific bit of metadata in the PDF identifying it as a ScanSnap PDF. Sadly, that metadata string changed.

The Finereader software is looking for “Mac OS X 10.5.8 Quartz PDFContext”, but under Snow Leopard, the string is set to “Mac OS X 10.6 Quartz PDFContext” instead.

There are ways to tweak PDF metadata, and one of them is by using pdftk.

I went to the pdftk site, all ready to download it and start OCRing my PDFs. I was greeted with less than optimal news: they have a version compiled for Panther, a version of OS X from several years ago.

I knew it wouldn’t work, but I gave it a try anyway: the app told me it needed Rosetta to run. I could have installed Rosetta at that point, but I figured I wanted a proper compiled version.

From there, I looked into compiling the app on OS X 10.6. I should have remembered my struggles with this several months ago on a Solaris Unix box when I found that pdftk depends on a monster called GCJ that required about forty other software packages to compile—it seemed a gargantuan task that I wasn’t ready to begin.

On a hunch, I inspected the content of a new pdf and an old pdf, the latter still acceptable to FineReader. Though much of the file was raw binary, the metadata was in text at the end. A short sed script was all it took to swap the nice text string for the offending 10.6 one.

In spite of my best efforts, FineReader still rejected my hand-tooled PDF file. It knew that it was a bogus file.

I have looked into Abbyy FineReader several times before, as well as Fujitsu’s ScanSnap support, and was unimpressed. For two vendors that produce products that are at the top of their class—FineReader is arguably the best OCR you can get for Mac, and ScanSnap is the best document scanner for the common man—they sure do have miserable customer support.

It is as if neither company cares a whit about the Macintosh platform or their customers. While most other vendors are busily patching their products and giving hourly updates on their Snow Leopard compatibility progress, Abbyy and Fujitsu just don’t seem to care that their best-of-breed combo suddenly doesn’t work on Mac.

Once they get this sorted out (hopefully in the next few months) I’ll give Snow Leopard another try. In the meantime, I’m sticking with good old Leopard.

I had promised that once I cleaned up the embarrassing bits of code I would post a perfect polished version here, but such promises are rarely fulfilled. A reader posted a comment asking for that source code, so I will post it here in its current state. The truth is, I have been using this script for months and, though it has some quirks, it works fine.

So this post is about Macintosh, AppleScript, and the ScanSnap-to-FineReader workflow. If these don’t interest you, better move on.

Update: The script on this page works only with Leopard (10.5). Get the Snow Leopard version here

The Original Problem

The Fujitsu ScanSnap S510m, my workhorse scanner, was designed to scan documents quickly and generate PDF files—this it does flawlessly. In order to provide OCR support, they have shipped a special version of FineReader, called FineReader for ScanSnap. The standard OCR configuration is to chain the output of the scanner to the FineReader program.

The problem is that this forces scanning and OCR to run in lockstep: you scan a document, you wait for OCR, and then you scan another document.

My desire was to write a simple AppleScript that would detach the “Scan a Document” process from the “OCR” process. By using this script, I can scan documents at whatever rate pleases me, and the OCR engine will chunk along at its own pace, consuming my scanned documents and producing OCR documents.

My Approach

I really looked hard at the OCR application, trying to find AppleScript hooks or special command line switches that might allow me to control it better. Sadly, it was not designed to be scriptable. The only thing I could do is call the FineReader application with a source file.

Given this limitation, I considered writing a script that would look at a particular folder, identifying new files as they appear and passing them on to FineReader.

Fortunately, AppleScript provides this kind of functionality with little effort in the form of Folder Actions. Perhaps the best way to see these in action (and try it out) is to see this post on Exploring the power of Folder Actions.

In order to achieve my goals, I did the following:

• Created a folder called “Pending Documents”
• Wrote the script to find the oldest-unprocessed-file and call FineReader with it
• Attached the script to the folder as a Folder Action

The Script

Let’s jump right in to the AppleScript. Download the script here.

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action ScriptsYou may have to create the Folder Action Scripts folder.
• Now open a Finder window, control-click and choose More / Configure Folder Actions…
• Check the Enable Folder Actions checkbox, if not checked
• Click the “+” in the bottom left
• Select a folder and click Open
• Choose the script Run OCR on New Folder Items and click Attach

Picky Details

As you can see in the source code, there were several issues to address:

• I had to make sure the script didn’t step on itself. If FineReader was running, I would wait until it was ready before processing.
• The script needed to determine which files had been processed already. This was handled fairly trivially by looking for a matching file with the processed by FineReader.pdf suffix. In other words, if I was looking at Scan001.pdf, I would see if there was a matching Scan001 processed by FineReader.pdf file.
• Part of checking for a source file’s “buddy” was stripping off the PDF suffix. This was done in a hackish way by using a one-line shell script, at lines 106-107.
• I thought it was important to verify that the source file was, indeed, a ScanSnap file—the FineReader will not process other PDF documents. This was done at lines 117-121 by looking at the Spotlight metadata for the Creator of the source file. That took some more shell scripting (133-154).
• The actual work was done by a single line, line 63.

The real work was fairly simple, while the bulk of the code was needed to polish pesky little details. Isn’t that the way code development often is?

If anyone has any improvements on my script, please let me know!