I’m serious. You don’t need to know anything at all about Linux to use one of these. Just download the image, install, and you suddenly have a full featured NAS file server, or you might have a database or a source code repository.

Last year I wrote an article on how to set up a NAS device using Ubuntu Linux. I have been a fan of Ubuntu since the start because it is a very easy distribution to install and configure. The down-side of using Linux has always been the fairly steep learning curve. Before you can get around to using the server, you need to get down in the weeds with configuration files and other stuff.

TurnKey Linux changes all of that.

Painless Installation

A few weeks back, I was setting up an aging PC as a standalone wiki server for a small office—this machine was going to provide a place for the office staff to document their procedures, how-tos, and other things.

I was about to set up an Ubuntu server, as I have done before many times, and install MoinMoin, like I did some months back. I remembered that it was a bit of a pain to get everything tweaked just right, so I did a quick check to see what kind of standalone wiki options were available online.

This is how I found TurnKey Linux. This project is all about single-purpose preconfigured Ubuntu server images.

One of those preconfigured images happens to be a MediaWiki appliance—the wiki engine behind Wikipedia—and I was in business.

The installation took about fifteen minutes, with very little user interaction. I answered a few basic questions and the installer took over from there. As soon as the install was done, the machine rebooted and displayed a message on the monitor with the IP addresses where you can browse to from any other machine.

Full Featured

The work that has gone in to these appliances is amazing. In fifteen minutes I had installed a complex configuration that has the Apache, PHP, MySQL, MediaWiki core, as well as maintenance utilities such as a neat tool that provides a Flash-based pure-AJAX-based SSH command line in a remote browser (i.e. your browser becomes a terminal). Even someone with Linux experience would have to spend quite a bit of time fiddling around with different packages and configuration options in other to provide the same functionality that TurnKey gives you out of the box.

As with most open source projects, the documentation is about 80% complete, with deep detail in some areas, but leaving others fairly sparsely documented. But don’t let this deter you: in most cases users know how to use the product they are installing (e.g. MediaWiki) but don’t want the hassle of configuring it on Linux. That’s where TurnKey shines.

Some Examples

In minutes, you can set up a NAS device. If you want to try advanced content management in your office, try Joomla or Drupal.

If you are working on a small project team and want to protect your source code, try Redmine or Trac and do your bug tracking using Bugzilla.

And while you are at it, you can document your organization’s working practices using a wiki such as MoinMoin or MediaWiki.

Don’t forget to back it up!

As with any computer, you should include your new TurnKey appliance in your backup strategy. The nice thing is that you don’t really need to care at all about backing up Linux or the other software; just back up the data. I don’t need to back up my entire MediaWiki machine; I just need to back up the database and image files. If anything goes wrong, you can rebuild the TurnKey appliance from scratch in minutes and then restore your data.

To save yourself some pain, keep notes on any small tweaks you made to the configuration.

One Machine, One Purpose

These disk images share common Ubuntu underpinnings, but they are referred to as Appliances because they turn your PC into a purpose-built appliance.

This means that if you want a content management system and you also want a ticket management system, you will need two old computers—not a rare commodity these days.

Take a look at what they have to offer and give TurnKey a shot—specialized software used in corporate environments is now within reach of small offices at the right price.

The problem is this: with Outlook we will be limited to 1000 messages in our in-box; sadly, many of us have tens of thousands of emails in our old GroupWise mail. Even after a fairly rigorous slash and burn mission, hacking out all of the low hanging fruit, there will be many thousands remaining and I don’t want to lose that information. It might be useful to search and find how I set up a Zebra bar code printer in 2003, no?

A bundle of different freeware glue tools came to my rescue. Read on to hear about the toolset that has made it so I can keep those messages for years to come.

Possible Solutions

Right out of the gate, I began looking for ways to migrate messages from one mail client to the other. Some apps have this built right in, and if not, there are scripts and utilities out there to do this; but I was hampered by a few key facts:

• I have no control over the email clients and their configuration. Even if there is a menu option for exporting GroupWise messages from version 7.2, I’m stuck at 6.4 and cannot use that option.
• GroupWise is a minor player in the email world. I’m not sure if Outlook would import from GroupWise, but I doubt it.
• They are replacing the client in one shot. There will be no interim period where both GroupWise and Outlook will be available.
• There is no getting around the hard limit of 1000 messages.
• I don’t want to spend money on this.

With these constraints in mind, I immediately thought about PDF documents. I then considered the following questions:

• How do I convert my email to PDF?
• How can I do this automatically with thousands of emails?
• Once I’m done, how do I search these documents?

Here’s what I did:

Conversion to PDF

The first part was easy. I downloaded one of the many free print-to-PDF products available.

I chose PDFCreator, because I am familiar with its use and I know that it does not munge the fonts.

Like many other PDF generation utilities, PDFCreator functions by providing a virtual printer to which any application can print. For example, to make a PDF of a web page, you use the Firefox Print menu and select PDFCreator from the drop-down list of available printers.

You are provided with a list of metadata fields that you can fill in, and these fields are used in the PDF generation.

Here’s what the PDFCreator screen looks like:

A word of caution: PDF Creator is free, but you must be careful to deselect their spammy toolbar options in two different places during the installation process. I don’t like software that comes with preselected toolbars to install (even nice ones like Google’s) because I’m certain that 95% of the folks who actually install the toolbar would never have chosen to do so if it were unchecked by default.

Running Everything Automatically

This was the interesting bit. I work with Windows machines at work, so there was no AppleScript option available. So I did the next best thing: I used AutoIT.

I will warn you that AutoIT is pretty much the Windows analog of AppleScript, without the cutesy pseudo English syntax. In other words, you will need to roll up your sleeves and get your hands a little dirty in order to put together a decent AutoIT script.

The payoff comes when you finish your work and compile it into a tight executable that you can share with your friends, allowing them to automate some complex series of button clicks and copy/paste operations.

I walked through the manual process of exporting an email to PDF and listed each action:

• Get the date, sender, and subject
• Create a filename based on date + sender + subject
• Launch the Print dialog
• Select PDFCreator
• Fill in the Document Title, Creation Date, and Subject in the PDFCreator dialog
• Fill in the full file path in the Save dialog

In addition, I wanted to make the script a little better by adding the following:

• Check that user has PDFCreator installed
• Verify that GroupWise is running and that the user has selected one or more messages
• Prompt the user for a target directory before processing the messages
• Sanitize the filenames by replacing illegal characters with underscores and truncating to meet maximum filename and path length in Windows
• Skip over files that have already been generated, quickly, so that one doesn’t need to worry about accidentally selecting messages that were already printed

There were other adjustments needed, but the process was the same: run the script, hit a problem, tweak the script a little to address the problem, and repeat.

Here’s a little bit of the AutoIT script:

You can see that it is a bit more intense than AppleScript, but remember that the full script wasn’t written in one go. I had a little short ten-line script that I kept tweaking as small problems cropped up until I had adjusted things to my liking.

Note that this is a GUI macro language. The machine starts clicking and typing away right in front of you and you probably shouldn’t interfere until your script finishes.

As of this afternoon, I have generated around 4,000 PDF documents for my email messages.

Searching All of Those Documents

This was the easiest part. These days there is an excellent tool available for searching documents on your desktop: Google Desktop. This product indexes every useful file on your desktop and provides a full Google search with a quick double-tap of the <control> key.

So you can enter a search like “Zebra bar code”

And the results look exactly like a Google web search, but it’s showing your desktop files. And you can see inline previews too.

Macintosh users can install Google Desktop as well, but all of these files should already be indexed and searchable by Spotlight.

Closing Thoughts

Whenever I reach for tools like this I feel a twinge of guilt—it’s outright hackery, isn’t it?

But there is a place for quick and dirty jobs in every workplace. I needed to get my files from one place to another, one time only. It just didn’t make sense to spend money or time on a more elegant solution.

Play around with each of these tools a little. Especially AutoIT—it’s a handy Swiss Army Knife to have at your disposal.

As it turns out, my original script does not work in Snow Leopard, and I promised that I would one day clean up and publish my new and improved version.

Chris posted a comment today as a gentle reminder, so here is the new and improved version without further delay…

The Details

Unfortunately, Snow Leopard came around and caused some indigestion. For starters, the ScanSnap Manager didn’t work correctly and Abbyy Finereader would not process anything made by the ScanSnap. A couple of months later they got everything straightened out and delivered new versions of each product.

The new version of the Abbyy Finereader product does not play well with my original script.

Since I cannot do without this important functionality, I rolled up my sleeves and rewrote most of the script. The new version works in Snow Leopard quite nicely with one small annoyance: you really don’t want to try to use the machine for anything other than scanning or OCR while it is going because the new Finereader version keeps bouncing the darned icon all the time it is running and that is quite annoying to watch.

Fortunately, I really don’t need to use my machine for anything else while it is chewing on the docs; I just wanted to be able to continue scanning at the same time!

Note: Before going forward, note that you will need to upgrade the ScanSnap Manager and Abbyy Finereader to the Snow Leopard versions first! Get the files here.

Here is a link to the new script…

And here’s the code itself:

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action Scripts
  You may have to create the Folder Action Scripts folder.
• Now open a Finder window and navigate to the parent folder of your scanned documents folder.
• Right click (control-click) the scanned documents folder and choose Folder Actions Setup…
• At this point if folder actions are not enabled, you will likely have to enable them and add the script manually.
  • Check Enable Folder Actions
  • Use the “+” buttons on the left and right sides to add the scan folder and then this script.
• Otherwise, a list of scripts will come up. Choose this script from the Choose a Script to Attach dialog.
• Close all windows.

That’s it! The script will be invoked automatically every time a new file appears in your scanned documents folder.

Please let me know if you have any ideas that can improve this script. I’m not an AppleScript guru, so someone might just know how to keep that annoying Finereader icon from jumping.

I was reading a post on a board I frequent where a person was describing exactly this kind of activity—removing sensitive information from PDF documents. Several suggestions were made, but one individual suggested opening the file in Acrobat Pro and replacing the sensitive text with good old Lorem Ipsum.

It was at that moment that I recalled a peculiar feature of the PDF file format: it is designed to support nondestructive updates, allowing people to make vast changes to a PDF document while still retaining the original document, fully intact. I did a few experiments and was surprised with the results.

A Brief Note on the PDF File Format

For the geeky types among us, one place to begin is this article:

Portable Document Format: An Introduction for Programmers

The key points to get out of the article is this: A PDF document is comprised of several distinct sections, a Header, a Body, an “xref” Table, and a Trailer. At the very end of the file you will find the character sequence %%EOF

The PDF standard was designed to allow multiple updates to a document, while retaining the original version. This is accomplished by appending anything new to the end of the document, after the original EOF tag. The document will now have two EOF tags: one indicating where the original document ended, and a new EOF tag indicating where the new changes end.

If we wish to revert PDF changes, it should be a simple matter of opening the PDF file in a binary editor, searching for the first EOF tag, and deleting everything following.

A Simple Experiment

Let’s start with a proper secret document containing missile plans…

Suppose we want to obscure some special information in paragraph 37. We can open the file in Acrobat Professional and use its text editing features to swap in the venerable Lorem Ipsum text.

Here’s what it looks like after the switch:

You can see here that the first seven lines of text starting on paragraph 37 have been replaced with appropriate unreadable text.

Now, open the new PDF file in a binary editor (since PDF files contain a mix of text and binary, the editor must be a binary editor).

Note the %%EOF character sequence embedded in the text. This is the first EOF tag, indicating where the original file ended. All we need to do is place the cursor to the right of the EOF and delete everything to the end of the file.

Once we have done so, it’s like magic:

The edits that replaced lines of paragraph 37 with gibberish have neatly been undone!

More Details

From the PDF Intro document linked earlier:

“The trailer, it turns out, plays an important role in the way PDF implements incremental updating. The key concept to understand here is that a PDF file is never overwritten, only added to. That goes for all portions of the PDF file – even the trailer itself, and the end-of-file marker. In other words, a multiply-updated PDF document may contain multiple trailers – and multiple end-of-file markers! (There may be numerous occurrences of %%EOF.) Each time the file is edited, an addendum is written to the tail of the file, consisting of the content objects that have changed, a new xref section, and a new trailer containing all the information that was in the previous trailer, as well as a /Prev key specifying the byte offset (from the beginning of the file) of the previous xref section. The cross-reference info will then be distributed across more than one xref section. To access all of the cross-references, the reader must walk the list of /Prev keys in all the trailers, in reverse order.

Space doesn’t permit a detailed exploration of updates here, but you can find several examples in Appendix A of the PDF 1.3 specification (available at http://partners.adobe.com/asn/developer).”

Summary

It is important to understand that the PDF standard allows for appended updates to files that leave the original document intact, regardless of how drastic the changes are. If you are intent on redacting text from PDF documents, do not depend on simply deleting the secrets using a PDF editor—you must use a proper redaction tool that addresses these issues correctly.

That said, I did some experimenting with a few utilities (Apple Preview, PDFpen, and Adobe Acrobat Pro) and found that some write the file from scratch each time, with no lingering cruft from former versions, while others respect the original intent of the PDF standard. This means that you can’t trust that older revisions are being retained in your file and you can’t trust that they aren’t.

Be conservative: use a redaction tool for secrecy and proper backups for versioning.

As it would turn out, there were several thousand other documents on a file server that shared the same funny behavior. By the time we were done struggling with this problem I had gained new respect for PDF corruption issues and their prevention.

The Problem

We were looking to load a few thousand of these scientific reports into a fancy-schmancy new database, with linguistics searching and other bells and whistles. Much to our chagrin, these documents just weren’t loading, and we couldn’t understand why. They were text documents, with some embedded images, but mostly straightforward text.

Here is an excerpt:

And you can tell that it is right and proper text because when I blow it up all the way, the fonts are nice and smooth—this isn’t just an image of text.

But if I copy and paste that particular paragraph into any handy editor (Notepad, in this case), this is what I see:

And as far as I know, at this point the actual text is beyond the reach of average folks like me. We tried, believe me we tried.

What went wrong?

A quick Google of the subject led us to understand that many PDF generation tools embed subsets of fonts, with nonstandard mappings from the text to the font.

This fellow explains it nicely:

“The PDF file does not contain all the information to extract the text. The problem is that a character in a PDF file may not contain information what “real” character it relates to. Some PDF generators do a pretty bad job when they embed fonts into PDF files. They use a proprietary encoding mechanism (e.g. 1 is A, 2 is B, 3 is C, …) in both the embedded font and when they place glyphs on the page. Without a table that implements the reverse (e.g. character code 1 is ‘A’) you cannot extract text from such a file.

There is nothing you can do (besides to complain to whoever created the PDF file, and the author of the software that created this file).”
— from khkremer on experts-exchange.com

As it would turn out, many of the reports had been generated by printing to Adobe Distiller from Microsoft Word. It would seem that the default settings used for Distiller included the “totally hose my document content” switch.

The Solution

We fretted over this quite a bit. These are important scientific reports, and there is no way to easily ungarble them. We finally ended up contacting the Abbyy Finereader folks and trying out their OCR toolkit for Linux: not only did this product make fast work of running optical character recognition on the sample document, but once we had a script running, we managed to blow through the 10,000 pages the trial license gave us, in a day or two.

Imperfect, at best

I am happy that we were able to salvage the bulk of the electronic knowledge found within those thousands of files, but our work barely scratched the surface.

For example, most of these documents have rich bookmarking of sections and keywording, such as this (content tastefully blurred on purpose).

In addition, scientific documents typically have loads of tables full of numbers. Though it is possible to mine this data with a good OCR tool (the FineReader API provides tools for just this purpose), the tables are far more difficult to extract correctly once the original text information is lost.

Final thoughts

I wrote a few weeks about document formats, mentioning the PDF/A document standard. This is worth investigating, regardless of what your document needs are.

If our thousands of files had been originally generated as PDF/A, it is certain that we would have been able to copy/paste from them without problem: PDF/A prohibits such font shenanigans as were perpetrated on our garbled reports.

In the end, our OCR sledgehammer approach worked like a charm, and is probably sufficient for our needs. Text mining is a pretty slushy business, so no-one will complain if there are a few typos on each page—if they find the doc in a search, they can print it and read it the old fashioned way.

Over the past couple of years I have considered several different approaches for keeping a grip on which computers had which service patch, which router is getting flaky, and which cable connects the library to the classroom at the end of the hall.

I have tried Excel spreadsheets, an Access database, even a spiral-bound notebook—none of them made the job any easier. A few weeks ago I thought about trying a Wiki and this has turned out to be a perfect fit!

If you are looking to keep a loose scrapbook of notes with lots of arbitrary categories and relationships between them, a wiki might do the trick. In this article I’ll cover two simple freeware wikis you can carry around on a thumb drive.

What’s in a Wiki?

All of us have used Wikipedia at one time or another, and though it may be regarded with disdain by high school teachers, when you consider how it works, Wikipedia is an amazing achievement. But what is the nature of a wiki?

One of the key features is that any page can be easily edited at any time (of course this can be limited by permissions). Another attribute is the ability to breathe life into a new page just by calling its name.

Between these two features, you get the essence of wiki-ness.

For example, if I have a page that discusses North American bears, I can type in a list of bears in a special format, often in jammed-together Wiki Words, like this:

• GrizzleyBear
• BlackBear
• BrownBear

As soon as I save the page, those bear names become hyperlinks. Even though I haven’t written any pages about the individual bears, whenever it finally suits me, I can click on BlackBear and accept the invitation to “Create a new page called BlackBear”

Better still, a friend who knows about black bears might click on BlackBear and write a beautiful page about the animals.

That’s what wikis are all about.

Back to the School Computers

In a matter of minutes I was able to make a page that described the building and listed the various rooms in the building. I was able to then click on each room and “auto-vivify” a page for the room.

From that point, it was easy to create custom pages for each computer in the building, with each page listing the machine’s stats. I also created pages for each network switch or router.

In a matter of two or three evenings I had the skeleton of a solid knowledge base populated—it’s a pretty fancy looking web site with dozens of pages that took little effort to put together.

Last night I noticed that one of the machines wasn’t connecting to the Internet, though it connects fine to internal servers. I popped open its page on the wiki and added a simple note at the bottom of the page:

2009-10-11 - This machine isn't able to connect to the Internet. Not sure why. It connects fine to internal servers.

A few weeks ago I replaced a fan in a network switch. An easy annotation on the wiki page for that device.

Personal Wikis

There are many uses for personal wikis, mostly centered around personal knowledge management and personal information management. People use wikis as a replacement for time and task management tools, as a place for gathering thoughts, as a sort of amorphous database, and many other things.

There are many different personal wikis available—here’s a short list of free ones. One nice simple wiki to try is TiddlyWiki. If you are looking for something with a bit more substance, you can try a portable version of MediaWiki—the engine behind Wikipedia—that runs off your thumb drive.

TiddlyWiki

This afternoon I downloaded the flyweight portable wiki called TiddlyWiki. This is an amazingly tight little application—it comes in the form of a single fat web page that you copy to your thumb drive. As you make edits to your TiddlyWiki, the single html page is saved with your changes. Since it’s a single fancy file, backups are dead easy.

Here’s what it looks like when you first launch the “empty.html” file:

After a half hour of twiddling around, I had thrown together this basic set of “Tiddlers”

In this screen shot you can see that there are now links that bring up custom “Tiddlers” for each computer and for each room. I have opened one of the little pages for Computer21.

They describe these pages as being comparable to note cards. All in all, it is tight and easy to use.

Want to give it a try? Download it from the TiddlyWiki site. You really need to play with it to get a feel for what it can do!

MediaWiki

If you are looking for something with a little more meat on it, you can run the Wikipedia engine on your USB drive.

The easiest way to set this up is to let MoWeS do everything for you. MoWeS stands for Modular Webserver System. It’s a free product that you can configure as a self-contained Apache web server with a variety of cool apps like MediaWiki, running off a thumb drive.

Here’s how to set up MediaWiki in five minutes:

• Go to the MoWeS Mixer
• The first time around choose “I do not have a MoWeS Portable II Package and want to obtain a new package” when prompted and click Go.
• On the software lists, check Apache2, MySQL5, PHP5, and MediaWiki
• Click Download Now
• At this point they ask you some kind of question in German, to filter spambots, but it seems to be a simple math problem. Fill in the answer and click Submit Query
  (“Zum Schutz vor Downloadrobotern geben Sie bitte das Ergebnis dieser Aufgabe ein: 5 + 8 = ?“)
• Unzip the downloaded zip file, mowes_portable.zip, and copy the files to your USB drive
• Open your thumb drive and double-click mowes.exe
• Select your language and accept the license
• Click install, and confirm when prompted

The installation process may take several minutes, but rest assured that it isn’t installing anything on your computer.

Note: I received two or three firewall warnings for the Apache web server and the MySQL database. I had to click the “Unblock” button for all of them before my new MediaWiki-on-a-stick would work correctly.

After all of the dust settled, I have this little window on my screen:

In order to shut down and close out, just click the End button.

Once your MediaWiki USB key is running, you can go to this web page:

http://127.0.0.1/mediawiki/index.php/Main_Page

It looks just like Wikipedia, doesn’t it?

What a truly amazing thing: you can carry around your own Wikipedia server on a USB key and plug it in any random machine and start it up.

Different Wiki Features

As you try out different wiki software, you will notice that there are plenty of differences in the features they support:

• Each wiki has a different kind of editor. Some are visual; others are simple text editors.
• The markup syntax you use for pages is different from wiki to wiki.
• Most wikis support features such as “category pages” that find all pages tagged with a category.
• Some support adding images and other content; others don’t. I imagine that TiddlyWiki probably has some means of embedding images, but I couldn’t find it.
• A quick glance at the MediaWiki screenshot above shows extended features such as the Discussion tab and the History tab.
• Some use the filesystem for their pages; others use a database.

Since I wanted a central wiki for the whole school, I chose a different product from the portable wikis I discussed here—I decided to run MoinMoin on a Ubuntu installation on an aging Gateway desktop machine. Nevertheless, the basic idea is still the same.

Once that arrangement becomes a little more stable I’ll write up a howto document, like the Linux NAS one from a few months back.

Other Sources

There are loads of different personal wiki options out there and many people have written how-to documents and tutorials. Here’s a few:

• Run Your Personal Wikipedia from a USB Stick (Lifehacker.com)
• Geek to Live: Set up your personal Wikipedia (Lifehacker.com)
• Wiki On A Stick (PmWiki.org)
• Getting Started with Wiki on a Stick (About.com)
• TiddlyWiki for the rest of us (giffmex)

Now prudence would lead one to take careful steps and wait until all of the wrinkles are ironed out before starting. I was almost not prudent enough this week.

Mac OS X Snow Leopard

So folks have been talking about the new Snow Leopard operating system for Mac. Over the past year, Apple has been positioning this version as more of a “under the hood” upgrade that tightens things up rather than a glitzy overhaul of the user interface. No matter what they said it was, I figured that it was newer, and therefore better, than the current OS—Leopard–and I had to have it.

I ordered my copy last week on Amazon and sat down with a smile as I awaited its arrival. And then I thought about doing a few quick Googles to see how other people have been making out with Snow Leopard. I immediately happened upon a few upgrade guides like this one, providing sage advice about the upgrade process. They recommended the “slash and burn” method, starting from a clean hard drive, and I felt that was a good idea. Nothing better than a wipe and fresh install to make your machine zip along twice as fast. And therein lies a tale.

The first sign of trouble

As I was reading up on the Snow Leopard upgrade process, I happened upon lists of “unsupported software” and casually glanced at the lists, expecting esoteric tools only used by three über geeks in the audio recording industry or perhaps some exotic ray-tracing software. Much to my surprise, I saw two of my favorite applications, in a very very short list of troublesome apps: Parallels and EyeTV.

I immediately checked the versions and breathed a sigh of relief when I saw that my EyeTV version was safe. But, Parallels was another story… They have no plans for patching Parallels 3 to work with Snow Leopard, and why should they, when they can sell us Parallels 4!

So, I ordered my fresh copy of Parallels 4, from Amazon with a twenty dollar rebate. When it arrived, I spent an evening upgrading Parallels, and I thought I was all set for Snow Leopard.

Preventative Measures

Following the advice of the upgrade websites, and prior experience, I used Carbon Copy Cloner to make a full backup of my hard drive on a spare external drive. On a hunch, I turned on the drive that I use for Time Machine and had it do one final “Time Machine” sweep through the system before bidding adiu to Leopard.

I knew that I had all of my installation media for stuff like iLife and Photoshop Elements, and I had all of my license keys in electronic form. It would be a simple matter of mounting the backup drive, copying over my loads of documents, and peering into them to find keys.

The first attempt

I boldly inserted the Snow Leopard disk and booted from the DVD drive, selecting the “Slash and Burn” method of installation. I reformatted the hard drive and went off for dinner while Snow Leopard installed.

Trouble

When I got home that evening, I started the lengthy process of installing stuff. I suddenly realized that it was not as easy as I had hoped: it’s one thing to reinstall something like Microsoft Office, but there seemed to be more loose ends than I had considered:

• How would I migrate my Mail settings from the old image to the new?
• What was the best way to migrate the Address Book contents?
• iTunes is great, but it has tendrils in everything. Can I simply copy my old library to the new without messing up my iPhone, Address Book, or other linked stuff?
• How about those nice password tools such as 1Password and SplashID that keep your passwords safe and sound? I had no clue how to get their contents from the backup. I wasn’t sure if it was even possible to do so—perhaps I was supposed to have exported the data beforehand.

It was becoming clearer to me that I had not done my homework at all.

More trouble

My initial shock at the depth of the upgrade process led me to start making a list of applications and looking at what I needed for each one. I soon found out that Snow Leopard support is somewhat spotty in many applications. In particular, the FineReader for ScanSnap software that I depend on so much for my scanning work flow is not fully supported. Fujitsu says that they will have an update soon and to keep checking their web site.

My password tool, 1Password, is another problem child. It works only on 32-bit Safari, and Snow Leopard now runs Safari in 64-bit mode. Of course, a new version is coming, and I will probably have to pay for it, but it is still in beta.

There was quite a bit of chatter on the Web about whether Adobe Photoshop Elements would work on Snow Leopard, and the responses seem split fifty-fifty for now.

Three very important tools were in danger of running in limited mode or not running at all, so I had to throw in the towel.

Time Machine saves the day!

As I sat, humbled, before my vanilla install of Snow Leopard, I admitted defeat. I slipped the Snow Leopard DVD back in the drive and rebooted from the DVD. This time, I selected the “Restore from Time Machine” option and turned on my Time Machine drive.

Guess what? It worked perfectly! Unlike many software products, Time Machine does exactly what it promises.

Within a few hours, my machine was fully restored to the way it looked seconds before I made my first attempt at Snow Leopard.

A Final Word

Learn from my mistakes, and my salvation by the full backup. As much as you can’t wait to upgrade, please do the following:

• Inventory all of your applications that you really need.
• Obtain the installation media (download or CD) for every single one.
• Obtain the keys for every single one.
• Investigate whether you need to export data from any of them, and make a checklist for these exports prior to upgrade.
• Check the “Unsupported Software” lists that are out there for any red flags.
• Check the web sites of your most important apps for their official word.
• And finally, do a complete backup!

It’s amazing how many applications and weird little utilities we forget we have. How could I have possibly remembered that I compiled a custom copy of the “rsync” executable for my backup workflow? I would have lost that and had to figure out how to rebuild it on Snow Leopard.

And I haven’t even talked about making sure your documents make it safely onto the new machine. That’s a whole ‘nother story.

In case I forgot to say it, please make a full backup.

[Update: I'm giving Snow Leopard a rest for a few months]

It has been said that Time Machine allows you to do a full restore from bare metal, and I’m living proof: I have done exactly that twice in the past week, with astounding success.

Encouraged by an episode of the Mac Geek Gab where they talked about their experiences upgrading their existing systems to Snow Leopard, I decided I would give the upgrade-in-place option a try. I expected some things to not work well and others to be quirky, but here’s what happened…

The actual install was painless, taking an hour or so to complete. I then began to kick the tires to see what was broken.

It was clear where those 64 bits went: apps like Safari were positively zippy, and I was pleasantly surprised with each new application I launched. All of my special settings seemed to make it through alive, including my password manager, though I did have to re-enter some of my registration keys. All of my mail and contacts made it through well. I was able to sync my iPhone without incident.

I found a few apps that weren’t working correctly and I looked for newer 10.6-compatible versions. I found newer versions of Yep and xGestures.

I did note that there is currently no ad blocker available for Safari that runs in 64-bit mode. This is disappointing because even though I understand that Apple wants us to see their ads, I can’t imagine that they really want us to suffer from the flickering jumping dreck that should have ended with the hated “punch the monkey” banners of years gone by. The fact of the matter is, if I want that 64-bit speed and snap, I guess I have to watch ads.

The Showstopper

I decided to scan a document to see just how difficult it would be to get my workflow going again. Michael F, below, wrote the truth about the situation: the scanner works fine in certain modes, but the OCR software doesn’t.

He pointed out that it was a problem of the FineReader software looking for a specific bit of metadata in the PDF identifying it as a ScanSnap PDF. Sadly, that metadata string changed.

The Finereader software is looking for “Mac OS X 10.5.8 Quartz PDFContext”, but under Snow Leopard, the string is set to “Mac OS X 10.6 Quartz PDFContext” instead.

There are ways to tweak PDF metadata, and one of them is by using pdftk.

I went to the pdftk site, all ready to download it and start OCRing my PDFs. I was greeted with less than optimal news: they have a version compiled for Panther, a version of OS X from several years ago.

I knew it wouldn’t work, but I gave it a try anyway: the app told me it needed Rosetta to run. I could have installed Rosetta at that point, but I figured I wanted a proper compiled version.

From there, I looked into compiling the app on OS X 10.6. I should have remembered my struggles with this several months ago on a Solaris Unix box when I found that pdftk depends on a monster called GCJ that required about forty other software packages to compile—it seemed a gargantuan task that I wasn’t ready to begin.

On a hunch, I inspected the content of a new pdf and an old pdf, the latter still acceptable to FineReader. Though much of the file was raw binary, the metadata was in text at the end. A short sed script was all it took to swap the nice text string for the offending 10.6 one.

In spite of my best efforts, FineReader still rejected my hand-tooled PDF file. It knew that it was a bogus file.

I have looked into Abbyy FineReader several times before, as well as Fujitsu’s ScanSnap support, and was unimpressed. For two vendors that produce products that are at the top of their class—FineReader is arguably the best OCR you can get for Mac, and ScanSnap is the best document scanner for the common man—they sure do have miserable customer support.

It is as if neither company cares a whit about the Macintosh platform or their customers. While most other vendors are busily patching their products and giving hourly updates on their Snow Leopard compatibility progress, Abbyy and Fujitsu just don’t seem to care that their best-of-breed combo suddenly doesn’t work on Mac.

Once they get this sorted out (hopefully in the next few months) I’ll give Snow Leopard another try. In the meantime, I’m sticking with good old Leopard.

I had promised that once I cleaned up the embarrassing bits of code I would post a perfect polished version here, but such promises are rarely fulfilled. A reader posted a comment asking for that source code, so I will post it here in its current state. The truth is, I have been using this script for months and, though it has some quirks, it works fine.

So this post is about Macintosh, AppleScript, and the ScanSnap-to-FineReader workflow. If these don’t interest you, better move on.

Update: The script on this page works only with Leopard (10.5). Get the Snow Leopard version here

The Original Problem

The Fujitsu ScanSnap S510m, my workhorse scanner, was designed to scan documents quickly and generate PDF files—this it does flawlessly. In order to provide OCR support, they have shipped a special version of FineReader, called FineReader for ScanSnap. The standard OCR configuration is to chain the output of the scanner to the FineReader program.

The problem is that this forces scanning and OCR to run in lockstep: you scan a document, you wait for OCR, and then you scan another document.

My desire was to write a simple AppleScript that would detach the “Scan a Document” process from the “OCR” process. By using this script, I can scan documents at whatever rate pleases me, and the OCR engine will chunk along at its own pace, consuming my scanned documents and producing OCR documents.

My Approach

I really looked hard at the OCR application, trying to find AppleScript hooks or special command line switches that might allow me to control it better. Sadly, it was not designed to be scriptable. The only thing I could do is call the FineReader application with a source file.

Given this limitation, I considered writing a script that would look at a particular folder, identifying new files as they appear and passing them on to FineReader.

Fortunately, AppleScript provides this kind of functionality with little effort in the form of Folder Actions. Perhaps the best way to see these in action (and try it out) is to see this post on Exploring the power of Folder Actions.

In order to achieve my goals, I did the following:

• Created a folder called “Pending Documents”
• Wrote the script to find the oldest-unprocessed-file and call FineReader with it
• Attached the script to the folder as a Folder Action

The Script

Let’s jump right in to the AppleScript. Download the script here.

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action ScriptsYou may have to create the Folder Action Scripts folder.
• Now open a Finder window, control-click and choose More / Configure Folder Actions…
• Check the Enable Folder Actions checkbox, if not checked
• Click the “+” in the bottom left
• Select a folder and click Open
• Choose the script Run OCR on New Folder Items and click Attach

Picky Details

As you can see in the source code, there were several issues to address:

• I had to make sure the script didn’t step on itself. If FineReader was running, I would wait until it was ready before processing.
• The script needed to determine which files had been processed already. This was handled fairly trivially by looking for a matching file with the processed by FineReader.pdf suffix. In other words, if I was looking at Scan001.pdf, I would see if there was a matching Scan001 processed by FineReader.pdf file.
• Part of checking for a source file’s “buddy” was stripping off the PDF suffix. This was done in a hackish way by using a one-line shell script, at lines 106-107.
• I thought it was important to verify that the source file was, indeed, a ScanSnap file—the FineReader will not process other PDF documents. This was done at lines 117-121 by looking at the Spotlight metadata for the Creator of the source file. That took some more shell scripting (133-154).
• The actual work was done by a single line, line 63.

The real work was fairly simple, while the bulk of the code was needed to polish pesky little details. Isn’t that the way code development often is?

If anyone has any improvements on my script, please let me know!

Their network security department very kindly (and politely) informed me that they had received a “cease and desist” order from a particular game publisher. They had included the game publisher’s email, complete with the incriminating evidence.

There it was: logs showing the MAC address of my cable modem being involved in suspicious BitTorrent activities.

Considering that at any time during the week there can be from two to six or seven different teenagers hanging out in my humble abode, carrying virus-ridden machines, the message was clear: I had to get serious about locking down network access

The Problem

I would have liked to have bought some net filtering software to slap on the offending machine and been done with it, however I knew that this was insufficient.

Even if this one event could be traced to a youthful source, a more ominous danger comes from the inevitable malware and viruses that teenagers collect on their machines as they swap cool stuff with their friends.

Complicating things, there are many devices on our home network: Besides their school laptops, the kids have video game consoles and one has an iPod touch, all with wifi access. Think about how many different gadgets are on your home network.

And shutting off access altogether was not an option—there is still schoolwork to be done!

The answer: A Private Network for the Kids

My solution was to put together an unusual network configuration using a second wireless router; I wanted the ability to manage every single kid-owned device at the flip of a switch, while leaving the grownups untouched.

I hooked the cable modem (red) to the main router, shown in green. I then plugged a second wireless router, shown in blue, into the first.

By doing this, you can see that there is one single wire connecting the entire blue network (the kids) to the green network. It was trivial to then configure the green router with appropriate access control and filtering for that one single device: the blue router.

Some quirky details

Home routers like these are, by default, configured with a NAT firewall. They work sort of like one-way mirrors: someone on the network can see out, but nobody can see in. As a result of this, the kids (blue devices) can see any device on the main router (green devices), such as our print server and the NAS device, but no one can see into the kids’ network.

As paradoxical as it seems, this is exactly what I wanted. By making the kids’ network a private network, it appears to the green router as a single device. When I am configuring access restrictions, I only need to control access for the blue router’s IP address or MAC address.

Many consumer-grade routers have flakey firmware that just doesn’t really behave well when you start doing things like turning on filtering for multiple machines. I simplified things by bringing down the number of controlled devices to one. In addition, if one were to try filtering on the IP addresses or MAC addresses of individual machines, this can be easily defeated by manually changing the IP address or MAC address. With my configuration, the MAC address being filtered is the blue router, locked away safely.

The Finer Points

If you want to set up a network like this, do the following:

• (Recommended) Reset the kids’ router. Hold the hard reset button on the router in while you turn on power; hold the button for 15 seconds or so.
• Hook the kids’ router up to a spare laptop using an Ethernet cable. (Turn off the wireless of the laptop for the time being).
• Use the laptop to navigate to the configuration web page (usually 192.168.1.1).
• Set the router’s own address to a different network from the main network, such as 192.168.2.1. This is critical.
• Configure the router’s gateway and DHCP server entries to all point to the main router (192.168.1.1). This tells the kids’ router to use the main router as a source for its DHCP lookups and such, rather than going to cable modem.
• Navigate to the configuration web page at the new address (192.168.2.1). You may need to close the browser and replug the Ethernet cable.
• Set up your wireless security for the kids however you like. Make sure to choose a different channel and SSID from your main router.
• Remove the laptop and plug the WAN port of the kids’ router into one of the LAN ports of the main router. Restart everything.
• Test both networks to make sure things work the way you think they should.
• (Optional) You might want to connect to the kids’ router and set it’s external IP address statically. Make sure that this is set to a number on the home network (e.g. 192.168.1.2).

Some notes:

• You can only maintain the kids’ router from a machine connected to the kids’ network; the home network cannot see the management screens. If you wish, you could enable remote management for the kids’ network only, since the main home router is still protecting the whole network from intruders.
• Computers on the kids’ network can see all devices, but they aren’t on the same network. This means that network printers and NAS devices are accessible, but you will have to attach to them using IP addresses. I was able to easily set up the machines on the 192.168.2.1 network to use a print server on 192.168.1.100.
• For machines that should have full access (a.k.a. yours), make sure that you either set the green network to be a higher priority or remove the blue network SSID entry altogether. I found out the hard way that my iMac would randomly pick the green or the blue depending on which one it saw first when it woke up.
• This does not wall off your main network; it simply provides a single point of control to the entire kids’ network. In other words, don’t depend on this setup to prevent malware on the kids machines from seeing your machine. You can, however, set up your PC to not trust the kids’ network.

Wireless Network Security

Regardless of how you set up your network, make sure you use at least WPA encryption (Never use WEP!). Make sure your passwords are solid.

Using DD-WRT on my new wireless router

In addition to the new network configuration, I went one step further and chose a main router that lends itself well to installation of open-source firmware. I ordered a Linksys WRT54GL from Amazon for a little over fifty bucks. I chose this one because, as a direct descendent of the venerable WRT54G, this router is very well suited for running alternative firmware such as DD-WRT, giving substantial control over things like, say, access control…

Within a half hour after my new router arrived, I had gone to the Supported Hardware page, obtained the latest build of DD-WRT, and replaced the Linksys firmware with the far-better open source code.

I won’t go into the specifics of installation here, but it isn’t very challenging. Check out the DD-WRT site for details.

Closing Thoughts

Make no mistake: we are responsible for whatever goes on our home networks. Just like your home telephone; if someone dials up some 900 number and rings up a thousand-dollar phone bill, the phone company won’t care a whit who did it, you will still pay. Likewise, regardless of who did the BitTorrent download, there is a certain degree of responsibility of the homeowner to lock down the network.

Another point: Without some degree of personal responsibility on the part of the kids in the house, this sort of activity would simply be an arms race of filtering and blocking versus hacking. My goal is to help keep the honest people honest and to make life more difficult for the viruses and malware.

Another issue altogether is the spam email that is carefully crafted to appear as if it has come from your bank, saying cheerfully “Your statement for May is available online, just click here to access!” … but whoever clicks will inevitably be providing their secrets to some ne’er-do-well in New Zealand who will promptly empty their accounts.

Here is a simple, quick, and free way to avoid phishing attacks as well as casual/accidental exposure to unwanted adult content.

OpenDNS

The service I am referring to is OpenDNS, a free domain name lookup service that you can use in lieu of your Internet Service Provider’s own DNS servers.

When your computer goes to a web site, the name of the web site must be converted to a numeric address, in much the same way that you use a telephone directory to look up a friend’s number.

This lookup service is typically provided by a server owned by your Internet Service Provider. The address to this server is automatically configured when your cable modem connects to the network the first time.

The way OpenDNS works is you change the Domain Name Server (DNS) setting in your router to now point to the OpenDNS servers instead of your ISP servers. By doing this, you have changed the default telephone directory used by your home network.

A Phone book with the Bad Numbers Missing

To take the phone book analogy further, imagine that in your new phone book, all of the phone numbers for shady businesses such as escort services and massage parlors have been replaced with a special number. When you dial that number, a pleasant older woman gives you a gentle scolding for trying to call such a business.

This is pretty much what happens with OpenDNS: when your browser asks for a page from www.naughtystuff.com, the OpenDNS server points you to a different place, a nice page from OpenDNS that says that the page is blocked and explains why.

One fix for your Entire Network

There are many options available for “net nanny” style software that can be installed on individual machines, such as the kids’ machine. These features are also embedded in modern versions of Windows and OS X. But, what about all of the little portable devices that find themselves into kids’ hands? How about their gaming consoles?

Since you configure OpenDNS at the network entry point to your home, the router, any device attached to your network is automatically covered.

Customizable Blocking

You can use OpenDNS without an account, just by pointing your router to their servers, but the real power comes when you register with them (for free) and make your own choices about what you want to see.

You can choose which parts of the Internet you don’t want to see using their online configuration tool. You can either use their “High/Moderate/Medium/Low/Minimal” options or you can pick and choose individual bits of stuff to allow or block.

Here’s a look at the categories available when you choose the custom blocking level:

Basic Setup (about 20 minutes)

• Configure your router to use the OpenDNS servers for DNS lookups.
• Create a free OpenDNS account.
• Install their small updater program on one machine on your network.
• Log in to your OpenDNS Dashboard on the web and configure your blocking settings to taste.

Why do you need the updater utility?

In order to provide the custom blocking, the OpenDNS servers need to know your main IP address assigned by your Internet Server Provider. The desktop utility simply informs OpenDNS of your new IP address if it ever changes.

What do users see if they go to a blocked page?

They see a page that indicates the site that was blocked, along with a short reason and a link they can click if they want access to the page. If they click that link and fill out the short form, you will get an email from OpenDNS with the user’s request.

The remainder of the “blocked” page is a search form with some sponsored links.

You can customize the message as well as the image shown on the web page. When someone reaches a blocked page in my network, they are greeted by a picture of our calico cat, Roxy.

Keeping the Honest People Honest

This approach to blocking unwanted web sites is not a complete solution for keeping your kids from where they shouldn’t go; it is more like a simple padlock: it keeps the honest people honest. A determined individual can easily get around this product using various techniques, but they have to make a conscious effort to do so.

The real strength of OpenDNS is that it helps avoid accidental exposure to unwanted web content and phishing sites.