There are plenty of different ways to catalog loosely associated knowledge of varying complexity—a few months back I discussed using a wiki for this—but some problems just don’t need that level of complexity and depth.

Some problems are more suited to random scribblings on a whiteboard, and that is where mind mapping software comes in.

What’s a Mind Map?

Imagine you and your family are going to fly to Rio de Janeiro this summer to visit distant relatives and you realize that there are about a thousand things to do  in preparation but you just can’t sort it all out.

You know that you need passports and you need to verify that everyone’s visa is still valid. There is the monumental task of deciding what to pack. You might want to make a checklist of places you want to visit. And you want to go hang gliding down to the beach, but there’s something nagging at you about whether or not your health insurance would cover a broken leg in a foreign land.

The problem is that it is difficult to keep the whole thing in your mind—if you concentrate on the luggage, you forget about the international driver’s license.

Mind maps allow you to visualize the whole thing at once, and you can slide stuff around and get it looking nice and pretty.

This is a relatively simple start at a mind map that represents the vacation. These maps are typically read from the top right going clockwise, though many have no specific sequence. In fact, you can do pretty much what you like with a mind map as long as it works for you.

Just Another Outliner?

At first glance these tools look like glorified outlining applications such as OmniOutliner, and they do serve admirably in this respect, but they are so much more. An outline gives you a very easy way to organize topics and thoughts, adding annotations and such along the way, but it isn’t nearly as easy to visualize and nonlinear concepts do not map well to an outline.

Consider the Rio Trip example above—you could put all of that information into an outline, but it would not be nearly as easy to process mentally.

And these mind maps look especially cool in presentations.

Slick Document Generation

Some of the commercial mind mapping products provide pretty good integration with Microsoft Office products.

Some time back, I needed to write up a set of style standards for Oracle’s PL/SQL programming language for our offshore team. Rather than just dive into Word and hope for the best, I used Mind Manager Pro from Mindjet to make the following map:

This tool supported attaching rich text to nodes in the map, so I used these notes to handle the actual code examples.

I was then able to put together a nice Word template that matched our corporate documents and I clicked the Export to MS Word button and had an instant document:

I have also used mind maps to auto-generate PowerPoint slide decks as well. These days I use Mind Manager on a daily basis at the office; it makes a great difference when I am trying to grasp complex topics with lots of strange dangly bits hanging off of the edges.

Nothing New

Mind maps have been around for a long time. A quick search of the ‘Net will show you that mind mapping software is quite plentiful and mature. There are good free products for PC and Mac available and there are many commercial products that take mind mapping a step further, often integrating with Microsoft Office.

Take a look at what Wikipedia has to say about mind mapping. Here’s a post from 43 Folders on the same topic. Peter Russell has useful information as well about them.

I learned about mind mapping from a friend at work who had been using them for years. After seeing him make some quick notes during a meeting, I was sold.

Closing Thoughts

I made the Rio de Janeiro example using XMind, the free “lite” version of XMindPro. This application is pretty full featured for a free basic version—the Pro version adds enterprise features such as import/export and collaboration.

There are XMind versions for Mac, PC, and Linux.

If you are looking for more, all of the commercial products offer a trial period. I use MindJet’s Mind Manager. Be warned, these tools are expensive, just like buying MS Office, but you might just find that they more than make up for their cost with your newly found productivity.

I’m serious. You don’t need to know anything at all about Linux to use one of these. Just download the image, install, and you suddenly have a full featured NAS file server, or you might have a database or a source code repository.

Last year I wrote an article on how to set up a NAS device using Ubuntu Linux. I have been a fan of Ubuntu since the start because it is a very easy distribution to install and configure. The down-side of using Linux has always been the fairly steep learning curve. Before you can get around to using the server, you need to get down in the weeds with configuration files and other stuff.

TurnKey Linux changes all of that.

Painless Installation

A few weeks back, I was setting up an aging PC as a standalone wiki server for a small office—this machine was going to provide a place for the office staff to document their procedures, how-tos, and other things.

I was about to set up an Ubuntu server, as I have done before many times, and install MoinMoin, like I did some months back. I remembered that it was a bit of a pain to get everything tweaked just right, so I did a quick check to see what kind of standalone wiki options were available online.

This is how I found TurnKey Linux. This project is all about single-purpose preconfigured Ubuntu server images.

One of those preconfigured images happens to be a MediaWiki appliance—the wiki engine behind Wikipedia—and I was in business.

The installation took about fifteen minutes, with very little user interaction. I answered a few basic questions and the installer took over from there. As soon as the install was done, the machine rebooted and displayed a message on the monitor with the IP addresses where you can browse to from any other machine.

Full Featured

The work that has gone in to these appliances is amazing. In fifteen minutes I had installed a complex configuration that has the Apache, PHP, MySQL, MediaWiki core, as well as maintenance utilities such as a neat tool that provides a Flash-based pure-AJAX-based SSH command line in a remote browser (i.e. your browser becomes a terminal). Even someone with Linux experience would have to spend quite a bit of time fiddling around with different packages and configuration options in other to provide the same functionality that TurnKey gives you out of the box.

As with most open source projects, the documentation is about 80% complete, with deep detail in some areas, but leaving others fairly sparsely documented. But don’t let this deter you: in most cases users know how to use the product they are installing (e.g. MediaWiki) but don’t want the hassle of configuring it on Linux. That’s where TurnKey shines.

Some Examples

In minutes, you can set up a NAS device. If you want to try advanced content management in your office, try Joomla or Drupal.

If you are working on a small project team and want to protect your source code, try Redmine or Trac and do your bug tracking using Bugzilla.

And while you are at it, you can document your organization’s working practices using a wiki such as MoinMoin or MediaWiki.

Don’t forget to back it up!

As with any computer, you should include your new TurnKey appliance in your backup strategy. The nice thing is that you don’t really need to care at all about backing up Linux or the other software; just back up the data. I don’t need to back up my entire MediaWiki machine; I just need to back up the database and image files. If anything goes wrong, you can rebuild the TurnKey appliance from scratch in minutes and then restore your data.

To save yourself some pain, keep notes on any small tweaks you made to the configuration.

One Machine, One Purpose

These disk images share common Ubuntu underpinnings, but they are referred to as Appliances because they turn your PC into a purpose-built appliance.

This means that if you want a content management system and you also want a ticket management system, you will need two old computers—not a rare commodity these days.

Take a look at what they have to offer and give TurnKey a shot—specialized software used in corporate environments is now within reach of small offices at the right price.

The problem is this: with Outlook we will be limited to 1000 messages in our in-box; sadly, many of us have tens of thousands of emails in our old GroupWise mail. Even after a fairly rigorous slash and burn mission, hacking out all of the low hanging fruit, there will be many thousands remaining and I don’t want to lose that information. It might be useful to search and find how I set up a Zebra bar code printer in 2003, no?

A bundle of different freeware glue tools came to my rescue. Read on to hear about the toolset that has made it so I can keep those messages for years to come.

Possible Solutions

Right out of the gate, I began looking for ways to migrate messages from one mail client to the other. Some apps have this built right in, and if not, there are scripts and utilities out there to do this; but I was hampered by a few key facts:

• I have no control over the email clients and their configuration. Even if there is a menu option for exporting GroupWise messages from version 7.2, I’m stuck at 6.4 and cannot use that option.
• GroupWise is a minor player in the email world. I’m not sure if Outlook would import from GroupWise, but I doubt it.
• They are replacing the client in one shot. There will be no interim period where both GroupWise and Outlook will be available.
• There is no getting around the hard limit of 1000 messages.
• I don’t want to spend money on this.

With these constraints in mind, I immediately thought about PDF documents. I then considered the following questions:

• How do I convert my email to PDF?
• How can I do this automatically with thousands of emails?
• Once I’m done, how do I search these documents?

Here’s what I did:

Conversion to PDF

The first part was easy. I downloaded one of the many free print-to-PDF products available.

I chose PDFCreator, because I am familiar with its use and I know that it does not munge the fonts.

Like many other PDF generation utilities, PDFCreator functions by providing a virtual printer to which any application can print. For example, to make a PDF of a web page, you use the Firefox Print menu and select PDFCreator from the drop-down list of available printers.

You are provided with a list of metadata fields that you can fill in, and these fields are used in the PDF generation.

Here’s what the PDFCreator screen looks like:

A word of caution: PDF Creator is free, but you must be careful to deselect their spammy toolbar options in two different places during the installation process. I don’t like software that comes with preselected toolbars to install (even nice ones like Google’s) because I’m certain that 95% of the folks who actually install the toolbar would never have chosen to do so if it were unchecked by default.

Running Everything Automatically

This was the interesting bit. I work with Windows machines at work, so there was no AppleScript option available. So I did the next best thing: I used AutoIT.

I will warn you that AutoIT is pretty much the Windows analog of AppleScript, without the cutesy pseudo English syntax. In other words, you will need to roll up your sleeves and get your hands a little dirty in order to put together a decent AutoIT script.

The payoff comes when you finish your work and compile it into a tight executable that you can share with your friends, allowing them to automate some complex series of button clicks and copy/paste operations.

I walked through the manual process of exporting an email to PDF and listed each action:

• Get the date, sender, and subject
• Create a filename based on date + sender + subject
• Launch the Print dialog
• Select PDFCreator
• Fill in the Document Title, Creation Date, and Subject in the PDFCreator dialog
• Fill in the full file path in the Save dialog

In addition, I wanted to make the script a little better by adding the following:

• Check that user has PDFCreator installed
• Verify that GroupWise is running and that the user has selected one or more messages
• Prompt the user for a target directory before processing the messages
• Sanitize the filenames by replacing illegal characters with underscores and truncating to meet maximum filename and path length in Windows
• Skip over files that have already been generated, quickly, so that one doesn’t need to worry about accidentally selecting messages that were already printed

There were other adjustments needed, but the process was the same: run the script, hit a problem, tweak the script a little to address the problem, and repeat.

Here’s a little bit of the AutoIT script:

You can see that it is a bit more intense than AppleScript, but remember that the full script wasn’t written in one go. I had a little short ten-line script that I kept tweaking as small problems cropped up until I had adjusted things to my liking.

Note that this is a GUI macro language. The machine starts clicking and typing away right in front of you and you probably shouldn’t interfere until your script finishes.

As of this afternoon, I have generated around 4,000 PDF documents for my email messages.

Searching All of Those Documents

This was the easiest part. These days there is an excellent tool available for searching documents on your desktop: Google Desktop. This product indexes every useful file on your desktop and provides a full Google search with a quick double-tap of the <control> key.

So you can enter a search like “Zebra bar code”

And the results look exactly like a Google web search, but it’s showing your desktop files. And you can see inline previews too.

Macintosh users can install Google Desktop as well, but all of these files should already be indexed and searchable by Spotlight.

Closing Thoughts

Whenever I reach for tools like this I feel a twinge of guilt—it’s outright hackery, isn’t it?

But there is a place for quick and dirty jobs in every workplace. I needed to get my files from one place to another, one time only. It just didn’t make sense to spend money or time on a more elegant solution.

Play around with each of these tools a little. Especially AutoIT—it’s a handy Swiss Army Knife to have at your disposal.

What do you do with 1,500 photographs?

In hope of sparing some folks hours of boredom I’d like to share my ideas on this topic here.

The Endless Stack of Photos

We have all been there. A friend or family member brandishes a stack of photos, saying “Let me show you photos of our trip to Ecuador…” (oh no… here it comes…). At this point, you reach for the photographs, but they hold the stack out of reach. They then turn each one over slowly, telling a long tale about every single image. “Oh, look at this monkey, it was so cute when he stole the candy out of little Billy’s hand and spit it into Aunt Sally’s hair.” and so on and so on.

You begin to look at the size of the stack and estimate how long this process will take.

Everyone has been on the receiving end of this treatment, but have you ever been the perpetrator?

It’s an easy trap to fall in to. To be honest, when you are showing photos to a friend, each image brings back a wave of pleasant memories and it is tempting to bask in the enjoyment of the memory, talking about how you felt at the time, as your friend’s eyes begin to aquire a glossy sheen.

Don’t Be That Person

Remember how it felt the last time you endured a four hour photo flipping marathon and have pity on those around you.

Here’s my strategy for pleasant photo sharing:

• Pare down the photo collection. Substantially.
• Create an attractive photo album with the finest photos of the lot.
• Hand the album to your friend and let them turn the pages.

Back to those 1,500 photos from India…

Here is one of the pages from the album I made from that trip.

As you can see, the end result is fairly simplistic, with a few very nice photos.

Paring Down the Stack

Even the most avid photographer understands that nobody wants to see a thousand photos. And don’t think that just because you made a slideshow with music instead of printing the photos that you are exempt. You can afford to cut the number down quite a bit.

Consider that a typical Hollywood motion picture contains less than 10 percent of the total footage filmed. Stanley Kubrick, ever the perfectionist, took this to the extreme with shooting ratios around 100 to 1. Following the analogy, in photography, it is quite reasonable to take dozens of photos for every single picture that you might share to others.

The real trick is deciding exactly how far to go with the selection process.

In my experience, you can weed out the bad photos for hours, and when you think the job is done, you can still go back and toss out a few dozen more.

I filter my photos in three major phases, using the five-star rating tool of my photo library software to help keep things in order. I personally use iPhoto, but any other good photo library suite should offer ratings and smart folders.

Phase 1: Removing obviously bad photos

This is a very quick pass through the whole collection. I start by selecting everything and marking all photos with a neutral rating of three stars.

I then find any photos that are underexposed or are blurry and give them one single star. Along the way, any photos that obviously have no useful content get the same treatment.

Phase 2: Identifying decent photos

I use a smart folder to show all photos with three stars or greater. This hides all of the junk from the first pass.

Now, I go through each photo and give it a deeper look. I sort them into three different stacks, giving two stars to anything that has useless or boring content and giving four stars to photos that I think are worth showing to people. Photos that don’t fit either description retain the neutral three-star rating. These are often repeats of the one good photo I tossed in the four-star stack.

Phase 3: Identifying the best of the best

I use a new smart folder to show only photos with four stars or better.

This is the hardest part. I go through the photos and try to find the absolutely best photo that expresses each experience or thought.

• If the photos are of places, then it makes sense to check that you have at least one photo of each important place you visited.
• If these are shapshots of friends and family, then you probably should verify that each person shows up in at least one of the pictures.

I often have a difficult time working through photo collections from visits to my wife’s family in Brazil: there are hundreds of people in the pictures and I often have doubts over who is family and who isn’t. Fortunately, my wife sits patiently with me and helps at this stage.

Look again at the photos of the elephant ride and the snake charmer. I probably have two dozen different shots of the snake charmers, while the elephant shot was a single dodgy photo taken by the tour guide. I was able to pick the very best snake charmer photo, but I had little choice with the other—there was no way I was going to omit a picture of me on an elephant so I used it. These are the kinds of tradeoffs we are dealing with.

You can see in this screenshot how I flagged most of the photos as three stars (I have already hidden all of the one-star dreck). There is one photo that has four stars, while the one next to it had an unappealing composition in my opinion, so it got two.

How far should you go?

I suppose this comes down to personal preference, but I like to keep things down to thirty or forty photos—from a starting point in the thousands. In a photo album, you can represent your whole trip in fifteen or twenty pages. This is far less intimidating than a big thick stack of photographs.

Of course, context is important. I will go out on a limb here and say that a new baby can be shown to co-workers in five photos.

In the end, I create a smart folder that shows only photos with five stars. And boy do they look good!

Tweak the best photos

My favorite tool of all for tweaking photos is the crop tool. A good crop can dramatically change the composition of a shot while still retaining the purity of the photo.

I will also straighten any slanting horizons and possibly fix funky light balance at this point. The tool set provided in iPhoto is quite adequate for these simple tasks.

Now create a slick photo album using any of the great tools available online.

Make the Photo Album

Again, I like using iPhoto. It allows you to easily create fancy albums using templates and so forth. Once you are done, you can buy a finished album with a few clicks.

Once you have your short list of photos, use a five-star smart folder as the source for the photo album. You can then spend a pleasant evening or two playing around with the layouts and composition and adding captions to your photos.

• Create a smart folder that shows only five-star photos
• Create a new photo album based on that smart folder
• Choose a pleasing layout
• Add your photos in varying page styles to the book
• Write some informative and/or witty captions for the photos

This is a screenshot of the iPhoto application, with the A-list photos along the top and the snake charmer page in the editing window.

You can even make albums like this online, without any editing software whatsoever…

This is a similar photo album creation tool that is run completely from the Winkflash website.

Now, finish picking out the features you want on your album (e.g. cover style) and place your order.
In my opinion, these albums come with the optimal number of pages (usually 20). Any more pages could make it tedious and boring. Fit your vacation into those 20 pages.

These books usually cost around forty bucks, but they are worth every penny.

The Finished Product

When everything is done, you will have a beautiful printed photo album that looks like you bought it at a book store.

I have seen photo albums from three different outfits up close.

MyPublisher

I have a couple of albums from these guys and they are near perfect. The pages look like thick magazine pages, with magazine-quality photos. I found that my white-on-black text bled a little.
Note that the leather used on the leather-bound books is paper thin.

These folks are always sending me coupons for 40% off, so it seems that you really don’t ever have to pay full price for their wares.

Apple iPhoto

These are identical to the MyPublisher books. Indeed, at one time iPhoto could send its output to either MyPublisher or Apple. I heard a rumor that MyPublisher did the books for Apple at one point.

The only down side to the Apple books is that you pay high shipping costs. Otherwise, the books are perfect.

Winkflash

A friend shared one of these with me. I happened to have my India album nearby, so we compared them. On the one hand, Winkflash is much cheaper; however, the image quality is not nearly as nice as the other books. Perhaps I was looking at a lower-end book from them, but the ink dots were a little coarse for my taste.

Summary

By putting a little effort in aggressive photo selection, basic image tweaks, and then taking advantage of the many photo book tools out there you can create a beautiful album that is a pleasure to leaf through.

These books have been available for several years; even so, whenever I hand one to a friend, they page through it, transfixed. People really love these albums and they actually enjoy looking through them.

Oh, and they make great gift ideas too!

What does this topic have to do with reducing paper in our lives? Believe me, printing one of these books is so much neater and cleaner than printing hundreds of loose photos. And you will enjoy them more.

As it turns out, my original script does not work in Snow Leopard, and I promised that I would one day clean up and publish my new and improved version.

Chris posted a comment today as a gentle reminder, so here is the new and improved version without further delay…

The Details

Unfortunately, Snow Leopard came around and caused some indigestion. For starters, the ScanSnap Manager didn’t work correctly and Abbyy Finereader would not process anything made by the ScanSnap. A couple of months later they got everything straightened out and delivered new versions of each product.

The new version of the Abbyy Finereader product does not play well with my original script.

Since I cannot do without this important functionality, I rolled up my sleeves and rewrote most of the script. The new version works in Snow Leopard quite nicely with one small annoyance: you really don’t want to try to use the machine for anything other than scanning or OCR while it is going because the new Finereader version keeps bouncing the darned icon all the time it is running and that is quite annoying to watch.

Fortunately, I really don’t need to use my machine for anything else while it is chewing on the docs; I just wanted to be able to continue scanning at the same time!

Note: Before going forward, note that you will need to upgrade the ScanSnap Manager and Abbyy Finereader to the Snow Leopard versions first! Get the files here.

Here is a link to the new script…

And here’s the code itself:

Installation

• Use the Script Editor to save this script as Run OCR on New Folder Items under User Home/Library/Scripts/Folder Action Scripts
  You may have to create the Folder Action Scripts folder.
• Now open a Finder window and navigate to the parent folder of your scanned documents folder.
• Right click (control-click) the scanned documents folder and choose Folder Actions Setup…
• At this point if folder actions are not enabled, you will likely have to enable them and add the script manually.
  • Check Enable Folder Actions
  • Use the “+” buttons on the left and right sides to add the scan folder and then this script.
• Otherwise, a list of scripts will come up. Choose this script from the Choose a Script to Attach dialog.
• Close all windows.

That’s it! The script will be invoked automatically every time a new file appears in your scanned documents folder.

Please let me know if you have any ideas that can improve this script. I’m not an AppleScript guru, so someone might just know how to keep that annoying Finereader icon from jumping.

Apparently this is not the case

It seems that the TSA has leaked their official document of airport security guidelines. ABC News says Online Posting Reveals a “How To” for Terrorists to Get Through Airport Security

A Rookie Mistake

Look at the screenshot of the document at the top of this post. Even though a certain part of the document has been blacked out, it is possible to select the text and copy/paste to find out what is hidden behind the black text.

What kinds of things are listed in this document?

• Photographs of all kinds of official ID cards. Ever wondered what a U.S. Senator’s ID card looks like?
• Procedures for calibrating equipment, such as where guns should be hidden for the testing and such.
• Guidelines for who gets searched and who doesn’t.
• Guidelines for what objects get searched and which don’t.
• And much much more!

In other words, this was a most unfortunate event.

See for yourself—ABC News (and others) have posted the document with redactions removed.

Easy as Pie

Here’s a screenshot of the original document, opened in Adobe Acrobat Professional.

As you can see, it was a trivial matter to use the TouchUp Object tool to gently slide the black rectangle off of the secret stuff (I have blurred the text here, though you can read it from ABC News if you wish).

If you are working with confidential documents that could potentially cause disaster if leaked, please learn how to redact your documents correctly!

I was reading a post on a board I frequent where a person was describing exactly this kind of activity—removing sensitive information from PDF documents. Several suggestions were made, but one individual suggested opening the file in Acrobat Pro and replacing the sensitive text with good old Lorem Ipsum.

It was at that moment that I recalled a peculiar feature of the PDF file format: it is designed to support nondestructive updates, allowing people to make vast changes to a PDF document while still retaining the original document, fully intact. I did a few experiments and was surprised with the results.

A Brief Note on the PDF File Format

For the geeky types among us, one place to begin is this article:

Portable Document Format: An Introduction for Programmers

The key points to get out of the article is this: A PDF document is comprised of several distinct sections, a Header, a Body, an “xref” Table, and a Trailer. At the very end of the file you will find the character sequence %%EOF

The PDF standard was designed to allow multiple updates to a document, while retaining the original version. This is accomplished by appending anything new to the end of the document, after the original EOF tag. The document will now have two EOF tags: one indicating where the original document ended, and a new EOF tag indicating where the new changes end.

If we wish to revert PDF changes, it should be a simple matter of opening the PDF file in a binary editor, searching for the first EOF tag, and deleting everything following.

A Simple Experiment

Let’s start with a proper secret document containing missile plans…

Suppose we want to obscure some special information in paragraph 37. We can open the file in Acrobat Professional and use its text editing features to swap in the venerable Lorem Ipsum text.

Here’s what it looks like after the switch:

You can see here that the first seven lines of text starting on paragraph 37 have been replaced with appropriate unreadable text.

Now, open the new PDF file in a binary editor (since PDF files contain a mix of text and binary, the editor must be a binary editor).

Note the %%EOF character sequence embedded in the text. This is the first EOF tag, indicating where the original file ended. All we need to do is place the cursor to the right of the EOF and delete everything to the end of the file.

Once we have done so, it’s like magic:

The edits that replaced lines of paragraph 37 with gibberish have neatly been undone!

More Details

From the PDF Intro document linked earlier:

“The trailer, it turns out, plays an important role in the way PDF implements incremental updating. The key concept to understand here is that a PDF file is never overwritten, only added to. That goes for all portions of the PDF file – even the trailer itself, and the end-of-file marker. In other words, a multiply-updated PDF document may contain multiple trailers – and multiple end-of-file markers! (There may be numerous occurrences of %%EOF.) Each time the file is edited, an addendum is written to the tail of the file, consisting of the content objects that have changed, a new xref section, and a new trailer containing all the information that was in the previous trailer, as well as a /Prev key specifying the byte offset (from the beginning of the file) of the previous xref section. The cross-reference info will then be distributed across more than one xref section. To access all of the cross-references, the reader must walk the list of /Prev keys in all the trailers, in reverse order.

Space doesn’t permit a detailed exploration of updates here, but you can find several examples in Appendix A of the PDF 1.3 specification (available at http://partners.adobe.com/asn/developer).”

Summary

It is important to understand that the PDF standard allows for appended updates to files that leave the original document intact, regardless of how drastic the changes are. If you are intent on redacting text from PDF documents, do not depend on simply deleting the secrets using a PDF editor—you must use a proper redaction tool that addresses these issues correctly.

That said, I did some experimenting with a few utilities (Apple Preview, PDFpen, and Adobe Acrobat Pro) and found that some write the file from scratch each time, with no lingering cruft from former versions, while others respect the original intent of the PDF standard. This means that you can’t trust that older revisions are being retained in your file and you can’t trust that they aren’t.

Be conservative: use a redaction tool for secrecy and proper backups for versioning.

[UPDATE: I spoke too soon—they only delivered half of the goods. See below.]

[UPDATE 2: Hurray! It's fixed! The birds are chirping and the sun is shining and life is good!]

When Snow Leopard came out back in August, I ordered my copy the first week and was so excited that I installed it the day it arrived. My joy was short-lived, however: the most important software package I use did not work!

I depend greatly on the OCR capabilities of the ABBYY FineReader software that comes with the ScanSnap scanners, and this was one of the many pieces of software that did not smoothly transition to Snow Leopard. I could scan documents, with limited functionality, but the OCR feature did not work.

Now that Fujitsu has released their official update, I will probably be installing Snow Leopard tomorrow evening. Now, do I do an upgrade or a full install? Hmmmm…

UPDATE

Well, they only delivered half of the goods.

I was reading through the seven-step process for updating the ScanSnap drivers and I arrived at step seven:

Step 7: The download for FineReader for ScanSnap update to Snow Leopard will be hosted by ABBYY but is not yet available. If you have already subscribed to be notified by Fujitsu regarding the Snow Leopard updates, an email will be sent to you when it is posted.

How displeasing. The only thing I really cared about was getting the OCR to work, and apparently Abbyy has not yet delivered their part (How hard can it be to update the part in your code that says “If the PDF metadata doesn’t match X then the document isn’t a ScanSnap doc” ?).

UPDATE 2

Instead of making us wait another month or two, ABBYY has delivered their patch in record time . I quickly installed the update and tested my ScanSnap script…

There are a few kinks to work out, but it is pretty clear that I will have my same old workflow AppleScript folder action up and running in short order. I will post the newer script as soon as I have it running properly.

And, as a side note, I had already purchased PDFpen from SmileOnMyMac as a backup plan. Their tool incorporates the OmniPage OCR engine, an engine that rivals that of ABBYY. My script was already running with PDFpen, but there were some issues with tables in documents that I forwarded on to SmileOnMyMac. One of these days I’ll post my scripts for PDFpen.

As it would turn out, there were several thousand other documents on a file server that shared the same funny behavior. By the time we were done struggling with this problem I had gained new respect for PDF corruption issues and their prevention.

The Problem

We were looking to load a few thousand of these scientific reports into a fancy-schmancy new database, with linguistics searching and other bells and whistles. Much to our chagrin, these documents just weren’t loading, and we couldn’t understand why. They were text documents, with some embedded images, but mostly straightforward text.

Here is an excerpt:

And you can tell that it is right and proper text because when I blow it up all the way, the fonts are nice and smooth—this isn’t just an image of text.

But if I copy and paste that particular paragraph into any handy editor (Notepad, in this case), this is what I see:

And as far as I know, at this point the actual text is beyond the reach of average folks like me. We tried, believe me we tried.

What went wrong?

A quick Google of the subject led us to understand that many PDF generation tools embed subsets of fonts, with nonstandard mappings from the text to the font.

This fellow explains it nicely:

“The PDF file does not contain all the information to extract the text. The problem is that a character in a PDF file may not contain information what “real” character it relates to. Some PDF generators do a pretty bad job when they embed fonts into PDF files. They use a proprietary encoding mechanism (e.g. 1 is A, 2 is B, 3 is C, …) in both the embedded font and when they place glyphs on the page. Without a table that implements the reverse (e.g. character code 1 is ‘A’) you cannot extract text from such a file.

There is nothing you can do (besides to complain to whoever created the PDF file, and the author of the software that created this file).”
— from khkremer on experts-exchange.com

As it would turn out, many of the reports had been generated by printing to Adobe Distiller from Microsoft Word. It would seem that the default settings used for Distiller included the “totally hose my document content” switch.

The Solution

We fretted over this quite a bit. These are important scientific reports, and there is no way to easily ungarble them. We finally ended up contacting the Abbyy Finereader folks and trying out their OCR toolkit for Linux: not only did this product make fast work of running optical character recognition on the sample document, but once we had a script running, we managed to blow through the 10,000 pages the trial license gave us, in a day or two.

Imperfect, at best

I am happy that we were able to salvage the bulk of the electronic knowledge found within those thousands of files, but our work barely scratched the surface.

For example, most of these documents have rich bookmarking of sections and keywording, such as this (content tastefully blurred on purpose).

In addition, scientific documents typically have loads of tables full of numbers. Though it is possible to mine this data with a good OCR tool (the FineReader API provides tools for just this purpose), the tables are far more difficult to extract correctly once the original text information is lost.

Final thoughts

I wrote a few weeks about document formats, mentioning the PDF/A document standard. This is worth investigating, regardless of what your document needs are.

If our thousands of files had been originally generated as PDF/A, it is certain that we would have been able to copy/paste from them without problem: PDF/A prohibits such font shenanigans as were perpetrated on our garbled reports.

In the end, our OCR sledgehammer approach worked like a charm, and is probably sufficient for our needs. Text mining is a pretty slushy business, so no-one will complain if there are a few typos on each page—if they find the doc in a search, they can print it and read it the old fashioned way.

Over the past couple of years I have considered several different approaches for keeping a grip on which computers had which service patch, which router is getting flaky, and which cable connects the library to the classroom at the end of the hall.

I have tried Excel spreadsheets, an Access database, even a spiral-bound notebook—none of them made the job any easier. A few weeks ago I thought about trying a Wiki and this has turned out to be a perfect fit!

If you are looking to keep a loose scrapbook of notes with lots of arbitrary categories and relationships between them, a wiki might do the trick. In this article I’ll cover two simple freeware wikis you can carry around on a thumb drive.

What’s in a Wiki?

All of us have used Wikipedia at one time or another, and though it may be regarded with disdain by high school teachers, when you consider how it works, Wikipedia is an amazing achievement. But what is the nature of a wiki?

One of the key features is that any page can be easily edited at any time (of course this can be limited by permissions). Another attribute is the ability to breathe life into a new page just by calling its name.

Between these two features, you get the essence of wiki-ness.

For example, if I have a page that discusses North American bears, I can type in a list of bears in a special format, often in jammed-together Wiki Words, like this:

• GrizzleyBear
• BlackBear
• BrownBear

As soon as I save the page, those bear names become hyperlinks. Even though I haven’t written any pages about the individual bears, whenever it finally suits me, I can click on BlackBear and accept the invitation to “Create a new page called BlackBear”

Better still, a friend who knows about black bears might click on BlackBear and write a beautiful page about the animals.

That’s what wikis are all about.

Back to the School Computers

In a matter of minutes I was able to make a page that described the building and listed the various rooms in the building. I was able to then click on each room and “auto-vivify” a page for the room.

From that point, it was easy to create custom pages for each computer in the building, with each page listing the machine’s stats. I also created pages for each network switch or router.

In a matter of two or three evenings I had the skeleton of a solid knowledge base populated—it’s a pretty fancy looking web site with dozens of pages that took little effort to put together.

Last night I noticed that one of the machines wasn’t connecting to the Internet, though it connects fine to internal servers. I popped open its page on the wiki and added a simple note at the bottom of the page:

2009-10-11 - This machine isn't able to connect to the Internet. Not sure why. It connects fine to internal servers.

A few weeks ago I replaced a fan in a network switch. An easy annotation on the wiki page for that device.

Personal Wikis

There are many uses for personal wikis, mostly centered around personal knowledge management and personal information management. People use wikis as a replacement for time and task management tools, as a place for gathering thoughts, as a sort of amorphous database, and many other things.

There are many different personal wikis available—here’s a short list of free ones. One nice simple wiki to try is TiddlyWiki. If you are looking for something with a bit more substance, you can try a portable version of MediaWiki—the engine behind Wikipedia—that runs off your thumb drive.

TiddlyWiki

This afternoon I downloaded the flyweight portable wiki called TiddlyWiki. This is an amazingly tight little application—it comes in the form of a single fat web page that you copy to your thumb drive. As you make edits to your TiddlyWiki, the single html page is saved with your changes. Since it’s a single fancy file, backups are dead easy.

Here’s what it looks like when you first launch the “empty.html” file:

After a half hour of twiddling around, I had thrown together this basic set of “Tiddlers”

In this screen shot you can see that there are now links that bring up custom “Tiddlers” for each computer and for each room. I have opened one of the little pages for Computer21.

They describe these pages as being comparable to note cards. All in all, it is tight and easy to use.

Want to give it a try? Download it from the TiddlyWiki site. You really need to play with it to get a feel for what it can do!

MediaWiki

If you are looking for something with a little more meat on it, you can run the Wikipedia engine on your USB drive.

The easiest way to set this up is to let MoWeS do everything for you. MoWeS stands for Modular Webserver System. It’s a free product that you can configure as a self-contained Apache web server with a variety of cool apps like MediaWiki, running off a thumb drive.

Here’s how to set up MediaWiki in five minutes:

• Go to the MoWeS Mixer
• The first time around choose “I do not have a MoWeS Portable II Package and want to obtain a new package” when prompted and click Go.
• On the software lists, check Apache2, MySQL5, PHP5, and MediaWiki
• Click Download Now
• At this point they ask you some kind of question in German, to filter spambots, but it seems to be a simple math problem. Fill in the answer and click Submit Query
  (“Zum Schutz vor Downloadrobotern geben Sie bitte das Ergebnis dieser Aufgabe ein: 5 + 8 = ?“)
• Unzip the downloaded zip file, mowes_portable.zip, and copy the files to your USB drive
• Open your thumb drive and double-click mowes.exe
• Select your language and accept the license
• Click install, and confirm when prompted

The installation process may take several minutes, but rest assured that it isn’t installing anything on your computer.

Note: I received two or three firewall warnings for the Apache web server and the MySQL database. I had to click the “Unblock” button for all of them before my new MediaWiki-on-a-stick would work correctly.

After all of the dust settled, I have this little window on my screen:

In order to shut down and close out, just click the End button.

Once your MediaWiki USB key is running, you can go to this web page:

http://127.0.0.1/mediawiki/index.php/Main_Page

It looks just like Wikipedia, doesn’t it?

What a truly amazing thing: you can carry around your own Wikipedia server on a USB key and plug it in any random machine and start it up.

Different Wiki Features

As you try out different wiki software, you will notice that there are plenty of differences in the features they support:

• Each wiki has a different kind of editor. Some are visual; others are simple text editors.
• The markup syntax you use for pages is different from wiki to wiki.
• Most wikis support features such as “category pages” that find all pages tagged with a category.
• Some support adding images and other content; others don’t. I imagine that TiddlyWiki probably has some means of embedding images, but I couldn’t find it.
• A quick glance at the MediaWiki screenshot above shows extended features such as the Discussion tab and the History tab.
• Some use the filesystem for their pages; others use a database.

Since I wanted a central wiki for the whole school, I chose a different product from the portable wikis I discussed here—I decided to run MoinMoin on a Ubuntu installation on an aging Gateway desktop machine. Nevertheless, the basic idea is still the same.

Once that arrangement becomes a little more stable I’ll write up a howto document, like the Linux NAS one from a few months back.

Other Sources

There are loads of different personal wiki options out there and many people have written how-to documents and tutorials. Here’s a few:

• Run Your Personal Wikipedia from a USB Stick (Lifehacker.com)
• Geek to Live: Set up your personal Wikipedia (Lifehacker.com)
• Wiki On A Stick (PmWiki.org)
• Getting Started with Wiki on a Stick (About.com)
• TiddlyWiki for the rest of us (giffmex)