Whiteboard Capture — Charles “Claw” Law

Background

The goal was to make it easy to capture whiteboard drawings. I find that drawing digitally can get in the way of the flow of ideas: the tool can slow you down, or you can get distracted polishing what is on the screen. For many people, pen and paper or a whiteboard is the easiest way to brainstorm and be creative.

I had both projects close to production quality before my children were born, when I paused work on them. Around that period, Microsoft released Office Lens to address a similar problem.

Cleanup

The first tool cleans up a whiteboard photograph. The original version followed a Microsoft Research paper. I subsequently rewrote every step using different techniques. The revised tool is equally fast and handles glare better.

The output can be a cleaned-up bitmap or a vector representation of the outline of each mark on the whiteboard.

Whiteboard capture interface showing the photographed drawing and its cleaned-up version — The original cleanup interface.

Vectorized outlines of the handwriting and diagrams from a whiteboard photograph — Vector output. Open either image to inspect it at full size.

A later publication of the related research is available from Microsoft: Whiteboard Scanning and Image Enhancement.

Recognition

The second tool estimates the original marker strokes from an image. The Jurassic Park lettering below illustrates the difference between the outline of a mark and the path a pen would take to draw it.

Show reconstructed marker strokes

Hover over the logo or select the checkbox to show the estimated strokes in red. The checkbox also works with touch and keyboard input.

Finding the outline of the letter “I” is relatively straightforward: it is the boundary of the white rectangle. But when someone writes an “I”, they draw a vertical line, not a rectangle. Recovering that line is what the recognition tool does.

The problem becomes harder at intersections. The “K” has several, and the outlines of neighboring letters can touch. The tool has to work out which paths correspond to the original strokes through those intersections.

From photographs to editable diagrams

Once the original marker coordinates have been estimated, they resemble the coordinates produced by touch gestures. That makes it possible to use tools designed for touch input with a photograph.

I connected the stroke-recognition output to GrafPad, a project I worked on that converts touch coordinates into shapes. With that combination, I converted a photograph of a whiteboard block diagram into a PowerPoint containing the same diagram as editable shapes.

The proof of concept worked, though it still needed more polishing.