Search a shelf of handwritten notebooks

The idea

A desktop app that takes a folder of scanned or photographed notebook pages and turns it into a searchable archive. It detects page edges, deskews, groups pages back into the notebooks they came from, runs handwriting recognition locally, and indexes the text. You type "the guy from the Lisbon conference" and get the page image, with the matching line highlighted in place.

The transcript is explicitly an index, not a replacement. Every hit opens the photograph of the page, because handwriting recognition at 85% accuracy is useless as a document and excellent as a search key.

Why build this

People with a decade of paper notebooks have a dead archive. They know the idea is in there, and they will flip pages for twenty minutes rather than give up — which is exactly the behaviour that says the problem is worth money. The existing options are bad: Evernote and OneNote will OCR handwriting but demand you live in their app and upload everything; Google Keep handles a photo at a time; dedicated scanners ship software that produces unsearchable PDFs.

What changed is that handwriting is no longer a research problem you need a cloud service for. A 7B vision-language model running on a laptop GPU now transcribes a page of messy cursive about as well as the paid cloud APIs did two years ago, and it does it with no per-page cost and nothing leaving the machine. That last part matters more than it sounds: notebooks contain therapy notes, salaries, medical details, and other people's secrets, and "it never leaves your laptop" is the only acceptable answer for a large slice of this audience.

Stack sketch

• Tauri 2 with a Svelte front end. A small native app that owns a local folder is the right shape here; a web app would mean uploading gigabytes of scans.
• Rust core for ingest, the database, and search: rusqlite with FTS5 for text, notify for watching the scan folder, image plus imageproc for page detection, perspective correction, and binarisation.
• A Python sidecar bundled with PyInstaller for the model work, talking to the Rust side over a local Unix socket. Qwen2.5-VL 7B through Ollama for full-page transcription, with TrOCR as the small fallback on machines that can't hold the bigger model.
• Page-level JSON from the model: lines, bounding boxes, and a per-line confidence, so the UI can highlight a hit on the image and grey out lines it doesn't trust.
• ocrmypdf in the same sidecar to emit a per-notebook PDF with an invisible text layer, so the archive is useful even if the app disappears.
• Scans arrive from whatever the user already has: a phone camera roll synced into the watch folder, a ScanSnap, or a flatbed. No capture UI in v1.

Scope for v1

In:

• Watch a folder, deduplicate by perceptual hash, and group pages into notebooks by EXIF timestamp clustering with manual override.
• Background transcription queue with visible progress and a hard pause, because this will peg a GPU for hours on a first import.
• Keyword search with image-anchored results, plus filters by notebook and date range.
• An inline correction editor: fix a line, and the fix goes to the index and is never overwritten by a re-run.
• Export: per-notebook searchable PDF and a Markdown file per page.

Out: live camera capture, sync between machines, tags and notebooks-as-folders, summarisation, any cloud account, and anything but Latin-script handwriting.

Where it could go

The natural second step is semantic search over the same corpus — fastembed for local embeddings, sqlite-vec for the index — followed by question-answering across notebooks with a local model. "What did I decide about the house in 2019?" is the query people actually want, and it tolerates imperfect transcripts far better than keyword search does, because retrieval only needs the gist.

The commercial path is professions that keep paper logbooks under obligation: field biologists, clinicians, maintenance engineers, lab notebooks that need dated, witnessed pages. They have the same archive problem plus a retention requirement, and they will pay per-seat for an on-premises version with audit logging and tamper-evident export. The consumer app is the proving ground and the funnel.

Watch out for

Accuracy on genuinely bad cursive is the whole product risk, and the honest mitigation is UX rather than modelling: always show the page, never present a transcript as the document, and surface per-line confidence so the user calibrates their trust instead of discovering the limits by being wrong. Check the model licence before you ship — several of the strongest open vision models carry terms that restrict commercial use or require attribution, and the one you benchmark on may not be the one you're allowed to bundle.