TinyDoc-VLM
Grounded document extraction SDK — schema-valid JSON, evidence boxes, confidence scores. Runs locally on free models. No API key.
🔥 Highlights
Beats the Free Competition
Field F1 0.870 on held-out SROIE receipts vs 0.376 for PP-OCR + heuristics and 0.227 for Tesseract + regex — measured on the same 100 documents.
Evidence for Every Field
Each extracted value ships with the OCR span and bounding box it came from — human-verifiable provenance, coverage 79.8%.
Schema-Valid Output
JSON Schema validation + sanitization on every result (validity 1.000 on eval), plus a calibrated confidence score per field.
Engine Ladder
Free ollama:qwen2.5vl:3b by default, quality tiers up to 7B, ONNX and OCR-fallback engines behind one interface.
One pip Install
pip install tinydoc — Python SDK with Pydantic-typed results, CLI, and a local Gradio demo.
Open & Free
Apache 2.0 license — free for commercial use. Local-first: no API key, no cloud, no telemetry.
🚀 Quick Start
pip install tinydoc
# Requirements: a running Ollama with qwen2.5vl:3b (free, local)
from tinydoc.pipeline import ReceiptPipeline
pipe = ReceiptPipeline("auto")
doc = pipe.extract("receipt.jpg")
# Schema-valid fields + confidence
print(doc.fields, doc.schema_valid, doc.confidence)
# Evidence: OCR span + bbox behind every value
for fr in doc.field_results:
print(fr.name, fr.value, fr.evidence["quote"], fr.evidence["bbox"])
# Or from the shell
tinydoc extract ./receipts/ --out results.jsonl --overlay overlays/
🏗️ How it works
Receipt / invoice image
↓
Engine (ollama:qwen2.5vl:3b — free & local, or 7B / ONNX / OCR-fallback)
↓
Raw model output → JSON extraction → collapse repetition → sanitize
↓
JSON Schema validation ── schema_valid: true/false
↓
Evidence anchoring (PP-OCR word boxes → best span + bbox per field)
↓
Confidence (evidence quality × schema × field rules)
↓
DocumentResult — fields + per-field {quote, bbox, score} + confidence
📊 Benchmarks (measured)
| Benchmark | Metric | Score | Status |
|---|---|---|---|
| SROIE receipts — TinyDoc pipeline (qwen2.5vl:3b) | Field F1 | 0.870 | ✅ n=100, committed artifacts |
| SROIE receipts — PP-OCR + heuristics baseline | Field F1 | 0.376 | ✅ same 100 docs, same scorer |
| SROIE receipts — Tesseract + regex baseline | Field F1 | 0.227 | ✅ same 100 docs, same scorer |
| Schema validity (pipeline, SROIE eval) | Rate | 1.000 | ✅ n=100 |
| Evidence coverage (RapidOCR, post A/B) | Rate | 0.798 | ✅ n=100 × 4 fields |
| FUNSD transfer probe (form → receipt-shaped keys) | Field F1 | 0.352 | ✅ n=50, nonstandard gold mapping |
| OCRBench — TinyDoc-VLM 256M research checkpoint | Accuracy | 0.0% | ⚠️ n=1000 — model retired from claims |
| DocVQA / CORD / PubTabNet | — | — | Not measured |
Every number above recomputes from committed artifacts in
evaluation/phase0/results/. The 256M checkpoint's earlier
benchmark figures were never measured and have been withdrawn — see
docs/BENCHMARKS.md.
🔗 Ecosystem
🐍 PyPI
pip install tinydoc — pypi.org/project/tinydoc
🤗 Model Hub
Research checkpoints (archived — see benchmarks) — eulogik/TinyDoc-VLM-256M
🤗 Space Demo
Try it live — eulogik/TinyDoc-VLM Space
📖 Documentation
Full docs — GitHub README
📄 Paper
Technical report — coming soon
Follow — @eulogik