Skip to content

scdenney/open-science-skills / codex/vlm-ocr

vlm-ocr

Verified

OCR scanned or image-only corpora with vision-language models in four phases: `triage` inventories the scans, identifies script and language, and shortlists models by script, hardware, and data constraints; `evaluate` picks a system by measured CER/WER against ground truth, using born-digital twins, a second reader, and decision rules set in advance; `run` builds the production pipeline; and `clean` removes model artifacts, audits what every cleaning rule deletes, corrects with rule-based and LLM passes, and records provenance. Born-digital documents with a text layer go to $doc-to-markdown; runs on Leiden's ALICE cluster add $vlm-ocr-alice.

FileKindSize
SKILL.mdskill md76.5 KB
agents/openai.yamlconfig221 B
references/model-routing.mdmarkdown11.3 KB
references/prompt-and-schema.mdmarkdown13.3 KB
references/prompt-templates-and-schema.mdmarkdown9.0 KB
scripts/ocr_client.pyexecutablescript15.5 KB
scripts/triage.pyexecutablescript14.8 KB

references/prompt-and-schema.md

View on GitHub
# VLM-OCR Prompt and Output Schema Reference

Worked examples of a language-specific transcription prompt and a per-page JSON output schema. Use as a starting point; adapt to your corpus, model, and language.

---

## (a) Language-Specific Transcription Prompt — Cyrillic (Pre-Reform Russian, 18th–19th c.)

The following prompt is designed for pre-reform Russian (before the 1918 orthographic reform), where the letters ѣ (yat), і (decimal i), ѳ (fita), and ѵ (izhitsa) appear alongside the modern Cyrillic alphabet. It enforces orthographic fidelity to the source, blocks modernization, and forces the model to mark uncertainty rather than guess. Levchenko (2025) documents that GPT-4o-class models insert archaic characters in ~59% of 18th-century Russian files *even when not prompted to do so*; this prompt does not eliminate that failure mode, but combined with stratified ground-truth validation (see SKILL.md, `run` phase §6) it at least prevents the opposite failure (silent modernization).

```
You are transcribing a scanned page from a pre-reform Russian printed
document. Your sole task is faithful transcription. Do not interpret,
translate, summarize, or normalize.

ALPHABET AND ORTHOGRAPHY
- Transcribe the exact characters visible on the page.
- The document uses pre-1918 Russian orthography. The following
  characters may appear and must be preserved when present:
    ѣ (yat), і (decimal i), ѳ (fita), ѵ (izhitsa),
    ъ at word ends, and archaic ligatures.
- Do NOT substitute modern equivalents (e.g., do not replace ѣ with е,
  і with и, ѳ with ф, or drop terminal ъ).
- Do NOT add any pre-reform character that is not visibly present on
  the page. If the page uses modern orthography, transcribe it as
  modern. Never "historicize" based on context or expectation.

LAYOUT
- Preserve reading order (top-to-bottom, left-to-right, columns in
  order).
- Mark layout features inline:
    [HEADING] ... [/HEADING] for titles and section heads
    [FOOTNOTE n] ... [/FOOTNOTE] for footnotes, numbered as printed
    [TABLE] ... [/TABLE] for tabular material (preserve row breaks
      with newlines and column breaks with tab characters)
    [FIGURE: brief literal description] for illustrations; do not
      invent captions
    [PAGE_NUMBER: n] for printed folio or page numbers
- Preserve hyphenation at line ends as printed. Do not silently merge
  across line breaks.

UNCERTAINTY
- If a character, word, or span is illegible, damaged, or genuinely
  ambiguous, wrap it in [UNCERTAIN: best_guess] tags. Do not guess
  silently. Do not produce a character you cannot see.
- If an entire region is unreadable (tear, stain, binding shadow),
  mark it [ILLEGIBLE: approximate_length_in_characters].

NEGATIVE CONSTRAINTS
- Do not translate to modern Russian or any other language.
- Do not modernize spelling, punctuation, or typography.
- Do not add commentary, explanation, headers, or markdown outside
  the tags defined above.
- Do not hallucinate text for blank pages, illustration-only pages,
  or pages containing only a page number. For those, return only
  the appropriate layout marker.

OUTPUT
Return the transcription as a single UTF-8 string using the layout
markers above. Do not wrap it in code fences. Do not add a preamble
or postscript.
```

Adaptations for other scripts follow the same structure: enumerate the expected character set, forbid substitution in either direction, define a fixed set of layout markers, and require explicit uncertainty tags. For English historical documents, enumerate the long s (ſ), ligatures (æ, œ, ct, st), and manicules. For Chinese, distinguish simplified from traditional, list expected variant characters, and instruct the model on handling mixed hanzi / hanja / kana and punctuation (。、「」).

---

## (b) Per-Page JSON Output Schema

The VLM returns the transcription as a string. A thin post-processing wrapper parses the inline layout and uncertainty markers into the structured record below. Store one record per page; aggregate into per-document and per-corpus reports as described in SKILL.md, `run` phase §7.

```json
{
  "page_id": "doc-00042_p017",
  "document_id": "doc-00042",
  "page_index": 17,
  "text": "[HEADING]О воспитаніи дѣтей[/HEADING]\n\nВоспитаніе [UNCERTAIN: юношества] есть ...",
  "confidence": 0.86,
  "uncertain_spans": [
    {
      "start_char": 48,
      "end_char": 58,
      "surface": "юношества",
      "best_guess": "юношества",
      "reason": "faded_ink"
    }
  ],
  "layout_markers": [
    {"type": "HEADING", "start_char": 0, "end_char": 28},
    {"type": "FOOTNOTE", "number": 1, "start_char": 412, "end_char": 530}
  ],
  "flags": {
    "handwritten": false,
    "low_resolution": false,
    "table": false,
    "figure": true,
    "blank": false,
    "page_number_only": false
  },
  "pipeline": {
    "model": "Qwen3-VL-32B-Instruct",
    "model_revision": "sha256:a1b2c3...",
    "quantization": "bf16",
    "dpi": 300,
    "prompt_id": "cyrillic-pre-reform-v1.2",
    "generation": {"temperature": 0.0, "max_tokens": 4096, "seed": 42},
    "software": {"transformers": "4.48.0", "vllm": "0.7.2"},
    "hardware": "1x A100-80GB",
    "timestamp_utc": "2026-04-18T14:32:11Z"
  }
}
```

Field notes:

- **Offset convention:** all `start_char`/`end_char` values are 0-based Unicode-codepoint offsets into the **marker-stripped** transcription (the `text` field after `[UNCERTAIN: ...]` and layout tags are removed), with `end_char` exclusive. Compute them during parsing, as the tags are stripped — offsets into the raw tagged string would shift with every tag.
- `confidence` is a model-reported or heuristic per-page score on [0, 1]; document the source in the pipeline record (self-reported log-probabilities, dictionary hit rate, or a learned classifier).
- `uncertain_spans` is the parsed form of `[UNCERTAIN: ...]` tags; always populated (empty array if none). `reason` is optional but encouraged (`faded_ink`, `torn`, `bleed_through`, `ambiguous_character`, `damaged_scan`).
- `layout_markers` is the parsed form of layout tags; downstream diagnostics consume this to distinguish body text from tables, footnotes, and figures.
- `flags` is a closed set of page-type indicators used by the diagnostic stage (SKILL.md, `run` phase §4) to route pages (OK / rule-fixable / LLM-fixable / manual review).
- `pipeline` carries every parameter needed to re-run the exact page through the same model; it is the minimal unit of machine-readable reproducibility metadata (SKILL.md, `run` phase §7).

---

## (c) Why This Schema Matters for Reproducibility (DA-RT)

The Data Access and Research Transparency (DA-RT) framework treats reproducibility as a first-class methodological obligation: other researchers must be able to verify, re-run, and build on the corpus. A structured per-page record — rather than a flat `.txt` file — makes this obligation operational in three specific ways.

1. **Machine-readable provenance.** Every page carries the exact model revision, quantization, DPI, prompt identifier, and generation parameters used to produce it. Barrie, Palmer and Spirling (2025) show that LM-based pipelines in political science often fail the most basic replication tests because this metadata is not captured; pinning revisions and recording generation parameters is their explicit recommendation. The schema embeds it per page.

2. **Separable quality signal.** `uncertain_spans`, `flags`, and `confidence` are first-class fields, not prose annotations. A downstream auditor can compute corpus-level statistics (fraction of pages flagged `low_resolution`, mean span-level uncertainty density, share of pages with tables) without re-parsing the transcription. This is what allows the corpus-level batch summary in SKILL.md, `run` phase §7 to be produced mechanically, and what lets van Strien et al. (2020) style downstream-task impact assessments proceed without rerunning OCR.

3. **Cleanly composable with the next phase.** The `clean` phase consumes `text`, `uncertain_spans`, and `flags` to route pages to the appropriate correction strategy (rule-based, constrained LLM decoding, or manual review) and records its own provenance in a parallel structure. `$methods-reporting` consumes the `pipeline` record directly into the methods section and replication archive. Without a structured schema, each downstream stage would need to re-parse free-text output and reconstruct provenance by hand, which is both error-prone and a barrier to pre-publication audit.

The schema itself is not a standard — adapt field names and add domain-specific fields as needed — but the principle is: every piece of information a re-runner or auditor would need should be a named field, not a convention buried in a text file.

---

## (d) Book-Page Prompt with a Page-Type Tag and a Box Delimiter (production, 2026)

This prompt pattern transcribed about 36,600 pages of South Korean school textbooks (1950s–2010s) with Qwen3.6-35B-A3B-FP8 in October 2026. The Korean text itself was arrived at in three rounds of a gate (SKILL.md, `evaluate` phase). It returns plain text with one tag line rather than JSON. The client (`scripts/ocr_client.py`) strips the tag into a `page_type` field and stores everything else per page, as in (b).

```
You are transcribing one page of a {LANGUAGE} {DOCUMENT TYPE}.

Line 1: exactly one tag of the form [PAGE_TYPE: X], where X is one of
- body: any page with content (running text, sidebars, source boxes, exercises, chapter openers, the preface)
- toc: table of contents
- front: cover, title page, author lists, editorial notes
- back: colophon, index, bibliography, answer keys, back cover
- blank: no text

Then transcribe all text on the page exactly as printed. The text contains {SCRIPTS, e.g. "Korean (hangul) and classical Chinese characters (hanja)"}. Do not translate, summarize, or correct anything.

This is a historical document, printed between {FIRST YEAR} and {LAST YEAR}. Reproduce the spelling character for character as printed, even where it differs from modern standard {LANGUAGE}: {6-10 REAL EXAMPLES FROM YOUR PAGES OF OLD SPELLINGS, NAMES, NUMERALS}. Never replace a printed spelling with its modern form, never correct what looks like a mistake, and never add characters that are not printed. Ignore letters from the facing page that show at the inner or outer edge of the image.

Order:
1. The main text column, in reading order. Headings printed in the margin beside a paragraph are headings of the main text: put each on its own line at the start of the paragraph it labels. Other marginal notes go after the BOXES line.
2. A line containing only === BOXES ===
3. Each sidebar, marginal note, source box, activity box, and figure or map caption, each as its own paragraph.
If the page has no boxes or captions, omit the === BOXES === line.

Omit running headers, running footers, and page numbers. For maps, diagrams and figures, transcribe only their title and caption; do not list place names, legend entries or labels. Transcribe tables as markdown tables. Use markdown headings only for headings printed on the page. Output only the transcription, with no commentary.
```

What each rule is for, and what it did not fix:

| Rule | Why | Residue to clean or report |
|---|---|---|
| Page-type tag | Assembly keeps `body` pages only, so tables of contents, colophons and indexes do not enter the corpus text. A blank page without the tag invites invented text: a small model wrote four characters of an unrelated script on a blank Polish page. | Check the tag distribution per book (body share well under 80 % means a misread book). |
| Main text, then `=== BOXES ===`, then boxes | A sidebar or caption never lands inside a main-text sentence. | Models sometimes emit a second delimiter; strip extras. |
| Spell as printed, with real examples from the pages | Without it, the model rewrote 1957 loanword spellings in modern form (폴투갈 → 포르투갈, 이태리 → 이탈리아, 세째 → 셋째): 69 modern forms against 27 printed ones on 48 sample pages. With the rule and examples: 42 against 45, and most of the remaining "modern" hits were what the books print. CER against a modern reference text cannot see this error. | A few one-syllable slips per old book remain; state it as a limitation. |
| Ignore the facing page | Single letters from the facing page bled into the box section. | A post-processing filter for single-letter lines at the page edge, audited so that it spares vertical quotations printed one character per line (SKILL.md, `clean` §2). |
| Maps: title and caption only | Models list map labels as text and once invented a legend. | Drop box paragraphs that are lists of short labels. |
| Margin-heading rule | 1970s–80s books set section headings in the margin; without the rule the model filed them as boxes. On one such book's sample pages it cut disagreement with a second reader from 0.18 to 0.04. | — |
| Omit running heads and page numbers | Removes most of them. | They survived on 8 of 12 reviewed pages when not the first or last line; post-processing must remove recurring heads (see SKILL.md, `clean`). |

Adapting it: name the scripts that occur, including secondary ones (telling a model the script raised its accuracy on Han characters by more than 20 points in GlotOCR Bench, arXiv 2604.12978). Take the spelling examples from your own sample pages, never from another period. Keep the page-type categories stable across prompt versions so the assembly code does not change. Store the prompt's hash with every page record.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall