GodModeAI2025/SkillSafeWerkstatt / query-llm-wiki
query-llm-wiki
UnverifiedSearch, filter, inventory, and answer from a verified SkillSafeWerkstatt release with claim-level source locators and read-only quality assessment. Use whenever the user asks about this wiki, requests original sources or citations, compares current and historical claims, filters metadata, inspects schema drift, or checks wiki quality without modifying it in Claude Code, Claude Cowork, or Codex.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.2 KB |
| evals/evals.json | config | 19.1 KB |
| references/answer-contract.md | markdown | 46.1 KB |
| references/query-filters.md | markdown | 2.6 KB |
| scripts/assess_quality.py | script | 1.8 KB |
| scripts/bm25.py | script | 8.2 KB |
| scripts/describe_actions.py | script | 4.4 KB |
| scripts/freshness.py | script | 3.6 KB |
| scripts/frontmatter_contract.py | script | 10.0 KB |
| scripts/identity_status.py | script | 6.0 KB |
| scripts/inventory_wiki.py | script | 6.5 KB |
| scripts/quality_core.py | script | 15.9 KB |
| scripts/search_core.py | script | 221.1 KB |
| scripts/search_wiki.py | script | 4.8 KB |
| scripts/sync_artifacts.py | script | 14.2 KB |
| scripts/trust_contract.py | script | 5.2 KB |
| scripts/verify_release.py | script | 12.1 KB |
| scripts/wiki_filters.py | script | 6.6 KB |
| scripts/wordforms.py | script | 56.0 KB |
scripts/search_core.py
View on GitHub#!/usr/bin/env python3
"""Shared, read-only search core of the query skill and of exported knowledge skills.
Two entrypoints rank a wiki release: `search_wiki.py` in the query skill and
`search_knowledge.py` in every frozen knowledge skill the maintenance skill
exports. Both are thin command-line wrappers around this module. They verify
their release before and after the search, then call `search()`. One core means
one ranking: an improvement reaches both places, and the exported copy cannot
drift away from the reader again.
This module only reads. It imports no release verifier and no writer. It takes a
release root plus options and returns plain report data. It also holds what
both wrappers share of their command line and report: the options, the request
intake, the selector, the report body and the printed form (`report_json`) in
which the evidence budget is counted. Printing, exit codes and integrity checks
stay with the wrappers.
Besides the ranked results, every report carries a `coverage` block that counts
what the search did not show: matches cut off by the limit, claims withheld per
page, query words that were dropped or found nowhere, and terms that occur only
in a layer that was not searched. The block only adds keys. It never changes
which results are returned, in which order, or with which score.
By default every query term also matches the forms of the same word that occur
in the searched files, and query words additionally match through two
disclosed heuristics, compound parts and one-edit typo corrections (see
wordforms.py). Each result names the rule that matched in `match_rules`, and
the coverage block lists the forms per term.
A search may carry up to four reformulations of the query, written by the
agent (synonyms, the wiki's own terms, a translation). One call reads the
release once and ranks the query and every reformulation over it. The query's
own `results`, `coverage` and `facets` stay exactly what they are without
reformulations; each result only names the searches that found it in
`found_by`. Pages that only reformulations return within the limit come back
separately in `reformulation_results`, taken in turn by rank from each
reformulation and never mixed into the query's ranking.
Every report also carries `absence_check`, the verdict on whether an absence
statement is allowed. With `exhaustive`, one call also ranks the sources and
scans every passage of the ranked files except navigation (page headers,
headings, claims, paragraphs, list items, tables, open questions) for all
content words of all formulations, in the members the ranking resolved plus
what only the scan sees: hyphen parts (`hyphen_part`), endings the word forms
do not cover (`ending`), separable verbs split in a main clause (`split_verb`),
the identifiers of claim and page metadata, and words only its segmentation
repairs. The hits come back under `exhaustive_hits`, cut
to a fixed budget of printed characters. The scan only adds keys: everything else equals the same
call with `include_sources`. Nothing is indexed or cached on disk.
"""
from __future__ import annotations
import json
import re
import unicodedata
from html import unescape as html_unescape
from pathlib import Path, PurePosixPath
from typing import Any, Iterable, Iterator, NamedTuple, Optional
import bm25
import freshness
import trust_contract
import wordforms
from frontmatter_contract import FrontmatterError, parse_document
from wiki_filters import matches_selector, validate_selector
H1 = re.compile(r"^#\s+(.+)$", re.MULTILINE)
HEADING = re.compile(r"^#{1,6}\s+(.+)$", re.MULTILINE)
TOKEN = re.compile(r"[^\W_]+(?:[-_][^\W_]+)*", re.UNICODE)
#: Digit groups split by a thousands separator, a full stop, an apostrophe or a
#: space, when every later group has exactly three digits ("2.500", "2 500",
#: "1.234.567"). `normalized` has already turned U+00A0, U+202F and U+2009 into
#: a space. A decimal comma ("2,50") is no separator, and no group starts inside
#: a number, a date ("31.12.2025") or a word.
DIGIT_GROUPS = re.compile(r"(?<![\w.,'’])\d{1,3}(?:[. '’]\d{3})+(?!\d)")
#: The separators DIGIT_GROUPS removes.
DIGIT_SEPARATOR = re.compile(r"[. '’]")
#: Units of two or more letters that a number may carry written straight after it
#: ("2500kWh"), normalised. A single letter stays on the number: "§ 14a" and
#: "§ 312g" name paragraphs, so "800W" is not split either.
NUMBER_UNITS = (
"kwhth", "kwhel", "kwh", "mwh", "gwh", "twh", "wh", "kwp", "mwp", "gwp", "kvar", "kva", "mva", "kw", "mw", "gw",
"kv", "va", "cent", "ct", "euro", "eur", "km", "kg", "cm", "mm", "qm", "m2", "m3", "ml", "std", "min", "sek",
"khz", "mhz", "ghz", "hz", "kbit", "mbit", "gbit", "kb", "mb", "gb", "tb", "bar", "mio", "mrd", "tsd",
)
#: A number with one of NUMBER_UNITS written straight after it.
NUMBER_UNIT = re.compile(r"(?<![\w-])(\d+)(" + "|".join(NUMBER_UNITS) + r")(?![^\W_])")
#: Where `number_spelling` can change anything: a separator before a group of three digits, or a unit after a digit.
NUMBER_SHAPE = re.compile(r"\d(?:[. '’]\d{3}(?!\d)|(?:" + "|".join(NUMBER_UNITS) + r")(?![^\W_]))")
#: The rule of a member that holds the same quantity as a query word in another
#: notation: another unit of the same dimension ("2,5 MWh" for "2500 kWh"), a
#: symbol for a unit word ("75 €", "20 %"), or a number word for digits before
#: the same word ("drei Monate" for "3 Monate").
QUANTITY_RULE = "quantity"
#: The first character of a quantity token. No token of a text starts with it,
#: so a quantity token never meets a written word.
QUANTITY_MARK = "="
#: Units a quantity token converts, normalised: written unit -> (canonical unit,
#: power of ten to it). Every factor is a power of ten, so the conversion is
#: exact. kWp, kWh_th and kWh_el stay apart: they are no plain kW and kWh.
QUANTITY_UNITS: dict[str, tuple[str, int]] = {
"wh": ("kwh", -3), "kwh": ("kwh", 0), "mwh": ("kwh", 3), "gwh": ("kwh", 6), "twh": ("kwh", 9),
"wattstunde": ("kwh", -3), "wattstunden": ("kwh", -3),
"kilowattstunde": ("kwh", 0), "kilowattstunden": ("kwh", 0),
"megawattstunde": ("kwh", 3), "megawattstunden": ("kwh", 3),
"gigawattstunde": ("kwh", 6), "gigawattstunden": ("kwh", 6),
"terawattstunde": ("kwh", 9), "terawattstunden": ("kwh", 9),
"w": ("kw", -3), "watt": ("kw", -3), "kw": ("kw", 0), "kilowatt": ("kw", 0),
"mw": ("kw", 3), "megawatt": ("kw", 3), "gw": ("kw", 6), "gigawatt": ("kw", 6),
"ct": ("eur", -2), "cent": ("eur", -2), "eur": ("eur", 0), "euro": ("eur", 0), "€": ("eur", 0),
"%": ("prozent", 0), "prozent": ("prozent", 0),
}
#: German number words, normalised, that stand for digits before a word.
#: "ein", "eine" and "einen" are articles and stay out.
NUMBER_WORDS = {
"eins": 1, "zwei": 2, "drei": 3, "vier": 4, "funf": 5, "sechs": 6, "sieben": 7, "acht": 8, "neun": 9,
"zehn": 10, "elf": 11, "zwolf": 12, "dreizehn": 13, "vierzehn": 14, "funfzehn": 15, "sechzehn": 16,
"siebzehn": 17, "achtzehn": 18, "neunzehn": 19, "zwanzig": 20, "dreissig": 30, "vierzig": 40,
"funfzig": 50, "sechzig": 60, "siebzig": 70, "achtzig": 80, "neunzig": 90,
}
#: The words a counted number may stand before, normalised, with the unit their
#: quantity token names. A closed list: a number before any other word is a
#: label ("Typ 2 Stecker", "Kapitel 4 Abrechnung", "Stufe 2 Tarif"), not a count.
COUNT_UNITS: dict[str, str] = {
form: unit
for unit, forms in (
("tag", "tag tage tagen tages"),
("werktag", "werktag werktage werktagen werktages"),
("kalendertag", "kalendertag kalendertage kalendertagen kalendertages"),
("woche", "woche wochen"),
("monat", "monat monate monaten monats"),
("quartal", "quartal quartale quartalen quartals"),
("jahr", "jahr jahre jahren jahres"),
("stunde", "stunde stunden"),
("minute", "minute minuten"),
("person", "person personen"),
("rate", "rate raten"),
)
for form in forms.split()
}
#: The values a number word names: only for them does a number before a count word give a quantity token.
NUMBER_WORD_VALUES = frozenset(NUMBER_WORDS.values())
#: What follows a number: a unit symbol, glued or after spaces, or one word after
#: white space (at most one line break). A hyphenated word ("4-Personen-Haushalt") is no follower.
QUANTITY_FOLLOWER = r"(?:[ \t]*([%€])(?![\w-])|(?:[ \t]+(?:\n[ \t]*)?|\n[ \t]*)([^\W\d_]+)(?![\w-]))"
#: The dash of a whole-euro amount ("75,- €", "75,-- EUR", "75,– €"), only before a money unit.
EURO_DASH = r",(?:--?|–)(?=[ \t]*(?:€|eur|euro)(?![\w-]))"
#: A number that may carry a quantity, with its follower: digits with an optional
#: decimal comma or whole-euro dash, as a whole token. Not inside a date, version or identifier
#: ("31.12.2025", "1.8.1", "Typ-2", "24/7", "12:30"), not after a dash ("1–3"), and not an ordinal ("1. April").
#: The first digit comes before the lookbehind, so the engine skips to digits fast.
QUANTITY_DIGITS = re.compile(
r"(\d(?<![\w.,'’/:⁄−+±=–—-]\d)\d{0,14})(?:,(\d{1,6})|" + EURO_DASH + r")?(?![\w-]|[.,]\d)" + QUANTITY_FOLLOWER
)
#: A money amount with its unit before it ("€ 75", "EUR 25,00"), not "Euro" (the
#: emission class "Euro 6"). Each unit comes before its lookbehind, so the engine searches for it as a literal.
QUANTITY_PREFIXED = re.compile(
r"(?:€(?<![\w-]€)|eur(?<![\w-]eur))[ \t]*(\d\d{0,14})(?:,(\d{1,6})|,(?:--?|–))?(?![\w%€-]|[.,]\d)"
)
#: Scale words: a number before them names no count of them.
SCALE_WORDS = frozenset({"hundert", "tausend", "tsd", "mio", "mrd", "million", "millionen", "milliarde", "milliarden"})
#: A scale word right after a prefixed money amount ("€ 1,5 Mio."): the token would lose the factor.
SCALE_AFTER = re.compile(r"[ \t]*(?:" + "|".join(sorted(SCALE_WORDS)) + r")(?![^\W\d_])")
#: Each number word as a whole token with a follower, the same boundaries; run only
#: for the words a text holds. The word comes before the lookbehind, so the engine searches for it as a literal.
NUMBER_WORD_AT = {
word: re.compile(word + r"(?<![\w.,'’/:⁄−+±=–—-]" + word + r")(?![\w-]|[.,]\d)" + QUANTITY_FOLLOWER) for word in NUMBER_WORDS
}
#: A number, a number word or "komma" right before a number: the number is part of a larger one.
NUMBER_BEFORE = re.compile(r"(?:\d|\b(?:" + "|".join(NUMBER_WORDS) + r"|komma))[ \t]+$")
#: A sign right before a number ("minus 5 %", "−5 %", "Veränderung - 5 %"), or a
#: range with a hyphen ("1 - 3 Monate"): the token would lose the sign or the range. A hyphen
#: that starts a line is a list marker and stays allowed ("- 14 Tage Frist").
SIGN_BEFORE = re.compile(r"(?:\bminus|\bplus|[−+±]|\S[ \t]+-)[ \t]*$")
#: The first number of a range right before a number: with a dash ("1 – 3
#: Monate"; a dash between words is a parenthesis and stays allowed: "Frist – wie
#: üblich – drei Monate"), with "bis" ("2 bis 4 Wochen", "10 € bis 20 €"), or
#: after "zwischen" with "und" ("zwischen 10 und 20 Prozent"). The second
#: number would lose the range; a first number with its own unit keeps its token.
RANGE_BEFORE = re.compile(
r"(?:(?:\d|\b(?:" + "|".join(NUMBER_WORDS) + r"))[ \t]*[–—]|(?:\d|[%€]|\b(?:" + "|".join(NUMBER_WORDS)
+ r"))[ \t]+bis|\bzwischen[ \t]+\S+(?:[ \t]+\S+)?[ \t]+und)[ \t]*$"
)
#: A legal citation right before a number ("§ 3", "Abs. 5", "Nr. 2"): the number names a provision, not a count.
CITATION_BEFORE = re.compile(r"(?:§|\babs\.|\babsatz|\bnr\.|\bnummer|\bart\.|\bartikel|\bziff\.|\bziffer|\bsatz)\s*$")
#: A definition heading of schema/CONCEPTS.md or schema/CLUSTERS.md: level two with a stable lowercase ID.
CONCEPT_HEADING = re.compile(r"^##\s+([a-z0-9]+(?:-[a-z0-9]+)*)\s*$")
#: A `- Field: value` line of a definition.
CONCEPT_FIELD = re.compile(r"^-\s+([A-Za-z]+):\s*(.*)$")
#: Any Markdown ATX heading line (unstripped): at most three spaces, one to six '#', then a space or the line end.
SCHEMA_HEADING = re.compile(r"^ {0,3}#{1,6}(?:\s|$)")
#: The opening line of a fenced block in a schema file, after stripping: three or more backticks or tildes.
SCHEMA_FENCE = re.compile(r"^(`{3,}|~{3,})")
CLAIM_BLOCK = re.compile(
r"<!--\s*claim\s*\n(?P<meta>.*?)-->\s*(?P<text>.*?)\s*<!--\s*/claim\s*-->",
re.DOTALL | re.IGNORECASE,
)
#: Version of the coverage block's shape. Version 2 counts word forms and adds
#: `forms`, `forms_total`, `form_only_terms` and the per-result `match_rules`.
#: Version 3 adds the per-search counts `formulations` and
#: `reformulation_results`; every report carries them, with or without
#: reformulations.
COVERAGE_FORMAT = "lmwiki-search-coverage/3"
#: At most this many reformulations per call, so at most five searches with the query.
MAX_ALSO = 4
#: The longest reformulation, counted after stripping surrounding whitespace.
MAX_FORMULATION_CHARS = 200
#: The largest request `--request -` reads from standard input.
MAX_REQUEST_BYTES = 65536
#: The keys a request may hold; any other key is refused.
REQUEST_KEYS = ("query", "also", "filter")
#: The fixed sentence that keeps a reader from taking the reformulation block for coverage.
REFORMULATION_BOUNDARY = (
"These pages were returned by the agent's reformulations, not by the words of the question, and are "
"ordered by their rank within each reformulation, not by score. They count as found only after the "
"page has been read and confirmed to answer the question; neither this list nor its absence shows "
"whether the wiki covers the topic."
)
#: A question mark directly before a letter: how Windows PowerShell 5.1 pipes a
#: non-ASCII character when its output encoding is ASCII ("K?ndigung",
#: "?kostrom", "Gr??e"). A damaged character at the end of a word ("s??") is
#: not caught, so the escapes stay the only reliable protection.
RE_ENCODED = re.compile(r"\?[^\W\d_]")
#: A lone surrogate code point: half of a character outside U+FFFF, which a
#: JSON escape can produce but no UTF-8 output can carry.
RE_SURROGATE = re.compile(r"[\ud800-\udfff]")
#: Why a request with a lone surrogate escape is refused, and what to send instead.
SURROGATE_DETAIL = (
"holds a lone surrogate escape; write a character outside U+FFFF as a surrogate pair, or as UTF-8"
)
#: How a reader learns to send text that no console code page can change.
ESCAPE_HINT = "escape every non-ASCII character as \\uXXXX"
#: How many forms the coverage block names per query term.
FORMS_LISTED = 10
#: How many matches beyond the limit the coverage block names with path and score.
OMITTED_TOP_LIMIT = 20
#: Searchable layers of a release, in the order the coverage block lists them.
SCOPES = ("wiki", "sources", "history")
#: The fixed sentence every coverage block carries, so no reader takes a count for a verdict.
COVERAGE_BOUNDARY = (
"These counts describe matches of the listed terms in the listed files, counting the forms "
"of the same word and the disclosed compound, irregular-form, quantity and typo matches listed "
"under forms. They do not show that the wiki holds nothing on the topic: a page that uses other "
"words, an irregular form outside the closed table, a number in a notation the search does not "
"convert, or another language is not counted."
)
#: Files that route a reader to content instead of holding it. Any index.md is one too.
NAVIGATION_FILES = frozenset({"WIKI.md", "meta/questions.md"})
class QueryError(ValueError):
"""The query holds no token the search can match."""
class RequestError(ValueError):
"""A request or its reformulations cannot be searched; the message is ready for the user."""
#: A character outside ASCII; every combining mark is one.
NON_ASCII = re.compile(r"[^\x00-\x7f]")
def normalized(value: str) -> str:
"""Casefold, decompose and drop combining marks, so "Zählerstand" matches "Zahlerstand".
Each distinct non-ASCII character is looked up once, and every mark found
is removed with one `str.replace`. Checking every character of a long
source text one by one made this the most expensive step of a search.
"""
decomposed = unicodedata.normalize("NFKD", value.casefold())
if decomposed.isascii():
return decomposed
for mark in [character for character in set(NON_ASCII.findall(decomposed)) if unicodedata.combining(character)]:
decomposed = decomposed.replace(mark, "")
return decomposed
#: The function-word tiers of wordforms.py, normalised like every token, so "für" is dropped as "fur".
FUNCTION_WORDS = wordforms.function_words(normalized)
#: The stopwords: left out of a query's terms unless nothing else is left, never a compound part.
FORM_STOPWORDS = FUNCTION_WORDS["stopword"]
#: Words that match only literally: the stopwords and the closed word classes.
CLOSED_CLASS = FUNCTION_WORDS["closed"]
# -- constants of the exhaustive scan and the absence check (Stufe 4) ----------
#: The most printed characters `exhaustive_hits` may take (owner decision E2),
#: counted exactly as `report_json` prints the value.
EXHAUSTIVE_BUDGET_CHARS = 100_000
#: The indent `report_json` prints a report with; the budget is counted in it.
REPORT_INDENT = 2
#: A hit prints at most this many characters of its passage, cut around the first match.
EXCERPT_CHARS = 400
#: A paragraph longer than this is cut into several passages, so a PDF
#: extraction without blank lines cannot become one passage per page. The cut
#: falls at the first sentence end once a passage has reached this size, so no
#: sentence is split; a table is cut at the row that reaches it.
PASSAGE_MAX_CHARS = 1_500
#: Where a paragraph that reaches no sentence end is cut all the same, at the
#: end of the line that reaches this size.
PASSAGE_HARD_MAX_CHARS = 3_000
#: How many pages without a returned hit absence_check names.
OMITTED_PAGES_LISTED = 20
#: Counted reformulations a complete absence check needs.
MIN_EXHAUSTIVE_REFORMULATIONS = 2
#: The longest query the search record quotes before it is cut.
SEARCH_RECORD_QUERY_CHARS = 200
#: The kinds of passage the scan cuts a file into; hits on one line sort in this order.
PASSAGE_KINDS = ("header", "heading", "claim", "paragraph", "open_question")
#: The scan's own function words, normalised (see wordforms.SCAN_FUNCTION_WORDS):
#: a passage need not hold them; the ranking and the precision gate never use them.
SCAN_FUNCTION_WORDS = frozenset(normalized(word) for word in wordforms.SCAN_FUNCTION_WORDS.split())
#: Words a passage never has to hold for a formulation to hit it: the closed class and the scan's function words.
SCAN_EXCLUDED = FUNCTION_WORDS["scan"]
#: A page or slide marker of an extracted source, the grammar of validate_extraction.py.
PAGE_MARKER = re.compile(r"<!--\s*(page|slide)\s*:\s*([^>]+?)\s*-->", re.IGNORECASE)
#: Why an absence check is not complete, in the order they are listed.
ABSENCE_REASONS = ("not_exhaustive", "too_few_reformulations", "no_content_words", "budget_exhausted")
#: What to do before an absence statement, in the order they are listed.
ABSENCE_STEPS = (
"read_withheld_claims", "read_results_without_claims", "read_reformulation_results", "read_exhaustive_hits",
"add_reformulations", "narrow_formulations",
)
ABSENCE_CHECK_FORMAT = "lmwiki-absence-check/1"
#: The fields of absence_check that the exhaustive scan fills, in their printed order; null without the scan.
SCAN_FIELDS = (
"reformulations_counted", "budget", "counts", "omitted_pages", "absent_terms", "formulations", "scopes", "terms",
)
SEARCH_RECORD_FORMAT = "lmwiki-search-record/1"
#: The fixed sentence every absence check carries: what a complete check shows, and what it does not.
ABSENCE_BOUNDARY = (
"A complete check means that every passage of the scanned files that holds all content words "
"of at least one searched formulation, in the forms listed under terms, itself or in its context, "
"was returned under exhaustive_hits. Context is the page header, the headings and label lines above "
"the passage, its parent list items and the first row of a split table, nothing else: a passage that "
"needs a neighbouring paragraph, or a sentence of its paragraph beyond a cut, or that words the answer "
"differently, with other words, an irregular form outside the closed table, a number in words or in a "
"unit the scan does not convert, or another language, is not a hit, and layers marked scanned: false were not checked. A table is read as a table "
"only as a pipe, grid, simple or multiline table or as raw HTML; one laid out otherwise is read as "
"paragraphs, without its header row."
)
#: Order of the layers among pages with otherwise equal hits.
SCOPE_RANK = {scope: position for position, scope in enumerate(SCOPES)}
#: A code fence: inside it no line is a heading or a marker.
FENCE = re.compile(r"^ {0,3}(```|~~~)")
#: A heading at column 0, as the ranking's HEADING counts it.
SCAN_HEADING = re.compile(r"^(#{1,6})\s+(.+?)\s*$")
#: A list item starts a passage of its own.
LIST_ITEM = re.compile(r"^\s*(?:[-*+]|\d{1,9}[.)])\s+")
#: The markers of a block quote at the start of a line: up to three spaces, then one or more ">", each with an optional space.
QUOTE_PREFIX = re.compile(r" {0,3}(?:>[ \t]?)+")
#: A possible sentence end inside a line: ., ! or ?, closing quotes or brackets, whitespace, the next word.
SENTENCE_GAP = re.compile(r"[.!?][\"'»«“”’‘)\]]*(\s+)(?=(\S))")
#: Characters that may close a sentence after its final punctuation.
SENTENCE_CLOSERS = "\"'»«“”’‘)]"
#: Characters that may open a word before its letters.
WORD_OPENERS = "\"'»«“”„‚‘(["
#: Abbreviations whose full stop ends no sentence, normalised; single letters,
#: numbers and words with an inner full stop ("z.B.") are caught by rule.
SENTENCE_ABBREVIATIONS = frozenset((
"abs abschn anh anl anm art aufl bd bzgl bspw bzw ca dr ebd einschl entspr erg evtl exkl ff gem ggf ggu hrsg "
"inkl insb jh kap lfd lit lt max mind mio mrd nr pkt prof sog spez std str tel tsd usw vgl vorl zt zzgl ziff "
"jan feb mar apr jun jul aug sep sept okt nov dez mo di mi do fr sa so hr "
"approx cf eg etc fig ie incl excl no nos pp vol vs").split())
#: A setext heading underline under a one-line paragraph: = for level 1, - for level 2.
SETEXT_UNDERLINE = re.compile(r"^ {0,3}(?:=+|-+)[ \t]*$")
#: A line that is only bold text, with an optional colon: a label that heads the passages after it.
BOLD_LABEL = re.compile(r"^(\*\*|__)(?=\S)(.+?)(?<=\S)\1:?$")
#: The longest one-line paragraph that may act as a label or a setext heading.
LABEL_MAX_CHARS = 100
#: A label's level on the heading stack: below every Markdown heading, so any heading or the next label ends it.
LABEL_LEVEL = 7
#: The scan's rule for a token that holds the word as one of its hyphen-separated parts
#: ("SMS" in "SMS-Warndienst"), and for the parts of a hyphenated search word written apart.
HYPHEN_PART_RULE = "hyphen_part"
#: The scan's rule for a token that shares its base with the term where the
#: stem rule cannot see it: the base is the word itself or the word without one
#: of INFLECTION_ENDINGS. That covers plurals in -er, with or without umlaut
#: ("Häuser", "Kinder"), in -s after a vowel ("Autos"), in -se ("Ergebnisse")
#: and in -nen ("Kundinnen"), the forms of a stem shorter than four letters
#: ("Tag", "Tage", "Tagen"), and the case endings -er and -em of an adjective
#: that no adjective suffix marks ("intelligentem" and "intelligenten",
#: "fester" and "festen"). A base of at least DEGREE_MIN_STEM letters is also
#: the word without one of DEGREE_ENDINGS, so the comparative and the
#: superlative meet the adjective and each other ("günstig", "günstigeren",
#: "günstigste", "am schnellsten", "spätestens"; English "cheaper" and
#: "cheapest"), or without one of FEMININE_ENDINGS, so a feminine form meets
#: the generic one ("Kundin", "Kundinnen" and the Binnen-I "KundInnen" for
#: "Kunde" and "Kunden", "Mitarbeiterinnen" for "Mitarbeiter"). It is a
#: heuristic: "Zahl" also meets "Zähler", so a hit through it is checked in the text.
ENDING_RULE = "ending"
#: The endings the rule removes from a term and from a scanned token to find their base.
INFLECTION_ENDINGS = ("e", "en", "n", "er", "ern", "es", "em", "s", "se", "sen", "nen")
#: The shortest stem the rule may leave or extend.
ENDING_MIN_STEM = 3
#: The comparative with an inflection ending and the superlative, German and English, that the rule removes and adds.
DEGREE_ENDINGS = (
"ere", "eren", "erer", "erem", "eres",
*(stem + ending for stem in ("st", "est") for ending in ("e", "en", "er", "em", "es", "ens")),
"est",
)
#: The feminine suffix, singular and plural, that the rule removes and adds.
FEMININE_ENDINGS = ("in", "innen")
#: The shortest base a degree or feminine ending may leave or extend ("Polster" is no superlative of "Pol").
DEGREE_MIN_STEM = 4
#: The scan's rule for a token that is another verb form of the term where the
#: word forms cannot see it: a regular verb whose stem has four letters
#: ("senkt" and "senken", "zahlte" and "zahlen", "prüft" and "geprüft"). One
#: of VERB_ENDINGS is removed from the term, another one added, or the stem
#: is put into a participle (ge-...-t). The rule also takes the regular forms
#: of an English verb for each other: -ed, -d after e, -ied for y, -ing and a
#: doubled final consonant ("cancel" and "cancelled", "apply" and "applied",
#: "charge" and "charging", "bill" and "billed"). It is a heuristic: "Markt"
#: also meets "Marken", so a hit through it is checked in the text.
VERB_ENDING_RULE = "verb_ending"
#: The verb endings the rule removes from a term and adds to its stem.
VERB_ENDINGS = ("e", "en", "n", "t", "et", "te", "ten", "ete", "eten", "st", "est")
#: The shortest stem the verb rule may leave or extend.
VERB_MIN_STEM = 4
#: The shortest base of an English verb form ("use" and "used").
ENGLISH_MIN_BASE = 3
#: Vowels, for the doubled consonant of an English verb form ("stop" and "stopped").
VOWELS = "aeiou"
#: The particles of separable verbs, which put ge- and zu- between particle and
#: stem ("abschalten", "abgeschaltet", "abzuschalten"). The verb rule also takes
#: a token for a form of such a verb, normalised; the longer particles come first.
SEPARABLE_PARTICLES = tuple(sorted(
{
normalized(word)
for word in (
"ab an auf aus bei dar durch ein empor entgegen fest fort frei her herab heran herauf heraus herbei "
"herein herum herunter hervor hin hinab hinauf hinaus hinein hinzu hoch los mit nach nieder um vor "
"voran voraus vorbei weg weiter wieder zu zurecht zurück zusammen"
).split()
},
key=lambda particle: (-len(particle), particle),
))
#: The shortest stem after a separable particle ("einbauen" and "eingebaut" share "bau").
SEPARABLE_MIN_STEM = 3
#: The scan's rule for a separable verb split in a main clause, the finite
#: verb first and the particle at the end of the clause: a passage that writes
#: "lehnt ... ab" or "passt ... an" holds "ablehnen" or "angepasst", and a
#: question that asks "Wann schaltet ... ab?" matches "abschalten",
#: "abgeschaltet" and "schaltet ... ab" for "schaltet" (and for the particle,
#: when it is a content word such as "zurück"). The scan pairs a particle that
#: ends a clause (before , . ; : ! ?, the end of the text or a conjunction)
#: with every word of its sentence before it that may be a finite verb; it is
#: a heuristic, so a hit through it is checked in the text.
SPLIT_VERB_RULE = "split_verb"
#: The mark that starts a split form in the scan's tokens ("+lehnt+ab"); TOKEN
#: never matches it, and the report prints the form as "lehnt … ab".
SPLIT_MARK = "+"
#: The separable particles as a set, for the clause end.
PARTICLE_SET = frozenset(SEPARABLE_PARTICLES)
#: How far before a particle the scan looks for its verb, in characters: far
#: enough for a long legal sentence, and a bound on the work for a text that
#: is one endless line.
SPLIT_LOOKBACK_CHARS = 500
#: A word that may be the finite verb of a split separable verb: at least
#: SEPARABLE_MIN_STEM letters, ending in one of VERB_ENDINGS (each ends in e, n
#: or t), or a form of TABLE_SPLIT_VERBS.
VERB_CANDIDATE = re.compile(r"(?<![^\W\d_])[^\W\d_]{%d,}[ent](?![^\W\d_])" % (SEPARABLE_MIN_STEM - 1))
def _table_split_verbs() -> dict[str, frozenset]:
"""The finite forms of the separable verbs of the irregular tables that end in none of e, n, t, with their particles.
A strong preterite or changed present ("zog", "schloss", "gab", "nahm")
is no VERB_CANDIDATE; it may still stand before its particle ("zog ...
um"). A form counts only for a particle that the tables join to it
("umzog", "abschloss"), so "zog ... mit" stays no split verb.
"""
found: dict[str, set[str]] = {}
for _present, changed, preterite, _participle, prefixes in wordforms.IRREGULAR_VERBS:
for prefix in prefixes.split():
if prefix not in PARTICLE_SET:
continue
for form in (*changed.split(), *preterite.split()):
if form[-1] not in "ent" and len(form) >= SEPARABLE_MIN_STEM and prefix + form in wordforms.IRREGULAR_FORMS:
found.setdefault(form, set()).add(prefix)
return {form: frozenset(particles) for form, particles in sorted(found.items())}
#: Per finite form of a separable verb of the irregular tables outside VERB_CANDIDATE, its particles.
TABLE_SPLIT_VERBS = _table_split_verbs()
#: Each form of TABLE_SPLIT_VERBS as a whole word, the longest first.
TABLE_SPLIT_VERB = re.compile(
r"(?<![^\W\d_])(?:" + "|".join(sorted(TABLE_SPLIT_VERBS, key=lambda form: (-len(form), form))) + r")(?![^\W\d_])"
)
#: Each separable particle as a whole word of a normalised text.
PARTICLE_WORDS = {particle: re.compile(r"(?<![\w-])" + particle + r"(?![\w-])") for particle in SEPARABLE_PARTICLES}
#: What follows a word that ends a clause: closing quotes or brackets, then a
#: punctuation mark, the end of the text, or a conjunction.
CLAUSE_END = re.compile(
r"[\"'»«“”’‘)\]]*(?:[ \t\n]*(?:[.,;:!?]|\Z)|[ \t\n]+(?:und|oder|aber|sondern|denn|bzw|sowie)(?![^\W_]))"
)
#: The scan's rule for a word of two or three letters inside a longer word,
#: which the compound rules (parts of four letters or more) never consider: as
#: the first part, when the rest, after a linking element, is one or more
#: words of the scanned files ("Gas" in "Gaskunden", "Gasversorger"), or as
#: the last part, with or without an inflection ending, after at least
#: SHORT_PART_LEAD letters ("Gas" in "Erdgas", "Tag" in "Feiertagen"). It is a
#: heuristic: "Eis" also meets "Beweis", so a hit through it is checked in the text.
SHORT_PART_RULE = "short_part"
#: The shortest and the longest word the rule applies to.
SHORT_PART_MIN = 2
SHORT_PART_MAX = 3
#: The fewest letters before a short word that is the last part of a longer word.
SHORT_PART_LEAD = 3
#: The scan's rule for a compound written with a suspension hyphen: "Grund- und
#: Leistungspreis" holds "Grundpreis", "Vertragsbeginn und -ende" holds
#: "Vertragsende". The scan joins the written part to every possible part of
#: the other word and matches a term against those joined forms only; it is a
#: heuristic, so a hit through it is checked in the text.
SUSPENDED_RULE = "suspended_compound"
#: The mark that keeps a joined form of a suspended compound apart from every
#: written token; TOKEN never matches it, and the report prints the form without it.
SUSPENDED_MARK = "~"
#: A suspension hyphen before a conjunction or a slash: "Grund- und", "Netz-, ", "Strom-/".
SUSPENDED_FRONT = re.compile(r"-(?:,?\s+(?:und|oder|bzw\.?|sowie|bis|and|or|&)\s+|\s*/\s*)(?=[^\W\d_])")
#: A suspension hyphen after a comma or a conjunction: "und -ende", ", -verlängerung".
SUSPENDED_BACK = re.compile(r"(?:,|\s(?:und|oder|bzw\.?|sowie|bis|and|or|&))\s+-(?=[^\W\d_])")
#: A conjunction a suspension hyphen may stand before or after.
SUSPENDED_CONJUNCTIONS = frozenset(("und", "oder", "bzw", "bzw.", "sowie", "bis", "and", "or", "&"))
#: A run of letters.
LETTERS = re.compile(r"[^\W\d_]+")
#: The shortest part of a word that a suspended compound may take over.
SUSPENDED_MIN_PART = 3
#: The scan's rule for a word with a footnote mark glued to it as one or two
#: plain digits, as MarkItDown writes <sup>2</sup> ("Kündigungsfrist2"). The
#: joined token is kept as well, so a name like "Fix12" still matches itself.
FOOTNOTE_RULE = "footnote_mark"
#: A token of at least four letters with one or two digits glued to its end.
FOOTNOTE_TOKEN = re.compile(r"([^\W\d_]{4,})\d{1,2}")
#: Superscript digits (¹ ² ³ ⁰ ⁴-⁹), which normalisation turns into plain digits glued to the word.
SUPERSCRIPT_DIGIT = re.compile("[¹²³⁰⁴-⁹]")
#: The border of a grid table (pandoc writes one for cells with several paragraphs): +---+---+ or +===+===+.
GRID_BORDER = re.compile(r"^ {0,3}\+(?:[-=:]+\+)+[ \t]*$")
#: The column rule of a simple or multiline table (pandoc's default for a Word table): two or more runs of dashes.
COLUMN_RULE = re.compile(r"^ {0,3}-{3,}(?: +-{3,})+[ \t]*$")
#: A full-width rule: the top and bottom border of a multiline table.
FULL_RULE = re.compile(r"^ {0,3}-{3,}[ \t]*$")
#: The most lines a multiline table's header may take.
TABLE_HEADER_MAX_LINES = 10
#: The start of a raw HTML table, as pandoc writes a table with merged cells or block content.
HTML_TABLE_START = re.compile(r"<table\b", re.IGNORECASE)
#: The end of a raw HTML table.
HTML_TABLE_END = re.compile(r"</table\s*>", re.IGNORECASE)
#: One row of an HTML table, up to its closing tag or the next row (HTML allows leaving </tr> out).
HTML_ROW = re.compile(
r"<tr\b[^>]*>(.*?)(?=</tr\s*>|<tr\b|</t(?:head|body|foot)\s*>|</table\s*>|\Z)", re.IGNORECASE | re.DOTALL
)
#: One cell of an HTML table row, header or data.
HTML_CELL = re.compile(r"<t[hd]\b[^>]*>(.*?)(?=</t[hd]\s*>|<t[hd]\b|\Z)", re.IGNORECASE | re.DOTALL)
#: An HTML heading on a line of its own.
HTML_HEADING = re.compile(r"^<h([1-6])\b[^>]*>(.*?)</h\1\s*>$", re.IGNORECASE)
#: The summary of an HTML details block on a line of its own, optionally after the opening <details>.
HTML_SUMMARY = re.compile(r"^(?:<details\b[^>]*>\s*)?<summary\b[^>]*>(.*?)</summary\s*>$", re.IGNORECASE)
#: The end of an HTML details block: the label its summary set ends here.
HTML_DETAILS_END = re.compile(r"^</details\s*>$", re.IGNORECASE)
#: Any HTML tag or comment, removed from the text of a table, heading or summary.
HTML_TAG = re.compile(r"<[^>]*>")
#: An inline HTML tag, which a browser shows without a space: removed from the text without one.
HTML_INLINE_TAG = re.compile(
r"</?(?:a|abbr|b|bdi|bdo|cite|code|data|del|dfn|em|i|ins|kbd|mark|q|s|samp|small|span|strong|time|u|var|wbr)\b[^>]*>",
re.IGNORECASE,
)
#: A source ID inside a wiki link or path, as the wiki contract names source files (`src-<hash>-<slug>.md`).
SOURCE_ID_IN_TEXT = re.compile(r"\bsrc-[0-9a-f]{16}(?![0-9a-z])")
#: A wiki link without a label; its target is an address, not text.
UNLABELLED_WIKILINK = re.compile(r"\[\[([^\[\]|]*)\]\]")
def number_spelling(value: str, apart: Optional[list[str]] = None) -> str:
"""A normalised text with every number in one spelling: digit groups joined, a unit split off (pure).
"2.500 kwh", "2 500 kwh" and "2500kwh" all become "2500 kwh", so a query
and a page that write one number differently meet. When `apart` is a
list, the groups of every number joined across a space are appended to
it: a space between two numbers ("100 200") is not always a separator.
"""
if NUMBER_SHAPE.search(value) is None:
return value
def join(match: "re.Match[str]") -> str:
written = match.group(0)
if apart is not None and " " in written:
apart.extend(DIGIT_SEPARATOR.split(written))
return DIGIT_SEPARATOR.sub("", written)
return NUMBER_UNIT.sub(r"\1 \2", DIGIT_GROUPS.sub(join, value))
def decimal_text(mantissa: int, exponent: int) -> str:
"""mantissa * 10**exponent written as a plain decimal with a point and no trailing zero (pure)."""
if mantissa == 0:
return "0"
while mantissa % 10 == 0:
mantissa //= 10
exponent += 1
digits = str(mantissa)
if exponent >= 0:
return digits + "0" * exponent
if len(digits) > -exponent:
return digits[:exponent] + "." + digits[exponent:]
return "0." + "0" * (-exponent - len(digits)) + digits
def spelled_quantities(spelled: str, words: Optional[Iterable[str]] = None) -> list[tuple[tuple[str, ...], str]]:
"""The quantities of a normalised text in one spelling (`number_spelling`), as (source tokens, quantity token) (pure).
A quantity is a number followed by a unit of QUANTITY_UNITS, converted to
its canonical unit ("2,5 mwh" and "2500 kwh" give "=2500kwh", "75,00 eur"
and "75 €" give "=75eur"), or a number that a number word names followed
by a count word of COUNT_UNITS ("drei monate" and "3 monaten" give
"=3monat", keyed by the unit the list names). A money amount may also
carry a whole-euro dash ("75,- €") or its unit before it ("€ 75", "eur
25,00"). A number that is only part of a quantity gives none: after a
citation, another number, a sign, or the first number of a range
(CITATION_BEFORE, NUMBER_BEFORE, SIGN_BEFORE, RANGE_BEFORE), or before a
scale word (a scale word is no unit; after a prefixed amount, SCALE_AFTER).
The source tokens are the tokens the number and the unit word give
as written ("9", "84" and "ct" for "9,84 ct"); a symbol gives none. `words`, the text's tokens when the caller has them,
saves looking for number words the text does not hold.
"""
found: list[tuple[tuple[str, ...], str]] = []
present = sorted(NUMBER_WORDS.keys() & (TOKEN.findall(spelled) if words is None else words))
matches: list[tuple["re.Match[str]", Optional[str]]] = [(match, None) for match in QUANTITY_DIGITS.finditer(spelled)]
if "€" in spelled or "eur" in spelled:
matches.extend((match, QUANTITY_MARK) for match in QUANTITY_PREFIXED.finditer(spelled))
if not matches and not present:
return found
for number_word in present:
matches.extend((match, number_word) for match in NUMBER_WORD_AT[number_word].finditer(spelled))
for match, word in matches:
if CITATION_BEFORE.search(spelled, max(0, match.start() - 12), match.start()):
continue
if NUMBER_BEFORE.search(spelled, max(0, match.start() - 16), match.start()):
continue
if SIGN_BEFORE.search(spelled, max(0, match.start() - 8), match.start()):
continue
if RANGE_BEFORE.search(spelled, max(0, match.start() - 40), match.start()):
continue
if word == QUANTITY_MARK:
# A prefixed money amount: the unit is known, a scale word after it would be lost.
if SCALE_AFTER.match(spelled, match.end()):
continue
whole, fraction = match.groups()
mantissa, exponent = int(whole + (fraction or "")), -len(fraction or "")
numbers = (whole, fraction) if fraction else (whole,)
unit_word = ("eur",) if spelled.startswith("eur", match.start()) else ()
found.append((numbers + unit_word, QUANTITY_MARK + decimal_text(mantissa, exponent) + "eur"))
continue
if word is None:
whole, fraction, symbol, unit_word = match.groups()
mantissa = int(whole + (fraction or ""))
exponent = -len(fraction or "")
numbers = (whole, fraction) if fraction else (whole,)
else:
symbol, unit_word = match.groups()
mantissa, exponent, numbers = NUMBER_WORDS[word], 0, (word,)
unit = symbol or unit_word
converted = QUANTITY_UNITS.get(unit)
if converted is not None:
canonical, shift = converted
exponent += shift
elif word is None and (mantissa not in NUMBER_WORD_VALUES or exponent):
# Digits no number word names gain nothing from a plain word after them.
continue
elif unit in COUNT_UNITS:
canonical = COUNT_UNITS[unit]
else:
continue
sources = numbers + ((unit,) if unit_word else ())
found.append((sources, QUANTITY_MARK + decimal_text(mantissa, exponent) + canonical))
return found
def superscript_tokens(repaired: str, written: Iterable[str]) -> list[str]:
"""The tokens a repaired text gives with every superscript glued to its word that `written` lacks (pure).
"m^2^" is split into "m" and "2" as written, and gives "m2" here, so a
search for "m²" (normalised "m2") meets it; "Kündigungsfrist^1^" keeps
the word apart from its footnote mark and adds "kundigungsfrist1".
"""
joined = wordforms.join_superscripts(repaired)
if joined == repaired:
return []
seen = set(written)
extra: list[str] = []
for token in TOKEN.findall(number_spelling(normalized(joined))):
if token not in seen:
seen.add(token)
extra.append(token)
return extra
def tokens(value: str, drop_stopwords: bool = True, quantities: Optional[list[str]] = None) -> list[str]:
"""The normalised whole words of a text, as the word-form layer reads it.
Layout and markup artefacts that split a word are repaired first
(`wordforms.repair_text`), numbers are brought into one spelling
(`number_spelling`), and a word with a superscript ("m^2^") also gives
its joined token ("m2") after the split ones. With `drop_stopwords`,
stopwords and single characters are left out. When `quantities` is a
list, the quantity tokens of the text are appended to it
(`spelled_quantities`), from the same spelling.
"""
repaired = wordforms.repair_text(value)
spelled = number_spelling(normalized(repaired))
values = TOKEN.findall(spelled)
if quantities is not None:
quantities.extend(token for _, token in spelled_quantities(spelled, values))
if "^" in repaired:
values.extend(superscript_tokens(repaired, values))
if drop_stopwords:
values = [value for value in values if value not in FORM_STOPWORDS and len(value) > 1]
return values
def frontmatter(text: str, source: str) -> tuple[dict[str, Any], str]:
document = parse_document(text, source)
return document.data, document.body
def candidate_paths(target: Path, include_sources: bool, include_history: bool) -> Iterable[Path]:
roots = [target / "WIKI.md", target / "wiki", target / "meta/questions.md"]
if include_sources:
roots.append(target / "sources")
if include_history:
roots.append(target / "meta/history")
seen: set[Path] = set()
for root in roots:
if root.is_file() and root.suffix.lower() == ".md":
paths = [root]
elif root.is_dir():
# Like the release verifier, only regular files count: a folder
# named like a page or a dangling link is not part of the release.
paths = [path for path in sorted(root.rglob("*.md")) if path.is_file()]
else:
paths = []
for path in paths:
resolved = path.resolve()
if resolved not in seen:
seen.add(resolved)
yield path
def as_list(value: Any) -> list[str]:
if isinstance(value, list):
return [str(item) for item in value]
return []
def alias_values(value: Any) -> list[str]:
"""Page aliases as a list: a list as before, a non-empty string as one alias (lint warns), anything else none."""
if isinstance(value, list):
return [str(item) for item in value]
if isinstance(value, str) and value.strip():
return [value]
return []
def split_pipe(value: str) -> list[str]:
return [item.strip() for item in value.split("|") if item.strip()]
def schema_definitions(text: str) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""Definitions of CONCEPTS.md or CLUSTERS.md in file order, and the field lines that belong to none (pure).
A level-two heading with a stable ID starts a definition and its `- Field: value`
lines follow. Any other ATX heading ends the open definition, so a section such as
`## Notes` can never pass its fields to the definition above it. Fenced blocks
(three or more backticks or tildes, closed only by a run of the same character at
least as long) and HTML comments over several lines are skipped, so a `#` line in
an example or a commented-out draft ends nothing; neither does a line indented by
four or more spaces. Returns (definitions, orphans): {"id", "line", "fields"} and
{"line", "field", "heading", "closed"}; `heading` is the nearest heading above the
orphan, `closed` the definition the old parse applied it to; lines are 1-based.
A repeated field keeps its last value. Lint, the search and the graph all read
both files through this one function.
"""
definitions: list[dict[str, Any]] = []
orphans: list[dict[str, Any]] = []
current: Optional[dict[str, Any]] = None
closer: Optional[dict[str, str]] = None
fence = "" # the run of backticks or tildes that opened the current fenced block
comment = False # inside an HTML comment that spans several lines
for number, raw_line in enumerate(text.splitlines(), 1):
line = raw_line.strip()
if fence:
if line.startswith(fence):
fence = ""
continue
if comment:
comment = "-->" not in line
continue
opening = SCHEMA_FENCE.match(line)
if opening:
fence = opening.group(1)
continue
if line.startswith("<!--") and "-->" not in line[4:]:
comment = True
continue
heading = CONCEPT_HEADING.match(line)
if heading:
current = {"id": heading.group(1), "line": number, "fields": {}}
definitions.append(current)
closer = None
continue
if SCHEMA_HEADING.match(raw_line):
if current is not None:
closer = {"heading": line, "closed": current["id"]}
elif closer is not None:
closer = {**closer, "heading": line}
current = None
continue
field = CONCEPT_FIELD.match(line)
if not field:
continue
if current is not None:
current["fields"][field.group(1).casefold()] = field.group(2).strip()
elif closer is not None:
orphans.append({"line": number, "field": field.group(1).casefold(), **closer})
return definitions, orphans
def controlled_concepts(path: Path) -> dict[str, dict[str, Any]]:
if not path.is_file():
return {}
definitions: dict[str, dict[str, Any]] = {}
for block in schema_definitions(path.read_text(encoding="utf-8"))[0]:
# A repeated ID replaces the earlier definition as a whole.
definitions[block["id"]] = {"id": block["id"], **block["fields"]}
return {
concept_id: {
**definition,
"terms": [definition.get("preferred", "")] + split_pipe(definition.get("aliases", "")),
}
for concept_id, definition in definitions.items()
if definition.get("status") == "active"
}
def matching_run(term_words: list[str], query_words: list[str]) -> Optional[int]:
"""Where a concept term first matches a run of query words, each the same word by spelling or inflection (pure).
Returns the start of the run, or None. Compound parts and typo
corrections never trigger a concept: a confirmed concept must not be
pulled in by a guess.
"""
width = len(term_words)
for start in range(len(query_words) - width + 1):
if all(
wordforms.same_word(query_words[start + offset], term_words[offset], CLOSED_CLASS)
for offset in range(width)
):
return start
return None
def triggered_concepts(
query_words: list[str], definitions: dict[str, dict[str, Any]]
) -> Iterator[tuple[str, list[str], list[str]]]:
"""Every concept the query words trigger, by id: its id, its terms and the query words of its matching run (pure).
A concept is triggered by the first of its terms, in definition order,
that matches a run of the query words (`matching_run`); the run is the
first one that term matches. A blank term, such as an empty Preferred
line, is no term: it never triggers and is not returned.
"""
for concept_id, definition in sorted(definitions.items()):
terms = [term for term in definition.get("terms", []) if term.strip()]
for term in terms:
term_words = tokens(term, drop_stopwords=False)
if not term_words:
continue
start = matching_run(term_words, query_words)
if start is not None:
yield concept_id, terms, query_words[start:start + len(term_words)]
break
def expand_query(query: str, definitions: dict[str, dict[str, Any]]) -> tuple[set[str], list[str], list[str]]:
"""The tokens the ranking matches, the concepts the query triggers, and the terms of those concepts."""
query_words = tokens(query, drop_stopwords=False)
expanded_tokens = set(tokens(query)) or set(query_words)
matched: list[str] = []
expanded_terms: list[str] = []
for concept_id, terms, _run in triggered_concepts(query_words, definitions):
matched.append(concept_id)
for term in terms:
if term not in expanded_terms:
expanded_terms.append(term)
expanded_tokens.update(tokens(term, drop_stopwords=False))
return expanded_tokens, matched, expanded_terms
def query_terms(query: str, expanded_terms: list[str]) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
"""The terms the ranking matches, in a stable order and with their origin, plus dropped words.
The term set equals the token set `expand_query` returns for the same query:
the query's own words without stopwords and single characters (or all of its
words when nothing else is left), then the tokens of every confirmed concept
term the query triggered. Dropped words are reported only when they were
actually left out of the ranking: a stopword that a confirmed concept term
brings back, such as "und" in "Arbeitspreis und Grundpreis", is ranked and
therefore listed as a term of origin `concept-expansion`, not as dropped.
"""
words = tokens(query, drop_stopwords=False)
kept = [word for word in words if word not in FORM_STOPWORDS and len(word) > 1]
terms: list[dict[str, str]] = []
seen: set[str] = set()
for word in kept or words:
if word not in seen:
seen.add(word)
terms.append({"term": word, "origin": "query"})
for expanded in expanded_terms:
for word in tokens(expanded, drop_stopwords=False):
if word not in seen:
seen.add(word)
terms.append({"term": word, "origin": "concept-expansion"})
dropped: list[dict[str, str]] = []
if kept:
for word in words:
if word in seen:
continue
seen.add(word)
dropped.append({"term": word, "reason": "stopword" if word in FORM_STOPWORDS else "single-character"})
return terms, dropped
def claim_fields(match: "re.Match[str]") -> dict[str, Any]:
"""One claim block matched by CLAIM_BLOCK: its metadata and its text with whitespace collapsed."""
metadata: dict[str, str] = {}
for raw_line in match.group("meta").splitlines():
if ":" not in raw_line:
continue
key, value = raw_line.split(":", 1)
metadata[key.strip().casefold()] = value.strip()
return {
"id": metadata.get("id", ""),
"kind": metadata.get("kind", ""),
"status": metadata.get("status", ""),
"sources": split_pipe(metadata.get("sources", "")),
"replaces": split_pipe(metadata.get("replaces", "")),
"contradicts": split_pipe(metadata.get("contradicts", "")),
"text": " ".join(match.group("text").split()),
}
def claims_in(body: str) -> list[dict[str, Any]]:
return [claim_fields(match) for match in CLAIM_BLOCK.finditer(body)]
#: A wiki link with a label, and a Markdown link: the label is what a reader sees.
LABELLED_WIKILINK = re.compile(r"\[\[[^\[\]|]*\|([^\[\]]*)\]\]")
#: A Markdown link; after a "!" it is an image, which `link_labels` leaves as written.
MARKDOWN_LINK = re.compile(r"\[([^\[\]]*)\]\([^()\s]*\)")
def link_labels(text: str) -> str:
"""A text with every labelled wiki link and Markdown link replaced by its label (pure).
An image ("") is no link and stays. MARKDOWN_LINK has no
lookbehind for the "!", which would keep the regular expression engine
from scanning fast for the "["; a match after a "!" is skipped instead,
and the search goes on right after its "[", where a lookbehind would.
"""
if "[" not in text:
return text
text = LABELLED_WIKILINK.sub(lambda match: match.group(1), text)
pieces: list[str] = []
done = 0
match = MARKDOWN_LINK.search(text)
while match is not None:
start = match.start()
if start and text[start - 1] == "!":
match = MARKDOWN_LINK.search(text, start + 1)
continue
pieces.append(text[done:start])
pieces.append(match.group(1))
done = match.end()
match = MARKDOWN_LINK.search(text, done)
if not pieces:
return text
pieces.append(text[done:])
return "".join(pieces)
def visible_body(body: str) -> str:
"""The body without claim metadata and without the targets of labelled links.
A link target is an address, not text. The word-form layer must not let
"[[wiki/concepts/kuendigungsfrist|Kündigungsfrist]]" count the word twice,
once as the folded slug and once as the label.
"""
return link_labels(CLAIM_BLOCK.sub(lambda match: match.group("text"), body))
def overlap(query_tokens: dict[str, str], text_tokens: set[str]) -> int:
"""How many query terms a text holds.
`query_tokens` maps every token that counts to the term it counts for, so
each term counts once, however many of its forms the text holds.
"""
return len({query_tokens[token] for token in text_tokens if token in query_tokens})
def wants_quantities(query_tokens: dict[str, str]) -> bool:
"""Whether a highlight map holds a quantity token, so texts must be read for theirs too."""
return any(token[:1] == QUANTITY_MARK for token in query_tokens)
def relevant_claims(
claims: list[dict[str, Any]],
query_tokens: dict[str, str],
concept_match: bool = False,
) -> list[dict[str, Any]]:
ranked: list[tuple[int, str, dict[str, Any]]] = []
with_quantities = wants_quantities(query_tokens)
for claim in claims:
quantities: Optional[list[str]] = [] if with_quantities else None
claim_tokens = set(tokens(str(claim.get("text") or ""), False, quantities)).union(quantities or ())
held = overlap(query_tokens, claim_tokens)
if held:
ranked.append((held, str(claim.get("id") or ""), claim))
ranked.sort(key=lambda item: (-item[0], item[1]))
if ranked:
return [claim for _, _, claim in ranked[:8]]
if concept_match:
return [claim for claim in claims if claim.get("status") in {"active", "disputed"}][:8]
return []
def best_snippet(body: str, query_tokens: dict[str, str]) -> str:
candidates: list[tuple[int, str]] = []
with_quantities = wants_quantities(query_tokens)
for line in body.splitlines():
compact = " ".join(line.strip().split())
if not compact or compact.startswith("#"):
continue
quantities: Optional[list[str]] = [] if with_quantities else None
line_tokens = set(tokens(compact, False, quantities)).union(quantities or ())
held = overlap(query_tokens, line_tokens)
if held:
candidates.append((held, compact))
if not candidates:
return ""
snippet = max(candidates, key=lambda item: (item[0], len(item[1])))[1]
return snippet if len(snippet) <= 320 else snippet[:317].rstrip() + "..."
def description_of(data: dict[str, Any]) -> str:
"""A page's description, or a source's extraction notes."""
return str(data.get("description") or data.get("extraction_notes") or "")
def document_fields(data: dict[str, Any], body: str) -> dict[str, list[str]]:
"""Split one document into the weighted fields BM25F scores over.
The quantity tokens of the title, aliases, description and body (the
headings are in the body) form a field of their own, so a query without
a quantity scores exactly as it would without them.
"""
quantities: list[str] = []
return {
"title": tokens(str(data.get("title") or ""), False, quantities),
"aliases": tokens(" ".join(alias_values(data.get("aliases"))), False, quantities),
"headings": tokens(" ".join(HEADING.findall(body)), False),
"description": tokens(description_of(data), False, quantities),
"body": tokens(body, False, quantities),
"quantities": quantities,
}
def facets(results: list[dict[str, Any]]) -> dict[str, dict[str, int]]:
output: dict[str, dict[str, int]] = {}
for field in ("type", "status", "language", "clusters", "concepts"):
counts: dict[str, int] = {}
for item in results:
raw = item.get(field)
values = raw if isinstance(raw, list) else [raw] if raw not in (None, "") else []
for value in values:
key = str(value)
counts[key] = counts.get(key, 0) + 1
output[field] = dict(sorted(counts.items(), key=lambda entry: (-entry[1], entry[0])))
return output
def scope_of(relative: str) -> str:
"""The layer a release-relative path belongs to."""
if relative.startswith("sources/"):
return "sources"
if relative.startswith("meta/history/"):
return "history"
return "wiki"
def is_navigation(relative: str) -> bool:
"""True for WIKI.md, every index.md, and meta/questions.md."""
return relative in NAVIGATION_FILES or PurePosixPath(relative).name == "index.md"
def read_document(path: Path, relative: str, text: Optional[str] = None) -> dict[str, Any]:
"""Parse one Markdown file into what ranking and accounting read from it.
Raises FrontmatterError when the file is outside the frontmatter contract.
"""
data, body = frontmatter(path.read_text(encoding="utf-8") if text is None else text, relative)
return parsed_document(data, body, path.stem)
def parsed_document(data: dict[str, Any], body: str, stem: str) -> dict[str, Any]:
"""`read_document` for a file whose frontmatter is already parsed; `stem` titles it when it has no title or H1."""
searchable_body = visible_body(body)
heading = H1.search(searchable_body)
title = str(data.get("title") or (heading.group(1).strip() if heading else stem))
fields = document_fields({**data, "title": title}, searchable_body)
return {
"data": data,
"raw_body": body,
"body": searchable_body,
"title": title,
"fields": fields,
"words": set().union(*fields.values()),
}
def unsearched_words(path: Path, relative: str) -> set[str]:
"""Every whole word of a file that is counted but not ranked.
A file outside the frontmatter contract or the UTF-8 encoding cannot be
ranked, but its words still exist; counting them keeps a damaged file in a
layer nobody searched from hiding a term or from failing the search. A file
that cannot be read at all contributes no words.
"""
try:
text = path.read_bytes().decode("utf-8", errors="replace")
except OSError:
return set()
try:
return read_document(path, relative, text)["words"]
except FrontmatterError:
return set(tokens(text, False))
def scope_paths(target: Path, scope: str) -> list[Path]:
"""The Markdown files of one layer that is counted without being searched."""
root = target / ("sources" if scope == "sources" else "meta/history")
if not root.is_dir():
return []
return [path for path in sorted(root.rglob("*.md")) if path.is_file()]
def has_phrase(query_phrase: str, body_phrase: str) -> bool:
"""Whether a body holds all query words in a row, the phrase signal of the ranking.
Both arguments are space-joined tokens. Only the literal query words
count, and only on whole-token boundaries: "tarif" is no phrase in
"festpreistarif" and none in "tarife".
"""
if not query_phrase:
return False
return f" {query_phrase} " in f" {body_phrase} "
def term_members(
ordered_terms: list[dict[str, str]],
resolver: wordforms.Resolver,
attested_elsewhere: Iterable[str] = (),
) -> dict[str, list[wordforms.Member]]:
"""Per query term, the tokens of the searched files that count for it, with rule and weight.
Query words get word forms and, after them, the heuristics; concept
expansions get word forms only. A term in `attested_elsewhere` occurs, as
itself or as a word form, in a layer that was not searched; it is no typo,
so it is never corrected to another word. `resolver` resolves over the
vocabulary of the searched files; one search builds it once for all its
queries.
"""
attested = set(attested_elsewhere)
return {
item["term"]: resolver.resolve(
item["term"], heuristics=item["origin"] == "query", typos=item["term"] not in attested
)
for item in ordered_terms
}
def layer_resolver(word_sets: list[set[str]]) -> wordforms.Resolver:
"""A word-form resolver over the vocabulary of one unsearched layer."""
vocabulary: dict[str, int] = {}
for words in word_sets:
for token in words:
vocabulary[token] = vocabulary.get(token, 0) + 1
return wordforms.Resolver(vocabulary, CLOSED_CLASS, FORM_STOPWORDS, heuristics=False)
def layer_tokens(term_names: list[str], resolver: wordforms.Resolver) -> dict[str, frozenset]:
"""Per term, the tokens of a layer that are the same word, from that layer's `layer_resolver`."""
return {
term: frozenset(token for token, _, _ in resolver.resolve(term, heuristics=False)) for term in term_names
}
def layer_members(term_names: list[str], word_sets: list[set[str]]) -> dict[str, frozenset]:
"""Per term, the tokens of an unsearched layer that are the same word: literal and word forms only."""
return layer_tokens(term_names, layer_resolver(word_sets))
#: The rules whose members count on a navigation page (WIKI.md, an index.md,
#: meta/questions.md): a term's literal token only. Navigation pages route to
#: content and are never cited; word forms must not lift an index into the
#: top, make a word look common, or count as the word in the coverage block.
NAVIGATION_RULES = frozenset({"exact"})
def members_for(members: list[wordforms.Member], navigation: bool) -> list[wordforms.Member]:
"""The members of a term that count on a page of one role: on a navigation page only NAVIGATION_RULES (pure)."""
return [member for member in members if member[1] in NAVIGATION_RULES] if navigation else members
def scoring_groups(
term_names: list[str],
members_of: dict[str, list[wordforms.Member]],
candidates: list[dict[str, Any]],
) -> tuple[list[bm25.TermGroup], list[bm25.TermGroup], dict[str, str]]:
"""The BM25 term groups for content pages and for navigation pages, and each term's group label.
Terms that are forms of one word ("Tarif" and "Tarife", or a concept term
the query already holds) form one group, so a page is not counted twice for
one word. On navigation pages a group holds only its members of
NAVIGATION_RULES. A group's document frequency counts the pages holding a
word form; only a group with no such page counts the pages holding a
heuristic member, so compound parts and typo corrections never make a word
look common.
"""
merged: dict[str, dict[str, Any]] = {}
label_of: dict[str, str] = {}
for term in term_names:
identity = term if term in CLOSED_CLASS else wordforms.stem_key(term)
entry = merged.setdefault(identity, {"label": term, "members": {}})
label_of[term] = entry["label"]
for token, rule, weight in members_of[term]:
current = entry["members"].get(token)
if current is None or weight > current[1]:
entry["members"][token] = (rule, weight)
content_groups: list[bm25.TermGroup] = []
navigation_groups: list[bm25.TermGroup] = []
for entry in merged.values():
members = entry["members"]
if not members:
continue
on_navigation = frozenset(token for token in members if members[token][0] in NAVIGATION_RULES)
word_forms = frozenset(token for token in members if members[token][0] in wordforms.WORD_FORM_RULES)
heuristics = frozenset(members) - word_forms
def holders(tokens_: frozenset) -> int:
return sum(
1
for candidate in candidates
if not candidate["words"].isdisjoint(on_navigation if candidate["navigation"] else tokens_)
)
frequency = holders(word_forms) or holders(heuristics)
weights = {token: members[token][1] for token in sorted(members)}
content_groups.append(bm25.TermGroup(entry["label"], weights, frequency))
navigation_groups.append(
bm25.TermGroup(entry["label"], {token: weights[token] for token in sorted(on_navigation)}, frequency)
)
return content_groups, navigation_groups, label_of
# -- requests and reformulations ----------------------------------------------
def _invalid_request(detail: str) -> RequestError:
return RequestError(f"request is invalid: {detail}")
def _invalid_reformulations(detail: str) -> RequestError:
return RequestError(f"reformulations are invalid: {detail}")
def request_from_stdin(stream: Any, *, verbatim: bool = False) -> dict[str, Any]:
"""Read one request from a wrapper's standard input and check it (see `read_request`).
Only the stream's bytes are read: the text layer decodes with the host's
locale, which differs between consoles. A terminal or a closed input is
refused instead of waiting for input that never comes.
"""
absent = _invalid_request(
"--request - reads one JSON object from standard input, but standard input is a terminal or closed"
)
buffer = getattr(stream, "buffer", None)
if stream is None or buffer is None:
raise absent
try:
interactive = bool(stream.isatty())
except (AttributeError, ValueError, OSError):
interactive = False
if interactive:
raise absent
try:
raw = buffer.read(MAX_REQUEST_BYTES + 1)
except (OSError, ValueError) as exc:
raise absent from exc
return read_request(raw or b"", verbatim=verbatim)
def _request_text(raw: bytes) -> str:
"""The request bytes as text: UTF-8 with or without BOM, or UTF-16 as PowerShell may send it."""
if raw.startswith(b"\xef\xbb\xbf"):
raw, encoding = raw[3:], "utf-8"
elif raw.startswith((b"\xff\xfe", b"\xfe\xff")):
encoding = "utf-16"
elif len(raw) > 1 and raw[1] == 0:
encoding = "utf-16-le"
elif raw[:1] == b"\x00":
encoding = "utf-16-be"
else:
encoding = "utf-8"
try:
return raw.decode(encoding)
except UnicodeDecodeError as exc:
raise _invalid_request(f"not UTF-8 text; send UTF-8, or {ESCAPE_HINT}") from exc
def read_request(raw: bytes, *, verbatim: bool = False) -> dict[str, Any]:
"""Decode and check one request object `{"query": ..., "also": [...], "filter": {...}}`.
Returns `query` exactly as given, `also` as a list (empty when absent) and
`filter` as the given object or None when absent. The selector itself is
validated by the wrapper, like `--filter-json`. Raises RequestError with a
message ready for the user. `verbatim` is for wording that must keep a
person's punctuation (findability questions): a `?` before a letter then
counts as re-encoded only where the JSON text carries it literally, so the
client sends a real one as `\\u003f`.
"""
if len(raw) > MAX_REQUEST_BYTES:
raise _invalid_request(f"request exceeds {MAX_REQUEST_BYTES} bytes")
text = _request_text(raw).lstrip("").strip()
if not text:
raise _invalid_request("empty input; pipe one JSON object to --request -")
def unique_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
found: dict[str, Any] = {}
for key, item in pairs:
if key in found:
raise _invalid_request(f"duplicate key '{key}'")
found[key] = item
return found
def no_constant(_name: str) -> Any:
raise _invalid_request("NaN and Infinity are not JSON")
try:
value = json.loads(text, object_pairs_hook=unique_keys, parse_constant=no_constant)
except json.JSONDecodeError as exc:
raise _invalid_request(f"not valid JSON: {exc.msg} (line {exc.lineno} column {exc.colno})") from exc
except RecursionError as exc:
raise _invalid_request("not valid JSON: nested too deeply") from exc
if not isinstance(value, dict):
raise _invalid_request("the top level must be a JSON object")
for key in value:
if key not in REQUEST_KEYS:
# The exhaustive scan is a command-line switch; the request keeps its three keys.
hint = "; pass --exhaustive on the command line" if key == "exhaustive" else ""
raise _invalid_request(f"unknown key '{key}'; allowed keys are {', '.join(REQUEST_KEYS)}{hint}")
query = value.get("query")
if not isinstance(query, str) or not query.strip():
raise _invalid_request("query must be a non-empty string")
also = value.get("also", [])
if not isinstance(also, list):
raise _invalid_request("also must be a list of strings")
for index, item in enumerate(also):
if not isinstance(item, str):
raise _invalid_request(f"also[{index}] must be a string")
selector = value.get("filter")
if "filter" in value and not isinstance(selector, dict):
raise _invalid_request("filter must be a JSON object (a selector)")
fields = [("query", query)] + [(f"also[{index}]", item) for index, item in enumerate(also)]
for field, item in fields:
if RE_SURROGATE.search(item):
raise _invalid_request(f"{field} {SURROGATE_DETAIL}")
if "�" in item or (RE_ENCODED.search(item) and (not verbatim or RE_ENCODED.search(text))):
raise _invalid_request(
f"{field} looks re-encoded on the way (a ? before a letter, or U+FFFD); {ESCAPE_HINT}"
+ (", and write a ? that really stands before a letter as \\u003f" if verbatim else "")
)
# The selector is printed back with the result, so it must be printable too.
if selector is not None and any(RE_SURROGATE.search(text) for text in _json_strings(selector)):
raise _invalid_request(f"filter {SURROGATE_DETAIL}")
return {"query": query, "also": list(also), "filter": selector}
def _json_strings(value: Any) -> list[str]:
"""Every string in a decoded JSON value, object keys included, collected without recursion."""
found: list[str] = []
pending = [value]
while pending:
item = pending.pop()
if isinstance(item, str):
found.append(item)
elif isinstance(item, dict):
pending.extend(item.keys())
pending.extend(item.values())
elif isinstance(item, list):
pending.extend(item)
return found
def check_also(also: list[str]) -> None:
"""Refuse more than MAX_ALSO reformulations or one longer than MAX_FORMULATION_CHARS after stripping."""
if len(also) > MAX_ALSO:
raise _invalid_reformulations(
f"at most {MAX_ALSO} reformulations may be given ({MAX_ALSO + 1} searches including the query); "
f"got {len(also)}"
)
for position, entry in enumerate(also, 1):
if not isinstance(entry, str):
raise _invalid_reformulations(f"reformulation {position} is not a string")
length = len(entry.strip())
if length > MAX_FORMULATION_CHARS:
raise _invalid_reformulations(
f"reformulation {position} is longer than {MAX_FORMULATION_CHARS} characters ({length})"
)
def formulation_key(text: str) -> str:
"""The words of a formulation in order, without case, accents or punctuation: what makes two the same search."""
return " ".join(TOKEN.findall(normalized(text)))
def plan_formulations(query: str, also: list[str]) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
"""The searches of one call: the query first, then every reformulation that adds a search.
A reformulation without a word, or with the same words in the same order
as the query or an earlier kept reformulation, is dropped and reported with
its 1-based `position` in the given list. Kept entries are numbered
consecutively from 0, the query. Raises RequestError through `check_also`.
"""
check_also(also)
kept: list[dict[str, Any]] = [{"index": 0, "text": query, "origin": "query"}]
seen = {formulation_key(query): 0}
dropped: list[dict[str, Any]] = []
for position, raw in enumerate(also, 1):
text = raw.strip()
key = formulation_key(text)
if not key:
dropped.append({"position": position, "text": text, "reason": "empty", "duplicate_of": None})
continue
if key in seen:
dropped.append({"position": position, "text": text, "reason": "duplicate", "duplicate_of": seen[key]})
continue
seen[key] = len(kept)
kept.append({"index": len(kept), "text": text, "origin": "also"})
return kept, dropped
# -- the command line both wrappers share ------------------------------------------
def add_search_arguments(
parser: Any, *, history: bool, limit_help: str, exhaustive_help: str, manifest_help: str
) -> None:
"""Add the search options of both wrappers to an argparse parser, in the order their help lists them.
A wrapper adds its own release option first (the query skill's
`--target`) and words three helps itself; `history` adds
`--include-history`, which only a release with history offers.
"""
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--query", help="Question or retrieval query")
group.add_argument(
"--request",
metavar="-",
help="Read the query, its reformulations and an optional selector as one JSON object from standard input; "
"only '-' is accepted",
)
parser.add_argument(
"--also",
action="append",
default=[],
metavar="REFORMULATION",
help="Reformulation of the query searched in the same call; repeatable, at most 4, each at most 200 characters",
)
parser.add_argument("--limit", type=int, default=12, help=limit_help)
parser.add_argument("--include-sources", action="store_true", help="Also search faithful source Markdown")
if history:
parser.add_argument("--include-history", action="store_true", help="Also search historical snapshots")
parser.add_argument("--exhaustive", action="store_true", help=exhaustive_help)
parser.add_argument("--filter-json", default=None, help="Validated metadata selector as JSON (default: all)")
parser.add_argument("--expect-manifest-sha256", default="", help=manifest_help)
def parse_search_arguments(parser: Any) -> Any:
"""The parsed command line of `add_search_arguments`, with the combinations argparse cannot refuse refused."""
args = parser.parse_args()
if args.request is not None and args.request != "-":
parser.error("--request accepts only '-' (standard input)")
if args.request is not None and args.also:
parser.error("--also cannot be combined with --request; put the reformulations into the request's also list")
return args
def query_from_arguments(args: Any, stdin: Any) -> tuple[str, list[str], Optional[dict[str, Any]]]:
"""The query, its reformulations and the request's filter from the parsed command line.
A wrapper calls this before it touches its release, so standard input is
always drained and invalid input never depends on the release. Raises
RequestError.
"""
if args.request is not None:
request = request_from_stdin(stdin)
query, also, request_filter = request["query"], request["also"], request["filter"]
else:
query, also, request_filter = args.query, list(args.also), None
check_also(also)
if request_filter is not None and args.filter_json is not None:
raise _invalid_request("filter is given both in the request and as --filter-json; give it once")
return query, also, request_filter
def selector_from(request_filter: Optional[dict[str, Any]], filter_json: Optional[str]) -> dict[str, Any]:
"""The validated selector: the request's filter, else --filter-json, else everything. Raises RequestError."""
if request_filter is not None:
try:
return validate_selector(request_filter)
except (ValueError, FrontmatterError) as exc:
raise _invalid_request(f"filter is not a valid selector: {exc}") from exc
try:
return validate_selector(json.loads(filter_json if filter_json is not None else '{"kind":"all"}'))
except (json.JSONDecodeError, ValueError, FrontmatterError) as exc:
raise RequestError(f"filter-json is invalid: {exc}") from exc
# -- one release, any number of queries ------------------------------------------
class LoadedRelease:
"""Everything a search reads from one release that does not depend on the query.
`load()` builds it once per call: every searched file read, parsed and
tokenized, the selector applied, the unsearched layers counted, the
vocabularies, the word-form resolvers and the BM25 corpus statistics.
`rank()` ranks any number of queries over it without reading a file again.
"""
def __init__(self, **fields: Any) -> None:
self.__dict__.update(fields)
def load(
root: Path,
*,
include_sources: bool = False,
include_history: bool = False,
selector: Optional[dict[str, Any]] = None,
history_shipped: bool = True,
path_prefix: str = "",
exhaustive: bool = False,
) -> LoadedRelease:
"""Read the release under `root` once for every query a call ranks (see `search` for the options).
With `exhaustive`, which needs `include_sources`, every ranked file is also
cut into passages and indexed for the exhaustive scan, from the same text
the ranking parses: each file is still read once. The scan index is not
built when any file is outside the frontmatter contract, because the
wrappers refuse the results then.
"""
if include_history and not history_shipped:
raise ValueError("this release ships no history layer to search")
if exhaustive and not include_sources:
raise ValueError("an exhaustive search also ranks the sources; pass include_sources=True")
selector = selector if selector is not None else {"kind": "all"}
concept_definitions = controlled_concepts(root / "schema/CONCEPTS.md")
searched_scopes = {"wiki": True, "sources": include_sources, "history": include_history}
# Sources are always counted, searched or not; history only when searched.
counted_scopes = {scope: searched_scopes[scope] or scope == "sources" for scope in SCOPES}
scope_files = {scope: 0 for scope in SCOPES}
searched = 0
filtered_out = 0
parser_errors: list[dict[str, str]] = []
# BM25 needs corpus-wide term rarity and average field lengths, and the
# word-form layer needs the vocabulary of every searched file, so every
# file is read before any of them is scored or counted.
searched_words: list[tuple[str, set[str], bool]] = []
candidates: list[dict[str, Any]] = []
scanned_files: list[tuple[str, str, str, dict[str, Any], bool]] = []
for path in candidate_paths(root, include_sources, include_history):
searched += 1
relative = path.relative_to(root).as_posix()
scope = scope_of(relative)
scope_files[scope] += 1
# The exhaustive scan segments the very text the ranking parses, so
# the file is read here, once, exactly as read_document reads it.
text = path.read_text(encoding="utf-8") if exhaustive else None
try:
document = read_document(path, relative, text)
except FrontmatterError as exc:
parser_errors.append({"path": path_prefix + relative, "error": str(exc)})
continue
# Counted before the selector applies: a metadata restriction narrows
# the results, but must not make an existing term look absent.
navigation = is_navigation(relative)
searched_words.append((scope, document["words"], navigation))
selected = matches_selector(document["data"], relative, selector)
if exhaustive and not parser_errors:
scanned_files.append((relative, scope, str(text), document, selected))
if not selected:
filtered_out += 1
continue
candidates.append(
{
"path": path,
"relative": relative,
"data": document["data"],
"title": document["title"],
"body": document["body"],
"raw_body": document["raw_body"],
"fields": document["fields"],
"words": document["words"],
"navigation": navigation,
}
)
unsearched: dict[str, list[set[str]]] = {}
for scope in SCOPES:
if counted_scopes[scope] and not searched_scopes[scope]:
for path in scope_paths(root, scope):
relative = path.relative_to(root).as_posix()
scope_files[scope] += 1
unsearched.setdefault(scope, []).append(unsearched_words(path, relative))
# Which tokens count for which term is decided on every searched file,
# before the selector, so a selector never changes what a word matches.
# On navigation pages only NAVIGATION_RULES count, so a member of another
# rule counts in the files of content_vocabulary only.
vocabulary: dict[str, int] = {}
content_vocabulary: dict[str, int] = {}
for _scope, words, navigation in searched_words:
for token in words:
vocabulary[token] = vocabulary.get(token, 0) + 1
if not navigation:
content_vocabulary[token] = content_vocabulary.get(token, 0) + 1
return LoadedRelease(
include_sources=include_sources,
history_shipped=history_shipped,
exhaustive=exhaustive,
# The scan index; None without `exhaustive` and after a parser error.
scan=(
ScanIndex(scanned_files, include_history, history_shipped, path_prefix, selector != {"kind": "all"})
if exhaustive and not parser_errors
else None
),
path_prefix=path_prefix,
concept_definitions=concept_definitions,
searched_scopes=searched_scopes,
counted_scopes=counted_scopes,
scope_files=scope_files,
searched=searched,
filtered_out=filtered_out,
parser_errors=parser_errors,
searched_words=searched_words,
candidates=candidates,
unsearched=unsearched,
# Unsearched layers count the literal token and its word forms only.
layer_resolvers={scope: layer_resolver(word_sets) for scope, word_sets in unsearched.items()},
vocabulary=vocabulary,
content_vocabulary=content_vocabulary,
resolver=wordforms.Resolver(vocabulary, CLOSED_CLASS, FORM_STOPWORDS),
corpus=bm25.Corpus(candidate["fields"] for candidate in candidates),
)
def candidate_phrase(candidate: dict[str, Any]) -> str:
"""A candidate's body as space-joined tokens for the phrase signal, built once per call."""
phrase = candidate.get("phrase")
if phrase is None:
phrase = candidate["phrase"] = " ".join(candidate["fields"]["body"])
return phrase
def candidate_claims(candidate: dict[str, Any]) -> list[dict[str, Any]]:
"""A candidate's claim blocks, parsed once per call and only for printed entries."""
claims = candidate.get("claims")
if claims is None:
claims = candidate["claims"] = claims_in(candidate["raw_body"])
return claims
class Ranking:
"""One query ranked over a loaded release.
`ordered` holds every match sorted by score, then path, as result objects
whose `snippet`, `claims`, `claims_total` and `claims_returned` are still
empty; `full(i)` returns entry i completed. `rank_of` maps each matched
path to its 1-based rank. `coverage`, `facets`, `matched_concepts` and
`expanded_terms` describe this query alone. `terms` (the ordered term
dicts) and `members` (term -> list of (token, rule, weight)) are internal:
the exhaustive scan matches every term in these members, so a report
never lists different word forms for one word in two places; it only adds
its own `hyphen_part` tokens and identifiers (see `scan_members`).
`quantities` (query word -> its quantity tokens, `query_quantities`) is
internal too: the scan matches them on every scanned page, whether or not
the ranking took them as members.
"""
def __init__(
self,
matches: list[tuple[dict[str, Any], dict[str, Any], bool]],
highlight: dict[str, str],
coverage: dict[str, Any],
matched_concepts: list[str],
expanded_terms: list[str],
terms: list[dict[str, str]],
members: dict[str, list[wordforms.Member]],
quantities: dict[str, list[str]],
) -> None:
self.terms = terms
self.members = members
self.quantities = quantities
self.ordered = [result for result, _, _ in matches]
self._pending = [(candidate, concept_match) for _, candidate, concept_match in matches]
self._highlight = highlight
self.rank_of = {result["path"]: number for number, result in enumerate(self.ordered, 1)}
self.coverage = coverage
self.facets = facets(self.ordered)
self.matched_concepts = matched_concepts
self.expanded_terms = expanded_terms
def full(self, index: int) -> dict[str, Any]:
"""Entry `index` of `ordered` with its snippet and claims, as a new object."""
result = dict(self.ordered[index])
candidate, concept_match = self._pending[index]
page_claims = candidate_claims(candidate)
claims = relevant_claims(page_claims, self._highlight, concept_match)
result["snippet"] = best_snippet(candidate["body"], self._highlight)
result["claims"] = claims
result["claims_total"] = len(page_claims)
result["claims_returned"] = len(claims)
return result
def query_quantities(query: str, ordered_terms: list[dict[str, str]]) -> dict[str, list[str]]:
"""Per query word, the quantity tokens of the quantities it is written in (pure).
"2500 kWh" gives "=2500kwh" to "2500" and to "kwh"; "75 €" gives
"=75eur" to "75" alone. A concept expansion gets none.
"""
words = {item["term"] for item in ordered_terms if item["origin"] == "query"}
found: dict[str, list[str]] = {}
for sources, token in spelled_quantities(number_spelling(normalized(wordforms.repair_text(query)))):
for source in sources:
if source in words and token not in found.setdefault(source, []):
found[source].append(token)
return found
def rank(loaded: LoadedRelease, query: str, *, limit: int) -> Ranking:
"""Rank one query over a loaded release and account for what its top `limit` does not show.
Raises QueryError for a query without a searchable token.
"""
query_token_set, matched_concepts, expanded_terms = expand_query(query, loaded.concept_definitions)
if not query_token_set:
raise QueryError("query must contain at least one searchable token")
ordered_terms, dropped_terms = query_terms(query, expanded_terms)
term_names = [item["term"] for item in ordered_terms]
searched_scopes = loaded.searched_scopes
counted_scopes = loaded.counted_scopes
include_sources = loaded.include_sources
vocabulary = loaded.vocabulary
content_vocabulary = loaded.content_vocabulary
candidates = loaded.candidates
files_with_term = {scope: 0 for scope in SCOPES}
term_files = {scope: {term: 0 for term in term_names} for scope in SCOPES}
def account(scope: str, words: set[str], tokens_of: dict[str, frozenset]) -> None:
present = [term for term in term_names if not tokens_of[term].isdisjoint(words)]
for term in present:
term_files[scope][term] += 1
if present:
files_with_term[scope] += 1
# Unsearched layers count the literal token and its word forms only.
for scope, word_sets in loaded.unsearched.items():
tokens_of = layer_tokens(term_names, loaded.layer_resolvers[scope])
for words in word_sets:
account(scope, words, tokens_of)
attested_unsearched = {
term for term in term_names if any(term_files[scope][term] for scope in loaded.unsearched)
}
members_of = term_members(ordered_terms, loaded.resolver, attested_unsearched)
# A quantity token becomes a member of a query word only where it bridges a
# notation: when at least one searched non-navigation page holds the token
# but none of the word's other members. That is decided once per query
# word, over all searched files, before the selector, like every other
# member. After that the member counts on every page that holds the token,
# also on a page that writes the quantity as the query does: such a page
# gains the member on top of the words it already holds and can rise
# ("2500 kWh" on a page with "2500 kWh" while another page writes
# "2,5 MWh"). A page without the token keeps its score.
quantity_members = query_quantities(query, ordered_terms)
for term, found in quantity_members.items():
known = {token for token, _, _ in members_of[term]}
added = [
(token, QUANTITY_RULE, wordforms.WEIGHTS[QUANTITY_RULE])
for token in found
if token in content_vocabulary and token not in known and any(
token in words and words.isdisjoint(known) for _scope, words, navigation in loaded.searched_words if not navigation
)
]
if added:
members_of[term] = sorted(members_of[term] + added, key=lambda member: (-member[2], member[0]))
def role_view(navigation: bool, rules: Optional[frozenset] = None) -> dict[str, frozenset]:
# Per term, the tokens that count on a page of this role (members_for), of `rules` only when given.
return {
term: frozenset(
token for token, rule, _ in members_for(members_of[term], navigation) if rules is None or rule in rules
)
for term in term_names
}
# The view of each page role (navigation or not): every token that counts,
# and the same word, the literal token and its word forms, never a compound part or typo.
counting = {navigation: role_view(navigation) for navigation in (False, True)}
same_word = {navigation: role_view(navigation, wordforms.WORD_FORM_RULES) for navigation in (False, True)}
member_tokens = counting[False]
same_word_searched: set[str] = set()
for scope, words, navigation in loaded.searched_words:
account(scope, words, counting[navigation])
same_word_searched.update(term for term in term_names if not same_word[navigation][term].isdisjoint(words))
corpus = loaded.corpus
matched_concept_set = set(matched_concepts)
query_phrase = " ".join(tokens(query, False))
content_groups, navigation_groups, label_of = scoring_groups(term_names, members_of, candidates)
groups = {False: content_groups, True: navigation_groups}
# What claims and snippets are matched with: every counting token mapped to the word it counts for.
highlight: dict[str, str] = {}
for term in term_names:
for token in sorted(member_tokens[term]):
highlight.setdefault(token, label_of[term])
# A file holding no token that counts for any term scores 0, because the
# ranking never turns a non-match into a match; it is not scored at all.
scored_tokens = frozenset().union(*member_tokens.values())
path_prefix = loaded.path_prefix
matches: list[tuple[dict[str, Any], dict[str, Any], bool]] = []
for candidate in candidates:
if candidate["words"].isdisjoint(scored_tokens):
continue
data = candidate["data"]
relative = candidate["relative"]
navigation = candidate["navigation"]
document_concepts = set(as_list(data.get("concepts")))
counted: dict[str, list[str]] = {}
score = bm25.score(
corpus,
candidate["fields"],
groups[navigation],
phrase_match=has_phrase(query_phrase, candidate_phrase(candidate)),
concept_overlap=len(document_concepts & matched_concept_set),
status=str(data.get("status") or ""),
counted=counted,
words=candidate["words"],
)
if score <= 0:
continue
allowed = counting[navigation]
counted_tokens = {term: set(counted.get(label_of[term], ())) & allowed[term] for term in term_names}
result = {
"path": path_prefix + relative,
"title": candidate["title"],
"id": str(data.get("id") or data.get("source_id") or ""),
"type": str(data.get("type") or ("source" if relative.startswith("sources/") else "")),
"status": str(data.get("status") or ""),
"date": str(data.get("date") or data.get("updated") or ""),
"description": description_of(data),
"language": str(data.get("language") or data.get("content_language") or ""),
"sources": as_list(data.get("sources")),
"clusters": as_list(data.get("clusters")),
"concepts": as_list(data.get("concepts")),
"tags": as_list(data.get("tags")),
"score": round(score, 2),
"trust_tier": trust_contract.trust_tier(data),
"stale_after": str(data.get(freshness.STALE_AFTER) or ""),
"past_expiry": freshness.is_stale(data),
# Filled by Ranking.full() for entries that are printed.
"snippet": None,
"claims": None,
"matched_terms": [term for term in term_names if member_tokens[term] & counted_tokens[term]],
"match_rules": [
{"term": term, "token": token, "rule": rule, "weight": weight}
for term in term_names
for token, rule, weight in members_of[term]
if token in counted_tokens[term]
],
"claims_total": None,
"claims_returned": None,
"navigation": navigation,
}
matches.append((result, candidate, bool(document_concepts & matched_concept_set)))
matches.sort(key=lambda match: (-float(match[0]["score"]), str(match[0]["path"])))
results = [result for result, _, _ in matches]
returned = results[:limit]
omitted = results[limit:]
def files_per_scope(term: str) -> dict[str, Optional[int]]:
return {scope: term_files[scope][term] if counted_scopes[scope] else None for scope in SCOPES}
def in_searched_files(term: str) -> bool:
return any(term_files[scope][term] for scope in SCOPES if searched_scopes[scope])
def counting_files(token: str, rule: str) -> int:
# The searched files in which a member counts: a member of
# NAVIGATION_RULES everywhere, any other only outside navigation pages.
return (vocabulary if rule in NAVIGATION_RULES else content_vocabulary).get(token, 0)
def counting_members(term: str) -> list[tuple[str, str, float, int]]:
found = [(token, rule, weight, counting_files(token, rule)) for token, rule, weight in members_of[term]]
return sorted((member for member in found if member[3]), key=lambda member: (-member[2], -member[3], member[0]))
def forms_of(term: str) -> list[dict[str, Any]]:
return [
{"token": token, "rule": rule, "weight": weight, "files": files}
for token, rule, weight, files in counting_members(term)[:FORMS_LISTED]
]
coverage = {
"format": COVERAGE_FORMAT,
"searched_files": loaded.searched,
"matched": len(results),
"returned": len(returned),
"truncated": len(results) > len(returned),
"omitted_count": len(omitted),
"omitted_top": [{"path": item["path"], "score": item["score"]} for item in omitted[:OMITTED_TOP_LIMIT]],
"scopes": {
scope: {
"searched": searched_scopes[scope],
"shipped": loaded.history_shipped if scope == "history" else True,
"files": loaded.scope_files[scope] if counted_scopes[scope] else None,
"files_with_a_query_term": files_with_term[scope] if counted_scopes[scope] else None,
}
for scope in SCOPES
},
"query_terms": [
{
**item,
"files": files_per_scope(item["term"]),
"forms": forms_of(item["term"]),
"forms_total": len(counting_members(item["term"])),
}
for item in ordered_terms
],
"dropped_terms": dropped_terms,
"unmatched_terms": [
item["term"] for item in ordered_terms if item["origin"] == "query" and not in_searched_files(item["term"])
],
"form_only_terms": [
item["term"]
for item in ordered_terms
if item["origin"] == "query" and item["term"] not in vocabulary and in_searched_files(item["term"])
],
# Decided on the same basis as the unsearched layer: the word itself and
# its word forms. A compound or typo match in the wiki is not the word.
"terms_only_in_unsearched": {
"sources": [
term
for term in term_names
if not include_sources and term_files["sources"][term] and term not in same_word_searched
],
},
"boundary": COVERAGE_BOUNDARY,
}
return Ranking(
matches, highlight, coverage, matched_concepts, expanded_terms, ordered_terms, members_of, quantity_members
)
def found_by(rankings: list[Ranking], path: str) -> list[dict[str, int]]:
"""Every search whose ranking holds `path`, with the page's rank there, in formulation order."""
return [
{"formulation": index, "rank": ranking.rank_of[path]}
for index, ranking in enumerate(rankings)
if path in ranking.rank_of
]
# -- the exhaustive scan (Stufe 4) -------------------------------------------------
class Passage(NamedTuple):
"""One unit of the exhaustive scan: a page header, heading, claim, paragraph, list item, table or open question.
`line` is the 1-based file line where it starts, `locator` the claim ID or
the source page or slide (`page=3`, or `page=3-4` for a paragraph that
runs across a page break), `status` the claim status, `tokens` its own
normalised words and identifiers, and `section` the path of its context,
outermost first: the headings and labels above it, then its parent list
items, then the first row of the table it continues (for a heading
passage: its ancestors only). `opens` is the path this passage heads for
the passages after it (a heading, a label, a list item, the first chunk
of a split table), with the tokens that path holds in `opens_tokens`.
`identifiers` are the tokens this passage holds only as an identifier,
from metadata (claim, source and page IDs) or the target of a wiki link
without a label, and not also as written text; they match only as whole
tokens and are never split at their hyphens.
"""
kind: str
line: int
locator: str
status: str
text: str
tokens: frozenset
section: tuple
opens: tuple = ()
opens_tokens: frozenset = frozenset()
identifiers: frozenset = frozenset()
def _letters_before(text: str, end: int) -> int:
"""Where the run of letters that ends at `end` starts (`end` when there is none)."""
start = end
while start > 0 and text[start - 1].isalpha():
start -= 1
return start
def _skip_space_before(text: str, end: int) -> int:
while end > 0 and text[end - 1] in " \t\n":
end -= 1
return end
def _word_before(text: str, end: int) -> int:
"""Where the word that ends at `end` starts, a hyphenated one ("e-mail") included."""
start = _letters_before(text, end)
while start >= 2 and start < end and text[start - 1] == "-" and text[start - 2].isalpha():
start = _letters_before(text, start - 1)
return start
def suspended_forms(text: str) -> set[str]:
"""The joined forms of every compound a normalised text writes with a suspension hyphen, marked (pure).
"grund- und leistungspreis" gives "grund" joined to every trailing part of
"leistungspreis" of at least SUSPENDED_MIN_PART letters ("grundpreis",
"grundspreis", ...), and "vertragsbeginn und -ende" gives every leading part
of "vertragsbeginn" joined to "ende" ("vertragsende", ...). Chains count
too ("grund-, arbeits- und leistungspreis", "strom-/gas-/wasserversorgung",
"beginn, -dauer und -ende"). Each form carries SUSPENDED_MARK, so it never
stands for a written token: the scan matches a term against these forms
only under SUSPENDED_RULE, where most of them meet nothing.
"""
found: set[str] = set()
longest = wordforms.MAX_COMPOUND
for match in SUSPENDED_FRONT.finditer(text):
end = match.start()
if not end or not text[end - 1].isalpha():
continue
other = LETTERS.match(text, match.end())
if other is None or len(other.group(0)) > longest:
continue
word = other.group(0)
parts = [word[cut:] for cut in range(1, len(word) - SUSPENDED_MIN_PART + 1)]
for _ in range(8):
start = _word_before(text, end)
written = text[start:end]
if not written or len(written) > longest:
break
found.update(SUSPENDED_MARK + written + part for part in parts)
# An earlier part of a chain: "grund-, arbeits- und ..." or "strom-/gas-/...".
before = _skip_space_before(text, start)
if before >= 2 and text[before - 1] in ",/" and text[before - 2] == "-":
end = before - 2
continue
break
for match in SUSPENDED_BACK.finditer(text):
other = LETTERS.match(text, match.end())
if other is None or len(other.group(0)) > longest:
continue
# Walk back over earlier suspended parts of a chain ("beginn, -dauer und -ende") to the written word.
end = _skip_space_before(text, match.start())
word = ""
for _ in range(8):
start = _word_before(text, end)
if start == end:
break
if start and text[start - 1] == "-" and (start < 2 or not text[start - 2].isalpha()):
end = _skip_space_before(text, start - 1)
if end and text[end - 1] in ",&":
end = _skip_space_before(text, end - 1)
continue
stop = end - 1 if end and text[end - 1] == "." else end
conjunction = _letters_before(text, stop)
if text[conjunction:stop] not in SUSPENDED_CONJUNCTIONS:
break
end = _skip_space_before(text, conjunction)
continue
word = text[start:end]
break
if not word or len(word) > longest:
continue
found.update(
SUSPENDED_MARK + word[:cut] + other.group(0) for cut in range(SUSPENDED_MIN_PART, len(word))
)
return found
def _sentence_start(text: str, end: int, floor: int = 0) -> int:
"""Where the sentence that holds `text[end]` starts in a normalised text, at `floor` at the earliest (pure).
A sentence ends at !, ?, ; or :, and at a full stop before whitespace or
the end of the text, except after a number ("1. januar"), a single letter
or an abbreviation ("abs.", "nr.").
"""
while True:
found = max(text.rfind(mark, floor, end) for mark in ".!?;:")
if found < 0:
return floor
if text[found] != ".":
return found + 1
if text[found + 1:found + 2].isspace():
start = found
while start > floor and not text[start - 1].isspace():
start -= 1
bare = text[start:found].lstrip(WORD_OPENERS)
if len(bare) >= 2 and "." not in bare and not bare[-1].isdigit() and bare not in SENTENCE_ABBREVIATIONS:
return found + 1
end = found
def separable_splits(
text: str, excluded: frozenset, words: Optional[Iterable[str]] = None
) -> list[tuple[str, list[str]]]:
"""Each separable particle that ends a clause of a normalised text, with the words before it that may be its verb (pure).
A particle ends a clause before a punctuation mark, a conjunction or the
end of the text, after any closing quotes or brackets ("lehnt sie mit
einem antwortcode ab."). Its verb is a word of the same sentence before
it, of at least SEPARABLE_MIN_STEM letters, outside `excluded`, that ends
in one of VERB_ENDINGS, as a finite verb does ("lehnt", "passen",
"schaltet"), or that is a strong form of the irregular tables the particle
joins to ("zog ... um", "schloss ... ab"; TABLE_SPLIT_VERBS), at most
SPLIT_LOOKBACK_CHARS characters before the particle.
Each word is listed once per particle, in text order, and
the particles in text order. `words`, the text's tokens when the caller
has them, only spares finding which particles occur.
"""
present = PARTICLE_SET.intersection(LETTERS.findall(text) if words is None else words)
if not present:
return []
ends: list[tuple[int, str]] = []
for particle in present:
for match in PARTICLE_WORDS[particle].finditer(text):
if CLAUSE_END.match(text, match.end()) is not None:
ends.append((match.start(), particle))
if not ends:
return []
# The possible verbs are found once; each particle takes those of its sentence.
spots = [(match.start(), match.group(0)) for match in VERB_CANDIDATE.finditer(text)]
table = [(match.start(), match.group(0)) for match in TABLE_SPLIT_VERB.finditer(text)]
if table:
spots = sorted(spots + table)
found: list[tuple[str, list[str]]] = []
# The window [floor, start) only moves forward, so two pointers bound it.
low = high = 0
for start, particle in sorted(ends):
floor = max(0, start - SPLIT_LOOKBACK_CHARS)
while low < len(spots) and spots[low][0] < floor:
low += 1
high = max(high, low)
while high < len(spots) and spots[high][0] < start:
high += 1
if low == high:
continue
sentence = _sentence_start(text, start, floor)
first = low
while first < high and spots[first][0] < sentence:
first += 1
verbs: list[str] = []
for _, word in spots[first:high]:
if word not in excluded and word not in verbs and particle in TABLE_SPLIT_VERBS.get(word, PARTICLE_SET):
verbs.append(word)
if verbs:
found.append((particle, verbs))
return found
def split_verb_tokens(text: str, words: Optional[Iterable[str]] = None) -> set[str]:
"""The marked split forms of the separable verbs a normalised text writes in a main clause (pure).
"der netzbetreiber lehnt sie ab." gives "+lehnt+ab" (and the same for
every other word before the particle that may be a finite verb). They
stand for the joined forms ("ablehnt"), which `ScanIndex.split_forms`
looks up for a term. `words` are the text's tokens when the caller has them.
"""
return {
SPLIT_MARK + verb + SPLIT_MARK + particle
for particle, verbs in separable_splits(text, CLOSED_CLASS, words)
for verb in verbs
}
def scan_words(repaired: str) -> frozenset:
"""The tokens the scan indexes for an already repaired text (pure).
They are the tokens of the word-form layer (`tokens(..., False, True)`),
plus the groups of every number joined across a space, which may have
been two numbers ("100 200") rather than one; plus, where a superscript
digit is glued to a word ("Kündigungsfrist²", which normalisation turns
into "kundigungsfrist2"), the word and the digit apart, while the joined
token stays; plus, where a superscript in pandoc's markup follows a word
("m^2^", "Kündigungsfrist^1^"), the joined token (`superscript_tokens`),
while the word and the mark stay apart; plus the marked joined forms of
compounds written with a suspension hyphen (`suspended_forms`); plus the
marked split forms of separable verbs (`split_verb_tokens`).
"""
apart: list[str] = []
text = normalized(repaired)
spelled = number_spelling(text, apart)
found = frozenset(TOKEN.findall(spelled))
if apart:
found = found.union(apart)
quantities = spelled_quantities(spelled, found)
if quantities:
found = found.union(token for _, token in quantities)
if "^" in repaired:
joined = superscript_tokens(repaired, found)
if joined:
found = found.union(joined)
if SUPERSCRIPT_DIGIT.search(repaired) is not None:
spaced = SUPERSCRIPT_DIGIT.sub(lambda match: " " + match.group(0), repaired)
found = found.union(TOKEN.findall(number_spelling(normalized(spaced))))
if "-" in text:
joined = suspended_forms(text)
if joined:
found = found.union(joined)
if not found.isdisjoint(PARTICLE_SET):
split = split_verb_tokens(text, found)
if split:
found = found.union(split)
return found
def html_text(fragment: str) -> str:
"""The text of an HTML fragment: tags and comments removed, entities decoded, whitespace collapsed (pure).
A subscript is joined to its word and a superscript written as pandoc's
^...^ (`wordforms.repair_markup`), and an inline tag ("<strong>",
"<em>", "<span>") is removed without a space, so "CO<sub>2</sub>" and
"<b>Kündigungs</b>frist" stay one word as a browser shows them; every
other tag stands for a space.
"""
if "<" in fragment:
fragment = HTML_INLINE_TAG.sub("", wordforms.repair_markup(fragment))
fragment = HTML_TAG.sub(" ", fragment)
if "&" in fragment:
fragment = html_unescape(fragment)
return " ".join(fragment.split())
def sentence_end(word: str, following: str) -> bool:
"""Whether `word`, followed by the word `following` ("" at a line end), ends a sentence (pure).
It must end in ., ! or ?, before any closing quotes or brackets, and the
next word must not start in lowercase. A full stop after a number ("am 1.
Januar"), a single letter ("S. 3"), a word with an inner full stop
("z.B.") or an abbreviation ("Abs. 2") ends nothing.
"""
core = word.rstrip(SENTENCE_CLOSERS)
if not core or core[-1] not in ".!?" or following[:1].islower():
return False
if core[-1] != ".":
return True
bare = core.rstrip(".").lstrip(WORD_OPENERS)
if len(bare) < 2 or "." in bare or bare[-1].isdigit():
return False
return normalized(bare) not in SENTENCE_ABBREVIATIONS
def sentence_pieces(line: str) -> list[tuple[str, bool]]:
"""A line cut after every sentence end, each piece with whether it ends a sentence (pure).
The last piece ends a sentence when the line ends with one; whitespace
between the pieces is dropped.
"""
pieces: list[tuple[str, bool]] = []
start = 0
if "." in line or "!" in line or "?" in line:
for match in SENTENCE_GAP.finditer(line):
begin = max(line.rfind(" ", start, match.start()), line.rfind("\t", start, match.start())) + 1
if sentence_end(line[max(begin, start):match.start(1)], match.group(2)):
pieces.append((line[start:match.start(1)], True))
start = match.end(1)
rest = line[start:]
last = rest[max(rest.rfind(" "), rest.rfind("\t")) + 1:]
pieces.append((rest, sentence_end(last, "")))
return pieces
def locator_span(first: str, last: str) -> str:
"""The locator of a passage that starts under `first` and ends under `last`: one of them, or a range like `page=3-4` (pure)."""
if first == last or not first:
return last
if not last:
return first
kind, _, _ = first.partition("=")
last_kind, _, value = last.partition("=")
return f"{first}-{value}" if kind == last_kind else f"{first}-{last}"
def readable(raw: str) -> str:
"""A fragment as the scan reads and shows it: links replaced by their labels, then split words joined (pure)."""
return wordforms.repair_text(link_labels(raw))
def inner_path(relative: str) -> str:
"""The path inside a history snapshot (after `meta/history/<snapshot-id>/`), otherwise the path itself."""
if relative.startswith("meta/history/"):
parts = relative.split("/", 3)
return parts[3] if len(parts) == 4 else ""
return relative
def scan_navigation(relative: str) -> bool:
"""True for WIKI.md and every index.md, decided inside a history snapshot on the inner path.
Unlike the ranking, the scan reads `meta/questions.md`: an open question
that names the asked word is exactly what an absence answer must disclose.
"""
return inner_path(relative) == "WIKI.md" or PurePosixPath(relative).name == "index.md"
def identifier_tokens(*values: Any) -> frozenset:
"""The normalised whole tokens of identifier values: claim, source and page IDs (pure)."""
joined = " ".join(str(value) for value in values if value)
if not joined:
return frozenset()
# IDs are almost always ASCII, where casefolding is all that normalized() does.
return frozenset(TOKEN.findall(joined.lower() if joined.isascii() else normalized(joined)))
def link_addresses(text: str) -> frozenset:
"""The tokens of unlabelled wiki-link targets and the source IDs they name (pure).
A link target is an address: it matches only as a whole token and is
never split at its hyphens, so "[[wiki/entities/tarif-lindenstrom-fix12]]"
never stands for the word "Tarif".
"""
if "[[" not in text:
return frozenset()
found: set[str] = set()
for target in UNLABELLED_WIKILINK.findall(text):
target = normalized(target)
found.update(TOKEN.findall(target))
found.update(SOURCE_ID_IN_TEXT.findall(target))
return frozenset(found)
def claim_identifiers(claim: dict[str, Any]) -> frozenset:
"""The IDs a claim block's metadata names: its own, its sources' (before the @ locator), replaced and contradicted claims."""
return identifier_tokens(
claim["id"],
*(entry.split("@", 1)[0] for entry in claim["sources"]),
*claim["replaces"],
*claim["contradicts"],
)
def _blank_or_marker(line: str) -> bool:
"""True for a blank line or a line that holds only page or slide markers."""
stripped = line.strip()
return not stripped or (stripped.startswith("<!--") and not PAGE_MARKER.sub("", stripped).strip())
def unquoted(line: str) -> str:
"""A line without the block-quote markers at its start ("> ", "> > ", ">>"), otherwise the line itself (pure)."""
if ">" not in line[:4]:
return line
match = QUOTE_PREFIX.match(line)
return line[match.end():] if match else line
def display_width(character: str) -> int:
"""The columns a character takes where pandoc lines up a table (pure).
Two for an East Asian wide or full-width character (the emoji ✅ and ❌,
CJK), none for a combining mark, an enclosing mark or a format character
(an NFD umlaut's U+0308, the soft hyphen, a zero-width space), one for
every other character.
"""
if unicodedata.category(character) in ("Mn", "Me", "Cf"):
return 0
return 2 if unicodedata.east_asian_width(character) in ("W", "F") else 1
def _column_cells(line: str, bounds: list[Optional[int]]) -> list[str]:
"""The cells of one line of a simple or multiline table, cut where the column rule's runs start.
The bounds are display columns, as pandoc lines the columns up: a wide
character takes two, a mark or format character none, and a character of
no width stays in the cell of the character before it.
"""
if not line.isascii():
cuts: list[Optional[int]] = [0]
inner = bounds[1:-1]
column = 0
for index, character in enumerate(line):
width = display_width(character)
if not width:
continue
while len(cuts) <= len(inner) and column >= inner[len(cuts) - 1]:
cuts.append(index)
column += width
cuts.extend([len(line)] * (len(inner) + 1 - len(cuts)))
bounds = [*cuts, None]
return [line[bounds[column]:bounds[column + 1]].strip() for column in range(len(bounds) - 1)]
def _grid_cells(line: str) -> list[str]:
"""The cells of one content line of a grid table, between its pipes."""
inner = line.strip()
inner = inner[1:] if inner.startswith("|") else inner
inner = inner[:-1] if inner.endswith("|") else inner
return [cell.strip() for cell in inner.split("|")]
def _table_rows(
lines: list[str], first: int, last: int, groups: list[list[int]], cells_of: Any
) -> tuple[list[tuple[int, str, list[str]]], list[str]]:
"""Each group of lines as one pipe-style row, with the page markers met up to it, and the markers after the last row.
The fragments of a cell are joined with a line break, so a hyphen at the
end of a fragment is repaired as in a paragraph.
"""
markers_at: dict[int, list[str]] = {}
for number in range(first, last + 1):
if "<!--" in lines[number]:
found = [f"{match.group(1).lower()}={match.group(2)}" for match in PAGE_MARKER.finditer(lines[number])]
if found:
markers_at[number] = found
rows: list[tuple[int, str, list[str]]] = []
pending: list[str] = []
done = first - 1
for group in groups:
for number in range(done + 1, group[-1] + 1):
pending.extend(markers_at.get(number, ()))
done = group[-1]
columns: list[list[str]] = []
for number in group:
line = PAGE_MARKER.sub(" ", lines[number]) if number in markers_at else lines[number]
for column, cell in enumerate(cells_of(line.rstrip())):
if column == len(columns):
columns.append([])
if cell:
columns[column].append(cell)
if any(columns):
rows.append((group[0], "| " + " | ".join("\n".join(column) for column in columns) + " |", pending))
pending = []
for number in range(done + 1, last + 1):
pending.extend(markers_at.get(number, ()))
return rows, pending
def _rule_groups(lines: list[str], start: int, single: bool) -> Optional[tuple[list[list[int]], int]]:
"""The rows from `start` up to a closing rule, and the rule's line, or None when no rule closes them.
Rows are separated by blank lines, unless `single` makes every line a row.
Two blank lines in a row, a heading at column 0 or a fence end the search
without a table.
"""
groups: list[list[int]] = []
current: list[int] = []
blank = False
for number in range(start, len(lines)):
line = lines[number].rstrip()
if not line.strip():
if blank:
return None
blank = True
if current:
groups.append(current)
current = []
continue
blank = False
if COLUMN_RULE.match(line) or FULL_RULE.match(line):
if current:
groups.append(current)
return (groups, number) if groups else None
if line.startswith("#") or FENCE.match(line):
return None
if single:
groups.append([number])
else:
current.append(number)
return None
def pandoc_table(lines: list[str], index: int) -> Optional[tuple[int, int, list[tuple[int, str, list[str]]], list[str]]]:
"""A table in one of pandoc's Markdown forms that starts at `lines[index]`, or None (pure).
The forms are the grid table (rows between +---+ borders, the header above
a +===+ border), the multiline table (pandoc's default for a Word table: a
full-width rule, the header, a column rule of dashes, rows separated by
blank lines, a closing rule), the simple table (a header line, a column
rule, one row per line, up to a blank line) and both without a header.
Returns the number of lines the table takes, the indent of its first line,
its rows as pipe-style text with the index of each row's first line and
the page markers met up to it (the header row first), and the markers
after its last row. Cells are cut at the column rule's runs or at the
grid's pipes, so a cell over several lines stays one cell.
"""
line = lines[index].rstrip()
left = line.lstrip()
depth = len(line) - len(left)
count = len(lines)
previous = lines[index - 1] if index else ""
if left[:1] == "+":
if GRID_BORDER.match(line) is None:
return None
groups: list[list[int]] = []
current: list[int] = []
header_end = 0
number = index + 1
while number < count:
row = lines[number].rstrip()
if GRID_BORDER.match(row):
if current:
groups.append(current)
current = []
if "=" in row and not header_end:
header_end = len(groups)
elif row.lstrip().startswith("|"):
current.append(number)
else:
break
number += 1
if current:
groups.append(current)
if not groups:
return None
if header_end > 1:
groups = [[number for group in groups[:header_end] for number in group]] + groups[header_end:]
rows, trailing = _table_rows(lines, index, number - 1, groups, _grid_cells)
return (number - index, depth, rows, trailing) if rows else None
if left[:1] == "-":
if FULL_RULE.match(line):
# A multiline table with a header: the header lines run up to the column rule.
for rule in range(index + 1, min(count, index + 2 + TABLE_HEADER_MAX_LINES)):
ruled = lines[rule].rstrip()
if not ruled.strip() or FULL_RULE.match(ruled):
return None
if COLUMN_RULE.match(ruled):
break
else:
return None
if rule == index + 1:
return None
found = _rule_groups(lines, rule + 1, False)
if found is None:
return None
body, close = found
groups = [list(range(index + 1, rule))] + body
elif COLUMN_RULE.match(line) and _blank_or_marker(previous):
# A table without a header: between two column rules, rows separated by blank lines or one per line.
found = _rule_groups(lines, index + 1, False)
if found is None:
return None
body, close = found
if all(len(group) == 1 for group in body) or not any(
not lines[number].strip() for number in range(index + 1, close)
):
body = [[number] for group in body for number in group]
rule = index
groups = body
else:
return None
starts = [match.start() for match in re.finditer(r"-+", lines[rule].rstrip())]
bounds: list[Optional[int]] = [0, *starts[1:], None]
rows, trailing = _table_rows(lines, index, close, groups, lambda text: _column_cells(text, bounds))
return (close - index + 1, depth, rows, trailing) if rows else None
if (
index + 1 < count and left and left[0] not in "#|<>" and LIST_ITEM.match(line) is None
and COLUMN_RULE.match(lines[index + 1].rstrip())
and (_blank_or_marker(previous) or FULL_RULE.match(previous.rstrip()))
):
# A simple table: one header line, the column rule, one row per line up to a blank line or a closing rule.
groups = [[index]]
close = index + 1
for number in range(index + 2, count):
row = lines[number].rstrip()
if not row.strip():
break
close = number
if COLUMN_RULE.match(row) or FULL_RULE.match(row):
break
groups.append([number])
if len(groups) == 1:
return None
starts = [match.start() for match in re.finditer(r"-+", lines[index + 1].rstrip())]
bounds = [0, *starts[1:], None]
rows, trailing = _table_rows(lines, index, close, groups, lambda text: _column_cells(text, bounds))
return (close - index + 1, depth, rows, trailing) if rows else None
return None
def scan_passages(relative: str, text: str, document: dict[str, Any]) -> list[Passage]:
"""Cut one file into the passages the exhaustive scan matches (pure).
`document` is what `read_document` returned for `text`, so the frontmatter
is parsed once. Lines are 1-based file lines, split on "\\n" only: a form
feed from a PDF extraction must not shift them. Trailing whitespace is
stripped from every line, so a CRLF file yields the same passages.
Only a blank line or a structural line (heading, list item, table, fence,
claim block, HTML table, heading or summary) ends a paragraph. A line with
a list marker inside a plain paragraph continues it ("ab dem\\n15. Januar"),
unless the paragraph ends with a colon or is a bold label line, and a
number other than 1 indented below a list item continues the item.
Block-quote markers come off every line first, so a quote is read like
its text: a ">" line ends a paragraph, and a label, heading, list or table
inside a quote counts. A page or
slide marker does not: it only sets the locator of what follows, so a
sentence broken across a page stays one passage, with the locator range
of both pages, and a hyphen at the line end before the marker is joined.
A paragraph longer than PASSAGE_MAX_CHARS is cut at a sentence end.
Context reaches a passage from the headings above it (Markdown, setext and
HTML headings, and labels: a one-line paragraph of at most LABEL_MAX_CHARS
characters that is only bold text or ends in a colon, or the summary of an
HTML details block, which lasts until the next heading or label), from its
parent list items, and, in a table cut at the size cap, from the table's
first row. A raw HTML table is read as a table: each row becomes a row of
cells, and its first row is the one a cut carries. So are pandoc's grid,
multiline and simple tables (`pandoc_table`): a row over several lines or
separated by blank lines stays one row of cells, and the header row is
the first row a cut carries. The page header and
every claim also carry the identifiers of their metadata as whole tokens:
page and source IDs, and the claim, source, replaced and contradicted
claim IDs. The targets of unlabelled wiki links are identifiers too
(`link_addresses`).
"""
data = document["data"]
body = document["raw_body"]
body_line = text.count("\n", 0, len(text) - len(body)) + 1
title = str(document["title"])
aliases = alias_values(data.get("aliases"))
description = description_of(data)
header_text = title + (" — " + ", ".join(aliases) if aliases else "") + (" — " + description if description else "")
listed_sources = " ".join(as_list(data.get("sources")))
header_ids = identifier_tokens(data.get("id"), data.get("source_id"))
if "src-" in listed_sources:
header_ids = header_ids.union(SOURCE_ID_IN_TEXT.findall(normalized(listed_sources)))
repair = wordforms.repair_text
header_words = scan_words(repair(title)).union(scan_words(repair(", ".join(aliases))), scan_words(repair(description)))
passages: list[Passage] = [
Passage("header", 1, "", "", header_text, header_words | header_ids, (), identifiers=header_ids - header_words)
]
# The title tokens read_document gave the title field, for an H1 that repeats the title.
title_tokens = document["fields"]["title"]
plain_kind = "open_question" if inner_path(relative) == "meta/questions.md" else "paragraph"
# The context stacks, each as parallel lists: headings and labels (level,
# text) and list items (indent, text). `section` caches their texts.
levels: list[int] = []
labels: list[str] = []
indents: list[int] = []
items: list[str] = []
section: tuple = ()
fenced = False
locator = ""
lines: list[str] = []
first_line = 0
length = 0
table = False
listed = False
indent = 0
table_row = ""
carried: tuple = ()
# The locators in effect at the first and at the last text of the open passage.
first_locator = ""
last_locator = ""
# The lines of a raw HTML table that is still open, as (number, line).
html_block: Optional[list[tuple[int, str]]] = None
def refresh() -> None:
nonlocal section
section = tuple(labels) + tuple(items) if items else tuple(labels)
def pop_headings(level: int) -> None:
while levels and levels[-1] >= level:
levels.pop()
labels.pop()
def written(addresses: frozenset, text_with_links: str) -> frozenset:
# The link targets this passage does not also hold as written text: its identifiers.
if not addresses:
return frozenset()
return addresses - scan_words(UNLABELLED_WIKILINK.sub(" ", text_with_links))
def heading(level: int, label: str, number: int) -> None:
indents.clear()
items.clear()
pop_headings(level)
refresh()
words = tokens(label, False)
# An H1 that repeats the title is carried by the header passage.
if words and not (level == 1 and words == title_tokens):
found = scan_words(label)
addresses = link_addresses(label)
identifiers = written(addresses, label)
found = found | addresses
passages.append(Passage("heading", number, locator, "", label, found, section, section + (label,), found, identifiers))
levels.append(level)
labels.append(label)
refresh()
def finish(cut: bool = False) -> None:
nonlocal lines, length, table, listed, carried
if not lines:
# Most calls come from a blank line after a passage that already ended.
carried = ()
listed = False
return
raw = "\n".join(lines)
# The repaired text is the display text too.
repaired = readable(raw)
found = scan_words(repaired)
addresses = link_addresses(raw)
identifiers = written(addresses, repaired)
if addresses:
found = found | addresses
if found:
shown = " ".join(repaired.split())
label = (
len(lines) == 1 and not listed and not table and not items and len(shown) <= LABEL_MAX_CHARS
and (shown.endswith(":") or BOLD_LABEL.match(shown) is not None)
)
if label:
pop_headings(LABEL_LEVEL)
refresh()
where = section + carried if carried else section
opens: tuple = ()
opens_tokens: frozenset = frozenset()
if listed or label:
opens, opens_tokens = where + (shown,), found
elif table and cut and not carried:
repaired_row = readable(table_row)
row_tokens = scan_words(repaired_row)
if row_tokens:
opens, opens_tokens = where + (" ".join(repaired_row.split()),), row_tokens
span = locator_span(first_locator, last_locator)
passages.append(Passage(plain_kind, first_line, span, "", shown, found, where, opens, opens_tokens, identifiers))
if label:
levels.append(LABEL_LEVEL)
labels.append(shown)
refresh()
elif listed:
indents.append(indent)
items.append(shown)
refresh()
if table and cut and opens:
carried = (opens[-1],)
if not (cut and table):
carried = ()
lines = []
length = 0
table = False
listed = False
def begin(number: int, is_table: bool, size: int) -> None:
nonlocal first_line, table, length, first_locator
first_line = number
table = is_table
length = size
first_locator = locator
def append(line: str, number: int, is_table: bool = False) -> None:
nonlocal length, table_row, last_locator
if is_table:
# A table row is one unit; the table is cut at the row that reaches the cap.
if not lines:
begin(number, True, len(line))
if not carried:
table_row = line
else:
length += 1 + len(line)
lines.append(line)
last_locator = locator
if length >= PASSAGE_MAX_CHARS:
finish(cut=True)
return
if (length + 1 + len(line) if lines else len(line)) < PASSAGE_MAX_CHARS:
# No cut can fall in this line: it joins the passage whole.
if lines:
length += 1 + len(line)
else:
begin(number, False, len(line))
lines.append(line)
last_locator = locator
return
for piece_number, (piece, ends) in enumerate(sentence_pieces(line)):
if not lines:
begin(number, False, len(piece))
lines.append(piece)
else:
length += 1 + len(piece)
if piece_number:
lines[-1] = f"{lines[-1]} {piece}"
else:
lines.append(piece)
last_locator = locator
# Cut at the first sentence end once the cap is reached; the rest of the line starts the next passage.
if length >= PASSAGE_MAX_CHARS and (ends or length >= PASSAGE_HARD_MAX_CHARS):
finish(cut=True)
def enter(depth: int) -> None:
# A new passage that is no list item leaves every list item indented as deep or deeper.
nonlocal indent
indent = depth
if indents and indents[-1] >= depth:
while indents and indents[-1] >= depth:
indents.pop()
items.pop()
refresh()
def table_line(row: str, number: int, depth: int) -> None:
# Consecutive rows form one passage.
if lines and not table:
finish()
if not lines:
enter(depth)
append(row, number, True)
def close_html() -> None:
# Read a raw HTML table as a table: each <tr> a row of cells, the first row carried across a cut.
nonlocal html_block, locator
block, html_block = html_block, None
raw = "\n".join(line for _, line in block)
start = block[0][0]
depth = len(block[0][1]) - len(block[0][1].lstrip())
markers = [(match.start(), f"{match.group(1).lower()}={match.group(2)}") for match in PAGE_MARKER.finditer(raw)]
# Text outside the rows (a caption, an unclosed fragment) is kept as a paragraph.
rest = html_text(HTML_ROW.sub(" ", raw))
if rest:
enter(depth)
append(rest, start)
finish()
for match in HTML_ROW.finditer(raw):
while markers and markers[0][0] < match.end():
locator = markers.pop(0)[1]
cells = [html_text(cell) for cell in HTML_CELL.findall(match.group(1))]
row = "| " + " | ".join(cells) + " |" if cells else html_text(match.group(1))
if row.strip("| "):
table_line(row, start + raw.count("\n", 0, match.start()), depth)
finish()
for _, value in markers:
locator = value
def continues(line: str, left: str) -> bool:
# A line with a list marker continues the open text instead of starting
# an item where a hard-wrapped extraction or pandoc's line wrapping put
# a number or a dash at the start of a line ("ab dem\n15. Januar",
# "mit Modul\n2. Ein Heizstab"): inside a plain paragraph, unless the
# paragraph ends with a colon or is a bold label line; inside a list
# item, only a number other than 1 indented deeper than the item's marker.
if not lines or table:
return False
if listed:
marker = left.split(None, 1)[0]
return marker[0].isdigit() and marker[:-1] != "1" and len(line) - len(left) > indent
last = lines[-1].rstrip()
if last.endswith(":"):
return False
return not (len(lines) == 1 and BOLD_LABEL.match(" ".join(readable(last).split())) is not None)
def handle(line: str, number: int) -> None:
nonlocal fenced, locator, lines, length, listed, html_block
line = line.rstrip()
left = line.lstrip()
if html_block is not None:
# Inside a raw HTML table: everything up to </table> belongs to it; a blank line ends an unclosed one.
if not left:
close_html()
finish()
return
html_block.append((number, line))
if HTML_TABLE_END.search(line):
close_html()
return
# Each pattern runs only where its first characters can match: the scan reads every line.
if left[:1] in ("`", "~") and FENCE.match(line):
finish()
fenced = not fenced
return
if not fenced:
marker = PAGE_MARKER.search(line) if "<!--" in line else None
if marker:
# A marker moves the locator for what follows; it never ends the passage it stands in.
before, after = line[: marker.start()], line[marker.end():]
if before.strip():
handle(before, number)
locator = f"{marker.group(1).lower()}={marker.group(2)}"
if after.strip():
handle(after, number)
return
atx = SCAN_HEADING.match(line) if line[:1] == "#" else None
if atx:
finish()
heading(len(atx.group(1)), " ".join(readable(atx.group(2)).split()), number)
return
if (
left[:1] in ("=", "-") and len(lines) == 1 and not table and not listed
and SETEXT_UNDERLINE.match(line)
):
label = " ".join(readable(lines[0]).split())
if label and len(label) <= LABEL_MAX_CHARS:
number = first_line
lines = []
length = 0
heading(1 if left[0] == "=" else 2, label, number)
return
if left[:1] == "<":
if HTML_TABLE_START.match(left):
finish()
html_block = [(number, line)]
if HTML_TABLE_END.search(line):
close_html()
return
tag = HTML_HEADING.match(left)
summary = None if tag else HTML_SUMMARY.match(left)
if tag or summary:
label = html_text(readable(tag.group(2) if tag else summary.group(1)))
if label:
finish()
heading(int(tag.group(1)) if tag else LABEL_LEVEL, label, number)
return
if HTML_DETAILS_END.match(left):
finish()
pop_headings(LABEL_LEVEL)
refresh()
return
if not left:
finish()
return
first = left[0]
if (first in "-*+" or first.isdigit()) and LIST_ITEM.match(line) and not continues(line, left):
finish()
enter(len(line) - len(left))
listed = True
append(line, number)
return
if first == "|": # a table row
table_line(line, number, len(line) - len(left))
return
if table:
finish()
if not lines:
enter(len(line) - len(left))
append(line, number)
def pandoc_rows(table: tuple[int, int, list[tuple[int, str, list[str]]], list[str]], start: int) -> None:
# A pandoc grid, multiline or simple table is read as a table: each row one row of cells, the header carried across a cut.
nonlocal locator
_, depth, rows, trailing = table
finish()
for offset, row, markers in rows:
if markers:
locator = markers[-1]
table_line(row, start + offset, depth)
finish()
if trailing:
locator = trailing[-1]
position = 0
line_at = body_line
def walk(segment: str, start: int) -> None:
raw_lines = segment.split("\n")
# Block-quote markers come off before every structural test, so a ">"
# line ends a paragraph and a label, heading, list or table inside a
# quote counts (pandoc writes an indented Word paragraph as a quote).
segment_lines = [unquoted(line) for line in raw_lines]
count = len(segment_lines)
index = 0
while index < count:
if html_block is not None:
# Inside a raw HTML table a line is passed on as written.
handle(raw_lines[index], start + index)
index += 1
continue
line = segment_lines[index]
if (
not fenced and html_block is None
and (line.lstrip()[:1] in ("-", "+") or (index + 1 < count and segment_lines[index + 1].lstrip()[:1] == "-"))
):
table = pandoc_table(segment_lines, index)
if table is not None:
pandoc_rows(table, start)
index += table[0]
continue
handle(line, start + index)
index += 1
def close() -> None:
if html_block is not None:
close_html()
finish()
for match in CLAIM_BLOCK.finditer(body):
walk(body[position:match.start()], line_at)
close()
# A claim block stands at column 0: it ends every list.
if items:
indents.clear()
items.clear()
refresh()
claim_line = line_at + body.count("\n", position, match.start())
claim = claim_fields(match)
claim_ids = claim_identifiers(claim)
addresses = link_addresses(match.group("text"))
# Tokens come from the text with its line breaks, so a hyphen at a line end is repaired as the ranking does.
claim_text = readable(match.group("text"))
words = scan_words(claim_text)
# Identifiers are the IDs and link targets the claim does not also hold as written text.
plain = scan_words(UNLABELLED_WIKILINK.sub(" ", claim_text)) if addresses else words
identifiers = (claim_ids | addresses) - plain
found = words | claim_ids | addresses
if found:
passages.append(
Passage("claim", claim_line, claim["id"], claim["status"], claim["text"], found, section, identifiers=identifiers)
)
line_at = claim_line + body.count("\n", match.start(), match.end())
position = match.end()
walk(body[position:], line_at)
close()
return passages
def concept_runs(query: str, definitions: dict[str, dict[str, Any]]) -> dict[str, list[str]]:
"""For each concept the query triggers, the query words of its matching run, as `expand_query` finds them (pure)."""
return {
concept_id: run for concept_id, _terms, run in triggered_concepts(tokens(query, drop_stopwords=False), definitions)
}
def content_words(terms: list[dict[str, str]]) -> list[str]:
"""The words a passage must hold for a formulation: its query words without closed-class and function words."""
return [item["term"] for item in terms if item["origin"] == "query" and item["term"] not in SCAN_EXCLUDED]
def alternatives(
words: list[str], runs: dict[str, list[str]], definitions: dict[str, dict[str, Any]]
) -> dict[str, list[tuple[Optional[str], tuple]]]:
"""Per content word, the word itself and then every confirmed concept term that may stand in for it."""
found: dict[str, list[tuple[Optional[str], tuple]]] = {}
for word in words:
options: list[tuple[Optional[str], tuple]] = [(None, (word,))]
listed = {(word,)}
for concept_id in sorted(runs):
if word not in runs[concept_id]:
continue
for term in definitions[concept_id].get("terms", []):
option = tuple(token for token in tokens(term, False) if token not in SCAN_EXCLUDED)
if option and option not in listed:
listed.add(option)
options.append((concept_id, option))
found[word] = options
return found
def hyphen_groups(units: list[str]) -> list[str]:
"""The written parts of a hyphenated token with every one-character part kept on its neighbour (pure).
A part of one character joins the part after it, or the part before it
when it is the last: "e-mail-adresse" gives "e-mail" and "adresse",
"typ-2-stecker" gives "typ" and "2-stecker", "vitamin-c" gives
"vitamin-c". So "E-Mail" stays one part and "E" is never one.
"""
groups: list[str] = []
pending = ""
for unit in units:
if pending:
unit = pending + "-" + unit
pending = ""
if len(unit) < 2:
pending = unit
continue
groups.append(unit)
if pending:
if groups:
groups[-1] = groups[-1] + "-" + pending
else:
groups.append(pending)
return groups
def hyphen_units(word: str) -> tuple:
"""The parts of a hyphenated content word that a passage may write apart, or () (pure).
A one-character part stays on its neighbour (`hyphen_groups`), so
"E-Mail-Adresse" gives "e-mail" and "adresse", and "E-Mobilität" gives
no parts at all: it never stands for "Mobilität" alone. Closed-class and
function words among the parts are dropped; at least two distinct parts
must remain.
"""
if "-" not in word and "_" not in word:
return ()
units = tuple(dict.fromkeys(unit for unit in hyphen_groups(wordforms.units_of(word)) if unit not in SCAN_EXCLUDED))
if len(units) < 2:
return ()
return units
def shown_token(token: str) -> str:
"""A member token as a report prints it: a joined form of a suspended compound without its mark, a split verb as "lehnt … ab"."""
if token[:1] == SUSPENDED_MARK:
return token[1:]
if token[:1] == SPLIT_MARK:
verb, _, particle = token[1:].partition(SPLIT_MARK)
return f"{verb} … {particle}"
return token
def ending_bases(spelling: str) -> set[str]:
"""The bases of a folded spelling: itself and itself without one of INFLECTION_ENDINGS (pure).
A base keeps at least ENDING_MIN_STEM letters, and a spelling shorter than
that has none. A base of at least DEGREE_MIN_STEM letters may also be the
spelling without one of DEGREE_ENDINGS ("gunstigsten" gives "gunstig") or
FEMININE_ENDINGS ("kundinnen" gives "kund"). Two words share a base when
the `ending` rule matches them.
"""
if not spelling.isalpha() or len(spelling) < ENDING_MIN_STEM:
return set()
bases = {spelling}
for endings, shortest in (
(INFLECTION_ENDINGS, ENDING_MIN_STEM), (DEGREE_ENDINGS, DEGREE_MIN_STEM), (FEMININE_ENDINGS, DEGREE_MIN_STEM),
):
bases.update(
spelling[: -len(ending)]
for ending in endings
if spelling.endswith(ending) and len(spelling) - len(ending) >= shortest
)
return bases
def ending_variants(spelling: str) -> set[str]:
"""The spellings that share a base with a folded spelling, as the `ending` rule compares (pure).
Every base of `ending_bases` with nothing or one of INFLECTION_ENDINGS
after it, and every base of at least DEGREE_MIN_STEM letters with one of
DEGREE_ENDINGS or FEMININE_ENDINGS: "intelligenten" gives "intelligentem"
and "intelligenter" (through "intelligent"), "tag" gives "tage", "auto"
gives "autos", "gunstiger" gives "gunstigste", "kunden" gives "kundin" and
"kundinnen". The spelling itself is left out.
"""
variants: set[str] = set()
for base in ending_bases(spelling):
variants.update(base + ending for ending in ("", *INFLECTION_ENDINGS))
if len(base) >= DEGREE_MIN_STEM:
variants.update(base + ending for ending in (*DEGREE_ENDINGS, *FEMININE_ENDINGS))
variants.discard(spelling)
return variants
def verb_stems(spelling: str) -> set[str]:
"""The verb stems a folded spelling may hold: without one of VERB_ENDINGS, or inside a participle ge-...-t (pure)."""
if not spelling.isalpha():
return set()
stems = {
spelling[: -len(ending)]
for ending in VERB_ENDINGS
if spelling.endswith(ending) and len(spelling) - len(ending) >= VERB_MIN_STEM
}
if spelling.startswith("ge") and spelling.endswith("t"):
inner = spelling[2:-2] if spelling.endswith("et") else spelling[2:-1]
if len(inner) >= VERB_MIN_STEM:
stems.add(inner)
return stems
def separable_stems(spelling: str) -> set[tuple[str, str]]:
"""The (particle, stem) pairs of a separable verb that a folded spelling may be a form of (pure).
A spelling that starts with one of SEPARABLE_PARTICLES is read three ways:
particle, stem and one of VERB_ENDINGS ("abschalten", "abschaltete");
particle, ge-, stem and -t, -et or -en, the participle ("abgeschaltet",
"angepasst", "angegeben"); and particle, zu-, stem and -n or -en, the
infinitive with zu ("abzuschalten"). Each stem keeps at least
SEPARABLE_MIN_STEM letters.
"""
pairs: set[tuple[str, str]] = set()
if not spelling.isalpha():
return pairs
for particle in SEPARABLE_PARTICLES:
if not spelling.startswith(particle):
continue
rest = spelling[len(particle):]
if len(rest) < SEPARABLE_MIN_STEM:
continue
stems = {
rest[: -len(ending)]
for ending in VERB_ENDINGS
if rest.endswith(ending) and len(rest) - len(ending) >= SEPARABLE_MIN_STEM
}
for infix, endings in (("ge", ("t", "et", "en")), ("zu", ("n", "en"))):
if rest.startswith(infix):
stems.update(
rest[2: -len(ending)]
for ending in endings
if rest.endswith(ending) and len(rest) - 2 - len(ending) >= SEPARABLE_MIN_STEM
)
pairs.update((particle, stem) for stem in stems)
return pairs
def verb_variants(spelling: str) -> set[str]:
"""The other verb forms of a folded spelling, as the `verb_ending` rule compares (pure).
Every stem of `verb_stems` with each of VERB_ENDINGS and as a participle
ge-...-t and ge-...-et: "senkt" gives "senken", "senkte" and "gesenkt".
Every pair of `separable_stems` with each of VERB_ENDINGS after the stem,
as a participle with ge- after the particle (-t, -et, -en), and as an
infinitive with zu- after the particle: "abschalten" gives "abgeschaltet"
and "abzuschalten", "abgeschaltet" gives "abschalten". Every base of
`english_bases` in the forms of `english_forms`: "cancelled" gives
"cancel" and "cancelling", "apply" gives "applied".
"""
variants: set[str] = set()
for stem in verb_stems(spelling):
variants.update(stem + ending for ending in VERB_ENDINGS)
variants.update(("ge" + stem + "t", "ge" + stem + "et"))
for particle, stem in separable_stems(spelling):
variants.update(particle + stem + ending for ending in VERB_ENDINGS)
variants.update(particle + "ge" + stem + ending for ending in ("t", "et", "en"))
variants.update((particle + "zu" + stem + "en", particle + "zu" + stem + "n"))
for base in english_bases(spelling):
variants.update(english_forms(base))
variants.discard(spelling)
return variants
def english_bases(spelling: str) -> set[str]:
"""The bases a folded spelling may be a regular English verb form of, itself included (pure).
-ied and -ies give the base in -y ("applied", "applies": "apply"); -ed
and -ing give the base without them, with an e added, and with a doubled
final consonant undone ("cancelled": "cancell", "cancelle", "cancel";
"charging": "charg", "charge"). A base keeps ENGLISH_MIN_BASE letters.
"""
if not spelling.isalpha():
return set()
bases = {spelling}
if spelling.endswith(("ied", "ies")):
bases.add(spelling[:-3] + "y")
for suffix in ("ed", "ing"):
if spelling.endswith(suffix):
base = spelling[: -len(suffix)]
bases.update((base, base + "e"))
if len(base) >= 2 and base[-1] == base[-2] and base[-1] not in VOWELS:
bases.add(base[:-1])
return {base for base in bases if len(base) >= ENGLISH_MIN_BASE}
def english_forms(base: str) -> set[str]:
"""The regular English verb forms of a base, the base included (pure).
The base, -ed and -ing; -d and -ing without the e after an e ("charge":
"charged", "charging"); -ied and -ies after a consonant and y ("apply":
"applied", "applies"); the final consonant doubled after one vowel
("cancel": "cancelled", "cancelling"; "stop": "stopped").
"""
forms = {base, base + "ed", base + "ing"}
last, before = base[-1], base[-2:-1]
if last == "e":
forms.update((base + "d", base[:-1] + "ing"))
elif last == "y" and before and before not in VOWELS:
forms.update((base[:-1] + "ied", base[:-1] + "ies"))
elif last not in VOWELS + "wxy" and before and before in VOWELS and base[-3:-2] not in VOWELS:
forms.update((base + last + "ed", base + last + "ing"))
return forms
def scan_members(ranking: Ranking, scan: "ScanIndex", vocabulary: dict[str, int]) -> dict[Any, list[wordforms.Member]]:
"""The members the scan matches per term: the ranking's, plus what only the scan can see.
Three additions, all for terms outside the closed class and function words:
- a scanned token that the ranking's vocabulary lacks and that is the
term itself or has its spelling or stem key gets the word-form rule
`exact`, `fold` or `stem`; that happens for an identifier from claim
metadata or the frontmatter, which the ranking never reads, and for a
word only the scan's segmentation repairs (a hyphen at a line end
before a page marker, a number in a table cell); for a query word, such
a token that the irregular tables pair with it gets `irregular` or
`derived`, as in the ranking (see `ScanIndex.scan_only_table_forms`);
- every joined token written as text that holds the term as a
hyphen-separated part of the same stem key ("sms" in "sms-warndienst")
is added under the heuristic rule `hyphen_part`, which the ranking does
not know;
- every scanned token that is the term with an inflection ending added
or removed where the stem rule cannot see it ("krankenhauser" for
"krankenhaus", "tage" for "tag") is added under the heuristic rule
`ending` (see `ScanIndex.ending_forms`);
- every scanned token that is another verb form of a four-letter stem
("senken" for "senkt", "zahlt" for "zahlen"), of a separable verb or of
an English verb ("cancelled" for "cancel") is added under the
heuristic rule `verb_ending` (see `ScanIndex.verb_forms`);
- every split form of a separable verb that a passage writes in a main
clause and that is a form of the term ("lehnt … ab" for "ablehnen") is
added under `split_verb` (see `ScanIndex.split_forms`);
- every scanned token that glues a one- or two-digit footnote mark to the
term's stem key ("kundigungsfrist2") is added under `footnote_mark`, and
every marked joined form of a suspended compound with the term's stem
key ("grund- und leistungspreis" for "grundpreis") under
`suspended_compound`;
- for a term of two or three letters, every scanned token that holds it
as its first or last part ("gaskunden", "erdgas" for "gas") is added
under `short_part` (see `ScanIndex.short_parts`).
For every term, also one of the closed class, each quantity token of its
query word (`Ranking.quantities`) that the scan holds is added under the
heuristic rule `quantity`, whether or not the ranking took it as a member.
"""
parts = scan.hyphen_parts()
footnotes = scan.footnote_marks()
suspended = scan.suspended()
members: dict[Any, list[wordforms.Member]] = {}
for item in ranking.terms:
term = item["term"]
listed = list(ranking.members.get(term, ()))
if term not in SCAN_EXCLUDED:
known = {token for token, _, _ in listed}
scan_only = scan.scan_only_forms(term, vocabulary)
if item["origin"] == "query":
scan_only += scan.scan_only_table_forms(term, vocabulary)
for token, rule in scan_only:
if token not in known:
member = (token, rule, wordforms.WEIGHTS[rule])
if rule == "exact":
listed.insert(0, member)
else:
listed.append(member)
known.add(token)
key = wordforms.stem_key(term)
for found, rule in (
(parts.get(key, ()), HYPHEN_PART_RULE),
(scan.ending_forms(term), ENDING_RULE),
(scan.verb_forms(term), VERB_ENDING_RULE),
(scan.split_forms(term), SPLIT_VERB_RULE),
(footnotes.get(key, ()), FOOTNOTE_RULE),
(suspended.get(key, ()), SUSPENDED_RULE),
(scan.short_parts(term), SHORT_PART_RULE),
):
for token in sorted(found):
if token not in known:
listed.append((token, rule, wordforms.WEIGHTS[rule]))
known.add(token)
known = {token for token, _, _ in listed}
for token in ranking.quantities.get(term, ()):
if token not in known and token in scan.own:
listed.append((token, QUANTITY_RULE, wordforms.WEIGHTS[QUANTITY_RULE]))
members[term] = listed
return members
def _word_sequence(text: str, words: frozenset, memo: dict[str, bool]) -> bool:
"""Whether a folded spelling is one or more of `words` joined by linking elements (the last one by its stem too)."""
known = memo.get(text)
if known is not None:
return known
found = text in words or wordforms.stem(text) in words
if not found and len(text) <= wordforms.MAX_COMPOUND:
for cut in range(wordforms.MIN_PART, len(text) - wordforms.MIN_PART + 1):
if text[:cut] not in words:
continue
if any(
text.startswith(link, cut) and len(text) - cut - len(link) >= wordforms.MIN_PART
and _word_sequence(text[cut + len(link):], words, memo)
for link in wordforms.LINKS
):
found = True
break
memo[text] = found
return found
def _linked_words(rest: str, words: frozenset, memo: dict[str, bool]) -> bool:
"""Whether the rest of a word after a short first part is a linking element and one or more `words`."""
return any(
rest.startswith(link) and len(rest) - len(link) >= wordforms.MIN_PART and _word_sequence(rest[len(link):], words, memo)
for link in wordforms.LINKS
)
def same_scan_word(left: str, right: str) -> bool:
"""Whether the reformulation count takes two content words for one word (pure).
It decides when a reformulation repeats a word of the question
(`counted_reformulations`), not what the scan matches. True for the same
word in another spelling or inflection (`exact`, `fold`
and `stem`: one stem key, so also a hyphenated word written joined or
hyphenated elsewhere), a verb form of the word forms (`verb`: a shared
verb key), a shared key in the irregular tables (`irregular`: "umzog" and
"umgezogen", "höchste" and "hoch"; `derived`: "umgezogene" and "Umzug"),
a shared base of the `ending` rule ("Kinder" and "Kind", "Tage" and
"Tag", "intelligentem" and "intelligenten"), and another verb form of a
four-letter stem or of a separable verb (`verb_ending`: "senkt" and
"senken", "abgeschaltet" and "abschalten"). The irregular check is
symmetric and so wider than the matching of ranking and scan, which pair
the comparison table one way ("gut" never meets "beste" there, and "hoch"
meets "höchste" only through `ending`): a wider check only discards a
reformulation as a repetition, which never makes an absence complete.
"""
if left == right:
return True
left_key, right_key = wordforms.stem_key(left), wordforms.stem_key(right)
if left_key == right_key:
return True
if left_key.isalpha() and right_key.isalpha():
if not {left_key, *wordforms.verb_keys(left_key)}.isdisjoint((right_key, *wordforms.verb_keys(right_key))):
return True
left_spelling, right_spelling = wordforms.spelling_key(left), wordforms.spelling_key(right)
if not set(wordforms.irregular_keys(left_spelling)).isdisjoint(wordforms.irregular_keys(right_spelling)):
return True
if not (left_spelling.isalpha() and right_spelling.isalpha()):
return False
return (
right_spelling in ending_variants(left_spelling) or left_spelling in ending_variants(right_spelling)
or right_spelling in verb_variants(left_spelling) or left_spelling in verb_variants(right_spelling)
)
def counted_reformulations(
formulations: list[dict[str, Any]],
contents: list[list[str]],
concepts: Optional[list[dict[str, frozenset]]] = None,
) -> list[bool]:
"""Which kept formulations count as reformulations for a complete absence check (pure).
The question never counts. A reformulation counts when it has a content
word and does not hold every content word of the question or of another
reformulation with content words, wherever that one stands; of several
reformulations that hold each other's words, only the first counts. The
same words in another order, spelling or inflection are no new search, and
neither are added words, which under the hit rule can only narrow what the
scan finds. So the order of the reformulations never changes the count.
Two words are the same when the scan matches them as forms of each other
(`same_scan_word`), or when both stand in a run of the same confirmed
concept in their own formulation (`concepts`: per formulation, the concept
ids per content word), because the scan then matches every term of that
concept for both. A hyphenated word is held where a word with the same
stem key is ("SMS-Warndienst", "SMSWarndienst", "SMS-Warn-Dienst"), or
where all of its parts are (`hyphen_units`: "SMS Warndienst"); the parts
of a hyphenated word hold what they hold alone.
"""
concepts = concepts or [{} for _ in contents]
def pool(index: int) -> list[tuple[str, frozenset]]:
words = [(word, concepts[index].get(word, frozenset())) for word in contents[index]]
words.extend((unit, frozenset()) for word in contents[index] for unit in hyphen_units(word))
return words
def same(left: tuple[str, frozenset], right: tuple[str, frozenset]) -> bool:
return not left[1].isdisjoint(right[1]) or same_scan_word(left[0], right[0])
def holds(index: int, other: int) -> bool:
"""Whether formulation `index` holds every content word of formulation `other`."""
words = pool(index)
for word in contents[other]:
item = (word, concepts[other].get(word, frozenset()))
if any(same(item, candidate) for candidate in words):
continue
units = hyphen_units(word)
if units and all(any(same((unit, frozenset()), candidate) for candidate in words) for unit in units):
continue
return False
return True
filled = [index for index, words in enumerate(contents) if words]
counted: list[bool] = []
for index, entry in enumerate(formulations[: len(contents)]):
counts = index > 0 and entry.get("origin") == "also" and bool(contents[index])
if counts and contents[0] and holds(index, 0):
counts = False
if counts:
for other in filled:
if other in (0, index) or not holds(index, other):
continue
# A strict superset never counts; of equal ones only the first does.
if not holds(other, index) or other < index:
counts = False
break
counted.append(counts)
return counted
def printed_chars(value: Any, depth: int) -> int:
"""The characters `value` takes when `report_json` prints it nested `depth` levels deep."""
text = report_json(value)
return len(text) + REPORT_INDENT * depth * text.count("\n")
def excerpt(text: str, wanted: frozenset) -> tuple[str, bool]:
"""A passage text, or a window of at most EXCERPT_CHARS characters around its first word in `wanted`."""
if len(text) <= EXCERPT_CHARS:
return text, False
room = EXCERPT_CHARS - 4
words = text.split(" ")
starts: list[int] = []
offset = 0
for word in words:
starts.append(offset)
offset += len(word) + 1
first = next((index for index, word in enumerate(words) if not wanted.isdisjoint(tokens(word, False))), 0)
def window(start: int) -> list[str]:
taken: list[str] = []
size = 0
for word in words[start:]:
grown = size + len(word) + (1 if taken else 0)
if grown > room:
break
taken.append(word)
size = grown
return taken
start = first
while start > 0 and starts[first] - starts[start - 1] <= 100:
start -= 1
taken = window(start)
if start + len(taken) <= first:
start = first
taken = window(start)
cut = False
if not taken:
taken = [words[start][:room]]
cut = True
remaining = cut or start + len(taken) < len(words)
return ("… " if start > 0 else "") + " ".join(taken) + (" …" if remaining else ""), True
class ScanIndex:
"""The passages of every scanned file and the posting lists the hit rule needs, built once per call."""
def __init__(
self,
files: list[tuple[str, str, str, dict[str, Any], bool]],
include_history: bool,
history_shipped: bool,
path_prefix: str,
mark_selected: bool,
) -> None:
self.path_prefix = path_prefix
self.mark_selected = mark_selected
self.scanned = {"wiki": True, "sources": True, "history": include_history}
self.shipped = {"wiki": True, "sources": True, "history": history_shipped}
self.counts = {
scope: {"files": 0, "navigation_skipped": 0, "passages": 0, "chars": 0} for scope in SCOPES
}
self.passages: list[Passage] = []
self.passage_page: list[int] = []
self.pages: list[dict[str, Any]] = []
self.page_passages: list[range] = []
self.own: dict[str, set[int]] = {}
self.header_pages: dict[str, set[int]] = {}
self.section_index: dict[str, set[tuple[int, tuple]]] = {}
self.section_passages: dict[tuple[int, tuple], set[int]] = {}
# Where a token stands only as an identifier (metadata, a link target), per
# passage, header page and context key. The hyphen_part rule never splits
# a token there; the same token written as text elsewhere is split.
self.address_only: dict[str, set[int]] = {}
self.header_address_only: dict[str, set[int]] = {}
self.section_address_only: dict[str, set[tuple[int, tuple]]] = {}
self._parts: Optional[dict[str, frozenset]] = None
self._stems: Optional[dict[str, frozenset]] = None
self._spellings: Optional[dict[str, frozenset]] = None
self._footnotes: Optional[dict[str, frozenset]] = None
self._suspended: Optional[dict[str, frozenset]] = None
self._split: Optional[dict[str, frozenset]] = None
self._scan_only: Optional[tuple[Any, dict[str, frozenset]]] = None
self._scan_only_table: Optional[tuple[Any, wordforms.Resolver]] = None
self._part_words: Optional[tuple[frozenset, list[tuple[str, str]]]] = None
for relative, scope, text, document, selected in files:
if scan_navigation(relative):
self.counts[scope]["navigation_skipped"] += 1
continue
self.counts[scope]["files"] += 1
self.counts[scope]["chars"] += len(text)
self.add_page(relative, scope, document, selected, scan_passages(relative, text, document))
def add_page(
self, relative: str, scope: str, document: dict[str, Any], selected: bool, passages: list[Passage]
) -> None:
page = len(self.pages)
self.pages.append(
{
"relative": relative,
"scope": scope,
"title": str(document["title"]),
"status": str(document["data"].get("status") or ""),
"selected": selected,
}
)
first = len(self.passages)
self.counts[scope]["passages"] += len(passages)
own = self.own
section_passages = self.section_passages
# Consecutive passages share their section; its prefixes are computed once per section.
prefixes: dict[tuple, list[tuple[int, tuple]]] = {}
openers: list[Passage] = []
for pid, passage in enumerate(passages, first):
for token in passage.identifiers:
self.address_only.setdefault(token, set()).add(pid)
for token in passage.tokens:
holders = own.get(token)
if holders is None:
own[token] = {pid}
else:
holders.add(pid)
keys = prefixes.get(passage.section)
if keys is None:
keys = prefixes[passage.section] = [
(page, passage.section[:depth]) for depth in range(1, len(passage.section) + 1)
]
for key in keys:
holders = section_passages.get(key)
if holders is None:
section_passages[key] = {pid}
else:
holders.add(pid)
if passage.kind == "header":
for token in passage.tokens:
self.header_pages.setdefault(token, set()).add(page)
for token in passage.identifiers:
self.header_address_only.setdefault(token, set()).add(page)
if passage.opens:
openers.append(passage)
# A heading, label, list item or table row is context only for the
# passages under it; one with nothing under it is not indexed.
for passage in openers:
key = (page, passage.opens)
if key in section_passages:
for token in passage.opens_tokens:
self.section_index.setdefault(token, set()).add(key)
for token in passage.identifiers:
self.section_address_only.setdefault(token, set()).add(key)
self.passages.extend(passages)
self.passage_page.extend([page] * len(passages))
# A heading that is not emitted (an H1 repeating the title) still
# leads its section: its words are the title's, held by the header.
self.page_passages.append(range(first, len(self.passages)))
def written_holders(self, token: str) -> set[int]:
"""The passages that hold `token` as written text, not only as an identifier."""
holders = self.own.get(token, set())
only = self.address_only.get(token)
return holders - only if only else holders
def hyphen_parts(self) -> dict[str, frozenset]:
"""Per stem key, the joined tokens written as text with a part of that key ("sms" -> "sms-warndienst"), built once.
Whether a token is text is decided per passage: a token that stands
somewhere only as an identifier or link target is still split where it
is written as text, and never where it is an address. A part of one
character is indexed only together with its neighbour
(`hyphen_groups`): "e-mail-adresse" is listed under "mail",
"adress" and "email", never under "e".
"""
if self._parts is None:
parts: dict[str, set[str]] = {}
for token in self.own:
if ("-" not in token and "_" not in token) or not self.written_holders(token):
continue
units = wordforms.units_of(token)
for unit in units:
if len(unit) >= 2 and unit not in SCAN_EXCLUDED:
parts.setdefault(wordforms.stem_key(unit), set()).add(token)
if any(len(unit) < 2 for unit in units):
for group in hyphen_groups(units):
if "-" in group:
parts.setdefault(wordforms.stem_key(group), set()).add(token)
self._parts = {key: frozenset(found) for key, found in parts.items()}
return self._parts
def stems(self) -> dict[str, frozenset]:
"""Per stem key, every scanned token with that key, built once and only for a hyphenated search word."""
if self._stems is None:
stems: dict[str, set[str]] = {}
for token in self.own:
stems.setdefault(wordforms.stem_key(token), set()).add(token)
self._stems = {key: frozenset(found) for key, found in stems.items()}
return self._stems
def spellings(self) -> dict[str, frozenset]:
"""Per folded spelling, the alphabetic tokens scanned in an ae/oe/ue spelling ("haeuser"), built once for the ending rule.
Every other alphabetic token is its own folded spelling and is looked up in `own` directly.
"""
if self._spellings is None:
spellings: dict[str, set[str]] = {}
for token in self.own:
if ("ae" in token or "oe" in token or "ue" in token) and token.isalpha():
key = wordforms.spelling_key(token)
if key != token:
spellings.setdefault(key, set()).add(token)
self._spellings = {key: frozenset(found) for key, found in spellings.items()}
return self._spellings
def ending_forms(self, term: str) -> frozenset:
"""The scanned tokens that share a base with `term` (`ending_variants`), compared in folded spelling.
"krankenhaus" finds "krankenhauser" and "krankenhausern", "tag" finds
"tage", "tagen" and "tages", "tage" finds "tag", "intelligenten"
finds "intelligentem" and "intelligenter", "auto" finds "autos"; a
base of fewer than ENDING_MIN_STEM letters is never used.
"""
if term in SCAN_EXCLUDED:
return frozenset()
return self._scanned(ending_variants(wordforms.spelling_key(term)), term)
def verb_forms(self, term: str) -> frozenset:
"""The scanned tokens that are another verb form of `term` (`verb_variants`), compared in folded spelling.
"senkt" finds "senken" and "gesenkt", "zahlen" finds "zahlt" and
"zahlte", "abschalten" finds "abgeschaltet" and "abzuschalten"; a
stem of fewer than VERB_MIN_STEM letters (SEPARABLE_MIN_STEM after a
separable particle) is never used.
"""
if term in SCAN_EXCLUDED:
return frozenset()
return self._scanned(verb_variants(wordforms.spelling_key(term)), term)
def split_index(self) -> dict[str, frozenset]:
"""Per joined form in folded spelling ("ablehnt"), the marked split forms that stand for it ("+lehnt+ab"), built once."""
if self._split is None:
joined: dict[str, set[str]] = {}
for token in self.own:
if token[:1] == SPLIT_MARK:
verb, _, particle = token[1:].partition(SPLIT_MARK)
joined.setdefault(wordforms.spelling_key(particle + verb), set()).add(token)
self._split = {key: frozenset(found) for key, found in joined.items()}
return self._split
def split_forms(self, term: str) -> frozenset:
"""The split forms a passage writes of a separable verb whose joined form is `term` or another form of it.
"ablehnen" finds "+lehnt+ab" (joined "ablehnt") and "+lehnen+ab";
"angepasst" finds "+passt+an"; "umgezogene" and "Umzug" find
"+zieht+um" through the verb forms of the irregular tables (never
through a noun of IRREGULAR_NOUNS: a split form is a verb). Only a
candidate that starts with one of SEPARABLE_PARTICLES can stand for a
split form.
"""
if term in SCAN_EXCLUDED:
return frozenset()
index = self.split_index()
if not index:
return frozenset()
spelling = wordforms.spelling_key(term)
found: set[str] = set()
for candidate in (spelling, *verb_variants(spelling), *(wordforms.irregular_variants(spelling) - wordforms.IRREGULAR_NOUN_FORMS)):
found.update(index.get(candidate, ()))
return frozenset(found)
def joined_forms(self, particle: str, verb: str) -> frozenset:
"""What a question's split verb ("schaltet … ab") matches: the joined forms written as text and the split forms of passages.
The candidates are the joined form ("abschaltet") and its other verb
forms ("abschalten", "abgeschaltet", "abzuschalten", and for a verb of
the irregular tables "umzog", "umgezogen"), compared in folded
spelling. A noun of IRREGULAR_NOUNS is no candidate: "ziehe … um" is
never reported as a split verb on "Umzug".
"""
spelling = wordforms.spelling_key(particle + verb)
candidates = {spelling, *verb_variants(spelling), *(wordforms.irregular_variants(spelling) - wordforms.IRREGULAR_NOUN_FORMS)}
found = set(self._scanned(candidates, ""))
index = self.split_index()
for candidate in candidates:
found.update(index.get(candidate, ()))
return frozenset(found)
def _scanned(self, candidates: set[str], term: str) -> frozenset:
"""The scanned tokens that are one of the folded `candidates`, without `term` and the excluded words."""
if not candidates:
return frozenset()
spellings = self.spellings()
found: set[str] = set()
for candidate in candidates:
if candidate in self.own:
found.add(candidate)
found.update(spellings.get(candidate, ()))
found.discard(term)
return frozenset(found - SCAN_EXCLUDED)
def footnote_marks(self) -> dict[str, frozenset]:
"""Per stem key of the letters, the scanned tokens that glue one or two digits to a word of four or more letters, built once.
"kundigungsfrist2" is listed under "kundigungsfrist": a footnote mark
that a converter wrote as a plain digit.
"""
if self._footnotes is None:
footnotes: dict[str, set[str]] = {}
for token in self.own:
if token[-1:].isdigit():
match = FOOTNOTE_TOKEN.fullmatch(token)
if match is not None:
footnotes.setdefault(wordforms.stem_key(match.group(1)), set()).add(token)
self._footnotes = {key: frozenset(found) for key, found in footnotes.items()}
return self._footnotes
def part_words(self) -> tuple[frozenset, list[tuple[str, str]]]:
"""The words a short part's rest may be, and the words a short part is looked for in, built once.
The first are the folded spellings and stem keys of every alphabetic
scanned token of at least wordforms.MIN_PART letters outside the
excluded words. The second are (token, folded spelling) for every
alphabetic scanned token, and for every part of a hyphenated token
written as text, of at most wordforms.MAX_COMPOUND letters.
"""
if self._part_words is None:
words: set[str] = set()
alpha: list[tuple[str, str]] = []
for token in self.own:
if token.isalpha():
if len(token) > wordforms.MAX_COMPOUND:
continue
spelling = wordforms.spelling_key(token)
alpha.append((token, spelling))
if len(spelling) >= wordforms.MIN_PART and token not in SCAN_EXCLUDED:
words.add(spelling)
words.add(wordforms.stem(spelling))
elif "-" in token and token[:1] != SUSPENDED_MARK and self.written_holders(token):
for unit in wordforms.units_of(token):
if unit.isalpha() and len(unit) <= wordforms.MAX_COMPOUND:
alpha.append((token, wordforms.spelling_key(unit)))
self._part_words = (frozenset(words), alpha)
return self._part_words
def short_parts(self, term: str) -> frozenset:
"""The scanned tokens that hold a term of two or three letters as a part (SHORT_PART_RULE).
"gas" finds "gaskunden" and "gasversorger" when "kunden" and
"versorger" are words of the scanned files, and "erdgas",
"flussiggas" and "erdgas-tarif" without that condition; "tag" finds
"feiertagen". A term outside those lengths, or excluded, finds nothing.
"""
if term in SCAN_EXCLUDED:
return frozenset()
key = wordforms.spelling_key(term)
if not key.isalpha() or not SHORT_PART_MIN <= len(key) <= SHORT_PART_MAX:
return frozenset()
words, alpha = self.part_words()
heads = tuple(key + ending for ending in ("", *INFLECTION_ENDINGS))
memo: dict[str, bool] = {}
found: set[str] = set()
for token, spelling in alpha:
if token in found or len(spelling) < len(key) + SHORT_PART_LEAD:
continue
if any(
spelling.endswith(head) and len(spelling) - len(head) >= SHORT_PART_LEAD for head in heads
) or (spelling.startswith(key) and _linked_words(spelling[len(key):], words, memo)):
found.add(token)
found.discard(term)
return frozenset(found - SCAN_EXCLUDED)
def suspended(self) -> dict[str, frozenset]:
"""Per stem key, the marked joined forms of suspended compounds (`suspended_forms`), built once."""
if self._suspended is None:
joined: dict[str, set[str]] = {}
for token in self.own:
if token[:1] == SUSPENDED_MARK:
joined.setdefault(wordforms.stem_key(token[1:]), set()).add(token)
self._suspended = {key: frozenset(found) for key, found in joined.items()}
return self._suspended
def scan_only_forms(self, term: str, vocabulary: dict[str, int]) -> list[tuple[str, str]]:
"""Scanned tokens the ranking's vocabulary lacks that are `term` itself or one of its word forms, with the rule.
The rule is `exact` for the term itself, `fold` for the same folded
spelling, and `stem` for the same stem key, as the word-form layer
names them. Closed-class and function words are never added.
"""
if self._scan_only is None or self._scan_only[0] is not vocabulary:
index: dict[str, set[str]] = {}
for token in self.own:
if token not in vocabulary and token not in SCAN_EXCLUDED:
index.setdefault(wordforms.stem_key(token), set()).add(token)
self._scan_only = (vocabulary, {key: frozenset(found) for key, found in index.items()})
found = self._scan_only[1].get(wordforms.stem_key(term), frozenset())
spelling = wordforms.spelling_key(term)
return [
(token, "exact" if token == term else "fold" if wordforms.spelling_key(token) == spelling else "stem")
for token in sorted(found)
]
def scan_only_table_forms(self, term: str, vocabulary: dict[str, int]) -> list[tuple[str, str]]:
"""Scanned tokens the ranking's vocabulary lacks that the irregular tables pair with `term`, with the rule.
A word that only the scan's segmentation repairs ("umge-", a page
marker, "zogen" give "umgezogen") is no word of the ranking, whose
resolver therefore never pairs it. Here it is paired the same way, by
a resolver over those tokens: `irregular` on the same side, `derived`
between a noun and its verb. Only for query words, like in the ranking.
"""
if not wordforms.irregular_keys(wordforms.spelling_key(term)):
return []
if self._scan_only_table is None or self._scan_only_table[0] is not vocabulary:
only = {
token: 1
for token in self.own
if token not in vocabulary and token not in SCAN_EXCLUDED and token.isalpha()
and wordforms.irregular_keys(wordforms.spelling_key(token))
}
self._scan_only_table = (vocabulary, wordforms.Resolver(only, CLOSED_CLASS, heuristics=False))
resolver = self._scan_only_table[1]
found = [(token, "irregular") for token in sorted(resolver.irregular(term))]
found.extend((token, "derived") for token in sorted(resolver.derived(term)))
return found
def unit_members(self, unit: str) -> list[wordforms.Member]:
"""The members of one part of a hyphenated search word written apart: its stem key alone or as a part."""
key = wordforms.stem_key(unit)
found = self.stems().get(key, frozenset()) | self.hyphen_parts().get(key, frozenset())
weight = wordforms.WEIGHTS
members = [(token, HYPHEN_PART_RULE, weight[HYPHEN_PART_RULE]) for token in sorted(found)]
ending = self.ending_forms(unit) - found
members.extend((token, ENDING_RULE, weight[ENDING_RULE]) for token in sorted(ending))
members.extend(
(token, SHORT_PART_RULE, weight[SHORT_PART_RULE]) for token in sorted(self.short_parts(unit) - found - ending)
)
return members
#: The matcher's key for a hyphen_part member that may only count where it is written as text.
TEXT_ONLY = "text-only"
class _Matcher:
"""The set algebra of the hit rule over one ScanIndex, with posting unions cached for one call.
A member key is a token, or (TEXT_ONLY, token) for a joined token that
the hyphen_part rule may only take where it is written as text.
"""
def __init__(self, scan: ScanIndex) -> None:
self.scan = scan
self._own: dict[frozenset, set[int]] = {}
self._context: dict[frozenset, set[int]] = {}
def holders(self, key: Any) -> Iterable[int]:
if key.__class__ is tuple:
return self.scan.written_holders(key[1])
return self.scan.own.get(key, ())
def header_pages(self, key: Any) -> Iterable[int]:
if key.__class__ is tuple:
only = self.scan.header_address_only.get(key[1])
pages = self.scan.header_pages.get(key[1], set())
return pages - only if only else pages
return self.scan.header_pages.get(key, ())
def section_keys(self, key: Any) -> Iterable[tuple[int, tuple]]:
if key.__class__ is tuple:
only = self.scan.section_address_only.get(key[1])
keys = self.scan.section_index.get(key[1], set())
return keys - only if only else keys
return self.scan.section_index.get(key, ())
def own(self, members: frozenset) -> set[int]:
found = self._own.get(members)
if found is None:
found = set()
for key in members:
found.update(self.holders(key))
self._own[members] = found
return found
def context(self, members: frozenset) -> set[int]:
"""Passages that hold a member themselves, under a heading holding one, or on a page whose header holds one."""
found = self._context.get(members)
if found is None:
scan = self.scan
found = set(self.own(members))
for key in members:
for page in self.header_pages(key):
found.update(scan.page_passages[page])
for section in self.section_keys(key):
found.update(scan.section_passages.get(section, ()))
self._context[members] = found
return found
class _Formulation:
"""What the scan needs of one kept formulation: content words, alternatives and their members.
The members are the ranking's (see `scan_members`). A hyphenated content
word also gets the alternative of its parts written apart (`hyphen_units`),
each part matched in its stem key, alone or as a part of a joined token;
`splits` maps the word to the keys of those parts.
"""
def __init__(
self,
text: str,
ranking: Ranking,
definitions: dict[str, dict[str, Any]],
scan: ScanIndex,
vocabulary: dict[str, int],
) -> None:
self.text = text
self.scan = scan
self.terms = ranking.terms
self.members: dict[Any, list[wordforms.Member]] = scan_members(ranking, scan, vocabulary)
self.content = content_words(ranking.terms)
# A separable verb the formulation splits ("Wann schaltet ... ab?")
# also matches its joined forms and the split forms of passages, for
# the verb and for a particle that is a content word ("zurück").
content = set(self.content)
weight = wordforms.WEIGHTS[SPLIT_VERB_RULE]
for particle, words in separable_splits(normalized(wordforms.repair_text(text)), SCAN_EXCLUDED):
for word in words:
if word not in content:
continue
found = scan.joined_forms(particle, word)
for term in (word, particle):
if term not in content:
continue
listed = self.members.setdefault(term, [])
known = {token for token, _, _ in listed}
listed.extend((token, SPLIT_VERB_RULE, weight) for token in sorted(found - known))
self.runs = concept_runs(text, definitions)
self.alternatives = alternatives(self.content, self.runs, definitions)
self._keys: dict[tuple[Any, bool], frozenset] = {}
self._every: Optional[frozenset] = None
self.splits: dict[str, tuple] = {}
for word in self.content:
units = hyphen_units(word)
if not units:
continue
# Keys of their own, so a part never takes the members of a term with the same spelling.
keys = tuple((HYPHEN_PART_RULE, unit) for unit in units)
for key, unit in zip(keys, units):
self.members[key] = scan.unit_members(unit)
self.splits[word] = keys
self.alternatives[word] = self.alternatives[word] + [(None, keys)]
def key_of(self, token: str, rule: str) -> Any:
"""The matcher's key for one member: a token taken by hyphen_part or short_part counts only where it is text."""
if token in self.scan.address_only and (
rule == SHORT_PART_RULE or (rule == HYPHEN_PART_RULE and ("-" in token or "_" in token))
):
return (TEXT_ONLY, token)
return token
def tokens_of(self, token: Any, word_forms_only: bool) -> frozenset:
"""The matcher's keys of a term's (or a part's) members, of the word forms only when asked, built once."""
found = self._keys.get((token, word_forms_only))
if found is None:
found = self._keys[(token, word_forms_only)] = frozenset(
self.key_of(member, rule)
for member, rule, _ in self.members.get(token, ())
if not word_forms_only or rule in wordforms.WORD_FORM_RULES
)
return found
def concepts(self) -> dict[str, frozenset]:
"""Per content word, the concepts in whose matching run it stands."""
return {word: frozenset(key for key, run in self.runs.items() if word in run) for word in self.content}
def hits(self, matcher: _Matcher, word_forms_only: bool) -> set[int]:
"""H_i: passages inside the context of every content word that hold at least one of them themselves."""
if not self.content:
return set()
contexts: list[set[int]] = []
owned: set[int] = set()
for word in self.content:
word_context: set[int] = set()
for _concept, option in self.alternatives[word]:
option_context: Optional[set[int]] = None
option_own: set[int] = set()
for token in option:
members = self.tokens_of(token, word_forms_only)
context = matcher.context(members)
option_context = set(context) if option_context is None else option_context & context
option_own |= matcher.own(members)
word_context |= option_context or set()
owned |= (option_context or set()) & option_own
contexts.append(word_context)
contexts.sort(key=len)
found = set(contexts[0])
for context in contexts[1:]:
found &= context
return found & owned
def every_member(self) -> frozenset:
"""Every member token of every content word and of its alternatives, built once."""
if self._every is None:
self._every = frozenset(
member
for word in self.content
for _concept, option in self.alternatives[word]
for token in option
for member, _, _ in self.members.get(token, ())
)
return self._every
def matched(self, matcher: _Matcher, pid: int) -> list[dict[str, Any]]:
"""Per content word, the member that carried it into a heuristic hit, and where it occurred."""
scan = matcher.scan
passage = scan.passages[pid]
page = scan.passage_page[pid]
sections = {(page, passage.section[:depth]) for depth in range(1, len(passage.section) + 1)}
entries: list[dict[str, Any]] = []
for word in self.content:
for concept, option in self.alternatives[word]:
# The first alternative whose context holds the passage carried the word.
if not all(pid in matcher.context(self.tokens_of(token, False)) for token in option):
continue
pool = [member for token in option for member in self.members.get(token, ())]
best: Optional[tuple[tuple[int, float, str], str]] = None
for token, rule, weight in pool:
key = self.key_of(token, rule)
if token in passage.tokens and (key.__class__ is not tuple or token not in passage.identifiers):
where = 0
elif not sections.isdisjoint(matcher.section_keys(key)):
where = 1
elif page in matcher.header_pages(key):
where = 2
else:
continue
rank = (where, -weight, token)
if best is None or rank < best[0]:
best = (rank, rule)
if best is None:
continue
entry: dict[str, Any] = {
"term": word,
"token": shown_token(best[0][2]),
"rule": best[1],
"in": ("text", "heading", "header")[best[0][0]],
}
if concept is not None:
entry["concept"] = concept
entries.append(entry)
break
return entries
def printed_claim_keys(
results: list[dict[str, Any]], block: list[dict[str, Any]], path_prefix: str
) -> dict[tuple[str, str], str]:
"""Every claim already printed, keyed by (release-relative path, claim ID or text), with where it is printed."""
printed: dict[tuple[str, str], str] = {}
for where, entries in (("results", results), ("reformulation_results", block)):
for entry in entries:
path = str(entry["path"])
relative = path.removeprefix(path_prefix)
for claim in entry.get("claims") or []:
key = (relative, str(claim.get("id") or "") or str(claim.get("text") or ""))
printed.setdefault(key, where)
return printed
def fit_prefix(entries: list[dict[str, Any]], budget: int) -> tuple[list[dict[str, Any]], bool]:
"""The longest prefix of page entries, hit by hit, that `report_json` prints within `budget` characters (pure).
Returns the entries of that prefix, the last one possibly with fewer
hits, and whether anything was left out. The size is counted where
exhaustive_hits stands in a report, one level deep: "[]", each entry with
its first hit two levels deep, each further hit four levels deep, and the
line breaks, commas and indents between them.
"""
returned: list[dict[str, Any]] = []
used = 2 # "[]"
for entry in entries:
emitted: Optional[dict[str, Any]] = None
for hit in entry["hits"]:
if emitted is None:
# A first entry adds two line breaks and the indents of the entry and of "]",
# a later one ",\n" and its indent.
cost = (8 if not returned else 6) + printed_chars({**entry, "hits": [hit]}, 2)
else:
# ",\n" and the indent of a hit.
cost = 10 + printed_chars(hit, 4)
if used + cost > budget:
return returned, True
used += cost
if emitted is None:
emitted = {**entry, "hits": [hit]}
returned.append(emitted)
else:
emitted["hits"].append(hit)
return returned, False
def _census(formulations: list[_Formulation], matcher: _Matcher, vocabulary: dict[str, int]) -> list[dict[str, Any]]:
"""Every term of every formulation except closed-class and function words, once per (term, origin).
Each entry names the formulations that hold the term, how many passages
and files hold one of its members per scanned layer (a hyphenated query
word also where a passage holds all of its parts), and its members.
"""
scan = matcher.scan
census: dict[tuple[str, str], dict[str, Any]] = {}
for index, formulation in enumerate(formulations):
for item in formulation.terms:
term, origin = item["term"], item["origin"]
if term in SCAN_EXCLUDED:
continue
key = (term, origin)
if key in census:
census[key]["formulations"].append(index)
continue
members = formulation.members.get(term, [])
holders = matcher.own(formulation.tokens_of(term, False))
split = formulation.splits.get(term) if origin == "query" else None
if split:
# A hyphenated word also occurs where a passage holds all of its parts.
apart: Optional[set[int]] = None
for part in split:
owned = matcher.own(formulation.tokens_of(part, False))
apart = set(owned) if apart is None else apart & owned
holders = holders | (apart or set())
passages = {scope: 0 for scope in SCOPES}
files: dict[str, set[int]] = {scope: set() for scope in SCOPES}
for pid in holders:
page = scan.passage_page[pid]
scope = scan.pages[page]["scope"]
passages[scope] += 1
files[scope].add(page)
ordered = sorted(members, key=lambda member: (-member[2], -vocabulary.get(member[0], 0), member[0]))
census[key] = {
"term": term,
"origin": origin,
"formulations": [index],
"passages": {scope: passages[scope] if scan.scanned[scope] else None for scope in SCOPES},
"files": {scope: len(files[scope]) if scan.scanned[scope] else None for scope in SCOPES},
"forms": [{"token": shown_token(token), "rule": rule} for token, rule, _ in ordered[:FORMS_LISTED]],
"forms_total": len(members),
}
return list(census.values())
def exhaustive_scan(
scan: ScanIndex,
rankings: list[Ranking],
kept: list[dict[str, Any]],
definitions: dict[str, dict[str, Any]],
vocabulary: dict[str, int],
printed: dict[tuple[str, str], str],
budget: int,
) -> dict[str, Any]:
"""Every passage that holds all content words of a kept formulation, ordered, cut to `budget` printed characters.
`kept` holds the kept formulations (index, text, origin), `rankings` their
rankings. Returns the returned page entries (`hits`), whether the
question's own hits fit into the budget (`narrowable`, internal), and the
fields of absence_check that describe the scan: the counted
reformulations, per-formulation counts, census, absent terms, counts,
budget, omitted pages and scopes.
"""
matcher = _Matcher(scan)
formulations = [
_Formulation(entry["text"], ranking, definitions, scan, vocabulary) for entry, ranking in zip(kept, rankings)
]
counted = counted_reformulations(
kept, [formulation.content for formulation in formulations], [formulation.concepts() for formulation in formulations]
)
by_formulation: list[set[int]] = []
word_form_hits: set[int] = set()
summary: list[dict[str, Any]] = []
for index, formulation in enumerate(formulations):
found = formulation.hits(matcher, False)
strong = formulation.hits(matcher, True)
by_formulation.append(found)
word_form_hits |= strong
summary.append(
{
"formulation": index,
"content_words": formulation.content,
"counted": counted[index],
"hits": len(found),
"hits_word_form": len(strong),
}
)
all_hits = sorted(set().union(*by_formulation)) if by_formulation else []
pages: dict[int, list[tuple[tuple[int, int, int], dict[str, Any]]]] = {}
references = 0
for pid in all_hits:
passage = scan.passages[pid]
page = scan.passage_page[pid]
member_of = [index for index, found in enumerate(by_formulation) if pid in found]
hit: dict[str, Any] = {
"kind": passage.kind,
"line": passage.line,
"locator": passage.locator,
"status": passage.status,
"formulations": member_of,
"strength": "word_form" if pid in word_form_hits else "heuristic",
}
key = (scan.pages[page]["relative"], passage.locator or passage.text)
if passage.kind == "claim" and key in printed:
hit["shown_in"] = printed[key]
references += 1
else:
if hit["strength"] == "heuristic":
hit["matched"] = formulations[member_of[0]].matched(matcher, pid)
wanted = frozenset().union(*(formulations[index].every_member() for index in member_of))
text, cut = excerpt(passage.text, wanted)
hit.update({"text": text, "text_chars": len(passage.text), "excerpt": cut})
pages.setdefault(page, []).append(((passage.line, PASSAGE_KINDS.index(passage.kind), pid), hit))
entries: list[dict[str, Any]] = []
for page, items in pages.items():
items.sort(key=lambda item: item[0])
hits = [hit for _, hit in items]
info = scan.pages[page]
entry: dict[str, Any] = {
"path": scan.path_prefix + info["relative"],
"scope": info["scope"],
"title": info["title"],
"status": info["status"],
"hits_total": len(hits),
}
if scan.mark_selected:
entry["selected"] = info["selected"]
entry["hits"] = hits
entries.append(entry)
entries.sort(
key=lambda entry: (
1 if entry["scope"] == "history" else 0,
-max(len(hit["formulations"]) for hit in entry["hits"]),
0 if any(0 in hit["formulations"] for hit in entry["hits"]) else 1,
0 if any(hit["strength"] == "word_form" for hit in entry["hits"]) else 1,
SCOPE_RANK[entry["scope"]],
entry["path"],
)
)
# Emit a strict prefix: hit by hit while the printed size stays within budget.
returned, stopped = fit_prefix(entries, budget)
used = printed_chars(returned, 1)
needed = printed_chars(entries, 1) if stopped else used
# Narrower reformulations can only help while the question's own hits fit:
# its hits do not depend on the reformulations.
narrowable = True
if stopped:
own_entries: list[dict[str, Any]] = []
for entry in entries:
own_hits = [hit for hit in entry["hits"] if 0 in hit["formulations"]]
if own_hits:
own_entries.append({**entry, "hits_total": len(own_hits), "hits": own_hits})
narrowable = not fit_prefix(own_entries, budget)[1]
hits_total = len(all_hits)
hits_returned = sum(len(entry["hits"]) for entry in returned)
returned_paths = {entry["path"] for entry in returned}
counts = {
"hits_total": hits_total,
"hits_returned": hits_returned,
"hits_omitted": hits_total - hits_returned,
"hits_word_form": len(word_form_hits.intersection(all_hits)),
"hits_heuristic": hits_total - len(word_form_hits.intersection(all_hits)),
"hits_already_printed": references,
"pages_with_hits": len(entries),
"pages_returned": len(returned),
"pages_partial": sum(1 for entry in returned if len(entry["hits"]) < entry["hits_total"]),
"pages_not_returned": len(entries) - len(returned),
}
omitted_pages = [
{"path": entry["path"], "hits": entry["hits_total"]} for entry in entries if entry["path"] not in returned_paths
][:OMITTED_PAGES_LISTED]
terms = _census(formulations, matcher, vocabulary)
absent = sorted(
{
entry["term"]
for entry in terms
if entry["origin"] == "query" and not any(entry["passages"][scope] for scope in SCOPES if scan.scanned[scope])
}
)
scopes = {
scope: {
"scanned": scan.scanned[scope],
"shipped": scan.shipped[scope],
**{
field: (scan.counts[scope][field] if scan.scanned[scope] else None)
for field in ("files", "navigation_skipped", "passages", "chars")
},
}
for scope in SCOPES
}
return {
"hits": returned,
"narrowable": narrowable,
"reformulations_counted": sum(counted),
"budget": {"limit_chars": budget, "used_chars": used, "needed_chars": needed},
"counts": counts,
"omitted_pages": omitted_pages,
"absent_terms": absent,
"formulations": summary,
"scopes": scopes,
"terms": terms,
}
def absence_steps(
reasons: list[str],
results: list[dict[str, Any]],
block: list[dict[str, Any]],
hits: list[dict[str, Any]],
narrowable: bool = True,
) -> list[str]:
"""What to do before an absence statement, from the closed vocabulary ABSENCE_STEPS, in its order.
`read_results_without_claims` names entries that print no claims (every
source, a page without claim blocks): their snippet is all the report
shows, and their text can answer in words the scan does not match.
`narrow_formulations` is listed only while `narrowable` holds, that is
while the question's own hits fit into the budget.
"""
printed = [entry for entry in list(results) + list(block) if not entry.get("navigation")]
conditions = {
"read_withheld_claims": any(entry["claims_returned"] < entry["claims_total"] for entry in printed),
"read_results_without_claims": any(not entry["claims_total"] for entry in printed),
"read_reformulation_results": bool(block),
"read_exhaustive_hits": any("text" in hit for entry in hits for hit in entry["hits"]),
"add_reformulations": "too_few_reformulations" in reasons or "no_content_words" in reasons,
"narrow_formulations": "budget_exhausted" in reasons and narrowable,
}
return [step for step in ABSENCE_STEPS if conditions[step]]
def absence_check(
scanned: Optional[dict[str, Any]], results: list[dict[str, Any]], block: list[dict[str, Any]]
) -> dict[str, Any]:
"""The verdict every report carries: whether an absence statement is allowed, and what is missing if not.
`scanned` is the return value of `exhaustive_scan`, or None when the call
ran without the exhaustive scan; then only `not_exhaustive` is listed and
every field that was not evaluated is null rather than zero.
"""
next_steps: Optional[list[str]] = None
if scanned is None:
reasons = ["not_exhaustive"]
else:
applies = {
"not_exhaustive": False,
"too_few_reformulations": scanned["reformulations_counted"] < MIN_EXHAUSTIVE_REFORMULATIONS,
"no_content_words": not any(entry["content_words"] for entry in scanned["formulations"]),
"budget_exhausted": scanned["budget"]["needed_chars"] > scanned["budget"]["limit_chars"],
}
reasons = [reason for reason in ABSENCE_REASONS if applies[reason]]
next_steps = absence_steps(reasons, results, block, scanned["hits"], scanned["narrowable"])
return {
"format": ABSENCE_CHECK_FORMAT,
"complete": not reasons,
"reasons": reasons,
"next_steps": next_steps,
**{field: None if scanned is None else scanned[field] for field in SCAN_FIELDS},
"boundary": ABSENCE_BOUNDARY,
}
def search_record(found: dict[str, Any], release: dict[str, Any]) -> dict[str, str]:
"""The one-line record of a search that an answer quotes; `release` is the release the wrapper verified (pure)."""
query = str(found["formulations"][0]["text"])
if len(query) > SEARCH_RECORD_QUERY_CHARS:
query = query[:SEARCH_RECORD_QUERY_CHARS] + "…"
scopes = found["coverage"]["scopes"]
exhaustive = found["exhaustive_hits"] is not None
check = found["absence_check"]
hits = f"{check['counts']['hits_returned']}/{check['counts']['hits_total']}" if exhaustive else "none"
line = " ".join(
[
SEARCH_RECORD_FORMAT,
f"release={release.get('version', '')}",
f"id={release.get('release_id', '')}",
f"manifest={str(release.get('manifest_sha256') or '')[:12]}",
f"query={json.dumps(query, ensure_ascii=False)}",
f"reformulations={len(found['formulations']) - 1}",
f"ranked={'+'.join(scope for scope in SCOPES if scopes[scope]['searched'])}",
f"exhaustive={'true' if exhaustive else 'false'}",
f"results={len(found['results'])}/{found['coverage']['matched']}",
f"hits={hits}",
f"complete={'true' if check['complete'] else 'false'}",
]
)
return {"format": SEARCH_RECORD_FORMAT, "line": line}
def report_body(
found: dict[str, Any], release: dict[str, Any], selector: dict[str, Any], scope_flags: dict[str, bool]
) -> dict[str, Any]:
"""The report a wrapper prints for a search, in the order both wrappers print it (pure).
`found` is what `search` returned, `release` the release the wrapper
verified and `selector` the validated selector. `scope_flags` holds the
effective layer options the wrapper offers, in its order (`include_sources`,
and `include_history` where history ships).
"""
return {
# The query exactly as given: plan_formulations keeps it as formulation 0.
"query": found["formulations"][0]["text"],
"formulations": found["formulations"],
"dropped_formulations": found["dropped_formulations"],
"release": release,
# The verdict and the record line come first: hosts may cut long output to a preview.
"search_record": search_record(found, release),
"absence_check": found["absence_check"],
"searched_files": found["searched_files"],
**scope_flags,
"selector": selector,
"filtered_out": found["filtered_out"],
"matched_concepts": found["matched_concepts"],
"expanded_terms": found["expanded_terms"],
"facets": found["facets"],
"results": found["results"],
"reformulation_results": found["reformulation_results"],
"coverage": found["coverage"],
# Last, so a cut preview still shows everything above.
"exhaustive_hits": found["exhaustive_hits"],
}
def report_json(value: Any) -> str:
"""A report, or any part of one, as both wrappers print it; the budget of exhaustive_hits is counted in it."""
return json.dumps(value, ensure_ascii=False, indent=REPORT_INDENT)
def search_loaded(
loaded: LoadedRelease,
query: str,
*,
also: Iterable[str] = (),
limit: int = 12,
exhaustive: bool = False,
exhaustive_budget: int = EXHAUSTIVE_BUDGET_CHARS,
) -> dict[str, Any]:
"""Rank `query` and its reformulations over a loaded release, fuse them, and add the absence check.
The steps after `load`: plan the formulations, rank each kept one, build
`results`, `reformulation_results` and `coverage` exactly as without the
exhaustive scan, then, with `exhaustive` (which needs `load(exhaustive=True)`),
scan every passage for all words of all formulations. `absence_check` is
computed in both modes; `exhaustive_hits` is None without the scan.
`exhaustive_budget` exists for tests only. Raises QueryError, RequestError
and ValueError like `search`.
"""
if limit < 1:
raise ValueError("limit must be at least 1")
if exhaustive and not loaded.exhaustive:
raise ValueError("an exhaustive search needs a release loaded with exhaustive=True")
also = list(also)
kept, dropped = plan_formulations(query, also)
if not formulation_key(query):
raise QueryError("query must contain at least one searchable token")
rankings = [rank(loaded, query, limit=limit)]
formulations = [kept[0]]
dropped_positions = {item["position"] for item in dropped}
kept_positions = [position for position in range(1, len(also) + 1) if position not in dropped_positions]
for entry, position in zip(kept[1:], kept_positions):
try:
ranking = rank(loaded, entry["text"], limit=limit)
except QueryError:
# Defensive: plan_formulations already drops text without a word.
dropped.append({"position": position, "text": entry["text"], "reason": "empty", "duplicate_of": None})
continue
formulations.append({**entry, "index": len(formulations)})
rankings.append(ranking)
dropped.sort(key=lambda item: item["position"])
first = rankings[0]
results: list[dict[str, Any]] = []
for index in range(min(limit, len(first.ordered))):
result = first.full(index)
result["found_by"] = found_by(rankings, result["path"])
results.append(result)
# Pages only reformulations return within the limit, taken in turn by
# rank: scores of different searches are not comparable, ranks are.
shown = {result["path"] for result in results}
pool: list[tuple[str, int, int]] = []
seen: set[str] = set()
navigation_skipped: set[str] = set()
for position in range(1, limit + 1):
for formulation, ranking in enumerate(rankings[1:], 1):
if position > len(ranking.ordered):
continue
item = ranking.ordered[position - 1]
path = item["path"]
if path in shown or path in seen:
continue
if item["navigation"]:
navigation_skipped.add(path)
continue
seen.add(path)
pool.append((path, formulation, position))
block: list[dict[str, Any]] = []
for path, via, position in pool[:limit]:
entry = rankings[via].full(position - 1)
entry["found_by"] = found_by(rankings, path)
entry["via"] = via
block.append(entry)
omitted = pool[limit:]
coverage = dict(first.coverage)
coverage["formulations"] = [
{
"formulation": index,
"matched": ranking.coverage["matched"],
"within_limit": min(ranking.coverage["matched"], limit),
"truncated": ranking.coverage["matched"] > limit,
"contributed": sum(
1
for entry in block
if any(hit["formulation"] == index and hit["rank"] <= limit for hit in entry["found_by"])
),
"query_terms": [
{key: value for key, value in item.items() if key != "forms"}
for item in ranking.coverage["query_terms"]
],
"dropped_terms": ranking.coverage["dropped_terms"],
"unmatched_terms": ranking.coverage["unmatched_terms"],
"form_only_terms": ranking.coverage["form_only_terms"],
"terms_only_in_unsearched": ranking.coverage["terms_only_in_unsearched"],
}
for index, ranking in enumerate(rankings)
]
coverage["reformulation_results"] = {
"returned": len(block),
"candidates": len(pool),
"truncated": len(pool) > len(block),
"omitted_count": len(omitted),
"omitted_top": [
{"path": path, "formulation": formulation, "rank": position}
for path, formulation, position in omitted[:OMITTED_TOP_LIMIT]
],
"navigation_skipped": len(navigation_skipped),
"boundary": REFORMULATION_BOUNDARY,
}
scanned: Optional[dict[str, Any]] = None
if exhaustive and loaded.scan is not None:
scanned = exhaustive_scan(
loaded.scan,
rankings,
formulations,
loaded.concept_definitions,
loaded.vocabulary,
printed_claim_keys(results, block, loaded.path_prefix),
exhaustive_budget,
)
return {
"searched_files": loaded.searched,
"filtered_out": loaded.filtered_out,
"formulations": [
{
"index": entry["index"],
"text": entry["text"],
"origin": entry["origin"],
"matched_concepts": ranking.matched_concepts,
"expanded_terms": ranking.expanded_terms,
}
for entry, ranking in zip(formulations, rankings)
],
"dropped_formulations": dropped,
"matched_concepts": first.matched_concepts,
"expanded_terms": first.expanded_terms,
"facets": first.facets,
"results": results,
"reformulation_results": block,
"coverage": coverage,
"absence_check": absence_check(scanned, results, block),
"exhaustive_hits": scanned["hits"] if scanned is not None else None,
"parser_errors": loaded.parser_errors,
}
def search(
root: Path,
query: str,
*,
also: Iterable[str] = (),
limit: int = 12,
include_sources: bool = False,
include_history: bool = False,
selector: Optional[dict[str, Any]] = None,
history_shipped: bool = True,
path_prefix: str = "",
exhaustive: bool = False,
exhaustive_budget: int = EXHAUSTIVE_BUDGET_CHARS,
) -> dict[str, Any]:
"""Rank the release under `root` for `query` and its reformulations, and account for what is not shown.
`also` holds at most MAX_ALSO reformulations (see `plan_formulations`).
`selector` must already be validated by `wiki_filters.validate_selector`.
`history_shipped=False` describes a release that carries no history layer,
such as an exported knowledge skill. `path_prefix` is prepended to every
path the report names; selectors, scopes and navigation are decided on the
release-relative path.
The release is read once and every formulation is ranked over it. The
query's `matched_concepts`, `expanded_terms`, `facets` (over every match),
`results` (at most `limit`) and `coverage` are exactly those of a search
without reformulations, plus `found_by` on each result and the coverage
keys `formulations` and `reformulation_results`. `reformulation_results`
holds at most `limit` pages that only reformulations returned within the
limit, without navigation pages. Also returned: `searched_files`,
`filtered_out`, `formulations`, `dropped_formulations`, `absence_check`,
`exhaustive_hits` and `parser_errors`. A caller must not report results
when `parser_errors` is non-empty.
`exhaustive=True` forces `include_sources` and scans every passage of the
ranked files except navigation for all words of all formulations, within
`exhaustive_budget` printed characters of `exhaustive_hits` (tests only;
the helpers never pass it). Everything else in the report stays what the
same call without the scan but with `include_sources` returns. Raises
QueryError for a query without a searchable token, RequestError for
invalid reformulations and ValueError for an impossible option.
"""
# Invalid input is refused before the release is read.
if limit < 1:
raise ValueError("limit must be at least 1")
if include_history and not history_shipped:
raise ValueError("this release ships no history layer to search")
also = list(also)
plan_formulations(query, also)
if not formulation_key(query):
raise QueryError("query must contain at least one searchable token")
loaded = load(
root,
include_sources=include_sources or exhaustive,
include_history=include_history,
selector=selector,
history_shipped=history_shipped,
path_prefix=path_prefix,
exhaustive=exhaustive,
)
return search_loaded(
loaded, query, also=also, limit=limit, exhaustive=exhaustive, exhaustive_budget=exhaustive_budget
)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |