K-Dense-AI/scientific-agent-skills / skills/markitdown
markitdown
VerifiedConverts heterogeneous documents and selected URIs to Markdown with Microsoft MarkItDown for text analysis, search, and LLM/RAG ingestion. Covers safe local conversion, streams, Office/PDF/data formats, batch workflows, plugins, vision OCR, Azure extraction, and the official MCP server.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 11.9 KB |
| references/api_reference.md | markdown | 11.6 KB |
| references/cloud_and_ocr.md | markdown | 13.8 KB |
| references/file_formats.md | markdown | 11.5 KB |
| references/mcp_and_plugins.md | markdown | 7.1 KB |
| references/migration.md | markdown | 9.0 KB |
| references/security.md | markdown | 10.2 KB |
| references/workflows.md | markdown | 8.0 KB |
| scripts/batch_convert.py | script | 10.3 KB |
| scripts/convert_literature.py | script | 11.8 KB |
| scripts/inspect_installation.py | script | 4.8 KB |
references/workflows.md
View on GitHub# Practical Workflows
All workflows target MarkItDown 0.1.8 and use `.markdown`, not the legacy `.text_content` alias.
## 1. One Trusted Local Document
```python
from hashlib import sha256
from pathlib import Path
from markitdown import MarkItDown
source = Path("protocol.docx").resolve(strict=True)
output = Path("protocol.md")
result = MarkItDown().convert_local(source)
output.write_text(result.markdown, encoding="utf-8")
provenance = {
"source": source.name,
"sha256": sha256(source.read_bytes()).hexdigest(),
"title": result.title,
"output": str(output),
}
print(provenance)
```
For very large files, compute the hash incrementally rather than loading the source again.
## 2. Binary Upload
Validate size and type before conversion. Use `BytesIO` plus explicit hints:
```python
from io import BytesIO
from markitdown import MarkItDown, StreamInfo
def convert_pdf_upload(payload: bytes, filename: str) -> str:
max_bytes = 25 * 1024 * 1024
if len(payload) > max_bytes:
raise ValueError("PDF exceeds the 25 MiB conversion limit")
if not payload.startswith(b"%PDF-"):
raise ValueError("Payload does not have a PDF signature")
result = MarkItDown().convert_stream(
BytesIO(payload),
stream_info=StreamInfo(
extension=".pdf",
mimetype="application/pdf",
filename=filename,
),
)
return result.markdown
```
Signature checks are only one validation layer; they do not make the parser sandboxed.
## 3. Batch Directory Conversion
```bash
python scripts/batch_convert.py inputs/ outputs/ \
--recursive \
--extensions .pdf .docx .pptx .xlsx .csv .html \
--manifest outputs/manifest.json
```
Properties:
- Local paths only
- Symlinks skipped
- Deterministic path ordering
- Relative directories preserved
- Output name retains the source suffix, e.g. `paper.pdf.md`
- Existing output skipped unless `--overwrite`
- Plugins off unless `--plugins`
- Nonzero exit when a conversion fails
Retry failed files after installing the missing format extra:
```bash
uv pip install "markitdown[pdf,docx,pptx,xlsx]==0.1.8"
python scripts/batch_convert.py inputs/ outputs/ --recursive --overwrite
```
Use a fresh output directory when comparing converter versions.
## 4. Literature Collection
Input naming convention:
```text
Author_Year_Title.pdf
```
Convert and build indexes:
```bash
python scripts/convert_literature.py papers/ literature/ \
--recursive \
--create-index
```
Organize by inferred year:
```bash
python scripts/convert_literature.py papers/ literature/ \
--recursive \
--organize-by-year \
--create-index
```
Each output contains provenance front matter:
```yaml
---
title: "Example title"
author: "Smith"
year: "2025"
source: "incoming/Smith_2025_Example_Title.pdf"
converted_at: "2026-10-01T17:00:00+00:00"
markitdown_version: "0.1.8"
---
```
The helper also writes `catalog.json` and `INDEX.md` when `--create-index` is requested. Metadata inferred from a filename is a convenience, not authoritative bibliographic metadata.
## 5. RAG Ingestion
Recommended sequence:
1. Validate and hash the source.
2. Convert with a pinned package/plugin version.
3. Keep source metadata separate from converted text.
4. Check non-empty output and expected section markers.
5. Sanitize active HTML and unsafe links if Markdown will be rendered.
6. Chunk by semantic headings with overlap.
7. Store source/hash/converter/page-or-section provenance with every chunk.
8. Mark chunks as untrusted source content in the retrieval prompt.
9. Retain the original source for verification.
Example conversion envelope:
```python
from dataclasses import asdict, dataclass
from hashlib import sha256
from importlib.metadata import version
from pathlib import Path
from markitdown import MarkItDown
@dataclass(frozen=True)
class ConvertedDocument:
source_name: str
source_sha256: str
converter_version: str
title: str | None
markdown: str
def convert_for_ingestion(path: Path) -> ConvertedDocument:
path = path.resolve(strict=True)
digest = sha256(path.read_bytes()).hexdigest()
result = MarkItDown().convert_local(path)
if not result.markdown.strip():
raise ValueError(f"Conversion produced no text: {path.name}")
return ConvertedDocument(
source_name=path.name,
source_sha256=digest,
converter_version=version("markitdown"),
title=result.title,
markdown=result.markdown,
)
```
Do not treat Markdown line numbers as stable page citations. MarkItDown does not expose PDF page coordinates.
## 6. Mixed-Format Study Folder
Use a two-pass workflow:
### Pass A: inventory
- Count files by extension and byte size.
- Detect duplicates by content hash.
- Exclude temporary, lock, and hidden files.
- Identify encrypted/unsupported inputs.
- Classify which files require external OCR/cloud processing.
### Pass B: convert
- Use local built-ins first.
- Route only failed scanned/complex files to an approved OCR/cloud path.
- Record the selected mode per file.
- Compare file count, failure count, and output count.
This minimizes cost and external data transfer.
## 7. Spreadsheet Orientation, Then Structured Analysis
Use MarkItDown to understand workbook organization:
```python
from markitdown import MarkItDown
preview = MarkItDown().convert_local("assay-results.xlsx").markdown
print(preview)
```
Then use a spreadsheet/dataframe library for calculations:
- Inspect formulas and cached values explicitly.
- Select the authoritative sheet/range.
- Preserve data types.
- Validate merged cells and hidden rows.
Do not parse a scientific numeric dataset back out of Markdown when the original workbook is available.
## 8. Presentation Review
```python
from markitdown import MarkItDown
result = MarkItDown().convert_local("lab-meeting.pptx")
```
Review:
- Slide boundaries
- Title hierarchy
- Speaker notes
- Text in charts/images
- Claims encoded only by color/position
If images carry essential meaning, add an approved vision description or OCR pass and label generated text as model-derived.
## 9. Scanned-PDF Escalation
1. Try built-in PDF extraction.
2. Detect unexpectedly short/empty output relative to page/file size.
3. Confirm the source is scanned by visual inspection.
4. Choose:
- Local OCR/layout parser for restricted data
- `markitdown-ocr` for approved vision-provider processing
- Document Intelligence for Azure layout/OCR
- Content Understanding for structured fields/multimodal processing
5. Validate a page sample and record the selected service/model/analyzer.
Do not automatically send failed local conversions to a cloud service.
## 10. Application-Controlled Remote Resource
The application—not MarkItDown—should:
- Validate destination and every redirect
- Enforce time and byte limits
- Restrict MIME types
- Apply authentication safely
- Produce a bounded `requests.Response`
Then:
```python
result = MarkItDown().convert_response(validated_response)
```
See `security.md` for the complete SSRF policy. A simple scheme check is not sufficient.
## 11. Deterministic Regression Corpus
Keep a small, redistributable corpus covering:
- Text-native and scanned PDFs
- DOCX headings/tables/equations
- PPTX notes/grouped shapes
- XLSX multiple sheets/merged cells
- CSV quoted fields and Unicode
- HTML links/lists/tables
- EPUB chapters
- JPEG/PNG metadata
For each fixture, assert:
- Conversion succeeds or fails with the expected category.
- Required headings/sentinel text are present.
- No source is unexpectedly routed to network/cloud.
- Output is UTF-8 and below a reasonable size.
- Package/plugin versions are recorded.
Avoid asserting a full byte-for-byte Markdown snapshot unless exact formatting stability is required; semantic assertions are less brittle.
## 12. Conversion Quality Report
Track per file:
- Source identifier and hash
- Bytes and extension
- Converter mode
- MarkItDown/plugin version
- Output characters/lines
- Optional title
- Warning/failure category
- Manual validation status
- External provider/analyzer, if any
The bundled batch manifest provides a baseline for these records.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |