Lee Powell · Architect of Scrivener and Scapple · Lumen & Lever

Most AI document pipelines fail before the model is ever called. Tables become paragraphs. Lists collapse into prose. Annotations are detached from context. Page references disappear. Source traceability is replaced by a confidence score. The structure that gave the document its meaning is gone before retrieval runs, and no retrieval recovers it.

This is the layer that gets underestimated. I have worked on it for a long time. Long before retrieval-augmented generation existed, I wrote the production C++ RTF reader and writer that ships inside Scrivener for Windows and Linux, used by hundreds of thousands of long-form writers across novels, dissertations, and screenplays. That code was eventually sold as a white-label engagement to Literature & Latte, who continue to maintain Scrivener today.

RTF is not a glamorous format. It is also not going away. It is still the wire format inside Microsoft Outlook for rich-text email. It is still produced by court reporting systems, medical records platforms, government archives, and twenty-year-old legal practice management systems. When a law firm pulls a thousand contracts out of an old document store, a meaningful portion of them are RTF. When a hospital exports decades of clinical notes, a meaningful portion are RTF. When you scrape an Outlook MSG file, the body is RTF.

The standard pipeline path converts these documents to plain text immediately. The structure goes. Tables become paragraphs. Section headings become bold lines indistinguishable from body emphasis. Numbered clauses lose their numbering. Footnotes lose their links to the text they annotate. The model performs less well, the answers are less reliable, and the cause sits a layer below where anyone is looking.

I built rtfstruct to fix that layer. It is a Python 3.11+ RTF reader and writer that produces a neutral document AST, preserves structure all the way through, exposes diagnostics rather than swallowing them, and supports clean roundtrip back to RTF. Apache-2.0. Part of Sourcetrace by Lumen & Lever, the document structure layer for AI pipelines that I now run as a consultancy.

What follows is why the layer matters, what is wrong with the existing tools, what rtfstruct does differently, and why ten years of writing production RTF code for a writing application turned out to be the right preparation for an AI ingestion problem.

Why structure is not optional

The phrase that captures the architectural mistake is "structure-before-model." When a workflow involves structurally rich documents, the first decision is not which model. It is what intermediate representation. A blood test report is not text. It is a structured clinical record with analytes, values, units, reference ranges, abnormal flags, methods, and trend history. The same is true of leases, bank statements, invoices, pathology reports, contracts. Each of them carries its meaning in the structure. Flatten the structure and the meaning becomes inferred rather than read.

The pipeline then asks the model to reconstruct probabilistically what the document already contained as deterministic structure. The result is silent error. Reference ranges honoured in one record and missed in another. Unit conversions correct for SI units and silently wrong for imperial. Clause cross-references followed in clean documents and lost in legacy ones. Nothing in the model's output makes the failure visible. The diagnostic surface was the source structure, and the source structure was discarded at ingestion.

The model should reason over the AST, not over the document. Where the source has structure, structure is the system. The model is the consultant the system calls when structure alone cannot answer the question.

That is the doctrine. It is also the reason a tool like rtfstruct exists at all. Flatten RTF to text before AI sees it and the model's job is harder, the evaluation surface is smaller, and the audit trail is unfit for production. Preserve structure into an AST and the same workflow ships, evaluates, and audits cleanly.

What the existing Python tools actually do

There are a handful of Python libraries in the RTF space. None of them does what an AI ingestion pipeline needs. The honest landscape:

The gap has sat in the Python ecosystem for years. There is no AST-first RTF reader and writer designed for structured pipelines, with first-class diagnostics and source spans, that is also open source. The closest thing is striprtf, which is excellent at exactly the opposite of what AI ingestion needs.

That is the gap rtfstruct fills.

What rtfstruct does differently

Four things matter. None of them are individually exotic. The combination is the point.

1. The AST is the public contract

rtfstruct parses RTF into a neutral document AST that preserves paragraphs, inline styles, lists, tables, links, fields, footnotes, endnotes, annotations, images, metadata, source spans, and recoverable diagnostics. Every other operation in the library is defined against the AST. JSON export, Markdown export, RTF roundtrip, and integration helpers all read from the AST. It is not an internal representation that gets discarded after parsing. It is the artefact.

from rtfstruct import parse_rtf

document = parse_rtf(r"{\rtf1\ansi Hello, \b world\b0!}")

print(document.to_json())
print(document.to_markdown())
print(document.to_rtf())

The AST distinguishes between a heading paragraph and a body paragraph, between a list item and a regular paragraph, between a footnote reference and a footnote body, between a table cell and a table row. None of that is in the rendered text. All of it is in the source RTF. A pipeline that sees the AST sees the document. A pipeline that sees flattened text sees a wall of words.

2. Diagnostics are returned with the document

RTF in production is messy. Twenty-year-old legacy documents have malformed control words, broken Unicode escapes, codepage mismatches. Most parsers either fail loudly or silently drop the affected content. Neither is what a production pipeline needs.

rtfstruct returns diagnostics as part of the document object. If a malformed Unicode escape is recovered, the recovered character comes back along with a diagnostic carrying the severity, code, message, and source location. The pipeline then makes explicit decisions: log it, surface it for human review, reject the document, or proceed with confidence flagged.

from rtfstruct import parse_rtf

document = parse_rtf(r"{\rtf1\u999999?}")

for diagnostic in document.diagnostics:
    print(diagnostic.severity.value, diagnostic.code, diagnostic.message)

The value of this is invisible until the production system encounters its first malformed document. After that it becomes the difference between a pipeline that fails opaquely and a pipeline that fails informatively.

3. Source spans map AST nodes back to byte offsets

For tools that need to highlight a region in the original RTF (legal review interfaces, document comparison tools, evidence-traceable AI systems), source spans are mandatory. rtfstruct supports them as an opt-in parser option. When enabled, every AST node carries a span pointing to the byte range in the source RTF that produced it. This is the foundation for the kind of source traceability that production AI systems need but rarely build, because retrofitting it later is structurally impossible.

from rtfstruct import ParserOptions, parse_rtf

document = parse_rtf(
    r"{\rtf1 Hello}",
    options=ParserOptions(track_spans=True),
)

4. Roundtrip without semantic loss

The reader and the writer share the same AST. A document parsed in, edited in place, and written back out preserves the structural choices it carried. This sounds straightforward and it is not. Most parsers that also write tend to lose information on the round trip. Inline style runs collapse, table cell properties drift, list numbering restarts. rtfstruct is tested for semantic roundtrip across inline formatting, metadata, fields and links, footnotes, annotations, lists, tables, images, and Unicode recovery.

from rtfstruct import read_rtf, write_rtf

document = read_rtf("input.rtf")
# inspect, modify, validate, classify...
write_rtf(document, "output.rtf")

Why ten years of Scrivener was the right preparation

Maintaining a parser in production for a long time produces a particular kind of engineering scar tissue. RTF is a 38-year-old format with hundreds of control words, dozens of edge cases that only appear in real documents, and at least three different lineages of generators producing subtly different output (Microsoft Word, OpenOffice, and various enterprise systems built up over decades). The official specification documents some of this. The rest is learned by debugging support tickets from a writer in Iceland whose decade-old document refuses to load correctly.

The Scrivener parser went through that learning the hard way. Pressure-tested across hundreds of thousands of writers, on Windows and Linux, on documents ranging from short stories to thousand-page novels, dissertations with mixed-language sections, screenplays with industry-specific formatting, academic papers with footnotes and citations and embedded equations. By the time it shipped to production it handled malformed documents, codepage drift, Unicode escapes outside valid ranges, and recovery from errors that simpler parsers would treat as fatal. That code was eventually sold as a white-label engagement to Literature & Latte.

rtfstruct is not a port of the Scrivener parser. It is a fresh codebase, written for Python, designed for structured AI pipelines rather than for an interactive writing application. The thinking behind it is shaped by the ten years I spent watching writers feed Scrivener documents that should not have parsed and watching the parser handle them anyway. The decisions about what to recover, what to flag, what to expose as diagnostics, and what shape the AST should take are decisions made before, in production, under load, with real users. That experience compresses years of design discovery into a starting point most parser projects do not have.

A note on provenance: I do not maintain Scrivener and have not seen its codebase in years. The intellectual property was sold to Literature & Latte. The lessons stayed with me. rtfstruct is a different library, for a different purpose, written for a different language, drawing on the same engineering instinct that produced the original.

What this is for, and what it is not

rtfstruct is for systems where document structure still matters. AI ingestion pipelines, RAG systems, legal discovery, banking and financial archives, forensic document tracing, publishing pipelines, long-form document intelligence, and any pipeline that processes RTF as input rather than discarding it as a legacy format.

It is not a plain-text stripper. If all you need is the words, striprtf is excellent and you should use it. It is not a Markdown converter for casual use. If your input is well-formed contemporary RTF and your output is human-readable Markdown, several lighter tools will do the job. It is not a renderer. There is no HTML output, no styled rendering, no display library. It is an AST reader and writer.

The library is at version 0.1, currently labelled pre-alpha because the API will continue to evolve as integration patterns surface. The core reader, AST, JSON exporter, Markdown exporter, and RTF writer are working today. Tests cover inline formatting, metadata, fields and links, footnotes, annotations, lists, tables, images, Unicode and codepage recovery, diagnostics, source spans, and semantic roundtrip. If you find a document it does not handle correctly, file an issue with the offending file and I will look at it.

The deeper thesis

RTF is the format I started with because it is where my engineering history sits. The thesis is bigger than RTF.

The AI industry treats document ingestion as a preprocessing step rather than as the architectural foundation it actually is. The model is given the leftovers and asked to reconstruct what was thrown away. The answer is not a better model. The answer is preserving structure all the way through. Tables stay tables. Clauses stay clauses. Source pages stay referenceable. Diagnostics surface where confidence is low. The model reasons over the structured representation. Validation runs deterministically. The human review checkpoint sees what the system saw. The audit trail traces every step back to the source byte range in the original document.

This is what Sourcetrace is. rtfstruct handles the RTF case. pdfstruct handles PDFs. Other formats follow the same pattern. The commercial work I do at Lumen & Lever applies the same discipline at the architectural level: helping executives and boards establish control over AI before the structural mistakes compound.

The tools are free. Their purpose is adoption, technical credibility, and the slow accumulation of evidence that the structure-before-model thesis is right. Build with them. File issues. Fork them if you find a better path.

Try it

rtfstruct is on GitHub under Apache-2.0. Documentation is on GitHub Pages. The library installs from source today and will be on PyPI when the API stabilises.

• Repository: github.com/keny369/rtfstruct
• Documentation: keny369.github.io/rtfstruct
• Sourcetrace RTF: lumenandlever.com/tools/sourcetrace-rtf

Lee Powell is the architect of Scrivener and Scapple, a former enterprise architect at Commonwealth Bank and Deutsche Bank, and the founder of Lumen & Lever, an AI governance consultancy advising executives and boards on structural AI readiness.