<p><a href="https://github.com/meownoirsoft/symport" rel="noopener noreferrer">Symport</a> is an AI document ingestion pipeline that turns a phone photo of any paper document — receipt, EOB, prescription, utility bill — into structured JSON, then stores it in Postgres with embeddings for semantic search. The full flow is: image upload → Sharp preprocessing → GPT-4o vision extraction → normalized JSON → Postgres + pgvector. I built it because I hate paper and I also lose paper.</p> <p>This post walks through how the pipeline actually works, including the prompt engineering decisions that make extraction reliable enough to trust and the fallback layers that keep the app useful when extraction fails.</p> <h2> TL;DR </h2> <ul> <li> <strong>Stack:</strong> Sharp for image preprocessing, GPT-4o for vision extraction, Prisma + Postgres + pgvector for storage and semantic search.</li> <li> <strong>The extraction prompt does most of the work:</strong> explicit date context to fight year hallucinations, constrained <code>type</code>/<code>category</code> enums for predictable downstream branching, and a strict "JSON only, no markdown" tail.</li> <li> <strong>User correction loop:</strong> Users can add freeform feedback ("the drug name is metformin, not metFORMIN") and re-run extraction; the feedback gets injected back into the system prompt.</li> <li> <strong>Schema choice:</strong> A single <code>extractedData</code> JSON column instead of per-type tables, with a denormalized <code>searchText</code> field for fast keyword search and an <code>embedding</code> column for semantic search.</li> <li> <strong>Two fallback layers:</strong> Document still saves if there's no API key, and still saves with an error summary if extraction throws — nothing is ever lost because AI had a bad day.</li> </ul> <h2> What the pipeline does </h2> <p>The flow is straightforward:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Image upload → sharpen + encode → GPT-4o vision → structured JSON → Postgres + embeddings </code></pre> </div> <p>A user photographs a receipt, an insurance EOB, a prescription, a utility bill — anything on paper. The app returns a structured JSON object with the relevant fields extracted, tagged, and ready to query. No manual data entry.</p> <h2> Step 1: Image preprocessing </h2> <p>Raw phone photos are large and often noisy. Before sending to the vision model, every image gets sharpened and re-encoded using Sharp:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="kd">const</span> <span class="nx">rawBuffer</span> <span class="o">=</span> <span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="k">await</span> <span class="nx">file</span><span class="p">.</span><span class="nf">arrayBuffer</span><span class="p">());</span> <span class="kd">const</span> <span class="nx">buffer</span> <span class="o">=</span> <span class="k">await</span> <span class="nf">sharpenAndEncode</span><span class="p">(</span><span class="nx">rawBuffer</span><span class="p">);</span> </code></pre> </div> <p>Sharp handles resizing, sharpening, and JPEG re-encoding in one pass. This serves two purposes: it reduces the payload size for the API call, and sharpening improves OCR accuracy on text-heavy documents like receipts. A blurry photo of small print is genuinely harder for vision models — a little preprocessing pays off.</p> <p>The processed image gets saved to disk as the source of truth, then the buffer goes to the extraction pipeline:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="kd">const</span> <span class="nx">filename</span> <span class="o">=</span> <span class="s2">`</span><span class="p">${</span><span class="nf">randomBytes</span><span class="p">(</span><span class="mi">12</span><span class="p">).</span><span class="nf">toString</span><span class="p">(</span><span class="dl">"</span><span class="s2">hex</span><span class="dl">"</span><span class="p">)}</span><span class="s2">.jpg`</span><span class="p">;</span> <span class="k">await</span> <span class="nf">writeFile</span><span class="p">(</span><span class="nx">fullPath</span><span class="p">,</span> <span class="nx">buffer</span><span class="p">);</span> <span class="kd">const</span> <span class="nx">extracted</span> <span class="o">=</span> <span class="k">await</span> <span class="nf">extractFromImageBuffer</span><span class="p">(</span><span class="nx">buffer</span><span class="p">);</span> </code></pre> </div> <p>Random hex filename prevents collisions and avoids leaking any metadata about the document in the path.</p> <h2> Step 2: The extraction prompt </h2> <p>This is where most of the real engineering lives. The system prompt does a lot of work to make the model's output consistent and parseable.</p> <p>The prompt has three parts assembled at startup:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="kd">const</span> <span class="nx">EXTRACTION_SYSTEM_HEAD</span> <span class="o">=</span> <span class="s2">`You are a document extraction assistant. Analyze the image and extract structured data. Current date context: We are in 2025. Use 2025 (not 2023 or other past years) for any ambiguous or partial dates when no stronger clue is present. Use context clues from the document text to infer the correct year: - "2025 taxes due in 2026" → tax year 2025 - "Plan year 2025", "Coverage year 2025" → use 2025 - "Due in 2026" on a tax-related doc often refers to tax year 2025 Respond with a single JSON object. Include "type", "category", "title", and "tags" in every response. - "type": one of rx_receipt, eob, utility_bill, general - "category": one of receipt, financial, medical, government, legal, identity, general - "title" (required, 2-5 words max): Short label only. No sentences. - "tags": array of 3–8 short labels. No spaces; use underscores if needed. `</span><span class="p">;</span> </code></pre> </div> <p>A few decisions worth calling out here:</p> <p><strong>Explicit date context.</strong> Vision models can hallucinate dates, especially on documents where the year is ambiguous. Anchoring the prompt with the current year and showing examples of how to reason about year context dramatically reduces date errors. Without this, a 2025 tax document might come back with 2023 dates because the model defaulted to its training data.</p> <p><strong>Constrained type and category values.</strong> Giving the model an explicit enum for <code>type</code> and <code>category</code> means you get predictable values you can branch on in code. Open-ended classification produces inconsistent strings that are annoying to handle downstream.</p> <p><strong>Short title constraint.</strong> "2-5 words max, no sentences" prevents the model from writing a summary disguised as a title. You want "Prescription receipt" not "This document appears to be a receipt from Walgreens for a prescription medication."</p> <p>The tail of the prompt closes with:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="kd">const</span> <span class="nx">EXTRACTION_SYSTEM_TAIL</span> <span class="o">=</span> <span class="s2">` Use null for missing values. Amounts as numbers. Dates as YYYY-MM-DD; use context clues for year. Output only valid JSON, no markdown or explanation.`</span><span class="p">;</span> </code></pre> </div> <p>"Output only valid JSON, no markdown or explanation" is load-bearing. Without it, GPT-4o will frequently wrap the response in a markdown code block. The extraction code handles that case anyway, but telling the model not to do it reduces the cleanup work:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="c1">// Strip optional markdown code block</span> <span class="kd">let</span> <span class="nx">jsonStr</span> <span class="o">=</span> <span class="nx">raw</span><span class="p">;</span> <span class="kd">const</span> <span class="nx">match</span> <span class="o">=</span> <span class="nx">raw</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="sr">/``</span><span class="err">` </span><span class="p">{</span><span class="o">%</span> <span class="nx">endraw</span> <span class="o">%</span><span class="p">}</span> <span class="p">(?:</span><span class="nx">json</span><span class="p">)?</span><span class="err">\</span><span class="nx">s</span><span class="o">*</span><span class="p">([</span><span class="err">\</span><span class="nx">s</span><span class="err">\</span><span class="nx">S</span><span class="p">]</span><span class="o">*</span><span class="p">?)</span> <span class="p">{</span><span class="o">%</span> <span class="nx">raw</span> <span class="o">%</span><span class="p">}</span> <span class="s2">```/); if (match) jsonStr = match[1].trim(); </span></code></pre> </div> <h2> Step 3: User feedback loop </h2> <p>One of the more useful features is the ability to correct extractions. If the model gets something wrong — misreads a drug name, gets the date wrong, miscategorizes the document — the user can add a correction note and re-run extraction. That feedback gets injected directly into the system prompt:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="k">if </span><span class="p">(</span><span class="nx">options</span><span class="p">?.</span><span class="nx">userFeedback</span><span class="p">?.</span><span class="nf">trim</span><span class="p">())</span> <span class="p">{</span> <span class="nx">systemContent</span> <span class="o">+=</span> <span class="s2">`\n\nIMPORTANT - User feedback on this document (apply these corrections): </span><span class="p">${</span><span class="nx">options</span><span class="p">.</span><span class="nx">userFeedback</span><span class="p">.</span><span class="nf">trim</span><span class="p">()}</span><span class="s2">`</span><span class="p">;</span> <span class="p">}</span> </code></pre> </div> <p>This means the model gets a second pass with explicit correction instructions. In practice it works well — "the drug name is metformin not metFORMIN" or "this is a 2025 EOB not 2024" gets applied reliably.</p> <p>The feedback also gets stored in the database as <code>extractionNotes</code> on the document, so you have a record of what was corrected.</p> <h2> Step 4: The data model </h2> <p>The Prisma schema keeps things straightforward:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="nx">model</span> <span class="nx">Document</span> <span class="p">{</span> <span class="nx">id</span> <span class="nb">String</span> <span class="p">@</span><span class="nd">id</span> <span class="p">@</span><span class="nd">default</span><span class="p">(</span><span class="nf">cuid</span><span class="p">())</span> <span class="nx">imagePath</span> <span class="nb">String</span><span class="p">?</span> <span class="nx">noteText</span> <span class="nb">String</span><span class="p">?</span> <span class="nx">status</span> <span class="nb">String</span> <span class="p">@</span><span class="nd">default</span><span class="p">(</span><span class="dl">"</span><span class="s2">pending</span><span class="dl">"</span><span class="p">)</span> <span class="nx">extractedData</span> <span class="nx">Json</span> <span class="nx">searchText</span> <span class="nb">String</span><span class="p">?</span> <span class="nx">embedding</span> <span class="nc">Unsupported</span><span class="p">(</span><span class="dl">"</span><span class="s2">vector(1536)</span><span class="dl">"</span><span class="p">)?</span> <span class="nx">tags</span> <span class="nb">String</span><span class="p">[]</span> <span class="nx">extractionNotes</span> <span class="nb">String</span><span class="p">?</span> <span class="nx">createdAt</span> <span class="nx">DateTime</span> <span class="p">@</span><span class="nd">default</span><span class="p">(</span><span class="nf">now</span><span class="p">())</span> <span class="nx">updatedAt</span> <span class="nx">DateTime</span> <span class="p">@</span><span class="nd">updatedAt</span> <span class="p">}</span> </code></pre> </div> <p>A few design choices here:</p> <p><strong><code>extractedData</code> is a JSON blob.</strong> Rather than creating separate tables for each document type (receipts, EOBs, utility bills), all extracted data lives in a single JSON column. This makes the schema flexible — different document types have different fields, and a rigid relational schema would be a constant maintenance burden as new types are added.</p> <p><strong><code>searchText</code> is denormalized.</strong> After extraction, key fields get pulled out and concatenated into a single <code>searchText</code> string for full-text search. This is faster to query than parsing JSON at search time:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="k">export</span> <span class="kd">function</span> <span class="nf">buildSearchText</span><span class="p">(</span><span class="nx">data</span><span class="p">:</span> <span class="nx">ExtractedDoc</span><span class="p">):</span> <span class="kr">string</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">parts</span><span class="p">:</span> <span class="kr">string</span><span class="p">[]</span> <span class="o">=</span> <span class="p">[];</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="nf">effectiveTitle</span><span class="p">(</span><span class="nx">data</span><span class="p">));</span> <span class="k">if </span><span class="p">(</span><span class="dl">"</span><span class="s2">summary</span><span class="dl">"</span> <span class="k">in</span> <span class="nx">data</span> <span class="o">&amp;&amp;</span> <span class="nx">data</span><span class="p">.</span><span class="nx">summary</span><span class="p">)</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="nc">String</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">summary</span><span class="p">));</span> <span class="k">if </span><span class="p">(</span><span class="dl">"</span><span class="s2">drug_name</span><span class="dl">"</span> <span class="k">in</span> <span class="nx">data</span> <span class="o">&amp;&amp;</span> <span class="nx">data</span><span class="p">.</span><span class="nx">drug_name</span><span class="p">)</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="nc">String</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">drug_name</span><span class="p">));</span> <span class="k">if </span><span class="p">(</span><span class="dl">"</span><span class="s2">insurer</span><span class="dl">"</span> <span class="k">in</span> <span class="nx">data</span> <span class="o">&amp;&amp;</span> <span class="nx">data</span><span class="p">.</span><span class="nx">insurer</span><span class="p">)</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="nc">String</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">insurer</span><span class="p">));</span> <span class="k">if </span><span class="p">(</span><span class="dl">"</span><span class="s2">tags</span><span class="dl">"</span> <span class="k">in</span> <span class="nx">data</span> <span class="o">&amp;&amp;</span> <span class="nb">Array</span><span class="p">.</span><span class="nf">isArray</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">tags</span><span class="p">))</span> <span class="p">{</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">push</span><span class="p">(...</span><span class="nx">data</span><span class="p">.</span><span class="nx">tags</span><span class="p">.</span><span class="nf">map</span><span class="p">(</span><span class="nx">t</span> <span class="o">=&gt;</span> <span class="nc">String</span><span class="p">(</span><span class="nx">t</span><span class="p">).</span><span class="nf">trim</span><span class="p">()).</span><span class="nf">filter</span><span class="p">(</span><span class="nb">Boolean</span><span class="p">));</span> <span class="p">}</span> <span class="k">return</span> <span class="nx">parts</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="dl">"</span><span class="s2"> </span><span class="dl">"</span><span class="p">);</span> <span class="p">}</span> </code></pre> </div> <p><strong><code>embedding</code> for semantic search.</strong> After the document is saved, an embedding gets generated from <code>searchText</code> and stored in a pgvector column. This enables semantic search — finding "cholesterol medication" when the document says "lipitor" — without a separate vector database. Just pgvector as a Postgres extension.</p> <h2> Step 5: Graceful degradation </h2> <p>The pipeline has two fallback layers. First, if there's no API key configured, the document still gets saved — just without extraction:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="k">if </span><span class="p">(</span><span class="o">!</span><span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">OPENAI_API_KEY</span><span class="p">)</span> <span class="p">{</span> <span class="k">await</span> <span class="nx">prisma</span><span class="p">.</span><span class="nb">document</span><span class="p">.</span><span class="nf">create</span><span class="p">({</span> <span class="na">data</span><span class="p">:</span> <span class="p">{</span> <span class="na">imagePath</span><span class="p">:</span> <span class="nx">filename</span><span class="p">,</span> <span class="na">status</span><span class="p">:</span> <span class="dl">"</span><span class="s2">pending</span><span class="dl">"</span><span class="p">,</span> <span class="na">extractedData</span><span class="p">:</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">general</span><span class="dl">"</span><span class="p">,</span> <span class="na">title</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Document</span><span class="dl">"</span><span class="p">,</span> <span class="na">summary</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Extraction skipped (no OPENAI_API_KEY)</span><span class="dl">"</span> <span class="p">},</span> <span class="na">searchText</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Extraction skipped</span><span class="dl">"</span><span class="p">,</span> <span class="na">tags</span><span class="p">:</span> <span class="p">[],</span> <span class="p">},</span> <span class="p">});</span> <span class="p">}</span> </code></pre> </div> <p>Second, if extraction throws, the document still gets saved with an error summary rather than failing the whole request:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="k">try</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">extracted</span> <span class="o">=</span> <span class="k">await</span> <span class="nf">extractFromImageBuffer</span><span class="p">(</span><span class="nx">buffer</span><span class="p">);</span> <span class="nx">extractedData</span> <span class="o">=</span> <span class="nx">extracted</span> <span class="k">as</span> <span class="nb">Record</span><span class="o">&lt;</span><span class="kr">string</span><span class="p">,</span> <span class="nx">unknown</span><span class="o">&gt;</span><span class="p">;</span> <span class="p">}</span> <span class="k">catch </span><span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span> <span class="nx">extractedData</span> <span class="o">=</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">general</span><span class="dl">"</span><span class="p">,</span> <span class="na">summary</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Extraction failed: </span><span class="dl">"</span> <span class="o">+</span> <span class="p">(</span><span class="nx">err</span> <span class="k">instanceof</span> <span class="nb">Error</span> <span class="p">?</span> <span class="nx">err</span><span class="p">.</span><span class="nx">message</span> <span class="p">:</span> <span class="dl">"</span><span class="s2">Unknown error</span><span class="dl">"</span><span class="p">),</span> <span class="p">};</span> <span class="p">}</span> </code></pre> </div> <p>The image is always saved. The extraction is best-effort. Users can re-trigger extraction manually, or add correction notes and re-run. Nothing gets lost because AI had a bad day.</p> <h2> The model </h2> <p>The extraction model is configurable via environment variable with <code>gpt-4o</code> as the default:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight typescript"><code><span class="kd">const</span> <span class="nx">model</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">OPENAI_EXTRACTION_MODEL</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">gpt-4o</span><span class="dl">"</span><span class="p">;</span> </code></pre> </div> <p>GPT-4o is the right choice here — it's genuinely better than smaller models at reading degraded document images, handwriting, and small print. For this specific task the quality difference is noticeable enough to justify the cost. Document extraction is a write-time operation (not a search-time one), so the latency and cost are acceptable.</p> <h2> What I'd do differently </h2> <p>A few things I'd change with hindsight:</p> <p><strong>Add a confidence score.</strong> The model sometimes hedges on fields it's uncertain about — a low-confidence flag on individual fields would let the UI highlight things that need user review rather than silently storing potentially wrong data.</p> <p><strong>Chunk large documents.</strong> A single-page receipt is fine. A multi-page insurance EOB or medical record is harder — the model gets less accurate as documents get longer or more complex. Chunking multi-page documents and merging the extracted JSON would improve accuracy on longer content.</p> <p><strong>Store the raw extraction response.</strong> Right now only the normalized result gets stored. Keeping the raw model output alongside it would make debugging extraction issues much easier.</p> <p>The full source is on GitHub at <a href="https://github.com/meownoirsoft/symport" rel="noopener noreferrer">github.com/meownoirsoft/symport</a>. The extraction logic lives in <code>lib/extract.ts</code> and the ingestion endpoint is <code>app/api/documents/route.ts</code> if you want to dig in.</p> <h2> FAQ </h2> <h3> Why GPT-4o for vision instead of a cheaper model or open-source alternative? </h3> <p>GPT-4o reads degraded phone photos, handwriting, and small print noticeably better than smaller or open-source vision models. For document extraction, getting the dates and amounts wrong is a much bigger problem than the per-call cost, so paying for the better model is worth it. Extraction runs once at write time, not on every read.</p> <h3> How do you stop the model from hallucinating dates or amounts? </h3> <p>The biggest wins are anchoring "current year" context in the system prompt with explicit examples, asking the model to use context clues from the document itself ("plan year 2025", "due in 2026" → tax year 2025), and constraining types/categories to enums so the model can't drift. The user-feedback loop catches anything that still slips through.</p> <h3> Why store extracted data as a single JSON column instead of typed tables? </h3> <p>Different document types have different fields — a prescription receipt has <code>drug_name</code> and <code>pharmacy</code>, an EOB has <code>insurer</code> and <code>claim_id</code>, a utility bill has <code>account_number</code> and <code>service_period</code>. A relational schema for every variant would be a constant migration treadmill. JSON keeps the schema flexible, and the denormalized <code>searchText</code> and <code>embedding</code> columns make queries fast where it matters.</p> <h3> What happens if the model returns invalid JSON? </h3> <p>The extraction code strips optional markdown code fences (<br> <br> <code>json ...</code><br> <br> ) and parses the rest. If parsing still fails, the document saves with an error summary in the <code>extractedData.summary</code> field rather than throwing — the user can re-run extraction or add a correction note. The image and metadata are never lost.</p> <h3> Can I use this with Anthropic's Claude vision instead of GPT-4o? </h3> <p>Yes. The extraction prompt is provider-agnostic and the model is configurable via <code>OPENAI_EXTRACTION_MODEL</code>. Swap the SDK call for the Anthropic SDK (or route through OpenRouter to avoid a code change) and Claude's vision models work as a drop-in alternative. The "JSON only, no markdown" instruction is even more important on Claude — it likes to explain itself by default.</p> <h3> How do you handle multi-page documents? </h3> <p>Today the pipeline treats each photo as a single document, which is fine for single-page items (receipts, prescriptions). For multi-page EOBs or medical records, the right next step is to chunk the document into pages, run extraction per page, and merge the resulting JSON into a single record. Adding that is on the to-do list.</p>