A multi-tenant pipeline that turns raw portfolio-company financials into a standardised, queryable Budget-vs-Actual — every line mapped to one canonical Chart of Accounts, every figure traceable back to its source cell, every mapping decision either made or confirmed by a human.
Deterministic code does the parsing. Claude only does two things — classify what a sheet is, and propose a canonical Chart-of-Accounts code for each line item. A human analyst confirms or overrides every mapping before it's trusted long-term — and that decision is written back into per-entity memory so the same label is never asked twice.
One workbook upload sets off a five-phase chain. Each phase writes its result to Postgres, then hands the next phase off via a Cloudflare Queue message.
bva_pendingEach phase completes, writes its status to pipeline_jobs, and pushes the next phase's payload onto the same Cloudflare Queue. Two of the five involve Claude — highlighted below.
display_rows; idempotent — re-running an already-normalised job short-circuits with no side effects.bva_pending instead of failing.Every other line of logic — parsing, persistence, aggregation, lineage — is deterministic code. These are the exact prompts and schemas, as written.
IGNORE — it's a derived output, not source datadoc_type derived deterministically — single type if all sheets agree, else MIXEDparent_label, indent_level, sheet, statement_typeREV, COGS, GP, OPEX, EBITDA, DA, EBIT, TAX, NI, BS, CF — dot notation for sub-codes, e.g. COGS.Teacher.LSBS.Asset.Current.Cash, BS.Asset.Fixed.PPE, BS.Liab.Current.AP, BS.Equity.RetainedCOGS.CSAT for a SaaS company vs. COGS.Teacher for a tutoring companyentity_profiles.confirmed_mappings are auto-mapped from memory — zero Claude callsThree lanes: the client, the Cloudflare Worker (ingestion + queue consumer + both deterministic parsing and the classify agent call), and FastAPI backed by Neon Postgres + R2.
calibrax-pipeline — max_batch_size: 1 (messages processed one at a time, not true batching despite the per-message loop in the consumer), max_retries: 3, no dead_letter_queue configured. R2 bucket is tnba-files, with CORS open to all origins for GET/PUT/HEAD/POST/DELETE — required for the client's direct presigned-URL upload in step 1 of the user journey.
The same pipeline described above, from the perspective of the portfolio analyst operating it rather than the system executing it.
Picks the portfolio company from /entities and uploads a budget, actuals, or mixed workbook. No sheet selection or formatting is required at this step.
Runs unattended — each sheet is typed, financial sheets are extracted, and rows are persisted. No analyst action is required at this stage.
Previously confirmed labels are reused from entity memory; unseen labels are mapped to a canonical code with a confidence score and a tier.
High-confidence mappings (tier 1) are auto-confirmed and aggregation runs immediately, provided both Budget and Actual rows exist for the entity.
Opens /mapping-review/{entity_id}, sorted by ascending confidence. Confirms or overrides tier-2 mappings; an override recomputes the BvA automatically.
Reads the statement view, traces any figure to its source cell via lineage and drilldown, and exports the BvA as a formatted workbook.
Tracked via status, current_phase, an append-only phase_logs array, and a phases_complete JSONB map.
bva_pending is a deliberate, non-error parked state — it fires when an entity has mapped rows from only one of ACTUAL or BUDGET. The job sits there correctly until the matching file arrives and re-triggers /pipeline/bva readiness, rather than treating an out-of-order upload as a failure.
Mapping runs automatically and BvA computes right after — review happens in parallel, sorted by ascending confidence so the most uncertain mappings surface first.
High confidence, unambiguous label. Analyst clicks confirm — no value changes, it's locked in as reviewed.
Plausible but ambiguous, or appears in an unexpected context. Already flowing into BvA — flagged for the analyst to verify against source before it's trusted.
Low confidence, wrong language, a reconciliation artifact, or unmapped (UNKNOWN). Analyst replaces the code directly.
entity_profiles.confirmed_mappings. The next file from the same company checks this memory before calling Claude at all — so mapping accuracy compounds per entity over time. /pipeline/backfill-suggestions retroactively applies a new confirmation to previously unmapped rows from earlier files.
Four read endpoints, each answering a different audit question — none re-run any computation; all read straight off persisted lineage. Lineage and drilldown are the two that carry figures back to an exact source cell, and are marked accordingly.
parse_formula_refs() — regex-extracts every Sheet!Cell reference out of any Excel formula string, including quoted sheet names and multi-cell SUM() expressionsperiod_variants() — matches a period against both "2025-01" and "Jan-25" label formats, since source sheets are never consistent about which they useOne row per portfolio company.
One row per upload.
One row per upload's journey through the 5 phases.
One row per parsed line item.
One row per label → canonical-code decision.
Raw uploaded workbooks — never re-read after parse extracts rows from them.
Rows shaded purple are the two agent call sites — everything else is deterministic.
| Function | Runs on | What it does | Writes / returns |
|---|---|---|---|
| classifyFile()Agent | Worker + Claude | Per-sheet doc-type, currency, period-range classification, statement boundary trimming. Persists incrementally after each sheet. | file_metadata.doc_typequeues: parse |
| parseFile() | Worker, deterministic | SheetJS heuristics — date/period headers, currency blocks, unit tags — across all financial sheets. | rows[]POST /pipeline/parse |
| run_parse() | FastAPI | Persists parsed rows as display_rows; on failure writes parse_debug for diagnosis. | display_rowsqueues: normalise |
| pipeline_normalise() | FastAPI | Idempotent — re-running a normalised job short-circuits. Prepares rows for canonical mapping. | pipeline_jobs.statusqueues: map |
| pipeline_map()Agent | FastAPI + Claude | Advisory-locks per file_id; checks confirmed_mappings memory first, batches only new labels (25 at a time) to Claude for canonical_code + confidence + tier. | mapping_recordsqueues: bva |
| pipeline_bva() | FastAPI | Readiness-gated: requires both ACTUAL and BUDGET mapped rows for the entity, else parks at bva_pending. Aggregates by canonical_code & period. | BvA aggregates |
| mapping_action() | FastAPI | Analyst confirms or overrides a mapping; override updates display_rows and writes the pair back into entity memory. | entity_profiles.confirmed_mappingsPOST /mapping-review/{id}/action |
| recompute_bva() | FastAPI | Re-runs aggregation for an entity after mapping corrections, without re-parsing source files. | BvA aggregatesPOST /recompute-bva/{entity_id} |
| backfill_suggestions() | FastAPI | Retroactively applies a newly-confirmed label→code mapping to previously unmapped rows from earlier files. | mapping_recordsPOST /pipeline/backfill-suggestions/{entity_id} |
| get_statement() | FastAPI | Serves Actual or Budget statement views by canonical code & period, with FX-to-USD rollups, independent budget-source path. | statement rowsGET /statement/{entity_id}?source=actual|budget |
| get_mapping_review() | FastAPI | Pulls mapping_records for the entity's latest files, sorted ascending by confidence — most uncertain first. | review queueGET /mapping-review/{entity_id} |
| get_canonical_codes() | FastAPI | Aggregate view per canonical code — label_count, avg_confidence, pending_count. | code summaryGET /lineage/canonical-codes/{entity_id} |
| get_lineage() | FastAPI | Every source row across every file that ever fed a given canonical code, with confidence and confirmation status. | source traceGET /lineage/{entity}/{code} |
| get_drilldown() | FastAPI | Expands a canonical code + period to exact source cells, parsing formula references via regex. | source rows + formula refsGET /drilldown/{entity}/{code}/{period} |
| get_provenance() | FastAPI | Cross-entity event log — file uploads and other tracked actions, filterable by entity/event type, paginated. | event logGET /provenance |
| export_bva() | FastAPI | Builds a formatted .xlsx of the entity's BvA via openpyxl (Font, PatternFill, Alignment), ready for download. | .xlsx fileGET /bva/export/{entity_id} |
Neither agent call output is trusted unconditionally. The following are the code-level controls applied to each call — response validation, blast-radius limits, and the conditions under which a call is skipped entirely.
Every label is checked against entity_profiles.confirmed_mappings first. Only labels with no prior confirmed match are ever sent to Claude — the agent never re-decides something a human already settled.
Labels chunk into batches of MAP_BATCH_SIZE = 25. Each batch is its own try/except — one malformed batch response doesn't take down the others, and successful batches commit immediately rather than waiting on the whole set.
If a batch throws (bad JSON, timeout, anything), it's flagged and logged but the loop continues. Only after all batches finish does the phase raise — and by then, every successful batch is already committed to mapping_records.
canonical_code defaults to 'UNKNOWN' if absent, confidence defaults to 0.5. The response is never trusted to be complete — every field read with .get() and a fallback.
Claude returns a suggestion string, but the actual tier (1 or 2) and confirmed flag are derived by code from the numeric confidence — tier 1 if confidence ≥ 0.85. The agent proposes; a fixed threshold decides.
Mappings at confidence ≥ 0.85 are inserted with confirmed = true immediately — no analyst action required. Only tier 2 (<0.85) sits in the review queue. This is a real throughput/risk trade-off, not an oversight.
pg_try_advisory_lock(hash(file_id)) — if a duplicate map invocation arrives (e.g. a redelivered queue message) while one is already running, it's skipped outright rather than queued or double-run.
Every UPDATE display_rows is scoped to rows that are still unmapped and not block_excluded — an agent response can never silently overwrite an already-mapped or deliberately-excluded row.
Auto-mapped-from-memory inserts use ON CONFLICT DO NOTHING — safe to re-run the same phase twice without duplicate rows, which matters because queue messages can be redelivered.
JSON.parse, a hard-coded skip-pattern prefilter applied before any sheet reaches Claude, and per-sheet incremental persistence so a failure mid-file does not discard earlier results. No batching, confidence-based auto-confirm, or advisory lock is applied, as each sheet's classification is independent and idempotent by design.
Each item below corresponds to a conditional already present in the codebase.
COALESCE(sheet_doc_type, file doc_type) so each sheet is treated by its own classification, not the file's overall label./recompute-bva/{entity_id} deletes and re-inserts that entity's variance_records from scratch — no partial/incremental drift.already_mapped, already_normalised) — redelivery doesn't reprocess or duplicate.max_retries: 3 before giving up. No dead-letter queue is configured, so a message that exhausts its retries is dropped with no automatic recovery path — the job is left at whatever pipeline_jobs.status its last successful phase reached.Each item below addresses a specific gap in the current implementation. None of the following is built; scope and priority require confirmation before development.
system_prompt as free text, and a returned canonical_code is never checked against anything — it is trusted as written, with only 'UNKNOWN' as a fallback. Replacing the free-text paragraph with a typed, versioned schema — a stable backbone of standard codes plus an extensible leaf layer for sector-specific sub-codes, structurally informed by IFRS/GAAP concepts without claiming statutory compliance — would let the harness validate a returned code against a closed vocabulary, compute rollups (e.g. Total OPEX) deterministically from a rollup_parent field instead of trusting the agent's aggregation, and stamp every mapping_record with the schema version active at mapping time so a future taxonomy revision cannot silently reinterpret historical BvA output.
rollup_parent) and per-term metadata. It is not a formal ontology: there are no logical axioms, no class/individual distinction, and no reasoner — validation is procedural (code-side checks), not inferential. A true ontology would require formalizing rollup_parent as an RDF/OWL relation, adding axioms (e.g. every REV.* code has normal_balance = credit as an enforced constraint), and introducing a reasoner or SHACL/SKOS validator — deliberately out of scope here, as the mapping problem is classification into a known structure, not inference over asserted facts.calibrax-pipeline is configured with max_retries: 3 and no dead_letter_queue binding. A message that fails three times — a transient Worker cold-start, a momentary Claude API timeout — is dropped silently rather than landing somewhere recoverable, leaving the job stalled at its last completed phase with no automated path back. Adding a dead-letter queue and a periodic sweep over pipeline_jobs rows stuck mid-phase would close this without changing any pipeline logic.Same three lanes as the current-state diagram earlier in this document. Solid boxes are unchanged. Dashed orange boxes are new. Dashed blue boxes are existing steps whose behavior changes.
Each card maps to one or more of the new/changed nodes above.
This is the single largest fidelity gain — it converts an unbounded, free-text output space into a closed, checkable one.
Removes an entire class of error from the agent's responsibility — aggregation becomes arithmetic, not inference.
Closes the gap directly raised in the earlier pipeline comparison — this is the validation step that was missing.
Reliability, not accuracy — but unrecovered failures are themselves a fidelity problem: a label that silently never gets mapped is indistinguishable from one mapped wrong.
Pinned versions and resource names, read directly from the repository — not approximated.