# Sibyla Project TODO — Development Plan v4.0 (FDR + Claude CLI)

Updated: 2026-07-29 (rev. 2 — reissued after the repository sync restored the
v3.2 backlog; incorporates the implemented Moloni/neutral-integration state and
the WhatsApp LID authorization plan)

This is the execution backlog for **Sibyla Development Plan v4.0**
(`Documents/2026-07-29 Sibyla Plano Desenvolvimento v4.0 (FDR + Claude CLI).md`).
It replaces the v3.2 backlog. The v4.0 center of gravity is the Fiscal Document
Registry (FDR) built by Luís Nascimento with Claude: its layer 1–4 table model
becomes the PostgreSQL schema, its corpus-validated rules become the deterministic
gates, and its skill — adapted to single-document contract mode — runs on headless
Claude CLI invoked by the Sibyla Worker. The .NET application is the production
harness: channels, storage, queueing, identity, audit, review UI, and export.

Input channels are unchanged (email/WhatsApp/Mattermost via Hermes Gateway/Apolo,
plus web upload). Physical documents are archived in Nextcloud. Excel is an
on-demand export (download or Graph refresh chosen per export), never a
continuous projection. Bank movements (FDR layer 5), the management layer
(layer 6), and ERP integrations are Phase 2.

Status legend:

- `[x]` Done and still valid for v4.0
- `[~]` In progress in the current working tree
- `[ ]` Missing or not started
- `[!]` Risk or blocker to resolve before production use

## Current Snapshot — foundations carried over from v3.2

These remain the harness for v4.0. Items built over the superseded financial
schema are valid as foundations but are re-based onto the v4.0 schema in P1-1/P1-4.

- `[x]` .NET 10 solution (API, Web, Worker, Domain, Infrastructure, Tests) with IIS and Windows Service hosting and publishing scripts.
- `[x]` OIDC/JWT authentication, global identity roles, users/companies/departments/memberships, and server-side `UserChannel` resolution with deny-by-default channel identity.
- `[x]` PostgreSQL job system: `FOR UPDATE SKIP LOCKED` claims, leases, heartbeats, retry backoff, cancellation, dead-lettering, job lanes, and postcondition enforcement.
- `[x]` Append-only audit protection (application + PostgreSQL trigger), correlation IDs, structured logging, readiness and health checks.
- `[x]` Immutable original storage under `/originais/<year>/<id>.<ext>`, SHA-256, company-scoped exact-hash duplicate prevention, local and Nextcloud/WebDAV storage clients.
- `[x]` Validated multi-file web upload with extension, MIME, size, and content policies.
- `[x]` `sibyla.channel-intake.v1` contract, dedicated `sibyla-apolo` service identity (`sibyla.channel-intake.write` only), opaque intake/candidate IDs, idempotent registration/upload/status operations, and the Apolo addendum v1.1.
- `[x]` Text-first PDF helper (`Sibyla.PdfTextExtractor`) and bounded visual-evidence staging (≤4 page images, size/pixel caps) with byte-safe fallback.
- `[x]` Restricted-runtime security patterns and F0 spike evidence: restricted local account execution, prompt-injection test, concurrent extraction test, post-reboot acceptance (to be re-executed against the Claude CLI runtime in P1-2).
- `[x]` Graph spike evidence: delegated workbook session, table `add-row`, throttling handling, ambiguous-timeout reconciliation, schema fingerprint fail-closed (reused by the on-demand Graph export in P1-8).
- `[x]` Review foundations: review queue and document detail pages, header/line correction, corrections stored as immutable revisions with actor/timestamp/reason (re-based in P1-6).
- `[x]` Deterministic gate foundations: NIF check-digit validation, plausible-date and known-supplier gates, configurable line-sum tolerance, effective-confidence combination, 100% auto-integration threshold (re-based and extended in P1-4).
- `[x]` `docs/hermes-documental-lessons-learned.md` process: human review findings → lessons file → versioned instruction bump → golden-set validation → production. Applies unchanged to the Claude skill.
- `[x]` Provider-neutral integration core and Moloni adapter implemented (Cegid refactored to the `Integration*` ledger, migration `20260723190000_MoloniLedgerConcurrency`; OAuth, purchase push as drafts, sales pull, fail-closed reconciliation, rollback runbook — see `docs/moloni-integration.md`). Local fake-test evidence only; **parked for Phase 2** — sandbox/live acceptance, OAuth credentials, rate limits, and alerting pending. No new ERP work in Phase 1.

## P1-0 — Contract and Schema Design Freeze

- `[ ]` Write the `sibyla.claude-extraction.v1` contract spec: classification (docClass, documentType, originClass, failure category), parties as printed with fiscal numbers, fiscal header/line fields, basic non-fiscal metadata, `readingMode`, per-field confidence, warning codes (`TEXT_LAYER_ABSENT_VISION_USED`, `UNREADABLE_AFTER_VISION`, `INSUFFICIENT_DATA`, `WRONG_DOCUMENT_TYPE`, `NOT_A_DOCUMENT`), optional operator comment; strict validation rules before persistence.
- `[ ]` Design the discard/purge lifecycle for junk intake (email logos, embedded images, unrelated files): `NOT_A_DOCUMENT` → DOCFAI/review; reviewer **Discard** (soft — excluded everywhere, bytes and audit retained); authorized-role **Purge** (bytes permanently deleted, audit tombstone keeps metadata + SHA-256, never automatic, never for items that reached fiscal/archive tables); retained hash auto-discards byte-identical re-submissions with an Information flag.
- `[ ]` Map every FDR layer 1–4 table and field to the PostgreSQL schema: DOCLOG, DOCFAI, DOCTYPE, DOCEFL, DOCRQE (live view), ENTMST, ENTALS, ITMMST, ITMALS, ENTITM, ITMCLS, MNGAPL, SNCACC, ENTBNK (schema only; populated in Phase 2), EXCRAT, FDCHDR, FDCDTL, plus `DOCARC` for non-fiscal basic metadata and the export-run ledger.
- `[ ]` Design permanent-code issuance (EC/EI/LG/FA/DT/EX/SN/CL/MA/IC/AL/IL + EntryCode `{PREFIX}{YY}{MM}{NNNN}`) as database-backed sequences per bucket; a key collision raises, never silently picks a winner; codes are never reassigned.
- `[x]` Resolve the `DOCTYPE` name collision — decided with Luís (2026-07-29): the FDCHDR/FDCDTL I/R/P/F/O column is named `FlowType` everywhere (schema, contract, UI, Excel export); `DOCTYPE` exclusively names the classification-rules table/sheet. Luís applies the same rename in the original Invoice Skill Build files, so export parity is preserved.
- `[ ]` Fix the seeded DocClass/DocumentType taxonomy as data (DOCTYPE rows): Payables, Receivables, Bank (log+archive only), Legal (Employment Contract, Customer Contract, Supplier Contract, Bank Loan Agreement, Lease, Leasing/ALD, Insurance Policy, License/Permit, NDA, Power of Attorney), Procurement (Estimate/Quotation, Purchase Order, Product Data Sheet, Proposal, Order Confirmation), Other → review. Fiscal eligibility stays bound to explicit rules so non-fiscal types can never leak into payables.
- `[ ]` Fix the Nextcloud business-copy layout per class: fiscal keeps `Empresa/Ano/Mês/Fornecedor/...`; decide Legal sub-path (by counterparty vs by subtype); Period always derives from the document date, never processing date or mtime.
- `[ ]` Decide with Luís: import FDR history (DOCLOG/FDCHDR/FDCDTL, 826+ headers) as opening balances vs fresh start with reference data only.
- `[ ]` Decide how the parked `Integration*` ledger's foreign keys to `Document`/`ExtractionRevision` are preserved or re-pointed to the FDR tables — the P1-1 migration must not orphan the implemented Cegid/Moloni integration capability.
- `[ ]` Review and sign off the contract and schema-mapping docs with Luís before migration work starts.

## P1-1 — Schema Migration and Reference-Data Seed

- `[ ]` Add EF Core migrations for the layer 1–4 tables, `DOCARC`, code sequences, and the export ledger; retire the superseded financial schema in the same migration train, applying the P1-0 decision so the parked `Integration*` ledger keeps valid references throughout.
- `[ ]` Implement the validation service (port of `validate_registry.py`): unique DOCLOG filenames; no (CodeName, DocumentID) collisions on payables; header↔detail within 2 cents and no zero-line rows; every CodeName resolves to ENTMST with non-blank FiscalNo; every (Period, Currency) in use has an EXCRAT row; every DOCLOG row has a DocumentType; Flag↔FlagCategory both-or-neither; every (DocumentType, OriginClass) has a DOCTYPE rule and matches its Treatment; synthetic-FiscalNo count reported as warning.
- `[ ]` Import Luís's live reference data from `Invoice Skill Build/Editor/Data/*.json` with permanent codes preserved: ENTMST, ENTALS, ITMMST, ITMALS, ENTITM, ITMCLS, MNGAPL, SNCACC, EXCRAT, DOCTYPE rules, DOCEFL rules.
- `[ ]` Execute the P1-0 history decision (import FDR document history or start fresh); if importing, run the validation service over the imported set and flag prototype-era quirks rather than silently normalizing them.
- `[ ]` Acceptance: seeded database reproduces the FDR reference sheets exactly through the P1-8 renderer; validation service passes clean.

## P1-2 — Claude CLI Runtime

- `[ ]` Implement the Worker extraction handler that stages the transient job directory (text layer and/or bounded page images only) and invokes headless `claude -p` with the versioned Sibyla skill under the restricted local account; stdout is the only channel back; bounded execution time; fail-closed cleanup.
- `[ ]` Validate every response strictly against `sibyla.claude-extraction.v1` before persistence; malformed, incomplete, oversized, or inconsistent responses fail safely to review, never partially persist.
- `[ ]` Persist per-run metadata on the immutable revision: model ID, skill version, instruction version, reading mode, token usage, reported cost, duration.
- `[ ]` Keep extraction concurrency configurable on the existing Extraction lane; on Anthropic API outage jobs stay queued and lease-retry — no fallback extractor, no silent degradation.
- `[ ]` Manage the Claude API key in the approved secret store only; define the egress policy for the CLI host account; add cost caps and usage alerting from day one.
- `[ ]` Re-execute the full security acceptance battery against the Claude CLI runtime on the target host: restricted account privileges, prompt injection embedded in a malicious PDF/image, concurrent runs without state corruption, post-reboot acceptance without interactive login.
- `[!]` Resolve the carried-over restricted-token/service-identity design (formerly `[BLOCKED SECURITY]`), now covering both the PDF helper and the Claude CLI child process: ACLs granting only helper read/execute and transient-job-directory access, no network-capable SIDs beyond what the CLI requires, no application/database/secret access. Production is gated on this.
- `[ ]` Acceptance: security battery green; concurrency and reboot drills pass on the target host.

## P1-3 — Production Skill v1

- `[ ]` Derive the production skill from the FDR skill in single-document contract mode: no batch pipeline, no JSON table writes, no filesystem archive moves — reading, classification, and extraction only.
- `[ ]` Preserve the corpus-validated rules verbatim: text-first reading with mandatory visual fallback before any unreadable verdict; explicit-statement-only classification for payment/collection notices (never terms-and-conditions boilerplate); Invoice-Receipt iff payment is proven; Cancelled Invoice included with negated amounts; payout/settlement records excluded; entity identity by tax ID, never printed name; "a wrong-but-plausible value is worse than an honest blank".
- `[ ]` Extend classification to the full P1-0 DocClass taxonomy, including basic-metadata extraction (parties, dates, description) for Legal and Procurement classes.
- `[ ]` Distill `Specs/lessons-learned.md` corpus knowledge into skill instructions v1; the skill remains stateless — lessons enter only through versioned skill updates.
- `[ ]` Build and version the golden document set, seeded from the FDR corpus: textless-but-legible scans, collection notices, foreign-currency invoices, missing supplier NIF, document-date typos, brand-vs-legal traps, plus Legal/Procurement samples.
- `[ ]` Require golden-set validation before any model, skill, or instruction change reaches production.
- `[ ]` Acceptance: golden-set precision recorded per field, document type, and reading mode; no regression versus prototype behavior on the corpus samples.

## P1-4 — Deterministic Pipeline (Persistence, Routing, Gates, Flags)

- `[ ]` Persist validated extraction results transactionally: DOCLOG row (with DOCFAI routing for failures), classification, and — per DOCTYPE rules — FDCHDR/FDCDTL rows or a DOCARC record.
- `[ ]` Route `NOT_A_DOCUMENT` results to DOCFAI/review at Low/Medium severity; auto-discard byte-identical re-submissions of previously discarded/purged items via the retained SHA-256, flagged at Information level.
- `[ ]` Implement entity resolution in .NET against live reference tables at write time (no multi-pass convergence): ENTALS aliases first, FiscalNo-anchored ENTMST matching, billed-to company excluded from the counterparty candidate pool.
- `[ ]` Add the `counterparty_not_internal_company` gate with the regression test where the internal company's NIF appears in the customer/`Contribuinte` field; intercompany documents require human confirmation.
- `[ ]` Support provisional counterparty entities with a pending tax identifier (synthetic `{CODENAME}001`): always unverified, flagged Medium, excluded from automatic approval until the real NIF is recorded.
- `[ ]` Implement VAT-splitting consistency gates: per-line VAT authoritative when stated; blended rate splits proportionally to line net amounts; reverse-charge/exemption record zero with a self-assessment flag; stamp duty on its own detail line, reconciled in the total but flagged non-VAT.
- `[ ]` Implement the currency gate: persist currency; when it differs from EUR (or the destination account currency), require a matching EXCRAT entry for the period before auto-approval; missing rate → visible `NO RATE`, fail closed to review.
- `[ ]` Implement date rules: Period = YYYYMM of DateDoc; two-tier AccountPeriod (≤10th ⇒ previous month; evidence-based service-period override); `DateDue` = stated, else DateDoc for invoice-receipts, else DateDoc+30d with an assumption flag; `DatePay` only with documentary proof.
- `[ ]` Implement EntryCode issuance per P1-0 design at persistence time.
- `[ ]` Implement the DOCEFL flag engine as a deterministic service (the prototype's `add_flag_evaluation.py` never existed): case-insensitive regex rules, every rule evaluated, worst match wins, `Unclassified` (Medium) fallback; AutoReviewable only for Information/Low; Bot approval cleared when a row stops being auto-reviewable; flag writes idempotent (strip prior marker before append); human review never destroyed.
- `[ ]` Implement semantic duplicate detection on (canonical CodeName, normalized DocumentID) — filename similarity is never evidence in either direction; exact matches are strong signals regardless of issue date.
- `[ ]` Add the periodic Maintenance job that re-detects duplicates and re-resolves aliases across the full history (cross-batch duplicates are invisible to any single pass).
- `[ ]` Compute DOCRQE as a live severity-tiered view (High/Warning/Medium/Low) from flags and gate outcomes — never a capture-forward snapshot; default ordering by severity.
- `[ ]` Never overwrite a specialized DocumentType or any human-origin classification with automated defaults; recataloging, re-extraction, and fingerprint recalculation preserve human decisions.
- `[ ]` Acceptance: gate unit/integration tests green, including the misattribution regression; a second run over processed documents produces no unintended changes.

## P1-5 — Reference-Layer Management UI

- `[ ]` CRUD and review screens for ENTMST (roles, provisional/pending-NIF state), ENTALS/ITMALS aliases, ITMMST, ENTITM, ITMCLS/MNGAPL/SNCACC (read-mostly), and DOCTYPE/DOCEFL rule administration with audit on every change.
- `[ ]` EXCRAT management: monthly rate entry from real authoritative sources (ECB/BdP), inverted-quote handling, EUR→EUR identity rows, carry-forward for the running month displayed and flagged as such; estimated rates impossible to enter as confirmed.
- `[ ]` Surface auto-created items (`AutoCreated`, flagged) and provisional entities in a review list; classification is sticky once set — manual edits survive reprocessing.
- `[ ]` Acceptance: a correction made in the UI affects the next processed document immediately, without any batch rerun.

## P1-6 — Review and Document UI

- `[ ]` Severity-ordered review queue over the live DOCRQE view with filters by DocClass, source, company, and flag category.
- `[ ]` Document detail: staged evidence (text/page images), extracted fields with confidence, flags, gate outcomes, revision history, job history, and audit timeline.
- `[ ]` Approve, reject, retry, reprocess, and needs-attention operations; corrections stored as immutable revisions with actor, timestamp, and reason (re-based from v3.2).
- `[ ]` Implement **Discard** (any reviewer; soft removal from all views/exports/archive, bytes retained) and **Purge** (authorized role only; permanent byte deletion of original and copies with an audit tombstone retaining metadata + SHA-256) per the P1-0 lifecycle design; purge is the sole deletion path, is audited, and is blocked for any document that ever reached a fiscal or archive table.
- `[ ]` Populate `SupersedesRevisionId` on reprocessed revisions and correlate reprocess audits to the reprocess operation, not the intake correlation (carried over from the v3.2 live-proof findings).
- `[ ]` Attributed, audit-backed document comment log preserved from the current implementation.
- `[ ]` Acceptance test: no automated recataloging, re-extraction, or recalculation ever overwrites a human-origin classification or correction.

## P1-7 — Archive (Nextcloud)

- `[ ]` On approval, write the business copy to Nextcloud under the P1-0 class-based layout; CodeNames only in filenames; `{Company}_{DocumentType}_{Entity}_{Period}_{NN}.ext` convention for fiscal documents; Year/Month always from the document date.
- `[ ]` Confirm the business copy before the document transitions to Archived; the immutable original is never moved or mutated.
- `[ ]` Reprocessing never duplicates a business copy; a Worker kill during archive copying recovers to exactly one confirmed copy.
- `[ ]` Acceptance: exactly-one-copy proof under crash/retry conditions.

## P1-8 — Excel Export (On Demand)

- `[ ]` Implement the export renderer with FDR sheet parity for layers 1–4 (render order, formats, permanent codes, values only, no formulas, no totals rows): DOCRQE, DOCLOG, FDCHDR, FDCDTL, ENTMST, ENTALS, ITMMST, ITMALS, ENTITM, ITMCLS, MNGAPL, SNCACC, EXCRAT, DOCFAI, DOCTYPE, DOCEFL.
- `[ ]` UI export button offering both targets per export: downloadable .xlsx snapshot, or Graph refresh of the shared workbook (delegated auth via DPAPI-protected MSAL token cache, workbook session, schema-fingerprint check, replace-table-contents semantics, EntryCode as the idempotency column).
- `[ ]` Record every export in the export-run ledger (actor, timestamp, target, row counts, outcome); no continuous synchronization remains anywhere; nothing writes back from Excel.
- `[ ]` Acceptance: exported workbook byte-comparable to the renderer golden output; repeated Graph refresh is idempotent; distinct-entity-string check confirms no formatting divergence against the database.

## P1-9 — Channels End-to-End, Durability Minimum, and Pilot

### Channels (carried from v3.2 F2)

- `[ ]` Configure the existing Hermes `orquestrador` (Apolo) profile to call the Sibyla channel-intake API per the addendum.
- `[ ]` Verify email, WhatsApp/Baileys, and Mattermost documents reach Sibyla with trusted sender/message metadata; Mattermost uses `group_sessions_per_user` isolation.
- `[ ]` Complete WhatsApp LID + E.164 dual-alias authorization per `docs/whatsapp-lid-and-e164-authorization-plan.md`: strict `digits@lid` grammar (byte-exact, no trimming), separate allowlist rows per alias, TDD test set, and the guarded/evidence-backed allowlisting and rollback procedure; intake stays fail-closed on identity rejection with no fallback path.
- `[ ]` Verify both email intake forms end-to-end: attached documents, and direct-download links from authorized sources (configured domain allowlist for supplier platforms — e.g. Moloni invoice-email links) under the addendum's safe-downloader rules; route login-required links and unlisted domains to manual triage; sender confirmations without internal details.
- `[ ]` Back up `HERMES_HOME` (including the Baileys session); verify gateway auto-start and reboot session preservation.
- `[ ]` Admin UI for users, companies, departments, memberships, channels, and allowlists; membership changes effective without a new token; cross-company/forged-metadata/disabled-membership tests.

### Durability minimum for go-live (from v3.2 F3)

- `[ ]` Expired-lease job recovery; dead-letter alerting and operator replay controls.
- `[ ]` Periodic PostgreSQL↔Nextcloud reconciliation.
- `[ ]` Off-host backups (PostgreSQL, Nextcloud, `HERMES_HOME`, DPAPI material, configuration) and a restore drill into an empty environment.
- `[ ]` Worker-kill drills during extraction and archive copy.
- `[ ]` Metrics: documents/day, auto-approval rate, reading mode, model cost, job duration/state, reconciliation divergence.

### Pilot and go-live

- `[ ]` Run the Luís + Miguel pilot with real mixed-class documents across all channels.
- `[ ]` Manually spot-check a sample of auto-approved successes against originals before enabling automatic approval — a plausible aggregate match rate is not evidence of correctness.
- `[ ]` Second-run idempotency check over the pilot corpus.
- `[ ]` Quantitative acceptance criteria (accuracy, review rate, latency, reliability, cost) recorded; explicit business, security, and operational go-live approval.

## Phase 2 — Parked (do not start without re-planning)

- `[ ]` FDR Layer 5 — Bank Movements: BNKACC/BNKMOV/BNKCHK/BNKMAT/BNKREC, statement parsers (BPI/BCP/Revolut), balance-continuity checks, tiered matchers, reconciliation; ENTBNK activation. Prototype matcher scripts serve as specs.
- `[ ]` FDR Layer 6 — Management: PAYCTR/PAYDTL, RCVCTR/RCVDTL, ENTBLC balances, payment/receivables control.
- `[ ]` ERP integrations — acceptance and activation of the **already-implemented** neutral core + Moloni adapter (see `docs/moloni-integration.md`): Moloni sandbox/live acceptance with real OAuth credentials, rate-limit measurement, operational alerting (`[!]` a Worker outage longer than 14 days expires the Moloni refresh token — alerting is mandatory before production), controlled draft-invoice pilot, and re-binding the `Integration*` ledger to the FDR schema per the P1-0 decision. No rebuild — the code is parked, not deleted.
- `[ ]` Deep Legal/Procurement metadata and alerts (F7 remainder): `EffectiveDate`/`ExpiryDate`/`RenewalDate`/`NoticePeriodDays`/`AutoRenews`, expiry-alert job, full-text search decision, employment-contract counterparty modeling, avença→invoice linkage.
- `[ ]` `financial-agent` proposal engine: target-host acceptance and operator UI (implementation exists and is parked); financial Q&A read models and endpoints; calibration expansion beyond the P1-3 golden set.

## Superseded Work — Do Not Continue

From the previous architectures (v3.2 list still applies):

- Redis queues, outbox dispatch, Kanban sync, folder watchers, `.eml` drops, direct IMAP.
- Whole-workbook Graph download/edit/upload; application-permission Graph assumptions; direct user-triggered Excel commits.
- Permission grants/document routes as primary ACL; mutable review fields on a single intake record.

Newly superseded by v4.0:

- `hermes -p documental-agent` as the extraction engine and `sibyla.extraction.v3.1` as the production contract (retained only as reference; the security harness and staging pipeline are reused by P1-2).
- The continuous `ExcelCommit` workflow on the document critical path (Prepared/Writing/WriteUnknown ledger); replaced by the P1-8 export ledger.
- The v3.2 financial document schema where the FDR schema replaces it (P1-1 migration train).
- Any documentation presenting the above as the target production design.

## Definition of Done for v4.0 Work

- `[ ]` Behavior matches a v4.0 acceptance requirement and does not preserve superseded architecture without an explicit reason.
- `[ ]` Claude reads and proposes; only deterministic .NET code decides, persists, and executes side effects after validation and approval.
- `[ ]` Scoped API operations resolve and enforce current server-side memberships.
- `[ ]` Business-significant state changes and external calls are auditable and correlated.
- `[ ]` Jobs are idempotent and tested for process termination, timeout, retry, and duplicate delivery; a second run produces no unintended changes.
- `[ ]` Human-origin classifications and corrections are never overwritten by automation.
- `[ ]` A wrong-but-plausible value is worse than an honest blank: missing rates, identifiers, or evidence fail closed to review, never get estimated.
- `[ ]` Database changes use reviewed EF Core migrations and PostgreSQL-specific integration tests.
- `[ ]` Skill, instruction, or model changes pass golden-set validation before production.
- `[ ]` Configuration contains no passwords, tokens, or client secrets in source control.
- `[ ]` Documentation and runbooks are updated with every configuration, deployment, or workflow change.
- `[ ]` `dotnet build GOTT.Sibyla.slnx` passes with no warnings; `dotnet test tests\Sibyla.Tests\Sibyla.Tests.csproj` passes with no failures.
- `[ ]` The relevant phase acceptance proof is recorded before the item is marked complete.