COUNTERSIGN: an agent fleet that cannot sign or pay

1Problem

An invoice arrives from a domain one character away from the real vendor's and says the bank account has changed. The control that should stop it is a person in accounts payable reading a PDF for about four minutes. The project README cites the FBI IC3 2025 report for $3,046,598,558 in reported losses across 24,768 complaints, with a $123,005 median, and Ardent Partners (2025) for $12.88 to $19.83 per manually processed invoice.

The failure is not carelessness. The check is not reproducible: nobody has time to find the vendor's real domain, the bank-detail change gets confirmed over the channel the attacker already controls, and afterwards nothing records what was checked. I wanted a system that does the checking work consistently and leaves a trace, without being a system that can be talked into moving money.

2Approach

The pipeline has seven stages. Ingest parses the invoice deterministically with Nutrient DWS; the model only maps spans that already exist onto named fields, each field keeps the page and box it came from as fractions of the page, and a value it cannot anchor comes back absent rather than invented. Identity uses SerpApi to ask whether the counterparty is the same legal entity or a namesake, whether coverage is adverse, and whether the address is a business or a mail drop. Domain uses name.com to check 40 confusable variants across eight attack classes against the live registry. Risk fuses the evidence into a verdict where every claim must cite an evidence span; a verdict that cites nothing is rejected and retried. Generate produces an out-of-band bank verification document with Foxit PDF Services, Deliver prepares a Foxit eSign envelope in draft and stops, and Persist writes the vendor record and an append-only audit log to Xano.

The central claim is checkable in code. In fleet/capabilities.py, signature.execute and payment.release are declared as human-only capabilities and are absent from the set the agents hold. They are declared on purpose, so that an attempt resolves to a named capability and is denied on the record instead of being inexpressible and therefore unauditable. A tool that is not in the mapping resolves to no capability and is denied, so the gate fails closed. tests/test_boundary.py fails if anyone grants an agent the power to sign, and tests/test_adversarial.py tries to break the boundary.

Four of the seven agents call Gemini 3.5 Flash or Flash-Lite on Vertex AI. Three deliberately do not: the domain sentinel is deterministic so its signal survives an audit, the envelope preparer is plumbing, and the injection screener stays outside a model because asking a model whether a document is manipulating a model puts the judgement inside the blast radius. IBANs, BICs, tax identifiers and mailboxes are masked before any prompt is built, so no account identifier enters a model context. The capability ledger, permission gate, injection armor, review queue and evidence-span verification come from my earlier quanta-gradesync project; the README states this is a new domain on that base, not a fork.

3Architecture

The envelope preparer requests the signature on every run and is refused every time. The refusal is an audit row, for example seq=5 envelope-preparer foxit_execute_signature deny, rather than a silent omission.

4Results

Labelled benchmark

Measurement: six invoices from one template, two runs each, live providers, known official domain held fixed. This measures reproducibility and the signal logic. It is not a claim about accuracy on real mail. A fabricated source was also rejected, and a legitimately granted call was still allowed, which rules out a gate that passes by denying everything.

Soak test

Measurement: demo/soak, 20-invoice corpus, 40 runs over 2 passes, five concurrent. The corpus is 80% legitimate because fraud is rare and a balanced set would misstate the problem. The generation and delivery stages were closed in the soak, so latency excludes them. These are my own in-repo measurements, not an external evaluation.

False-positive trajectory

Measurement: the three figures are quoted from the README, which gives the trajectory but not the per-round corpus or run counts. Only the final 28.1% (9/32) is backed by the report file in the repository.

One live run

In the worked case name.com is the impersonated vendor and narne.com the fictional fraudulent sender, using an rn for m substitution. The run checked 24 names, found 8 confusables already registered, scored the invoice high at 1.00 on 3 signals from 17 evidence items, left Foxit envelope 35674834 in draft, wrote 10 audit rows to Xano, and was refused the signature. Nothing was mocked. I state registry facts only and make no claim about whoever holds that domain.

5What did not work

The false-positive rate is still 28.1%

After two rounds of fixes, 9 of 32 legitimate runs were flagged and 2 of 32 would have blocked a real payment. All five flagged legitimate cases are listed in the soak report, and the blocked one is a legitimate sibling TLD. A domain registry can say a variant is taken, not whether name.net belongs to the same company as name.com. I left this named rather than tuning it away, and I would not deploy the control as it stands.

The soak found defects the benchmark did not

The benchmark passed 6/6 while two real bugs sat underneath. The vendor baseline was written from whatever invoice arrived first, so a supplier whose first invoice was fraudulent would have had the attacker's account enrolled as its baseline. And billing subdomains were compared to the official domain by string equality, so every legitimate invoice from invoices.name.com carried the vendor's own confusable surface as a finding against it. The README ties the drop from 80.7% to 28.1% to fixing defects like these; it does not break the drop down per defect.

Verdicts moved between runs

One case, a clean legitimate name.com invoice, returned review on one pass (score 0.35) and clear on the next (score 0) with the same single signal. That is the 19/20 stability figure. Four times during the build, a fact a rule could settle had been given to a model and came out differently between runs, or not at all: the sender domain, the bank-detail change, the IBAN, and the sender-versus-official comparison. Settled signals now travel with the evidence and take precedence over the model's draft.

Providers and integrations

In the soak, 23 stage executions were degraded or failed, with 4 SerpApi faults and 1 Vertex fault recorded, and every run still produced a verdict. Doctavian is integrated but was never configured: its OAuth consent screen was in testing mode and rejected my Workspace account with 403 access_denied, so Foxit PDF Services covers generation. Foxit's envelope fetcher also could not download from every public host; raw.githubusercontent.com worked while w3.org and africau.edu failed with a generic download error.

Open evaluation limits

The benchmark is six invoices from one template, accuracy is measured given a known official domain, and the domain sweep reports registered or available, never who owns it. I do not know the hackathon result.

6Lessons

• Declare the forbidden capabilities and deny them on the record. An attempt that resolves to a named capability is auditable; one that cannot be expressed leaves no trace.
• Fail closed on anything undeclared, and test the gate in both directions: a granted call must still pass, or a deny-everything gate would score perfectly.
• A control is judged by how often it flags real work. The 8/8 detection figure means little next to a 28.1% false-positive rate, and a benchmark with two negatives did not show it.
• Do not give a model facts a rule can settle. Four separate cases pointed the same way, and the fix each time was to let the settled signal outrank the model's draft.
• Run a repeated soak on a realistic class balance early. It found the baseline-enrolment and subdomain defects that a single labelled pass missed.