Trust, but verify

Say I hand you a copybook parser and tell you it's correct. How would you check? You could read the code — but the bugs that matter here aren't the kind you spot by reading; two of the worst ones in this project's history (last post) read as perfectly reasonable code and still computed silent nonsense. You could run its own test suite — but a test suite written by the same person who wrote the parser encodes the same assumptions, so it agrees with the parser precisely where you'd most want an outside opinion.

The only verification worth anything is one that doesn't derive from the thing being verified. For a byte layout that turns out to be a high bar, because the obvious sources of truth are all somewhere downstream of the same understanding. So PICASSO is checked three ways, and the thing they have in common is that none of them is PICASSO.

One: diff it against a real compiler

The most authoritative statement of what a copybook means is what an actual COBOL compiler does with it. A compiler has computed these offsets, correctly, for decades — it is the reference implementation of the rules a parser is trying to reproduce. So the strongest check available is a differential oracle against one.

PICASSO carries a harness (tools/layout-oracle) that does exactly this against GnuCOBOL — cobc, a real, open-source COBOL compiler, configured for IBM/MVS sizing. For each copybook it generates a tiny program that includes the layout, compiles it, and asks the compiler for the authoritative LENGTH OF every field and of the whole record. Then it diffs those lengths against the offsets PICASSO computed independently. Two engines, no shared code, same input — do they agree?

Run across the broad external corpus PICASSO is exercised on — several hundred real copybooks pulled from open-source projects like AbsaOSS/cobrix, JRecord, and AWS CardDemo — PICASSO's record length matches the compiler on every copybook cobc can build, with exactly one systematic exception I'll come to. That's the check I trust most, because a compiler is not going to be politely wrong in the same way I am. And it's not a one-time audit: every new layout feature — REDEFINES, SYNC alignment, group-level USAGE inheritance, OCCURS DEPENDING ON — is gated on an oracle pass before it's allowed to merge. If the compiler disagrees, the feature isn't done.

Two: check values a third party published

A compiler agreeing on lengths is powerful, but it's still about layout. I also wanted proof that the actual decoded values come out right — and, again, not values I'd generated myself.

This is where DTAR020 earns its place a second time. It's a real extract (379 records, 10,233 bytes, EBCDIC, packed decimal, undelimited, headless — every hard property at once). Crucially, it's been used as a worked example by an unrelated tool, CobolToJson, whose README publishes what specific fields of specific records decode to. So PICASSO's test suite decodes the file and asserts those fields against CobolToJson's own published output — a number a different team printed, from a different codebase, before PICASSO existed. Then it re-encodes the whole file and checks it reproduces the original 10,233 bytes exactly. Independent values on the way in, byte-for-byte identity on the way out, including the negative packed-decimal values that are easiest to get subtly wrong.

Three: match a human who did it by hand

The third check closes the loop back to where the series started. CATALOG-74's specs.js is a byte layout a developer — me — worked out by hand and typed into a config file, back before the parser existed. It's the exact artefact PICASSO is meant to replace. So a parity test points PICASSO at CATALOG-74's own copybooks and asserts it reproduces those hand-typed offsets, byte-for-byte.

This one is a different kind of evidence from the other two. The compiler is more authoritative than a human and CobolToJson is an independent machine, but the hand-transcription is the real-world baseline the tool has to beat to justify existing: if PICASSO can't reproduce what a careful human already got right, it's not fit to replace them. It does reproduce it — with one telling exception, which is the most interesting result in the whole exercise.

The place PICASSO is right and the human wasn't

In Part 1 I promised this story. Here it is. BALANCE-REC has a field:

NET-BALANCE   PIC S9(7)V99  SIGN IS TRAILING SEPARATE.

That SIGN IS TRAILING SEPARATE reserves a whole extra byte, belonging to the same data item, to hold a literal sign character. It's one field, ten bytes wide, that decodes straight to a signed decimal. But my hand-transcription didn't model signed types at all, so I fudged it — I split it into two synthetic fields:

{ name: 'netBalance', start: 30, len: 9, type: 'int' },
{ name: 'sign',       start: 39, len: 1, type: 'str' },

...and then needed a special helper elsewhere in the code to glue the digits and the sign back into an actual signed number at read time. It works. It's also a workaround for a limitation in my hand-rolled codec, leaking into what's supposed to be a faithful description of the record.

PICASSO models it the way COBOL actually defines it: one NET-BALANCE field spanning all ten bytes, decoding directly to a signed decimal. No synthetic sign field, no glue helper. And the parity test doesn't quietly tolerate the difference as a near-miss — it asserts the merged single field on purpose, as the parser getting right exactly what the human cut a corner on. When your automated tool disagrees with the hand-made baseline, the honest move is to figure out who's correct and encode the answer, not to loosen the test until they match. Here, the tool was correct.

The discrepancy kept on purpose

Which brings me to the single systematic disagreement with the compiler, because the right way to handle a known difference is to document it loudly, not paper over it. It's about COMP-5.

PICASSO sizes COMP-5 the IBM way — the same 2/4/8-byte widths as COMP-4, fixed by digit count, widening only the value range and not the storage. That's what mainframe data actually uses, and it's the target PICASSO is built for. Native GnuCOBOL, by contrast, sizes COMP-5 byte-granularly, so a small item can come out a byte narrower. When the oracle run first flagged this, it would have been easy to "fix" PICASSO to match the compiler and make the diff go green. That would have been wrong — it would have made PICASSO agree with GnuCOBOL's native mode and disagree with the mainframe extracts it's supposed to read. So the divergence stays, deliberately, and it's written down in the code, the README, and the oracle's own notes as a dialect assumption rather than a bug. A green diff is not the goal. A correct layout is, and sometimes those point in opposite directions — the job is to know which is which and say so.

Why three, and why disjoint

Any one of these checks has a blind spot. The compiler is authoritative on layout but I only run it where cobc can build the copybook. The published values are independent but cover the fields one tool happened to print. The hand-transcription is the real target but a human is fallible — as NET-BALANCE proves. The point isn't any single check; it's that their blind spots don't overlap. A layout error big enough to matter would have to survive a real compiler's arithmetic and a third party's decoded values and a human's hand-count, all at once, all agreeing on the same wrong answer. For a byte offset, that conjunction is about as close to proof as this kind of tool gets — and it's proof that never once consults PICASSO's own opinion of itself.

There's a fair question hanging over all of this, and I've been putting it off for four posts. I keep saying "PICASSO computes," "the parser models," "the code refuses" — but I didn't write that code. The next post is about who did, and what that was actually like.