Let the real files find the bugs

There's a particular kind of bug I've learned to be afraid of. Not the crash — the crash is honest. It stops, it points at a line, someone fixes it. The one to fear is the calculation that runs cleanly to completion and hands back a number that is simply wrong, with the same confident face it wears when it's right.

A copybook parser lives entirely in that danger zone. Its output is byte offsets, and a byte offset is either exactly correct or it silently poisons every field after it. Read one COMP-3 field as three bytes instead of two — the mistake from the last post — and the extract still decodes. You still get numbers. They're just shifted by one byte from there to the end of the record, and they look every bit as plausible as the real ones. Nobody sees a stack trace. They see a report where the totals are a little off, three weeks later, and no idea why.

The rule: fail loudly

So the governing principle of the whole project is stated as a prohibition: never silently miscompute. If PICASSO cannot lay a field out with certainty — an unsupported USAGE, a construct it doesn't model, a copybook shape it wasn't built for — it does not guess, does not skip the clause and hope, does not fall back to a best effort. It throws a named, specific error and stops. A refusal is recoverable: you read it, you understand it, you decide what to do. A silent wrong answer is not, because you never find out it happened.

That sounds obvious written down. It is emphatically not what the code did at first, and the gap between the two is instructive — because the silent-miscompute holes were exactly the ones that unit tests sailed straight past.

The bugs that passed every test

Early on, several constructs PICASSO didn't support weren't rejected — they were quietly ignored, treated like the genuinely harmless clauses (VALUE, JUSTIFIED) that carry no storage and can safely be skipped. The trouble is that these ones do carry storage, so skipping them didn't produce an error. It produced a layout that was wrong and said nothing:

• OCCURS and REDEFINES both used to fall through to the skip-a-token path. A repeating field or an overlapping one would parse "successfully" — with silently wrong offsets for every field after it. They're now named rejections; support came later, deliberately, one at a time.
• An OCCURS length that overflowed. The arithmetic that multiplies out a repeat count could overflow an integer and wrap to a small positive number — a pathologically large table computing a plausible tiny length. It's now a checked overflow that fails loud rather than wrapping.
• An elementary item with subordinates. A field that had both a PIC of its own and children under it used to silently drop the children and shift every following offset. Now it's a named error.
• A mistyped COMPUTATIONAL. COMPUTATIONAL3 with a dropped hyphen, or a COMP-7 that doesn't exist, used to be silently skipped — leaving the field mis-sized as plain DISPLAY. Now an unrecognized computational usage is rejected by name.

Every one of these had passing unit tests around the code near it. None of the tests caught the hole, because a unit test checks the case you thought of, and by definition you didn't think of the case where your own code quietly does the wrong thing. What caught them was pointing the parser at input that didn't care what I'd thought of.

Enter a real 1990s mainframe file

PICASSO's test suite has plenty of synthetic copybooks — small ones I wrote to exercise a specific rule. They're useful and they're clean, and being clean is their weakness: a synthetic copybook only ever contains the problems its author already knew to put in it. So I went and got one I hadn't written.

DTAR020 is a genuine COBOL copybook from an actual reporting system, dated 19/12/90, credited in its own header to "Bruce Arthur," reused as a teaching example across open-source COBOL tooling for years. It comes with a real 379-record, 10,233-byte binary extract — the actual bytes, straight off the system. I bundled both into PICASSO unmodified: sequence numbers intact, header comments intact, nothing cleaned up to suit the parser. That decision is the entire point. The moment you tidy a file to make it parse, it stops being a test of reality and becomes a test of your tidying.

Running the untouched file surfaced four separate gaps, none of which any synthetic copybook had, because synthetic copybooks don't have these problems by construction:

1. The source was fixed-format

Every copybook I'd written was free-format — code starting at the left margin. Real COBOL source from the era is fixed-format: columns 1–6 are a line-sequence number, column 7 is a comment/continuation flag, and the code lives in columns 8–72. DTAR020's lines begin 000900, 001000, 001100. Fed those, the parser read the sequence numbers as level numbers and produced one garbage field — silently, of course. It now detects fixed-format per line, strips the sequence area, honours the column-7 comment flag, and ignores the identification area in columns 73–80. Both formats can even mix in one file.

2. The text was EBCDIC

The one PIC X field decoded to öùöøôõõø — the right eight bytes read through the wrong character set. The file is EBCDIC cp037, not ASCII, and PICASSO was Latin-1 only. The fix isn't just "add EBCDIC"; it's applying the code page per field, so the packed-decimal fields are left strictly alone. Translate the whole record and you'd corrupt every COMP-3 field in it — the same mistake as pulling a mainframe file over FTP in ASCII mode. (I gave EBCDIC and packed decimal a proper airing in Part 2.)

3. The records had no delimiters

10,233 bytes is exactly 379 × 27. There is nothing between the records — no newline, no marker, just a bare concatenation you slice by the layout's own width. The codec had been splitting on newlines, a Unix convention a mainframe file has never heard of, so it handed the layout all 10,233 bytes as one oversized record. Worse, this file carries 21 0x0D bytes inside its COMP-3 fields (0x0D is the digit 0 with a negative sign nibble) — so any newline-normalisation would have eaten real data and shifted every record after it. Undelimited slicing is now a supported mode. It's deliberately not auto-detected, because a delimited file's length can divide evenly by the record length too, and guessing would return a plausible count of misaligned records — the silent wrong answer again.

4. It was headless

DTAR020 opens at level 03. There is no 01 wrapping it, because it's a copy member — designed to be pulled into a record the including program names, which PICASSO never sees. A common, legitimate shape. The parser now supplies the 01 the compiler would have inherited and flags that it did so, rather than choking on the missing top level. (Two 01 records in one copybook is still refused — those are alternative layouts, not one record, and merging them would describe neither.)

Why this is the method, not an anecdote

Four gaps, one file, none of them things I'd have invented a test for — because each was a way the real world is shaped that my imagination had smoothed over. That's not luck. It's the predictable result of a rule I now hold to deliberately: synthetic tests confirm what you already understand; real files tell you what you don't. The synthetic suite is still there and still valuable — it pins down the rules I do know, and it runs in a second. But it is structurally incapable of surprising me, and the bugs that matter for a data tool are the surprising ones.

The two disciplines reinforce each other. Fail-loudly means that when a real file hits something unhandled, I get a clean, named refusal pointing straight at the construct — not a mystery number I have to reverse-engineer weeks later. So every real file I run is either a clean pass or a precise signpost to the next gap. That loop — run something real, read the loud failure, close the gap, run it again — is how DTAR020 went from four rejections to a file that now round-trips all 379 records byte-for-byte through nothing but the public API. It's also how the broader corpus of a few hundred real copybooks got worked through, one honest failure at a time.

None of which, on its own, proves the offsets are actually right. A parser can fail loudly and still confidently compute the wrong layout for something it thinks it understands. Round-tripping a file to the same bytes it came in as is a strong signal, but it's still the tool checking its own work. To really trust the numbers I needed a source of truth that isn't PICASSO — three of them, in fact. That's the next post.