What's actually inside a copybook

In the first post I described the ritual: a person, a copybook, and a ruler, counting bytes by eye into a config file. I want to spend this post on why that count is so easy to get wrong — because the reasons are the whole reason PICASSO has to exist. If a copybook really were one character per byte, you wouldn't need a parser. You'd need a text editor and some patience.

It isn't. Let me take a real one apart.

The shape of the thing

A copybook is a small schema written in COBOL's data-definition sub-language. Here's a genuine one — DTAR020, the output of a 1990s reporting system, dated 19/12/90, which PICASSO carries as a sample because it's real and unforgiving:

*   RECORD LENGTH IS 27.
    03  DTAR020-KCODE-STORE-KEY.
        05 DTAR020-KEYCODE-NO   PIC X(08).
        05 DTAR020-STORE-NO     PIC S9(03)   COMP-3.
    03  DTAR020-DATE            PIC S9(07)   COMP-3.
    03  DTAR020-DEPT-NO         PIC S9(03)   COMP-3.
    03  DTAR020-QTY-SOLD        PIC S9(9)    COMP-3.
    03  DTAR020-SALE-PRICE      PIC S9(9)V99 COMP-3.

Two kinds of line. Some entries have subordinates and no PIC clause of their own — DTAR020-KCODE-STORE-KEY is one; it's a group item, a heading that owns the fields indented under it and takes up exactly as many bytes as they do together. Everything with a PIC is an elementary item, an actual field holding actual data. The two-digit numbers at the front — 03, 05 — are level numbers: they encode the nesting purely by magnitude. A higher number is subordinate to the nearest lower one before it, which is how 05s belong to an 03 without a single bracket. So far, countable.

Then you hit the PIC clauses, and the counting quietly falls apart.

PIC: the part everyone thinks they understand

A PIC (picture) clause describes a field's contents one symbol at a time. The vocabulary is small:

• 9 — a decimal digit position.
• X — any character.
• (n) — repeat the previous symbol n times, so 9(7) is seven digits and X(08) is eight characters. Purely shorthand: 9(3) and 999 are the same field.
• V — an implied decimal point. It marks where the decimal sits for arithmetic but occupies no byte of its own. S9(9)V99 is eleven digits with the point understood two places from the right; there is no . character stored anywhere.
• S — the field is signed. On its own it also takes no byte; the sign has to live somewhere, and where it lives is a rabbit hole I'll come back to.

Already there are two ways to miscount before you've even reached the interesting part. The V is invisible — forget it's not a byte and every offset after the field is off by one. The S is worse, because whether it costs a byte depends on a different clause somewhere else on the line.

DISPLAY: the one honest case

By default a numeric or text field is stored as DISPLAY: one byte per digit or character, human-readable, the digit 7 literally stored as the character 7. This is the mental model everyone brings, and it's correct — for DISPLAY. PIC X(08) is eight bytes. PIC 9(5) is five. If every field were DISPLAY, the ruler would work and I'd have nothing to write about.

The trouble is that storing numbers as text is wasteful, and mainframes have counted bytes since bytes were expensive. So the interesting fields aren't DISPLAY at all.

COMP-3: nine digits in five bytes

Look back at the copybook. Every numeric field is tagged COMP-3 — packed decimal, the workhorse numeric format of mainframe data. Packed decimal stores two decimal digits in each byte, one per four-bit nibble, and spends the final nibble on the sign (hex C for positive, D for negative). So the width isn't the digit count — it's the digit count plus one for the sign, divided into pairs and rounded up: ceil((digits + 1) / 2) bytes.

Run that on DTAR020-QTY-SOLD PIC S9(9): nine digits, plus a sign nibble, is ten nibbles, is five bytes. Not nine. A field a character-counter writes down as nine is a little over half that on disk.

Now do the whole record the naive way and the honest way, side by side:

field              PIC             counted   actual   offset
KEYCODE-NO         X(08)  DISPLAY     8       8      0
STORE-NO           S9(03) COMP-3      3       2      8
DATE               S9(07) COMP-3      7       4     10
DEPT-NO            S9(03) COMP-3      3       2     14
QTY-SOLD           S9(9)  COMP-3      9       5     16
SALE-PRICE         S9(9)V99 COMP-3   11       6     21
                                    ----     ----
                                     41       27

Forty-one versus twenty-seven. A human eyeballing this file lays out a 41-byte record, reads the very first COMP-3 field two bytes too wide, and shifts every field after it. The extract still parses — it just returns numbers that are quietly, plausibly wrong. That is the failure mode this entire project is aimed at, and it's baked into the most common numeric format on the platform.

Binary COMP: same trap, different arithmetic

Packed decimal isn't the only compact form. Plain COMP (also written COMP-4, COMP-5, or BINARY) stores a number as a true binary integer, big-endian, two's-complement — and its width is fixed in steps by the digit count, not by counting characters: 1–4 digits fit in a 2-byte halfword, 5–9 in a 4-byte fullword, 10–18 in an 8-byte doubleword. So PIC S9(4) COMP and PIC S9(1) COMP are both two bytes; PIC 9(9) COMP is four. A different rule from COMP-3, the same lesson: the byte width is a function you have to know, and it has nothing to do with the number of 9s you can see.

(These bytes are raw binary — they are never run through a character table, which matters the moment EBCDIC enters the room.)

EBCDIC: even the text isn't the text

The PIC X(08) field at the front is DISPLAY text — eight bytes, honest width. But on a real mainframe extract, those eight bytes aren't ASCII. They're EBCDIC, IBM's pre-ASCII character encoding, where the letter A is 0xC1 and the digit 1 is 0xF1 — nothing like the ASCII you'd assume. DTAR020's key field holds the value "69684558". Read those exact bytes as Latin-1, the way any modern tool does by default, and you get öùöøôõõø. Right length, correct offset, complete garbage value.

So even the one field whose width is honest still needs decoding through the right code page (cp037, in DTAR020's case) — and, crucially, only the text fields. Run the whole record through a code-page translation and you'll corrupt every COMP-3 field, because packed-decimal nibbles aren't characters and don't survive being reinterpreted as any. The encoding has to be applied per field or not at all. That's a distinction a ruler can't even represent.

Signs: the digit that's secretly also a sign

Back to that S. A signed DISPLAY number has to keep its sign somewhere, and COBOL gives you options that occupy different amounts of space:

• SIGN IS SEPARATE — the sign gets its own byte, a literal + or - before or after the digits. PIC S9(7) SIGN IS TRAILING SEPARATE is eight bytes, not seven. (This is the exact clause that trips up CATALOG-74's hand-transcription — a story for Part 4.)
• Overpunch (zoned, the default when you don't say SEPARATE) — no extra byte at all. The sign is folded into the zone nibble of one of the digits, usually the last. The digit still reads as a digit to anything that ignores the top nibble, but a positive 3 is stored as the byte for C and a negative 3 as the byte for L. The value 123 negative ends in the character L; positive, it can end in C — or in the plain digit itself.

An overpunched sign is genuinely invisible to a character count: the field is exactly as wide as its digits, but the last character isn't the digit it looks like. Miss it and you don't shift an offset — you decode a value with the wrong sign, or read a letter where you expected a number. Nothing about the width tells you it's there; only the S and the absence of SEPARATE do.

OCCURS and REDEFINES: structure that multiplies and overlaps

Two more clauses bend the layout in ways a flat reading can't follow. OCCURS repeats an item: 05 LINE-ITEM OCCURS 3 TIMES over a 20-byte group is 60 bytes, three copies laid end to end, and if the count itself is stored in another field (OCCURS DEPENDING ON) the record's length isn't even fixed — it's data you have to read first. REDEFINES is the opposite: it lays a second interpretation over the same bytes, two field definitions describing one region of storage, so the naive "add up every field" arithmetic double-counts and walks off the end of the record.

Both are cases where the offset of a field depends on more than the fields before it — on a repeat count, on an overlay, sometimes on a value inside the record. No amount of careful counting gets there, because the thing you'd need to count isn't a length; it's a rule.

Why the offsets are computable anyway

Here's the payoff, and the reason a parser is the right answer rather than a more careful human. Every single one of these rules is deterministic. Packed-decimal width is a formula. Binary width is a lookup by digit count. The implied decimal is a position, not a byte. EBCDIC is a fixed table. The sign lives in a place the clauses tell you exactly. OCCURS multiplies; REDEFINES overlays. None of it is ambiguous — it's just spread across half a dozen interacting rules that a person applies by hand, in their head, field after field, and gets right almost every time. Almost.

A compiler doesn't count characters; it applies those rules mechanically and never tires on field forty. PICASSO does the same thing: it reads the copybook the way the compiler would and computes the layout — 27 bytes for DTAR020, packed fields and implied decimals and EBCDIC text all accounted for — instead of asking a human to hold six rules in their head and hope. The information was always fully specified. It just was never one byte per character, and pretending otherwise is where the bugs come from.

Which raises the obvious question for a tool whose entire job is these offsets: how do you know it got them right? That's the next two posts. First, what happens when you point it at a real file — and then how you prove the answer against something that isn't yourself.