What is EBCDIC, and why is it still here?

Open one of the RRC's EBCDIC files in a text editor and you get a screen of this:

ÖÁÊÊÅÙ@@@@@@@ðððððððððÁÂÃÄÅ@@@@@@ñòóôõö

Nothing is wrong with the file. You are looking at perfectly good data in a character encoding that predates ASCII's dominance and never went away.

Where it comes from

EBCDIC — Extended Binary Coded Decimal Interchange Code — is IBM's character encoding, introduced with the System/360 in the 1960s. It descends from punched card encodings rather than from teleprinter codes, which is why its layout looks arbitrary to anyone raised on ASCII: the letters are not contiguous, and the gaps are where the card columns used to be.

ASCII won everywhere except on IBM mainframes, where EBCDIC remains native. Texas regulatory data has been processed on mainframes continuously since the 1970s. The RRC has been migrating datasets to ASCII for years — which is why several datasets are published twice, once in each encoding — but a dozen are still EBCDIC-only, including several production ledgers, the statewide oil and gas well databases, the well status files, the P-4 certificate file and the statewide field data.

If both an ASCII and an EBCDIC version of a dataset exist, take the ASCII one. They contain the same records.

Code page 037

EBCDIC is not one encoding. It is a family of code pages that agree on letters and digits and disagree about punctuation. Code page 037 — CP037, sometimes IBM-037 — is the US/Canada page, and it is what US mainframe data including the RRC's uses.

The structural logic is easier to see in hexadecimal than in a table. Every character's high nibble is its zone and the low nibble is its digit:

Three things fall out of that table immediately.

The alphabet has holes. a through i are contiguous, then there is a jump to j. A through I, jump, J through R, jump, S through Z. Any code that assumes letter - 'A' gives an alphabet index is broken on EBCDIC.

Digits sort above letters. In ASCII, '9' < 'A'. In EBCDIC, 'Z' < '0'. Sort order is not preserved by conversion, so a file that was sorted on the mainframe is not sorted after you convert it, and vice versa.

Space is 0x40, not 0x20. Fixed-width records are padded with 0x40. If you see a file full of @ characters when you open it, you are looking at EBCDIC spaces decoded as Latin-1 — 0x40 is @ in ASCII.

That last observation is the fastest way to identify an EBCDIC file by eye: runs of @ where blanks should be, capital letters with accents where digits should be (0xF0–0xF9 are ð–ù in Latin-1), and no line breaks anywhere.

No line endings

Mainframe files are records, not lines. A dataset of 350-byte records is 350 bytes, then the next 350 bytes, with nothing in between. There is no 0x0A, no 0x0D, and no end-of-record marker at all — the record length is metadata that lives in the layout document, not in the file.

This is why "open it in a text editor" fails twice over: wrong encoding, and the whole file is one enormous line.

It also means a conversion has to know the record length. Converting byte-for-byte gives you correct characters in one unbroken stream; you have to slice it yourself:

RECORD_LENGTH = 350

with open(path, "rb") as f:
    while True:
        chunk = f.read(RECORD_LENGTH)
        if len(chunk) < RECORD_LENGTH:
            break
        record = chunk.decode("cp037")
        ...

Some RRC EBCDIC files do carry record delimiters, because they were produced by a process that added them. Check before you assume: read the first few kilobytes and look for 0x25, the EBCDIC line feed, or 0x15, its newline. If neither appears, the file is unlined and you need the record length from the layout manual.

Converting it

Python ships a CP037 codec, so the simple case is one call:

text = raw_bytes.decode("cp037")

For bulk conversion of hundreds of megabytes, a 256-byte translation table is faster, because bytes.translate runs in C over the whole buffer at once:

# EBCDIC_TO_ASCII is a 256-byte table indexed by EBCDIC byte value
def ebcdic_to_ascii_bytes(data: bytes) -> bytes:
    return data.translate(EBCDIC_TO_ASCII)

Both approaches appear in the upstream pipeline: the streaming P-5 loader decodes with cp037 directly, while the bulk converter uses a translation table and writes Latin-1 output. They agree on every byte that matters.

The conversion is lossless in the direction that matters. Every EBCDIC byte maps to exactly one output byte, so record positions are preserved — byte 47 of the EBCDIC record is byte 47 of the converted record. That is essential, because everything downstream is positional.

The failure modes

Wrong code page. CP037 is not CP500, CP1047 or any of the national variants. They agree on letters and digits and disagree on punctuation — square brackets, the exclamation mark, the caret, the not-sign. Decode US data with a European page and the names and numbers all look right while a handful of punctuation characters are quietly wrong. This is the worst kind of bug because nothing looks broken.

Decoding as UTF-8. Most EBCDIC bytes are not valid UTF-8, so a UTF-8 read either throws or, with errors="replace", fills your data with U+FFFD. Once replacement characters are in, the original bytes are gone.

Assuming the file is text. Opening in text mode with the platform default encoding, letting an editor "fix" the line endings, or running it through a tool that strips high bytes — all destroy the file. Always open EBCDIC in binary mode.

Losing the record alignment. If you strip bytes, normalise whitespace, or convert line endings anywhere in the pipeline, every fixed-width field after the edit shifts. Positional parsing then produces plausible-looking garbage: numbers that are off by a digit, names that start one character late.

Signed numbers survive the trip

One thing you will notice in converted EBCDIC — and in the ASCII files too — is numeric fields ending in a letter or a brace: 0321234567C, 00000123}. That is not corruption. It is a COBOL zoned decimal sign, and it is a direct consequence of the EBCDIC zone-nibble structure described above. The next article explains it in full, and the explanation is genuinely satisfying once you have the table on this page in front of you.

What EZRRC does

EZRRC does not convert EBCDIC. The upstream pipeline does that, once, and loads the result into the warehouse; EZRRC serves the decoded rows and, separately, the pristine original file exactly as the RRC published it. If you want to check the conversion yourself, download the original and compare — that is what it is there for.

Next: fixed-width COBOL record layouts, implied decimals, and the zoned-decimal sign encoding.