Parsing fixed-width exports and log lines safely

Two kinds of text look trivial to convert and are not: an export where every field sits at a fixed position, and a log where every line follows a pattern. In both cases the conversion usually succeeds. A program reads the output without an error, and the damage is in the values: an identifier whose leading zeros vanished, a name that gained or lost spaces, a field cut one character early after a letter with a diacritic, a dash that turned into literal text, or a damaged line that someone patched into a record nobody wrote. This guide takes the two source formats and the two target formats in turn, using the specifications, and the final part is a test designed to go red.

Fixed-width: the layout lives outside the file

CSV has a documented format, and the log formats below have documentation too. A fixed-width file has no header that says where fields start. The only description is a column specification that comes with the file, either as two boundary positions or as a position plus a width, and it is the single source of truth. No public standard for this layout turned up in the search for this guide, so everything about it below is practice rather than a cited rule, and the first job is to read the spec literally.

Three questions decide whether the cut is right. First, does counting begin at one or at zero, and is the end position included? Position six through twenty is a different width under each convention, and an off-by-one shifts every later column by a character. Second, are the positions counted in characters or in bytes? The two agree only for plain ASCII. UTF-8 spends between one and four octets on a character (RFC 3629), so a byte-based spec applied to Cyrillic or accented text cuts through the middle of a letter, and a character-based spec applied to bytes does the opposite. When a spec says bytes and the data contains anything non-ASCII, there are two possible readings, and a converter that picks one silently is guessing. Third, what should happen to a line whose length differs from the layout?

Legacy exports raise a fourth question, the encoding. Python's codec documentation groups the old single-byte code pages into families, and one of them is the IBM EBCDIC family, with cp037 listed for English (Python codecs documentation). A file in such a code page has to be decoded first; cutting it as if it were ASCII produces garbage whose column widths still look plausible. When the encoding is unknown, ask the sender instead of trying codecs until the output looks readable.

Padding, trimming and numbers

A fixed-width field is padded to its width, usually with spaces on the right for text and zeros on the left for numbers. Trimming the padding is a decision, not a default. A name column almost always wants its right-hand spaces gone, but a code column such as 00042 loses information if the zeros are removed, and a field that is deliberately all spaces means "empty", which is different from a missing line. Apply trimming per column, only for columns the layout names, and keep numbers as text unless the spec describes a numeric type. A spreadsheet will happily turn 00042 into 42 the next time someone opens the CSV, so the safe CSV is one where the consumer is told which columns are text.

Two number conventions are worth recognising from mainframe-era exports. An implied decimal point means the digits 000750 with two decimal places stand for 7.50 although no point appears in the file. It can only be applied when the spec states how many decimals the column has; applying it to a column that does not use it divides real values by a hundred. A signed field may carry its sign inside the last digit, as a letter or a brace instead of a plain minus. Decoding that needs a table taken from the system that wrote the file, and a converter without the table should stop rather than guess. These two conventions come from long practice with such files and are not tied to one public document here.

Writing CSV that a parser reads back

RFC 4180 records what most implementations do. Each record sits on its own line, and the separator is the CRLF pair; the final record may leave it off. A header line, when present, is shaped like any record. All lines should carry the same count of fields, and a space inside a field belongs to the field. A field with an embedded comma, quotation mark or newline needs a pair of double quotes around it, and within that pair every double-quote character is typed twice. The remaining fields may be written bare, or every field may be wrapped.

Each of those rules is a place where a hand-built string goes wrong. A city name such as "Sofia, Bulgaria" shifts every later field by one if it is written bare. A note with a quoted word inside it ends the field early unless the quotes are doubled. A field holding a line break splits a record in two for any reader that reads line by line. Trailing spaces from the fixed-width cut stay in the field, which is correct, and they survive only if nobody downstream trims them. The standard says nothing about a byte order mark and names US-ASCII as the common character set, with others allowed through a charset parameter, so state the encoding to whoever receives the file rather than assuming it.

Log lines: one format per source

For logs, "split on spaces" fails the moment a field contains one. The combined access log of Apache is defined by a format string made of directives, and the Apache documentation walks through each field (Apache log files).

The nginx documentation lists a predefined combined format with the same fields in the same order: client address, user, local time, request line, status, body bytes, then the referer and agent headers (nginx log module). The shapes match, but the servers are separate programs and their handling of an absent value is documented separately: Apache writes a hyphen where information is unavailable, and nginx logs a hyphen when a variable is not found. A parser should treat the hyphen as "absent" for the fields whose definition allows it and keep it as text anywhere else.

Escaping matters as much as the field list. Apache escapes quotes and backslashes in logged strings with a backslash, and writes other special bytes as \xhh sequences (mod_log_config). nginx escapes the same way by default, as \xXX, and has a json setting for the escape parameter that produces JSON string escaping instead (nginx log module). A field read by finding the next space, or the next quote without looking at the backslash before it, is cut in the wrong place the first time an agent string contains a quote.

Syslog has a grammar too

The syslog protocol in RFC 5424 defines a header of priority, version, timestamp, host name, application name, process id and message id, followed by structured data and then the message. A lone hyphen is the defined "nil value" for the fields that allow it, and it is also what a message without structured data carries in that position. The priority is the facility multiplied by eight plus the severity, so the value 165 is facility 20 with severity 5, and the severities run from 0 for emergency to 7 for debug. The timestamp is a full date and time with a zone designator or offset, and can itself be the nil value. Structured data is a series of bracketed elements with named parameters whose values are quoted, and inside a value a quote, a backslash and a closing bracket must each be escaped with a backslash. The message may begin with a byte order mark that marks it as UTF-8. Older syslog layouts, which have no version number and a different timestamp, do not fit this grammar and must not be forced into it.

JSON Lines as the target

The JSON Lines description sets three requirements: UTF-8 encoding with no byte order mark, each line a valid JSON value, and a line terminator of the line-feed character, with the carriage-return pair also accepted because surrounding white space is ignored. Objects are the usual choice; an empty line counts as no value at all, which is why a converter should never emit one for a skipped input line. The conventional extension is .jsonl.

Records read one at a time, so a single bad line does not have to spoil the file. This is where a deliberate policy matters. When a line does not match the pattern, because a field is missing, a quote is unclosed or a status is not a number, the options are to guess a repair, to drop the line or to keep it unchanged. Guessing invents a record. Dropping loses one, and nobody will ever count the gap. Keeping the original line as a record with a raw key costs one line of output and keeps the evidence. The same reasoning applies to types: a field declared numeric by the format becomes a JSON number, every other field keeps its original text, and a timestamp is copied with its own zone offset rather than converted.

Checking the result

Do not trust a conversion because it ran. Build a small input that contains the awkward cases on purpose and compare the output with what you worked out by hand: a text field with a comma and a quote, a code with leading zeros, a line one character short, a name with an accented letter, a request line containing spaces, an agent string with an escaped quote, a hyphen in a field where the grammar allows absence and one where it does not, and a line from a different format. Then break the converter, for example by splitting on spaces, and confirm that the check fails. A test that passes on a broken converter measured nothing. Finally, count: the output must hold exactly one record for every input line that is not empty, the unparseable ones included as raw records.

When this does not apply

The limits of this guide: plain-text fixed-width files with a column specification, CSV as RFC 4180 describes it, and the Apache, nginx and RFC 5424 line shapes. Binary record layouts, packed decimal fields, spreadsheets with multiple sheets, CSV dialects that use semicolons or tabs, multi-line log events such as stack traces, and custom application formats need their own rules. The statements about fixed-width practice, implied decimals and signed digits come from experience with such files and not from a cited standard. Server versions differ, and a site may have changed its log format string, so always read the actual configuration. Nothing here was run against real exports; the examples are illustrations drawn from the specifications.

Where our skills fit

Our fixed-width-to-csv skill takes the text and a column specification and returns CSV. It cuts exactly at the stated positions, trims only the columns the specification names and keeps zeros and inner spaces. It applies an implied decimal only when told how many places, and it quotes fields that contain commas or quotes. It answers with a fixed refusal line, instead of a table, when a line does not fit, when a byte-based specification meets non-ASCII text, or when it meets an encoded sign. Our log-lines-to-jsonl skill reads access logs, syslog or key and value logs by the format's grammar and writes one JSON record per line. It turns hyphens into null only where the grammar defines them as absent, keeps timestamps as written, and keeps a non-matching line as a raw record. More write-ups from the same workshop bench sit under our building notes.