BUILDING Parts of this page describe what is not built yet, and each says so.

Verify your audit log

Your audit log is a chain: each entry carries the hash of the one before it. You can check, on your own machine and without trusting us, that your copy is consistent with itself: every hash recomputes and every entry names the one before it. A later copy checked against an earlier copy’s head hash also shows that nothing before that head was rewritten. What a clean check does not show is at the end of this page.

Three ways to get a copy

1. From the console. On the Audit log screen, choose Export. The file downloads in your browser, from the first entry.
2. From a terminal. rl audit export -out audit.ndjson writes the same file, from the first entry or from -from-seq, and checks as it reads that every entry names the one before it. It makes the file, and checks that it can, before it asks for the first page, so a destination it cannot write fails before the log is touched. The file is readable only by you, and nothing is left behind if the chain does not link. Change its mode yourself if you mean to share it, because it holds the whole unfiltered chain.
3. Delivered to a bucket in your own account. Delivering the log to your own bucket as it is written is not switched on yet, and neither are the signed checkpoints that need it. Check a signed checkpoint says what one shows.

The first two produce the same file format. Every page of a thousand entries you read or export writes one entry to the log, and both say so. Checking a file you already hold, on your own machine, writes nothing.

The file format

One JSON object per line. The first line is a manifest: its kind, the format’s version, the range the file covers (from_seq and through_seq), the head the chain had when you took the copy (head_seq and head_hash), how many entries it holds, and whether it reaches the head. Every other line is one entry: its sequence number, its time, the organisation and project it belongs to, who did it, what they did and to what, the document before and after, the request and address it came from, the hash of the entry before it and its own hash. An entry written by a machine credential, a service principal’s secret or a token a CI job exchanged, also carries via_json: the door that credential came through, as a JSON object in a string. It is left out of an entry that has no door.

The manifest’s version is 1 for a file in which no entry has a door, written exactly as before, and 2 for a file in which one does. A checker that reads only version 1 keeps reading every file of the first kind and refuses a file of the second kind, rather than reporting it as altered.

The before and after documents are strings, as the server stores them: canonically equivalent to what was hashed, and not byte for byte the same. A check writes them in canonical form before it hashes them, which is why spacing, key order and number spelling inside them do not matter to it.

The chain rule

An entry’s hash is sha256(previous hash || canonical entry), where || joins the bytes. The hash before the first entry is thirty-two zero bytes. The canonical entry is built the same way every time:

1. One byte holding the version: 1 for an entry with no door, and 2 for an entry that has one.
2. One byte holding the number of fields: 13 for version 1 and 14 for version 2.
3. The thirteen fields, in this order, each as four bytes holding its length, big-endian, followed by its text as UTF-8: the sequence number in decimal digits, with no sign and no leading zero, the organisation id as a lower-case hyphenated UUID, the project id the same (empty when the entry names no project), the actor kind, the actor id, the action, the resource kind, the resource id, the before document, the after document, the request id, the address and the time. A version 2 entry has a fourteenth field after the time: its door, in canonical form (below), which is always a JSON object with at least one key.
4. The before and after documents, and the door, in canonical form, or empty when there is none, which means nothing but spaces, tabs, line feeds and carriage returns: object keys sorted by their UTF-8 bytes and no whitespace. A string is escaped only for the quote, the backslash, \b, \f, \n, \r and \t, and for any other control character below U+0020, which is written as \u00 and two lower-case hex digits; everything else, including <, >, & and characters outside ASCII, is written as it is. Numbers are written plainly: the exponent expanded, no leading plus or leading zeros, the zeros of a fraction kept, and the sign of a zero dropped.
5. The time in UTC, written exactly as 2026-10-01T12:00:00.100000Z: six fractional digits, with an upper-case T and Z. A time written any other way, with fewer digits when the last are zeros for example, is not accepted.

A hash is exactly 64 lower-case hex digits. The version of an entry is read off the entry itself, so taking the door off a version 2 entry, or putting one on a version 1 entry, changes the bytes that are hashed and is a hash_mismatch at that entry.

What the hash does not cover

Some of what is in the file is not read by the hash, so editing it passes both checkers:

• The project slug. The hash covers project_id, not the slug written beside it.
• The spacing, key order and number spelling inside before_json and after_json. Texts that are canonically equivalent hash the same.
• The manifest’s complete, page_size and max_pages.

The manifest’s records, through_seq, head_seq and head_hash are checked against the entries. What a check does catch is a change to the value of any field the hash covers: each is a hash_mismatch at that entry.

Both checkers also read every line strictly. A line that repeats a key, that has a key differing from one in the format only by case, or that has any key outside the format’s exact set is refused, with the line and the key named, because a reader that took the last duplicate or ignored case would read a different entry from one that did not. Text that is not UTF-8 is refused, and so is nesting deeper than 500 levels. A refusal is not a finding about the chain: it means nothing was checked.

Check it with rl

rl audit verify audit.ndjson recomputes every hash from the entry’s own fields and checks that each entry names the one before it, that sequence numbers run on with no gap (and no repeat inside a file), and that the file’s own count and end are what its manifest says. It stops at the first break and names its kind and the sequence number. Given several files, or a directory of them, it orders them by their manifests. A directory is searched all the way down, so a copy of files from your own storage is checked by naming the directory you copied. Expect them nested as audit/v1/<organisation>/<yyyy>/<mm>/<dd>/<first sequence>.ndjson, with the first sequence padded to twenty digits; below the top of a directory only files ending in .ndjson are read, and a checkpoints folder is skipped. An entry that two files both hold is checked once and accepted when the two copies are the same entry, but the same sequence number with a different hash is a hash_mismatch. A copy that starts after the first entry needs -resume-hash, the hash of the entry before its first; without it, rl says that nothing before the first entry was checked.

A clean result says consistent, never intact. Its first line reads consistent: N records (seq A through B), every hash recomputed and every link held. and its second says that this shows the copy is internally consistent, not who wrote it, and that a later copy checked against an earlier copy’s head hash is what shows nothing before it was rewritten. With -o json the answer is the key consistent, and the comparison with us is a separate key, server_agrees.

By default it does not ask us anything, and it says so. With -against-server it also reads our entry at your last sequence number and compares the hashes. That shows we still hold what you hold; it is the same server that wrote the chain answering about itself, so it does not show the chain was never rewritten before you took the copy. That read writes one entry to the log. Its exit codes:

There is no public download of rl yet, which is why there is a second way.

Check it without rl

The program below is a Python program of about seven hundred lines that uses only Python’s standard library, so you can read every line of it before you run it, and it shares no code with ours: it was written from the rule above. python3 verify.py audit.ndjson checks a file, and --resume-hash, directories and entries that two files both hold work as they do with rl. It prints the same two lines when the copy is consistent, and it exits 0 then, 1 on a break, 2 when you invoked it wrongly or a path cannot be opened, and 3 when it cannot read the file, so that nothing was checked. It never contacts us, so it has no comparison with the server to make.

#!/usr/bin/env python3
"""Check a RunningLake audit export with nothing but Python's standard library.

    python3 verify.py [--resume-hash HEX] FILE_OR_DIRECTORY...

The file is an export of the audit chain, as the console or `rl audit export` writes it:
newline-delimited JSON, a manifest line first and then one line per record.

This is a second implementation, written from the published encoding and sharing no code with
the server's. It recomputes every hash from the record's own fields, so a change to the value of
any hashed field is found without trusting the hash the record was exported beside.

SEVERAL FILES, A DIRECTORY, AND ENTRIES THAT APPEAR TWICE
  Files are read in the order of their manifests: first sequence ascending, and of two files that
  start together the longer first. A directory is searched all the way down: at its top every file is
  taken, below it only the files ending in .ndjson, so a copy synced from your own storage, which
  nests its files as audit/v1/<organisation>/<yyyy>/<mm>/<dd>/<first sequence>.ndjson, is checked by
  naming the directory you synced. A directory called checkpoints, and anything whose name starts
  with a dot, is skipped.
  Delivery is at-least-once, so two files can hold the same entries. An entry that repeats one
  already accepted, with the same sequence, organisation, previous hash and hash, is skipped (its hash
  is still recomputed). The same sequence with a different hash is a hash_mismatch: a fork is never
  forgiven. The last 65536 entries are remembered for this, so memory stays bounded; a repeat from
  further back, or a repeat inside one file, is out_of_order.

WHAT A CLEAN RESULT SHOWS
  The hash is not keyed. Anyone holding the file can edit a record and recompute that record's
  hash and every hash after it, and the result is still consistent. So "consistent" means the
  copy agrees with itself: every hash recomputes and every record names the one before it. It
  does not show who wrote the copy. What shows that nothing before a point was rewritten is a
  later copy checked against an earlier copy's head hash (the hash of its last record), which is
  what --resume-hash is for.

WHAT THE HASH DOES NOT COVER
  Editing any of these passes this verifier, because the hash does not read them:
    the `project` slug (the hash covers project_id, not the slug beside it);
    the spacing, key order and number spelling of before_json and after_json (texts that are
    canonically equivalent hash the same: the file carries the JSON as the server stores it,
    which is not byte-for-byte what was hashed);
    the manifest's complete, page_size and max_pages.
  The manifest's records, through_seq, head_seq and head_hash are checked against the records.
  A copy whose manifests name no head (a segment cut from the chain names none) can be consistent and still end
  before the chain does, so it says so on a line of its own: entries after it are not covered.

HOW A LINE IS READ
  Each line is read as an ordered list of keys and refused, with exit 3 and the line and key
  named, when it repeats a key, carries a key that differs from one of the format's only by
  case, or carries any key outside the format's exact set. Text must be UTF-8, and a line, or a
  before_json or after_json, nested more than 500 levels deep is refused.

THE EXPORT'S VERSION
  The manifest's version is 1 for a file in which no record carries a via_json, and 2 for a file in
  which one does. A version 1 file has exactly the keys it always had, so a verifier that reads only
  version 1 keeps reading every export of a chain no machine credential has written to, and refuses,
  rather than reports a break on, one that has. In a version 2 file a record that has a door carries
  the key via_json (its canonical JSON, as text) and a record that has none omits the key.

THE RULE
  hash = sha256(previous_hash || canonical_bytes(record))
  The hash before sequence 1 is thirty-two zero bytes.

CANONICAL BYTES OF A RECORD
  A record is written in one of two versions, and which one is read off the record itself: a
  record with a via_json (the door a machine credential came through) is version 2, and a record
  without one is version 1. A verifier that meets a version it does not know refuses; it does not
  report a break.
  one byte    the version, 1 (no via_json) or 2 (a via_json)
  one byte    the number of fields, 13 for version 1 and 14 for version 2
  then the fields in this order, each a 4-byte big-endian length followed by its bytes
  (UTF-8 text):
     1 seq             decimal digits, no sign and no leading zero
     2 org_id          lowercase hyphenated uuid
     3 project_id      the same, or empty when the record names no project
     4 actor_kind
     5 actor_id
     6 action
     7 resource_kind
     8 resource_id
     9 before          canonical JSON of before_json, or empty when it is empty
    10 after           canonical JSON of after_json, or empty when it is empty
    11 request_id
    12 ip
    13 at              the instant in UTC, exactly 2026-10-01T12:00:00.100000Z: six fractional
                       digits, upper-case T and Z
    14 via             version 2 only: the canonical JSON of via_json, which is always a JSON
                       object with at least one key

  A hash is exactly 64 lower-case hex digits. "Empty" for before_json and after_json is nothing
  but spaces, tabs, line feeds and carriage returns.

CANONICAL JSON
  object keys sorted by their UTF-8 bytes; no whitespace; strings escaped only for the quote,
  the backslash, \\b \\f \\n \\r \\t and the other control characters below U+0020 as \\u00XX in
  lower case (everything else, including < > & and non-ASCII, stays as it is); numbers are plain
  decimal with the exponent expanded, a leading plus and leading zeros removed, trailing zeros
  of a fraction kept, and the sign of a zero dropped.

Exit codes: 0 consistent, 1 a break (the kind and the sequence are printed), 2 a usage mistake
or a path that cannot be opened, 3 a file this verifier cannot read (so nothing was checked).
"""

import hashlib
import json
import os
import re
import struct
import sys

VERSION = 1
FIELD_COUNT = 13
VIA_VERSION = 2
VIA_FIELD_COUNT = 14
# The export file's own versions: 1, and 2 when a record carries a via_json.
EXPORT_VERSIONS = (1, 2)
EXPORT_KIND = "runninglake.audit.export"
GENESIS = b"\x00" * 32
MAX_NUMBER_DIGITS = 512
MAX_DEPTH = 500
INT64_MAX = 2 ** 63 - 1
# How many of the latest accepted entries are remembered so that a later file repeating them can be
# recognised. A repeat further back than this is out_of_order.
OVERLAP_WINDOW = 1 << 16
JSON_SPACE = " \t\r\n"

MANIFEST_KEYS = (
    "kind", "version", "from_seq", "through_seq", "head_seq", "head_hash",
    "complete", "records", "page_size", "max_pages",
)
RECORD_KEYS = (
    "seq", "at", "org_id", "project_id", "project", "actor_kind", "actor_id", "action",
    "resource_kind", "resource_id", "before_json", "after_json", "request_id", "ip",
    "prev_hash", "hash",
)
# Version 2 of the export adds one key, the door.
RECORD_KEYS_VIA = RECORD_KEYS[:12] + ("via_json",) + RECORD_KEYS[12:]

_SEQ = re.compile(r"(0|[1-9][0-9]*)")
_TIME = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9]{2}):([0-9]{2}):([0-9]{2})\.[0-9]{6}Z")
_HEX32 = re.compile(r"[0-9a-f]{64}")
_SURROGATE = re.compile("[\ud800-\udfff]")


class Unreadable(Exception):
    """A file this verifier cannot read; nothing was checked."""


class Break(Exception):
    def __init__(self, kind, seq, expected="", actual="", detail=""):
        super().__init__(kind)
        self.kind, self.seq, self.expected, self.actual, self.detail = kind, seq, expected, actual, detail

    def __str__(self):
        s = "audit chain broken at seq %s: %s" % (self.seq, self.kind)
        if self.expected or self.actual:
            s += " (expected %s, got %s)" % (self.expected, self.actual)
        if self.detail:
            s += ": " + self.detail
        return s


class LineFault(Exception):
    """What is wrong with one line. refuse: the verifier cannot check the file (exit 3); otherwise
    the line is a malformed record, which is a finding (exit 1)."""

    def __init__(self, refuse, msg):
        super().__init__(msg)
        self.refuse, self.msg = refuse, msg


# ---------------------------------------------------------------------------
# Canonical JSON
# ---------------------------------------------------------------------------


class Number(str):
    """A number kept as the literal it was written with."""


def _number(literal):
    neg = False
    s = literal
    if s.startswith("-"):
        neg, s = True, s[1:]
    elif s.startswith("+"):
        s = s[1:]
    mant, exp_part, has_exp = s, "", False
    for marker in ("e", "E"):
        if marker in s:
            mant, exp_part = s.split(marker, 1)
            has_exp = True
            break
    exp = int(exp_part) if has_exp else 0
    int_part, _, frac_part = mant.partition(".")
    digits = int_part + frac_part
    point = len(int_part) + exp
    if len(digits) + abs(exp) > MAX_NUMBER_DIGITS:
        raise ValueError("a number of unreasonable magnitude: " + literal)
    while point <= 0:
        digits = "0" + digits
        point += 1
    while point > len(digits):
        digits += "0"
    whole, frac = digits[:point], digits[point:]
    whole = whole.lstrip("0") or "0"
    out = whole + ("." + frac if frac else "")
    if neg and (whole + frac).strip("0"):
        out = "-" + out
    return out


_ESCAPES = {'"': '\\"', "\\": "\\\\", "\b": "\\b", "\f": "\\f", "\n": "\\n", "\r": "\\r", "\t": "\\t"}


def _unpaired(s):
    """An unpaired surrogate is not a character; the replacement character is what stands for it."""
    return _SURROGATE.sub("\ufffd", s)


def _string(s):
    out = ['"']
    for ch in _unpaired(s):
        if ch in _ESCAPES:
            out.append(_ESCAPES[ch])
        elif ch < " ":
            out.append("\\u%04x" % ord(ch))
        else:
            out.append(ch)
    out.append('"')
    return "".join(out)


def _write(v, out):
    if v is None:
        out.append("null")
    elif v is True:
        out.append("true")
    elif v is False:
        out.append("false")
    elif isinstance(v, Number):
        out.append(_number(str(v)))
    elif isinstance(v, str):
        out.append(_string(v))
    elif isinstance(v, list):
        out.append("[")
        for i, e in enumerate(v):
            if i:
                out.append(",")
            _write(e, out)
        out.append("]")
    elif isinstance(v, dict):
        out.append("{")
        for i, k in enumerate(sorted(v, key=lambda k: k.encode("utf-8"))):
            if i:
                out.append(",")
            out.append(_string(k))
            out.append(":")
            _write(v[k], out)
        out.append("}")
    else:
        raise ValueError("unexpected %s" % type(v).__name__)


def _refuse_constant(name):
    raise ValueError("%s is not JSON" % name)


def _last_key_wins(pairs):
    d = {}
    for k, v in pairs:
        d[_unpaired(k)] = v
    return d


def depth_of(text):
    """The deepest nesting of [ and { outside strings, without parsing (so without recursion)."""
    if text.count("[") + text.count("{") <= MAX_DEPTH:
        return 0
    depth = deepest = 0
    in_string = False
    i, n = 0, len(text)
    while i < n:
        c = text[i]
        if in_string:
            if c == "\\":
                i += 1
            elif c == '"':
                in_string = False
        elif c == '"':
            in_string = True
        elif c in "[{":
            depth += 1
            deepest = max(deepest, depth)
        elif c in "]}":
            depth -= 1
        i += 1
    return deepest


def canonical_json(raw):
    """The canonical text of a JSON document; empty input is empty (a missing value)."""
    if not raw.strip(JSON_SPACE):
        return ""
    doc = json.loads(raw, object_pairs_hook=_last_key_wins, parse_int=Number, parse_float=Number,
                     parse_constant=_refuse_constant)
    out = []
    _write(doc, out)
    return "".join(out)


# ---------------------------------------------------------------------------
# Canonical record, and the chain rule
# ---------------------------------------------------------------------------

_DAYS = (31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)


def canonical_time(text):
    """The instant exactly as it is hashed. Only that form is read: anything else is a different
    spelling of an instant, and two readers that accept different spellings disagree."""
    m = _TIME.fullmatch(text)
    if not m:
        raise ValueError("at %r is not in the form YYYY-MM-DDTHH:MM:SS.ffffffZ (UTC, six fractional "
                         "digits), which is the form that is hashed" % text)
    year, month, day, hour, minute, second = (int(g) for g in m.groups())
    leap = year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)
    days = _DAYS[month - 1] + (1 if month == 2 and leap else 0) if 1 <= month <= 12 else 0
    if not (1 <= day <= days and hour <= 23 and minute <= 59 and second <= 59):
        raise ValueError("at %r is not a time that exists" % text)
    return text


def _refuse_repeats(pairs):
    """An object that repeats a key is not one the server writes (jsonb keeps one), and the canonical form keeps the last, so
    a second spelling of the door would hash like the first and read differently to a reader that keeps the first."""
    seen = set()
    for k, _ in pairs:
        k = _unpaired(k)
        if k in seen:
            raise ValueError("via_json repeats the key %r: a reader that keeps the first and one that keeps the "
                             "last would read two different doors" % k)
        seen.add(k)
    return dict(pairs)


def canonical_via(raw):
    """The canonical text of a record's door: empty when it has none, else a non-empty object. A door that is
    anything else is not one a writer produces, so a record carrying it cannot be hashed."""
    if raw.strip(JSON_SPACE):
        json.loads(raw, object_pairs_hook=_refuse_repeats, parse_int=Number, parse_float=Number,
                   parse_constant=_refuse_constant)
    via = canonical_json(raw)
    if via and (via[0] != "{" or via == "{}"):
        raise ValueError("via_json must be a non-empty JSON object")
    return via


def canonical_bytes(r):
    via = canonical_via(r.get("via_json", ""))
    fields = [
        r["seq"],
        r["org_id"],
        r["project_id"],
        r["actor_kind"],
        r["actor_id"],
        r["action"],
        r["resource_kind"],
        r["resource_id"],
        canonical_json(r["before_json"]),
        canonical_json(r["after_json"]),
        r["request_id"],
        r["ip"],
        canonical_time(r["at"]),
    ]
    version, count = VERSION, FIELD_COUNT
    if via:
        fields.append(via)
        version, count = VIA_VERSION, VIA_FIELD_COUNT
    assert len(fields) == count
    out = bytearray([version, count])
    for f in fields:
        b = f.encode("utf-8")
        out += struct.pack(">I", len(b))
        out += b
    return bytes(out)


def chain_hash(prev, record):
    return hashlib.sha256(prev + canonical_bytes(record)).digest()


# ---------------------------------------------------------------------------
# Reading a line strictly
# ---------------------------------------------------------------------------


class _Pairs(list):
    """An object as the ordered list of its (key, value) pairs, so that a repeated key survives."""


def _decode(raw, where):
    try:
        return raw.decode("utf-8")
    except UnicodeDecodeError:
        raise Unreadable("%s is not valid UTF-8, so it cannot be read as JSON (an export is UTF-8 "
                         "text; the file may have been re-encoded or damaged). Nothing was checked" % where)


def strict_object(text, allowed, version=VERSION):
    """The top-level (key, value) pairs of a line that is exactly one JSON object, or a LineFault.
    The order of the checks is part of the contract (it is the Go verifier's): nesting, JSON,
    one object, a lone surrogate, then the keys."""
    d = depth_of(text)
    if d > MAX_DEPTH:
        raise LineFault(True, "is nested %d levels deep: no export of this format is, and this "
                              "reader stops at %d" % (d, MAX_DEPTH))
    try:
        doc = json.loads(text, object_pairs_hook=_Pairs, parse_constant=_refuse_constant)
    except RecursionError:
        raise LineFault(True, "is nested too deeply to read")
    except ValueError as e:
        raise LineFault(False, "is not JSON (%s)" % e)
    if not isinstance(doc, _Pairs):
        raise LineFault(False, "is JSON but not one object")
    for k, v in doc:
        if _SURROGATE.search(k) or (isinstance(v, str) and _SURROGATE.search(v)):
            raise LineFault(False, "carries a \\u escape of a lone UTF-16 surrogate, which is not "
                                   "text (a decoder would silently read it as U+FFFD)")
    seen = set()
    for k, _ in doc:
        if k in seen:
            raise LineFault(True, "repeats the key %r: a reader that keeps the first and one that "
                                  "keeps the last would see two different records" % k)
        if k not in allowed:
            for a in allowed:
                if a.lower() == k.lower():
                    raise LineFault(True, "has the key %r, which differs only by case from %r: a "
                                          "case-insensitive reader would take it for that field and "
                                          "a case-sensitive one would not" % (k, a))
            raise LineFault(True, "has the key %r, which is not part of version %d of the format"
                            % (k, version))
        seen.add(k)
    return list(doc)


def _is_int64(v):
    return type(v) is int and -(2 ** 63) <= v <= INT64_MAX


def manifest_fault(pairs):
    for k, v in pairs:
        if k in ("kind", "from_seq", "through_seq", "head_seq", "head_hash"):
            ok = isinstance(v, str)
        elif k == "complete":
            ok = v is True or v is False
        elif k == "version":
            ok = v is None or _is_int64(v)
        else:
            ok = _is_int64(v)
        if not ok:
            raise LineFault(True, "has a %s of the wrong type (%.40s)" % (k, json.dumps(v)))


def _seq_of(text):
    """A sequence as the file writes it: plain decimal, within int64."""
    if not _SEQ.fullmatch(text):
        raise ValueError("seq %r is not a plain decimal number" % text)
    n = int(text)
    if n > INT64_MAX:
        raise ValueError("seq %r is not a number this reader can hold" % text)
    return n


# ---------------------------------------------------------------------------
# Reading an export
# ---------------------------------------------------------------------------


def read_manifest(path):
    with open(path, "rb") as f:
        first = f.readline()
    if not first.strip(b" \t\r\n"):
        raise Unreadable("%s is empty: an export starts with a manifest line" % path)
    text = _decode(first, "%s line 1 (the manifest)" % path)
    try:
        pairs = strict_object(text, MANIFEST_KEYS)
        manifest_fault(pairs)
    except LineFault as e:
        if e.refuse:
            raise Unreadable("%s line 1 (the manifest) %s. Nothing was checked" % (path, e.msg))
        raise Unreadable("%s does not start with a manifest line (its first line %s): it is not an "
                         "export of the audit chain" % (path, e.msg))
    m = {"kind": "", "version": None, "from_seq": "", "through_seq": "", "head_seq": "",
         "head_hash": "", "complete": False, "records": 0, "page_size": 0, "max_pages": 0}
    m.update(pairs)
    if m["kind"] != EXPORT_KIND:
        raise Unreadable("%s is not an export of the audit chain" % path)
    if m["version"] is None:
        raise Unreadable("%s has a manifest with no version, so it cannot be read with confidence" % path)
    if m["version"] not in EXPORT_VERSIONS:
        raise Unreadable("%s is export version %r and this verifier reads versions %s: the chain "
                         "is not broken, the verifier is old"
                         % (path, m["version"], " and ".join(str(v) for v in EXPORT_VERSIONS)))
    try:
        m["_from"] = _seq_of(m["from_seq"])
        if m["_from"] < 1:
            raise ValueError("below 1")
    except ValueError:
        raise Unreadable("%s: the manifest's from_seq %r is not a sequence" % (path, m["from_seq"]))
    try:
        m["_through"] = _seq_of(m["through_seq"])
    except ValueError:
        m["_through"] = 0  # only orders files that start together; the file's own check finds it
    return m


class Walker:
    def __init__(self, resume=None):
        self.resume = resume
        # sequence -> (prev_hash, hash) of the latest accepted entries.
        self.seen = {}
        # After its first line, each line of a file must carry the sequence after the one before
        # it, in that file. A repeat between files is tolerated; a repeat inside one is not.
        self.in_file = False
        self.file_next = 0
        self.started = False
        self.anchored = False
        self.prev = GENESIS
        self.next = 0
        self.org = None
        self.first = 0
        self.last = 0
        self.count = 0
        # The last sequences that a file's manifest named as the chain's head, in a file whose last entry that was (its hash is checked against
        # the manifest's). The copy reaches the head only when the walk ended on one of them: the files may overlap or sit inside one another,
        # so the last file need not be the one that ends the chain.
        self.head_claims = set()

    def begin_file(self):
        """The first line of a file may be any sequence the chain has reached."""
        self.in_file = False

    def duplicate(self, seq, prev_hash, hash_, rec):
        """An entry whose sequence has already been passed. It is the same entry, and skipped, when
        its organisation (checked before), previous hash and hash equal the accepted entry's and the
        hash still recomputes from its own fields. A different hash at the same sequence is a fork
        and a hash_mismatch."""
        seen = self.seen.get(seq)
        if seen is None:
            raise Break("out_of_order", seq, str(self.next), str(seq))
        if seen[1] != hash_:
            raise Break("hash_mismatch", seq, seen[1].hex(), hash_.hex())
        if seen[0] != prev_hash:
            raise Break("link_mismatch", seq, seen[0].hex(), prev_hash.hex())
        try:
            want = chain_hash(prev_hash, rec)
        except (ValueError, KeyError, TypeError, RecursionError) as e:
            raise Break("malformed", seq, detail=str(e) or type(e).__name__)
        if want != hash_:
            raise Break("hash_mismatch", seq, want.hex(), hash_.hex())

    def add(self, rec):
        try:
            seq = _seq_of(rec["seq"])
            canonical_time(rec["at"])
        except ValueError as e:
            raise Break("malformed", self.next, detail=str(e))
        for k in ("prev_hash", "hash"):
            if not _HEX32.fullmatch(rec[k]):
                raise Break("malformed", seq, detail="%s is not 64 lower-case hex digits" % k)
        prev_hash, hash_ = bytes.fromhex(rec["prev_hash"]), bytes.fromhex(rec["hash"])
        if not self.started:
            self.started = True
            self.org = rec["org_id"]
            self.first = self.next = seq
            if seq < 1:
                raise Break("malformed", seq, detail="a sequence starts at 1")
            if seq == 1:
                self.prev, self.anchored = GENESIS, True
            elif self.resume is not None:
                self.prev, self.anchored = self.resume, True
            else:
                self.prev = prev_hash
        if rec["org_id"] != self.org:
            raise Break("wrong_org", seq, self.org, rec["org_id"])
        if self.in_file and seq < self.file_next:
            raise Break("out_of_order", seq, str(self.file_next), str(seq))
        if self.in_file and seq > self.file_next:
            raise Break("sequence_gap", seq, str(self.file_next), str(seq))
        if seq < self.next:
            self.duplicate(seq, prev_hash, hash_, rec)
            self.in_file, self.file_next = True, seq + 1
            return
        if seq > self.next:
            raise Break("sequence_gap", seq, str(self.next), str(seq))
        if prev_hash != self.prev:
            raise Break("link_mismatch", seq, self.prev.hex(), prev_hash.hex())
        try:
            want = chain_hash(self.prev, rec)
        except (ValueError, KeyError, TypeError, RecursionError) as e:
            raise Break("malformed", seq, detail=str(e) or type(e).__name__)
        if want != hash_:
            raise Break("hash_mismatch", seq, want.hex(), hash_.hex())
        self.prev, self.next, self.last = hash_, seq + 1, seq
        self.count += 1
        self.in_file, self.file_next = True, seq + 1
        self.seen[seq] = (prev_hash, hash_)
        self.seen.pop(seq - OVERLAP_WINDOW, None)


def verify_file(path, manifest, walker):
    n = 0
    last_seq = 0
    last_hash = ""
    walker.begin_file()
    with open(path, "rb") as f:
        f.readline()  # the manifest
        line_no = 1
        for raw in f:
            line_no += 1
            if not raw.strip(b" \t\r\n"):
                continue
            where = "%s line %d" % (path, line_no)
            text = _decode(raw, where)
            try:
                pairs = strict_object(text, RECORD_KEYS_VIA if manifest["version"] >= 2 else RECORD_KEYS,
                                      manifest["version"])
                for k, v in pairs:
                    if not isinstance(v, str):
                        raise LineFault(False, "has a %s that is not text" % k)
            except LineFault as e:
                if e.refuse:
                    raise Unreadable("%s %s. Nothing was checked" % (where, e.msg))
                raise Break("malformed", walker.next, detail="%s %s" % (where, e.msg))
            rec = dict.fromkeys(RECORD_KEYS_VIA, "")
            rec.update(pairs)
            if not rec["org_id"]:
                raise Unreadable("%s carries no org_id, which every hash covers: it cannot be "
                                 "recomputed. Export it again" % where)
            for k in ("before_json", "after_json", "via_json"):
                d = depth_of(rec[k])
                if d > MAX_DEPTH:
                    raise Unreadable("%s carries a before_json, after_json or via_json nested %d levels deep: no "
                                     "export of this format is, and this reader stops at %d. Nothing "
                                     "was checked" % (where, d, MAX_DEPTH))
            try:
                walker.add(rec)
            except Break as b:
                if b.kind == "malformed" and not b.detail.startswith(where):
                    b.detail = "%s: %s" % (where, b.detail)
                raise
            n += 1
            last_seq, last_hash = _seq_of(rec["seq"]), rec["hash"]
    if manifest["records"] != n:
        raise Break("malformed", last_seq, detail="%s: the manifest says it holds %s records and it holds %d"
                    % (path, manifest["records"], n))
    if n and manifest["through_seq"] != str(last_seq):
        raise Break("malformed", last_seq, detail="%s: the manifest says it ends at seq %s and it ends at seq %d"
                    % (path, manifest["through_seq"], last_seq))
    if n and manifest["head_seq"] == str(last_seq) and manifest["head_hash"] != last_hash:
        raise Break("hash_mismatch", last_seq, manifest["head_hash"], last_hash,
                    "the last record is the head the manifest names, and its hash is not the head's")
    if n and manifest["head_seq"] == str(last_seq):
        walker.head_claims.add(last_seq)


def expand(paths):
    """Files as given; a directory as the files inside it. At the top of a directory every file is
    taken; below it only *.ndjson, because a copy synced from your own storage nests its segments by
    date (audit/v1/<org>/<yyyy>/<mm>/<dd>/<from>.ndjson) and carries other things beside them. A
    checkpoints directory is skipped wherever it is, and so is anything whose name starts with a dot.
    A directory is not followed through a symbolic link."""
    out = []
    for p in paths:
        if os.path.isdir(p):
            root = os.path.normpath(p)
            found = []

            def fail(e):
                raise e

            for here, dirs, names in os.walk(root, onerror=fail):
                dirs[:] = [d for d in dirs if not d.startswith(".") and d != "checkpoints"]
                for n in names:
                    full = os.path.join(here, n)
                    if n.startswith(".") or not os.path.isfile(full):
                        continue
                    if here != root and not n.endswith(".ndjson"):
                        continue
                    found.append(full)
            if not found:
                raise FileNotFoundError("%s holds no files" % p)
            out += sorted(found)
        elif os.path.isfile(p):
            out.append(p)
        else:
            raise FileNotFoundError("%s does not exist" % p)
    return out


def main(argv):
    args = list(argv)
    resume = None
    if args and args[0] == "--resume-hash":
        if len(args) < 2 or not re.fullmatch(r"[0-9a-fA-F]{64}", args[1]):
            print("--resume-hash is the hash of the record before the first one: 64 hex digits", file=sys.stderr)
            return 2
        resume = bytes.fromhex(args[1])
        args = args[2:]
    if not args:
        print("usage: verify.py [--resume-hash HEX] FILE_OR_DIRECTORY...", file=sys.stderr)
        return 2
    try:
        paths = expand(args)
    except OSError as e:
        print("error: %s" % e, file=sys.stderr)
        return 2

    walker = Walker(resume)
    try:
        # Ordered by what their manifests say: first sequence ascending, and of two files that start
        # together the longer first, so that a file inside another is read after it and is
        # recognised as a repeat.
        files = sorted(((read_manifest(p), p) for p in paths), key=lambda mp: (mp[0]["_from"], -mp[0]["_through"]))
        if resume is not None and files[0][0]["_from"] == 1:
            print("error: --resume-hash given, but the first file starts at seq 1", file=sys.stderr)
            return 2
        for manifest, path in files:
            verify_file(path, manifest, walker)
    except Unreadable as e:
        print("error: %s" % e, file=sys.stderr)
        return 3
    except Break as b:
        print("error: %s" % b, file=sys.stderr)
        return 1
    except OSError as e:
        print("error: %s" % e, file=sys.stderr)
        return 2
    except Exception as e:  # nothing here may end in a traceback that reads as a finding
        print("error: could not check: %s: %s. Nothing was checked" % (type(e).__name__, e), file=sys.stderr)
        return 3
    if walker.count == 0:
        print("error: the files hold no records, so there is nothing to check", file=sys.stderr)
        return 3

    print("consistent: %d records (seq %d through %d), every hash recomputed and every link held."
          % (walker.count, walker.first, walker.last))
    print("This shows the copy is internally consistent, not who wrote it; a later copy checked against "
          "an earlier copy's head hash is what shows nothing before it was rewritten.")
    if not walker.anchored:
        print("note: the file starts at seq %d and no --resume-hash was given, so nothing before it "
              "was checked." % walker.first)
    if walker.last not in walker.head_claims:
        print("head: the copy does not say it reaches the chain's head, so entries after it are not covered.")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Check a signed checkpoint

Once delivery to your own bucket is running and signing is on for your environment, we also sign the head of your log, at most once a day and only when the log has gained an entry since the previous checkpoint, and write the signed document beside your entries as audit/v1/<organisation>/checkpoints/<yyyy-mm-dd>.json, with its signature in the .sig file of the same name. A day without a checkpoint is not by itself a fault. The document is one line of JSON: kind, version, your org_id, an entry number seq, that entry’s hash, verified_at (the time we verified it) and key_id. There is none to check today, because delivery to your own bucket is not switched on yet.

What a checkpoint shows: the document is, byte for byte, what a key we hold signed, and the private half of that key never leaves the key service of our cloud account. What it does not show by itself: that the head it names is true. We hold the key, so whoever could make us sign could make us sign a false head. Two things help you catch it. The checkpoint sits in a bucket in your account under Object Lock, which keeps the version we wrote from being deleted or altered for as long as your bucket’s retention runs; it does not by itself stop us writing a newer version under the same name, so list the object’s versions, and a checkpoint with more than one version is a finding: keep every version and tell us. And its entry number and hash are ones you can compare with a copy of your log that you took yourself: if we rewrote the log before that entry after you took your copy, the signed head would disagree with it. A checkpoint says nothing about entries after the one it names.

To check one you need openssl, jq and the list of keys we sign with, at runninglake.com/.well-known/runninglake-audit-keys.json. Keep a copy of that list and fetch it again now and then, so a change to it is visible: a key that stops signing should stay in it, marked retired, and nothing yet enforces that. Use only a key whose environment is prod if your organisation runs in production; today the list holds a key for our test environment only. Copy the two checkpoint files byte for byte. The document is written before its signature, so a copy taken between the two has no signature; copy both again a little later. Then, with the key_id from the document, and the list saved as runninglake-audit-keys.json (replace KEY_ID with that key id and <yyyy-mm-dd> with the day in the file’s name):

jq -r --arg id KEY_ID '.keys[] | select(.key_id == $id) | .public_key' runninglake-audit-keys.json > key.pem
openssl dgst -sha256 -verify key.pem -signature <yyyy-mm-dd>.sig <yyyy-mm-dd>.json

The second prints Verified OK when the document is exactly what that key signed. Anything else is a finding: keep both files and tell us. If it verifies, compare the document’s org_id, seq and hash with the same three fields of the entry with that sequence number in an export of your log, checked as above. The signature is ECDSA over P-256 and SHA-256, DER-encoded, and key_id is the SHA-256 of the key’s DER-encoded public key in lower-case hex.

What each break means

A file with no version in its manifest, or a version the checker does not know, is not reported as a break. The chain is not broken; the checker is old, or the file is not an export. Both checkers refuse it and exit 3, as they do for every refusal listed under the exit codes above.

What a check cannot prove

• The hash is not keyed or signed. Anyone holding the file can edit an entry and recompute its hash and every hash after it, and the result is still consistent; whoever can edit our database can do the same there. So a clean check shows a copy agrees with itself, and not who wrote it or that it is the record as it was first written. A signed daily checkpoint is meant to close part of that gap, and what one shows and what it does not is above.
• Keep each copy’s head hash, which is in the first line of the file as head_hash. A later copy that continues from it (give it to -resume-hash), or that holds the same hash at the same sequence number, shows that nothing before that head has been rewritten since you took the earlier copy.
• Comparing your last entry with the one we hold, with rl audit verify -against-server, shows that we still hold what you hold. It is the same server that wrote the chain answering about itself, and the comparison is recorded in the log.
• A copy that ends early looks consistent. The manifest says where the chain was when you took the copy; an entry written after that is not in your file. When the manifest names no head, or one beyond where the file ends, a check run on its own (rl audit verify without -against-server, or the script above) says so on a last line that begins head:: the copy does not say it reaches the chain’s head, so entries after it are not covered.
• It cannot show you an entry that was never written. It shows that the record is consistent, not that the record is everything that happened. The audit chain says what the chain does cover.