Pipeline

The config schema's version, which is independent of the HTTP API's (rule unsupported-api-version).

Values: runninglake.io/v1alpha1

The kind this file declares. A kind that disagrees with the file's path is refused (rule kind-mismatch).

Values: Pipeline

No description

The resource's name, which is also its path segment. It must be the name this file's path gives it (rule name-mismatch).

No description

No description

No description

No description

No description

No description

Schedule is when the pipeline runs by itself: a mapping, with cron required. Absent, it runs only when started. Recorded with the pipeline and checked here; nothing schedules a pipeline yet, so nothing runs on it.

Cron is when it runs: five fields (minute, hour, day of month, month, day of week), or @hourly, @daily, @weekly, @monthly or @yearly. A seconds field and Quartz's L, W, # and ? are refused.

Timezone is the IANA zone the cron is read in, such as Europe/London. Default UTC. Abbreviations such as EST, and Local, are refused: they name no place's rules.

Paused stops the schedule from starting runs. Resuming does not replay the paused period. A run started by hand still runs.

Catchup is what happens to runs missed while the platform could not start them: latest (default) runs the newest one, all runs the newest 5, none runs none.

Overlap is what a scheduled start does while a run is still active: skip (default) records it skipped, queue waits up to maxQueuedRuns, cancel_previous cancels the active run.

MaxConcurrentRuns is how many runs may be active at once. Default 1, and only a partitioned pipeline may say more: two runs of one table would write it twice.

MaxQueuedRuns is how many runs overlap: queue may hold waiting. Default 20.

Jitter delays each scheduled start by up to this long, such as 30s or 2m, at most 5m; the same delay every time for one pipeline. Default none.

No description

No description

No description

Name is the table this asset writes, spelled as its Table declares it: the namespace levels, then the table, joined by dots, such as gold.finance.daily_revenue, or gold.daily_revenue for a table directly in a layer. Each level is [a-z][a-z0-9_-]{0,62}; a level beginning with "_" is the platform's, the _files segment a volume lives under among them, and is refused. Once the tree declares a Layer, the name must be a Table the tree declares, in this pipeline's routing.format, and no other pipeline may write it: running never creates a table. Nothing runs a pipeline yet, so nothing writes it.

SQL is this asset's statement, written inline. Set exactly one of sql and sqlFile. A value that reads as a file path (one word ending in .sql) is refused: sql is never read as a path, so the file it names would never be read.

SQLFile is the path of a file holding this asset's statement, relative to the pipeline's directory: models/daily_revenue.sql names pipelines/<pipeline>/models/daily_revenue.sql. Set exactly one of sql and sqlFile. Once the project has declared a layer, a commit is refused when the path leaves the directory or is not a .sql file outside files/, when its name is one the platform does not write, when the folder holds no such file, or when the file is over 256 KiB or not UTF-8 text. The statement itself is not checked here.

PythonFile is the path of a Python script, relative to the pipeline's directory, that is this asset's body in place of a statement: python/clean_sales.py names pipelines/<pipeline>/python/clean_sales.py. Set exactly one of sql, sqlFile and pythonFile. An asset with a pythonFile is a code step: the table named by name is the only table its script writes, by appending to it (materialization: append, in a pipeline whose routing.format is ICEBERG), and no SQL engine is asked to run it. Nothing runs a code step yet, so a pipeline with one is recorded and checked and cannot be started. Once the project has declared a layer, a commit is refused when the folder holds no such file, when the path is not a .py file outside files/, or when the file is over 256 KiB or not UTF-8 text.

Code is what a code step declares about itself: the tables and volumes its script may read, the registered connections it may call, whether it is safe to run again, and optionally where it must run. Allowed only with pythonFile. Leaving it out declares a script that reads nothing, calls nothing and makes no promise about being run again. Recorded with the pipeline and checked for its shape: nothing runs a code step yet, so nothing here limits a script today.

Reads are the tables and volumes the script may read. Each is checked for being a name the platform can address, and not yet for being declared.

Tables are the tables the script may read, each spelled as its Table declares it (gold.finance.daily_revenue, or gold.daily_revenue for a table directly in a layer).

Volumes are the volumes whose files the script may read, each spelled layer.schema.volume (bronze.pos.landing).

Connections are the names of the registered connections the script may call, such as fx-rates. Each name is spelled as a resource name is: lower-case letters, digits, hyphens and underscores, at most 63 characters. Recorded only: the registry these names resolve against is not built, so a name is checked for its shape and not for being a connection anyone registered.

RetrySafe declares that running the script again after a failure is safe: it writes the same rows for the same inputs and calls nothing that cannot be repeated. Default false. Recorded; nothing reads it yet.

Deterministic declares that the script returns the same rows for the same inputs. Default false. Recorded; nothing reads it yet.

Runtime pins where the step runs: `function` for a short, small task on the cloud's own functions, `pod` for a heavy or long one on Kubernetes. Leave it out to let the platform pick. Any other word is refused (rule code-step-runtime). Recorded; nothing reads it yet.

The flow this asset is generated from: flows/<name>.yaml in the pipeline's folder, with the same name as the SQL file, models/<name>.sql. The platform writes both. The pipeline stops compiling when the SQL file and the flow no longer agree.

How this asset's result is written to its table: `snapshot` replaces the table's rows with the result, `append` adds the result's rows, `merge` updates the rows whose uniqueKey matches a result row and inserts the rest, and `scd1` does that for the columns that changed and also removes the rows a true-or-false column of the result marks as deleted. The words are lowercase, and any other word is refused (rule materialization-unknown). The word decides which engines may run the asset, because a replace has to land as one commit and an append does not. Nothing executes a pipeline yet, so this declares the write; it does not perform it, and `scd1` is declared and checked and cannot be started yet (a pipeline that names it is refused when a run is planned).

No description

DependsOn names the other assets of this same pipeline that this asset depends on, each spelled as its name is, so a pipeline's assets form a graph. A name this pipeline does not declare, the asset's own name, and a cycle are refused.

The operation class this asset asks of an engine, which the router decides eligibility on. Leave it out and it is derived from materialization: `snapshot` is OVERWRITE, `append` is INSERT and `merge` is MERGE. A declared class is one of SELECT, INSERT, OVERWRITE, PARTITION_OVERWRITE, CTAS, MERGE, UPDATE or DELETE, spelled in capitals (rule operation-class-unknown), and must agree with the materialization (rule operation-class-mismatch).

Business rule ids this asset implements, such as BR-4, each once. Recorded with the pipeline, and read back when the pipeline is opened again as a draft: the draft carries the words of the rules its assets name here.

How long one attempt of this asset's task may run before it is stopped, as a duration in hours, minutes or seconds such as 45m, 2h or 1h30m. Default 1h. A timeout written here may not be longer than the pipeline's runTimeout, nor longer than 24h, the most the plane runs a task for. Recorded with the pipeline; nothing enforces it yet.

How this asset's task is tried again when it fails. It replaces the pipeline's retries whole rather than merging with it: a field left out here takes its default, not the pipeline's value. Absent, the pipeline's retries apply; `retries: {}` takes every default instead, so a failed task is not tried again. Recorded with the pipeline; nothing enforces it yet.

How many times a task is tried in all, its first attempt included, from 1 to 10. Default 1: a failed task is not tried again. Recorded with the pipeline; nothing enforces it yet.

How long to wait after the first failed attempt before trying again, as a duration such as 30s, 2m or 1m30s. Default 30s. Recorded with the pipeline; nothing enforces it yet.

What each wait is multiplied by to give the next, so the waits grow: 2 doubles them and 1 keeps them equal. At least 1. Default 2. Recorded with the pipeline; nothing enforces it yet.

The longest any one wait may grow to, as a duration such as 10m or 1h. It may not be shorter than initialInterval. Default 10m. Recorded with the pipeline; nothing enforces it yet.

How far each wait is moved at random, as a fraction of it either way, from 0 to 1: 0.2 moves a 60s wait to anywhere from 48s to 72s, and 0 moves nothing. Default 0.2. Recorded with the pipeline; nothing enforces it yet.

Whether an attempt stopped by its timeout is tried again. Default false: only an attempt that failed is tried again, and one that timed out ends the task. Recorded with the pipeline; nothing enforces it yet.

No description

No description

The check's class: not_null, range, unique, referential_integrity, freshness or distribution_stability. Recorded with the pipeline; nothing checks the word or runs a check yet.

No description

No description

No description

The pipeline's design document: DESIGN.md at the root of the pipeline's folder. The platform writes it and serves it back as written. A pipeline that names one stops the commit if the file is missing or unusable.

No description

The table format: iceberg, hudi or ducklake, in any case. Required (rules format-required and format-unknown). Choose it once: a plan that changes the format of a table the platform has already reported is refused (rule format-is-immutable), and a Pipeline's is held to its tables' once the tree declares a Layer (rule asset-format). A change pushed straight to the repository's branch does not go through a plan, so it is not refused, and does not change a table that already exists.

An engine to pin, or empty to leave the choice to the router. On a pipeline, the authoring gate refuses a pin no rule permits for its format and operations, and names the rule that fired. Nothing runs a pipeline yet, so nothing is routed today.

A release-train FAMILY to pin, written YYYY.MM (for example 2026.11), or empty to follow the train of the plane the environment runs on. A Pipeline's setting; on a Table it is read by nothing. A pin names a family and not a serial: the serials inside a family are the same engines and move with the plane, and a family does not move while a pipeline pins it. A serial is refused (rule train-pin-is-a-family), and so is a family this control plane's runtime index does not hold (train-unknown) or one outside its support window, the current family plus two (train-out-of-support). When a family is pinned, spec.routing.engine is judged against the engines that family runs, and not against the plane's.

Who reads or writes the table, and how fresh they need it. Recorded; nothing reads it yet, so a consumer constrains no engine and no storage mode today.

No description

Either read or write. Recorded; nothing reads it yet.

How fresh this consumer needs the table, as a Go duration string, e.g. "5m". Recorded; nothing reads it yet.

No description

No description

The monthly cap in US dollars. Leave it out for no cap; 0 is a cap of zero, and the two are different values. Recorded with the pipeline; nothing enforces it yet.

What should happen when the cap is reached: `hold` is meant to queue the job and notify the owner. Recorded with the pipeline; nothing enforces it yet, so no job is held today.

How a failed task is tried again, for each asset of this pipeline that sets no retries of its own. An asset's own retries replaces this block whole rather than merging with it: a field the asset's block leaves out takes its default, not the value set here. With retries set nowhere, a failed task is not tried again. Recorded with the pipeline; nothing enforces it yet.

How many times a task is tried in all, its first attempt included, from 1 to 10. Default 1: a failed task is not tried again. Recorded with the pipeline; nothing enforces it yet.

How long to wait after the first failed attempt before trying again, as a duration such as 30s, 2m or 1m30s. Default 30s. Recorded with the pipeline; nothing enforces it yet.

What each wait is multiplied by to give the next, so the waits grow: 2 doubles them and 1 keeps them equal. At least 1. Default 2. Recorded with the pipeline; nothing enforces it yet.

The longest any one wait may grow to, as a duration such as 10m or 1h. It may not be shorter than initialInterval. Default 10m. Recorded with the pipeline; nothing enforces it yet.

How far each wait is moved at random, as a fraction of it either way, from 0 to 1: 0.2 moves a 60s wait to anywhere from 48s to 72s, and 0 moves nothing. Default 0.2. Recorded with the pipeline; nothing enforces it yet.

Whether an attempt stopped by its timeout is tried again. Default false: only an attempt that failed is tried again, and one that timed out ends the task. Recorded with the pipeline; nothing enforces it yet.

How long one run of this pipeline may take from when it starts, as a duration in hours, minutes or seconds such as 12h, 90m or 1h30m; a run still going then ends timed out. Default 24h. Recorded with the pipeline; nothing enforces it yet.

How long a run may wait to start, queued or held, as a duration in hours, minutes or seconds such as 6h or 90m; a run still waiting then is cancelled. Default 24h. Recorded with the pipeline; nothing enforces it yet.

What should happen to a run's output when a quality check fails: block_all, publish_passing_quarantine_failing or publish_all_and_alert. Recorded with the pipeline; nothing reads it yet and no default is applied, so nothing is held or quarantined today.

Whether drift on this resource should be reported or corrected. Recorded with the pipeline; nothing reads it yet — the reconciler's drift sweep does not consult it, so `correct` corrects nothing today.

Partitions splits the pipeline's output so more than one run may be active at once: absent, the pipeline is unpartitioned and runs one at a time (spec.schedule.maxConcurrentRuns above 1 is refused without it). Recorded with the pipeline; the scheduler and plane do not target or bind a single partition yet (8.1.41-8.1.43), so this declares the shape without changing how a run executes today.

Kind is time or static.

Granularity is hourly, daily, weekly or monthly. Required for kind time; refused for kind static. An hourly partition is always UTC: timezone must be empty or UTC.

Start is the first partition's date, YYYY-MM-DD. Required for kind time; refused for kind static.

Timezone is the IANA zone partition boundaries fall on, such as Europe/London. Default UTC. Refused for kind static, and for hourly granularity unless it is empty or UTC.

Column is the declared Table's own column this pipeline's assets partition by, checked against that table's schema. Required for kind time; refused for kind static.

Values is the closed list of partitions for kind static. Required (at least one) for kind static; refused for kind time.

DependsOnPast admits partition N only once partition N-1 succeeded (Airflow's meaning). Default false. Refused on a pipeline with no partitions: with one partition there is no N-1 to wait for.

Parameters declare, by name and type, the values a run of this pipeline takes: at most 32. Recorded with the pipeline and checked here; no run can be started with values yet, so declaring one changes no run today. A statement written inline reads one with getvariable('name'), the name as a string, and a name that is not declared is refused there; a statement kept in a file is not checked yet. secretRef is reserved and refused: a parameter cannot be a secret yet.

Name is what a statement reads with getvariable('name'): lower case, a letter first, letters, digits and underscores, 63 characters at most. A name that starts with rl_ is the platform's own and is refused here.

Type is string, integer, decimal, boolean, date, timestamp or enum. An integer is a 64-bit whole number, a decimal has at most 29 digits before the point and 9 after it, a date is YYYY-MM-DD, a timestamp is RFC 3339 with an offset and is kept in UTC, and an enum is one of the values listed.

Description says what the parameter is for. Says what the value means to whoever starts a run.

Required means a run cannot start without a value. A required parameter has no default.

Default is the value used when a run is started with none, written as text in the type's spelling and held to the type and to the constraints below. Left out, an optional parameter is empty to the statement (NULL) when no value is given.

MaxLength is a string's longest length in characters, from 1 to 4096. Left out, 4096.

Pattern is a string's shape, a regular expression in RE2 syntax of at most 256 characters. A value must match it whole.

Min is the lowest value an integer, decimal, date or timestamp parameter may have, written as text in the type's spelling.

Max is the highest value an integer, decimal, date or timestamp parameter may have, written as text in the type's spelling.

Scale is the most digits a decimal may have after the point, from 0 to 9. Left out, 9.

Values are an enum's choices: at least one and at most 64, each up to 256 characters, none repeated.

SecretRef is reserved. A parameter that sets it is refused: a secret cannot be passed as a parameter yet.

Triggers declare the outside calls that may start this pipeline, reviewed as a change set like the SQL they feed. Today the only kind is a webhook: a name, a map from the pipeline's parameters to a place in the sender's JSON body (or to a constant), and ceilings that can only lower an endpoint's. A trigger never names a project, an environment or a pipeline, so a payload can never choose them. Recorded with the pipeline and checked here; a person can create an endpoint for a trigger in the console, but the address a sender calls is not served yet, so declaring one starts nothing today.

Webhooks are at most 16. Each has a name that is unique among them.

Name is lower case, a letter first, then letters, digits and hyphens, 63 characters at most, such as stripe-payout. An endpoint is bound to it by name.

Parameters map a parameter of this pipeline to where its value comes from: {from: <RFC 6901 pointer>} reads the sender's JSON body, such as /data/object/arrival_date; {value: <constant>} is fixed. At most 32. Pointers only: no filter, no template, no expression, at most 16 levels deep, with ~0 and ~1 as the only escapes and no - token. A required parameter with no default must be mapped. A parameter read FROM the body must be constrained (a string by a pattern or an enum, a number or a date by a min and a max), because the signature proves who sent a body and not that its contents are safe to size work or name a file by.

From is a pointer. In a storage rule it points into the batch: /batch/id, /batch/files, /batch/bytes, /batch/firstArrival, /batch/lastArrival, /batch/closeReason or /rule/name; and, from a rule whose batches hold one file, /objects/0/size, /objects/0/key or /objects/0/uri. In a webhook trigger it points into the sender's JSON body.

Value is a constant.

Limits lower an endpoint's ceilings for this trigger. They can only lower: an endpoint starts at most 30 runs an hour and has at most 5 active, by default.

StartsPerHour is 1 to 30.

ActiveRuns is 1 to 5.

Automation starts this pipeline when its upstream assets update, instead of only on its own schedule or by hand. Absent, nothing starts it but its schedule and a direct start. Recorded with the pipeline; the upstream evaluator that watches materialisations and starts a run does not exist yet (8.1.9, on the ledger 8.1.49 writes), and neither does the cross-pipeline cycle check, so this declares the shape without changing when a run starts today.

Condition is eager (fires when any of Assets has materialised since this pipeline's last run, and no upstream run is in progress) or on_cron (at the schedule's tick, fires only if every one of Assets has updated). on_cron needs spec.schedule.cron.

Assets are the upstream tables that trigger this pipeline, each named as its Table declares itself: layer.table or layer.schema.table. Empty defaults to the tables spec.sources names, which are what this pipeline's own statements read.

Debounce waits this long after an upstream update before firing, in case more follow. Default 5m.

MaxWait bounds how long Debounce can keep deferring while updates keep arriving: at least Debounce.