BUILDING Parts of this page describe what is not built yet, and each says so.

RunService

Beta. A change to anything on this page is announced at least 90 days before it takes effect.

RunService starts, watches and controls the runs of a project's pipelines, and reads a project's pipelines and the files of a pipeline's folder.

RunService is in BETA: complete enough to try, still settling, and most of its calls are marked "Not released yet" below. A change to a beta call is announced at least 90 days before it takes effect, with the Deprecation and Sunset response headers.

Authenticate with a bearer credential. For automation use a service principal: its secret, or the short-lived token a CI job exchanges for it. A personal API token also works, but it belongs to a person and is not for automation. A service principal can hold only the actions pipeline.run, run.read and run.cancel, so a call that needs another action (placing a hold, a backfill, reading a pipeline's files) is for people.

Every call names its project by slug and, where an environment matters, the environment by name. The calls that only read (every Get and List except GetPipelineFile) may be made with GET as well as POST. Every start takes an idempotency_key: the same key with the same body answers with the run it made, with replayed set.

A call that is not released yet is marked so below. It answers Unimplemented with the reason not_built (HTTP 501), so a client written against this reference gets that answer, and not a 404, until the call is released.

Calls

23 calls. A call marked Not released yet is published and answers not_built until the control plane serves it. GET says the call may also be sent as a GET, with the request in the message query parameter.

Call	GET	Today
`CancelBackfill`	No	Not released yet
`CancelRun`	No	Released
`ConfirmBackfill`	No	Not released yet
`GetBackfill`	Yes	Not released yet
`GetPipeline`	Yes	Released
`GetPipelineFile`	No	Released
`GetPipelineSchedule`	Yes	Not released yet
`GetRun`	Yes	Released
`GetRunMetrics`	Yes	Not released yet
`ListAssetMaterializations`	Yes	Not released yet
`ListBackfills`	Yes	Not released yet
`ListHolds`	Yes	Released
`ListPartitions`	Yes	Not released yet
`ListPipelines`	Yes	Released
`ListRuns`	Yes	Released
`ListScheduleTicks`	Yes	Not released yet
`PlaceHold`	No	Released
`PreviewSchedule`	No	Not released yet
`ReleaseHold`	No	Released
`RepairRun`	No	Not released yet
`ResumeBackfill`	No	Not released yet
`StartBackfill`	No	Not released yet
`StartRun`	No	Released

CancelBackfill

Not released yet: it answers not_built. It is sent as a POST.

CancelBackfill stops a backfill. Needs run.cancel.

Request: CancelBackfillRequest. Response: CancelBackfillResponse. Path: /runninglake.v1.RunService/CancelBackfill

CancelRun

Released. It is sent as a POST.

CancelRun asks a run to stop. Needs run.cancel. A queued or held run ends at once; an admitted or running one moves to cancelling, its live attempts are stopped and the run ends cancelled. A run that has ended, or is already cancelling, is returned as it is. The reason is optional, at most 500 characters, and is kept in the audit log only. It answers Unimplemented on a control plane whose orchestrator is not switched on.

Request: CancelRunRequest. Response: CancelRunResponse. Path: /runninglake.v1.RunService/CancelRun

ConfirmBackfill

Not released yet: it answers not_built. It is sent as a POST.

ConfirmBackfill starts a backfill whose check the caller has read. Needs pipeline.backfill.

Request: ConfirmBackfillRequest. Response: ConfirmBackfillResponse. Path: /runninglake.v1.RunService/ConfirmBackfill

GetBackfill

Not released yet: it answers not_built. It may also be sent as a GET.

GetBackfill returns one backfill. Needs run.read.

Request: GetBackfillRequest. Response: GetBackfillResponse. Path: /runninglake.v1.RunService/GetBackfill

GetPipeline

Released. It may also be sent as a GET.

GetPipeline returns one pipeline's summary and its applied specification. Needs run.read.

Request: GetPipelineRequest. Response: GetPipelineResponse. Path: /runninglake.v1.RunService/GetPipeline

GetPipelineFile

Released. It is sent as a POST.

GetPipelineFile returns the text of one file of a pipeline's folder, read from git at the commit where the pipeline's files last changed (the commit the folder's listing was last changed at, which is not the branch head). Needs pipeline.file.read at project scope: owner, admin and engineer, never a service principal. The call stores no text, and reading a file is recorded in the audit log. A file that looks like it holds a credential is withheld, not sent. A refusal is a Connect error with the reason in the rl-error-reason header and no text; see GetPipelineFileResponse for the words.

Request: GetPipelineFileRequest. Response: GetPipelineFileResponse. Path: /runninglake.v1.RunService/GetPipelineFile

GetPipelineSchedule

Not released yet: it answers not_built. It may also be sent as a GET.

GetPipelineSchedule says when a pipeline next fires and why it is in its state. Needs run.read.

Request: GetPipelineScheduleRequest. Response: GetPipelineScheduleResponse. Path: /runninglake.v1.RunService/GetPipelineSchedule

GetRun

Released. It may also be sent as a GET.

GetRun returns one run with its tasks and attempts. Needs run.read at project scope.

Request: GetRunRequest. Response: GetRunResponse. Path: /runninglake.v1.RunService/GetRun

GetRunMetrics

Not released yet: it answers not_built. It may also be sent as a GET.

GetRunMetrics returns a run's measured figures per attempt, task and engine (CPU seconds, peak memory, spill, bytes scanned, rows, log volume), each absent when the runner did not report it. Needs run.read.

Request: GetRunMetricsRequest. Response: GetRunMetricsResponse. Path: /runninglake.v1.RunService/GetRunMetrics

ListAssetMaterializations

Not released yet: it answers not_built. It may also be sent as a GET.

ListAssetMaterializations returns the runs that last materialised each asset. Needs run.read.

Request: ListAssetMaterializationsRequest. Response: ListAssetMaterializationsResponse. Path: /runninglake.v1.RunService/ListAssetMaterializations

ListBackfills

Not released yet: it answers not_built. It may also be sent as a GET.

ListBackfills returns a pipeline's backfills. Needs run.read.

Request: ListBackfillsRequest. Response: ListBackfillsResponse. Path: /runninglake.v1.RunService/ListBackfills

ListHolds

Released. It may also be sent as a GET.

ListHolds returns the holds that bear on a project, newest first: its own and its environments' and pipelines', and the organisation-wide ones. Needs run.read.

It answers unimplemented while emergency holds are not switched on for this control plane.

Request: ListHoldsRequest. Response: ListHoldsResponse. Path: /runninglake.v1.RunService/ListHolds

ListPartitions

Not released yet: it answers not_built. It may also be sent as a GET.

ListPartitions returns a backfill's partitions. Needs run.read.

Request: ListPartitionsRequest. Response: ListPartitionsResponse. Path: /runninglake.v1.RunService/ListPartitions

ListPipelines

Released. It may also be sent as a GET.

ListPipelines returns an environment's pipelines as summaries. What the applied configuration holds (name, owner, schedule, whether the controller has read it) is sent; the run statistics are computed on the server and are left unset until a run can exist. Needs run.read at project scope.

Request: ListPipelinesRequest. Response: ListPipelinesResponse. Path: /runninglake.v1.RunService/ListPipelines

ListRuns

Released. It may also be sent as a GET.

ListRuns returns a project's runs, newest first. Needs run.read at project scope.

Request: ListRunsRequest. Response: ListRunsResponse. Path: /runninglake.v1.RunService/ListRuns

ListScheduleTicks

Not released yet: it answers not_built. It may also be sent as a GET.

ListScheduleTicks returns what the scheduler decided at each fire time. Needs run.read.

Request: ListScheduleTicksRequest. Response: ListScheduleTicksResponse. Path: /runninglake.v1.RunService/ListScheduleTicks

PlaceHold

Released. It is sent as a POST.

PlaceHold holds the runs of an organisation, project, environment or pipeline until it is released or expires, at most 72 hours on. Needs run.hold, with no step-up: an emergency stop must be instant. Naming no project holds the whole organisation, which only an owner or an admin may do. A run that starts under a hold is held, never refused; a run already running continues. The same idempotency_key by the same caller with the same request returns the hold it made.

It answers unimplemented while emergency holds are not switched on for this control plane.

Request: PlaceHoldRequest. Response: PlaceHoldResponse. Path: /runninglake.v1.RunService/PlaceHold

PreviewSchedule

Not released yet: it answers not_built. It is sent as a POST.

PreviewSchedule says when a cron expression would fire. Needs run.read.

Request: PreviewScheduleRequest. Response: PreviewScheduleResponse. Path: /runninglake.v1.RunService/PreviewSchedule

ReleaseHold

Released. It is sent as a POST.

ReleaseHold ends a hold early. Needs run.hold.release, with step-up, which only a signed-in person can give: no API token can release a hold. An owner or an admin may release any hold; an engineer only one they placed themselves. Releasing resumes the runs it held.

It answers unimplemented while emergency holds are not switched on for this control plane.

Request: ReleaseHoldRequest. Response: ReleaseHoldResponse. Path: /runninglake.v1.RunService/ReleaseHold

RepairRun

Not released yet: it answers not_built. It is sent as a POST.

RepairRun reruns the failed part of a finished run. Needs pipeline.run.

Request: RepairRunRequest. Response: RepairRunResponse. Path: /runninglake.v1.RunService/RepairRun

ResumeBackfill

Not released yet: it answers not_built. It is sent as a POST.

ResumeBackfill reruns a backfill's failed partitions. Needs pipeline.backfill.

Request: ResumeBackfillRequest. Response: ResumeBackfillResponse. Path: /runninglake.v1.RunService/ResumeBackfill

StartBackfill

Not released yet: it answers not_built. It is sent as a POST.

StartBackfill starts, or with validate_only only checks, a backfill of a pipeline's partitions. Needs pipeline.backfill.

Request: StartBackfillRequest. Response: StartBackfillResponse. Path: /runninglake.v1.RunService/StartBackfill

StartRun

Released. It is sent as a POST.

StartRun records one run of a pipeline now. Needs pipeline.run at project scope; no step-up, so a token can call it. It starts the whole pipeline: assets and partition_key are not accepted yet, and a request that sets one is refused with invalid_argument. parameters are checked against the pipeline's declarations; a refusal is invalid_argument with a BadRequest detail naming each parameter and never a value.

Today no data plane reports that it can run a pipeline's tasks, so a start is refused with failed_precondition, its message says why, and nothing is recorded. A start that is accepted records a queued run; nothing moves a queued run yet.

Request: StartRunRequest. Response: StartRunResponse. Path: /runninglake.v1.RunService/StartRun

Messages

Each field is named as it is written in JSON. A request may leave out a field it does not need. A response leaves out a field that has its default value, so read a field that is missing as that default.

AssetMaterialization

Field	Type
`asset`	string
`runId`	string
`at`	timestamp (an RFC 3339 string)

Backfill

Field	Type
`id`	string
`pipeline`	string
`status`	string

BackfillPartition

Field	Type
`key`	string
`status`	string
`runId`	string

BackfillTrigger

Field	Type
`backfillId`	string
`partitionKey`	string
`index`	integer
`count`	integer

CancelBackfillRequest

Field	Type
`project`	string
`backfillId`	string
`cancelRunning`	boolean
`reason`	string
`idempotencyKey`	string

CancelBackfillResponse

Field	Type
`backfill`	Backfill

CancelRunRequest

Field	Type
`project`	string
`runId`	string
`reason`	string
`idempotencyKey`	string

CancelRunResponse

Field	Type
`run`	Run

ConfirmBackfillRequest

Field	Type	Meaning
`project`	string	No description
`backfillId`	string	No description
`estimateDigest`	string	estimate_digest names the check the caller read, so a changed estimate is refused.

ConfirmBackfillResponse

Field	Type
`backfill`	Backfill

GetBackfillRequest

Field	Type
`project`	string
`backfillId`	string

GetBackfillResponse

Field	Type
`backfill`	Backfill

GetPipelineFileRequest

GetPipelineFileRequest names one file of a pipeline's folder, as the listing (GetPipeline's source.files) named it. The listing is the allowlist: a path it does not hold is refused, and so is one it holds with a problem, so a request cannot ask git for a name of its own choosing.

Field	Type	Meaning
`project`	string	No description
`environment`	string	No description
`name`	string	name is the pipeline, as GetPipelineRequest.name is: the bare name.
`path`	string	path is exactly the PipelineFile.path the listing gave, relative to the pipeline's folder.
`sha256`	string	sha256 is the PipelineFile.sha256 of the row the person opened, 64 lowercase hex characters. Required: an empty or malformed one is invalid_argument. It binds the answer to the row that was clicked, so a listing that moved between the click and the read is refused (file_listing_moved) and not answered with another version's text.

GetPipelineFileResponse

GetPipelineFileResponse is one file's text. A refusal is never a reply: it is a Connect error, so a client cannot show a field of a refusal by mistake.

THE REASONS A REFUSAL CARRIES, in the rl-error-reason header, and the ONLY ones. Every one but file_path rides FailedPrecondition: - file_path (InvalidArgument): a segment or the path is malformed (empty, over 255 bytes, a NUL or control character, absolute, "..", or not equal to its cleaned form), or sha256 is not 64 lowercase hex characters. - file_not_listed: the path is not in the listing (this includes a file past the first 100), or the listing names a kind of file this call does not read. - file_listing_moved: the request's sha256 is not the listing's now. Reload the list. - file_too_big: the listing marks the file too_big, or git holds more than 262144 bytes at the path. - file_not_text: the listing marks the file not_text, or git's bytes are not UTF-8 or hold a NUL. - file_unreadable_name: the listing marks the file bad_name; its stored path is not one git holds. - file_source_unavailable: the control plane could not read the file from git just now: the listing names no commit, the commit or the file is not in the clone, the remote is not one it can read, or the read was cancelled or timed out. - file_source_mismatch: the bytes git holds at that commit do not hash to the listed sha256. No text. - file_withheld: the text looks like it holds a password or key, or the check could not run. The message does not say which.

Two more refusals carry no file reason. A caller without pipeline.file.read gets permission_denied from the interceptor, and a pipeline the caller cannot see, or that is not declared, gets not_found. A read the control plane could not record in the audit log is Unavailable with the reason unavailable_retry and no text: nothing is returned that was not recorded. A caller reading too fast, or too many reads at once, is ResourceExhausted.

Field	Type	Meaning
`content`	string	content is the file's text: valid UTF-8, no NUL, at most 262144 bytes. Never truncated: a file over the limit is refused (file_too_big), not cut.
`sha256`	string	sha256 is the SHA-256 of the bytes read, 64 lowercase hex characters. It equals the request's: the control plane checked.
`commitSha`	string	commit_sha is the 40-hex commit the bytes were read from: the one at which the listing's files last changed, not the branch head.

GetPipelineRequest

Field	Type
`project`	string
`environment`	string
`name`	string

GetPipelineResponse

Field	Type	Meaning
`pipeline`	PipelineSummary	No description
`specJson`	string	spec_json is the applied specification as JSON text. New fields may appear in it as pipelines learn to declare more, so read the ones you know. A specification holds references to secrets, never their values.
`source`	PipelineSource	source is what the control plane read from the pipeline's folder in git: the files beside pipeline.yaml. UNSET means no read of the folder has been recorded for this pipeline yet, which is not the same as an empty folder; an empty folder is a source with no files.
`observedGeneration`	int64 (a string of digits)	observed_generation is the generation of the declaration the pipeline controller last looked at. Unset when it has never looked. A source whose generation is higher than this was read from git before the controller observed it.
`parameters`	PipelineParameters	parameters are the pipeline's declared parameters, read from the applied declaration: what a run may be started with, for the pipeline page, the start form and rl. UNSET means they could not be read (the stored declaration is unreadable, or the control plane is older than this field); SET AND EMPTY means the pipeline declares none. A declaration proposed in a change set and not yet applied is not here.
`automation`	PipelineAutomationDetail	automation is the upstream trigger in full. UNSET when the pipeline declares none, its declaration could not be read, or the environment's tables could not be listed to resolve it.

GetPipelineScheduleRequest

Field	Type
`project`	string
`environment`	string
`pipeline`	string

GetPipelineScheduleResponse

Field	Type
`schedule`	PipelineSchedule

GetRunMetricsRequest

GetRunMetricsRequest names the scope of the read. The field that says whose figures to read is added when the call is released.

Field	Type
`project`	string

GetRunMetricsResponse

GetRunMetricsResponse carries nothing yet: the figures are added when the call is released.

No fields.

GetRunRequest

Field	Type
`project`	string
`runId`	string

GetRunResponse

Field	Type
`run`	Run

Hold

Field	Type	Meaning
`id`	string	No description
`kind`	string	kind is why runs are held: emergency, for a hold a person placed. The set of words may grow.
`reason`	string	No description
`scope`	string	scope is what the hold covers: org, project, environment or pipeline.
`project`	string	project, environment and pipeline name what the hold covers; each is empty where the scope is wider.
`environment`	string	No description
`pipeline`	string	No description
`placedBy`	string	placed_by is the principal that placed it, as <kind>:<id>.
`placedAt`	timestamp (an RFC 3339 string)	No description
`expiresAt`	timestamp (an RFC 3339 string)	expires_at is when the hold ends by itself, at most 72 hours after placed_at.
`cancelActive`	boolean	cancel_active says the runs in flight in the scope were also asked to cancel when the hold was placed.
`active`	boolean	active is true while the hold is not released and not expired.
`releasedAt`	timestamp (an RFC 3339 string)	No description
`releasedBy`	string	released_by is the principal that released it, or system:hold-expiry when it expired.
`releaseReason`	string	No description

ListAssetMaterializationsRequest

Field	Type
`project`	string
`environment`	string
`pipeline`	string
`pageSize`	integer
`pageToken`	string

ListAssetMaterializationsResponse

Field	Type
`materializations`	list of AssetMaterialization
`nextPageToken`	string

ListBackfillsRequest

Field	Type
`project`	string
`environment`	string
`pipeline`	string
`pageSize`	integer
`pageToken`	string

ListBackfillsResponse

Field	Type
`backfills`	list of Backfill
`nextPageToken`	string

ListHoldsRequest

Field	Type	Meaning
`project`	string	No description
`environment`	string	environment narrows the list to the holds that bear on one environment of the project.
`pageSize`	integer	No description
`pageToken`	string	No description
`activeOnly`	boolean	active_only lists only the holds in force.

ListHoldsResponse

Field	Type
`holds`	list of Hold
`nextPageToken`	string

ListPartitionsRequest

Field	Type
`project`	string
`backfillId`	string
`pageSize`	integer
`pageToken`	string

ListPartitionsResponse

Field	Type
`partitions`	list of BackfillPartition
`nextPageToken`	string

ListPipelinesRequest

Field	Type
`project`	string
`environment`	string
`pageSize`	integer
`pageToken`	string

ListPipelinesResponse

Field	Type
`pipelines`	list of PipelineSummary
`nextPageToken`	string

ListRunsRequest

Field	Type	Meaning
`project`	string	No description
`environment`	string	No description
`pipeline`	string	No description
`statuses`	list of RunStatus	No description
`triggers`	list of TriggerKind	No description
`principal`	string	No description
`since`	timestamp (an RFC 3339 string)	No description
`until`	timestamp (an RFC 3339 string)	No description
`backfillId`	string	No description
`rootRunId`	string	No description
`view`	RunView	No description
`pageSize`	integer	No description
`pageToken`	string	No description
`storageRule`	string	storage_rule lists only the runs a storage rule's batches started, by the rule's name within `environment`, which must then be set. It needs read access to the project's configuration as well as to its runs (a service principal is refused). A rule that does not exist in that environment, or that is hidden from the caller, is NotFound; a rule that was deleted can no longer be named, and its runs are still listed without this filter. It cannot be combined with `principal`. Like `principal`, it filters the newest runs the server examines, so a rule that has not fired lately can give a short or empty page that still carries a next_page_token: follow the token.

ListRunsResponse

Field	Type
`runs`	list of Run
`nextPageToken`	string

ListScheduleTicksRequest

Field	Type
`project`	string
`environment`	string
`pipeline`	string
`pageSize`	integer
`pageToken`	string

ListScheduleTicksResponse

Field	Type
`ticks`	list of ScheduleTick
`nextPageToken`	string

PipelineAutomation

PipelineAutomation is what a pipeline's list row and header need of its upstream trigger.

Field	Type	Meaning
`condition`	string	condition is eager (start a run when a watched table is written, once the writes have gone quiet and no run is in the way) or on_cron (the pipeline's schedule decides when, and a tick fires only when every watched table has been updated since the last one).
`registered`	boolean	registered says an evaluator holds this generation of the declaration. UNSET means it could not be read. False means nothing evaluates the trigger yet: the declaration is accepted and stored, and no run is started by a table being written. It is false on a control plane that does not run the evaluator, and until the pipeline's next pass after the evaluator is turned on.

PipelineAutomationDetail

PipelineAutomationDetail is the rest of a pipeline's upstream trigger, for the pipeline page.

Field	Type	Meaning
`watches`	list of string	watches are the tables the trigger watches, as dotted names (layer.table or layer.schema.table), at most 64: the ones the declaration names, or, when it names none, the tables the pipeline's sources read that this environment declares. THE DEFAULT LIST IS WHAT THE CALLER CAN SEE: a table in a layer the caller cannot see is left out of it for that caller, though the trigger watches it, so two callers can be shown different defaults for one pipeline.
`watchesDefaulted`	boolean	watches_defaulted is true when the declaration names no table and watches is the default.
`unresolved`	list of string	unresolved are the names the declaration lists that are not tables this environment declares, as the caller can see them: a table in a layer the caller cannot see is listed here for that caller.
`debounce`	duration (a string such as "30s")	debounce is how long the writes must have been quiet before a run starts; the default (five minutes) is counted when the declaration says none, so it is always set.
`maxWait`	duration (a string such as "30s")	max_wait bounds how long continuing writes may defer a run. Unset means no bound.

PipelineFile

PipelineFile is one file in a pipeline's folder. The listing never carries the file's text: GetPipelineFile sends it, one file at a time.

Field	Type	Meaning
`path`	string	path is relative to the pipeline's folder, with forward slashes.
`role`	string	role is what kind of file it is: sql, python, scala, design (DESIGN.md, at the root of the folder), flow (flows/<name>.yaml) or other (anything under files/). The platform writes the design and flow files; a request cannot add or change one.
`sizeBytes`	int64 (a string of digits)	size_bytes is the file's size in bytes. Unset when it is not known.
`sha256`	string	sha256 is the SHA-256 of the file's bytes, as 64 lowercase hex characters.
`usedBy`	list of string	used_by are the assets of this environment that name the file as their sqlFile (a flow file: as their flow), in the order the pipeline declares them. The match is on the cleaned path: spaces around the name, a leading "./" and doubled or trailing slashes are removed first, and a name that is empty, absolute or climbs out of the folder with ".." matches nothing. Case is not folded. DESIGN.md is named by the pipeline's designDoc and not by an asset, so it has no users here. Empty means no asset names it, or that this could not be worked out; problem says unused only when it was worked out.
`problem`	string	problem is empty or one word: too_big (over 256 KiB, so GetPipelineFile refuses it), not_text (not UTF-8 text, or holds a NUL), bad_name (the name is not one the platform writes: a character outside lowercase letters, digits, underscore and hyphen, more than one dot, no dot, an upper-case extension, a dot in a directory name, or a path over 255 bytes; it is listed and is not counted in the digest), unused (a SQL file or a flow file that no asset in this environment names, or DESIGN.md when the pipeline declares no designDoc), model_differs (a flow file whose generated SQL file does not match it) or flow_unreadable (a flow file that is not in the format the platform writes, so it cannot be compared with its SQL file). The last two are set on the flow file and never on the SQL file, and an asset that names such a flow does not compile: its pipeline's Compiled condition is false with the reason GeneratedFileMismatch.
`generatedFrom`	string	generated_from is the path of the flow file this file is generated from, relative to the pipeline's folder, as flows/<name>.yaml. It is set on a SQL file that an asset of this environment names as its sqlFile while that same asset also names a flow, and it is empty for every other file. Such an asset does not compile while the SQL file and the flow differ. Empty also means that this could not be worked out, as used_by does, so an empty value is not proof that a file was written by hand.

PipelineParameter

PipelineParameter is one declared parameter AS DECLARED: its name, its type and its rules. It is never a value of any run, and never a secret: a parameter that names a secret is refused where the declaration is written. Every value in it is text in the type's own spelling (a decimal 0.10, a date 2026-01-31, a timestamp in UTC), as the declaration wrote it.

Field	Type	Meaning
`name`	string	No description
`type`	string	type is string, integer, decimal, boolean, date, timestamp or enum.
`description`	string	No description
`required`	boolean	required means a run cannot start without a value. A required parameter has no default.
`defaultValue`	string	default_value is what a run gets when it is started with none. Unset means there is no default, so an optional parameter is empty (NULL) to the statement.
`maxLength`	integer	max_length and pattern are a string's: the longest length in characters, and a regular expression (RE2) a value must match whole. Unset and empty when the declaration left them out.
`pattern`	string	No description
`min`	string	min and max are the bounds of an integer, decimal, date or timestamp, and scale a decimal's most digits after the point. Unset when left out.
`max`	string	No description
`scale`	integer	No description
`values`	list of string	values are an enum's choices, in declaration order.

PipelineParameters

PipelineParameters is the declared parameters of one pipeline, in declaration order, at most 32. It is a message and not a repeated field so that "none declared" and "not read" are two different answers.

Field	Type
`parameters`	list of PipelineParameter

PipelineSchedule

Field	Type
`pipeline`	string

PipelineSource

PipelineSource is the folder of one pipeline, as it was last read from git.

Field	Type	Meaning
`digest`	string	digest is one fingerprint of every file in the folder, as "v1:" or "v2:" and 64 lowercase hex characters. The tag is "v2:" for a folder that holds a flow file (flows/<name>.yaml) or the design document (DESIGN.md), and "v1:" for every other folder, so a folder that holds neither keeps the fingerprint it always had; taking the last of them away returns the fingerprint of what remains, under "v1:". It changes when the path or the bytes of any file change, and for nothing else; a file marked bad_name is left out of it. An empty folder has the fingerprint of no bytes at all.
`generation`	int64 (a string of digits)	generation is the generation of the pipeline this folder was read with.
`files`	list of PipelineFile	files are the folder's files in a fixed order: the design document, then flow files, then SQL, then Python, then Scala, then other, and by path within each. At most 100 are listed; warnings says when the folder is larger.
`warnings`	list of string	warnings are facts about the folder as a whole. A word is one of: too_many_files (more than 100 files, so only the first 100 by path are listed) or too_large (more than 4 MiB in total, so only the first 100 by path are listed and the listing keeps no file's text). At most ONE word is sent: the file count is judged first, so a folder past both limits says too_many_files. Both limits count every file the folder looked at, those marked bad_name included. The digest covers every file except any marked bad_name.

PipelineSummary

PipelineSummary is a pipeline's row: what it is, how it is triggered and how it has done.

Field	Type	Meaning
`name`	string	No description
`owner`	string	No description
`paused`	boolean	paused is true when the schedule is paused. A manual or API start still runs.
`cron`	string	cron and timezone are the declared schedule, empty for a pipeline with none.
`timezone`	string	No description
`nextRunAt`	timestamp (an RFC 3339 string)	No description
`recentStatuses`	list of RunStatus	recent_statuses are the last runs' statuses, newest first, at most ten.
`latestStatus`	RunStatus	No description
`latestRunAt`	timestamp (an RFC 3339 string)	No description
`succeeded30d`	integer	succeeded_30d and finished_30d are the runs that finished in the last thirty days, computed on the server; absent when the server has not counted them.
`finished30d`	integer	No description
`medianDuration`	duration (a string such as "30s")	No description
`ready`	boolean	ready is false until the controller has read the declaration and every table it writes is ready; it does not yet mean a run can start. ready_reason says why not.
`readyReason`	string	No description
`automation`	PipelineAutomation	automation is the upstream trigger the pipeline declares: it starts the pipeline when a table it watches is written by another pipeline. UNSET means the pipeline declares none, or its declaration could not be read. A pipeline can have a schedule too.

PlaceHoldRequest

Field	Type	Meaning
`project`	string	No description
`environment`	string	No description
`pipeline`	string	No description
`reason`	string	reason is why, 3 to 500 characters; it is in the audit log and shown on every run the hold holds.
`idempotencyKey`	string	No description
`expiresInSeconds`	integer	expires_in_seconds is how long the hold lasts: 1 to 259200 (72 hours).
`cancelActive`	boolean	cancel_active also cancels the runs already in flight (admitted or running) in the hold's scope, after the hold is in force, each by the cancel door with the class hold_emergency. Needs run.cancel as well as run.hold. Queued and held runs are not cancelled: the hold holds them and they resume when it ends. A retry with the same idempotency_key cancels the runs the first call did not reach.

PlaceHoldResponse

Field	Type	Meaning
`hold`	Hold	No description
`runsCancelled`	integer	With cancel_active, the runs in flight the hold covers, as this call reached them. runs_cancelled is how many it moved to cancelling. runs_not_cancelled is how many it could not (an error, or more runs than one call walks): the hold is in force either way, and a retry with the same idempotency_key tries them again. runs_not_permitted is how many are in projects where the caller may not cancel (run.cancel at that project, which for a hold on the whole organisation can be less than at the organisation): a retry does not change that, and they are not cancelled. A run that was already cancelling or finished is in none.
`runsNotCancelled`	integer	No description
`runsNotPermitted`	integer	No description

PreviewScheduleRequest

Field	Type
`project`	string
`cron`	string
`timezone`	string
`count`	integer

PreviewScheduleResponse

Field	Type
`fires`	list of ScheduleFire

ReleaseHoldRequest

Field	Type	Meaning
`project`	string	project is empty for an organisation-wide hold.
`holdId`	string	No description
`idempotencyKey`	string	idempotency_key is required. A retry by the same person of a release that succeeded returns the hold it released; anyone else's retry, or a hold that expired, answers failed_precondition.
`reason`	string	reason is why it is released early, up to 500 characters; empty is recorded as "released early".

ReleaseHoldResponse

Field	Type
`hold`	Hold

RepairRunRequest

Field	Type	Meaning
`project`	string	No description
`runId`	string	No description
`mode`	string	mode (FAILED_AND_DOWNSTREAM or ALL) says how much of the run to repair, and version (CURRENT or ORIGINAL) says which revision of the pipeline to repair it with. The spelling is final when RepairRun is released.
`taskKeys`	list of string	No description
`version`	string	No description
`idempotencyKey`	string	No description

RepairRunResponse

Field	Type
`run`	Run

RepairTrigger

Field	Type
`repairedRunId`	string

ResumeBackfillRequest

Field	Type
`project`	string
`backfillId`	string
`idempotencyKey`	string

ResumeBackfillResponse

Field	Type
`backfill`	Backfill

Run

Run is one execution of a pipeline's declared graph, as the control plane last heard of it.

Field	Type	Meaning
`id`	string	No description
`project`	string	No description
`environment`	string	No description
`pipeline`	string	No description
`status`	RunStatus	No description
`errorClass`	string	error_class is a short class word, never an error message. Empty until the run ends in one. The set of words may grow: show a word you do not know as it came.
`cancelClass`	string	cancel_class says who or what ended a cancelled run (a person, a queue timeout, a hold that expired). Empty otherwise.
`trigger`	Trigger	No description
`idempotencyKey`	string	No description
`parameters`	list of RunParameter	parameters are the values the run was started with, in name order, with the type and where each came from. GetRun and StartRun carry them; ListRuns does not.
`createdAt`	timestamp (an RFC 3339 string)	No description
`admittedAt`	timestamp (an RFC 3339 string)	No description
`startedAt`	timestamp (an RFC 3339 string)	No description
`finishedAt`	timestamp (an RFC 3339 string)	No description
`hold`	RunHold	hold is set while the run is held, and says why.
`planeId`	string	plane_id and train say where the run ran: empty until a plane takes it.
`train`	string	No description
`configCommit`	string	config_commit and config_generation say which revision of the pipeline's declaration the run was planned from, so a later edit never rewrites what the run was.
`configGeneration`	int64 (a string of digits)	No description
`rootRunId`	string	root_run_id and parent_run_id group a repair's runs with the run they repair. Empty otherwise.
`parentRunId`	string	No description
`tasks`	list of RunTask	tasks are carried by GetRun and by ListRuns with RUN_VIEW_WITH_TASK_STATES.
`taskCount`	integer	task_count is the number of tasks the run has, which may exceed len(tasks) when the list was cut for size.
`parameterValuesClearedAt`	timestamp (an RFC 3339 string)	parameter_values_cleared_at is when the values in parameters were cleared, once the run's restart window had ended or a person asked for it. Unset while the values are kept. After it is set every parameter keeps its name, type and source and has an empty value: an empty value on a run with this set is a cleared one, never a value that was empty. GetRun and StartRun carry it; ListRuns does not.

RunAttempt

RunAttempt is one try of a task.

Field	Type	Meaning
`number`	integer	No description
`status`	AttemptStatus	No description
`cause`	string	cause is why the attempt exists (first, user_retry, platform_retry, repair).
`queuedAt`	timestamp (an RFC 3339 string)	No description
`startedAt`	timestamp (an RFC 3339 string)	No description
`finishedAt`	timestamp (an RFC 3339 string)	No description
`engine`	string	engine and engine_reason say which engine ran it and which rule chose it.
`engineReason`	string	No description
`errorClass`	string	No description
`rowsOut`	int64 (a string of digits)	No description
`bytesScanned`	int64 (a string of digits)	No description
`cpuCoreSeconds`	number	No description
`peakMemoryBytes`	int64 (a string of digits)	No description

RunHold

RunHold says why a run is held.

Field	Type	Meaning
`kind`	string	kind is why the run is held, for example emergency, trial_exhausted or spend_cap. The set of words may grow: show a word you do not know as it came.
`holdId`	string	No description
`placedBy`	string	No description
`reason`	string	No description
`expiresAt`	timestamp (an RFC 3339 string)	No description
`blockingRunId`	string	No description

RunParameter

RunParameter is one parameter a run was started with. A secret's value is empty, and so is every value once the run's parameter_values_cleared_at is set.

Field	Type
`name`	string
`type`	string
`value`	string
`secret`	boolean
`source`	string

RunTask

RunTask is one node of a run's graph.

Field	Type	Meaning
`key`	string	No description
`name`	string	name is resolved from the run's own revision, so a rename never rewrites history.
`dependsOn`	list of string	No description
`status`	TaskStatus	No description
`errorClass`	string	No description
`maxAttempts`	integer	No description
`timeout`	duration (a string such as "30s")	No description
`attempts`	list of RunAttempt	No description

ScheduleFire

Field	Type
`at`	timestamp (an RFC 3339 string)

ScheduleTick

Field	Type
`scheduledFor`	timestamp (an RFC 3339 string)
`outcome`	string
`reason`	string

ScheduleTrigger

Field	Type	Meaning
`scheduledFor`	timestamp (an RFC 3339 string)	scheduled_for is the fire time the tick was for, which may precede created_at by a skew.

StartBackfillRequest

Field	Type	Meaning
`project`	string	No description
`environment`	string	No description
`pipeline`	string	No description
`idempotencyKey`	string	No description
`validateOnly`	boolean	validate_only checks the backfill and returns its estimate without starting it.

StartBackfillResponse

Field	Type
`backfill`	Backfill

StartRunRequest

Field	Type	Meaning
`project`	string	No description
`environment`	string	No description
`pipeline`	string	No description
`idempotencyKey`	string	idempotency_key makes a retried start the same start: the same body answers with the run it made and replayed = true; a different body with the same key is refused.
`parameters`	map of string to string	parameters are the values for the pipeline's declared parameters, as text in each type's spelling. A pipeline that declares none takes none. StartRun does not start a pipeline that declares one on a plane that does not report it can bind them.
`assets`	list of string	assets will limit the run to these assets and what they need. It is not accepted yet: leave it empty to start the whole pipeline, or the request is refused with invalid_argument.
`partitionKey`	string	partition_key is not accepted yet: leave it unset, or the request is refused with invalid_argument.

StartRunResponse

Field	Type
`run`	Run
`replayed`	boolean

StorageEventTrigger

StorageEventTrigger says which batch of files started a run, and which storage rule's batch it was. It is set only on a run whose trigger kind is STORAGE_EVENT, and only while the rule exists and the caller may see it: the caller needs read access to the project's configuration (a service principal never has it) and the rule must not be hidden from them. For any other caller, or once the rule is deleted, the trigger keeps its kind and principal (system:storagerule:<rule id>, which names the rule by id) and carries no detail. It carries counts and times and never a file's name or path.

Field	Type	Meaning
`rule`	string	rule is the rule's resource name, projects/{project}/environments/{environment}/storagerules/{rule}. The rule is always in the run's own project and environment.
`ruleId`	string	rule_id and rule_name are the rule's id and its name within its environment.
`ruleName`	string	No description
`batch`	string	batch is the batch's resource name, projects/{project}/environments/{environment}/storagerules/{rule}/batches/{batch}, and batch_id is its id.
`batchId`	string	No description
`files`	integer	files is the number of the batch's new events: the files the rule recorded for the first time. A repeated delivery of a file, and a file the rule had already processed, are not counted. bytes is the size of those new events that had one: an event with no size (a deleted file) counts in files and adds nothing to bytes.
`bytes`	int64 (a string of digits)	No description
`firstArrival`	timestamp (an RFC 3339 string)	first_arrival and last_arrival bound when the plane received the batch's events, the repeated and already-processed ones included.
`lastArrival`	timestamp (an RFC 3339 string)	No description
`closeReason`	string	close_reason is why the batch closed: quiet, full, max_wait, plane_limit, rule_changed or backfill. The set of words may grow: show a word you do not know as it came.

Trigger

Trigger is why a run exists: its kind and who caused it, with the facts of that kind.

Field	Type	Meaning
`kind`	TriggerKind	No description
`principal`	string	principal is `user:<id>`, `token:<id>`, `agent:<id>`, `service:<id>` (a service principal) or `system:<name>`.

UpstreamTrigger

Field	Type
`runId`	string
`pipeline`	string
`asset`	string

WebhookRunTrigger

WebhookRunTrigger says which webhook endpoint's delivery started a run whose kind is WEBHOOK. It is set only on such a run. endpoint_id is the endpoint's id, and webhook_id is the delivery id the sender gave it (its Webhook-Id header): the one identifier the sender's own log and this run share, and the key a retry of the same delivery is recognised by. The run's principal is the service principal the endpoint belongs to. It carries no payload, no header and no parameter value.

Field	Type
`endpointId`	string
`webhookId`	string

Values

An enumeration is sent and read as one of these words. A client should treat a word it does not know as something newer than itself.

AttemptStatus

AttemptStatus is one attempt's state.

ATTEMPT_STATUS_UNSPECIFIED, ATTEMPT_STATUS_QUEUED, ATTEMPT_STATUS_RUNNING, ATTEMPT_STATUS_SUCCEEDED, ATTEMPT_STATUS_FAILED, ATTEMPT_STATUS_CANCELLED, ATTEMPT_STATUS_TIMED_OUT

RunStatus

RunStatus is a run's state. A run in SUCCEEDED, FAILED, CANCELLED or TIMED_OUT has finished.

RUN_STATUS_UNSPECIFIED, RUN_STATUS_QUEUED, RUN_STATUS_HELD, RUN_STATUS_ADMITTED, RUN_STATUS_RUNNING, RUN_STATUS_CANCELLING, RUN_STATUS_SUCCEEDED, RUN_STATUS_FAILED, RUN_STATUS_CANCELLED, RUN_STATUS_TIMED_OUT

RunView

RunView says how much of a run a list carries.

RUN_VIEW_UNSPECIFIED, RUN_VIEW_BASIC, RUN_VIEW_WITH_TASK_STATES

TaskStatus

TaskStatus is one task's state within a run.

TASK_STATUS_UNSPECIFIED, TASK_STATUS_PENDING, TASK_STATUS_QUEUED, TASK_STATUS_RUNNING, TASK_STATUS_RETRYING, TASK_STATUS_SUCCEEDED, TASK_STATUS_FAILED, TASK_STATUS_TIMED_OUT, TASK_STATUS_CANCELLED, TASK_STATUS_UPSTREAM_FAILED, TASK_STATUS_SKIPPED

TriggerKind

TriggerKind is why a run exists.

TRIGGER_KIND_UNSPECIFIED, TRIGGER_KIND_SCHEDULE, TRIGGER_KIND_MANUAL, TRIGGER_KIND_API, TRIGGER_KIND_BACKFILL, TRIGGER_KIND_UPSTREAM, TRIGGER_KIND_REPAIR, TRIGGER_KIND_AGENT, TRIGGER_KIND_STORAGE_EVENT, TRIGGER_KIND_WEBHOOK

All services