Skip to content
RunningLake
Company Console ↗ Request early access
Contents: Reference

BUILDING Parts of this page describe what is not built yet, and each says so.

The API

Everything the console and the rl command do goes through one API, and a script can call it directly. This page is for the person writing that script.

Where it is

The API is served at the address your console is served from. A call is an HTTP POST to /<package>.<Service>/<Method> with a JSON body, in the Connect protocol, so a plain HTTP client works and you need none of our libraries. In the example below, $RL_API is that address with no slash at the end, and $RL_TOKEN is the credential from the next section.

Who is calling

  • A CI job holds no secret. It proves who it is with the identity token its CI system issues, and exchanges that for a short-lived token of a service principal. CI with GitHub Actions and CI with GitLab show the whole job.
  • A server or an orchestrator holds the secret of a service principal: create the principal in the console under API access, then mint a secret for it. A secret is shown once, so put it in your secret manager and never in a repository.
  • A person at a terminal can use a personal API token, which belongs to that person. It works, and it is not for automation: a script that others depend on should run as a service principal.

A service principal can hold only three actions: pipeline.run, run.read and run.cancel. A call that needs another action, such as placing a hold or reading a pipeline’s files, is for people. Every call needs a bearer credential in the Authorization header.

Make a call

Listing the runs of an environment needs the permission to read runs, which a service principal holds as “Read runs”: 

curl -s "$RL_API/runninglake.v1.RunService/ListRuns" \
  -H "Authorization: Bearer $RL_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Connect-Protocol-Version: 1" \
  -d '{"project":"analytics","environment":"dev","pageSize":5}' \
  | jq '.runs[] | {id, pipeline, status}'

The answer is JSON in the Connect encoding: field names are camelCase (nextPageToken), an enumeration is its word (RUN_STATUS_SUCCEEDED), and a 64-bit integer is a string of digits. A call that only reads may also be sent as a GET, with the request in the message query parameter; the reference says which.

Errors

A refusal is a Connect error: a JSON body with code, message and details, and response headers that say what happened.

  • rl-error-reason is a short stable word, for what a program may branch on. Match on it, not on the message.
  • rl-action names the permission that was refused, when one was.
  • rl-request-id is on every answer, a success included. Quote it when you ask us about a call.
  • rl-retry-after-ms says how long to wait, in milliseconds, when you are calling too fast. The standard Retry-After says the same in whole seconds, rounded up.

New reasons can be added, so treat a reason you have not seen as you would an error you do not know: fall back on the Connect code. A call the reference marks “Not released yet” answers with the code unimplemented, HTTP 501, and the reason not_built.

Pages

A call that lists takes pageSize and pageToken and answers nextPageToken. With no pageSize you get 50, and the most you can ask for is 200: a larger number is treated as 200, not refused. nextPageToken is empty on the last page. Send it back, unchanged, as pageToken to get the next one.

A token belongs to the query that made it: the organisation, project, environment and filters. Used with other filters, or if it is not a token we issued, it is refused with the code invalid_argument and the reason invalid_page_token. Start again from the first page.

Idempotency

A call that makes something takes an idempotencyKey: 1 to 128 characters, letters, digits and . _ : -. A key is yours: it is scoped to the caller that sent it and to the pipeline, so two callers who send the same key do not meet. If a connection drops and you send the same key with the same body again, you get the run the first call made, with replayed set. The same key with a different body is refused.

8 calls of RunService take a key. 4 are released today: CancelRun, PlaceHold, ReleaseHold, StartRun. This is what StartRun does today, in the words of the reference:

StartRun records one run of a pipeline now. Needs pipeline.run at project scope; no step-up, so a token can call it. It starts the whole pipeline: assets and partition_key are not accepted yet, and a request that sets one is refused with invalid_argument. parameters are checked against the pipeline's declarations; a refusal is invalid_argument with a BadRequest detail naming each parameter and never a value.

Today no data plane reports that it can run a pipeline's tasks, so a start is refused with failed_precondition, its message says why, and nothing is recorded. A start that is accepted records a queued run; nothing moves a queued run yet.

Limits

Two refusals look alike and are not. Both are the Connect code resource_exhausted, and the reason tells them apart.

  • rate_limited: you are calling faster than you may. Slow down, and wait as long as Retry-After or rl-retry-after-ms says.
  • quota_exceeded: a bound is full, such as the work already running. Slowing down does not help, and the refusal carries no wait: the remedy is to wait for work to finish.

What is published

One service is published: RunService as beta. 10 of the 23 calls of RunService are released today; the rest answer not_built. Everything else the console calls is internal, and may change without notice.

The reference lists every published call, and the OpenAPI document is the same description as a file, for a generator or a linter of your own. How the API changes says how much notice a change gets.