BUILDING Parts of this page describe what is not built yet, and each says so.
Job logs
A run is one execution of a pipeline, made of tasks, which are its steps. A job is one attempt at one task, or one query; it is not a pipeline, and not one of the control plane’s own background jobs. Its logs are what it wrote while it ran: a task’s output and error output, or what the engine recorded about a query, such as its output, its profile or its stage detail.
Jobs run on a plane, the Kubernetes cluster where your queries run and your files live. RunningLake’s control plane is the other half, with the console and the API; the data plane page explains both.
No plane keeps job logs yet. Nothing runs a pipeline yet. The first logs to be kept are designed to be a query’s. This page says how job logs are designed to be kept, so that you can judge the design before there is anything to read. The first four sections are that design, apart from the sentences in them that say what is true today; the four after them say what is built and what is not.
What is built is small. A project can declare how long it wants job logs kept. A run’s Logs tab and a query’s Logs section in the SQL workspace say why there is nothing to show. The control plane answers a question about a query’s logs only for whoever ran it. And the AI policy accepts a setting that would allow log excerpts to be sent to a model, and turning it on needs an owner or an admin to approve.
Where job logs are kept
On a plane in your own cloud account, job logs are designed to be kept in that account. While a job runs, its lines are designed to be in the plane’s system bucket, a separate bucket from the one where the plane keeps the project’s data, and then to be moved into that data bucket, into a folder of the project’s environment that holds nothing but job logs. In the design, a line is your data in either bucket, and can repeat values from your tables. The control plane is designed to keep no copy of a log, though on a plane we host the logs are in our account; the data plane page says whose account a plane is in. The control plane does hold the text of the queries you submit, as that page also says. The design is written for AWS, the one cloud a plane is built for today.
In the design, a job’s output is kept as separate streams, one for each place it came from, and which streams a job has depends on the engine that ran it: a task’s output and its error output are two, and an engine may add more, such as a profile. In the design each stream, apart from an engine’s own event log, is cut into numbered segments, so that a browser asks for the part it is showing and not the whole log.
In the design, a log can have gaps: lines are lost if a node stops or a log file rotates before the plane reads them, a very long line or a very long stream is cut, and lines that fail to move expire after about 3 days. The plane is designed to report the lines it knows it dropped, which can be fewer than were lost, and the viewer is designed to say when a stream has a gap.
Two keys, and the time between them
In the design, two encryption keys protect a job log, one after the other. The plane’s key is one key for the whole plane, so on a plane shared by several organisations it is shared by them. In the design, the lines of one organisation that are still waiting to be moved sit in a folder of the system bucket named by the identifiers of the organisation, the project and the environment. The control plane is designed to sign a short-lived link to a job log, one that works for whoever holds it until it expires, only inside a folder whose path it builds itself from its own records, never inside one that a plane or a request supplies. A project’s key is made for that project alone: the control plane creates it in the plane’s account, through its role, when the project is set up on the plane. Both keys are in the plane’s cloud account, so on a plane in your own account they are in your account.
While a job runs, the plane is designed to keep its lines in the plane’s system bucket, a separate bucket from the one that holds the project’s data, under the plane’s key, because the plane’s agent, the program in your cluster that opens the connection to the control plane, cannot write under a project’s key. When the job ends, and about every 15 minutes while it runs, the plane is designed to move the lines into the job-log folder of the project’s environment in the data bucket, under the project’s key, and delete the originals.
In the design, two things follow, and you should know both: a running job’s most recent lines can sit under the plane’s key for about 15 minutes, and a little longer when a move runs behind; and if a move fails, those lines stay under the plane’s key until they expire, which the design puts at about 3 days after they were written, and the system bucket’s lifecycle rule that removes them runs once a day, so allow a day more. Either way the lines are designed to be encrypted at rest. What differs is which key would have to be disabled: the project’s key for lines that have moved, and for the rest the plane’s key, which also encrypts the plane’s disks, its catalog database and its system bucket, and is the default key of its data bucket where the plane’s setup created it, so disabling it would break the plane. On a plane in your own account you could disable either; on a plane we host both keys are ours, and neither is yours to disable.
Lines that have moved are designed to be under the project’s key. In the design, disabling the project’s key makes them unreadable, though S3 can go on reading, for a time that has not been measured, with a key derived from it that S3 has already cached. The role the control plane assumes in the plane’s account can enable a disabled key again, and can cancel a scheduled deletion until its waiting period, 30 days unless a shorter one is set, has passed. After that, in the design, the lines are unrecoverable. The same role can also rewrite a key’s policy, so a denial of those two acts would not hold on its own, and add grants on the key. A key you registered yourself would change this only if its policy denied that role all five: enabling the key, cancelling its deletion, rewriting its policy, adding grants and scheduling its deletion. Registering your own encryption key is built in the console and is not switched on yet. In the design, neither a disable nor a destroy of the project’s key reaches lines that are still waiting to be moved. The data plane page says where keys stand.
What reaches the control plane
The control plane is the half of RunningLake that holds your configuration and your audit log. Your plane is designed to send it identifiers, numbers and a few closed words: which job and which stream, how many lines, segments and bytes, the number of the first line the plane marks as an error, how many lines were dropped, what state the stream is in and when its lines finished moving, and a few more counts of the same kind. In the design, the messages your plane sends about a job’s logs are shaped so that a line does not fit in them, and the control plane checks the shape of each field; but a shape limits form and not meaning, which makes a line reaching the control plane in one of those messages unlikely and does not prove it impossible.
For a search, and for a log rule, the plane is also designed to report the line numbers and counts of the matches, never the matched text. A log rule is a pattern you declare; in the design the control plane sends a notice when one appears in a log, with a count and a line number and never a line. A count can still tell someone whether a value appeared in a log. Your plane is not designed to send a query’s execution profile or an engine’s event record.
The one route by which whole lines are designed to reach the control plane is a test of a log rule: the lines you paste are sent to it, and it is designed to match them and not store them, though on a plane we host the logs are in our account.
In the other direction the control plane is designed to send your plane identifiers, the number of days to keep a log, and what you type to search a log, which it is designed not to store. In the design the text travels in the body of a request and not in an address, and the control plane logs nothing of it. Your audit log is designed to record a search by a digest of its text and its length, not the text. The links the control plane is designed to sign for a job log go to your browser, and, for a search of a whole log, to your plane.
These messages, and the commands for a search or a retention period, are designed to travel on the connection your plane opens to the control plane. On our QA deployment, which is the one that exists, that connection is not encrypted in transit, and encrypting it and authenticating both ends of it are not built yet. In the design, signed links and search text go down it, so the design depends on that protection. The security page says what protects it.
No staff screen and no staff call is designed to read a job log. That is not a promise that nobody on our staff can reach one. The control plane holds a role in your plane’s account that can read and write its buckets, including what is under a project’s key, and a staff member can mint an owner’s token into your organisation with a command on the control plane’s host, as the security page says. On a plane we host, the plane’s buckets and keys are ours.
How you read a log, and where a line can go next
In the design, you read a log in the console. No plane keeps job logs yet. Today there is nothing to read, and the console says the plane keeps no job logs.
In the design, your browser fetches each segment straight from the bucket that holds it, through a link that RunningLake’s control plane is designed to create and that is designed to work for about five minutes. In the design the link names one segment of one stream; there is one more for the stream’s index, which holds its counts, and one for downloading the whole stream. The log is designed not to pass through the control plane or be stored there, though on a plane we host the bucket it is read from is ours. On a plane in your own account that bucket is yours. The bucket needs a cross-origin rule that lets the console, in your browser, read it with a signed link. On AWS the plane’s setup adds one to a bucket it creates, and the rule it adds admits any origin, not only the console’s address, because the signed link, and not the page that asks, is what authorises a read. A signed link works for whoever holds it until it expires, from any program and not only from a browser. If you gave the plane a bucket that already existed, you add a rule yourself. Without one, the console is designed to say so.
A line is designed to go to your screen, your clipboard or your disk. In the design, a link you
share to a line carries the stream and the line number, never the line. In the design, two
routes take a line further, and each is a choice you make. In the design, lines you paste to
test a log rule are sent to the control plane, as the section above says. And in the design a
project can allow a bounded excerpt around a failed task’s first error to be sent to a model
provider, through the logExcerpts setting of the AI
policy. The setting is accepted today, is off by default and needs an owner or an admin to
approve. In the design the plane reads the excerpt and masks it before it goes, and the excerpt
goes from the plane and not through the control plane, though on a plane we host the plane that
reads and sends it is ours. In the design the masking replaces the personal data it recognises
but not a person’s name in ordinary text, so it does not guarantee that nothing identifying is
left. The masking is not built, and in the design no excerpt is sent until it is. In the design
only a model call that starts in the plane carries an excerpt, and the calls made under
the trial carry none. No model is called yet. No excerpt is read or sent
under that setting yet.
Who may read
Who may ask about a query’s logs is built. Only the person who ran a query can ask what logs it has, by the rule that already decides who may read the query’s result. A colleague is told what anyone is told about a query that does not exist: not found. That is only about the logs. The query’s own record, with who submitted it, is listed to every role that may read runs. Today the answer for the person who ran it is that the plane keeps no job logs, so nothing is delivered to anyone.
In the design, a run’s logs go to anyone who may read the run, as long as nothing in the run’s environment is hidden from them; the permissions page says who may read a run. A layer is a named area of your lake, such as bronze or gold. One marked restricted is left out of the control plane’s lists, lookups and counts of layers, schemas, tables, views and volumes for anyone its grants do not name. That hides listings, not data: the plane’s catalog does not apply it, so an engine that reaches the catalog can still list the layer, and the roles the plane reads your files with, the project’s and the catalog’s, can read the layer’s files. A log can repeat values from tables, so in the design a person from whom any layer of the environment is hidden in that way may not read any run’s log there. An environment that has declared no restricted layer hides nothing from anyone, so in the design whoever may read a run there may read its logs.
In the design, that rule looks at layers, not at the columns and rows a reader’s own grants allow. So in the design, where no layer is hidden from a reader, a run’s log can show a value that the reader’s grants on that table would refuse. The permissions page does not list that exception, because today there is no run log to read.
A run is designed to execute as its pipeline, not as the person who started it, so starting a run grants no right to read its log. In the design, a read that the control plane cannot decide is refused. Nothing runs a pipeline yet. Reading a run’s logs is not built yet.
How long they are kept
A project can declare how long it wants job logs kept, with a JobLogPolicy, a file in its configuration repository: 7, 30, 90 or 365 days, and 90 when it says nothing. An environment may set its own period. Through a change set, making the period longer needs an owner or an admin to approve it, even in an environment that otherwise applies changes straight away; making it shorter in the project’s own file does not. An environment’s own period is judged without sight of the project’s, so setting one for the first time asks for approval unless it is 7 days. A project of an organisation on the trial is refused 365 days when the change is planned. The JobLogPolicy reference has the fields.
Nothing applies that period yet; until it does, logs that have moved are not removed. In the design, a log has moved once the plane has written its lines under the project’s key in the data bucket; the Two keys section says how.
A policy you declare today shows as Degraded on purpose in
rl status and in the console. That is expected, not a
fault: the declaration is recorded and checked, and nothing applies it.
In the design, a period is the longest a log is meant to be kept, not the least, and removal is not instant: allow about a day more. In the design lines that fail to move expire after about 3 days, plus up to a day, whatever the period; a longer period reaches only logs that move after the change, and a shorter one reaches older logs too.
Erasing logs
In the design, an owner or an admin with a fresh sign-in can delete one run’s logs that have moved into the project’s data bucket. The deletion is designed to be recorded in your audit log: who deleted which run’s logs, when, and the reason the person types, in their own words. The deletion itself is designed to copy no log line into the entry, but a reason is free text, and the audit log is append-only, so it is kept as typed. Deleting one run’s logs on request is not built yet. In the design there is no call to delete a query’s logs on request; once they have moved, they are removed after the retention period ends, and become unrecoverable if the project’s key is destroyed.
Deleting a project schedules its key for deletion, once the project has been taken down on the plane, if it was ever applied to one. That key also protects the project’s other files in the bucket where the plane keeps the project’s data, so they become unreadable along with its logs, though S3 can go on reading for a time that has not been measured, as the Two keys section says of a disabled key. They become unrecoverable once the key’s waiting period has passed (30 days unless a shorter one is set), unless the role the control plane assumes in the plane’s account cancels the deletion before then. The files are not deleted. A deletion is refused on a plane whose agent does not report that it can remove a project’s namespace, which is every plane today, so none has been carried out.
More that is not built
Four more parts of the design have a call declared. For a query, the control plane can answer the call to follow a running log from the lines its plane’s collector writes, as far behind as the collector’s flush; for a task of a run it is the first item below. The sections above say where the rest stands.
- Following a running task’s log is not built yet.
- Searching a whole log is not built yet.
- Per-run figures beside the logs, such as peak memory and bytes scanned, are not built yet. A query’s own record already carries its CPU, memory and bytes scanned.
- Log rules, which act when a pattern appears in a log, are not built yet.