BUILDING Parts of this page describe what is not built yet, and each says so.
Lineage
Lineage is the part of Governance that says where an asset’s data comes from and where it goes. It works on tables and views, at table level, and every edge it draws is one that your project declares in git or one that the control plane read from your SQL. Each edge says which, and the explorer says what it did not read.
What an edge is
An edge joins two assets of one environment, tables or views, and points from the one that feeds to the one that is fed. The control plane keeps the edges itself, by id. It works the declared ones out from your project’s declarations when a change is applied, and reads your SQL for the parsed ones. Nothing is sent up from your data plane for lineage, and nothing is read from your data.
An edge says how it is known and which file holds what gives it. A change that removes or moves a declaration, or that edits a statement, ends its edge, and an edge that has ended is not drawn.
An environment whose edges have not been worked out is said to be not computed, in words, and never drawn as empty: an empty graph would say nothing is connected, and nobody has looked. That is true of an environment whose change is held for review until it is approved, and, briefly, of one that existed before lineage was recorded, until the next sync works it out; it needs no change to be applied.
How an edge is known
Every edge carries one of three marks, in words and in line style, so that colour is never the
only thing that says it. declared is a solid line: a person listed it in git.
parsed is a dashed line: the control plane read it from a statement’s SQL.
observed is a dotted line: it was seen in a run.
The control plane says, with every answer, whether it records each mark, and the explorer’s
legend and rl lineage print that. Declared is always “Recorded”. Parsed is
“Recorded” when the last pass over this environment read SQL, all of it or as much as one pass
takes; an asset whose SQL is new, changed or retried says so itself. It is “Not recorded yet”
before a pass has read any, and “Could not run” when the control plane’s SQL parser could not be
used for SQL that is new or has changed, to those it is told to, as the section on what is parsed says.
Observed is not recorded. A client never says so on its own authority.
What your project declares
Two things in git give a declared edge, and nothing else does.
- A view’s reads. A view lists the tables and views it reads. Each one that your environment declares is an edge from it to the view. The view’s author lists them, and nothing checks the list against the view’s SQL: the explorer says so on the edge.
- A pipeline’s order. When an asset of a pipeline depends on another asset of the same pipeline, the explorer draws an edge from the table the first writes to the table the second writes. The edge says that the pipeline runs one after the other, and never that either reads the other. The pipeline is how the edge is known and is not an asset of its own, so the explorer does not say which pipeline writes a table.
A pipeline declares no read of another table. What its SQL reads is not declared and is never called so: it is known only by reading the SQL, and a table it reads that way is a parsed edge, below.
What is parsed
The control plane reads two kinds of SQL: a view’s SQL, and the SQL a pipeline’s asset holds inline. For each table or view your environment declares that a statement reads, it draws an edge from that table or view to the table the asset writes, or to the view. A statement that reads two tables gives two edges.
A parsed edge says whose SQL read it, the pipeline or the view, and which file holds it. It never says which commit it was read at, and it never shows the SQL. A pair of assets can hold a declared edge and a parsed one: they are two edges, drawn as two lines and listed as two rows, because they are known in two ways, and the one hop on a record says both.
A statement is read as the environment’s last synced revision holds it. An environment held for review, or one whose first sync has not run, is not read. When a statement changes, when an address it read moves, is removed or is replaced, or when a table now exists at an address it could not resolve, its parsed edges end at once and are drawn again when the statement has been read, so no edge is one an older statement gave. Until then the asset that statement writes says “Not read yet”.
Some SQL is not read, and the explorer says so of the asset that SQL writes, in words, and never as a statement that reads nothing. A statement that is new or changed and waits is “Not read yet”. One the parser refused, because it is outside the dialect the control plane reads or too large, is “Could not be parsed”. SQL a pipeline names in a file is “Kept in a file, not read”: the control plane holds no file text, so it does not read a statement kept in a file. What an asset whose SQL is not read reads is not shown, and the explorer says how many of the assets in the graph it holds are like that, counting only the assets it shows. An asset whose own SQL is not read says at least for what it reads upstream, because its SQL may read more than the edges show. Where it says that no column is traced, and in every answer that holds no edge and names the parsed mark, the explorer also says that SQL kept in a file is not read, so what it reads is not shown.
A statement can read a table your environment does not declare, such as a source. That is not an edge, and it is not counted. What a statement reads that nothing declares is not shown.
When the parser could not be used for SQL that is new or has changed, the parsed mark reads “Could not run” and the explorer says that parsed edges may be missing; with every statement already read, stopping the parser changes nothing a reader can see. A statement loses its parsed edges until it is read again when its SQL changes, when an address it read moves, is removed or is replaced, and when a table now exists at an address it could not resolve. While the parser cannot be used they are not redrawn, and the mark says Could not run to a viewer who can see that SQL. The mark is told by its cause: a control plane with no parser, and one whose parser is too old for this release, read “Could not run” to everyone; a parser that gave no answer, and a store that refused the write, read it only to someone who can see SQL that is new or changed and was waiting when the attempt failed, and everyone else reads “Recorded”, so the mark never tells a reader that SQL they may not see was edited. The declared edges are drawn as they always are, and the graph is never empty because of it.
A parsed edge adds context and decides nothing: no change waits for review because of one, and no access depends on one. Impact lists what a parsed edge reaches as it lists what a declared one does, and does not say which kind of edge brought an asset into the list.
Reading it
Lineage is an entry in the Governance group. It reads the project and environment named at the foot of the sidebar. A record’s Lineage tab, for a table and for a view, and the one hop on its Overview and in the Data lake read the same answer at depth one and link to the explorer. With no asset in the address the explorer makes no call: it asks you to choose an asset and offers a search by name over the tables and views you may see.
The address holds the asset, the direction and the depth, and nothing else, so a link opens the same view. You choose upstream, downstream or both, and a depth from 1 to 10, and 2 when you choose none. A depth outside 1 to 10 is refused by name and never cut to fit.
The graph has one tab stop. The arrow keys move between assets, Enter shows the impact of the one you are on, and Escape clears it. Show as list gives the same edges as a table, one row for each edge and mark, with how each is known and the file that holds it. The list is the form that holds every edge: a picture can leave one out, a cycle among them.
The control plane cuts an answer at 100 assets and says so, with how many it holds, never with a total: a total would be wrong the moment part of the answer is hidden. Narrow it by direction or depth. There is no page two, because a page of a graph is not a graph.
What you may see
Lineage is read with lineage.read, which every role holds, a viewer included, and it reads like the Data assets list: what your credential may see. An API token can be made narrower than the person who made it, and a token without lineage.read is refused, with the permission named.
An asset you may not see is not in the answer, and neither is any edge to it, a parsed one included. The walk runs over what you may see only, so an asset that is reached only through a hidden one is not reached. The asset next to a cut says “Some lineage is not visible to you.” and nothing else: no count, no name, no kind and no direction. A request for an asset that is hidden from you, one that does not exist and one in another environment get the same answer. What the SQL of an asset you may not see reads is not said, and no figure of statements is sent.
One limit follows, and it is not fixed. When the SQL of an asset you may not see changes, the parsed edge it gave to an asset you may see ends, and it is drawn again only once the parser has read the new SQL. While the parser cannot be used you see neither that edge nor the “Some lineage is not visible to you.” note it would have raised, and nothing says why: saying so would tell you that SQL you may not see was edited.
Impact
Choose an asset in the graph and Impact lists every asset downstream of it, at any depth, by kind, with each one’s owner, steward and whether it is certified. The depth you drew does not limit it. When part of what depends on the asset is not visible to you, Impact says so, counts only what it holds, and reads at least. Impact only reads; it decides nothing.
A certified table’s breaking change
A change that removes, renames or retypes a column a certified table declares in its own file, deletes that file, or changes the namespace or the name the file declares, waits for approval before it lands, and the change set’s page shows what is downstream of the table. Adding a column to the table’s own file does not wait, except while the control plane cannot tell whether the table is certified, when it waits too. A column added inside a nested type reads as that column retyped, and a column renamed in a file that gives its columns no ids reads as removed. An environment’s own patch of the table waits when it changes the table’s schema, namespace or name in any way and the table is certified in that environment. A commit pushed to your repository is not a change set, so no approval reason holds it: an environment that does not review projects it without an approval, and one that reviews holds what the branch brings until a person approves that transition. Certification says which tables count, which environment’s mark decides, and who approves it.
Under the diff, Impact asks for each table the change set holds what the explorer would say, as you: an asset you may not see is not in it. It says how many assets the table has downstream, and says at least when part of what depends on it is not visible to you, lists them by kind with each one’s owner, steward and certification, and then says what the explorer says of parsing, from the same answer: that the SQL was not read, that the parser could not run, that no column is traced, and that SQL kept in a file is not read. It does not say which change the table is held for: that is the reason in the change set’s rules. An environment’s patch of a table is read in that environment, and the section says so unless it is the environment you are in; a table of another environment than the one you are in has no link into the explorer, which reads the one you are in. It reads at most five tables and says how many more the change set names. When Impact cannot be read, or the table is not one you may see, the section says so in one sentence, and says that the change still waits for an approver only while the change set is proposed: a draft, a change set that has been approved and has its pull request open, and one that has ended say nothing of waiting. It decides nothing: the reason in the change set’s rules is what holds the change.
From the command line
rl lineage prints the edges around an asset, with how each is known, and
rl lineage impact lists what is downstream of it. Both name the asset the way
rl asset does, take -project and -env as it does, and
take -o json for the answer the console reads. rl lineage takes
-direction and -depth, and prints the direction and depth the
control plane applied. A parsed edge’s evidence names the file that holds its statement and
the pipeline or view whose SQL read it. Each prints which marks the control plane records,
from the answer, Could not run among them, and says not reported
when an answer carries none. The
command-line reference lists each command, the permission it
needs and the exit codes.
What is not built
These are the things you might expect of lineage that it does not do.
- A statement kept in a file is not read yet, so what it reads is not shown.
- A parsed edge names the file of its statement and not the commit it was read at yet.
- Observed lineage is not recorded yet.
- When a change that removes, renames or retypes a column of a certified table merges, its certification does not fall due yet.
- A change to a top-level column’s required flag or defaults, to a column’s or a table’s doc, to a table’s labels, annotations, identifier fields, schema id, column order, partitions, properties, write order, routing format, routing engine or routing consumers, the adding of column ids to a file that had none or the dropping of them from one that had them, and the creation of a table, are not escalated for review yet.
- The reason a change set gives counts the columns a change removes, renames or retypes and does not name them yet.