Skip to content
RunningLake
Company Console ↗ Request early access
Contents: Data in

BUILDING Parts of this page describe what is not built yet, and each says so.

Sources

A Source is a database or another system that RunningLake is meant to read from and land in your lake as tables. You declare one as a file in your project’s repository, like a table or a pipeline, and it is planned, reviewed and applied as any other change is.

This page says what a Source does today, which is less than what it is for. A Source can be declared, planned in a change set from the console form or with rl, applied, and then listed as pending. The control plane records its state. Nothing connects to your database, tests a connection or captures a change yet. Four of the types have a page of their own: PostgreSQL change capture, MySQL change capture, JDBC extraction and object storage sync.

Declare a source

Save this as sources/orders-db.yaml:

apiVersion: runninglake.io/v1alpha1
kind: Source
metadata:
  name: orders-db
spec:
  type: postgres_cdc
  host: orders-db.internal.example.com
  port: 5432
  database: orders
  credentials:
    user: rl_replication
    secretRef: orders-db-reader
  read:
    tables:
      - public.orders
      - public.customers
  land:
    schema: bronze.orders_db
  mode: continuous
  • type is the kind of source: postgres_cdc, mysql_cdc, jdbc, api, object_storage or stream. This page uses postgres_cdc.
  • host, port and database say where your database is.
  • engine is the database a jdbc source reads. The jdbc type names its engine, postgres or mysql. Any other engine, SQL Server and Oracle included, is refused by the rule source-type-field.
  • credentials.user is the database user the source is meant to be read with. It is a role name, held only to the shape of a user name.
  • credentials.secretRef is the name of a SecretReference. See Credentials.
  • read.tables lists the tables to read, each written schema.table, except public.rl_heartbeat, which is the heartbeat table and is refused. A pattern such as public.* is refused: every name is one you typed.
  • land.schema is the declared schema the tables land in. Land it in the bronze layer: rl diff does not check which layer it is in.
  • mode is continuous for change capture and batch for the types that read on a schedule, and it has to agree with the type.

The Source kind is the generated reference, with every field and the schema to point your editor at. rl diff checks the file offline, with the rules the control plane runs when it plans a change, and the change is planned and reviewed as Review and approval describes. Planning a change that declares a Source takes the source.connect permission, which owners, admins and engineers hold.

The reference also lists fields for features that are not built: apply, schedule, incremental, drift and network are checked and recorded, and nothing acts on them.

For a Source, rl diff also prints a warning, kind-not-built. It is a warning and not an error, and it says: “A Source says where data is brought in from. This build records the declaration and nothing acts on it yet. No ingestion is configured from it, so nothing is loaded. The file is valid and is kept as written.”

What lands, and what you declare

A Source is meant to land in bronze. For each name in read.tables you declare one table in the schema land.schema names, called by the last part of the name in lower case: public.orders lands in bronze.orders_db.orders. In a repository that declares layers, rl diff refuses a name that cannot be a table’s name and refuses two names that would make one table, and it never rewrites either (the rule source-landing-name). A repository that declares no layer is not held to this rule. Declare the schema and the tables in the same change as the source. If your repository declares layers, rl diff refuses a Source until they are declared, and it names the file to write for each (the rules undeclared-schema and source-landing-table). The Add a source form in the console plans the source and its tables as one change set, with the schema when it is new and your environment declares layers, and you review it in Changes.

A landing table declares no columns of its own. When the change is applied, the control plane creates each table with eight columns the platform adds, _rl_op, _rl_pos, _rl_tx, _rl_commit_ts, _rl_seq, _rl_sync_id, _rl_ingested_at and _rl_rescued, and no column of your source’s. A table that already exists is not given them, so declare a source’s tables in the change that declares the source.

In a repository that declares layers, rl diff refuses a landing table that declares columns (the rule landing-table-shape). In one that declares none, nothing refuses it, and a table that declares columns is created as it declares itself, without the eight. The columns of your source are meant to be discovered and not declared: the plane is meant to add them as it finds them, without their names passing through the control plane. Nothing adds the columns of your source to its tables yet.

Credentials

A Source has no field for a password. credentials.secretRef names a SecretReference, a file that says where the password lives in your own store; the password stays there, and RunningLake keeps the name. rl diff refuses a Source whose reference no file declares (the rule secret-reference), and a password written where the name goes is refused, because no SecretReference has that name. A push made straight to your repository is not scanned for secrets, and credentials.user is held only to the shape of a user name, so keep every password out of every file. Secrets explains how to declare a reference and how to check that your plane can find the secret. credentials.user is plain configuration, because a user name is a role and not a secret.

Nothing resolves the secret reference a Source names yet.

Prepare your database

A database has to be set up before it can be read this way. The control plane writes the steps, and the SQL in them, from your declaration, so the console and the command line print one text, and you run it yourself on your own database, with your own administrator. The console shows the steps on the form that adds a source, once the name, the database user and the tables are typed, and on the Settings tab of a saved source, with the SQL as the control plane wrote it and a Copy button on each step that has SQL. The console always writes the steps for a database you run yourself: for RDS, Aurora or Cloud SQL, ask for them on the command line with -flavour. The console can ask the control plane to test your connection, and today no plane can answer it; it ticks no step: each step is yours to do, and nothing checks that you did. On the command line, for the file above: 

rl diff
rl source prereqs -project payments -env dev -f sources/orders-db.yaml
rl source prereqs -project payments -env dev -f sources/orders-db.yaml -sql

The first command checks your working tree offline, the file included. It needs the SecretReference the file names to be declared in the same tree and, if your repository declares layers, the schema and tables the file lands in. The second prints the steps, numbered, each with the check that would verify it, and says which of them need the database to restart. The third prints only the SQL, so it can be piped to your database’s client. For this file it is: 

ALTER SYSTEM SET wal_level = 'logical';
-- Then restart PostgreSQL. Check afterwards:
SHOW wal_level;

CREATE ROLE "rl_replication" WITH LOGIN REPLICATION;
-- The password is chosen in your own store and set there. It is not written here.

GRANT USAGE ON SCHEMA "public" TO "rl_replication";
GRANT SELECT ON "public"."orders", "public"."customers" TO "rl_replication";

-- The plane is meant to update this one small table, in the publication below, so that the database keeps moving its change log forward when nothing else is writing. No plane does yet.
-- Check that no table of this name exists before you run this: a table that already has this name and other columns is left as it is, and the grants below still apply to it.
CREATE TABLE IF NOT EXISTS "public"."rl_heartbeat" (
  "id" integer PRIMARY KEY,
  "beat_at" timestamptz NOT NULL
);
INSERT INTO "public"."rl_heartbeat" ("id", "beat_at") VALUES (1, now()) ON CONFLICT ("id") DO NOTHING;
GRANT USAGE ON SCHEMA "public" TO "rl_replication";
GRANT SELECT, INSERT, UPDATE ON "public"."rl_heartbeat" TO "rl_replication";

CREATE PUBLICATION "rl_orders_db" FOR TABLE "public"."orders", "public"."customers", "public"."rl_heartbeat";
-- A publication of this name that already exists, made before the heartbeat table, takes it with:
-- ALTER PUBLICATION "rl_orders_db" ADD TABLE "public"."rl_heartbeat";

DO $rl$
DECLARE t regclass;
BEGIN
  FOR t IN
    SELECT c.oid::regclass FROM pg_class c
    WHERE c.oid IN ('"public"."orders"'::regclass, '"public"."customers"'::regclass)
      AND NOT EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = c.oid AND i.indisprimary)
  LOOP
    EXECUTE format('ALTER TABLE %s REPLICA IDENTITY FULL', t);
  END LOOP;
END
$rl$;

-- The plane is meant to create the replication slot rl_orders_db on its first start. No plane does yet.
-- It needs one free slot and one free WAL sender. Check how many are free:
SELECT current_setting('max_replication_slots')::int - count(*) AS free_slots FROM pg_replication_slots;
SELECT current_setting('max_wal_senders')::int - count(*) AS free_senders FROM pg_stat_replication;

Read it before you run it. It changes wal_level, which takes a restart. It makes a role that can log in and stream changes, with no password: choose one in your own store and set it there. It gives the role USAGE on their schemas and on public and SELECT on exactly the tables the file names, makes one small table of one row, public.rl_heartbeat (the plane is meant to update it so that the database keeps moving its change log forward when nothing else is writing; no plane does yet), gives the role SELECT, INSERT and UPDATE on it and no write on any other table, makes a publication of the tables the file names and that one, gives REPLICA IDENTITY FULL to a table with no primary key so that a change can name its row (that makes PostgreSQL write the whole old row to the log on each update and delete, and its documentation calls it a fallback for a table with no suitable key), and ends by counting the free replication slots and WAL senders. That identity does not make a key: a table with no primary key can still hold identical rows, and the PostgreSQL change capture page says what that can cost the first copy and what the connection test is meant to warn about. Every name is quoted as PostgreSQL quotes an identifier, and nothing in it carries a password. If you run it with psql, use -v ON_ERROR_STOP=1, so that it stops at the first statement that fails; and check first that no table called public.rl_heartbeat exists, because the statement that makes it leaves a table of that name, with other columns, as it is.

A note for a security reviewer: the role is created to log in and has no password, so password authentication fails for it until you set one in your own store. If an entry in your pg_hba.conf uses the trust method for the address the plane connects from, anyone who can connect from that address is let in without a password. Set the password, and use one hostssl entry (the server must have ssl on) with scram-sha-256, naming the plane’s address, this database and this role, above any broader entry: PostgreSQL uses the first record that matches.

With -f, rl sends the control plane only the source’s name, type, database user, tables and, for the jdbc type, its engine, never its host, port, database or credential reference. A source that is already saved is named instead of given as a file: rl source prereqs orders-db. The steps differ with the kind of database, and -flavour takes self_managed (the default), rds, aurora or cloud_sql; PostgreSQL change capture and MySQL change capture say what changes.

Read what is recorded

Three commands read what the control plane records about the sources of an environment. They need config.read, which every role holds, and they print JSON with -o json. 

rl source list -project payments -env dev
rl source get -project payments -env dev orders-db
rl source kinds -project payments -env dev

rl source list shows each source’s type, state, lag, last sync and where it lands. rl source get shows one: its state, with a sentence that says what the state means, the name of its credential reference, and its slot. rl source kinds shows the types of source and which of them the control plane lists as available in your environment (today PostgreSQL change capture, the jdbc type, object storage and MySQL change capture; the control plane records a Source of any of the four and no plane reads one yet, and none on the shared trial plane), with the control plane’s own sentence for each it does not: “This kind of source is not built yet.” A figure the plane has not reported is a dash and never a zero. In the console, Sources shows the same, and Add a source opens a choice of types and, for PostgreSQL change capture, MySQL change capture, the jdbc type and object storage, the form that plans the change. Sources are not offered on the shared trial plane.

For all four, the control plane records the declaration, and no plane reads, connects or captures yet.

A Source reads pending, or paused when its file says paused, and nothing reports a state beyond those yet. Pending means only that the source is applied and nothing has been reported for it.

Test a connection

A connection test cannot run yet: the console asks the control plane for one, the control plane asks a plane that says it can test, and no plane says so. The form in the console says Not tested beside Plan until a test of that draft has run, and no rl command tests a connection. A test is designed to run from your plane and not from RunningLake, and to report a verdict and a class of failure and never a database’s own message, because that can quote a table name or a value.

Not built yet

  • MySQL, SQL Server and Oracle are not read yet.
  • Reading tables on a schedule, with the jdbc type, is not built yet.
  • API connectors are not built yet.
  • Object storage sync is not built yet.
  • Schema changes are not detected yet.
  • Streams need the Flink engine, which no plane runs yet.

Only four of the six types have a form in the console: PostgreSQL change capture, the jdbc type, MySQL change capture and object storage. What a form plans is a declaration.