BUILDING Parts of this page describe what is not built yet, and each says so.

JDBC extraction

JDBC extraction is the source type jdbc, which the console calls database table (JDBC). A plane is meant to read tables from a database on a schedule, in full or by a cursor column, and append them to bronze. What you can do with it today is declare it and prepare the database for it. Reading a table with the jdbc type is not built yet. Sources says what works and how to declare one. A source that follows every change as it happens, and does not read on a schedule, is a change capture source: PostgreSQL change capture or MySQL change capture.

Declare a JDBC source

Save this as sources/warehouse-db.yaml:

apiVersion: runninglake.io/v1alpha1
kind: Source
metadata:
  name: warehouse-db
spec:
  type: jdbc
  engine: postgres
  host: warehouse-db.internal.example.com
  port: 5432
  database: sales
  credentials:
    user: rl_reader
    secretRef: warehouse-db-reader
  read:
    tables:
      - public.orders
      - public.customers
  land:
    schema: bronze.warehouse_db
  mode: batch
  incremental:
    cursor: updated_at
    lookback: 10m
  schedule:
    cron: "0 * * * *"

type is jdbc, and mode is batch, the only mode this type takes.
The jdbc type names its engine, postgres or mysql. Any other engine, SQL Server and Oracle included, is refused by the rule source-type-field. The engine decides the language of the grants below.
host, port and database are required for this type. credentials.user is the database user the source is meant to be read with, held only to the shape of a user name, and credentials.secretRef names a SecretReference.
read.tables lists the tables to read, at most 256. A pattern such as public.* is refused: every name is one you typed. Write each table with its schema in front, or its database for MySQL, as in public.orders: the steps for your database need both parts.
land.schema is the declared schema the tables land in. Each name lands in a table of that schema called by the last part of the name in lower case, so public.orders lands in bronze.warehouse_db.orders. What lands, and what you declare says what to declare beside the source, and what rl diff refuses in a repository that declares layers.
incremental and schedule are the next two sections. Both are optional.

A jdbc source can also carry drift, what the source is meant to do when its own schema changes. Nothing detects a schema change yet, so today this records what the source is meant to do when one is detected. The Add a source form in the console plans a file of this kind.

How it reads

A jdbc source is read one of two ways. The setting is for the whole source, every table in it.

The whole table each time is the default. It is meant to copy every row of each table on every sync and append the copy to bronze under a new sync id, so bronze stays append-only.
Only the rows past a cursor column is incremental. It is meant to read the rows whose cursor column is above the largest value already in bronze, less the lookback. The cursor’s value is meant to stay in the plane. A read by cursor is not meant to see rows deleted in your database; use change capture or a full read for them.

incremental.cursor is the column an incremental read orders by, and every table in the source is meant to have it. It is required when incremental is set, and only the jdbc type may set it. incremental.lookback is how far behind the cursor’s largest value to read again, to catch rows committed out of order, such as 10m or 2h; leave it out for none. A lookback is a number and a unit, such as 10m, 2h, 90s or 1h30m; the plane is meant to count whole seconds. One that is not a length of time is a warning and not a refusal: the finding says it is not a length of time, such as 10m, 2h or 90s.

The schedule

schedule has the form of a pipeline’s schedule, a cron expression and optionally a time zone, and the control plane checks it when you plan the change. A Source differs from a pipeline in three ways: it takes only the overlap skip, so a sync that is still running makes the next tick skipped; it syncs one at a time, so maxConcurrentRuns above 1 is refused; and it has no queue, so maxQueuedRuns is refused. Leave schedule out for a source that is synced only when it is started.

The control plane records the schedule. Nothing syncs a jdbc source yet: a schedule in the file is recorded, and nothing starts a sync from it. The control plane has a scheduler that is built to open a sync when a schedule comes due, and it is switched off in every deployment today. A person can also ask for a sync by hand with Sync now on the source’s page in the console, and the control plane refuses it when the plane bound to the environment does not report that it can run a source’s sync. No plane reports that yet.

What your database needs

rl source prereqs writes two steps for a jdbc source, in the order you do them. The SQL of the second is written in the language of the engine the file names:

The steps for a jdbc source
Step	What it makes true
`tcp`	The plane must be able to open a connection to the host on the declared port. This step has no SQL: it is for your network.
`table_select`	The login may read every table the source reads.

The steps create nothing: the user is yours. A read needs SELECT on each table and, for PostgreSQL, USAGE on the table’s schema. The second step is one grant of SELECT for each table, and for PostgreSQL one grant of USAGE for each schema before them:

The grant for each engine
Engine	You write	The step writes, for the user `rl_reader`
`postgres`	`public.orders`	`GRANT USAGE ON SCHEMA "public" TO "rl_reader"` and `GRANT SELECT ON "public"."orders" TO "rl_reader"`
`mysql`	`sales.orders`	GRANT SELECT ON `sales`.`orders` TO 'rl_reader'@'%', with a comment that the `%` lets the user connect from any host: name the plane’s address instead if you restrict hosts.

No connection test is built for this type yet, so nothing has checked that a plane can reach this database or read these tables.

The credential

The secret stays in your store. This source names the reference and holds no value; a plane is meant to read the value from your store when it connects, and nothing connects yet. Secrets explains how to declare a reference and how to check that your plane can find the secret.

Remove a source

Remove a source as you remove any file: delete sources/warehouse-db.yaml, then plan and apply the change. Today that retires the source and removes the state the control plane recorded for it. It does not touch your database. The tables declared for the source are files of their own, and they stay unless the same change deletes them.

Not built yet

Reading a table with the jdbc type is not built yet.
Nothing syncs a jdbc source yet: a schedule in the file is recorded, and nothing starts a sync from it.
No connection test is built for this type yet, so nothing has checked that a plane can reach this database or read these tables.
Nothing detects a schema change yet, so today this records what the source is meant to do when one is detected.