BUILDING Parts of this page describe what is not built yet, and each says so.

Object storage sync

Object storage sync is the source type object_storage. A plane is meant to copy files from a storage location or an SFTP drop into a volume on a schedule. What you can do with it today is declare it. Copying a file is not built yet. Sources says what works and how to declare one. What a landed file is meant to start is declared in Storage rules.

Declare an object storage source

Save this as sources/partner-drop.yaml. The host key fingerprint in it is a placeholder: replace it with your server’s fingerprint.

apiVersion: runninglake.io/v1alpha1
kind: Source
metadata:
  name: partner-drop
spec:
  type: object_storage
  url: "sftp://drop.partner.example.com/outbound/"
  hostKeyFingerprint: "SHA256:AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8"
  credentials:
    secretRef: partner-drop-key
  filePattern: "*.csv"
  read:
    objects:
      - "claims/"
      - "returns/"
  land:
    volume: bronze.partner.inbound
  mode: batch
  schedule:
    cron: "0 2 * * *"

• type is object_storage, and mode is batch, the only mode this type takes. This type has no host, port, database or credentials.user: it connects through url.
• url is the address of the storage location or the SFTP drop. It carries no password, and a user name alone may be in it.
• hostKeyFingerprint pins the SFTP server’s host key, and only an sftp address has one.
• credentials.secretRef names a SecretReference. The gate asks for one whatever the address is.
• read.objects lists the prefixes to read, at most 256. Each is written relative to url, such as claims/, and names a folder: claims and claims/ are the same one. A wildcard does not go in a prefix: put it in filePattern.
• filePattern says which files under each prefix are meant to be copied. It is optional.
• land.volume is the declared volume the files land in, written layer.schema.volume. A volume is declared in a file of its own, and this source does not declare it.
• schedule is optional. See The schedule.

The address and a prefix are joined with one slash and nothing is decoded: url s3://partner-storage/exports/ and prefix daily/ name s3://partner-storage/exports/daily. To copy the folder url names, end url one folder higher and name that folder as the prefix.

The address and the host key

The schemes a plane is meant to copy from are s3, gs, abfs, abfss and sftp, in lower case, then :// and a name. Another scheme is accepted with a warning, and nothing here checks that it can be read.

Host key. The server’s host key fingerprint, as ssh-keygen -l prints it: SHA256: and 43 characters. Check it with whoever runs the server.

Copy the second field of the line ssh-keygen -l prints, which starts SHA256:. An MD5 fingerprint, a padded or lower case value and a whole key are not accepted. The fingerprint is asked for every sftp address, in whatever case its scheme is written: a source declared without one is warned about, and a change that declares one without it is refused. The plane is meant to connect only to the host key you pin, to refuse a server whose key changes, and never to trust the first key it sees.

Which files

filePattern says which files under each prefix are meant to be copied, written relative to the prefix: * stands for any run of characters except /, ? for one character, and each folder level needs its own *, such as *.csv or */*.csv. Nothing else is special: **, [ ], { } and a backslash are refused, and so are a pattern that starts or ends with /, one longer than 1024 bytes, and one with a . or .. or empty folder name.

Leave it empty and every file below each prefix is meant to be copied, at any depth, the folders within a prefix included.

A file is meant to be copied only if its whole path below the prefix matches, and capital letters count. *.csv matches files directly in the prefix and not in a folder within it; */*.csv matches files one folder down.

Where files land

Files are meant to land in the volume's files folder, <layer>/<schema>/_files/<volume>/, keeping each prefix as written: <prefix>/<path>, where <path> is the file's path below the prefix. The address's own path is not kept.

A volume takes one source. A source that names a volume another source already lands in is refused when you plan it, and the refusal names the other source's file. Like the other landing rules, this one is judged only in a repository that declares layers.

The copy rules

A copy is meant not to overwrite or delete a file: a file removed at the source is meant to stay in the volume, and a file that changed at the source is meant to be refused and counted. The run is meant to end failed while other new files are still copied. This is a check the copy makes before it writes, not a lock on the storage.

A copy is meant to compare checksums where both sides keep one and sizes alone where one side does not (an object uploaded in parts or encrypted with a key often has none), so a changed file of the same size may not be noticed.

Because a changed file stays different at the source, a copy that follows these rules would refuse it again on every later run, and each of those runs would end failed until the file at the source is put back as it was. That is a known gap: a record of which files a copy refused is not built yet, and a way to accept the new version is not built yet.

The credential

The secret stays in your store. This source names the reference and holds no value; a plane is meant to read the value from your store when it connects, and nothing connects yet. Secrets explains how to declare a reference and how to check that your plane can find the secret.

For an sftp address the credential is meant to be a private key without a passphrase. A copy is not meant to take a password, and Plan does not check which the reference holds.

A copy over sftp cannot run until the plane can check the server’s key against your fingerprint, which is not built yet.

The schedule

schedule has the form of a pipeline’s schedule, a cron expression and optionally a time zone, and the control plane checks it when you plan the change. Like a jdbc source, a source of this type takes only the overlap skip, syncs one at a time and has no queue. Leave schedule out for a source that is synced only when it is started. Nothing syncs an object storage source yet: a schedule in the file is recorded, and nothing starts a sync from it. The control plane’s scheduler is switched off, and it is built to open syncs only for database table sources: even switched on it would skip an object storage source’s tick. Sync now on the source’s page is refused for an object storage source, because only database sources can be synced today.

What the gate says

rl diff judges the file offline with the rules the control plane runs when it plans a change. For an object storage source it says the following. Where a rule has two severities, a warning is what a tree that already holds the source is given, so that it can still be removed, and a refusal is what a change that creates the source or writes the field is given.

Remove a source

Remove a source as you remove any file: delete sources/partner-drop.yaml, then plan and apply the change. Today that retires the source and removes the state the control plane recorded for it. It does not touch your storage or your drop, and it does not remove files already landed in the volume.

Not built yet

• Nothing copies files for an object storage source yet: no plane connects to your storage or your drop, copies a file into the volume or starts a sync from a schedule.
• A copy over sftp cannot run until the plane can check the server’s key against your fingerprint, which is not built yet.
• No plane has an identity to write a volume with yet.
• A record of which files a copy refused is not built yet, and a way to accept the new version of a changed file is not built yet.
• No connection test is built for this type yet, so nothing has checked that a plane can reach this location or read these prefixes.