BUILDING Parts of this page describe what is not built yet, and each says so.

Storage rules

A storage rule watches one place in your lake for files that members upload, gathers them into batches, and acts on each batch; today the one action that runs records it. This page says what a rule is, every field of it, what it may do, what leaves your account, how to try a file against it before one arrives, and which clouds raise events. The events a rule records carry ids and counts, and never a file name.

A rule

A rule is a file in your project’s repository, like a table or a pipeline. Save this as storagerules/partner-drop.yaml:

apiVersion: runninglake.io/v1alpha1
kind: StorageRule
metadata:
  name: partner-drop
spec:
  watch:
    volume: bronze.partners.drops
    prefix: incoming/
  filters:
    events: [created]
    names: ["*.csv", "**/*.csv.gz"]
    exclude: ["*.tmp"]
    size: {min: 1KiB, max: 5GiB}
  batching: {debounce: 5m, maxFiles: 1000, maxWait: 1h}
  ceilings: {startsPerHour: 30, activeRuns: 5}
  actions:
    - record: {}
  enabled: true

rl diff checks the file offline, exactly as the control plane would, and a rule is a change like any other: it is planned, reviewed and applied as Review and approval describes. The StorageRule kind is the generated reference, with the schema to point your editor at. These are its fields:

The gate also refuses what no schema can say, such as a rule with no watch, a maxWait shorter than the debounce, or a place that is not declared in the same environment, and it names the field. An overlay may patch only enabled and batching for one environment.

The place it watches

A rule watches one place: a volume, or a schema’s uploads volume, and optionally a folder of it. A rule reacts to files written under the place. Members write there, in the console or with rl volume upload, and the rule does not check who wrote a file. A rule that watches a volume is exactly as visible as the volume, and one that watches a schema is visible to whoever holds a grant on the whole schema or layer.

Filters

A filter narrows the files a rule acts on: the kind of change, name patterns and exclusions, and a size range. In a name pattern * and ? stop at a slash and ** crosses it. A file that a filter refuses is counted and never listed.

Batching

Files gather into a batch, which closes after the debounce without a new file, when it holds maxFiles, or after maxWait, whichever comes first. A batch holds at most 1,000 files. Each field is optional: left out, a batch closes after 5 quiet minutes, at 1,000 files, or after 1 hour.

Ceilings and approval

Ceilings are the most a rule may start: batches an hour, and runs going at once. A rule may lower a ceiling, or raise it up to the plane’s maximum of 600 starts an hour and 50 active runs, and a raise needs a second person. Nothing starts a run from a file yet, so nothing enforces a ceiling today.

A change to a rule needs a second person when it creates a rule that is on or switches one on, adds or changes an action that moves data, changes the lake, starts a pipeline or sends a message, points such a rule at another place, raises a ceiling, or sets or removes one. It does not when it changes filters or batching, adds a tag, records only, switches a rule off, creates a rule that is off, or lowers a ceiling the rule already had. A change that cannot be read needs one too.

Actions

A rule has one to eight actions, run in order for each batch. Once more than one kind of action runs, one that fails stops the ones after it for that batch. The closed list is:

• record records the batch, and nothing else happens. It is the only action that runs today.
• startPipeline starts a run of a pipeline with values from the batch. Starting a pipeline from a file is not built yet.
• promote appends the batch’s files to a declared table. Promote is not built yet.
• move, quarantine and tag copy, move or tag the batch’s files. Moving, quarantining and tagging files are not built yet.
• email, slack, teams and webhook send a message of ids and counts. Messages from a rule are not built yet.

Naming any action but record is refused when the change is checked, and the violation names the work that will accept it. A dry run never says such an action would run.

What crosses the boundary

The events a rule records carry no file name. The plane reads the events of your own storage, matches them against your rules and reports ids, counts, sizes and times to the control plane, which keeps a keyed digest of each event so that a repeat is recognised, and never the name of a file. A rule’s Events tab says so: to find a file, match the time and the size against the files in your own storage. Two other places see a name. A name you type or pick to try a rule travels to the control plane to be matched and is not kept, and the audit entry of an upload records its path.

Try a file

The dry run asks what one file would do under a rule, before one arrives: which filters it passes, what batching would do, which actions would run, and why nothing would. In the console it is Test this rule on a rule’s page. With rl:

# What would one file do under this rule? Nothing is sent to your plane and nothing starts.
rl storage-rule test partner-drop -project analytics -env dev -object incoming/orders.csv -size 12MiB

It writes nothing and starts nothing, and it prints no file name. The exit code is the verdict: 0 when at least one action would run, 1 when none would, and 2 when nothing was evaluated, such as a rule that is not there. A rule whose plane has not confirmed that it holds it, or does not report that it records, exits 1 and says which, so a script never reads a mistyped name as a verdict.

From the command line

rl storage-rule list lists the rules of an environment, one line each, with the state of each, and rl storage-rule get reads one: its state, where it watches, and what its plane reports. rl storage-rule events lists what the control plane recorded for a rule. A figure that a plane has not reported is a dash, never a zero. rl get StorageRule reads the file itself. The command-line reference lists each command, the permission it needs and its exit codes.

Which clouds raise events

On AWS a plane can be set up so that its storage sends its events through Amazon EventBridge to a queue in your account, which the plane’s agent can read. This is off until the plane is set up for it. A bucket you bring yourself must send notifications to EventBridge, and a rule’s state says so when it does not. Whether a plane is reading events for a rule is on the rule’s page.

• Google Cloud planes do not raise storage events yet.
• Azure planes do not raise storage events yet.

Back to Start here