Skip to content
RunningLake
Company Console ↗ Request early access
Contents: Reference

RUNNING

StorageRule

A StorageRule as RunningLake's configuration repository declares it, at storagerules/<name>.yaml. Generated from the Go types the control plane decodes this file into, so it checks the shape the gate reads — field names and types, and which fields may exist. The gate also applies rules no schema can express — required fields inside spec among them — so a file this schema accepts can still be refused by `rl diff`. Take #/properties/spec for the spec alone.

File: storagerules/<name>.yaml

Fields

FieldTypeRequiredMeaning
apiVersion string Yes

The config schema's version, which is independent of the HTTP API's (rule unsupported-api-version).

Values: runninglake.io/v1alpha1
kind string Yes

The kind this file declares. A kind that disagrees with the file's path is refused (rule kind-mismatch).

Values: StorageRule
metadata object Yes
No description
metadata.name string Yes

The resource's name, which is also its path segment. It must be the name this file's path gives it (rule name-mismatch).
metadata.labels map of string No
No description
metadata.annotations map of string No
No description
spec object No

StorageRuleSpec is a rule that watches one declared place in your lake for files and acts on them in order. Recorded and checked here; nothing watches a place or runs an action yet, so a rule does nothing.
spec.watch object No

Watch is the one place the rule watches: a volume, or a schema's uploads volume.
spec.watch.volume string No

Volume is a declared volume, layer.schema.volume, such as bronze.partners.drops.
spec.watch.schema string No

Schema is a declared schema, layer.schema, whose uploads volume is watched.
spec.watch.prefix string No

Prefix narrows the watch to a folder of the place, such as incoming/: at most 255 bytes, never .. and never beginning with _ or a dot.
spec.filters object No

Filters narrow the files the rule acts on.
spec.filters.events array of string No

Events are which changes start a batch: created, deleted, or both. Default created.
spec.filters.names array of string No

Names are at most 16 globs over the path relative to the watched place: * and ? stop at /, ** crosses it. Default every file.
spec.filters.exclude array of string No

Exclude are at most 16 globs whose files are left alone.
spec.filters.size object No

Size bounds a file's size, such as {min: 1KiB, max: 5GiB}; at most 5TiB.
spec.filters.size.min string No
No description
spec.filters.size.max string No
No description
spec.batching object No

Batching gathers files into one batch before the actions run.
spec.batching.debounce string No

Debounce waits this long after the last file before the batch starts: 0s to 1h.
spec.batching.maxFiles integer No

MaxFiles starts the batch once it holds this many files: 1 to 1000.
spec.batching.maxWait string No

MaxWait starts the batch this long after its first file however files keep coming: at least the debounce, at most 6h.
spec.ceilings object No

Ceilings bound how much the rule may start. Each may be lowered, or raised up to the plane maximum: 600 starts an hour and 50 active runs.
spec.ceilings.startsPerHour integer No

StartsPerHour is at most how many batches start in an hour: 1 to 600.
spec.ceilings.activeRuns integer No

ActiveRuns is at most how many runs the rule may have going at once: 1 to 50.
spec.actions array of object No

Actions are what the rule does with a batch, 1 to 8 of them, run in order; a failed action stops the later ones for that batch. Each entry names exactly one action. Only record runs today; each other action is refused, naming the work that will accept it.
spec.actions[].record object No

Record records the batch and does nothing else: record: {}.
spec.actions[].startPipeline object No

StartPipeline starts a run of a pipeline with values from the batch. Refused until the starting of pipelines by storage events is built.
spec.actions[].startPipeline.pipeline string No

Pipeline is the pipeline to start, by name.
spec.actions[].startPipeline.parameters map of object No

Parameters map a parameter of the pipeline to a value: {from: <pointer>} reads the batch, such as /batch/id or /batch/files; {value: <constant>} is fixed.
spec.actions[].startPipeline.parameters.*.from string No

From is a pointer. In a storage rule it points into the batch: /batch/id, /batch/files, /batch/bytes, /batch/firstArrival, /batch/lastArrival, /batch/closeReason or /rule/name; and, from a rule whose batches hold one file, /objects/0/size, /objects/0/key or /objects/0/uri. In a webhook trigger it points into the sender's JSON body.
spec.actions[].startPipeline.parameters.*.value any value No

Value is a constant.
spec.actions[].promote object No

Promote appends the batch's files to a declared table. Refused until promote is built.
spec.actions[].promote.table string No

Table is the declared table, layer.schema.table.
spec.actions[].move object No

Move copies the batch's files to a declared volume, then deletes them. Refused until file actions are built.
spec.actions[].move.volume string No

Volume is the destination, by its own declared name: layer.schema.volume.
spec.actions[].quarantine object No

Quarantine moves the batch's files to the volume's quarantine area. A destination overrides the default quarantine area the plane otherwise derives from the file's own volume. Refused until file actions are built.
spec.actions[].quarantine.volume string No

Volume is the destination, by its own declared name: layer.schema.volume.
spec.actions[].tag object No

Tag sets object tags on the batch's files. Refused until file actions are built.
spec.actions[].tag.tags map of string No

Tags are the keys and values to set. A key may not begin aws: (AWS reserves it for its own system tags) or rl- (the platform's own prefix).
spec.actions[].email map of any value No

Email sends a message of ids and counts. Refused until the message outbox exists.
spec.actions[].slack map of any value No

Slack posts a message of ids and counts. Refused until the message outbox exists.
spec.actions[].teams map of any value No

Teams posts a message of ids and counts. Refused until the message outbox exists.
spec.actions[].webhook map of any value No

Webhook sends a signed POST of ids and counts. Refused until the message outbox exists.
spec.enabled boolean No

Enabled turns the rule on. Default true. An environment may set its own.

A nested field is written with its parents, spec.a.b; an item of a list is [] and a value of a map is *. Required means the parent object must carry the field; the rules a schema cannot express, such as a field that is required only inside spec, are checked by rl diff.

The schema

This page is read from the kind’s published schema, at https://runninglake.com/schema/v1alpha1/storagerule.json. An editor that follows it checks a file of this kind as you type. To see the fields of a kind’s spec from the command line, run rl schema with the kind’s name: it works offline and prints what the gate accepts. The command reference has the rest.

All kinds