Data engineering in 2026
Where data engineering is going, and what we do about it.
Generative AI is now part of every data platform. The work is moving from writing pipelines by hand to designing, governing and paying for them. Here are five shifts that define 2026, what RunningLake does about each, and what we deliberately leave out.
01 · Data types managed
AI-native pipelines and data for LLMs
What is changing. Pipelines now feed retrieval-augmented generation and other LLM applications, so documents, embeddings and vector search sit beside tables. Models also need to understand the data: explicit keys, relationships and well-described assets.
What RunningLake does. Your documents stay in your own Volumes. A pipeline step cuts them into chunks and writes one row per chunk to an ordinary Iceberg table in your bucket, and each embedding is a column of numbers in that table, governed like any other column. Similarity search runs in SQL on the engines. A vector database such as Pinecone or Milvus receives a copy through a registered destination; it is never where your data lives.
AI-ready data PLANNED Reverse ETL PLANNED
What we leave out: a vector database of our own, and a fourth table format. A copy that lives somewhere else is a second place to secure.
02 · Pipeline creation
AI copilots and declarative orchestration
What is changing. AI writes the boilerplate: SQL, PySpark, unit tests and documentation. The engineer's job becomes the workflow around it, declared in files that a machine can validate, and the judgment no model has: business context, and why a job is slow.
What RunningLake does. Tables, pipelines and grants are files in git with a published JSON Schema, and rl diff checks them with the same gate the server uses. The Pipeline Agent drafts a pipeline as a change set, with a test beside each transform, and a person submits it. Ask why a run was slow, and the answer comes from the run's own numbers: skew, spill, small files. And rl mcp gives the coding assistant you already use the same validate-and-plan path, with your own permissions.
Config in git RUNNING Pipeline Agent PLANNED Generated tests PLANNED Slow-run diagnosis PLANNED rl mcp PLANNED
What we leave out: autonomy beyond proposing. An agent drafts, and a person submits every change, through the same checks as their own edit.
03 · Target storage
The open lakehouse and zero ETL
What is changing. Open table formats on object storage give one copy of the data the governance of a warehouse and the flexibility of a lake, and native integrations replace hand-written extraction code.
What RunningLake does. Iceberg in your own bucket is the default, and a customer who leaves keeps working tables. Connectors are designed to be declared configuration that runs in your cloud account and lands a changelog in bronze, so there is no extraction code to maintain; Postgres change capture comes first. Connectors are not built yet. You can declare a Source, the place a connector is meant to read from, and apply it today; Sources are not offered on a trial. Nothing connects to your database, tests a connection or captures a change yet. New Iceberg format features are measured engine by engine on every train, and written only when the one format floor moves.
Postgres CDC and upload BUILDING Iceberg v3 features PLANNED Policy at the catalog PLANNED
What we leave out: querying your operational databases in place, and a second write format for one table. Data is meant to be copied into open tables, where the catalog is meant to apply your policies.
04 · Pipeline quality
Data contracts and reliability
What is changing. Data contracts have become production agreements between the teams that produce data and the teams that depend on it, enforced before a change breaks anyone downstream, alongside automated lineage.
What RunningLake does. A table's contract lives in git with it: its keys and relationships, its checks, how fresh it must be, its owner and who depends on it. A change that would break it waits for the owners of what depends on it, and a run that would break it publishes nothing, so your dashboards keep the last good data. Contracts import from and export to the Open Data Contract Standard.
Data contracts PLANNED
What we leave out: constraints we cannot keep. No engine enforces a key on an open table, so a declared key is a promise the contract's checks verify on every publish.
05 · Cost management
FinOps as part of the architecture
What is changing. AI and real-time workloads make cloud cost grow fast, so cost awareness is now designed in, not reviewed at the end of the month.
What RunningLake does. The bill is in cloud dollars. Work is added up by project, environment, submitter and engine; not yet by table. The cost page projects the month as a range with its basis, flags unusual spending, and shows advice with the dollars at stake: snapshots piling up, a warm pool kept warm for nobody. Table maintenance appears as its own line. No advice is applied until you plan it.
Cost attribution BUILDING Cost per table PLANNED Forecast and alerts PLANNED Cost advice PLANNED Maintenance line PLANNED
What we leave out: an alert that acts. It never throttles, suspends or re-routes your work, and there is no credit unit anywhere.
Questions
Does RunningLake include a vector database?
No. Embeddings are columns of ordinary Iceberg tables in your own bucket, governed like any other column, and similarity search runs in SQL. When an application needs a vector database such as Pinecone or Milvus, RunningLake keeps it in sync through a registered destination, and your tables stay the record. This is planned, not built yet.
What is a data contract in RunningLake?
A table’s contract lives in git with it and names its keys and relationships, its checks, how fresh it must be, its owner and who depends on it. A change that would break it waits for the owners of what depends on it, and a run that would break it publishes nothing. Contracts import from and export to the Open Data Contract Standard (ODCS). This is planned, not built yet.
How does RunningLake forecast cloud spend?
The cost page projects the month as a range, and says what it was projected from. It is a projection, not a promise, and unusual spending raises an alert that never stops, slows or moves your work by itself. This is planned, not built yet.
Can AI coding assistants work with RunningLake?
Yes. Tables, pipelines and grants are files in git with a published JSON Schema, and rl diff checks them with the same gate the server uses. rl mcp gives an assistant such as Claude Code, Cursor or Copilot that same validate-and-plan path and nothing more, with your own permissions. A person submits every change. Configuration in git runs today; rl mcp is planned, not built yet.
Does RunningLake do zero ETL?
Connectors are designed to be declared configuration that runs in your own cloud account and lands a changelog in bronze, so there is no extraction code to write or maintain; Postgres change capture comes first. Connectors are not built yet. You can declare a Source, the place a connector is meant to read from, and apply it today; Sources are not offered on a trial. Nothing connects to your database, tests a connection or captures a change yet. RunningLake does not query your operational databases in place: the data is meant to be copied into open tables, where the catalog is meant to apply your policies. Postgres change capture and file upload are being built; policy at the catalog is planned, not built yet.
Which of this can I use today?
Each answer above says where its subject stands, and every capability on this page carries a status chip. A capability not marked RUNNING is not built yet, and the ones new on this page are not yet in the build order on the company page.