BUILDING Parts of this page describe what is not built yet, and each says so.
Migrating in
Migrating in has two parts that are built. The usage analyser: you run a query in your own Databricks or Snowflake account and send us the file it writes, and it shows what that work costs today, beside a range for the work it can price on RunningLake, and says what each side leaves out. It is an estimate from a usage export, not a quote. And the inventory reader: you run a second query in your own Snowflake account, and it reads the tables and roles in the file into a plan.
The analyser holds no Databricks or Snowflake login and takes none: you run the query yourself and send the file. The inventory reader holds no Snowflake login either. A shorter page, for someone deciding whether to try it, is Migrating in.
Run the export
There is one query for each source. Each selects one row for each day over the last 365 complete days and for each piece of work. It selects no statement text, no person and no table name, and no name of anything you called.
The queries were written from the vendor documentation and have not been run against a real account.
| Export from | Required columns | Optional columns |
|---|---|---|
| Databricks | usage_date, sku_name, usage_unit, usage_quantity |
job_id, warehouse_id, cluster_id, list_price_usd |
| Snowflake | usage_date, warehouse_id, credits |
bytes_scanned, execution_seconds, query_count |
An export is at most 16 MiB and at most 5,000 workloads, and an organisation keeps at most 200 analyses. They are limits, not promises that a file of that size will be read. What a Databricks export leaves out is the machines under classic clusters, storage and data transfer. What a Snowflake export leaves out is storage, data transfer, and the credits of serverless features and of the cloud services layer. The analysis says so.
To run the Databricks query, use a SQL warehouse of your workspace, as someone who can read the billing system tables, and download the result as a CSV file with a header row.
-- RunningLake usage export for Databricks.
--
-- Run it in a SQL warehouse of your workspace, as someone who can read the billing system
-- tables, and download the result as a CSV file with a header row. RunningLake reads the file in
-- memory and does not keep it.
--
-- WHAT IT SELECTS. One row for each day over the last 365 complete days and for each piece of
-- work: the day, the SKU class, the unit and how much of it was used, the job, SQL warehouse or
-- cluster the usage belongs to (by id), and that usage priced at Databricks' effective list
-- price, in dollars. A cluster id is kept only for usage that belongs to no job and no SQL
-- warehouse, because a job's cluster is made again for every run. It selects no statement text,
-- no person and no table name, and no name of anything you called: only ids and Databricks' own
-- SKU classes.
--
-- WHAT THE DOLLARS ARE. list_price_usd is the day's usage multiplied by Databricks' list price,
-- as one amount in dollars. It is the LIST price: a price negotiated in your contract is not in
-- it, and the analyser asks you for the rate you pay if you want rows with no dollars priced at
-- yours. It is empty, and never 0, when any of that day's usage had no list price, or when the
-- price rows did not match the usage exactly once each, because a total of only the priced usage
-- is a total of other work. A correction Databricks made to earlier usage arrives as a record
-- with a negative quantity, and it nets against the usage it corrects.
--
-- EMPTY CELLS. A cell with nothing in it is empty and not the word null, because the SQL
-- editor writes a missing value as that word in a downloaded file.
--
-- WHAT IT LEAVES OUT. The machines under classic clusters, storage and data transfer, which
-- Databricks does not bill and a usage export does not hold. The analysis says so.
--
-- This query was written from the vendor's documentation and not yet run against a real account.
-- It reads system.billing.usage and system.billing.list_prices.
WITH usage_by_day AS (
SELECT
u.usage_date AS usage_date,
u.sku_name AS sku_name,
u.usage_unit AS usage_unit,
COALESCE(u.usage_metadata.job_id, '') AS job_id,
COALESCE(u.usage_metadata.warehouse_id, '') AS warehouse_id,
CASE
WHEN u.usage_metadata.job_id IS NULL AND u.usage_metadata.warehouse_id IS NULL
THEN COALESCE(u.usage_metadata.cluster_id, '')
ELSE ''
END AS cluster_id,
SUM(u.usage_quantity) AS usage_quantity,
COUNT(*) AS usage_records
FROM system.billing.usage AS u
WHERE u.usage_date >= date_sub(current_date(), 365)
AND u.usage_date < current_date()
GROUP BY 1, 2, 3, 4, 5, 6
),
list_priced_by_day AS (
SELECT
u.usage_date AS usage_date,
u.sku_name AS sku_name,
u.usage_unit AS usage_unit,
COALESCE(u.usage_metadata.job_id, '') AS job_id,
COALESCE(u.usage_metadata.warehouse_id, '') AS warehouse_id,
CASE
WHEN u.usage_metadata.job_id IS NULL AND u.usage_metadata.warehouse_id IS NULL
THEN COALESCE(u.usage_metadata.cluster_id, '')
ELSE ''
END AS cluster_id,
SUM(u.usage_quantity) AS usage_quantity,
COUNT(*) AS usage_records,
COUNT(p.pricing.effective_list.default) AS priced_records,
SUM(u.usage_quantity * p.pricing.effective_list.default) AS list_price
FROM system.billing.usage AS u
LEFT JOIN system.billing.list_prices AS p
ON u.sku_name = p.sku_name
AND u.cloud = p.cloud
AND u.usage_unit = p.usage_unit
AND p.currency_code = 'USD'
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE u.usage_date >= date_sub(current_date(), 365)
AND u.usage_date < current_date()
GROUP BY 1, 2, 3, 4, 5, 6
)
SELECT
d.usage_date AS usage_date,
d.sku_name AS sku_name,
d.usage_unit AS usage_unit,
CAST(d.usage_quantity AS DECIMAL(38, 9)) AS usage_quantity,
d.job_id AS job_id,
d.warehouse_id AS warehouse_id,
d.cluster_id AS cluster_id,
CASE
WHEN p.usage_records = d.usage_records
AND p.priced_records = d.usage_records
AND p.usage_quantity = d.usage_quantity
THEN CAST(CAST(p.list_price AS DECIMAL(38, 12)) AS STRING)
ELSE ''
END AS list_price_usd
FROM usage_by_day AS d
LEFT JOIN list_priced_by_day AS p
ON d.usage_date = p.usage_date
AND d.sku_name = p.sku_name
AND d.usage_unit = p.usage_unit
AND d.job_id = p.job_id
AND d.warehouse_id = p.warehouse_id
AND d.cluster_id = p.cluster_id
ORDER BY usage_date, sku_name, job_id, warehouse_id, cluster_id
To run the Snowflake query, use a worksheet of your account, with a role that can read the SNOWFLAKE.ACCOUNT_USAGE views, and download the result as a CSV file with a header row.
-- RunningLake usage export for Snowflake.
--
-- Run it in a worksheet of your account, with a role that can read the SNOWFLAKE.ACCOUNT_USAGE
-- views, and download the result as a CSV file with a header row. RunningLake reads the file in
-- memory and does not keep it.
--
-- WHAT IT SELECTS. One row for each day over the last 365 complete days and for each warehouse:
-- the day, the warehouse's id, the compute credits it used, and the bytes its queries scanned,
-- the seconds they ran and how many there were. It selects no statement text, no person and no
-- table name, and no name of anything you called: a warehouse is its id and nothing else. Days
-- are the days of your session's time zone. Snowflake's pseudo-warehouse for cloud services
-- only has no warehouse id above zero and is left out.
--
-- WHAT THE CREDITS ARE. credits is the compute credits of the warehouse (CREDITS_USED_COMPUTE).
-- It counts credits and not dollars, so the analyser asks you for the rate you pay per credit,
-- from your contract, to price them. It is not the credits you were billed for: the cloud
-- services layer, which Snowflake bills only above an allowance, and serverless features are
-- not in it.
--
-- WHAT IT LEAVES OUT. Storage, data transfer, and the credits of serverless features (Snowpipe,
-- automatic clustering, tasks and the like) and of the cloud services layer. The analysis says
-- so. The days of a warehouse that ran no query have zeros for the three query columns, because
-- that is what they were.
--
-- This query was written from the vendor's documentation and not yet run against a real account.
-- It reads two ACCOUNT_USAGE views, which can lag by a few hours, and a year of query history
-- can take a minute to scan on a busy account.
WITH credits_by_day AS (
SELECT
TO_DATE(start_time) AS usage_date,
warehouse_id AS warehouse_id,
SUM(credits_used_compute) AS credits
FROM snowflake.account_usage.warehouse_metering_history
WHERE start_time >= DATEADD(day, -365, CURRENT_DATE())
AND start_time < CURRENT_DATE()
AND warehouse_id > 0
GROUP BY 1, 2
),
work_by_day AS (
SELECT
TO_DATE(start_time) AS usage_date,
warehouse_id AS warehouse_id,
SUM(bytes_scanned) AS bytes_scanned,
SUM(execution_time) / 1000 AS execution_seconds,
COUNT(*) AS query_count
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD(day, -365, CURRENT_DATE())
AND start_time < CURRENT_DATE()
AND warehouse_id > 0
GROUP BY 1, 2
)
SELECT
c.usage_date AS usage_date,
c.warehouse_id AS warehouse_id,
c.credits AS credits,
COALESCE(w.bytes_scanned, 0) AS bytes_scanned,
COALESCE(w.execution_seconds, 0) AS execution_seconds,
COALESCE(w.query_count, 0) AS query_count
FROM credits_by_day AS c
LEFT JOIN work_by_day AS w
ON c.usage_date = w.usage_date
AND c.warehouse_id = w.warehouse_id
ORDER BY c.usage_date, c.warehouse_id
Send it
In the console, open Migrate and choose Analyse a usage export. Say where the export is from, add the file and, if you want dollars for work the export counts in units, type what you pay per unit, from your contract. The analyser never fills in a rate for you, and without one those units stay as units, with no dollars.
The command line does the same: rl migrate analyse -source snowflake -rate RATE export.csv sends the
file and prints the analysis. Flags go before the file. The call is made once and never retried, because a second
call would be a second analysis. rl migrate analyses lists the analyses your organisation keeps, and
rl migrate analysis ID reads one back; with -csv it takes the analysis as a file.
The rl command line has every flag and exit code. Who may send an export, read an
analysis and take its file is in Permissions.
Read the result
An analysis shows two sides beside each other, each with what it leaves out. Today’s side is what the export states, averaged to a month over the period it covers. Each figure says where it came from: dollars the export states at the vendor’s list price, before any price you negotiated; units at the rate you typed; or units alone, with no dollars.
The RunningLake side is a range from low to high, with the rate card it was priced with and the region it assumed. The floor and the management fee are lines of their own. The floor is the cost of keeping one node on for every hour of the month. The fee is the management fee of the Team tier, and its rate is on the pricing page. A Business or Enterprise platform fee is not in the range. The RunningLake side does not price storage, or the services a plane keeps running whether or not any work runs.
Work the analyser cannot place is listed with its reason and kept out of both sides: either the work is of a kind RunningLake does not run yet, or the export does not say what kind of work it is.
It does not subtract one side from the other, turn the two into a ratio, or promise that a move will cost what the range says.
Confidence and coverage
Each priced workload has a confidence and the reason for it, and the range lists what drives it. The paragraph at the top of an analysis says that confidence beside the RunningLake range, and the RunningLake total repeats it with the reason; when only part of the work is priced, that paragraph says it of the priced part. Work that was not priced has no confidence: it says it was not priced and gives the reason. One engine, Trino, has had its scan speed measured on a RunningLake plane, on one machine type, an AWS c7i.xlarge. A Snowflake export that states the bytes scanned gets a range for its SQL warehouse work, at low confidence. The compute part of the range is the bytes in the export divided by that speed and priced at that machine's on-demand list price in us-east-2, an AWS region, which is the region the analysis states among its assumptions, with its low end at the typical speed measured and its high end at a slow one, and a range appears only when the control plane has a rate card loaded that is not out of date. The bytes in the export may not be counted the same way as the engine counts them, and the export does not say whether they are before or after compression, so the range could be too high or too low. The measurement was made on one sample table, read repeatedly from one file, on a node that looked idle, with warm caches, and on Trino 483 alone. It is not shown to hold for wide, nested, small-file or many-partition tables. Real tables may cost more than the range says, and the range does not bound work dominated by skewed joins. The high end of the range is the slowest of the cases the range was built from, at its second-slowest of ten runs. A skewed join was measured, was slower still, and is left out. That high end is not a worst case for your tables, and the low end is not a floor. A narrow range does not mean a small uncertainty: the gap between its two ends exists only because two speeds measured on the engine are apart, and nothing in the range allows for the bytes in the export not being comparable to the engine's. Cold start, memory growth and a busy shared node were not measured.
A Databricks export states no bytes scanned, so its work is not priced. Scheduled work, which the analyser would run on DuckDB, is not priced, because DuckDB has not been measured. When part of the work is not priced, the RunningLake side shows no total.
Read an inventory
A usage export says what your work costs. An inventory says what you have: the tables and roles of a Snowflake account. The inventory reader keeps a migration plan with one item for each table and one for each role you created.
You run one query in your own Snowflake account, with a role that can read the SNOWFLAKE.ACCOUNT_USAGE views, and download the result with its header line. It returns one column named record, and each line of it is one JSON object. The query is in the console, on the page that starts a migration. There you choose where the inventory is from, say which project and environment the tables would land in, and add the file.
The query selects the names, types, row counts and sizes of your tables and their columns, your roles, and the privileges granted to each role directly. It selects no person, no view text, no policy name or body, and no row of any table. It also sends the name of each view and of what depends on what, and which tables and columns carry a policy, never the policy's name or body. The control plane keeps the number of views and no name of a view or a dependency. Only names, types, counts, sizes and a few yes-or-no flags leave your account.
It leaves out privileges on anything but a database, a schema or a table, privileges on objects that do not exist yet, and which role holds which role. It leaves out the SNOWFLAKE and SNOWFLAKE_SAMPLE_DATA databases, databases shared with you, the personal databases whose names begin USER$, and every INFORMATION_SCHEMA schema; the file counts the databases it left out. The views it reads can lag by a few hours, so an object made this morning may not be in the file.
The inventory query was written from the vendor documentation and has not been run against a real account. The type names the mapping recognises come from the same documentation; a type reported under another name is refused.
The first line of the file counts what follows, so a download that was cut short is refused and not half read. When the reader refuses a file it says so in a sentence of its own, names the line when it can, and nothing from the file is kept. An inventory is at most 16 MiB, 20,000 tables, 2,000 columns in a table and 5,000 roles, and a plan holds at most 25,000 items.
The item of a table says the layer, schema and table name it would have here. A database or schema named bronze, raw, silver, staging, gold or marts proposes the layer (raw, staging and marts become bronze, silver and gold) and the other name becomes the schema; a pair with no such name waits for a person to choose the layer. Other names are lowercased only when they are all capitals or all small letters; a name with both cases waits for a person. A table that is external, temporary, dynamic, hybrid or managed as Iceberg, or has no columns, waits for a person too. A name the platform cannot hold, or two names that would become one, stops the item being mapped, and the item says why. A column type with an equivalent here is mapped. A type that loses something, such as a time zone, a timestamp finer than a microsecond or JSON held as text, makes the table wait for a person. A type with no equivalent, such as a geography, a geometry or a vector, refuses it. A table with a policy on it waits for a person too.
Each role you created becomes one item that waits for a person; roles Snowflake ships are counted and get no item. Direct privileges on tables that will exist here are listed as the grants they would become, for select only. Insert, update, delete and truncate, ownership and every other privilege are counted and not turned into grants, and so is a privilege on a table that waits for a person for a policy or a name collision. An organisation keeps at most 10 migration plans, and one that has that many is refused another.
What is kept and what is recorded
Of a usage export (the analyser keeps no names):
The export is read in memory and not kept. What is kept is the monthly total for each workload that has dollars, from the export or from a rate you typed, the units the export counts in aggregate, the kind of work each workload is, the id the export gives it, the period it covers and how many rows it had, who sent it and when, and the figures the analyser worked out from them, with the rate card and assumptions it used. The audit log keeps a keyed fingerprint of the file. An analysis also keeps the kind of work, the range and confidence worked out for each workload, the rate card, the region and the assumptions it was priced with, and who sent it and when. An id is kept exactly as the file gives it: the queries select ids and SKU classes, and in a usage export no name of anything you created.
When the analyser refuses a file it says so, and that nothing from the file was kept.
Of an inventory (names are kept):
What you move has names, so an inventory is kept on its own terms. The file is read in memory and not kept. What is kept is the name of each table, column and role it lists, the database and schema each table is in, the type of each column, the privileges each role holds on tables, what each would become here and why an item needs a person, the row count and size of each table, who sent it and when, with the target you chose. No statement text is read, and no user or email field is selected, but a table, column or role may still be named after a person. The audit entries a migration writes keep a keyed fingerprint of the file and counts, the kind of a principal a person chooses, and no name. Everyone who may read the project can read the names, except those under a layer they may not see, and an item with no layer proposed is shown to everyone who may read the project. A person can choose who the grants of a role are for; the principal they type is kept with the role, may be an email address, is read by everyone who can read the names, and is erased when the migration is deleted. A person can also say that an item was done some other way, with rl migrate handled-outside or the API, and must write how; that note is your own words and may name anyone, it is kept with the item, is read by everyone who can read the names and is erased when the migration is deleted, and an audit entry says only that the statement was made or taken back, and whether the item is a table or a role. RunningLake does not do the item or check the note. Deleting a migration erases its plan and its items, and the names they hold, from the database the control plane keeps. A migration with a change set still draft, proposed or open cannot be deleted until that change set is withdrawn or closed. A change set already made from its items is not changed. Its documents, in the control plane's database and, once approved, in your configuration repository, keep the table, column and team names and the principal the grants of a role go to, which may be an email address, and deleting the migration does not reach them. A backup of that database made before the delete still holds the names and notes until that backup is removed. The audit entry of the delete keeps that it was deleted and how many items it had, and no name.
Sending an export, and taking an analysis out as a file, are each recorded in the audit log. The entry holds counts and the period, and never an id or a figure from the file. Importing an inventory is recorded too: the entry holds the source, counts and a keyed fingerprint of the file, and never a name.
You can delete an analysis, or a migration with the names it holds, in the console, with rl migrate delete, or with the API. Deleting an analysis erases its totals and the figures worked out from them. The audit log keeps that it was deleted, and what it already kept when the analysis was made (the source, the period, how many rows and workloads, and a keyed fingerprint of the file), and no figure. An organisation keeps at most 200 analyses, and one that has that many is refused another. Deleting an analysis frees a place under the limit of 200 analyses, and deleting a migration frees a place under the limit of 10 migration plans. Closing an account does not erase them by itself yet: delete a migration first if you want its own copy of the names and notes gone. The change sets made from it keep theirs.
Not built yet
Reading a Snowflake inventory, and mapping each table and role in it to what it would become here, is built. So is reading a usage export, and so is planning a mapped table as a change set that a person reviews and approves.
- Copying the data of a table is not built yet.
- Translating the SQL of views is not built yet.
- Moving jobs and models in is not built yet.
- Reading a Databricks inventory is not built yet.
- Reading a dbt manifest is not built yet.
- dbt has no usage export: its models arrive as a manifest in a migration, which is not built yet.