Environment: dex 1.6.3, BigQuery connector, dbt 1.11.
Problem
The same project measured before and after converting 13 staging models from
views to incremental (merge, with a watermark filter that prunes to a
three-day slice of a DAY-partitioned source):
|
before |
after |
transform build estimate |
6.90 GB |
6.70 GB |
| billed |
4.75 GB |
3.45 GB |
Billed fell 27%. The estimate moved 3%. dbt has no dry run for a
MERGE, so an incremental node is priced as though it read the whole target
relation, regardless of how aggressively its batch filter prunes.
Why it matters
- The gap is structural, not noise, and it widens over time. Billed tracks
the incremental window; the estimate tracks accumulated history. On a growing
warehouse the two diverge without bound.
- The estimate is what the guard enforces against. A build that bills 3.45 GB is
refused at a 5 GB ceiling, and nothing in the envelope lets an operator tell an
over-estimate from a genuinely expensive build.
- Incremental models are the normal shape for any warehouse large enough for the
cost guard to matter. This is the case where the estimate is least useful and
most consulted.
Proposal
dbt materializes an incremental run's batch as a temp table before merging.
Dry-run that compiled SELECT rather than pricing the target relation: it is
the statement that actually scans, and dbt will happily compile it.
Failing that, label incremental nodes' contributions in per_table_bytes as
upper bounds, so an operator can discount them deliberately instead of guessing.
Acceptance
For an incremental node whose batch filter prunes partitions, per_table_bytes
reflects the pruned scan rather than the full relation.
Related: #278 asks a refusal to cite the historical estimate-to-billed ratio,
which softens this. It does not fix the estimate.
Sibling: #324 is the downstream consequence — a cumulative daily ceiling checked
against estimates this pessimistic refuses work that real spending would not
have come close to.
Environment: dex 1.6.3, BigQuery connector, dbt 1.11.
Problem
The same project measured before and after converting 13 staging models from
views to incremental (
merge, with a watermark filter that prunes to athree-day slice of a DAY-partitioned source):
transform buildestimateBilled fell 27%. The estimate moved 3%. dbt has no dry run for a
MERGE, so an incremental node is priced as though it read the whole targetrelation, regardless of how aggressively its batch filter prunes.
Why it matters
the incremental window; the estimate tracks accumulated history. On a growing
warehouse the two diverge without bound.
refused at a 5 GB ceiling, and nothing in the envelope lets an operator tell an
over-estimate from a genuinely expensive build.
cost guard to matter. This is the case where the estimate is least useful and
most consulted.
Proposal
dbt materializes an incremental run's batch as a temp table before merging.
Dry-run that compiled
SELECTrather than pricing the target relation: it isthe statement that actually scans, and dbt will happily compile it.
Failing that, label incremental nodes' contributions in
per_table_bytesasupper bounds, so an operator can discount them deliberately instead of guessing.
Acceptance
For an incremental node whose batch filter prunes partitions,
per_table_bytesreflects the pruned scan rather than the full relation.
Related: #278 asks a refusal to cite the historical estimate-to-billed ratio,
which softens this. It does not fix the estimate.
Sibling: #324 is the downstream consequence — a cumulative daily ceiling checked
against estimates this pessimistic refuses work that real spending would not
have come close to.