Skip to content

Plan typed table and incremental materialization operations - #1651

Draft
dataders wants to merge 7 commits into
databricks:mainfrom
dataders:codex/typed-ddl-lifecycle-plans
Draft

Plan typed table and incremental materialization operations#1651
dataders wants to merge 7 commits into
databricks:mainfrom
dataders:codex/typed-ddl-lifecycle-plans

Conversation

@dataders

@dataders dataders commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • resolve table and incremental control flow into typed, serializable operation plans
  • distinguish logical Iceberg from Delta-backed UniForm and native managed Iceberg
  • reject Delta-only partition overwrite for managed Iceberg and prevent managed Iceberg from selecting REPLACE ON
  • retain explicit catalog provider, physical provider, DBR runtime/version, named capabilities, and selected SQL renderer variants
  • plan config-change inclusion, insert-by-name, tags/catalog legality, column comments, constraints, and optimization before execution
  • keep legacy Jinja fallbacks while ensuring typed leaf renderers do not probe provider, catalog, runtime, version, or capabilities

Dependency stack

Planning boundary

The plan owns decisions that change strategy, operation ordering, staging/replacement safety, overwrite behavior, operation inclusion, or SQL renderer variant. Relations, compiled SQL, resolved columns and predicates, and literal tag/property values remain late-bound execution inputs.

Validation

  • full unit suite: 1,376 passed, 4 skipped
  • final changed-surface suite: 191 passed, 2 skipped
  • renderer tests deliberately fail if typed paths probe format, catalog, runtime capability, or re-plan creation
  • Ruff fatal-error and unused-import checks passed; git diff checks passed
  • live Databricks validation remains blocked by the invalid configured access token

This remains a draft while dependency PRs and live Databricks validation continue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant