ClickHouse: Support scalar expressions in WITH clause by alrevuelta · Pull Request #2333 · apache/datafusion-sqlparser-rs

alrevuelta · 2026-05-07T10:43:43Z

ClickHouse's WITH clause supports a reversed-order form <expression> AS <identifier> (alongside, and freely interleaved with, traditional CTEs). Currently the parser hard-errors on the leading non-identifier token:

WITH 42 AS answer SELECT answer FROM t
-- Expected: identifier, found: 42

Changes

New WithItem enum (Cte(Cte) | Named { expr, alias }); With::cte_tables: Vec<Cte> becomes With::items: Vec<WithItem>.
New dialect flag Dialect::supports_with_clause_scalar_expression (default false, true for ClickHouseDialect only).
New parse_with_item routes each WITH-list element: when the leading token can't begin a CTE name, it parses <expr> AS <ident> directly; otherwise it tries the CTE form via maybe_parse and falls back. parse_cte itself is unchanged.

Tests

Added in tests/sqlparser_clickhouse.rs, covering: literal scalar (the issue's example), string literal, aggregate, scalar subquery, bare-identifier disambiguation, mixing scalar + CTE in one list, lambda, and a negative test confirming non-ClickHouse dialects still reject the form.

iffyio · 2026-05-07T11:35:22Z

+#[test]
+fn parse_with_clause_named_expression_unsupported_in_other_dialects() {
+    // The named-expression form is only enabled for ClickHouse; other
+    // dialects should still reject `WITH 42 AS answer …`.
+    let res = sqlparser::parser::Parser::parse_sql(
+        &GenericDialect {},
+        "WITH 42 AS answer SELECT answer FROM t",
+    );
+    assert!(res.is_err(), "expected parse error, got {res:?}");
+}


I think we can skip this, no need to test what the parser does for an invalid query for unsupported dialects

removed
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43L1902

iffyio · 2026-05-07T11:35:42Z

+}
+
+#[test]
+fn parse_with_clause_named_expression_ast() {


we can merge this with the test above since they cover the same feature

merged
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43L1887

iffyio · 2026-05-07T11:36:46Z

+#[test]
+fn parse_with_clause_named_expression() {
+    // Plain literal scalar.
+    clickhouse().verified_stmt("WITH 42 AS answer SELECT answer FROM t");


for the tests, lets use all_dialects_where instead of hardcoding it to clickhouse

done
3b86842#diff-b9a99857d69356e86daf74627f3d5da1f187d71e35dde75b0d924ed668ab1a43R1850

iffyio · 2026-05-07T11:40:19Z

+        // CTE form must start with an identifier. If the leading token
+        // can't begin one (e.g. `42`, `(SELECT …)`, `(x, y) -> …`), this
+        // is unambiguously the named-expression form.
+        if matches!(self.peek_token().token, Token::Word(_)) {
+            if let Some(cte) = self.maybe_parse(|p| p.parse_cte())? {
+                return Ok(WithItem::Cte(cte));
+            }
+        }


Can this be simplified to use peek_subquery_or_cte_start? e.g.

that we do if self.dialect.supports() and !self.peek_subquery_or_cte_start { parse expr } else { parse cte }

tbh, not sure i fully understand this. with that, will WITH 42 AS answer SELECT answer FROM t parse? peek_subquery_or_cte_start returns false on 42 (not (SELECT / (WITH), so we'd go to parse_cte, which then errors on 42 not being an identifier?

iffyio · 2026-05-07T11:40:52Z

+    /// Parse a single item in a `WITH` clause.
+    ///
+    /// In standard SQL this is always a CTE (`name [(cols)] AS (query)`).
+    /// Dialects that enable [`Dialect::supports_with_clause_scalar_expression`]
+    /// — currently only ClickHouse — also accept the reversed form
+    /// `<expression> AS <identifier>`, which can be freely interleaved with
+    /// CTEs in the same comma-separated list.


Suggested change

/// Parse a single item in a `WITH` clause.

///

/// In standard SQL this is always a CTE (`name [(cols)] AS (query)`).

/// Dialects that enable [`Dialect::supports_with_clause_scalar_expression`]

/// — currently only ClickHouse — also accept the reversed form

/// `<expression> AS <identifier>`, which can be freely interleaved with

/// CTEs in the same comma-separated list.

/// Parse a single item in a `WITH` clause.

fixed and minor renaming
3b86842#diff-fc6a8d66b8cb6bd48a119a70e489cbcef82e7f551e7cf08e8e972ef4e774ef49R14642

iffyio · 2026-05-07T11:46:09Z

+#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum WithItem {


repr wise can we do something like?

pub enum WithExpression { Cte(...), Cse(ExprWithAlias), // can we reuse exprwithalias? }

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R776-R782

iffyio · 2026-05-07T11:47:04Z

+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum WithItem {
+    /// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.


Suggested change

/// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.

/// A common table expression.

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R776-R782

iffyio · 2026-05-07T11:47:36Z

+    /// `<expr> AS <alias>` — binds an expression (literal, scalar subquery,
+    /// lambda, …) to a name visible in the surrounding query.
+    ///
+    /// See ClickHouse's [common scalar expressions][1].
+    ///
+    /// [1]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions


Suggested change

/// `<expr> AS <alias>` — binds an expression (literal, scalar subquery,

/// lambda, …) to a name visible in the surrounding query.

///

/// See ClickHouse's [common scalar expressions][1].

///

/// [1]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions

/// A common scalar expression.

///

/// [Clickhouse]: https://clickhouse.com/docs/sql-reference/statements/select/with#common-scalar-expressions

iffyio · 2026-05-07T11:49:35Z

-    pub cte_tables: Vec<Cte>,
+    /// The items declared by this `WITH` clause: traditional CTEs and,
+    /// for dialects that support it, named expressions.
+    pub items: Vec<WithItem>,


Suggested change

pub items: Vec<WithItem>,

pub exprs: Vec<WithExpression>,

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R757-R758

iffyio · 2026-05-07T11:49:46Z

+    /// The items declared by this `WITH` clause: traditional CTEs and,
+    /// for dialects that support it, named expressions.


Suggested change

/// The items declared by this `WITH` clause: traditional CTEs and,

/// for dialects that support it, named expressions.

/// The expressions declared by this `WITH` clause.

fixed
3b86842#diff-ff51937f34d0076928ac31baffb41b26e4cbc0ec6dd8496105a7f61188f92ed1R757-R758

alrevuelta · 2026-05-11T11:17:54Z

@iffyio thanks for the comments, fixed.

there is only this that i dont fully understand if possible.

Allow WITH using scalars for ClickHouse dialect

6589675

iffyio reviewed May 7, 2026

View reviewed changes

Implement comment fixes

3b86842

	/// A traditional common table expression: `name [(cols)] AS [MATERIALIZED] (query)`.
	/// A common table expression.

		/// The items declared by this `WITH` clause: traditional CTEs and,
		/// for dialects that support it, named expressions.

	/// The items declared by this `WITH` clause: traditional CTEs and,
	/// for dialects that support it, named expressions.
	/// The expressions declared by this `WITH` clause.

Conversation

alrevuelta commented May 7, 2026

Changes

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alrevuelta commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants