Is it possible to run SQL jobs on serverless (jobs, not SQL)

[scdmitry]

hi, while it's trivial to run on SQL warehouses, there are cases where scaling of serverless jobs (contrasted to fixed sizes of SQL warehouses), as well as some configurations (dynamic partition overwrite, for example) are needed.

I was wondering if someone looked into this and whether there are some known limitations about such approach.
I could imagine a crude way of parsing manifest, then constructing a Databricks workflow and slapping an environment on top of it, and deploying + running that, but I'm wondering if i'm missing some already supported way in the adapter, or if reaching something like that is a low hanging fruit and I just need to look better?

[benc-db]

Jobs clusters do not have a thrift server which is how we execute SQL in dbt-databricks. As much as this is not the answer you want to hear, it is not a company priority to support dbt SQL models on job clusters; on the other hand, I believe we are working internally to address your valid complaint about SQL Warehouse scaling.

[scdmitry]

Thank you for the answer! If I could add a couple more subquestions (partially related), sorry if this will get a little convoluted.

While the adapter has been absolutely excellent for SQL, I was also looking into python models to partially deal with my challenge - e.g. I was wondering if I can run some python models on serverless jobs, and wrap the SQL in them. I don't think that's a good approach, and would prefer to stick to SQL and will stick to it, but as I was testing, I saw what ended up running, I looked into the jinja template used for python jobs (https://github.com/databricks/dbt-databricks/blob/main/dbt/include/databricks/macros/adapters/python.sql#L42 if I'm not mistaken).

Putting SQL aside, we do have some pyspark models that would be very difficult to port to SQL. But the jinja template has some challenging parts in the end (overwriteSchema). Would you have any recommendation about how to approach a python model if that template is not compatible with what I'm running? Is there a way to change which macro is used when a python model is ran?

[benc-db]

Putting SQL aside, we do have some pyspark models that would be very difficult to port to SQL. But the jinja template has some challenging parts in the end (overwriteSchema). Would you have any recommendation about how to approach a python model if that template is not compatible with what I'm running? Is there a way to change which macro is used when a python model is ran?

Please open an issue with details about this, we might be able to make it more configurable. Alternatively, if you include your own macro with a matching name, I think you'll macro will get picked up in favor of the Databricks one.

[sd-db]

Hi @scdmitry closing the issue as the original intent is not resolvable at client side/fixed at server side. Feel free to reopen if you still have a concern.