DBT Models are great in separating SQL into their own enclosures which can be summoned anytime by Directed A-cyclic Graph code scripted in Python.
I have mostly witnessed plain SQL sprinkled with a few built-in DBT methods such as ref() and source() within the model files. I did wonder if the logic within these models can be changed during run time like we do with DAG code (obviously via Airflow’s variables).
With SQL injection option out of the window; Conditional block provided by Jinja was the obvious answer.
Jinja?
For Novice users; Jinja is a templating language that references certain variables; values for which are then rendered during runtime! In short, it is possible to advice DBT on how to handle certain conditions that might occur during runtime.
Photo by Nathan Bingle on Unsplash
This link should provide more enlightenment over Jinja templating.
Where do we use this?
There are specific use cases where this trick comes in handy:
- Adhoc code execution: when DBT is expected to run a specific logic only on a special occasion or for a bug that doesn’t occur too often. An if-else block here could house code for both the usual as well as the Adhoc logic.
- Re-instating only certain parts of a dataset: This could be useful to re-instate only a specific section or partition of the table to cut down on costs or while dealing with with large datasets.