Tag: Workflows

Amidst the ever-changing technological landscape, some battle-tested principles remain very relevant. The Unix Philosophy is one such guiding framework, offering not just historical context, but actionable insights into personal data management in today’s world. It represents a path ...

A Use Case: Machine Learning (ML) is a dynamic field where models are continuously improved and updated. Consider an ML engineer at a tech company that deploys models for image recognition. Every time the engineer updates or improves a model, they must ensure it meets the required accuracy and pe...

When it comes to orchestration frameworks for data engineering, there are many different options. Airflow is either loved or hated based on who you ask, as I’ve discussed in a previous post. As someone who has used Airflow for close to 6 years now, I haven’t seen enough bad to put m...

Situation: While working on working multiple countries/regions file (with one file dedicated to each country), we encountered the requirement to execute a specific set of data cleaning, validation, transformation and enrichment task for each of these countries. To streamline this process and enable ...