As data teams scale up on the cloud, data platform teams need to ensure the workloads they are responsible for are meeting business objectives, our main mission here at Sync. At scale with dozens of data engineers building hundreds of production jobs, controlling their performance at scale is untenable for a myriad of reasons from technical to human.
The missing link today is the establishment of a closed loop feedback system that helps automatically drive pipeline infrastructure towards business goals. That was a mouthful, so let’s dive in and get more concrete about this problem.
The problem for data platform teams today
Data platform teams have to manage fundamentally distinct stakeholders from management to engineers. Oftentimes these two teams have opposing goals, and platform managers can be squeezed by both ends.