About me
Hello I’m Hugo Lu, a Data Engineer who’s also worked in Finance and now CEO@ Orchestra. Orchestra is a data release pipeline tool hat helps Data Teams release data into production reliably and efficiently. I write about what good looks like in Data.
Introduction
In the last part of Ingestion I covered a few ways of thinking about how to structure your data ingestion. We saw there are two main factors to consider: speed / latency and throughput / volume. There are also considerations around the destination you’re using — sending data to a data lake is different to moving it to a data warehouse for example. In this article, we’ll cover some different technical ways of achieving some of these methods, and we’ll ignore streaming for now which deserves its own topic.