Data ingestion remains one of the most debated topics in the data engineering community. Despite the rise of managed services like Fivetran and open-source alternatives like Airbyte, many teams still struggle to achieve reliable, scalable ingestion pipelines. This post examines the core reasons: schema evolution complexities, API rate limits, and the mismatch between declarative configurations and real-world data sources. It argues that no single tool fully addresses the diversity of data sources and the need for custom transformations. For data leaders, this means investing in a flexible ingestion layer that combines best-of-breed tools with in-house expertise. The signal is especially relevant as organizations scale their data platforms and face the limits of off-the-shelf solutions.
A deep dive into why data ingestion remains a challenge despite Fivetran and Airbyte, with insights for production data stacks.