Choosing the right ETL tool or orchestration engine is critical for building efficient data pipelines. This article presents a practical benchmark of eight popular solutions, including Airflow, using a standardized environment: CentOS 7.9, Xeon Gold 5218, 64GB RAM, with a MySQL to ClickHouse data pipeline processing 10 million rows. The author evaluates each tool on performance, scalability, ease of use, and ecosystem integration. Key findings include performance comparisons for batch processing, scheduling capabilities, and operational complexity. For data engineers and architects, this benchmark provides actionable insights to make informed decisions when selecting tools for their data infrastructure. The article also discusses trade-offs between different approaches, such as Airflow's scheduling strengths versus its limitations as a pure sync tool.
Hands-on benchmark of 8 ETL tools and schedulers with real-world data, covering performance, scalability, and ease of use.