Now Apache Airflow is a platform that is used to programmatically authoring, scheduling, and monitoring workflows. It basically solved the issues that come with long-running cron tasks that execute bulky scripts. Needless to say that this ended up being far more than a lifesaver. The came along Data Engineer Maxime Beauchemin, who created an open-sourced Airflow with the idea that it would allow them to quickly author, iterate on, and monitor their batch data pipelines. Just as how you would get more hands to bail out water pouring into a ship to remain afloat. A dedicated workforce tasked with creating automated processes by writing scheduled batch jobs. This meant more data engineers, data scientist and analyst. There was massive expansion, which is always a good thing, but so was the volume of data they had to deal with. In 2015, a company we all know now, Airbnb, encountered an issue. To understand Airflow we must first start with the problem that saw to its creation.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |