Apache Airflow or Apache Beam for data processing and job scheduling

Question

I\'m trying to give useful information but I am far from being a data engineer.

I am currently using the python library pandas to execute a long series of transformation

Answer 1

I am doing the same as you with airflow, and I've got very good results. I am not very sure about the following: Beam is machine learning focused and airflow is for anything you want. Finally you can create a hive with kubernetes +airflow.