Back to Blog
Airflow kubernetes executor logs7/7/2023 ![]() ![]() ![]() Using KEDA with Airflow KEDA enables elastic from and to zero scaling of Airflow CeleryWorkers on Kubernetes Code samples and configuration can be found here. Before starting the container, a git pull of the dags repository will be performed and. Dags: By storing dags onto persistent disk, it will be made available to all workers. ![]() The Kubernetes executor will create a new pod for every task instance. Message: "Killing container with id docker://airflow-worker:Container failed liveness probe. The KubernetesExecutor natively runs any task in your DAG as a pod on Kubernetes. The kubernetes executor is introduced in Apache Airflow 1.10.0. Looking at the stackdriver logs for GKE I see logs for the container being created, then started, then a bunch of these:įieldPath: "ntainers" We are also seeing issues with the scheduler being stuck in a crash loop and having the workload suspended. We are unable to use our composer environment at the moment because kubernetes is constantly restarting the worker pods. ![]()
0 Comments
Read More
Leave a Reply. |