Failed to extract xcom from airflow pod - Kubernetes Pod Operator

Question

While running a DAG which runs a jar using a docker image,
xcom_push=True is given which creates another container along with the docker image in a sing

Answer 1

I want to point out the error I faced regarding xcom and KubernetesPodOperator although it was not the same cause as the OP. Just in case anyone stumbles on this question since this is the only one regarding KPO and XCom.

I am using Google Cloud Platform (GCP) Cloud Composer, it uses a slightly older than latest Airflow version, hence when i referred to official GitHub, it mentions to use do_xcom_push whereas the old Airflow uses the arg xcom_push instead!

Answer 2

If xcom_push is True then KubernetesPodOperator creates one more sidecar container (airflow-xcom-sidecar) in Pod along with the base container(actual worker container). This sidecar container reads data from /airflow/xcom/return.json and returns as xcom value. So in your base container you need to write the data you want to return in /airflow/xcom/return.json file.

Answer 3

This happened because the result of the task execution is not being pushed to the xcom in the expected path required by the KubernetesPodOperator plugin. Take a look at the following unit test from the Airflow repository to check how it should be implemented (source code snippet included below for your convenience, followed by the link to the repository):

    def test_xcom_push(self):
        return_value = '{"foo": "bar"\n, "buzz": 2}'
        k = KubernetesPodOperator(
            namespace='default',
            image="ubuntu:16.04",
            cmds=["bash", "-cx"],
            arguments=['echo \'{}\' > /airflow/xcom/return.json'.format(return_value)],
            labels={"foo": "bar"},
            name="test",
            task_id="task",
            xcom_push=True
        )
        self.assertEqual(k.execute(None), json.loads(return_value))

https://github.com/apache/incubator-airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321

edit: it is worth mentioning that the result pushed to the xcom must be a json.