how to configure some external jars library to the flink docker container

问题

I am running a flink docker image with the following configuration.

version: '2.1'
services:
  jobmanager:
    build: .
    image: flink
    volumes:
      - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    hostname: "jobmanager"
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink
    volumes:
    - .:/usr/local/lib/python3.7/site-packages/pyflink/lib
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - jobmanager:jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

the running log is as below

    taskmanager_1  | 2020-10-11 10:34:03,714 INFO  org.apache.flink.runtime.taskexecutor.TaskManagerRunner      [] -  Classpath: 
  /opt/flink/lib/flink-csv-1.11.0.jar:
/opt/flink/lib/flink-json-1.11.0.jar:
/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:
/opt/flink/lib/flink-table-blink_2.12-1.11.0.jar:
/opt/flink/lib/flink-table_2.12-1.11.0.jar:
/opt/flink/lib/log4j-1.2-api-2.12.1.jar:
/opt/flink/lib/log4j-api-2.12.1.jar:
/opt/flink/lib/log4j-core-2.12.1.jar:
/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:
/opt/flink/lib/flink-dist_2.12-1.11.0.jar:::

from the logs you can see some default libraries loaded into the system, but I want to add some jars like flink-jdbc_2.11-1.9.0.jar, which is in my local filesystem.

my local jar folder is /usr/local/lib/python3.7/sitepackages/pyflink/lib

I tried

volumes:
  - /opt/flink/lib/:/usr/local/lib/python3.7/site-packages/pyflink/lib

the error is

ERROR: for keras-flask-deploy-webapp-master_jobmanager_1  Cannot start service jobmanager: Mounts denied: 
The path /opt/flink/lib is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.

I tried this but still not working

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib

[ERROR] Flink distribution jar not found in /opt/flink/lib.
taskmanager_1  | [ERROR] The execution result is empty.

How would I link the jdbc jar library to the flink 's docker container?

回答1:

The workaround in this case can be to attach the volume with your specific jars to some temporary location in container and override the run command to copy the attached files into the /opt/flink/lib flink classpath folder:

volumes:
  - /usr/local/lib/python3.7/site-packages/pyflink/lib:/opt/flink/lib-custom
command: cp -a /opt/flink/lib-custom/. /opt/flink/lib/ && jobmanager

I would say tweaking jobmanager container would be enough if you run the application from there, but if not - repeat the same in taskmanager as well.

来源：https://stackoverflow.com/questions/64303382/how-to-configure-some-external-jars-library-to-the-flink-docker-container

标签

Docker

containers

apache-flink