Multiple services with different dockerfiles on GAE Flexible

Question

I\'m using Google AppEngine Flexible with python environment. Right now I have two services: default and worker that share the same codebase, configured by app.yaml

Answer 1

Since the Dockerfile name cannot be changed, the only way to not have to modify the Dockerfile would be to store each service in its own, separate directory. Clean separation, each service has its own Dockerfile and/or startup configuration.

But this raises a question: how to deal with the code shared by multiple services? Using symlinks (which works great for sharing code across standard env services) doesn't work for the flexible env services, see Sharing code between flexible environment modules in a GAE project.

I see a few possible approaches, none really ideal, but maybe more appealing than what you currently have:

• hard-link each and every shared source code file (since hardlinking directories is not possible). A bit tedious and error-prone, but you only have to do that once per file
• package and publish your shared code as an external library, added to the requirements.txt file of each service using it
• split the shared code in a separate repository and have a copy of that repository in each service using it (maybe as a git submodule if using git?). You just need to ensure at the service deployment time that the shared repository is pulled at the proper version - can be quite reliably done through automation. A bit more complicated if you have uncommited changes in this repo - you'd have to patch the same changes in all services.
• have multiple copies of the Dockerfiles with different names which you simply copy over instead of always editing the same file. Symlinking instead of copying might work as well, since the symlink doesn't need to be followed outside of the service directory, if it's just replicated as a symlink it'll work.

Answer 2

Yes, as you mentioned, I think using the --image-url flag, is a good option here.

Specify a custom runtime. Build the image locally, tag it, and push it to Google Container Registry (GCR) then, deploy your service, specifying a custom service file, and specifying the remote image on GCR using the --image-url option.

Here's an example that accomplishes different entrypoints in 2 services that share the same code: ...this is assuming that the "flex" and not "standard" app engine offering is being used.

lets say you have a: project called my-proj with a default service that is not important and a second service called queue-processor which is using much of the same code from the same directory. Create a separate dockerfile for it called QueueProcessorDockerfile and a separate app.yaml called queue-processor-app.yaml to tell google app engine what i want to happen.

QueueProcessorDockerfile

FROM node:10
# Create app directory
WORKDIR /usr/src/app
COPY package.json ./
COPY yarn.lock ./
RUN npm install -g yarn
RUN yarn
# Bundle app source
COPY . .
CMD [ "yarn", "process-queue" ]

*of course i have a "process-queue" script in my package.json queue-processor-app.yaml

runtime: custom
env: flex
... other stuff...
...

1. build and tag the docker image Check out googles guide here -> https://cloud.google.com/container-registry/docs/pushing-and-pulling docker build -t eu.gcr.io/my-proj/queue-processor -f QueueProcessorDockerfile .
2. push it to GCR docker push eu.gcr.io/my-proj/queue-processor
3. deploy the service, specifying which yaml config file google should use, as well as the image url you have pushed gcloud app deploy queue-processor-app.yaml --image-url eu.gcr.io/my-proj/queue-processor

Answer 3

So i had a very similar issue with my Java applications. We were looking to migrate from Heroku to GAE and were attempting to simulate the Heroku Procfile with GAE services. Effectively what we did was to create separate directories in our application src/main/appengine/web and src/main/appengine/worker where each directory conainted the app.yaml and Dockerfile specific to the process. Then using the mvn appengine:deploy capabilities, we specified the -Dapp.stage.dockerDirectory and -Dapp.stage.appEngineDirecory respectively for each service we wanted to deploy. Then using just some parameters we were able to basically script out parallel deployments of each service from the same code base. Not sure if this works in your situation, but it was very useful for us: Here are the two example commands in their entirety:

Web Process: mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/web -Dapp.stage.appEngineDirectory=src/main/appengine/web -Dapp.stage.stagingDirectory=target/appengine-web -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}

Worker Process: mvn appengine:deploy -Dapp.stage.dockerDirectory=src/main/appengine/worker -Dapp.stage.appEngineDirectory=src/main/appengine/worker -Dapp.stage.stagingDirectory=target/appengine-worker -Dapp.deploy.projectId=${project-id} -Dapp.deploy.version=${project-version}