问题
We are using Artifactory pro 5.10.4 version. We are migating from a simple installation on a VM and apache reverse proxy for docker registries to docker swarm cluster. We are convinced that Traefik is the best alternative when it comes to dynamic configuration. We are facing some docker push with traefik. Our compose file for Artifactory service is:
version: '3.7'
services:
artifactory:
image: docker.bintray.io/jfrog/artifactory-pro:5.10.4
environment:
EXTRA_JAVA_OPTIONS : >
'-Xms2g'
'-Xmx2g'
'-Xss256k'
'-XX:+UseG1GC'
'-Dcom.sun.management.jmxremote'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Djava.util.logging.config.file=/var/opt/jfrog/artifactory/etc/logging.properties'
volumes:
- /mnt/nfs-dev/artifactory:/var/opt/jfrog/artifactory
- /data/artifactory/cache/filestore:/var/opt/jfrog/artifactory/cache/filestore
- /data/artifactory/logs:/var/opt/jfrog/artifactory/logs
networks:
- proxy
ports:
- 8081:8081
deploy:
labels:
- traefik.enable=true
- traefik.http.routers.artifactory.rule=Host(`artifactory.example.com`) && (PathPrefix(`/artifactory`))
- traefik.http.routers.artifactory.tls=true
- traefik.http.routers.artifactory.entrypoints=websecure
- traefik.http.routers.artifactory.service=artifactory
- traefik.http.services.artifactory.loadbalancer.server.port=8081
- traefik.http.routers.nx-docker-router.rule=Host(`docker-test.example.com`)
- traefik.http.routers.nx-docker-router.tls=true
- traefik.http.routers.nx-docker-router.entrypoints=websecure
- traefik.http.routers.nx-docker-router.middlewares=testheader
- traefik.http.routers.nx-docker-router.middlewares=docker-redirect
- traefik.http.middlewares.docker-redirect.redirectregex.regex=^https://docker-test.example.com/(v1|v2)/(.*)
- traefik.http.middlewares.docker-redirect.redirectregex.replacement=/artifactory/api/docker/docker-repo/$$1/$$2
- traefik.http.middlewares.testheader.headers.customrequestheaders.X-Forwarded-Port=443
- traefik.http.middlewares.testheader.headers.customrequestheaders.X-Forwarded-Proto=https
- traefik.http.middlewares.testheader.headers.customrequestheaders.X-Artifactory-Override-Base-Url=https://docker-test.example.com/artifactory
- traefik.http.routers.nx-docker-router.service=nx-docker-service
- traefik.http.services.nx-docker-service.loadbalancer.server.port=8081
mode: replicated
replicas: 1
restart_policy:
condition: any
networks:
proxy:
external: true
We have a configuration that points artifactory.example.com=docker-test.example.com=docker swarm manager node. The name of the repo in Artifactory is "docker-repo". With the configuration above docker pull/login works fine except for docker push. The push ends with the error:
The push refers to repository [docker-test.my-nx.com/openjdk]
1467e455c6a3: Pushing [==================================================>] 335.7MB/335.7MB
042ae34a965b: Pushing [==================================================>] 40.12MB/40.12MB
d4e82c28f86e: Pushing [==================================================>] 148.4MB/148.4MB
EOF
the /var/log/message file indicates:
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.810434220+02:00" level=debug msg="Pushing layer: sha256:1467e455c6a34d120f1f6be534a403eea4a6c01b707135958ac0fba6e344a1e7"
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.823999807+02:00" level=debug msg="Pushing layer: sha256:042ae34a965b67022e809160737c63873e241d4931908b7b8283cdfb918eb269"
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.841103402+02:00" level=debug msg="activateDeviceIfNeeded(c18f9432f75625f1dfadf6275537e28229a335013cb42d9cf77fcd17e1de84a2)" storage-driver=devicemapper
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.841171352+02:00" level=debug msg="cancelDeferredRemovalIfNeeded START(docker-253:1-69978887-c18f9432f75625f1dfadf6275537e28229a335013cb42d9cf77fcd17e1de84a2)" storage-driver=devicemapper
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.841258883+02:00" level=debug msg="cancelDeferredRemovalIfNeeded END(docker-253:1-69978887-c18f9432f75625f1dfadf6275537e28229a335013cb42d9cf77fcd17e1de84a2)" storage-driver=devicemapper
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.843697947+02:00" level=debug msg="Pushing layer: sha256:d4e82c28f86e452aed539f62d03baca2a2efd3637e46d184514255a66703e3e1"
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.861477572+02:00" level=error msg="Upload failed: EOF"
Oct 15 15:02:38 test-docker dockerd: time="2020-10-15T15:02:38.861563681+02:00" level=info msg="Attempting next endpoint for push after error: EOF"
The traefik access.log shows:
=example.com HTTP/1.1" 200 103 "-" "-" 24196 "nx-docker-router@docker" "http://10.1.7.245:8081" 1167ms
10.0.0.2 - - [15/Oct/2020:13:53:46 +0000] "HEAD /artifactory/api/docker/docker-repo/v2/test-alpine/blobs/sha256:46b040a27ad3cb3c9d50a40a0ca1349e94a0742177e0caedea5c22a1fbe42c9a HTTP/1.1" 404 0 "-" "-" 24198 "nx-docker-router@docker" "http://10.1.7.245:8081" 15ms
10.0.0.2 - - [15/Oct/2020:13:53:46 +0000] "HEAD /artifactory/api/docker/docker-repo/v2/test-alpine/blobs/sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c HTTP/1.1" 200 0 "-" "-" 24197 "nx-docker-router@docker" "http://10.1.7.245:8081" 15ms
10.0.0.2 - - [15/Oct/2020:13:53:48 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24202 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:53:51 +0000] "POST /artifactory/api/docker/docker-repo/v2/test-alpine/blobs/uploads/ HTTP/1.1" 202 0 "-" "-" 24204 "nx-docker-router@docker" "http://10.1.7.245:8081" 12ms
10.0.0.2 - - [15/Oct/2020:13:53:53 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24206 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:53:58 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24207 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:54:02 +0000] "POST /artifactory/api/docker/docker-repo/v2/test-alpine/blobs/uploads/ HTTP/1.1" 202 0 "-" "-" 24209 "nx-docker-router@docker" "http://10.1.7.245:8081" 10ms
10.0.0.2 - - [15/Oct/2020:13:54:03 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24211 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:54:08 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24212 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:54:13 +0000] "GET /api/overview HTTP/1.1" 200 452 "-" "-" 24213 "api@internal" "-" 0ms
10.0.0.2 - - [15/Oct/2020:13:54:17 +0000] "POST /artifactory/api/docker/docker-repo/v2/test-alpine/blobs/uploads/ HTTP/1.1" 202 0 "-" "-" 24215 "nx-docker-router@docker" "http://10.1.7.245:8081" 13ms
When everything works fine we should see "PATCH" and "PUT" requests, it's as if the docker client is not sending the PATCH and PUT requests. I have tested a PUT request with curl and it is not filtered by traefik and rearch Artifactory sever and ends with internal error 500 (java exception like shown below) The following curl command:
curl -v -k "Content-Type:application/octet-stream" --location-trusted -H "X-Forwarded-Proto: https" -H "X-Artifactory-Override-Base-Url: http://example.com/artifactory" -X PUT -u user:pwd https://example.com/v2/testgb/blobs/uploads/fe98aa56212374607e41eb7068bd
results in :
2020-10-15 14:01:25,849 [http-nio-8081-exec-14] [ERROR] (o.a.r.c.e.GlobalExceptionMapper:48) - null
java.lang.NullPointerException: null
at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.finishPatchUpload(DockerV2LocalRepoHandler.java:227)
at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlob(DockerV2LocalRepoHandler.java:187)
at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.uploadBlob(DockerV2Resource.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.lambda$doFilter$1(ArtifactoryAuthenticationFilterChain.java:133)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.artifactory.webapp.servlet.authentication.ArtifactoryBasicAuthenticationFilter.doFilter(ArtifactoryBasicAuthenticationFilter.java:84)
at org.artifactory.addon.docker.rest.DockerV2AuthenticationFilter.doFilter(DockerV2AuthenticationFilter.java:206)
at org.artifactory.webapp.servlet.authentication.ArtifactoryAuthenticationFilterChain.doFilter(ArtifactoryAuthenticationFilterChain.java:150)
at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:311)
at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:208)
at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:167)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:74)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:279)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
I also compared headers with apache, there is a little difference:
Traefik http POST headers
X-Forwarded-Host: test-docker-dev.example.com.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: c6a5aca31f36
X-Real-Ip: 10.0.0.2
X-Artifactory-Override-Base-Url:https%3a//test-docker.example.com/artifactory
Apache http POST headers
X-Forwarded-Port:443
X-Forwarded-Proto:https
X-Artifactory-Override-Base-Url:https%3a//test-docker.example.com.com/artifactory
X-Forwarded-For: MY IP
X-Forwarded-Host:test-docker.example.com.com (le serveur sur lequel apache tourne).
X-Forwarded-Server:127.0.0.1
I tried to hardcode some headers like (X-Real-Ip) in traefik using middleware but it doesn't work. I think the big question that we have to understant is why Docker engine is not sending back the "PUT" request after receiving a 202 response and the location header. Is there a way to debug docker engine and have more information than just "level=error msg="Upload failed: EOF"
Any help will be appreciated. Thanks.
来源：https://stackoverflow.com/questions/64373392/docker-push-fail-to-push-to-artifactory-docker-registry-behind-traefik-reverse-p