问题
I am developing a Node multi-container application on a Linux Mint Docker host, with Docker and Docker Compose installed via Snapcraft. There are four containers, and two of them have bind-mount volumes against project folders on the host. This is all pretty standard stuff.
A couple of days ago, some of the containers started to fail with permission errors. Here is my Docker Compose command, and the logs from the failing containers:
$ docker-compose -f docker-compose.yml -f docker-compose-dev.yml up --no-build
Creating network "frontend_default" with the default driver
Creating frontend_mysql_1 ... done
Creating frontend_reverse-proxy_1 ... done
Creating frontend_api_1 ... done
Creating frontend_frontend_1 ... done
Attaching to frontend_reverse-proxy_1, frontend_mysql_1, frontend_api_1, frontend_frontend_1
api_1 | npm ERR! code EACCES
api_1 | npm ERR! syscall open
api_1 | npm ERR! path /root/.config/configstore/update-notifier-npm.json
api_1 | npm ERR! errno -13
api_1 | npm ERR! Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'
api_1 | npm ERR! You don't have access to this file.
api_1 | npm ERR!
api_1 | npm ERR! at Object.openSync (fs.js:440:3)
api_1 | npm ERR! at Object.readFileSync (fs.js:342:35)
api_1 | npm ERR! at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)
api_1 | npm ERR! at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)
api_1 | npm ERR! at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)
api_1 | npm ERR! at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)
api_1 | npm ERR! at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)
api_1 | npm ERR! at processTicksAndRejections (internal/process/task_queues.js:76:11)
api_1 | npm ERR! Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'
api_1 | npm ERR! You don't have access to this file.
api_1 | npm ERR!
api_1 | npm ERR! at Object.openSync (fs.js:440:3)
api_1 | npm ERR! at Object.readFileSync (fs.js:342:35)
api_1 | npm ERR! at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)
api_1 | npm ERR! at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)
api_1 | npm ERR! at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)
api_1 | npm ERR! at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)
api_1 | npm ERR! at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)
api_1 | npm ERR! at processTicksAndRejections (internal/process/task_queues.js:76:11) {
api_1 | npm ERR! stack: "Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'\n" +
api_1 | npm ERR! "You don't have access to this file.\n" +
api_1 | npm ERR! '\n' +
api_1 | npm ERR! ' at Object.openSync (fs.js:440:3)\n' +
api_1 | npm ERR! ' at Object.readFileSync (fs.js:342:35)\n' +
api_1 | npm ERR! ' at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)\n' +
api_1 | npm ERR! ' at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)\n' +
api_1 | npm ERR! ' at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)\n' +
api_1 | npm ERR! ' at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)\n' +
api_1 | npm ERR! ' at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)\n' +
api_1 | npm ERR! ' at processTicksAndRejections (internal/process/task_queues.js:76:11)',
api_1 | npm ERR! errno: -13,
api_1 | npm ERR! syscall: 'open',
api_1 | npm ERR! code: 'EACCES',
api_1 | npm ERR! path: '/root/.config/configstore/update-notifier-npm.json'
api_1 | npm ERR! }
api_1 | npm ERR!
api_1 | npm ERR! The operation was rejected by your operating system.
api_1 | npm ERR! It is likely you do not have the permissions to access this file as the current user
api_1 | npm ERR!
api_1 | npm ERR! If you believe this might be a permissions issue, please double-check the
api_1 | npm ERR! permissions of the file and its containing directories, or try running
api_1 | npm ERR! the command again as root/Administrator.
api_1 |
api_1 | npm ERR! A complete log of this run can be found in:
api_1 | npm ERR! /root/.npm/_logs/2020-05-26T12_31_11_538Z-debug.log
api_1 | mysql_1 | /bin/bash: /usr/local/bin/docker-entrypoint.sh: Permission denied
reverse-proxy_1 | /bin/sh: can't open '/entrypoint.sh': Permission denied
frontend_reverse-proxy_1 exited with code 2
frontend_1 | npm ERR! code EACCES
frontend_1 | npm ERR! syscall open
frontend_1 | npm ERR! path /root/.config/configstore/update-notifier-npm.json
frontend_1 | npm ERR! errno -13
frontend_1 | npm ERR! Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'
frontend_1 | npm ERR! You don't have access to this file.
frontend_1 | npm ERR!
frontend_1 | npm ERR! at Object.openSync (fs.js:440:3)
frontend_1 | npm ERR! at Object.readFileSync (fs.js:342:35)
frontend_1 | npm ERR! at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)
frontend_1 | npm ERR! at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)
frontend_1 | npm ERR! at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)
frontend_1 | npm ERR! at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)
frontend_1 | npm ERR! at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)
frontend_1 | npm ERR! at processTicksAndRejections (internal/process/task_queues.js:76:11)
frontend_1 | npm ERR! Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'
frontend_1 | npm ERR! You don't have access to this file.
frontend_1 | npm ERR!
frontend_1 | npm ERR! at Object.openSync (fs.js:440:3)
frontend_1 | npm ERR! at Object.readFileSync (fs.js:342:35)
frontend_1 | npm ERR! at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)
frontend_1 | npm ERR! at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)
frontend_1 | npm ERR! at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)
frontend_1 | npm ERR! at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)
frontend_1 | npm ERR! at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)
frontend_1 | npm ERR! at processTicksAndRejections (internal/process/task_queues.js:76:11) {
frontend_1 | npm ERR! stack: "Error: EACCES: permission denied, open '/root/.config/configstore/update-notifier-npm.json'\n" +
frontend_1 | npm ERR! "You don't have access to this file.\n" +
frontend_1 | npm ERR! '\n' +
frontend_1 | npm ERR! ' at Object.openSync (fs.js:440:3)\n' +
frontend_1 | npm ERR! ' at Object.readFileSync (fs.js:342:35)\n' +
frontend_1 | npm ERR! ' at Configstore.get all [as all] (/usr/lib/node_modules/npm/node_modules/configstore/index.js:30:25)\n' +
frontend_1 | npm ERR! ' at Configstore.get (/usr/lib/node_modules/npm/node_modules/configstore/index.js:74:27)\n' +
frontend_1 | npm ERR! ' at UpdateNotifier.check (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:78:16)\n' +
frontend_1 | npm ERR! ' at module.exports (/usr/lib/node_modules/npm/node_modules/update-notifier/index.js:151:17)\n' +
frontend_1 | npm ERR! ' at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/bin/npm-cli.js:85:48)\n' +
frontend_1 | npm ERR! ' at processTicksAndRejections (internal/process/task_queues.js:76:11)',
frontend_1 | npm ERR! errno: -13,
frontend_1 | npm ERR! syscall: 'open',
frontend_1 | npm ERR! code: 'EACCES',
frontend_1 | npm ERR! path: '/root/.config/configstore/update-notifier-npm.json'
frontend_1 | npm ERR! }
frontend_1 | npm ERR!
frontend_1 | npm ERR! The operation was rejected by your operating system.
frontend_1 | npm ERR! It is likely you do not have the permissions to access this file as the current user
frontend_1 | npm ERR!
frontend_1 | npm ERR! If you believe this might be a permissions issue, please double-check the
frontend_1 | npm ERR! permissions of the file and its containing directories, or try running
frontend_1 | npm ERR! the command again as root/Administrator.
frontend_1 |
frontend_1 | npm ERR! A complete log of this run can be found in:
frontend_1 | npm ERR! /root/.npm/_logs/2020-05-26T12_31_12_821Z-debug.log
frontend_1 | frontend_mysql_1 exited with code 126
I am not sure what would cause that, as nothing major has changed on the dev machine. I did some rooting around, and found that AppArmor had started failing. Here's some logs from dmesg on the host:
May 26 13:31:15 dev-VirtualBox kernel: [ 136.787188] audit: type=1400 audit(1590496275.941:68): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-nodemon.json" pid=6516 comm="node" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:16 dev-VirtualBox kernel: [ 137.403740] audit: type=1400 audit(1590496276.557:69): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-npm.json" pid=6252 comm="npm" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:17 dev-VirtualBox kernel: [ 137.959766] audit: type=1400 audit(1590496277.113:70): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-nodemon.json" pid=6683 comm="node" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:17 dev-VirtualBox kernel: [ 138.388630] audit: type=1400 audit(1590496277.541:71): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=6561 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:19 dev-VirtualBox kernel: [ 140.765452] audit: type=1400 audit(1590496279.917:72): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-npm.json" pid=6861 comm="npm" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:20 dev-VirtualBox kernel: [ 140.975698] audit: type=1400 audit(1590496280.129:73): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-npm.json" pid=6942 comm="npm" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:20 dev-VirtualBox kernel: [ 140.996244] audit: type=1400 audit(1590496280.149:74): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=7082 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:20 dev-VirtualBox kernel: [ 141.399142] audit: type=1400 audit(1590496280.553:75): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-nodemon.json" pid=7162 comm="node" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:20 dev-VirtualBox kernel: [ 141.521377] audit: type=1400 audit(1590496280.673:76): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/root/.config/configstore/update-notifier-nodemon.json" pid=7185 comm="node" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:24 dev-VirtualBox kernel: [ 145.403602] audit: type=1400 audit(1590496284.557:77): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=7312 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:32 dev-VirtualBox kernel: [ 153.298729] audit: type=1400 audit(1590496292.453:78): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=7452 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:31:46 dev-VirtualBox kernel: [ 166.995319] audit: type=1400 audit(1590496306.149:79): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=7610 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:32:12 dev-VirtualBox kernel: [ 193.228513] audit: type=1400 audit(1590496332.381:80): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/usr/local/bin/docker-entrypoint.sh" pid=7788 comm="docker-entrypoi" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 26 13:32:55 dev-VirtualBox kernel: [ 236.569562] audit: type=1400 audit(1590496375.721:81): apparmor="DENIED" operation="open" profile="snap.docker.compose" name="/proc/7907/mounts" pid=7907 comm="python3" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
So, it looks like an AppArmor problem. It is possible that AppArmor has had a system update, but I have not changed it specifically, as far as I know. What can I do to restore normal Docker operation?
回答1:
This problem was quite hard to track down, as there does not seem to be much information about for it. My sense is that it affects Docker only in Snap, and not Docker installed in other ways. The problem is discussed in this forum thread.
The problem stems from a buggy kernel, in my case 5.3.0-53, and dropping back down to the previously installed version solves the problem. For me that is 5.3.0-51. This bug report indicates that the problem is also exhibited in 5.4.0-31, and that can be fixed by dropping back to 5.4.0-29. This explains how the problem manifested on its own - a new kernel was delivered via a system update.
I used this answer to amend Grub so it boots into an older kernel. Here are the steps to take for readers with the same issue:
List your currently kernels, so you can identify the previously installed kernel:
dpkg -l linux-{image,headers}-"[0-9]*" | awk '/ii/{print $2}'
Confirm which kernel is the problem:
uname -r
You will need to change the boot option in /etc/default/grub, specifically a value called GRUB_DEFAULT. This is normally 0 to mean "boot latest kernel":
GRUB_DEFAULT=0
You need to change this so it points to a specific kernel. My value is thus:
GRUB_DEFAULT="Advanced options for Linux Mint 19.3 Cinnamon>Linux Mint 19.3 Cinnamon, with Linux 5.3.0-51-generic"
These are menu strings, which we're asking Grub to auto-select for us. To discover what yours are, have a look in /boot/grub/grub.cfg, and search for the keyword submenu to find the top-level element, i.e. "Advanced options for Linux Mint 19.3 Cinnamon" in this case. From that point, search for menuentry to find the second-level element, which is "Linux Mint 19.3 Cinnamon, with Linux 5.3.0-51-generic" in this case. There are many submenus - you need the version for the penultimate version of the kernel.
Add these two string together, put a > between them, wrap them in quotes, and use that as your GRUB_DEFAULT value. As per the useful answer I linked to earlier, it is sensible to take a backup of your Grub file before rebooting, in case you get something wrong:
sudo cp /etc/default/grub /etc/default/grub.bak
Finally, you should regen the Grub menu after your changes:
sudo update-grub
Once you reboot, you should find that Docker is back to normal. It is to be hoped that a new kernel will be released, at which point Grub will need to be manually switched back to track the latest version (assuming it shall be fixed in the next Linux release).
来源：https://stackoverflow.com/questions/62022953/various-docker-container-paths-have-started-failing-with-permission-errors-on-li