Ubuntu 24.04 Docker Watchtower Problems

By hernil

August 9, 2024

If you’ve seen this kind of output from your Watchtower logs recently you might have ended up in the right place.

Error response from daemon: Cannot kill container: af791ab8aad584fa7488eba4cdad91ab7252d9f2cb04041c2a36ce80ae546145: 
Cannot kill container af791ab8aad584fa7488eba4cdad91ab7252d9f2cb04041c2a36ce80ae546145: 
unknown error after kill: runc did not terminate successfully: 
exit status 1: unable to signal init: permission denied

The problem

Watchtower is granted some access to the Docker socket which it uses to send signals that interact with running containers. Spesifically it needs to stop or kill running containers that use outdated images so that it can spin them up again fully updated. Apparmor is used to "confine programs to a limited set of resources." ¹ Somehow there are Apparmor restrictions set on a component of Docker or it’s accompanying tools (the culprit seems to be confinement of runc which is the default Docker container runtime) that seem to block the signals Watchtower are trying to send to running containers.

The solution

NOTE:

I use Docker as provided in the base Ubuntu apt repository. That means I install it using sudo apt install docker.io. This is not the reccomended way according to Docker itself². For me I prefer trading away bleeding edge functionality for the extra layer of integration and testing that using the distro packages allow. Yes, I’m aware of how ironic typing this out in this post is :-) And no, I have not tested whether this fix applies to (or is even needed for) the packages provided directly from upstream Docker.

Someone in the comment thread for this bug that knows a whole lot more about Apparmor than I do has suggested a temporary fix in the form of an Apparmor profile for Docker. I’ll mention that I cannot wouch for this configuration. It solves the mentioned problem, but it could also disable or modify other parts of Apparmor that I am not aware of. Apparmor is a pretty important part of the system security so apply this at your own risk.

#include <tunables/global>


profile docker-default flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer=unconfined,
  # runc may send signals to container processes
  signal (receive) peer=runc,
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read,tracedby,readby) peer=docker-default,
}

Now restart Apparmor so the new profile is loaded.

sudo systemctl restart apparmor.service

And you should probably restart the Docker service as well. I did, just to be sure.

sudo systemctl restart docker.service

Hopefully Watchtower should now just work again.

Input or feedback to this content? Reply via email!

The problem

The solution

Related Articles