Securing the Docker Ecosystem: Part 1: Strategies to Secure the Docker Daemon

Recently, two key advancements, the acceleration of the software development cycle and the increasing complexity of the application stack, have triggered the need for faster and easier ways of pushing code into production. In this context, more lightweight, flexible, and resource-efficient approaches, such as “containers,” have become increasingly popular. To this end, more and more organizations have adopted Docker technology as the de facto standard in their application container space.

This is the first article of a three-part series on improving the security posture of your Docker ecosystem. If you haven’t read the other articles yet, here are the links to the second and third blog posts.

Why Docker?

Docker enables the creation of containers that can hold what’s required for a discrete application or technology stack. It is written in the Go programming language and leverages many features of the Linux kernel to deliver its functionalities. The container lifecycles can be managed easily with tooling available on the Docker platform.

With the help of containers, developers can bundle an application with its code, system libraries, runtime, and tools into a standalone unit capable of running on any platform. Since a stack of required components can be easily assembled, run, and even destroyed as needed, the system can be kept uncluttered of packages and data that are no longer needed. Containers also make it easy to manage workloads dynamically; they enable greater application portability and speed up application delivery.

Unlike Virtual Machines, containers do not embed their own kernel but run directly on the host machine’s kernel. They can share other system resources to manage interactions between the containers and the host machine.

This combination of characteristics culminates in a lightweight, shareable, and versatile tool that inherently promotes an agile and secure posture, enabling developers to evade many problems associated with code duplication.

Unfortunately, some of the characteristics that make Docker containers so popular with developers during development and testing also raise concerns, particularly with regard to their security. The reality is that with new technology - Docker and container technologies are only a few years old - emergent security issues are not only unsurprising but seriously challenging as well.

Concerns have been raised about the security properties of containers, as well as the manner in which they are accessed, used, managed and administered, especially in production environments. The most pressing requirements from the community of users regard more effective articulation of optimal and safe Docker use; who can access containers, who is allowed to do what when a container is running, and who can add networking or load balancing.

Courtesy of the cooperative efforts of the global security community, many of these problems have been addressed by the emergence of open-source and proprietary tools that articulate security in their automated deployment and management of containers - Kubernetes being a great example - however, there is plenty more work to do.

A Brief Intro to Docker Daemon

On the whole, the architecture of a Docker ecosystem is complex, with containers, repositories, and orchestrators, as well as a high level of automation. Docker uses a client-server architecture, where the Docker client talks to the Docker daemon using a REST API over UNIX sockets or via a network interface. The daemon (dockerd) does all the heavy lifting of building, running, and distributing Docker containers. It runs on the host operating system, listens for Docker API requests, and manages Docker objects like images, containers, networks, and volumes. Daemons can communicate with each other to manage Docker services.

Within the Docker architecture, vulnerabilities can be introduced from multiple directions. Therefore, it serves to pay attention to the configuration of the engine, the build-time, and the runtime setups of the Docker environment. This means that you can either use them in your continuous integration pipelines or as the “bare bones” of your production infrastructure.

Attackers will attempt to leverage the complicated maze of controls and potential misconfigurations to escalate privileges, break out of isolation, compromise the underlying host, and move laterally within and across the network. This is why it’s essential to be prepared, to map all the weakest points within your infrastructure requiring special maintenance, and to ensure that your environment is as robust as it can be.

In this new series of articles, we explore the security implications of the three main aspects of the Docker ecosystem: i) the daemon, ii) the build phase, and iii) the runtime. Each article will explore one aspect in detail and provide useful strategies to help you boost the security of that aspect of your organization’s Docker ecosystem.

In this first article, we talk about some effective strategies to mitigate the security concerns around the Docker Daemon. In part two, we will discuss how to improve the security of container builds. The final part will address the security of container runtime. To strengthen the overall security of your Docker ecosystem, make sure you bookmark all three articles!

10 Proven Strategies to Secure Your Docker Daemon

The Docker daemon is the pivotal point of the Docker architecture. That’s why it’s critical to improve its security posture by reviewing its rich settings and tuning its configuration.

Here are ten proven strategies to improve the defense posture of the Docker daemon in your organization.

1. Keep your version up-to-date

As with any other software that runs in your system, staying current on updates reduces the risk of being compromised by bad actors who (may) gain unauthorized access to your Docker ecosystem. A recent example of such an event is the runC vulnerability, published in 2019. This vulnerability allowed a malicious container to break out of isolation, thus gaining execution rights over the host. Luckily, the problem was quickly identified and patched in version 18.09.2.

Monitor the Docker releases and make sure to use the most up-to-date version.

$ docker version

2. Restrict access to the daemon

Only administrative users should have access to the Docker daemon.

The Docker daemon mostly runs with root privileges (the experimental rootless mode that is now available in the latest versions is discussed later in this article). If an attacker, either positioned in a container or on the host, gets access to the Docker daemon, they can abuse it to escalate their privileges and quickly take full control of the host system.

In the most common configuration, the UNIX users of the group called “docker” can use the daemon. Allowing someone into the “docker” group is equivalent to giving them root access to the underlying operating system. Therefore, you should carefully review the members of the docker group and remove any untrusted users.

$ getent group docker

3. Protect the UNIX socket from unintended access

The non-networked /var/run/docker.sock UNIX socket is used by default to locally access the Docker Engine API. The safe default settings on Linux include a socket file owned by the “root” user and a “docker” group allowing solely “root” read and write file permission (ug+rw).

$ ls -l /var/run/docker.sock

4. Do not use an unencrypted network socket

Never expose the Docker daemon’s TCP port without enabling authentication of the clients via TLS.

It is conventional to use port 2375 for un-encrypted/un-authenticated communication and port 2376 for encrypted communication with the daemon. Anyone who can reach the unencrypted port also has full access to the Docker daemon and hence has full access to the hosts’ operating system. Even if inbound connections are blocked by a firewall, the daemon is still prone to Server-Side Request Forgery attacks, Privilege Escalation, and Container Breakout attacks staged from within compromised containers.

A developer who only exposes the unencrypted/unauthenticated port to localhost on the local workstation might appear to be part of a safe setup, but this actually makes the system prone to Cross-Site Request Forgery attacks. This dangerous default setting existed in Docker CE for Windows until the release of version 0.5.2.

$ cat /etc/docker/daemon.json

If you need to open a Docker server to TCP connections, protect the daemon by only allowing TLS to connect with clients authenticated by a certificate signed by that CA as explained in the official Docker documentation.

If you need to develop on a remote Docker server, learn how to use the “ssh://” URL scheme in your local DOCKER_HOST environment variable without exposing the server on the network (see the documentation).

5. Protect the key files

The Docker daemon depends on a number of key files and directories that should be kept safe from unauthorized access, some of which have already been discussed in the previous paragraphs. Because this daemon runs with root privileges, check that the Docker files are owned by root and can’t be manipulated by any other user.

$ sudo ls -ld \
    /usr/sbin/runc \
    /usr/bin/containerd \
    /var/run/docker.sock \
    /etc/sysconfig/docker \
    /etc/default/docker \
    /etc/docker \
    /var/lib/docker

$ sudo find /usr /etc \
    -name docker.service -or \
    -name docker.socket \
    -ls 2>/dev/null

Optionally, you can also configure the Linux Auditing System to audit the files above and report any suspicious activity.

6. Use the “rootless” mode

When possible, run the Docker daemon in rootless mode.

Rootless mode is an experimental feature introduced in Docker Engine version 19.03 to run the Docker daemon and containers as a non-root user. Docker’s previous solution used the daemon’s userns-remap option, which takes advantage of user namespaces to map the root user in the inner namespace to an unprivileged range in the parent namespace, reducing the potential security exposure at container runtime.

The rootless mode works in a similar way, except that the Docker daemon also runs as a non-root user. This defuses most of the Privilege Escalation threat scenarios and mitigates the impact of attackers breaking out of the containerized environment. It also enables multiple per-user Docker environments to run in the same shared host.

Read more about this neat functionality here.

7. Ensure a global seccomp profile is enabled

Make sure the default seccomp profile is enabled.

Seccomp is a Linux kernel feature used to restrict the system calls that can be made from a process. It is used by Docker as a very low-level filter that reduces the kernel’s surface area accessible by the containers.

A custom seccomp profile can be applied by using the daemon’s seccomp-profile option, but in most cases, using the default option is adequate. Custom per-container seccomp profiles can be also applied at runtime.

$ docker info --format '{{ .SecurityOptions }}'

Read more about how Docker uses seccomp here.

8. Consider applying a SELinux policy for Docker containers

Security-Enhanced Linux (SELinux) is a Linux kernel security module that allows the use of Mandatory Access Controls (MAC) on system resources.

If you want to apply a policy to your containers, you should enable it at the host level and then configure the selinux-enabled option in the daemon.

After enabling it at the system level, custom per-container SELinux policies can be applied at runtime. Some Linux distributions, such as Red Hat, come with SELinux policies for Docker.

$ docker info --format '{{ .SecurityOptions }}'

9. Use secure registries to pull and push images

Insecure registries do not use TLS, nor do they have an invalid TLS certificate. Such registries should not be used as they are prone to man-in-the-middle (MITM) attacks by malicious actors who are able to modify the network traffic.

$ docker info --format '{{.RegistryConfig.InsecureRegistryCIDRs}}'

10. Consider disabling inter-container communication

If not required, you should restrict the network traffic between containers.

By default, unrestricted network traffic is enabled amongst all containers on the same host in the default network bridge. However, a bad actor positioned in a compromised container could leverage this functionality to abuse other services exposed by other containers within the container network on the same host.

To disable the inter-container communication, configure the daemon with the icc flag set to false. Note that this configuration can be overridden by containers that are run with the deprecated --link option.

$ docker network ls -q | xargs docker network inspect -f '{{ .Name }}: {{ .Options }}'

You should also consider using user-defined bridge networks to enforce network isolation among containers as a more flexible solution.

Docker Daemon Security Starts with Developer Training

The technical strategies we have discussed in this article are proven to work in the real world. However, if your developers don’t have the proper training, it’s unlikely they’ll be able to implement secure Docker architecture effectively since they won’t know which issues to look out for or how to address them effectively. As the threat landscape continues to expand, your organization needs a robust defense strategy, where defensive programming methodologies and best practices are a cornerstone in the SDLC. However, old-fashioned AppSec training programs with little or no focus on real-world threat scenarios are simply inadequate and potentially downright dangerous, proffering a false sense of security through compliance, not actual security.

SecureFlag’s 100% practical training combines the best of in-class training with computer-based training to provide a robust solution for modern, security-conscious organizations. After completing their training with SecureFlag, your developers and software engineers will be able to confidently develop secure applications. Equally important, the organization will strengthen its SDLC toolkit, minimize remediation costs, and boost its market competitiveness.

For more information about SecureFlag’s unique “hands-on” AppSec training approach, get in touch with us today.