Welcome back to Securing the Docker Ecosystem - our three-part series on improving the security posture of your Docker ecosystem. In each article, we address one particular aspect of Docker through a security lens.
The first article is about preventing and mitigating potential Docker Daemon security issues, while the third one dives into the Docker runtime.
In this second article, we will focus on Docker container builds and how you can make them more secure by correctly implementing proven procedures. But before we get into the how let’s look at the what: Docker Container Builds and the common security concerns around them.
Security Concerns in Docker Container Build
The build phase involves building Docker images from a Dockerfile and a set of files (called context).
Why is it important to consider security at the build phase? After all, it is the Daemon that is the critical element of the Docker architecture. Right? Well, yes, the Daemon is certainly important, but it is by no means the only important element of the entire Docker ecosystem. In fact, some technical experts go so far as to say that Docker security begins at the image. An image is a read-only template with instructions for creating a Docker container.
The security and quality of Docker images available for assembling applications is a concern because often, developers are unable to verify the origin and the integrity of images available for download, say, from the Docker Hub registry. They’ll initially start by downloading a container base image, and then download and use libraries, and then add their own code… they have accessed a “supply chain” that may well have vulnerabilities in any part of the stack that touches the network. And in the rush to containerisation, they may leave their networks vulnerable to security risks from these insecure images.
“… internal users could download a popular Docker image, insert malicious code or back doors, then re-upload the image with a similar name and gain access to production containers”
– Jay Lyman, 451 Research
In 2015, BanyanOps found that about 40% of Docker images distributed through Docker Hub had high-priority vulnerabilities. At the time, the repository stored about 95,000 Docker images. By June 2020, the size of this repository had grown to 3.5 million images. What does this have to do with the security risks in the Docker build phase?
Consider these facts based on a vulnerability analysis carried out by a trio of computer security researchers from the Norwegian University of Science and Technology (NTNU):
- Out of the 2,500 Docker images they analysed, they found that only 17.8% (430) contained no known vulnerabilities.
- 8 out of 10 “community” images were found to be extremely vulnerable.
- To their surprise, they found that “certified” images were even more vulnerable. These images are supposed to follow recommended best practices, pass a functional API test suite, and complete a vulnerability scanning assessment to make them suitable for enterprise use, but in this case, 82% of them contained at least one high or critical vulnerability.
- “Official” images, albeit the most secure among the four image categories, were also found to be vulnerable, with 45% of them having at least one critical or high vulnerability.
Further, the researchers found the most common and severe vulnerabilities in the JavaScript libraries and Python packages.
Admittedly, the study was limited to a small sample set, but it still delivers a good lesson - don’t ignore the security risks of images in the Docker Container Build! -. Docker and container technologies are only a few years old, so security technologies and best practices - not to mention the documentation around them - are constantly evolving. So instead of relying on them to improve the security posture of your Docker container build, it’s important to follow some security strategies and thumb rules as well.
One of these strategies, the Docker Content Trust (DCT) feature in the Docker Engine, addresses some concerns around image and application container security by allowing developers to verify the integrity and the publisher of all data received from a registry over any channel. Essentially, DCT provides the ability to use digital signatures for data sent to and received from remote Docker registries, so developers can verify image publishers and ensure that images have not been tampered with. This strategy is included towards the end of this article.
Let’s get started with 5 security best practices for building secure Docker images and evaluating third-party image safety and compliance.
5 Strategies to Secure Your Docker Container Build
Keep your images clean
Keeping your images clean and lean will help to increase the maintainability and security of your container pool. As a general rule, make sure that you:
- Use only minimised and trusted parent images. Carefully select the parent image of your containers based on the size and the trustworthiness of the publisher.
- Always review your builds, and uninstall any unnecessary software at build time.
Query the list of packages using the package manager in your container.
The below example lists the software installed by the Alpine Linux’s package manager.
$ docker exec CONTAINER_ID apk info
- Do not run unnecessary processes in a container. Whenever possible, stick with the pattern of running a single, stateless process per container.
- Remove the SUID and SGID bit to executables at build time. SUID and SGID executables run with the file owner’s privileges, rather than the privilege of the current user, and they are often leveraged to launch Privilege Escalation attacks.
Add this command to your Dockerfiles to remove the SUID and SGID bits from the executables.
RUN find / -perm /6000 -type f -exec chmod a-s {} \; || true
- Use multi-stage builds. This neat feature was added in Docker Engine 17.05 to create optimised and easy-to-maintain Dockerfiles. Use this to separate any untidy build phases, such as the compilation of source code, from the final phase of the real image build.
- Unlike other software running in your systems, it’s considered best practice for Docker images to pin parent images and software versions to increase the predictability of the containers you are building. To enforce this, avoid using parent images with the
latesttag and do not upgrade the system’s packages at build time.
Use a non-root user
Docker builds and runs the container with the privileges declared in the last USER command in the Dockerfile. If this is not set, it defaults to the high-privileged “root” user.
Follow the principle of “least privilege”, and drop the container privileges as soon as possible in the build process by adding the USER command to your Dockerfiles.
To identify all the containers that run as “root”:
$ docker ps -a -q | xargs docker inspect -f '{{.Id}}: User {{if .Config.User}}{{.Config.User}}{{else}}root{{end}}'
To check if an image has been built using the USER command:
$ docker history IMAGE_ID | grep USER
Protect build-time secrets
Secret strings, such as credentials and authentication tokens, should never be stored in Dockerfiles or application code.
Contrary to popular belief, it is also unsafe to pass them as environment variables at build time using --build-arg. The build-time environment variables were not designed to handle secrets, and sensitive material can be easily disclosed by running the docker history command against the image.
Juggling build-time variables between ARG and ENV commands in the Dockerfile is also an anti-pattern. Once a variable is defined with ENV, it persists in the running container and can be read by anyone who gains access to the container with the right privileges.
Review your Dockerfiles and CI configuration to make sure no secrets are passed at build time using environment variables.
Inspect your images to identify any ARG command that declares variables with suspicious names.
$ docker history IMAGE_ID | grep ARG
From an architectural point of view, secrets management is outside Docker’s scope and should be delegated to some dedicated platform component by using AWS IAM, Kubernetes Secrets, or external vaults.
However, starting with Docker version 18.09 and API version 1.39, Docker provides a neat --secret argument to the docker build command to secure the passage of sensitive material at build time.
Read more about Docker secrets build information here.
Do not ADD resources
Avoid using ADD in Dockerfile to copy files from the context into the image at build time.
Unlike COPY, the Dockerfile ADD command can fetch remote resources via URL at build time. This could be abused in systems that dynamically generate Dockerfiles, such as Docker-based Continuous Integration or Platform Management systems, to download and install malicious code at build time.
Check if an image has been using ADD.
$ docker history IMAGE_ID | grep ADD
If possible, stick to using the COPY command.
Use Docker Content Trust
Docker does not verify the integrity and publisher of images in the registry by default. This leaves Docker operators vulnerable to running untrusted and potentially malicious images.
With the release of version 1.8, Docker introduced Content Trust, which supports digital signing and authentication of images. Do consider enabling this feature in your corporate Docker build pipeline.
Trust for an image tag is managed through the use of Docker Content Trust signing keys. Back up the root key somewhere safe. Since it is only required to create new repositories, it makes sense to store it offline in hardware.
Within the Docker CLI, you can sign and push a container image with the docker trust command syntax. To sign an image, a Docker Registry with a Notary server attached is a pre-requisite.
To sign a Docker Image, you will need a delegation key pair, which can be generated either locally, using docker trust key generate, or by a certificate authority.
Content Trust is disabled by default in the Docker Client.
To enable it, set the DOCKER_CONTENT_TRUST environment variable to 1. This prevents the client from working with unsigned images.
Read more about Content Trust.
Build Security into Your Docker Container Build Process with Developer Training
Would you prefer to resolve issues (“after the fact”) or avert them?
Assign developers to put out fires, or prevent these fires in the first place?
Spend time and money fixing problems lower down the SDLC pipeline, or create a process whereby potential problems are caught and fixed fast and before the crash?
Smart business leaders always select Door #2. Here’s why!
Security bugs during the Docker container build phase tend to increase over time. Fixing them later slows down development, stresses out your technical team, and increases remediation costs. A more efficient way to reduce security issues downstream is to prevent them, or at least identify and remediate them as early as possible. And this can only happen if your developers and technologists have the right skills and knowledge - for which real-world, hands-on training is absolutely vital.
SecureFlag provides world-class training to teach developers modern secure coding practices through 100% hands-on exercises. They learn defensive programming based on real-world vulnerabilities and famous breaches, while you get a more robust SDLC toolkit to help you meet your business and technology goals.
The risks are real, and old-fashioned training (multiple-choice questions anyone?!) just does not cut it. For a free demo of our unique training approach for 10+ technologies, get in touch with us today.
