How to Squash OpenShift Docker Build Layers to Remove Secrets

When you build a Docker image, a series of layers get cached to speed up subsequent builds. However, this caching can have adverse effects if an image layer contains secrets, such as credentials. A malicious person or process could find the layer containing your secrets and discover its contents.

Luckily, when performing Docker builds in OpenShift, you can quickly destroy secret layers by setting imageOptimizationPolicy to SkipLayers in your BuildConfig.

Let’s first consider the following Dockerfile:

FROM registry.redhat.io/ubi8/ubi:8.2

# Comes from an OCP secret also called ‘nexus-creds’
COPY nexus-creds nexus-creds

RUN curl -u $(cat nexus-creds/username):$(cat nexus-creds/password) -O http://example-nexusrepo-sonatype-nexus-demo.apps-crc.testing/repository/example-artifacts/jq && \
    chown :0 jq && \
    chmod 010 jq && \
    mv jq /usr/local/bin && \
    rm -rf nexus-creds

This Dockerfile installs jq to a UBI8-based image. However, rather than download jq from their releases page, it instead downloads from a local Nexus instance. It is common in the enterprise to download tools from a repository within the corporate network rather than from an upstream repository. However, to download from the corporate repository, you must first authenticate by providing your credentials. Credentials are provided in the example Dockerfile by copying them from a file in the build context (nexus-creds).

Here’s the million-dollar question: How do you remove these credentials from the image so that they cannot be revealed? If you think the answer is to remove the credentials in a RUN command (rm -rf nexus-creds), you are partially correct. That is a necessary step and will remove the credentials from the final container. However, it will not remove the credentials from the COPY layer. The COPY layer and the credentials it contains will still remain in the container registry. As a result, an attacker could steal your credentials by accessing this layer.

Below is partial output of the “oc start-build” command that builds the example Dockerfile. Here, you’ll see each of the layers that got pushed to the internal registry, one of them being the COPY layer, highlighted below.

Pushing image image-registry.openshift-image-registry.svc:5000/demo/example-image:latest ...
Getting image source signatures
Copying blob sha256:264aaa453271eaaf6bc92b521f62e38cbda4f01e1a3fbf50abdb8b466be58a64
Copying blob sha256:7f24bdb73d536337c7d82dee9c11e09acd3bacb0e1a76e98c6e82a8208e82284
Copying blob sha256:1c808c9ef8fef442a30a026190afef3c609840a0d53214d9054649394d26ba60
Copying blob sha256:1d1edefff77e9c0fd0518961b3e80aaffaab14e4b44b54b7a0b97cb0661cd39c

So, if removing the credentials in a RUN statement is only partially correct, what else can you do to prevent credentials from being stolen? Luckily, Red Hat addresses this by allowing you to provide a BuildConfig setting called imageOptimizationPolicy. When setting this to SkipLayers, the OpenShift build will squash each of the layers added to the base image (in other words, it will squash everything down to the FROM command). Since running “rm -rf nexus-creds” after the COPY command removes the credentials from the final layer, you can rest assured that the credentials will be removed entirely from both the image and the registry itself by squashing the layers with imageOptimizationPolicy.

Below is an example of a BuildConfig that uses the imageOptimizationPolicy: SkipLayers setting.

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: example-image
spec:
  output:
    to:
      kind: ImageStream
      name: example-image
  source:
    binary: {}
    type: Binary
    secrets:
      - destinationDir: nexus-creds
        secret:
          name: nexus-creds
  strategy:
    dockerStrategy:
      imageOptimizationPolicy: SkipLayers
      pullSecret:
        name: pull-secret
      dockerfilePath: Dockerfile
    type: Docker

When this BuildConfig is applied, the resulting image will have only three layers (two from the base image and one from the image we are building off of that base). Below is the ending output of “oc start-build” after the imageOptimizationPolicy setting is applied.

Pushing image image-registry.openshift-image-registry.svc:5000/demo/example-image:latest ...
Getting image source signatures
Copying blob sha256:7f24bdb73d536337c7d82dee9c11e09acd3bacb0e1a76e98c6e82a8208e82284
Copying blob sha256:264aaa453271eaaf6bc92b521f62e38cbda4f01e1a3fbf50abdb8b466be58a64
Copying blob sha256:9e60edd0480d9cd1c51f682a2978582a42951172f507f57da0b18656f71df037

Only three layers this time instead of four! If you were to access your image registry or pull this image down from outside the cluster, there would be no trace of the COPY layer or your credentials for this given image and tag.

Thanks for Reading!

Hopefully, this helps you prevent secrets from being exposed within intermediate layers stored in your image registry. For more information, be sure to check out Red Hat’s documentation on imageOptimizationPolicy and squashing layers in Docker builds.