Why is Container Security Important? #

  • Shared OS Kernel: Containers share the host OS kernel, so vulnerabilities here can affect all containers.
  • Use of Base Images: Untrusted or tampered images can include malicious code, so it’s important to use signed images, verify checksums, and pull from trusted sources.
  • Use of Third-party Dependencies: Containers often include external libraries which may have security issues if not regularly scanned and updated.
  • Fast Lifecycle: Containers are short-lived and rapidly deployed. Security must be integrated into the build and deployment process (DevSecOps).
  • Platform Security: Containers must meet compliance requirements (e.g., PCI DSS, HIPAA, GDPR). Implement logging, access control, and auditing from the ground up.

What are the Best Practices with Securing Containers? #

  • Use Official Images: Use official Container images from trusted sources
  • Regular Updates and Patch Management: Keep Docker Engine and container images up to date with security patches
  • Minimize Attack Surface: Use minimal base images (e.g., Alpine), and remove unnecessary packages/tools
  • Image Scanning: Scan images using tools like Docker Scout, Trivy, or Clair to detect vulnerabilities early
  • Run as Non-Root User: Never run containers as root—define a dedicated user in the Dockerfile
  • Principle of Least Privilege: Grant only the minimum permissions required to the container and its processes
  • Limit Linux Capabilities: Drop unused capabilities using the --cap-drop flag for tighter control
  • Avoid Hardcoding Secrets: Do not store passwords, tokens, or API keys in images or environment variables
  • Secrets Management: Use Docker Secrets, HashiCorp Vault, or AWS Secrets Manager to handle credentials securely
  • Network Segmentation: Use Docker’s network features (e.g., bridge, overlay) to isolate containers and limit lateral movement
  • Restrict External Access: Only expose necessary ports and endpoints
  • Container Hardening: Configure and enforce security policies (e.g., Docker Bench Security)
    • Key practices include restricting container capabilities, enforcing user namespaces, and reducing default privileges.
  • Observability: Implement monitoring, logging and tracing to detect and respond to security incidents promptly
    • Collect and centralize container logs using tools like Fluentd, ELK, or Datadog
    • Use tools like Jaeger or Grafana Tempo to trace and troubleshoot security incidents

How can you run a Container as Non-Root User? #

  • Avoid root user: Limit security risks by not running containers as root
    FROM node:14

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

# Create a non-root user and group
RUN groupadd -r appgroup && useradd -r -g appgroup -d /usr/src/app appuser

# Change ownership of the app directory
RUN chown -R appuser:appgroup /usr/src/app

# Switch to the non-root user
USER appuser

EXPOSE 3000

CMD ["node", "app.js"]
  • Create group: Set up a system group named appgroup
  • Create user: Add a system user named appuser
  • Assign user to group: Place appuser in the appgroup
  • Set home directory: Assign /usr/src/app as the home directory for appuser

How do you remove Linux capabilities from a container? #

  • Examples: Example Linux Capabilities:
    • CAP_CHOWN: Change file ownership
    • CAP_SETUID: Change user ID
    • CAP_NET_BIND_SERVICE: Bind to ports < 1024 (e.g., 80, 443). Needed by web servers like NGINX or Apache
    • CAP_NET_RAW: Use raw sockets
    • CAP_SYS_TIME: Set the system clock
    • CAP_SYS_ADMIN: Powerful and wide-ranging, includes mounting filesystems, chroot, etc.
    • CAP_AUDIT_WRITE: Write records to the kernel audit log
  • NOT all capabilities are needed: Typical applications do NOT need all capabilities
    • Recommended: Selectively add the capabilities that are needed
  • --cap-drop: Use Docker option --cap-drop to remove Linux capabilities from a container
    • Drop all capabilities: --cap-drop=ALL removes all privileged operations
    • Add specific capability: --cap-add=NET_ADMIN allows network administrative tasks
  • Example command:
    docker run --cap-drop=ALL --cap-add=NET_ADMIN myimage
  • Always start with --cap-drop=ALL and selectively add only what's absolutely necessary using --cap-add.
  • Review application needs carefully before allowing elevated capabilities.

How do you check the Health of a Container? #

  • Ensure Smooth Container Operations: Health check is crucial for maintaining the reliability and stability of your container
  • HEALTHCHECK Instruction: Add a health check in the Dockerfile to periodically test if the container is healthy Example:
    # Define the health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=5s \
  CMD replace_with_your_command || exit 1
  • CMD replace_with_your_command: The command to run for the health check. It should return a zero exit code if healthy, or a non-zero code if unhealthy
    • CMD replace_with_your_command || exit 1 is a common shell scripting pattern - Run this command. If it fails (i.e., returns a non-zero exit code), then exit the script immediately with exit code 1.
  • --interval=30s: Sets the time between consecutive health checks
    • The container will perform a health check every 30 seconds
  • --timeout=10s: Specifies the maximum time allowed for the health check command to complete
    • If the health check doesn't respond within 10 seconds, it's considered a failure
  • --retries=3: Determines the number of consecutive failures needed to consider the container as unhealthy
    • After 3 failed health checks, Docker marks the container as unhealthy
  • --start-period=5s: Provides a grace period after container start before initiating health checks
    • Docker waits for 5 seconds after the container starts before performing the first health check

Below are the three primary methods to check the health of a container

  • HTTP Endpoint
  • TCP Port
  • Custom Script

HTTP Endpoint: This method involves sending an HTTP request to a specific endpoint within the container to verify if the web server or application is responding correctly

  • Example:
    # Use an appropriate base image
FROM nginx:latest

# Define the health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl --fail http://localhost:80/ || exit 1

# Expose port 80
EXPOSE 80

TCP Port: This method checks whether a specific TCP port is open and accepting connections, ensuring that the service within the container is listening correctly

  • Example:
    # Use an appropriate base image
FROM postgres:latest

# Define the health check
# nc -z localhost 5432 || exit 1
# check if a service is listening on port 5432 
# and to exit with an error if it is not.
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD nc -z localhost 5432 || exit 1

# Expose port 5432
EXPOSE 5432

Custom Script: This method utilizes a custom script to perform more complex or specific health checks tailored to the application's logic within the container

  • healthcheck.sh: Custom Script

    #!/bin/bash

# Example health check: Verify if the application process is running
if pgrep httpd > /dev/null; then
  exit 0
else
  exit 1
fi

  • Example:

    # Use an appropriate base image
FROM httpd:latest

# Copy the custom health check script into the container
COPY healthcheck.sh /usr/local/bin/healthcheck.sh

# Update package lists before installing procps
RUN apt-get update && apt-get install -y procps

# Make the script executable
RUN chmod +x /usr/local/bin/healthcheck.sh

# Define the health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD /usr/local/bin/healthcheck.sh

# Expose necessary ports
EXPOSE 8080

Explain Docker Content Trust (DCT) and how it enhances container security? #

  • Question: How can the user of an image be certain that the image was not modified after it was published by the image publisher?
    • Enable Docker Content Trust (DCT)
  • Docker Content Trust (DCT): Security feature that lets you verify the integrity & publisher of image
    • The image is published by a trusted source
    • The image hasn’t been tampered with
  • Notary: Tool used by DCT to cryptographically sign and verify container images
  • Enabling DCT: Create an Environment Variable DOCKER_CONTENT_TRUST with value 1
    export DOCKER_CONTENT_TRUST=1
    • Only signed images are pulled or pushed (when enabled)
      • Prevents pulling unsigned images
      • Prevents pushing images without a valid signature
  • Example: Pulling an Image
    export DOCKER_CONTENT_TRUST=1
docker pull alpine:latest
    • If the image is signed, it will be pulled.
    • If not signed, it will fail with: Error: remote trust data does not exist for alpine: notary.docker.io

Verifying Trust Data

$ docker trust inspect --pretty myrepo/myapp
Signatures for myrepo/myapp:

# SIGNED TAG shows which tags are signed (e.g., latest)
SIGNED_TAG      DIGEST                       SIGNERS
latest          7b0f2b97d114b57a6b8f8c7e     alice

# SIGNERS are authorized entities who signed the image
List of signers and their keys:

SIGNER              KEYS
alice               03d4dc9e7ad6c8c9e173aa6c80f2b67

Administrative keys for myrepo/myapp:

  Repository Key:    6c3253b7a2d49f99d5e0b96b2db3a1ed
  Root Key:          2fb4456e0bb79e62f4aa5ce7e2580b31

What are CVEs (Common Vulnerabilities and Exposures)? #

  • CVE: Publicly disclosed security vulnerability in software or hardware.
  • Uniquely Identified: Each CVE is assigned a unique identifier and a brief description of the issue.
  • Standardized Information Sharing: Maintained by the MITRE Corporation and used globally to standardize and share information about cybersecurity threats.
  • Each CVE typically includes:
    • ID: e.g., CVE-2023-45678
    • Description: Brief summary of the vulnerability
    • Affected Products: Software, OS, or libraries impacted
    • References: External advisories or patch info (e.g., NIST, GitHub)
    • CVSS Score: A severity rating from 0 to 10 (see below)
      • 7.0–8.9 (High Impact), 9.0–10.0 (Critical)
  • Why CVEs Matter to Developers & DevOps: Helps you identify vulnerabilities in open-source libraries, Docker images, and IaC
  • Real-World Example: CVE-2021-44228 The Log4Shell vulnerability in Apache Log4j
    • Affected millions of Java applications
    • CVSS Score: 10.0 (Critical)
    • Allowed remote code execution without authentication

What is Docker Image Vulnerability Scanning? #

  • Docker Images can contain vulnerabilities: Docker images, built in layers, can inherit vulnerabilities from their base images, third-party dependencies, and even misconfigurations in the Dockerfile
    • Vulnerable Base Images: Using outdated or unpatched base images can expose your application to known exploits
    • Outdated Dependencies: Libraries and packages within your image that haven't been updated can contain security flaws
    • Hardcoded Secrets: Storing sensitive information like API keys, passwords, or access tokens directly in the image is a major security risk
    • Excessive Permissions: Running containers as a root user or with unnecessary capabilities increases the potential impact of a compromise
    • Unnecessary Components: Including build tools, development libraries, or other components not required for runtime increases the attack surface.
  • Scan to detect early: Scanning helps detect and mitigate these risks early
    • Shift Left: Identifying and fixing vulnerabilities early in the development lifecycle (shifting left) prevents them from reaching production environments, where they can be much more costly and disruptive
  • Popular Tools:
    • Docker Scout (by Docker Inc.): Docker-native vulnerability scanning. Integrated into Docker Desktop and CLI.
    • Trivy (Aqua Security): Fast, easy-to-use, and supports scanning of OS packages, application dependencies, ..
    • Clair (by CoreOS) : Static analysis of vulnerabilities in containers. Open-source project that scans Docker images by comparing installed packages against known vulnerabilities.
    • Snyk: Scans code, containers, and dependencies for known CVEs (Common Vulnerabilities and Exposures)
    • Grype: A vulnerability scanner for container images and filesystems
  • Best Practice:
    • Scan early and often – Integrate scans into your CI/CD pipelines

How can you scan Docker images with Trivy? #

Trivy is a simple yet powerful vulnerability scanner for containers. It checks both operating system packages and software dependencies (like Python packages or Node.js modules).

Basic Scan: To scan a Docker image, run:

trivy image <image_name>

# Example
# Pull an example image
docker pull nginx:latest

# Scan the nginx:latest image
trivy image nginx:latest
  • Output: Lists found vulnerabilities (e.g., CVE identifiers, severity levels, and potential fixes)

Fix Vulnerabilities: After running a Trivy scan, you may see a list of vulnerabilities along with recommended fixes

  • Upgrade the Base Image: Use a more recent or stable base image that has fewer known vulnerabilities
  • Apply Security Patches: Update OS packages
  • Update Application Dependencies: Modify requirements.txt, package.json, or equivalent files to use newer, patched versions
  • Remove Unnecessary Packages: Remove libraries not needed by your application to reduce the attack surface
  • Implement Security Best Practices:
    • Use multi-stage builds to keep the final image lightweight
    • Run as a non-root user to minimize privileges
    • Store secrets outside the image (e.g., Docker Secrets or an external vault)
  • Rescan: After making changes, run Trivy again to confirm that vulnerabilities are resolved