GitLab Runner with Kubernetes executors using DIND services without enabling Privileged mode fails to run

Issue

When using the Docker or Kubernetes executor for building container images, you have two options:

  1. DIND as a service with privileged mode enabled.
  2. Socket binding where privileged mode does not need to be enabled.

Do not mix these options. Using socket binding with the DIND service and disabling privileged mode will not work.

As per the doc: the socket binding method:

If you bind the Docker socket you can’t use docker:24.0.5-dind as a service. Volume bindings also affect services, making them incompatible.

You may see errors like this in the job log:

ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Or this in the runner logs:

mount: permission denied (are you root?)

Environment

  • Using the Kubernetes executor on GitLab Runner v17.0 and newer with DIND services and having privilege mode disabled.

  • Impacted offerings:

    • GitLab.com
    • GitLab Dedicated
    • GitLab Self-Managed

  • Impacted versions:

  • GitLab Runner v17.0 and newer

Cause

A combination of bug fixes and security and feature enhancements have created more restricted functionality.

Resolution OR Workaround

  • If you do not want to enable privileged mode or use socket binding for building container images, consider transitioning to Kaniko which addresses the security concerns with DIND and socket binding. Note, some features like DAST and Code Quality still require DIND.
  • Enable privileged mode as described in the Kubernetes DIND docs if using DIND as a service.

Additional information

GitLab uses specific runners with privileged mode enabled for the gitlab-org/gitlab repository. Some discussion in this issue.

On this page...