Migration failure: rake aborted trying to backup the secrets.yml due to Device or resource busy

Description

Webservice and Sidekiq pods stuck in CrashLoopBackOff or failing after upgrade.

Missing Rails.application.credentials.active_record_encryption_primary_key for production environment. The secret will be generated and stored in config/secrets.yml.
Missing Rails.application.credentials.active_record_encryption_deterministic_key for production environment. The secret will be generated and stored in config/secrets.yml.
Missing Rails.application.credentials.active_record_encryption_key_derivation_salt for production environment. The secret will be generated and stored in config/secrets.yml.
Creating a backup of secrets file: /srv/gitlab/config/secrets.yml: /srv/gitlab/config/secrets.yml.orig.1740655144
Device or resource busy @ rb_file_s_rename - (/srv/gitlab/config/secrets.yml, /srv/gitlab/config/secrets.yml.orig.1740655144)

Environment

  • Self-managed GitLab Helm chart deployment

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

  • 17.8.0 and later
  • 8.8.0 and later (Helm chart)

Solution

  • Environments with the shared-secrets job disabled will need to manually create the three new secrets by following the instructions in the release notes.
  • Environments with the shared-secrets job enabled can re-deploy the chart as a workaround.

Cause

Gitlab 17.8 added three new secrets. The shared-secrets job auto-generates the new secrets. The secrets must be manually added to the GitLab Rails secrets section if the shared-secrets job is disabled.

Auto-generation can fail if Device or resource busy errors occur while backing up secrets.yml. This might happen if the mount is in read-only mode or the file is being read from at the time the automation runs.

On this page...