Resolve CI job token signing key is not set error

Description

  • After upgrading Gitlab server, some actions result in the error:
    • RuntimeError (CI job token signing key is not set)
  • Other errors related to ci_job_token_signing_key may be seen.

Environment

LDAP is enabled.

Impacted offerings:

  • GitLab Self-Managed

Impacted versions:

  • 17.7.0 and later

Solution

To resolve the issue, re-run the migration. On a Rails node, run the following commands:

# Re-run the migration

gitlab-rails db:migrate:down VERSION=20241017160504
gitlab-rails db:migrate:up VERSION=20241017160504

# Restart the GitLab application

gitlab-ctl restart

Cause

This happens when the 20241017160504 migration does not complete properly, which fails to create the ci_job_token_signing_key column in the application settings table.

  • When LDAP is enabled, the 8_devise.rb initializer checks whether the GitLab license supports multiple LDAP servers.
  • This has a side effect of checking Gitlab::CurrentSettings.usage_ping_features_enabled?, which loads ApplicationSetting.
  • Loading ApplicationSetting then loads attr_encrypted, which then loads the schema for application_settings into the schema cache.
  • The migrations run to add the encrypted_ci_job_token_signing_key column (20241017160107) and generate the key (20241017160504).
  • However, when the 20241017160504 migration is loaded, the cache is already populated with the state of the table before the column has been added. As a result, the update does nothing.

Additional Information

To check if the ci_job_token_signing_key column exists in the table:

  1. Open a Rails console session
  2. Execute Gitlab::CurrentSettings.ci_job_token_signing_key

Related Links

Related issue: #38397

On this page...