CI jobs stuck in "Running" status after upgrading to GitLab 17.7
Issue
CI jobs are stuck in the Running status after upgrading to GitLab 17.7.
In addition, the production_json.log contains the entries with the following error:
...
"exception.class": "RuntimeError",
"exception.message": "CI job token signing key is not set",
...And the runner job log trace is empty.
Environment
-
Impacted offerings:
- GitLab Self-Managed
-
Impacted versions:
- GitLab v17.7.0
Cause
GitLab 17.7.0 introduced a dynamic JSON web token (JWT) for CI_JOB_TOKEN to improve the security of CI Jobs.
The GenerateCiJobTokenSigningKey database migration introduced to generate the signing key for these JWTs may not successfully generate a signing key in some cases, even though it is marked as ran in the output of sudo gitlab-rake db:migrate:status.
Workaround
Confirm you are hitting this issue by running the following in the rails console:
Gitlab::CurrentSettings.ci_job_token_signing_keyIf the output returns nil, you will need to generate a key manually. Run the following in the rails console to generate a new token signing key:
settings = Gitlab::CurrentSettings.current_application_settings
settings.ci_job_token_signing_key = OpenSSL::PKey::RSA.new(2048).to_pem
settings.save