Troubleshooting common SSL certificate verification errors

Issue

GitLab is returning one of the following errors when trying to establish a TLS secured connection with a particular resource.

  • SSL certificate problem: unable to get local issuer certificate
  • certificate verify failed (unable to get local issuer certificate)
  • unable to verify the first certificate
  • certificate signed by unknown authority
  • SSL certificate problem: self signed certificate in certificate chain

Cause

These errors can occur when a GitLab component tries to establish an SSL/TLS connection to an external resource where the SSL certificate being served is not properly trusted or is different to what is being expected.

Common situations where this issue occurs include:

  • The network connection to the resource is proxied in some manner where a different, untrusted, SSL certificate is being served earlier within the network path.
  • GitLab connects to an Elasticsearch server or other integration that uses either a self-signed, internal CA signed or custom SSL certificate chain.
  • A purchased SSL certificate signed by a publicly trusted CA has an issue with the certificate chain being served, or the CA isn't part of the ca-cert bundle included with GitLab.

GitLab has its own ca-cert bundle that you can add certs to by placing the individual custom certs in the /etc/gitlab/trusted-certs directory. The Linux package ships with the official Mozilla collection of trusted root certification authorities which are used to verify certificate authenticity.

Resolution

Ensure the certificate is trusted by GitLab

  1. On the GitLab instance, add the full certificate chain of the public certificate for the connected resource to the /etc/gitlab/trusted-certs/ directory. On multi-node installations, make sure to copy the certificate into the /etc/gitlab/trusted-certs/ directory on all nodes.

  2. Apply your changes with sudo gitlab-ctl reconfigure.

Custom certificate chains

Due to a known issue, when using a custom certificate chain, the server, intermediate, and root certificates must be put into separate files in the /etc/gitlab/trusted-certs directory.

This applies in both cases where GitLab itself, or external resources GitLab must connect to, are using a custom certificate chain.

For example, for GitLab itself you can use:

```
/etc/gitlab/trusted-certs/example.gitlab.com.crt
/etc/gitlab/trusted-certs/example.gitlab.com_intermediate.crt
/etc/gitlab/trusted-certs/example.gitlab.com_root.crt 
```

For external resources GitLab must connect to, you can use:

```
/etc/gitlab/trusted-certs/external-service.gitlab.com.crt
/etc/gitlab/trusted-certs/external-service.gitlab.com_intermediate.crt
/etc/gitlab/trusted-certs/external-service.gitlab.com_root.crt 
```

Apply your changes with sudo gitlab-ctl reconfigure.

Verifying the SSL certificate being served

Serving the full certificate chain is recommended in order to prevent SSL errors when clients connect. The full certificate chain order should consist of the server certificate first, followed by all intermediate certificates, with the root CA last.

From inside the GitLab environment, verify if the full certificate chain is being served, and the order in which they're being served, by running the following command:

echo | /opt/gitlab/embedded/bin/openssl s_client -showcerts -connect <hostname>:<port>

Related Links

On this page...