Gitaly crashloops with unclean Gitaly shutdown: ... checking bundled Git binary ...: permission denied

Issue

Gitaly is stuck in a crashloop after installing or upgrading to GitLab version 17+.

The following error message is seen in /var/log/gitlab/gitaly/current:

unclean Gitaly shutdown: creating Git command factory: setting up Git execution environment: constructing Git environment: checking bundled Git binary "gitaly-git-v2.45": permission denied

Environment

  • noexec mount option set on /var partition.

  • Impacted offerings:

    • GitLab Self-Managed

  • Impacted versions:

    • 17.0 to current

Cause

The noexec option prevents the execution of executable binaries or linked executables. Gitaly creates symbolic links to Gitaly binaries in a directory in /var/opt/gitlab/gitaly/run, attempts to execute them and is denied the permission to, resulting in Gitaly crashing.

Resolution

In the short term:

  1. Edit the /etc/fstab file to remove the noexec mount option from /var.
  2. Reload systemd so it picks up the new /etc/fstab content: sudo systemctl daemon-reload
  3. Remount /var: sudo mount -o remount /var
  4. Check mount options on /var to ensure noexec has been removed: mount | grep var

In the longer term:

  1. Move /var/opt/gitlab to a different partition without noexec.

Additional information

  • The noexec option is set on /var by default in Linux distributions which are configured to comply with CIS Benchmarks.
  • The noexec option can also cause Git hooks to fail with messages similar to the following:
    • The '/var/opt/gitlab/gitaly/run/gitaly-1994402/hooks-344860756.d/pre-receive' hook was ignored because it's not set as executable.\\nhint: You can disable this warning with git config advice.ignoredHook false

Related links

On this page...