SSH Error: REMOTE HOST IDENTIFICATION HAS CHANGED

Description

Users attempting to pull from or push to GitLab repositories via SSH encounter the error message: REMOTE HOST IDENTIFICATION HAS CHANGED
Git operations over SSH are blocked with warnings about potential man-in-the-middle attacks
The client's SSH known_hosts file contains a different key for the GitLab instance hostname/IP address
Subsequent Git commands using SSH authentication fail until the issue is resolved

Environment

Self-managed GitLab instances where the server has been recreated, migrated, or reinstalled while maintaining the same hostname/IP address
Git clients using SSH for repository operations (clone, pull, push, fetch)
Systems using OpenSSH client for Git SSH authentication

Impacted offerings:

GitLab Self-Managed

Solution

Option A: Copy SSH host keys to new host (recommended)

On the original GitLab server, copy the SSH host keys:

# Create a backup directory
sudo mkdir -p /root/ssh_backup

# Copy all host key files
sudo cp /etc/ssh/ssh_host_* /root/ssh_backup/

# Set permissions for backup files
sudo chmod 600 /root/ssh_backup/ssh_host_*_key
sudo chmod 644 /root/ssh_backup/ssh_host_*_key.pub

Transfer these key files to the new GitLab server:

# Using rsync (from original server)
sudo rsync -avPz /root/ssh_backup/ssh_host_* root@new-gitlab-server:/etc/ssh/

# Or create a tarball and transfer via another method
sudo tar -czf /root/ssh_keys.tar.gz -C /root/ssh_backup .

On the new GitLab server, set proper permissions for the transferred keys:

# Ensure correct permissions
sudo chmod 600 /etc/ssh/ssh_host_*_key
sudo chmod 644 /etc/ssh/ssh_host_*_key.pub
sudo chown root:root /etc/ssh/ssh_host_*

Restart the SSH service to apply the imported keys:

For systemd-based systems
```
sudo systemctl restart sshd
```
For init.d-based systems
```
sudo service sshd restart
```

Option B: Remove SSH known_hosts entries (for each client)

Edit the known_hosts file to remove the conflicting GitLab entry:

a. Automatically remove the offending line for your GitLab instance
```
# Replace gitlab.example.com with your actual GitLab instance hostname
ssh-keygen -R gitlab.example.com
```
b. Or open the file in a text editor to manually remove the line
```
vim ~/.ssh/known_hosts
```

The next Git operation over SSH will prompt to accept the new key:

# Example: Clone a repository via SSH
git clone git@gitlab.example.com:namespace/project.git

You will see a prompt similar to:

The authenticity of host 'gitlab.example.com' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Verify the fingerprint with your GitLab administrator before accepting.

Cause

When creating a new instance that reuses an IP address or hostname of a previously known host, the SSH host keys do not match those stored in the client's known_hosts file. This mismatch occurs because:

SSH host keys are generated during the initial setup of the operating system
New instances generate SSH keys when openssh-server is installed
The client's SSH security mechanism detects when different host keys are offered for a known server

This is a security feature designed to protect against man-in-the-middle attacks, but it requires resolution when legitimate host changes occur.

Additional Information

The error is a security feature, not a bug
The error is not specific to GitLab
For environments with frequent instance recycling, consider:
- Using configuration management tools to maintain consistent host keys
- Implementing a DNS-based solution where new instances get new hostnames
- Setting up a central authority for host key verification