Updating the Geo nodes

Depending on which version of Geo you are updating to/from, there may be different steps.

General update steps

In order to update the GitLab Geo nodes when a new GitLab version is released, all you need to do is update GitLab itself:

Log into each node (primary and secondaries)
Update GitLab
Update tracking database on secondary node when the tracking database is enabled.
Test primary and secondary nodes, and check version in each.

Special update notes for 9.0.x

IMPORTANT: With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are required in order to update the secondary nodes and keep the Streaming Replication working. Downtime is required, so plan ahead.

The following steps apply only if you upgrade from a 8.17 GitLab version to 9.0+. For previous versions, update to GitLab 8.17 first before attempting to upgrade to 9.0+.

Make sure to follow the steps in the exact order as they appear below and pay extra attention in what node (primary/secondary) you execute them! Each step is prepended with the relevant node for better clarity:

[secondary] Login to all your secondary nodes and stop all services:
```
sudo gitlab-ctl stop
```
[secondary] Make a backup of the recovery.conf file on all secondary nodes to preserve PostgreSQL's credentials:
```
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
[primary] Update the primary node to GitLab 9.0 following the regular update docs. At the end of the update, the primary node will be running with PostgreSQL 9.6.
[primary] To prevent a de-synchronization of the repository replication, stop all services except postgresql as we will use it to re-initialize the secondary node's database:
```
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
[secondary] Run the following steps on each of the secondaries:
1. [secondary] Stop all services:
```
sudo gitlab-ctl stop
```
2. [secondary] Prevent running database migrations:
```
sudo touch /etc/gitlab/skip-auto-migrations
```
3. [secondary] Move the old database to another directory:
```
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
4. [secondary] Update to GitLab 9.0 following the regular update docs. At the end of the update, the node will be running with PostgreSQL 9.6.
5. [secondary] Make sure all services are up:
```
sudo gitlab-ctl start
```
6. [secondary] Reconfigure GitLab:
```
sudo gitlab-ctl reconfigure
```
7. [secondary] Run the PostgreSQL upgrade command:
```
  sudo gitlab-ctl pg-upgrade
```
8. [secondary] See the stored credentials for the database that you will need to re-initialize the replication:
```
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
9. [secondary] Create the replica.sh script as described in the database configuration document.
10. [secondary] Run the recovery script using the credentials from the previous step:
```
sudo bash /tmp/replica.sh
```
11. [secondary] Reconfigure GitLab:
```
sudo gitlab-ctl reconfigure
```
12. [secondary] Start all services:
```
sudo gitlab-ctl start
```
13. [secondary] Repeat the steps for the rest of the secondaries.
[primary] After all secondaries are updated, start all services in primary:
```
sudo gitlab-ctl start
```

Check status after updating

Now that the update process is complete, you may want to check whether everything is working correctly:

Run the Geo raketask on all nodes, everything should be green:
```
sudo gitlab-rake gitlab:geo:check
```
Check the primary's Geo dashboard for any errors
Test the data replication by pushing code to the primary and see if it is received by the secondaries

Update tracking database on secondary node

After updating a secondary node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, and it is required since 10.0.

Run database migrations on tracking database
```
sudo gitlab-rake geo:db:migrate
```
Repeat this step for every secondary node