Upgrading postgresql database server did not upgrade pg_dump, resulting in a version mismatch

Question

Here's an interesting problem I just finished working on.   I figured I would share it with the Atlassian community.

(1) After upgrading a legacy server from postgresql server 9.1 to 9.4, everything worked, psql was upgraded and all was fine with the world.

(2) ...Except the backups which had been failing.  The pg_dump utility was failing to dump the database because of a version mismatch similar to the error below:

pg_dump: server version: 9.4.4; pg_dump version: 9.1.6
pg_dump: aborting because of server version mismatch

no_longer_in_sudoers_file · Accepted Answer

Here was the solution...

A little investigation showed the following to be true:

root@serverNameRedacted:/usr/lib/postgresql# psql --version
psql (PostgreSQL) 9.4.4

root@serverNameRedacted:/usr/lib/postgresql# pg_dump --version
pg_dump (PostgreSQL) 9.1.6

root@serverNameRedacted:/usr/lib/postgresql# which pg_dump
/usr/bin/pg_dump

root@serverNameRedacted:/usr/lib/postgresql# ls -la /usr/bin/pg_dump
lrwxrwxrwx 1 root root 37 Jun  4 10:13 /usr/bin/pg_dump -> ../share/postgresql-common/pg_wrapper

root@serverNameRedacted:/usr/lib/postgresql# dpkg -l | grep postgres
ii  pgdg-keyring                            2014.1                            keyring for apt.postgresql.org
rc  postgresql-9.1                          9.1.18-0ubuntu0.12.04             object-relational SQL database, version 9.1 server
ii  postgresql-9.3                          9.3.9-1.pgdg12.4+1                object-relational SQL database, version 9.3 server
rc  postgresql-9.4                          9.4.4-1.pgdg12.4+1                object-relational SQL database, version 9.4 server
ii  postgresql-client-9.1                   9.1.6-0ubuntu0.12.04             front-end programs for PostgreSQL 9.1
ii  postgresql-client-9.3                   9.3.9-1.pgdg12.4+1                front-end programs for PostgreSQL 9.3
ii  postgresql-client-9.4                   9.4.4-1.pgdg12.4+1                front-end programs for PostgreSQL 9.4
ii  postgresql-client-common                169.pgdg12.4+1                    manager for multiple PostgreSQL client versions
ii  postgresql-common                       169.pgdg12.4+1                    PostgreSQL database-cluster manager
ii  postgresql-contrib-9.3                  9.3.9-1.pgdg12.4+1                additional facilities for PostgreSQL

I do know that the postgres 9.4 utilities are installed...

root@serverNameRedacted:/usr/lib/postgresql# ls -la /usr/lib/postgresql/9.4/bin/
total 1896
drwxr-xr-x 2 root root   4096 Sep  5 14:07 .
drwxr-xr-x 3 root root   4096 Sep  5 14:07 ..
-rwxr-xr-x 1 root root  68088 Jun 10 12:32 clusterdb
-rwxr-xr-x 1 root root  68152 Jun 10 12:32 createdb
-rwxr-xr-x 1 root root  63872 Jun 10 12:32 createlang
-rwxr-xr-x 1 root root  68536 Jun 10 12:32 createuser
-rwxr-xr-x 1 root root  63896 Jun 10 12:32 dropdb
-rwxr-xr-x 1 root root  63872 Jun 10 12:32 droplang
-rwxr-xr-x 1 root root  63864 Jun 10 12:32 dropuser
-rwxr-xr-x 1 root root  68424 Jun 10 12:32 pg_basebackup
-rwxr-xr-x 1 root root 351864 Jun 10 12:32 pg_dump
-rwxr-xr-x 1 root root  89304 Jun 10 12:32 pg_dumpall
-rwxr-xr-x 1 root root  30944 Jun 10 12:32 pg_isready
-rwxr-xr-x 1 root root  47560 Jun 10 12:32 pg_receivexlog
-rwxr-xr-x 1 root root  51944 Jun 10 12:32 pg_recvlogical
-rwxr-xr-x 1 root root 154904 Jun 10 12:32 pg_restore
-rwxr-xr-x 1 root root 519376 Jun 10 12:32 psql
-rwxr-xr-x 1 root root  68120 Jun 10 12:32 reindexdb
-rwxr-xr-x 1 root root  68248 Jun 10 12:32 vacuumdb

I could change the symlink to the 9.4 version (as suggested on stackoverflow.  see http://stackoverflow.com/questions/12836312/postgresql-9-2-pg-dump-version-mismatch):

sudo ln -s /usr/lib/postgresql/9.3/bin/pg_dump /usr/bin/pg_dump --force

But if you don't need the older version (9.1) for migration or what not it would be better to just remove the old software:

apt-get remove postgresl-client-9.1
apt-get install postgresql-client-9.4

Depending on the mission-specific needs, the end result is the same:

root@serverNameRedacted:/usr/lib/postgresql# pg_dump --version
pg_dump (PostgreSQL) 9.4.4