Postgres 9.3 terminated by signal 11: Segmentation fault after BitBucket upgrade

[FosterMooreAdmin]

We are getting segmentation faults in our postgres 9.3 instance after upgrading from Bitbucket 4.5.2 to 4.11.2. As shown in the logs, you can see nothing was being reported prior to the upgrade on Saturday.

2016-12-14 09:08:18 root@git.prd.fm.internal [pg_log]$ ls -lhtr

total 788K

-rw------- 1 postgres postgres    0 Dec  8 00:00 postgresql-Thu.log

-rw------- 1 postgres postgres    0 Dec  9 00:00 postgresql-Fri.log

-rw------- 1 postgres postgres 111K Dec 10 23:40 postgresql-Sat.log

-rw------- 1 postgres postgres 193K Dec 11 23:40 postgresql-Sun.log

-rw------- 1 postgres postgres 190K Dec 12 23:40 postgresql-Mon.log

-rw------- 1 postgres postgres 208K Dec 13 22:44 postgresql-Tue.log

-rw------- 1 postgres postgres  60K Dec 14 09:14 postgresql-Wed.log

2016-12-14 09:33:51 root@git.prd.fm.internal [pg_log]$ 

The actual log details are:

 

< 2016-12-14 07:44:39.290 NZDT >LOG:  database system is ready to accept connections

 

< 2016-12-14 08:44:40.003 NZDT >LOG:  server process (PID 28179) was terminated by signal 11: Segmentation fault

 

< 2016-12-14 08:44:40.003 NZDT >LOG:  terminating any other active server processes

 

< 2016-12-14 08:44:40.003 NZDT >WARNING:  terminating connection because of crash of another server process

 

< 2016-12-14 08:44:40.003 NZDT >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 

< 2016-12-14 08:44:40.003 NZDT >HINT:  In a moment you should be able to reconnect to the database and repeat your command.

 

< 2016-12-14 08:44:40.004 NZDT >WARNING:  terminating connection because of crash of another server process

 

< 2016-12-14 08:44:40.004 NZDT >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 

...

and then it auto-recovers:

< 2016-12-14 08:44:40.013 NZDT >HINT:  In a moment you should be able to reconnect to the database and repeat your command.

< 2016-12-14 08:44:40.020 NZDT >LOG:  all server processes terminated; reinitializing

< 2016-12-14 08:44:40.080 NZDT >FATAL:  the database system is in recovery mode

< 2016-12-14 08:44:40.080 NZDT >LOG:  database system was interrupted; last known up at 2016-12-14 08:39:59 NZDT

< 2016-12-14 08:44:40.080 NZDT >LOG:  database system was not properly shut down; automatic recovery in progress

< 2016-12-14 08:44:40.091 NZDT >LOG:  redo starts at 13/F5BB6E58

< 2016-12-14 08:44:40.110 NZDT >LOG:  record with zero length at 13/F5D36E48

< 2016-12-14 08:44:40.110 NZDT >LOG:  redo done at 13/F5D36E18

< 2016-12-14 08:44:40.110 NZDT >LOG:  last completed transaction was at log time 2016-12-14 08:44:38.411565+13

< 2016-12-14 08:44:40.191 NZDT >LOG:  database system is ready to accept connections

< 2016-12-14 08:44:40.191 NZDT >LOG:  autovacuum launcher started

This merry go-round repeats every hour. Yesterday however, the database server barfed and went into recovery mode bringing BitBucket down. We added more RAM but still get the same symptoms in the logs, just not the collapse into recovery mode.

Is anyone aware of this issue and does anyone have any work-around/fix or knowledge about a pending fix?

[Ben Humphreys · Accepted solution]

Hi,

We have had a few reports of this and are currently investigating. The symptoms suggest it is certainly a bug in PostgreSQL and other customers have reported upgrading to PostgreSQL 9.3.15 has resolved the problem.

We will be trying to isolate the bug such that we can say exactly which PostgreSQL versions are impacted and which contain the fix. I will update this answer if more information becomes available.

Regards,
Ben
Bitbucket Server 

[Ben Humphreys]

Also see https://confluence.atlassian.com/display/BitbucketServerKB/Postgres+segmentation+fault+after+upgrading+to+Bitbucket+4.11+and+up

Updates will be made to that knowledge base document if more information becomes available.

[FosterMooreAdmin]

Thanks for confirming this Ben. I upgraded both our UAT and production systems and have had no further error reports appearing in the postgres logs so can confidently say that upgrading to 9.3.15 (from 9.3.5) has solved the problem.