Postgres 9.3 terminated by signal 11: Segmentation fault after BitBucket upgrade

We are getting segmentation faults in our postgres 9.3 instance after upgrading from Bitbucket 4.5.2 to 4.11.2. As shown in the logs, you can see nothing was being reported prior to the upgrade on Saturday.

2016-12-14 09:08:18 root@git.prd.fm.internal [pg_log]$ ls -lhtr

total 788K

-rw------- 1 postgres postgres    0 Dec  8 00:00 postgresql-Thu.log

-rw------- 1 postgres postgres    0 Dec  9 00:00 postgresql-Fri.log

-rw------- 1 postgres postgres 111K Dec 10 23:40 postgresql-Sat.log

-rw------- 1 postgres postgres 193K Dec 11 23:40 postgresql-Sun.log

-rw------- 1 postgres postgres 190K Dec 12 23:40 postgresql-Mon.log

-rw------- 1 postgres postgres 208K Dec 13 22:44 postgresql-Tue.log

-rw------- 1 postgres postgres  60K Dec 14 09:14 postgresql-Wed.log

2016-12-14 09:33:51 root@git.prd.fm.internal [pg_log]$ 


The actual log details are:
 

< 2016-12-14 07:44:39.290 NZDT >LOG:  database system is ready to accept connections

 

< 2016-12-14 08:44:40.003 NZDT >LOG:  server process (PID 28179) was terminated by signal 11: Segmentation fault

 

< 2016-12-14 08:44:40.003 NZDT >LOG:  terminating any other active server processes

 

< 2016-12-14 08:44:40.003 NZDT >WARNING:  terminating connection because of crash of another server process

 

< 2016-12-14 08:44:40.003 NZDT >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 

< 2016-12-14 08:44:40.003 NZDT >HINT:  In a moment you should be able to reconnect to the database and repeat your command.

 

< 2016-12-14 08:44:40.004 NZDT >WARNING:  terminating connection because of crash of another server process

 

< 2016-12-14 08:44:40.004 NZDT >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

 
...
and then it auto-recovers:

< 2016-12-14 08:44:40.013 NZDT >HINT:  In a moment you should be able to reconnect to the database and repeat your command.

< 2016-12-14 08:44:40.020 NZDT >LOG:  all server processes terminated; reinitializing

< 2016-12-14 08:44:40.080 NZDT >FATAL:  the database system is in recovery mode

< 2016-12-14 08:44:40.080 NZDT >LOG:  database system was interrupted; last known up at 2016-12-14 08:39:59 NZDT

< 2016-12-14 08:44:40.080 NZDT >LOG:  database system was not properly shut down; automatic recovery in progress

< 2016-12-14 08:44:40.091 NZDT >LOG:  redo starts at 13/F5BB6E58

< 2016-12-14 08:44:40.110 NZDT >LOG:  record with zero length at 13/F5D36E48

< 2016-12-14 08:44:40.110 NZDT >LOG:  redo done at 13/F5D36E18

< 2016-12-14 08:44:40.110 NZDT >LOG:  last completed transaction was at log time 2016-12-14 08:44:38.411565+13

< 2016-12-14 08:44:40.191 NZDT >LOG:  database system is ready to accept connections

< 2016-12-14 08:44:40.191 NZDT >LOG:  autovacuum launcher started


This merry go-round repeats every hour. Yesterday however, the database server barfed and went into recovery mode bringing BitBucket down. We added more RAM but still get the same symptoms in the logs, just not the collapse into recovery mode.

Is anyone aware of this issue and does anyone have any work-around/fix or knowledge about a pending fix?

 

1 answer

1 accepted

This widget could not be displayed.

Hi,

We have had a few reports of this and are currently investigating. The symptoms suggest it is certainly a bug in PostgreSQL and other customers have reported upgrading to PostgreSQL 9.3.15 has resolved the problem.

We will be trying to isolate the bug such that we can say exactly which PostgreSQL versions are impacted and which contain the fix. I will update this answer if more information becomes available.

Regards,
Ben
Bitbucket Server 

Also see https://confluence.atlassian.com/display/BitbucketServerKB/Postgres+segmentation+fault+after+upgrading+to+Bitbucket+4.11+and+up

Updates will be made to that knowledge base document if more information becomes available.

Thanks for confirming this Ben. I upgraded both our UAT and production systems and have had no further error reports appearing in the postgres logs so can confidently say that upgrading to 9.3.15 (from 9.3.5) has solved the problem.

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Aug 21, 2018 in Bitbucket

Branch Management with Bitbucket

As a project manager, I have discovered that different developers want to bring their previous branching method with them when they join the team. Some developers are used to performing individual wo...

1,320 views 8 11
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you