Bitbucket services not starting after updating SSL certificate

[Brett Carpenter]

Bitbucket 6.3.0 running on Debian Jessie. When SSL certificates were renewd bitbucket started reporting "500" error. After restarting Bitbucket service error log showed this error: 

2019-06-02 07:31:57,629 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2019-06-02 07:32:02,303 ERROR [spring-startup] com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:375)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:204)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:445)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:516)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:116)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.<init>(ExtendedHikariDataSource.java:49)
at com.atlassian.stash.internal.hikari.HikariDataSourceFactory.create(HikariDataSourceFactory.java:70)
at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:228)
at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:721)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:534)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:745)
... 100 frames trimmed
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
... 24 common frames omitted
2019-06-02 07:32:05,002 ERROR [spring-startup] com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Web interface shows:

A fatal error has occurred

The following problem occurred, which prevents Atlassian Bitbucket from starting:

• The database, as currently configured, is not accessible. 
  Communications link failure
  
  The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

It seems this issue is mostly caused by mysql-connector-java not being the correct version or being in the correct location. This set-up didn't change when the SSL certificate was renewed, however mysql-connector-java-5.1.47.jar is in /var/atlassian/application-data/bitbucket/lib/

What should I try to get Bitbucket back up and running?  

[Stephen Sifers]

Hello Brett and welcome to the Community!
Thank you for providing the log output along with the steps you took prior to the connection issue with MySQL. The logs are pointing to an issue with the connection (or lack thereof) to MySQL. Here are a few items I would proceed to check:

• If MySQL is on another host, ensure they’re able to communicate and the MySQL user is granted access from the Bitbucket server.
• https://dev.mysql.com/doc/refman/8.0/en/account-names.html
• The path you specified for your MySQL driver is in the application directory and not the home directory. Please ensure your driver is in the proper location for Bitbucket to use:
• Bitbucket Server home directory
• Additionally, it's always going to re-check that MySQL and Bitbucket are configured for each other properly, you can review the documentation for this at Connecting Bitbucket Server to MySQL

If you’re still receiving errors after attempting the above, please reach back out and let us know.

Regards,

Stephen Sifers