Bitbucket Migrate Database tool fails

[John Goetz]

I'm moving from a RHEL 6 / MySQL 5.7 BitBucket server database to a CentOS7 / MySQL 8 database.

I have two instances of BitBucket 7.4.0 running on different servers; the first is pointed at the old database, and the second is pointing at the new database.

When I attempt to run the Migrate Database wizard from the old server, it refuses to connect, generating this error message:

The configuration entered is not valid. A database connection could not be established. Please check you configuration and try again. 

Show Details displays:

The configuration entered is not valid. A database connection could not be established. Please check your configuration and try again. com.atlassian.stash.internal.db.DefaultDatabaseManager.validateConfiguration(DefaultDatabaseManager.java:217) com.atlassian.stash.internal.migration.DefaultDatabaseMigrationService.validateConfiguration(DefaultDatabaseMigrationService.java:85) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ... Failed to obtain JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) com.atlassian.stash.internal.db.DefaultDatabaseValidator.validate(DefaultDatabaseValidator.java:43) com.atlassian.stash.internal.db.DefaultDatabaseManager.validateConfiguration(DefaultDatabaseManager.java:214) ... Could not create connection to database server. sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ... null com.mysql.jdbc.ConnectionImpl.getServerCharacterEncoding(ConnectionImpl.java:3286) com.mysql.jdbc.MysqlIO.sendConnectionAttributes(MysqlIO.java:1987) com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1913) ...

From the command line on the old database server I can connect via telnet on port 3306 and also via mysql to the new database server; therefore it's not a firewall or networking issue.

The logs show this:

2020-07-24 11:08:40,539 WARN [http-nio-7990-exec-9] ,username> @1WONQ3Ix668x1986x0 rbvr07 192.168.200.117 "POST /admin/db/edit HTTP/1.1" c.a.s.i.db.DefaultDatabaseManager A connection could not be opened with the DataSource
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at com.atlassian.stash.internal.db.DefaultDatabaseValidator.validate(DefaultDatabaseValidator.java:43)
at com.atlassian.stash.internal.db.DefaultDatabaseManager.validateConfiguration(DefaultDatabaseManager.java:214)
at com.atlassian.stash.internal.migration.DefaultDatabaseMigrationService.validateConfiguration(DefaultDatabaseMigrationService.java:85)
at com.atlassian.stash.internal.web.admin.DbConfigurationController.testAndMigrateDbConfig(DbConfigurationController.java:93)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.AuthorizationFailureInterceptor.doFilterInternal(AuthorizationFailureInterceptor.java:39)
at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:26)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:48)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:84)
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.lang.Thread.run(Unknown Source)
... 309 frames trimmed
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2575)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:155)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:146)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:205)
at org.springframework.jdbc.datasource.SingleConnectionDataSource.initConnection(SingleConnectionDataSource.java:224)
at org.springframework.jdbc.datasource.SingleConnectionDataSource.getConnection(SingleConnectionDataSource.java:164)
at org.springframework.jdbc.datasource.DelegatingDataSource.getConnection(DelegatingDataSource.java:99)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 36 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.mysql.jdbc.ConnectionImpl.getServerCharacterEncoding(ConnectionImpl.java:3286)
at com.mysql.jdbc.MysqlIO.sendConnectionAttributes(MysqlIO.java:1987)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1913)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1290)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2493)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526)
... 53 common frames omitted

I'm not sure where to go from here.

I tried copying the data over with Navicat, but the process errors on the DATABASECHANGELOG table.

I then tried copying the DATABASECHANGELOG by itself, which worked but then BitBucket wanted to run the database migration to create new tables and so crashed with a fatal database error.

What else should I try?
Thanks for any help or hint you can give!

[Evan Slaughter · Accepted solution]

Hi John,

Taking a look at the stack trace provided, my top two thoughts are either an issue with networking (which it sounds like you may have traced down to likely not being the issue) or an issue with the database connection string not matching what is expected by the MySQL server you're migrating to.

As far as the database connection goes - in the stack trace I can see it getting through the validate connection methods all the way up until it tried to establish a connection to the specified data source using the standard spring DataSourceUtils.getConnection method. At which point it blew up with that MySQLNonTransientConnectionException. 

All of that to say that what we're seeing here is likely not something that is being blocked on the application layer. It may be useful to check to see if you can find anything in the MySQL logs to confirm that it's getting some kind of request, or see if it's getting an exception on that end too and it's kicking it back for a different reason than what we're seeing on the Bitbucket side of things.

If you're still running into issues, since it's possible the issue is related to database connection strings here (and they're a bit of a sensitive subject) - feel free to send us a support-zip in a support ticket and we'll be happy to take a look.

Thanks,

Evan Slaughter

[Evan Slaughter]

Also - for what it's worth: there may be other contextual reasons for your team using MySQL, but this could also be a good opportunity to consider migrating to a different database type.

There are quite a few issues we've seen come up when using MySQL as the external database (mostly in larger instances of Bitbucket) that get outlined here. But if your team is open to the possibility of using a different external database type moving forward, this could be a good opportunity to make the switch as part of the maintenance, as it could help future-proof your instance against some of these other known issues with MySQL.

Thanks,

Evan Slaughter