Bitbucket doesnt start anymore

[michael.klinger]

Hey Atlassian Community, 

We are running our Bitbucket instance within a docker container. And a separate docker container with a postgres database. After restarting our System, Bitbucket shows the Setup page again. It looks like it doesnt find the volumes or db credential anymore ? 

The Logs below: 

 Kind Regards 

And many thanks

[Caterina Curti]

Hi @michael.klinger ,

The screenshot is not readable so I cannot tell what the error is exactly from the visible stacktrace in the log.

Can you paste it here?

Was there any change in the container itself?

If it is not been deleted and recreated, I don't see why there should be a problem after a restart of the system (assuming that's a restart of the server).

But you are correct when you say that the problem could be with either the volumes used by the container or the database.

Let's have a look at the logs.

Thanks,

Caterina - Atlassian

[michael.klinger]

Sorry for the picture. I thought you can click on to make it bigger.

Logs below

[michael.klinger]

By the Way . Many thanks for your fast reply :) 

[michael.klinger]

2020-01-08 05:37:05,261 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication on e34616a921fc with PID 45 (/opt/bitbucket/app/WEB-INF/classes started by bitbucket in /var/atlassian/bitbucket)
2020-01-08 05:37:05,305 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to default profiles: default
2020-01-08 05:37:10,334 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 6.7.1 (bdc7899 built on Fri Oct 04 06:21:58 UTC 2019)
2020-01-08 05:37:10,335 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_151-b12
2020-01-08 05:37:17,860 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 18.239 seconds (JVM running for 25.43)
2020-01-08 05:37:31,765 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/bitbucket
2020-01-08 05:37:55,765 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/bitbucket/shared
2020-01-08 05:38:03,441 WARN [spring-startup] com.hazelcast.instance.Node [172.18.0.15]:5701 [bitbucket] [3.11.1] No join method is enabled! Starting standalone.
2020-01-08 05:39:11,416 INFO [spring-startup] c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:44821
2020-01-08 05:39:18,678 INFO [spring-startup] c.a.s.i.s.g.t.DefaultTranscodeServer Callback socket listening at 127.0.0.1:39123
2020-01-08 05:39:35,939 WARN [FelixStartLevel] o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.
2020-01-08 05:40:08,423 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] c.a.plugin.osgi.factory.OsgiPlugin Unable to start the plugin container for plugin 'com.atlassian.bitbucket.server.bitbucket-emoticons'
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'twemojiProvider' defined in URL [bundle://47.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to in
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:304)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:57)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:322)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:287)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 11 frames trimmed
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:184)
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider.<init>(TwemojiProvider.java:47)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 9 common frames omitted
2020-01-08 05:40:09,084 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] o.e.g.b.e.i.u.c.RunnableTimedExecution Closing runnable for context NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.bitbucket.server.bitbucket-emoticons, config=osgibundle:/META-INF/spring/*.xml) did not finish in 500ms; consid
2020-01-08 05:40:09,087 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] o.e.g.b.e.i.s.ExtenderConfiguration Application context refresh failed (NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.bitbucket.server.bitbucket-emoticons, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'twemojiProvider' defined in URL [bundle://47.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to in
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:304)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:57)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:322)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:287)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 11 frames trimmed
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:184)
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider.<init>(TwemojiProvider.java:47)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 9 common frames omitted
2020-01-08 05:40:09,092 ERROR [ThreadPoolAsyncTaskExecutor::Thread 13] o.e.g.b.e.i.d.s.DependencyWaiterApplicationContextExecutor Unable to create application context for [com.atlassian.bitbucket.server.bitbucket-emoticons], unsatisfied dependencies: none
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'twemojiProvider' defined in URL [bundle://47.0:0/META-INF/spring/atlassian-plugins-components.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to in
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:304)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:57)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:322)
at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:287)
at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 11 frames trimmed
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider]: Constructor threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:184)
... 9 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.atlassian.bitbucket.internal.emoticons.twemoji.TwemojiProvider.<init>(TwemojiProvider.java:47)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:172)
... 9 common frames omitted
2020-01-08 05:40:29,312 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/plugin/connect/confluence/ in bundle com.atlassian.plugins.atlassian-connect-plugin; it does not exist
2020-01-08 05:40:29,757 WARN [spring-startup] c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist
2020-01-08 05:40:31,865 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2020-01-08 05:40:40,920 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2020-01-08 05:40:40,925 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2020-01-08 05:40:40,938 WARN [spring-startup] c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist

[Caterina Curti]

Hi @michael.klinger ,

Can you try to clear the plugin directory to see if this helps?

Here are the steps:

1. Stop Bitbucket Server
2. Delete the following directories :
   1. <Bitbucket_Home_Directory>/plugins/.bundled-plugins
   2. <Bitbucket_Home_Directory>/plugins/.osgi-cache
3. Start Bitbucket Server

Where <Bitbucket_Home_Directory> refers to the Bitbucket home, depending on how your containers are setup this could be within the container or in a volume. 

If this does not help it would be good to also get the details of how the containers are setup.

Thanks,
Caterina

[michael.klinger]

Thanks :) 

I can only find the .osgi-cache folder in the Bitbucket Home Directory.

I deleted the .osgi-cache folder, unfortunately without success.

I am using a docker-compose.yml to deploy the whole stack (jira, confluence , bitbucket).

Here are the parts of the docker-compose for bitbucket and the bitbucket postgres db:

bitbucket-db:
    image: postgres:9.4
    networks:
      - bitbucket
    volumes:
      - bitbucket-db:/var/lib/postgresql/data
    secrets:
       - atlassian_bitbucket_postgres_password
    env_file:
      - bitbucket.env
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "50"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
          - node.platform.os == linux
      restart_policy:
        condition: any
      resources:
        limits:
          memory: 256M
        reservations:
          memory: 32M
   
  

  bitbucket-app:
    image: teamatldocker/bitbucket:6.7.1
    depends_on:
      - bitbucket-db
    networks:
      - proxy
      - bitbucket
    ports:
      - "7999:7999"
    volumes:
      - bitbucket-app:/var/atlassian/bitbucket
    env_file:
      - bitbucket.env
    logging:
      driver: "json-file"
      options:
        max-size: "500k"
        max-file: "50"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
          - node.platform.os == linux
      restart_policy:
        condition: any
      resources:
        limits:
          memory: 3072M
        reservations:
          memory: 512M

[michael.klinger]

I am using version 6.7.1

[Caterina Curti]

Hi Michael,

Do you see any timeout related errors in the logs? I just noticed that some log lines seem to be truncated.

If that's the case, can you try to add the -Datlassian.plugins.enable.wait at startup? The default value is 300 so we can try to set it to 600.

Here is our doc for that:

Unable to start the plugins due to timeout

The teamatldocker image is not the official one provided by Atlassian (which is available at https://hub.docker.com/r/atlassian/bitbucket-server/) so I'm not sure what would be the best way to specify this as the JVM_SUPPORT_RECOMMENDED_ARGS environment variable in this context. There is not much documentation on the Dockerhub page (https://hub.docker.com/u/teamatldocker).