Node Sidecar went offline

[Mousa Ebrahimi]

Hi

I am trying to upgrade Bitbucket from v7.21.0 to v8.2.0 on RHEL 7. I made sure all the pre-requirements are available during the build like JDK Git version etc that mentioned in the Bibucket upgrade doco.

Bitbucket 8.2.0 installs on the server and load as normal, able to login and run initial tests, but after some time the application goes offline and the server BB logs:

2022-07-28 12:47:00,831 INFO  [mesh-heartbeat-timer:thread-1]  c.a.s.i.s.g.m.DefaultMeshSidebandRegistry Sidecar (http://localhost:7777): Reopening sideband channel

2022-07-28 12:47:00,999 INFO  [mesh-grpc-request:thread-26]  c.a.s.i.mesh.DefaultMeshNodeRegistry Node Sidecar (http://localhost:7777) went offline

2022-07-28 12:47:01,008 INFO  [mesh-grpc-request:thread-28]  c.a.s.i.s.g.m.DefaultMeshSidebandRegistry Sidecar (http://localhost:7777): Sideband channel closed because node is unavailable

Please note we dont have any mesh nodes setup, Its a single server env.

Am I missing something ?

Thank you

Moses

[Christian Glockner]

Hi Moses,

We've got a knowledge base article on that: https://confluence.atlassian.com/bitbucketserverkb/error-repository-not-available-offline-occurs-when-accessing-a-repository-in-bitbucket-server-1155145973.html

As for the reason for Sidecar stopping, the Mesh logs should show.

Cheers,

Christian

Premier Support Engineer

Atlassian

[Mousa Ebrahimi]

Hi Christian, 

Thank you very much for your reply.

The link is the similar to what we experience at the moment in our testing environment.

But at the moment, it is keeps happening on the test server, and every time I restart either the server or the Bitbucket to bring the service back up. 

If this was happening once in a while that was fine, but since its keep happening I can not proceed with rolling this to production as its not stable.

Is there any way to disable mesh on version 8.2.+? below is the output of mesh log from the server.

2022-07-31 11:47:26,496 INFO  [main] - c.a.b.mesh.boot.MeshApplication Starting MeshApplication using Java 11.0.15 on <Host> with PID 4524 (/apps/bitbucket/atlassian-bitbucket-8.2.0/app/WEB-INF/mesh/mesh-app.jar started by bitbucket in /data/bitbucket-data/mesh)

2022-07-31 11:47:26,507 INFO  [main] - c.a.b.mesh.boot.MeshApplication The following 1 profile is active: "sidecar"

2022-07-31 11:47:33,172 INFO  [main] - c.a.b.m.g.b.DefaultGitBinaryHelper Using /apps/tools/git/git-2.36/bin/git (2.36.0)

2022-07-31 11:47:33,489 INFO  [main] - c.a.b.mesh.app.BuildInfoLogger Starting Mesh 1.2.0 (de66ba5 built on 2022-06-23T01:05:20.216Z)

2022-07-31 11:47:33,490 INFO  [main] - c.a.b.mesh.app.BuildInfoLogger JVM: Red Hat, Inc. OpenJDK 64-Bit Server VM 11.0.15+9-LTS

2022-07-31 11:47:33,724 WARN  [main] - c.a.b.m.t.ResourceThrottleStrategyProvider [scm-hosting] This machine's total memory cannot safely support the configured minimum number of tickets (2 configured, 1 possible). Selecting fixed throttling instead

2022-07-31 11:47:33,895 INFO  [main] - c.a.b.m.g.t.DefaultTranscodeServer Callback socket listening at

2022-07-31 11:47:35,409 INFO  [main] - c.a.b.m.g.hook.DefaultGitHookService Git hook callback socket listening on

2022-07-31 11:47:35,449 INFO  [main] - c.a.b.mesh.app.HomeLockAcquirer Successfully acquired lock on home directory /data/bitbucket-data/mesh

2022-07-31 11:47:37,248 INFO  [main] - c.a.bitbucket.mesh.grpc.GrpcServer gRPC server started (port: 7777, SSL: false, maxDirectMemory: 718 MB)

2022-07-31 11:47:37,260 INFO  [main] - c.a.b.mesh.boot.MeshApplication Started MeshApplication in 12.44 seconds (JVM running for 14.014)

2022-07-31 11:47:53,409 INFO  [grpc-server:thread-1] 16MBAHB0x707x2x1 127.0.0.1 "ManagementService/Sideband" (>0 <0) c.a.b.m.m.DefaultSidebandManager : Sideband connected 

Thank you

Moses

[Christian Glockner]

Hi Moses,

This suggests you have very little (probably too little) RAM on this machine:

[scm-hosting] This machine's total memory cannot safely support the configured minimum number of tickets (2 configured, 1 possible). Selecting fixed throttling instead

How much RAM do you have on that machine, and what's the configured heap size for Bitbucket Server?

Cheers,

Christian

Premier Support Engineer

Atlassian

[Mousa Ebrahimi]

Hi Christian,

That would probably be it. Its t3a medium instance with 4gb ram, and bb mem heap been set to 2gb and es mem heap been set to 2gb as well.

I downsized es mem heap to 1 and build the server again to check if it makes any difference. But please advise whats the recommended specs for testing env.

Cheers 

Moses

[Christian Glockner]

Hi Moses,

In a test environment I would expect that at least for Bitbucket Server you would be able to get by with the default heap size (512 initial, 1024 max), which would free up some memory already.

Cheers,

Christian

Premier Support Engineer

Atlassian