Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

Bitbucket Cluster - Node Authentication Failed

Thomas Langhorne
Thomas Langhorne November 2, 2023

Having trouble getting clustering working for bitbucket.

Running with Bigbang, istio is the service mesh, and matallb is the load balancer


Helm Chart Version 1.16.6
Bitbucket Application Version: 8.5.0
Helm chart Values:

values:
  image:
    repository: docker.io/atlassian/bitbucket
    tag: "8.5.0"
  serviceAccount:
    clusterRole:
      create: true
    clusterRoleBinding:
      create: true
  database:
    type: "postgresql"
    url: "jdbc:postgresql://modbucket-postgres-postgresql:5432/bitbucket"
    driver: "org.postgresql.Driver"
    credentials:
      secretName: atlassian-postgresdb
      usernameSecretKey: postgres-user
      passwordSecretKey: postgresql-password
  bitbucket:
    clustering:
      enabled: true
    hazelcastService:
      enabled: true
    additionalEnvironmentVariables:
      - name: "http_proxy"
        value: "REDACTED"
      - name: "https_proxy"
        value: "REDACTED"
      - name: "no_proxy"
        value: "REDACTED"
    additionalJvmArgs:
      - -Dcom.redhat.fips=false
      - -Djava.net.preferIPv4Stack=true
      - -Datlassian.dev.mode=false
      - -Datlassian.plugins.enable.wait=300
  ingress:
    nginx: false
    # Disable SSL Configuration as Istio Ingress Controller will provide SSL configuration
    # If this is "true" then default Bitbucket security configuraitons will prevent web page resolution
    https: false
  monitoring:
    exposeJmxMetrics: true
    jmxExporterImageRepo: docker.io/bitnami/jmx-exporter
  podAnnotations:
    traffic.sidecar.istio.io/excludeOutboundPorts: "80,443"
  podLabels:
    app: modbucket
  volumes:
    localHome:
      persistentVolumeClaim:
        create: true
        resources:
          requests:
            storage: 10Gi
      mountPath: "/var/atlassian/application-data/bitbucket"
    sharedHome:
      nfsPermissionFixer:
        imageRepo: docker.io/library/alpine
        imageTag: 3.18.2
      persistentVolumeClaim:
        storageClassName: ceph-filesystem
        create: true
        resources:
          requests:
            storage: 10Gi
      mountPath: "/var/atlassian/application-data/shared-home"



The issue I am experiencing is that the Scaled Nodes:
1) The additional nodes don't show up under the webpage/admin/clustering
2) logs have the error Node authentication failed: Unexpected bytes from remote node, closing socket


I will post the logs as a reply

Answer
Watch
Like Be the first to like this
371 views

1 answer

0 votes
Thomas Langhorne
Thomas Langhorne November 2, 2023

Posting the logs for additional details

Log from my 2nd pod, that fails Node authentication

INFO:root:User is currently root. Will downgrade run user to bitbucket
INFO:root:Running Bitbucket with command '/opt/atlassian/bitbucket/bin/exec-bitbucket-node.sh', arguments ['/opt/atlassian/bitbucket/bin/exec-bitbucket-node.sh']
Starting Atlassian Bitbucket as the current user
Starting Atlassian Bitbucket as the current user
/opt/atlassian/bitbucket/bin/_exec-webapp.sh: line 62: \nStarting Bitbucket webapp at http://localhost:${bitbucket.http.port}${bitbucket.context}: bad substitution
2023-11-02 16:30:33,819 INFO  [main]  c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.5.0 using Java 11.0.20.1 on modbucket-bitbucket-1 with PID 7 (/opt/atlassian/bitbucket/app/WEB-INF/classes started by bitbucket in /var/atlassian/application-data/bitbucket)
2023-11-02 16:30:33,828 INFO  [main]  c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to 1 default profile: "default"
2023-11-02 16:30:34,636 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 8.5.0 (d5f13a2 built on Mon Oct 10 02:01:27 UTC 2022)
2023-11-02 16:30:34,637 INFO  [main]  c.a.b.i.boot.log.BuildInfoLogger JVM: Eclipse Adoptium OpenJDK 64-Bit Server VM 11.0.20.1+1
2023-11-02 16:30:35,927 INFO  [main]  c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.147 seconds (JVM running for 4.003)
2023-11-02 16:30:38,807 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/bitbucket
2023-11-02 16:30:42,612 INFO  [spring-startup]  c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /var/atlassian/application-data/shared-home
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/atlassian/bitbucket/app/WEB-INF/lib/hazelcast-3.12.12.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-11-02 16:30:44,766 WARN  [spring-startup]  c.h.s.d.integration.DiscoveryService [10.42.1.101]:5701 [modbucket-bitbucket] [3.12.12] Cannot fetch the current zone, ZONE_AWARE feature is disabled
2023-11-02 16:30:44,814 WARN  [spring-startup]  c.h.kubernetes.KubernetesClient Cannot fetch public IPs of Hazelcast Member PODs, you won't be able to use Hazelcast Smart Client from outside of the Kubernetes network
2023-11-02 16:30:44,840 WARN  [hz.hazelcast.cached.thread-3]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:60645 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:31:08,683 INFO  [spring-startup]  c.a.s.i.s.g.m.DefaultSidecarManager Sidecar started after 8281ms
2023-11-02 16:31:13,103 INFO  [spring-startup]  c.a.s.i.hook.DefaultHookService Hook callback socket listening on 127.0.0.1:43777
2023-11-02 16:31:16,855 INFO  [hz.hazelcast.event-1]  c.a.s.i.s.g.m.DefaultMeshSidebandRegistry Sidecar (http://localhost:7777): Opening sideband channel
2023-11-02 16:31:18,541 WARN  [spring-startup]  c.a.p.o.c.felix.ExportsBuilder Cannot deep scan non file 'jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/rome-1.18.0.jar!/rome-utils-1.18.0.jar'
2023-11-02 16:31:18,588 WARN  [spring-startup]  c.a.p.o.c.felix.ExportsBuilder Cannot deep scan non file 'jar:file:/opt/atlassian/bitbucket/app/WEB-INF/lib/atlassian-tenancy-compatibility-plugin-4.0.0.jar!/atlassian-tenancy-api-4.0.0.jar'
2023-11-02 16:31:20,599 WARN  [FelixStartLevel]  o.e.g.b.e.i.s.ExtenderConfiguration Gemini Blueprint extensions bundle not present, annotation processing disabled.
2023-11-02 16:31:29,488 WARN  [ThreadPoolAsyncTaskExecutor::Thread 13]  c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.admin.criteria.WritableDirectoryExistsCriteria' for component importation purposes. Skipping...
2023-11-02 16:31:29,488 WARN  [ThreadPoolAsyncTaskExecutor::Thread 13]  c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.setup.settings.SettingsManager' for component importation purposes. Skipping...
2023-11-02 16:31:29,488 WARN  [ThreadPoolAsyncTaskExecutor::Thread 13]  c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.user.SignupManager' for component importation purposes. Skipping...
2023-11-02 16:31:29,488 WARN  [ThreadPoolAsyncTaskExecutor::Thread 13]  c.a.p.s.s.r.i.ComponentImportBeanFactoryPostProcessor Unable to load class 'com.atlassian.confluence.util.UserChecker' for component importation purposes. Skipping...
2023-11-02 16:31:42,256 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle com.atlassian.bitbucket.server.bitbucket-frontend; it does not exist
2023-11-02 16:31:42,898 WARN  [spring-startup]  c.a.p.c.r.OsgiBundledPathScanner Cannot scan directory /page/ in bundle com.atlassian.bitbucket.server.bitbucket-policies as it does not exist
2023-11-02 16:31:42,923 WARN  [spring-startup]  c.a.p.c.r.StaticPluginPathScanner Cannot scan directory /static/bitbucket/internal/feature/admin/ratelimit/ in web static as it does not exist
2023-11-02 16:31:44,202 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle com.atlassian.oauth.atlassian-oauth-service-provider-plugin; it does not exist
2023-11-02 16:31:44,481 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2023-11-02 16:31:44,483 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.de_DE; it does not exist
2023-11-02 16:31:44,486 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2023-11-02 16:31:44,488 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.fr_FR; it does not exist
2023-11-02 16:31:44,493 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /extension/build-status/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2023-11-02 16:31:44,495 WARN  [spring-startup]  c.a.s.i.p.s.OsgiBundledPathScanner Cannot scan directory /com/atlassian/oauth/shared/servlet/ in bundle tac.bitbucket.languages.ja_JP; it does not exist
2023-11-02 16:31:44,627 INFO  [spring-startup]  c.a.s.internal.scm.PluginScmService Bitbucket is using /usr/bin/git version 2.37.7
2023-11-02 16:31:45,082 INFO  [spring-startup]  c.a.b.internal.ssh.server.SshServer Starting SSH server on port 7999...
2023-11-02 16:31:45,190 INFO  [spring-startup]  c.a.b.internal.ssh.server.SshServer Started SSH server successfully.
2023-11-02 16:31:45,559 INFO  [spring-startup]  c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: KnownIssueCheck
2023-11-02 16:31:45,559 INFO  [spring-startup]  c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for StartupChecksJob
2023-11-02 16:31:45,677 INFO  [spring-startup]  c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: VersionCheck
2023-11-02 16:31:45,681 INFO  [spring-startup]  c.a.b.i.s.i.e.DefaultIndexEventQueueProcessor Event queue processor has been started
2023-11-02 16:31:45,682 INFO  [spring-startup]  c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: SearchIndexCheck
2023-11-02 16:31:45,682 INFO  [spring-startup]  c.a.b.i.s.c.c.DefaultClusterJobManager Registering job for SearchSynchronizeJob
2023-11-02 16:31:45,682 INFO  [spring-startup]  c.a.b.i.s.c.j.c.HealthCheckRunner New health check registered: AuthenticationCheck
2023-11-02 16:32:00,627 INFO  [Caesium-1-4]  c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search
2023-11-02 16:32:00,652 WARN  [Caesium-1-4]  c.a.b.i.s.i.IndexingSynchronizationService Unable to establish a connection to the search server during index synchronisation.
2023-11-02 16:32:00,653 WARN  [Caesium-1-4]  c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob  Job id: b259e598-83f0-471c-86b0-b88f5b6c4d80
2023-11-02 16:32:32,406 WARN  [pool-6-thread-1]  c.a.u.p.MpacApplicationCacheManager Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: org.apache.http.conn.ConnectTimeoutException: Connect to marketplace.atlassian.com:443 [marketplace.atlassian.com/104.192.142.15, marketplace.atlassian.com/104.192.142.16, marketplace.atlassian.com/104.192.142.17] failed: connect timed out
2023-11-02 16:33:17,458 WARN  [pool-6-thread-1]  c.a.u.p.MpacApplicationCacheManager Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: org.apache.http.conn.ConnectTimeoutException: Connect to marketplace.atlassian.com:443 [marketplace.atlassian.com/104.192.142.16, marketplace.atlassian.com/104.192.142.17, marketplace.atlassian.com/104.192.142.15] failed: connect timed out
2023-11-02 16:35:44,508 WARN  [hz.hazelcast.cached.thread-5]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:50155 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:37:44,496 WARN  [hz.hazelcast.cached.thread-2]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:58749 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:39:44,498 WARN  [hz.hazelcast.cached.thread-2]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:49205 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:41:44,496 WARN  [hz.hazelcast.cached.thread-4]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:36281 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:43:44,492 WARN  [hz.hazelcast.cached.thread-2]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:53859 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:45:44,492 WARN  [hz.hazelcast.cached.thread-4]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:49217 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:47:44,492 WARN  [hz.hazelcast.cached.thread-5]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:37833 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
2023-11-02 16:49:44,495 WARN  [hz.hazelcast.cached.thread-4]  c.a.s.i.c.DefaultClusterJoinManager CONNECT(10.42.1.101:58063 -> 10.42.3.112:5701): Node authentication failed: Unexpected bytes from remote node, closing socket
Reply

Suggest an answer

Log in or Sign up to answer
Bitbucket
Bitbucket
DEPLOYMENT TYPE
SERVER
VERSION
8.5.0
TAGS
AUG Leaders

Atlassian Community Events

    See all events