Bitbucket DC (2 nodes) behind HAProxy getting "Too many redirects"

[kns_jeinhorn]

I am setting up a new Bitbucket Data Center environment.  I am using 6 virtual machines:

• juptsthaka01 - running haproxy and keepalived
• juptsthaka02 - running haproxy and keepalived
• juptstsrch01 - running elasticsearch
• juptstpgsql01 - running PostgreSQL
• juptstbitb01 - running Bitbucket Data Center 7.21
• juptstbitb02 - running Bitbucket Data Center 7.21

Pretty picture:

keepalived on the two haproxy/keepalived servers is working fine and manages virtual IP xx.yy.zz.46, and I don't think the keepalived.conf files are relevant to my problem.

THE PROBLEM IS:  My ability to connect and work in BitBucket is buggy and inconsistent. 

• Sometimes I enter my credentials on the login page and get "This page isn't working.  juptstbitbucket.mycomp.com redirected you too many times.  Try clearing your cookies.  ERR_TOO_MANY_REDIRECTS" (clearing cookies does not help) and then 2 seconds later, the login is successful . . . but later, pages stop loading.
• Other times, I enter my credentials on the login page, and the login page re-displays, with no error, and this repeats indefinitely.

I have scoured countless Atlassian pages (many of them erroneous) about how to achieve this setup, including:

• https://confluence.atlassian.com/bitbucketserver/secure-bitbucket-behind-haproxy-using-ssl-779303273.html   -   This seems to apply to environments with a single back-end server, and has no information about cookies / sticky sessions 

• https://confluence.atlassian.com/bitbucketserver/install-bitbucket-data-center-872139817.html - BETTER 

• https://confluence.atlassian.com/bitbucketserver/migrate-server-xml-customizations-to-bitbucket-properties-897811761.html  
• https://confluence.atlassian.com/bitbucketserver0721/configuration-properties-1115666478.html 

Here is my haproxy.conf from juptsthaka01:

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global

log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
ssl-default-bind-options no-sslv3
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# Stop "Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you
# should set it to at least 2048." Warning
tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
# Use the log line that was set in the global section. Putting a "log global" directive
# in the defaults section is equivalent to putting it in all of the subsequent proxy sections.
log global
# By default, output from HAProxy is minimal. Adding the line "option httplog" enables
# more verbose HTTP logging.
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
# Workaround for Chrome 35-36 bug. See http://blog.haproxy.com/2014/05/26/haproxy-and-http-errors-408-in-chrome/
errorfile 408 /dev/null

frontend bitb_http_frontend
bind xx.yy.zz.46:80
bind xx.yy.zz.46:443 ssl crt /etc/ssl/local/juptstbitbucket.pem ciphers RC4-SHA:AES128-SHA:AES256-SHA
default_backend bitb_http_backend

backend bitb_http_backend
mode http
option httplog
option httpchk GET /bitbucket/status
option forwardfor
option http-server-close
# Uncomment the following line for HAProxy 1.5.
# (Ref. https://confluence.atlassian.com/bitbucketserver/install-bitbucket-data-center-872139817.html)
appsession BITBUCKETSESSIONID len 52 timeout 1h
balance roundrobin
cookie BITBUCKETSESSIONID prefix
# The following 3 lines are for HAProxy 1.6+. If you're on 1.5, uncomment them. <<<< WTF
#stick-table type string len 52 size 5M expire 30m
#stick store-response set-cookie(BITBUCKETSESSIONID)
#stick on cookie(BITBUCKETSESSIONID)
server bitb01 juptstbitb01.mycomp.com:7990 check inter 10000 rise 2 fall 5
server bitb02 juptstbitb02.mycomp.com:7990 check inter 10000 rise 2 fall 5
server bitbbackup01 juptstbitb01.mycomp.com:7990 backup
server bitbbackup02 juptstbitb02.mycomp.com:7990 backup

listen admin
mode http
bind *:8090
stats enable
stats uri /

Note that the sample code on this page:   https://confluence.atlassian.com/bitbucketserver/install-bitbucket-data-center-872139817.html     has the instruction "The following 3 lines are for HAProxy 1.6+. If you're on 1.5, uncomment them", which is NONSENSE.  Since my haproxy version is 1.5, I left the "stick" lines commented out.

[ftwit-admin@juptsthaka01 ~]$ sudo haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

Here is the relevant portion of bitbucket.properties.  (I tried both with and without the "server.redirect-port=443"):

#
# 2022-05-17 (JRE): Use context path /bitbucket
server.context-path=/bitbucket
#
# 2022-06-17 (JRE): Implementing haproxy front end
server.port=7990
server.secure=true
server.scheme=https
server.proxy-port=443
#server.redirect-port=443
server.proxy-name=juptstbitbucket.mycomp.com
#

According to this:    https://confluence.atlassian.com/bitbucketserver0721/configuration-properties-1115666478.html       server.session.cookie.name has default value BITBUCKETSESSIONID

I would be tremendously grateful if someone could identify the flaws in my config files.

Many, Many Thanks !!!!!

Janet

[kns_jeinhorn · Accepted solution]

The source of my struggles was the very, very, very bad statement on Atlassian's page  https://confluence.atlassian.com/bitbucketserver/install-bitbucket-data-center-872139817.html     

As I mentioned, the page has these very, very, very bad instructions:  

"The following 3 lines are for HAProxy 1.6+. If you're on 1.5, uncomment them"

I am using haproxy 1.5 (1.5.18, specifically) and after I UNCOMMENTED the three "stick" lines, my configuration worked.