Process polkitd using 100% of CPU on Bamboo Server

[peterst28]

Our Bamboo Server (Atlassian Bamboo version 5.14.4.1 build 51417 - 12 Jan 17) on Ubuntu 16.04.2 LTS has been running painfully slowly for a couple of weeks.  I logged into the server, ran top, and saw the following:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1351 bamboo    20   0  222692   3376   1024 S 100.0  0.1  40:39.93 polkitd
 1350 bamboo    20   0  222692   3384   1024 S  99.7  0.1  40:38.74 polkitd

I have tried to figure out what these processes are doing, but I have seen nothing interesting.

ps:

ubuntu@ip-172-31-32-221:~$ sudo ps -ef | grep 1351
bamboo    1351     1 93 16:24 ?        00:43:54 ./polkitd -c oybmfqrnv.conf -t 1

Strace:

Looks like it's just waiting.

ubuntu@ip-172-31-32-221:~$ sudo strace -f -p 1350
strace: Process 1350 attached with 4 threads
strace: [ Process PID=1359 runs in x32 mode. ]
[pid  1357] futex(0x2648a94, FUTEX_WAIT_PRIVATE, 13, NULL <unfinished ...>
[pid  1350] futex(0x7f1a7e7d29d0, FUTEX_WAIT, 1357, NULL <unfinished ...>
[pid  1358] select(4, [3], NULL, NULL, {593, 77339} <unfinished ...>
[pid  1359] gettimeofday(0x7f1a7d7cfcf0, NULL) = 0
strace: [ Process PID=1359 runs in 64 bit mode. ]
[pid  1359] gettimeofday({1497373791, 989477}, NULL) = 0

lsof:

ubuntu@ip-172-31-32-221:~$ sudo lsof -p 1351
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
polkitd 1351 bamboo  cwd    DIR  202,1     4096 269040 /var/tmp
polkitd 1351 bamboo  rtd    DIR  202,1     4096      2 /
polkitd 1351 bamboo  txt    REG  202,1  1200944 262342 /var/tmp/polkitd
polkitd 1351 bamboo    0r   CHR    1,3      0t0      6 /dev/null
polkitd 1351 bamboo    1w   CHR    1,3      0t0      6 /dev/null
polkitd 1351 bamboo    2u   REG  202,1    11194    880 /tmp/tmpfj1lOZF (deleted)
polkitd 1351 bamboo    3u  IPv4  16649      0t0    TCP ip-172-31-32-221.ec2.internal:40513->ns3004370.ip-176-31-117.eu:http (ESTABLISHED)

tcpdump:

Not much happening.

ubuntu@ip-172-31-32-221:~$ sudo tcpdump port 40513
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:13:38.807610 IP ip-172-31-32-221.ec2.internal.40513 > ns3004370.ip-176-31-117.eu.http: Flags [.], ack 3445976208, win 596, options [nop,nop,TS val 676992 ecr 1367780029], length 0
17:13:38.889476 IP ns3004370.ip-176-31-117.eu.http > ip-172-31-32-221.ec2.internal.40513: Flags [.], ack 1, win 310, options [nop,nop,TS val 1367792571 ecr 670748], length 0

[Gonchik Tsymzhitov]

Looks like it was this one

https://access.redhat.com/discussions/3534761