The Atlassian Marketplace server is not reachable after certificates installation.

[Alex Makhov]

Hi all,

After installing confluence I have a error: "The Atlassian Marketplace server is not reachable. " . In logs I see:

2022-04-12 16:29:32,170 WARN [http-bio-8090-exec-1] [atlassian.confluence.cache.TransactionalCacheFactory] warning Transactional cache update outside transaction. All updates to this cache should be performed fro
m a thread with a valid transaction context.
-- referer: http://10.205.8.196:8090/authenticate.action?destination=%2Fplugins%2Fservlet%2Fupm | url: /rest/stp/1.0/license/status | userName: admin
2022-04-12 16:29:33,066 WARN [http-bio-8090-exec-5] [atlassian.upm.pac.PacClientImpl] unknown Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: javax.net.ssl.SSLPeerU
nverifiedException: peer not authenticated
-- referer: http://10.205.8.196:8090/authenticate.action?destination=%2Fplugins%2Fservlet%2Fupm | url: /plugins/servlet/upm | userName: admin
2022-04-12 16:29:35,365 WARN [http-bio-8090-exec-1] [atlassian.confluence.cache.TransactionalCacheFactory] warning Transactional cache update outside transaction. All updates to this cache should be performed fro
m a thread with a valid transaction context.
-- referer: http://10.205.8.196:8090/plugins/servlet/upm | url: /rest/stp/1.0/license/status | userName: admin
2022-04-12 16:29:36,681 WARN [http-bio-8090-exec-2] [atlassian.upm.pac.PacClientImpl] unknown Error when querying application info from MPAC: com.atlassian.marketplace.client.MpacException: javax.net.ssl.SSLPeerU
nverifiedException: peer not authenticated
-- referer: http://10.205.8.196:8090/plugins/servlet/upm | url: /rest/plugins/1.0/product-version | userName: admin

I've tried to install certificates like it's recommended in the article:

https://confluence.atlassian.com/confkb/the-atlassian-marketplace-server-is-not-reachable-due-to-peer-not-authenticated-321850263.html

But it didn't help. Also I don't understand why in the article there are 3 certificates to download:

openssl s_client -connect marketplace.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > marketplace.atlassian.com.crt
openssl s_client -connect plugins.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > plugins.atlassian.com.crt
openssl s_client -connect api.media.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > api.media.atlassian.com.crt

But only 2 certs to isntall:

keytool -import -alias marketplace.atlassian.com:443 -keystore /path/to/truststore -file /path/to/marketplace.atlassian.com.crt
keytool -import -alias plugins.atlassian.com:443 -keystore /path/to/truststore -file /path/to/plugins.atlassian.com.crt

I've tried to isntall all 3, or only 2 as it's in the article, but no changes in logs.

The active java parameters from ps aux:

conflue+ 351657 92.4 10.4 12548036 850720 pts/2 Sl 17:13 2:01 /srv/confluence/jre//bin/java \
-Djava.util.logging.config.file=/srv/confluence/conf/logging.properties \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx10240m \
-XX:MaxPermSize=1024m -Djava.awt.headless=true -Dhttp.proxyHost=proxy.my.server.net \
-Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.my.server.net -Dhttps.proxyPort=3128 \
-Dhttp.nonProxyHosts=*.my.server.net|*.services.lan|*.corp.mycompany.com|localhost|ius-app009.services.lan|*.atlassian.com \
-Djava.endorsed.dirs=/srv/confluence/endorsed -classpath /srv/confluence/bin/bootstrap.jar:/srv/confluence/bin/tomcat-juli.jar \
-Dcatalina.base=/srv/confluence -Dcatalina.home=/srv/confluence -Djava.io.tmpdir=/srv/confluence/temp org.apache.catalina.startup.Bootstrap start

 Could you please advise me how can I fix the issue?

[Fabio Racobaldo _Herzum_]

Hi @Alex Makhov ,

please could you check if your proxy is correctly configured?

Confluence, in order to reach the Atlassian Marketplace, should be able to reach the external network.

Dhttp.proxyHost=proxy.my.server.net \
-Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.my.server.net -Dhttps.proxyPort=3128

[Alex Makhov]

Thank you for your reply.

Proxy settings look okay, names were changed to hide real servers names.

I also tried to play with nonProxyHosts parameters with included and excluded *.atlassian.com but results are the same.

Now all 3 certificates are added. Do you know if it's okay?

# keytool -list -keystore cacerts -storepass mypass | grep atlas
marketplace.atlassian.com:443, Apr 12, 2022, trustedCertEntry, 
api.media.atlassian.com:443, Apr 12, 2022, trustedCertEntry, 
plugins.atlassian.com:443, Apr 12, 2022, trustedCertEntry, 

[Fabio Racobaldo _Herzum_]

Please verify that your proxy can reach the Atlassian marketplace. 

You don't need to add cert to cacert for Atlassian Marketplace.

[Alex Makhov]

I can reach the Marketplace with curl using the same proxy. IS it a valid test? 

# curl -I -x proxy.my.server.net:3128 https://marketplace.atlassian.com/ 
HTTP/1.1 200 Connection established

HTTP/2 200
date: Wed, 13 Apr 2022 09:12:02 GMT
content-type: text/html
content-length: 7385
last-modified: Mon, 11 Apr 2022 08:01:54 GMT
x-amz-version-id: 4_mKr_HAwJqJFoo5ToPz3G8TfRTjpF7i
server: globaledge-envoy
etag: W/"b0a440bfaef381ad841baab0e51a7dec"
via: 1.1 e0a78b49206aba2a7e76eb45b9688a8e.cloudfront.net (CloudFront),1.1 varnish (Varnish/6.5)
x-amz-cf-pop: IAD89-P2
x-amz-cf-id: 42SOMhOXJNjJ9nFg_5T-W1XqkZhT3DkDUPSpss6nMCIAMVX5xtnWLg==
access-control-allow-origin: *
x-varnish: 344030221 340280408
age: 120
vary: Accept-Encoding,Origin
x-cache: HIT
accept-ranges: bytes
x-envoy-upstream-service-time: 92
expect-ct: report-uri="https://web-security-reports.services.atlassian.com/expect-ct-report/marketplace-athreos", max-age=86400 
strict-transport-security: max-age=63072000; preload
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
atl-traceid: 0812b4e359233bc6
report-to: {"group": "endpoint-1", "max_age": 600, "endpoints": [{"url": "https://dj9s4kmieytgz.cloudfront.net"}], "include_subdomains": true} 
nel: {"report_to": "endpoint-1", "max_age": 600, "include_subdomains": true, "failure_fraction": 0.001}

[Alex Makhov]

I also see this message in the logs:

2022-04-13 10:24:39,760 WARN [scheduler_Worker-10] [atlassian.upm.notification.PluginLicenseNotificationCheckerImpl] updateRemotePluginLicenseNotifications Automatic license update check failed: com.atlassian.upm
.mac.HamletException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

[Fabio Racobaldo _Herzum_]

@Alex Makhov , curl check should be a valid check. Did u try to restart your application?

[Alex Makhov]

Yes, I've restarted it several times with different proxy parameters, but results are the same. 

Now my proxy settings are below:
JVM_PROXY_OPTS="-Dhttp.proxyHost=proxy.my.server.net -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.my.server.net -Dhttps.proxyPort=3128"

But no changes.

[Fabio Racobaldo _Herzum_]

are u sure that you imported the cert into the correct cacert associated to application?

[Alex Makhov]

I have only one cacerts file. It's in the /srv/confluence/jre/lib/security directory.

JRE_HOME="/srv/confluence/jre/"

Can I specify this file directly with the option?

-Djavax.net.ssl.keyStore=path/to/cacerts

And also, which certificates must be installed and in which order?

As I said there are 3 certs to download:


openssl s_client -connect marketplace.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > marketplace.atlassian.com.crt
openssl s_client -connect plugins.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > plugins.atlassian.com.crt
openssl s_client -connect api.media.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > api.media.atlassian.com.crt

but only 2 to install:

keytool -import -alias marketplace.atlassian.com:443 -keystore /path/to/truststore -file /path/to/marketplace.atlassian.com.crt
keytool -import -alias plugins.atlassian.com:443 -keystore /path/to/truststore -file /path/to/plugins.atlassian.com.crt

[Alex Makhov]

@Fabio Racobaldo _Herzum_ 

When I run tcpdump on the confluence host, I see some traffic to the post 3128, so proxy settings look working.

# tcpdump port 3128 -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:51:55.937359 IP (tos 0x0, ttl 64, id 30206, offset 0, flags [DF], proto TCP (6), length 60)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [S], cksum 0xa102 (incorrect -> 0xa1b9), seq 3864229577, win 64240, options [mss 1460,sackOK,TS val 2814714464 ecr 0,nop,wscale 7], length 0 
10:51:55.938745 IP (tos 0x28, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [S.], cksum 0x2311 (correct), seq 2301375038, ack 3864229578, win 28960, options [mss 1460,sackOK,TS val 913184399 ecr 2814714464,nop,wscale
7], length 0
10:51:55.938755 IP (tos 0x0, ttl 64, id 30207, offset 0, flags [DF], proto TCP (6), length 52)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [.], cksum 0xa0fa (incorrect -> 0xc105), seq 1, ack 1, win 502, options [nop,nop,TS val 2814714466 ecr 913184399], length 0
10:51:55.940273 IP (tos 0x0, ttl 64, id 30208, offset 0, flags [DF], proto TCP (6), length 165)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [P.], cksum 0xa16b (incorrect -> 0x1080), seq 1:114, ack 1, win 502, options [nop,nop,TS val 2814714467 ecr 913184399], length 113
10:51:55.941464 IP (tos 0x28, ttl 58, id 23632, offset 0, flags [DF], proto TCP (6), length 52)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [.], cksum 0xc1a5 (correct), seq 1, ack 114, win 227, options [nop,nop,TS val 913184400 ecr 2814714467], length 0
10:51:55.966080 IP (tos 0x28, ttl 58, id 23633, offset 0, flags [DF], proto TCP (6), length 91)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [P.], cksum 0xb7b7 (correct), seq 1:40, ack 114, win 227, options [nop,nop,TS val 913184406 ecr 2814714467], length 39
10:51:55.966086 IP (tos 0x0, ttl 64, id 30209, offset 0, flags [DF], proto TCP (6), length 52)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [.], cksum 0xa0fa (incorrect -> 0xc04b), seq 114, ack 40, win 502, options [nop,nop,TS val 2814714493 ecr 913184406], length 0
10:51:55.992909 IP (tos 0x0, ttl 64, id 30210, offset 0, flags [DF], proto TCP (6), length 240)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [P.], cksum 0xa1b6 (incorrect -> 0x1441), seq 114:302, ack 40, win 502, options [nop,nop,TS val 2814714520 ecr 913184406], length 188
10:51:56.002377 IP (tos 0x28, ttl 58, id 23634, offset 0, flags [DF], proto TCP (6), length 59)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [P.], cksum 0x6262 (correct), seq 40:47, ack 302, win 235, options [nop,nop,TS val 913184415 ecr 2814714520], length 7
10:51:56.002385 IP (tos 0x0, ttl 64, id 30211, offset 0, flags [DF], proto TCP (6), length 52)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [.], cksum 0xa0fa (incorrect -> 0xbf5b), seq 302, ack 47, win 502, options [nop,nop,TS val 2814714529 ecr 913184415], length 0
10:51:56.002529 IP (tos 0x0, ttl 64, id 30212, offset 0, flags [DF], proto TCP (6), length 52)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [F.], cksum 0xa0fa (incorrect -> 0xbf59), seq 302, ack 47, win 502, options [nop,nop,TS val 2814714530 ecr 913184415], length 0
10:51:56.002544 IP (tos 0x28, ttl 58, id 23635, offset 0, flags [DF], proto TCP (6), length 52)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [F.], cksum 0xc06e (correct), seq 47, ack 302, win 235, options [nop,nop,TS val 913184415 ecr 2814714520], length 0
10:51:56.002547 IP (tos 0x0, ttl 64, id 30213, offset 0, flags [DF], proto TCP (6), length 52)
nlut-confluence11.services.lan.55046 > proxy.my.server.net.3128: Flags [.], cksum 0xa0fa (incorrect -> 0xbf58), seq 303, ack 48, win 502, options [nop,nop,TS val 2814714530 ecr 913184415], length 0
10:51:56.003812 IP (tos 0x28, ttl 58, id 53739, offset 0, flags [DF], proto TCP (6), length 52)
proxy.my.server.net.3128 > nlut-confluence11.services.lan.55046: Flags [.], cksum 0xc062 (correct), seq 48, ack 303, win 235, options [nop,nop,TS val 913184416 ecr 2814714530], length 0
^C
14 packets captured
19 packets received by filter
0 packets dropped by kernel

So probably it's certificates issue, 
Could you please advise which certificates must be added?
All 3  
marketplace.atlassian.com.crt
plugins.atlassian.com.crt
api.media.atlassian.com.crt

?

[Fabio Racobaldo _Herzum_]

Hi @Alex Makhov ,

try to import marketplace.atlassian.com.crt and marketplace-cdn.atlassian.com.crt :

openssl s_client -connect marketplace.atlassian.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > marketplace.atlassian.com.crt
openssl s_client -connect marketplace-cdn.atlassian.com:443 -servername marketplace-cdn.atlassian.com < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > marketplace-cdn.atlassian.com.crt

[Alex Makhov]

Certificates are imported, service restarted, but nothing changed, still cannot reach marketplace.

# keytool -list -keystore cacerts | grep atlassian 
Enter keystore password: changeit
marketplace.atlassian.com:443, Apr 14, 2022, trustedCertEntry,
marketplace-cdn.atlassian.com:443, Apr 14, 2022, trustedCertEntry,

[Alex Makhov]

I've made a fresh installation to a VM which can reach https://marketplace.atlassian.com without proxy 

$ curl -I https://marketplace.atlassian.com/
HTTP/2 200

But result is the same - "The Atlassian Marketplace server is not reachable"

SoI see 2 possible reasons:

1. The application works with wrong  trust store

2. Required certificates are not installed. 

Now  two certificates are imported on the new test machine.

Should I change them or add more certs?

[Alex Makhov]

@Fabio Racobaldo _Herzum_ 

I'm confused, I have a new machine which doesn't use proxy, I've installed all certificates:

$ keytool -list -keystore cacerts | grep -A 1 atla
Enter keystore password: changeit
marketplace.atlassian.com:443, Apr 15, 2022, trustedCertEntry, 
Certificate fingerprint (SHA1): 2E:63:8C:61:CC:EC:56:3F:A5:0A:5C:C2:3B:63:EC:DF:44:97:0C:AA
--
api.media.atlassian.com:443, Apr 20, 2022, trustedCertEntry, 
Certificate fingerprint (SHA1): 85:7C:4C:1E:4F:DA:ED:62:3E:2F:3E:F6:52:FC:D5:BD:76:11:71:3C
--
plugins.atlassian.com:443, Apr 20, 2022, trustedCertEntry, 
Certificate fingerprint (SHA1): 2E:63:8C:61:CC:EC:56:3F:A5:0A:5C:C2:3B:63:EC:DF:44:97:0C:AA
--
marketplace-cdn.atlassian.com:443, Apr 15, 2022, trustedCertEntry, 
Certificate fingerprint (SHA1): 61:97:B7:83:C0:2B:DE:A9:88:84:4A:ED:29:E4:D2:BD:43:B0:53:0E

I've added new JAVA options for a trust store:

-Djavax.net.ssl.trustStore=/srv/confluence/jre/lib/security/cacerts -Djavax.net.ssl.trustStorePassword=changeit

and restarted the confluence application. But I have the same:
The Atlassian Marketplace server is not reachable.

Do you have any ideas why it still happens?  

[katarina.puchrikova]

@Alex Makhov have you managed to solve this out?