CVE-2021-44228 Atlassian using log4j 1.2.17

[niclas.grimskar@citynetwork.eu]

Hi.

A Major vulnerability has been published named CVE-2021-44228, and looking into our Atlassian products, a fairly old version of log4j is used all across the different products.

What actions does Atlassian recommend to mitigate this vulnerability?

[andrewbrock]

BitBucket server bundles elasticsearch, which does have the affected log4j artifacts present.

[Systems and Infrastructure]

In the FAQ for this CVE Atlassian is saying that Bitbucket Server & Data Center are not affected but I was just thinking the same. Elasticsearch in Bitbucket 7.6.10LTS comes with log4j-core-2.11.1.jar. And according to Apache this version is vulnerable. Should Atlassian not recommend the replacement of this lib?

[John Price]

I wonder the same thing.  For example, we also use SonarQube, which is not vulnerable per the vendor, but it uses ElasticSearch and so they recommended a change to the JVM arguments to mitigate.

[andrewbrock]

Here's what I've done in the absence of any official confirmation/guidance.

In the BitBucket home directory there's a file called shared/search/jvm.options

Change this:

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

...to this:

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
# CVE-2021-44228 mitigation
-Dlog4j2.formatMsgNoLookups=true


This next one may be unnecessary, but just to be safe, I edited bitbucket/7.5.0/bin/_start-webapp.sh and changed this:


#JVM_SUPPORT_RECOMMENDED_ARGS=

...to this:

JVM_SUPPORT_RECOMMENDED_ARGS=-Dlog4j2.formatMsgNoLookups=true

[Radek Janata]

Thanks, @andrewbrock, this has been so far the most useful information related to potential vulnerability in ElasticSearch embedded in Bitbucket.

Regarding this vulnerability in Bitbucket, you can also check dedicated thread:
https://community.atlassian.com/t5/Trust-Security-questions/Log4J-vulnerability/qaq-p/1885867

[dylan-nicholson]

Has anyone found a way of upgrading the log4j2 package instead? We're trying to remove all copies of the jar with the vulnerability in it (but it's definitely needed for the elasticsearch service to start).

At it is, the elasticsearch service that comes with bitbucket only listens on the loopback address, so it can't be access externally.  At worst, somebody might be able to interactively login to the bitbucket server as a low-privileged user, send a message to the elasticsearch service and execute code in the context of that service's credentials, but there's no good reason to have any low-privilege users that allow interactive login to the bitbucket server anyway.  

[John Price]

I've read the FAQ at https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html but I'm not clear whether plugins for Data Center/Server could be affected.  We have a ton of plugins.  Anyone know?  It seems plugins would just leverage the main log4j component installed with Jira/Confluence, but I'm not sure.

[Christian Bär]

+1, good question. I think we both are assuming, that plugins just leverage the main loh4j component, but it would be nice if someone from Atlassian would acknowledge that. 

Atlassian !?

[Rick Carini]

In the newly released document, Multiple Products Security Advisory - Log4j Vulnerable To Remote Code Execution - CVE-2021-44228 | Atlassian Support | Atlassian Documentation (accessed, 14-Dec-2021, 12:30 AM PST)

Atlassian mentions that they have “...identified third-party apps that are vulnerable”.

DATA CENTER AND SERVER APPS

Atlassian is also scanning and reviewing data center and server apps. Similar to cloud apps, Atlassian has yet to discover apps developed by Atlassian that are vulnerable to CVE-2021-44228, but have identified third-party apps that are vulnerable. Each vulnerable DC or server app will be given the same expedited deadline as cloud apps. DC and server apps that fail to address the vulnerability within this expedited timeframe will be removed from the marketplace, and then Atlassian will inform customers who have vulnerable apps installed.

Finally, Atlassian is encouraging all cloud, DC and server apps vulnerable to CVE-2021-44228 to rotate their shared secret, and to directly communicate with customers themselves about their efforts to mitigate the situation.

[Daniel Eads]

Hi all,

Daniel from Atlassian Support here. I'd just like to provide you with this preliminary FAQ related to the log4j zero-day. Our Security team is currently investigating the impact of the Log4j remote code execution vulnerability (CVE-2021-44228) and determining any possible impacts. In the meantime, hopefully this FAQ will help address some initial questions you may have.

Thanks,
Daniel Eads | Atlassian Support

[krandell]

We do not have JMS Appender enabled in our configuration and were still hit by a malware attack on our Confluence server yesterday.

It was the same malware that hit us in August due to this vulnerability:

https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html

Obviously we have since upgraded, currently on 7.13.0

Given that log4j 1.2 was end of life in 2015 and has other security vulnerabilities logged against it, I'm shocked that it's still in use.

https://logging.apache.org/log4j/1.2/

[Rafael Corredor]

Hi @Daniel Eads,

Is there any official way to follow this topic? I am already watching some pages in the community, but it is not possible to watch the initial faq, https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html.

How it is assumed we can be aware of any update related Atlassian investigation?

Thank you

[Matt Baillargeon]

Has there been any updates yet from the Security Team at Atlassian? Curious if it will be something minor (replace log4j-1.2 jar file) or a full upgrade is required.

[Daniel Eads]

Hi all,

Daniel with Atlassian Support here to let you know our security team has finished its investigation. We have an official response statement here on Community, which you can access at this link.

Additionally there is more information available on our advisory page, as well as the previously-published FAQ:

• https://confluence.atlassian.com/display/SECURITY/Multiple+Products+Security+Advisory+-+Log4j+Vulnerable+To+Remote+Code+Execution+-+CVE-2021-44228
• https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html

Thanks,
Daniel Eads | Atlassian Support

[andrewbrock]

Hi,

Is there a reason why BitBucket Server isn't mentioned anywhere in either of those links? What about the bundled elasticsearch product?

[Evan Underwood-Harley]

@Daniel Eads - "Atlassian has yet to discover apps developed by Atlassian that are vulnerable to CVE-2021-44228, but have identified third-party apps that are vulnerable"

Can you share which third party apps are vulnerable?

[David Busby]

AS the third party has their own legal rights and perview over their response to the matter, I am not sure this will be possible but will be happily surprised if this can be provided, at least, to assure that Plugins are or are not affected, which I am taking the inference to mean that "third-party apps" refers to plugins which deployments may or may not be using.

[Daniel Eads]

@andrewbrock The advisory has been updated with additional information about Bitbucket Server and the bundled elasticsearch:

https://confluence.atlassian.com/security/multiple-products-security-advisory-log4j-vulnerable-to-remote-code-execution-cve-2021-44228-1103069934.html

[Vijay Sv]

You need to check-in <installation directory>\lib\log4j-1.2.17-atlassian-3.jar.

Jira 8.13.x is using log4j version 1.2.17.

CVE-2021-44228 is affected with version 2 of log4j between versions 2.0-beta-9 and 2.14.1. It is not present in version 1 of log4j and is patched in 2.15.0.

[Vijay Sv]

btw i am talking Jira Server version.

[Leon Lehmann]

Mh for log4j v1.2.17 there exists a RCE vulnerability since 2019: https://nvd.nist.gov/vuln/detail/CVE-2019-17571

[Peter]

Just to note.... 2.15 has another vulnerability.  2.16 is minimum version (for now).

[Tobias]

2.16. is also not enough at all, there was another vulnerability found: https://www.zdnet.com/article/apache-releases-new-2-17-0-patch-for-log4j-to-solve-denial-of-service-vulnerability/

Apache already released version 2.17.0 - as a reference of @Vijay Sv there is an existing vulnerability wat @Leon Lehmann already mentioned. Furthermore the support for the version 1.x already ended in August 2015.

So what will Atlassian do in the future? Do you as Atlassian team assume responsibility for a possible attack? @Daniel Eads 

[Daniel Eads]

Hi @Tobias , please refer to the Atlassian advisory for impact on Atlassian products, and then elastic's announcement for more impact information related to the bundled elasticsearch product in Bitbucket Server. Both these articles take the information from the initial CVE-2021-44228 and follow-up CVE-2021-45046 into consideration.

Edit: Our security team has updated the FAQ (not the advisory itself) to explicitly include CVE-2021-45105 and indicate no impact.

[Clark Everson]

Hi niclas.grimskar@citynetwork.eu 

Honestly there was a more recent security incident than that: https://confluence.atlassian.com/security/multiple-products-security-advisory-unrendered-unicode-bidirectional-override-characters-cve-2021-42574-1086419475.html

The only path for it is to upgrade to the recommended versions. If you want to have an easier path forward then using LTS version (for Confluence it would be 7.13.x as the latest) would be the easiest path as these are supported for 2 years.

Upgrading will take care of all security issues that are currently known but as the CVE from November has no other path this would be the best path for any old ones you have as well as the newest ones.

CVEs can be tracked here: https://confluence.atlassian.com/security/atlassian-security-229839985.html

Usually they have a temporary fix but in general the long term fix is upgrade. LTS's make that easier as security issues are patched on them for two years as long as you install the patch, which usually involves much less testing then full version upgrades as they are designed that you can just install the patch and be safe.

Best,

Clark

[Christian Elsner]

(...)
Best Regards

[Clark Everson]

Hi @Christian Elsner 

Thanks for the information, honestly I couldn't find any documentation on it when I looked. However, for this user, the upgrade path to LTS would still be the recommended route because when they have a fix an LTS just requires you to do the patch upgrade and will continue to do so for two years from release.

Mitigations are only temporary and usually cause loss of features, Keeping LTS's updated resolves these issues.

Best,

Clark

[Vincent Kopa (Ovyka)]

Hi

You say "there was a more recent security incident than that" ?

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228 is far more recent than https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42574

[Shannon Fabert]

The fix for the unicode bidirectional threat does not address CVE-2021-044228.  It does mitigate CVE-2021-42574. Per another thread, Atlassian products are not affected by log4j issue because it is running on version 1 not version 2.  Upon further research, Atlassian is still gathering information on using log4j 2.

[Ramiro Encinas]

There is a jar (\Atlassian\JIRA\atlassian-jira\WEB-INF\lib\log4j2-stacktrace-origins-2.2-atlassian-2.jar) that apparently refers to the version 2.x

[Tom Bell]

Same in our version of Jira v8.13 LTS.  Do we know if the JMSAppender vulnerability applies to log4j v2 versus 1.2?

[Maurice H.]

check: https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html for latest answers to the topic

[Damian Rosair]

Official Atlassian FAQ for the vulnerability:
https://confluence.atlassian.com/kb/faq-for-cve-2021-44228-1103069406.html

[Shyam]

How do we know what is the version of log4j used by Atlassian DC servers especially Jira , Bitbucket and Confluence. Does it display in the UI of the server properties? 

[Tobias]

Hello together, 

now I think Atlassian has to investigate fast because there are new findings that V.1.x is not safe enough. 

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046 

This is the CVE which is the main reason for 2.16.0.
For this kind of attack, either the "log4j2.noFormatMsgLookup" property nor the 2.15.0 helps.

[Bill Bailey]

There is another thread related to this topic/ https://community.atlassian.com/t5/Data-Center-questions/Is-Confluence-Data-Center-server-vulnerable-to-CVE-2021-44228/qaq-p/1884158 

Seems the version Atlassian is using is not impacted.