Why Atlassian web sites are so unstable?

[Sergey Svishchev]

Time and time again, CAC is showing shocked Atlassian logo instead of documentation. Often, several sites at once go down.

What's up, Atlassian?..

[Joe Clark · Accepted solution]

Hi Sergey,

CAC is currently suffering from some ongoing performance issues for which the root cause has not yet been identified. Within the last 48hrs a number of changes have been made which should improve the recent poor performance.

When several sites go down at once, it may be related to an outage in our unified login system Atlassian ID (https://id.atlassian.com). The development team who maintain Atlassian ID have some planned improvements to improve the resilience of systems that integrate with it.

Hope this is helpful!

Joe

[Sergey Svishchev]

Joseph,

Are those issues something that a customer should be aware of or they are specific to Atlassian's environment?

Maybe Atlassian could share their experience in running a large public installation of Confluence (see https://answers.atlassian.com/questions/34250/confluence-100k-usersfor context).

[Joe Clark]

Hi again, Sergey,

The root cause is still unknown, and the CAC architecture is probably very different from a typical customer's environment, so learnings from this outage may not be 100% applicable to other users.

However, I can point out that the immediate performance problems were solved temporarily by putting a Varnish cache server in front of CAC for anonymous users - this was only possible due to our SSO configuration with Atlassian ID.

[Sergey Svishchev]

Interesting. Is there anything special about Varnish that made it suitable for the task?

I am currenlty using nginx (with disk cache) and it does the job quite well -- plus, it is extremely useful when occasional feature starts misbehaving and needs to be blocked :-)

Perhaps you can also answer https://answers.atlassian.com/questions/256355/? Our instance isn't quite at 100K users yet but I'd take anything that helps increase the app's performance.

[Joe Clark]

I believe Varnish was chosen because a high percentage of CAC's traffic comes from anonymous, unauthenticated sources. This means that in certain circumstances, the rendered pages can be served directly from a proxy cache rather than causing code execution within the Confluence app.

[Joe Clark]

I'm not sure what to say about https://answers.atlassian.com/questions/256355/- lazy loading of the Confluence editor should be enabled by default in recent versions of Confluence.

[Sergey Svishchev]

OK, now the graphs are gone from status.a.c :)