OnDemand products are very slow

[René Mulder]

We are currently evaluating Confluence, JIRA, GreenHopper, Bamboo and Bitbucket. The evaluation will run for two more weeks, but we are already experiencing that the responsiveness of the entire stack is not what we expected.

We're very enthusiastic about the functionality, but performance is currently balancing between just-workable and not-workable. i.e. more than 5 seconds to load a page.

Browsers tested are latest versions of chrome, firefox and safari. Results are comparable

Is there anything we can do to improve this? for example, is our deployment running at an inefficient data center. Is there a difference in performance between the OnDemand evaluation and the OnDemand subscription?

[Peter Abramowitsch · Accepted solution]

I also am seeing really dismal performance with Atlassian OnDemand (confluence+jira+agile) and was hoping we wouldnt have to host the product ourselves. I have attached a net trace which hopefully will show your team which requests are the culprit. I know there was some discussion about the activity stream a few years back, but this is now.... Any help appreciated as 36 second page loads are not acceptible.

[Harry Chan]

Most of the time here is spent on GET batch.js and GET batch.css. It's either huge latency (not likely) or just slow server response (likely).

[Martin_Kirk]

Maybe they should also cut down on the amount of Javascript.. 

Chrome Dev tools counted 36 MB just load load the frontpage / dashboard... 

[Jónas Tryggvi Jóhannsson]

Solving this is simple for Atlassian; stop serving server.css and server.js from their own servers and move assets to a CDN. This will ensure that everyone is fetching the JavaScript and Stylesheet assets from a server that is optimized for that and as close to the end user as possible latency wise, while we only fetch generated pages from Atlassian servers.

[Harry Chan]

I think all OnDemand products are hosted in Contegix at St. Louis. Atlassian can clarify on that one.

Where are you located? Are you facing network - latency / download / upload issues? It may be that you are too far away from that data center and will need a different solutions - self hosted or 3rd party hosted.

You can also try to get Atlassian support to tune / tweak your instance if it helps.

[Joe Clark]

Just chiming in to say that the OnDemand data center is in Ashburn, Virginia, on the US East Coast.

Contegix still offers its own premium service for Atlassian hosting: http://www.contegix.com/solutions/atlassian-managed-application-hosting/

[René Mulder]

Hi Joseph,

Thanks for chiming in. Good to hear something from someone from Atlassian. Your response worries me a bit though. Pointing to Contegix suggests that Atlassian means for their own OnDemand offering to be so slow. Surely that can't be the case and you are working on this right?

Regards,

Rene

[Joe Clark]

Hi!

Sorry if I gave that impression. I wasn't meaning to make any insinuations as to whether or not third party hosting services are faster or slower than Atlassian's own hosting services. I just wanted to highlight that even though Atlassian's own hosting isn't in the Contegix Datacenter any more, that option is still available for customers who want it through Contegix directly.

I don't have any information to share about OnDemand performance improvements, sorry, but I would be surprised if the team responsible for it wasn't already working out how to address the concerns raised here.

[Richard Donkin]

Would be good to have an Atlassian update on the work on these performance problems, since they've been going on for a year and many people have reported them.

I can understand temporary issues, but to have these still going on a year later makes it seem like Atlassian is not serious about the OnDemand service...

[Richard Donkin]

There's an open issue on JIRA that's being worked on now, which seems to be behind at least part of the poor performance of OnDemand, but maybe not all: https://jira.atlassian.com/browse/JRA-37337- have a look, and if you are seeing this issue, add a comment (and maybe Watch and Vote).

[Andrew Sinclair]

I am in Australia the home of Atlassian and I can definitely say Atlassian On Demand is by far the slowest of all the SaaS services I use!

[Noam Lewis]

One more thing - the performance seems to be inconsistent. There are times where it's great. It's not consistently slow.

[Noam Lewis]

Latency is not the issue. In our experience the OnDemand servers are simply slow. I opened a ticket and they did certain things that were supposed to increase performance for our instance, but it still has an ocassional (not rare) hiccup where you wait 15-30 seconds for a page to load. To be fair, sometimes it's fast. But sometimes very slow too.

[Noam Lewis]

JIRA OnDemand is very slow for us as well. We're evaluating it and this could be the reason for not continuing past the evaluation.

[Warren Poon]

Has anyone contacted JIRA support about this and gotten an explanation? I'm glad to find out that we're not the only ones suffering from this lag. The hosting cost for OnDemand doesn't seem worth it since it's quite rudimentary to spool up an instance w/ any Atlassian products you have licenses for. On the other hand, they're obviously hosting in the cloud as well, so it's similarly rudimentary for them to spool up more servers if processing power is the bottleneck.

I'd like to know what the source of the latency is.

[Adam Mosam]

Just saw Renee's post - did you hear back from JIRA support?

[René Mulder]

Harry, Thanks for your response. I did open a support request with Atlassian.

I'm located in The Netherlands, but part of the problem was indeed in the proxy server I was behind.

From a faster network, responsiveness was better, especially after disabling the 'Activity Stream', which as I learned, traditionally takes a long time to load.

[Richard Donkin]

JIRA OnDemand is still extremely slow sometimes. We're in the UK, and loading a new screen takes tens of seconds sometimes - for example, searching for a single word across 500 issues took 27.3 seconds - when refreshing to repeat search, it took 21.2 seconds (JQL is text ~ "keyword").

• Trello, which we use for non-software tasks and product planning, takes just 2 seconds for the same search across several thousand items - 15 times faster and searching more data!

The ping time to our JIRA server is between 90 and 140ms, with no packet loss, so it's not the network, just the very slow JIRA app.

JIRA is by far the slowest SaaS app we use. I would really like to use it in more areas as the functionality is good. Because JIRA is so slow, we are likely to use a non-Confluence wiki, as by all accounts this has the same performance problems - direct revenue loss to Atlassian.

[Glownet]

Same problem with really impossible slowness of Confluence OnDemand here in Russia.

[Shek Singhal]

We are using Atlassian JIRA OnDemand from the SF Bay Area as well. It is very slow when loading a page type for the first time after browser open (but this is when it matters the most). The cause seems to be the server-side generation of the batch.js file. Our network connection has no problem in downloading the 611KB compressed batch.js file (The batch.js file is 2.2MB uncompressed, 611KB gzip compressed).

The cause of the latency does not seem to be in downloading this 611KB file, but rather about 5 seconds spent waiting for the 2.2MB batch.js uncompressed file to be created. There is considerable lag in generating this batch.js file.

The generation of the batch.js file needs to be optimized. See Network tab on Chrome:
http://img820.imageshack.us/img820/4572/6mr.gif

[anu]

Hi - sorry for the poor experience you have had with our OnDemand service so far. We recognize that this is a serious issue and are working on ways to improve our performance and make it better.

As part of this exercise, we've fixed the issue at https://jira.atlassian.com/browse/JRA-37337and are deploying this into production shortly.

We currently have a dedicated team working on identifying root causes for slow performance on our OnDemand service and fixing those. If you are interested in helping us test these fixes, please reach out to abharadwaj @ atlassian.com

- Anu

[Stephan Vierkant]

Come oooooooooooooooooooooooooooooooooon!

Waiting for 30+ seconds is not unusual. We are getting more and more frustrated by it and are looking for alternatives. Why does this take so long?

[Jónas Tryggvi Jóhannsson]

OnDemand products are unusable most of the time for us now. I can't belive that this has been an issue for a year without any fixes from Atlassian.

[Stephan Vierkant]

JIRA is too slow for us (The Netherlands). Waiting for 15-50 (!) seconds for one page is not unusual. Other websites (all around the world) are much faster.

Atlassian, please stop thinking US-only!

[Jordan Ambra]

Atlassian is located in Australia. Also, it's slow for us here in the US too.

[Yves Berquin [Matrix Requirements]]

I am an on-demand customer from Belgium.

The traceroute to my atlassian.netinstance reveals it's located in the US, apparently in a NTT Australia network location in Denver, Colorado. It takes a little over 100ms to cross the Atlantic. I am using it from two different cities, and get the same results.

There will never be fast responses for us until Atlassian install some of their servers in Europe.

I suspect they are also not using enough server horse power, because during our morning (when the Americans sleep) it seems quite a lot faster than during our evening.

[Harry Chan]

Latency is 1 thing, but it's not the bulk of the problem here - it's mostly a lack of memory and CPU power per instance.

Considering the pricing Atlassian offers the OnDemand products at - it's hard. We try to keep prices more reasonable due to the added cost of trying to make it run more smooth and offer memory upgrades to customers.

[Igor Silva]

We bought a Jira onDemand license and since the first day, the problem persisted, the solution is very slow, until now some of my users are refusing to use the solution in this conditions. We need answers.

[Paul Zehle]

We (my customers, partners and my own company; each with own onDemand-Accounts) also have serious problems with the page loading speed for onDemand products.

Sometimes it tooks over 10 seconds to refresh/load pages! Working in flow is unthinkable most of the time.

Our companies are based in Berlin, Germany and we use different ISPs.

Unfortunately this problem exists for months now...

[Nicki Messerschmidt]

I want to chime in to this. We are trying to use Jira ondemand and it is sometimes "slow as hell" (like my coworker likes to say).

Most of the time it is near unbearable.

Our instance is https://link-m.atlassian.net

Our internet connection is usually 10-20ms (to 8.8.8.8) but about 100-200ms to link-m.atlassian.net. So there is network latency but on the atlassian side...

[Harry Chan]

The added latency is likely offset by compression and doesn't seem to be the culprit here.

[Melvin Gruesbeck]

I am a potential buyer testing your system. It is still slow as a slug. Dec. 29, 2013

[dauciunas]

OnDemand is slow - especially after some period of inactivity. 

Looks like it needs to warm up.. but still page load time average at about 5 seconds. 

using from London and Vilnius

[Josh McCullough]

I live next door to Ashburn, VA (where the data center is, supposedly), and JIRA is so slow it is not usable at all on my 50Mb FiOS connection!

[Philippe Schueller]

Same experience here with JIRA... very, very long load times. Initial loading of the Dashboard frequently takes over 30 seconds while other pages are usually around 5-15 seconds. Once cached it is a bit faster, but still rather sluggish. We have been using the service for over a year now and have noticed little improvement.

[JeanA]

I'm going to add to what Anu said above. We take this seriously and have rolled-out recent infrastructure changes adding CDN support for static content for JIRA and Confluence in OnDemand. Our tests show a 1.5-2x improvement in end-user page load times. The product related changes to increase the surface area of resources that are cachable are rolling out over the next couple of weeks.

The infrastructure and product teams are continuing to work on performance that will not just help OnDemand but the download versions as well. Thanks for all the feeback!

[Stephan Vierkant]

Is this an improvement that relates to JRA-37337? If so, I'm not sure a CDN will help: even a dial-up internet connection from Ugunda will serve a static file faster than 60+ seconds. A 2x improvement sounds impressive, but that's still more than 30 seconds.

[Julia Wester [Wittified]]

that's good that the download versions are being looked at as well. In my experience, the load time of the first GET is too long in Confluence even for the server version. On our on-demand version we are still very regularly seeing extremely slow response times.

[Barry Mason]

I've cancelled JIRA Agile and as soon as I find an alternative for the bug tracker, that will be cancelled as well. I've sent a tweet and email to JIRA asking them about it and never recieved a reply. It really is just too slow and it doesn't sound like anything will be done about it anytime soon.

Can anyone reccommend an alternative?

[Anders Kreinøe]

YouTRACK looks like it could be an interesting alternative. I have not testet it, but i have considered it a lot while waiting on Jira page loads.

http://www.jetbrains.com/youtrack/