Last week I posted an analysis of negative reviews of large paid Marketplace apps and asked whether app downtime was a real problem. Four people took the time to show me it was not the question I thought it was. I said I would publish the data, so here it is, and then one open question instead of another hypothesis of mine.
The data: github.com/VovikP/jira-app-reviews — 431 one- and two-star reviews from 22 of the largest paid apps, with dates and app names, plus two scripts: one regenerates the data from the public API, the other reproduces every number I quote below, keyword list and permutation-test parameters included. No key needed. Re-running the fetch today returned all 431 plus one row where the author had since changed 2 stars to 1.
Four things I had wrong. Three came from this thread; I found the fourth while writing this up.
Nick pointed out that an unauthenticated probe from outside cannot separate a rate limit, or one tenant's trouble, from a real outage. That is decisive against the thing I was actually building — one unauthenticated GET per vendor — and it is why I stopped. It is not an argument that no tracker could work: Hugo sketched one later in this thread that would, with real workflows, several regions and tenant-awareness. That is a different and much larger piece of engineering, and it is not a side project.
Richard's account of how a Statuspage actually gets updated — a person decides, after monitoring or tickets suggest something is wrong, because publishing an outage that never happened is worse than publishing late — is his, from running these systems, and I am repeating it rather than verifying it. I had been reading the page as a monitoring feed that was underperforming. On his account it was never trying to be one.
Hugo pushed on the statistics, so I tested my own claim instead of defending it. I had said the technical share of complaints is rising. This data cannot show that. Everything before 2025 gives 50.8% (n=364), 2025-2026 gives 61.2% (n=67): +10.4 points at a permutation p of 0.15. Narrow the earlier window to 2019-2024 and it is 52.2% (n=320), so +9.0 points at p=0.22. The remaining 44 reviews predate 2019 and sit at 40.9%. Neither comparison is significant and 2020 sits as high as 2025, so the claim is retracted.
I should be careful about how far that goes in the other direction. Failing to detect a rise is not the same as showing there is none, and here the sample is weak enough that the distinction matters: at these sizes the test has 34% power against the difference it is looking at, so a real effect of that size would be missed about two times in three. The script prints that figure. And “technical” here means my keyword list, not an objective category: a wider list moves the same comparison to +9.7 points at p=0.19. The p-value does not make the variable less of a judgement call. Both lists are in the script.
The fourth is mine: while preparing this post I found that I had labelled that earlier bucket “2019-2024” in my replies above, when the figures actually cover everything before 2025, back to 2010. The conclusion is unchanged, but the window was mislabelled and the corrected numbers are in the README.
Aaron suggested two disputed reviews might be a coordinated campaign. I checked the app they both belong to, and this time defined the test before running it. Its full negative history is 183 reviews over 93 months, averaging 1.97 a month. May, June and July 2025 ran 3, 3 and 2. A month with three or more happens in 27 of those 93 months, and a Poisson process at that rate predicts 31.5%. The three-month total of 8 is matched or beaten by 29 of the 91 three-month windows in the app's history. So that stretch sits around the two-thirds mark of the app's own normal range — there is no unusual concentration to explain.
What that does and does not settle: it argues against a campaign large enough to show up as a spike. It says nothing about a campaign of two reviews, which would look exactly like this. Your localized-error explanation remains equally consistent with everything I can see, and nothing here separates the two. The function is in the script if you want to point it at another app.
So, the open question. I went in with an answer and got corrected, which was the useful outcome but not an efficient way to find a real problem. So this time I am not proposing anything.
If you administer Jira Cloud with paid apps: what actually costs you time?
Not what is annoying in principle — what you personally lost an afternoon to in the last few months. Configuration that will not stay put, a migration that ate a week, permissions that behave differently in one app than another, reporting you rebuild by hand every month, something you gave up on entirely.
I will collect whatever comes back and post the summary here, whether or not anything in it turns out to be buildable. If the honest answer is “nothing, it mostly works”, that is a useful result too — it is roughly what Sebastian already said about his own instance, and it deserves to be counted rather than argued with.
