Should bounties be eliminated from the new infrastructure?

[Norman Abramovitz]

I do not think bounties should be eliminated when Answers changes to use Confluence Questions. I know for me that a bounty got me to:

1. I at least looked at the question because it made me curious what the real issue is
2. A bounty is not for the answer rate, but for the looking at rate. Even if I knew that I did not know answer to the question, I sometimes stil read the question. Even if the question was answered I would sometimes go back and read the accepted answer.
3. A bounty atracted my eye to the question, so indirectly it raises the question priority.
4. When I got caught up with trying to reach the next Karma level (for those great Atlassian gifts), I would go the extra mile on answering the question with bounties.
5. I do not think bounties should raise the answer rate since the timeframe is based upon on when I factually login to look at questions to answers. I answer questions in the late evening or early morning times.

How has bounties affected you?

Has bounties caused anyone to have a negative reaction to the question?

1. When I notice a person using bounties all the time, I sometimes skip over the question.

Actually, I rather see badges elminiated since I never answered a question because I wanted a badge. I am always surprised that I got a badge. A badge is something the system or other people have given to me indirectly. Now if a badge earned me some karma points, then my opinion would change.

[Joe Clark · Accepted solution]

Norman, these are all great questions. Thankyou for asking them. I think many of them are less about bounties and more broadly about understanding the usage patterns of Answers in general - we at Atlassian should always be concerned about the overall 'health' and experience of the answering community.

I have left them as open questions for now, since even without this additional layer of analysis, I am already in favour of having bounties as a feature in CQ. This feature is being tracked by the CQ team as https://jira.atlassian.com/browse/CQ-791- I'm going to see if we can actually move this in to a "must have" for Answers.

One of my long term goals is to have this kind of 'health' analysis available for all users to see as a dashboard or something, rather than relying on sporadic analysis of the DB from within Atlassian.

I think allowing a minimum bounty to be configured would be useful, too.

My analysis only shows the bounty offered in addition to the default 25 karma for accepting an answer.

[Norman Abramovitz]

It is true the questions are about usage patterns with an emphasis towards bounty usage. I was looking for some additional 'facts' that was qualitatively measurable that could assist in this discussion.

Thank you for your time and being open to discuss this topic.

[Norman Abramovitz]

I would have edited my previous comment, but somehow editing seems to be broken now. I do not get my old text anymore to fix things up, so I do not know if I am editing my comment or creating a new comment.

Well, anyhow, you might want to update your FAQ about bounties in a similar to the way you discussed badges.

[Joe Clark]

Yes, I just noticed that editing comments is broken, too. I'll raise a JIRA issue to track it, but it's unlikely that we'll push out a fix for it unless our timeline for Confluence Questions changes drastically.

I have a FAQ update on my todo list :) We're now aiming to get bounties in for the migration.

[Joe Clark]

Hi All!

Your responses to this question made me question my initial plan to make a decision on bounties based purely on eyeballing the site and using intuition.

I've now run some SQL queries on a database export from last night, and wanted to share the results with you:

How many bounties have been offered?

select count(*) from forum_questionbountyoffer;

count

4938

How many questions have had a bounty offered on them?

select count(distinct question_id) from forum_questionbountyoffer;

count

4908

How many questions have had a bounty offered as a percentage of all questions asked?

How many questions are there in total?

select count(id) from forum_node where node_type = 'question';

count

73772

Now do the math...

4908 / 73772 * 100% = 6.65%

How many users have offered a bounty, as a percentage of all users who've ever asked a question?

How many users have ever asked a question?

select count(user_ptr_id) 
from forum_user
where user_ptr_id in (
    select distinct author_id 
    from forum_node 
    where node_type = 'question'
);

count

30101

How many users have offered a bounty?

select count(distinct user_id) from forum_questionbountyoffer;

count

2890

Now do the math...

2890 / 30101 * 100% = 9.6%

What bounties are offered?

select amount as bounty, count(amount) as times_offered
from forum_questionbountyoffer
group by amount
order by amount asc;

bounty times_offered

1 2216

2 312

3 395

4 82

5 733

6 97

7 10

8 19

9 7

10 418

11 15

12 5

13 1

15 81

16 17

17 1

19 2

20 154

21 6

22 1

23 1

24 2

25 75

26 6

27 1

28 1

29 2

30 58

31 2

34 1

35 8

36 4

37 3

40 23

42 1

45 4

46 2

47 1

50 95

51 2

55 1

56 2

57 1

60 2

65 1

66 1

70 4

75 5

76 1

77 1

80 1

91 1

94 1

100 34

104 1

145 1

150 3

200 7

300 1

500 2

1893 1

2012 1

6761 1

What's the answer rate for bountied questions?

How many questions are there?

(from earlier): 73772

How many questions have an answer?

select count(q.id) from forum_node q
where node_type = 'question'
and q.id in (select distinct parent_id from forum_node where node_type = 'answer')

count

62385

Now do the math...

62385 / 73772 * 100% = 84.5%

How many bountied questions are there ?

(from earlier): 4908

How many bountied questions have an answer?

select count(distinct q.id) from forum_node q
where node_type = 'question'
and q.id in (select distinct parent_id from forum_node where node_type = 'answer')
and q.id in (select distinct question_id from forum_questionbountyoffer);

count

4426

Now do the math...

4426 / 4908 * 100% = 90.18%

What about if we exclude outler bounties (eg. bounties more than 500 karma) and unhelpful bounties (eg. bounties less than 5 karma):

select count(distinct question_id) from forum_questionbountyoffer where amount >= 5 and amount <= 500

count

1927

And how many are answered?

count

1789

Now do the math...

1789 / 1927 * 100% = 92.84%

What's the 'accepted answer' rate for bounties?

How many questions are there?

(from earlier): 73772

How many questions have an accepted answer?

select count(q.id) from forum_node q
where node_type = 'question'
and q.id in (select distinct parent_id from forum_node where node_type = 'answer' and marked = true)

count

36348

Now do the math...

36348 / 73772 * 100% = 49.27%

How many bountied questions are there?

(from earlier): 4908

How many bountied questions have an accepted answer?

select count(distinct q.id) from forum_node q
where node_type = 'question'
and q.id in (select distinct parent_id from forum_node where node_type = 'answer' and marked = true)
and q.id in (select distinct question_id from forum_questionbountyoffer);

count

3044

Now do the math...

3044 / 4908 * 100% = 62.02%

How many views do bountied questions get?

What's the average view rate for all questions?

select avg(extra_count)
from forum_node
where node_type = 'question';

avg

417.67

What's the average view rate for bountied questions?

select avg(extra_count)
from forum_node
where node_type = 'question'
and id in (select distinct question_id from forum_questionbountyoffer)

avg

398.49

What's the average view rate for bountied questions, excluding outliers?

select avg(extra_count)
from forum_node
where node_type = 'question'
and id in (select distinct question_id from forum_questionbountyoffer where amount >= 5 and amount <= 500)

avg

398.49

[Joe Clark]

My key takeaways:

• Answers has really good base answer rates, still! 84% answer rate and 50% accept rate are pretty darn good!

• The answer rate and accept rate are 10% better for questions with bounties! Wow, I did not expect that. A great result!

• The feature usage for bounties is pretty low - about 10% of users and 10% of questions. A good rule of thumb is that a feature should be used by about ~10% of users or else it should be cut. Given that bounties are a feature that I would expect to have less usage then normal, I think this is pretty good - it gets used regularly enough to be worthwhile.

• View statistics were not influenced by bounties, but this is not an accurate statistic, since our view counts do not distinguish real users from search bots and RSS feeds.
• The majority of bounties are offered for extremely low amounts (less than 5 karma). I do not think this is useful. Some bounties are extremely high and are probably people switching karma between two accounts (eg. because they changed email address).

[JamieA]

Thanks Joe, great intel. Very open, and minimal bullshit. You are living the values ;-)

Some of the numbers don't seem to actually lead to the numbers in the conclusions, maybe just my misreading or copy/paste issue.

I'll be happy whatever you decide, given that you have all the facts.

[Norman Abramovitz]

Interesting numbers. Your criteria that bounties should effect answer rate at least has been met.

Can you tell if the karma level of the persons that answered, commented and accepted answers? Assuming a higher karma level means more experience (major assumption here), does the person with more experience answer bounty questions more often than non-bounty questions? Does the person with more experience comment on bounty questions more often than non-bounty questions?

Again, if we could assume that a karma level means a better quality of answer, could we infer the quality of the answers overall on answers? Are your more experienced (higher karma) people still answering questions? Are the bounty questions getting a better quality of answer(ie higher karma persons)?

Could you tell 'newness' of the person offering bounties on answers? If you compare the creation date of the account to the creation date of the question, we might be able to infer the type of person asking the questions? Are bounties being given out by people who are new atlassian products (or at least new answers)?

I am looking to give you some more food for thought about bounties with qualitative numbers behind them.

Does bounty questions attracts your more experienced users?

Are bounties created by your new users to answers?

I am hoping answers to these questions would weigh on your decision about bounties.

[Norman Abramovitz]

I think maybe bounty numbers might be low because you are displaying the default 25 points plus the bounty. When I initially saw the bounty, I thought it was 25 points plus another 26 points (ie bounty of 1 karma). You could raise the minimum bounty karma as well. It depends on the type of users that actually use the bounty.

[Joe Clark]

No need to apologize!

That's interesting but from our POV, as users, that's beside the point. I don;t really care why it might be slow, just that it not be slower than answers.

Sure, I get that. I guess my point was that people sometimes look at CAC as an example of Confluence not performing well, without taking into consideration that it's basically the biggest and most heavily-trafficked Confluence server on the public Internet. AAC's usage is much further within the "normal" bounds for Confluence operation and I don't envisage any major problems (although my gut has already been proven wrong once in this thread, so who knows if that means anything!). At any rate, we are definitely doing some performance testing with the AAC production data as a sample set before going live, and super bad performance would definitely stop the migration until it was addressed. I think anything less than a 10% performance degradation would still be acceptable to go ahead, with some room for future improvement.

[JamieA]

Hi @Joseph.. sorry for the accusative nature of my, err, accusation. The reason I am struggling with it is that, given a question with a bounty, personally I am more likely to answer it, and to try to provide a fuller answer. Not that much more inclined, but a bit. So I'm surprised if that does not also apply to others, and that that doesn't influence the stats. I do think that it increases the quality of answers, but you won't see that with a quantitive analysis.

> CAC has an order of magnitude increase in traffic compared to AAC (something like 3-5x, I think), based on uniques and total hits, so it's not an ideal comparison

That's interesting but from our POV, as users, that's beside the point. I don;t really care why it might be slow, just that it not be slower than answers.

I see that you're going to do this and it makes sense from a commercial perspective, I'm just asking you to keep an eye on performance.

> I'm happy to have an honest discussion about it here... open company, no bullshit and all that :-)

Yup... "answers is my girlfriend" etc. I do seem to spend more time with answers than her ...

[Joe Clark]

I'm waiting for a recent export of the production DB and then will post some analysis here. The last one I did was from an export that was ~ 6 months old, so I think it would be good to re-assess with the most recent state.

[Joe Clark]

@JamieI haven't "massaged" any stats, but I will admit to performing only a cursory analysis of bounties usage. I'm happy to have an honest discussion about it here... open company, no bullshit and all that :-)

If the data shows that bounties did increase the view rate, is that a successful case for bounties? The asker may still not have gotten a good answer, even if 5x the number of people viewed it. My gut says that if the bounties don't increase the answer rate, they're just a distraction.

CAC has an order of magnitude increase in traffic compared to AAC (something like 3-5x, I think), based on uniques and total hits, so it's not an ideal comparison. The earlier performance problems with AAC were 'solved' by throwing hardware at it, the underlying codebase is still littered with performance issues.

We have provided the Confluence Questions dev team with a full dump of the AAC prod database, which they are using to import into CQ for performance testing. I feel pretty confident that performance will be as good, if not better, than it is currently.

Also with Confluence Data Center on the horizon, we're going to gain some additional headroom for more scaling if needed.

[Norman Abramovitz]

I look forward to the statistics and discussion. I am hoping more people will express if bounties made a difference in using answers.

[JamieA]

I agree with Norman. Badges blow... I have no idea what badges I have, they're not shown anywhere prominently... and I would not do something to get a badge.

In the FAQ it says that bounties didn't significantly change the answer rate. I struggle a bit to believe that. I'd also like to see how much they influenced the view rate. Maybe the stats are being massaged a bit to fit with the roadmap for Confl Qs. Fair play, I would do the same.

Even if they did not change the answer rate, perhaps the bounty questions were that much harder, and they would not have had any answers without a bounty. Or perhaps the quality was higher as people tried to get the bounty.

[JamieA]

Another concern I have is that performance of CAC is pretty bad, along with frequent outages. After some great work by the Answers team, the performance of AAC is now pretty good, and I'd hate to lose all that.

[Mirek]

For my perspective .. It could stay, but .. please rescrict ability to create a bountry for people that do not have enough karma. I see a lot of questions with bountries from people that have 1 karma and bountry is set to 25, 50 .. As I remember if person have 1 karma (or less than a bounty) nad he acceppt the answer you will not recive 25 or 50 karma..

-Mirek

[Joe Clark]

Hey Norman! Thanks for your input - it's interesting to get your perspective on bounties. I want to reply to your question with some stats + data from the Answers DB, but I'm out of the office right now and can't get to it.

When I'm next in, I'll extract some and post it here and we can discuss it some more. I'm open to the possibility of keeping bounties alive, but my initial investigation into the raw numbers didn't seem to indicate it was worth it..