Indexing forever!

[Andrew DeFaria]

I recently checked in a change to Perforce and I go to Crucible/Fisheye and attempt a review. I cannot find my changelist. It's not there. I see it says "The repository 'Perforce' is being scanned, information from the repository may not be up to date (Status message: Indexing revision metadata item 3041 of 138940)". Well that'll probably take a long time. It took a day or two to index all of the non-metadata. Does one need to wait so long for this updating to take place before one can submit a review? How do I explain this to the users:

User: I want to do a review but I can't find my changelist. Where is it?

Me: Try back in a few hours/days...

That doesn't appear to me to be an acceptable answer. The server was saying "Indexing revision metadata" with different numbers last night when I went home. Apparently it's still not done. So I tried stopping and restarting it and the above is what I get. Exactly how long does all this indexing take?

[Radu Dumitriu · Accepted solution]

So, your problem is that you have a big Perforce repo. Now, I am not an expert on FE and I didn't work with Perforce, but just googling a bit the FE documentation I came with the following stuff:

https://confluence.atlassian.com/display/FISHEYE/Perforce

Check: Block size, Start Revision (you can start at a meaningful revision), most probably Throttle connections-per-sec

Must read: https://confluence.atlassian.com/display/FISHEYE/Improve+FishEye+scan+performance

Take a better look at the no 4 / no 2 (already mentioned) / no 3 / no 7

Have fun.

[Andrew DeFaria]

Thanks but I understand how to configure the repository already. As for various possibilities of improving performance, do you really think that potential customers should be spending their time tweaking such tunables just to evaluate the product? I don't. Out of the box it should work well. After all it's trying to showcase your product in order to get a sale. IOW while I'll spend some time on this, I'm not dedicating my working days to it just to eval the product.

Said differently, Swarm, from Perforce, already does this without the protracted indexing, storage and CPU costs and demands on my time to understand the underlying database and tweak the tunables.

I'm not saying that this is necessarily a bad product or anything like that - but as somebody just trying to evaluate the product to make a recommendation... Well which would you choose given the info presented so far? If Atlassian wishs to make a better showing then I would say that at the least this indexing process should either complete very early in the evaluation period or the evaluation license should either be extended or not even started until the indexing has finished. AFAICT this is just a normal Perforce repo - not like it's Google's repo (http://www.perforce.com/blog/110607/how-do-they-do-it-googles-one-server-trick)!

[Andrew DeFaria]

Our evaluation is expiring in 15 days and I still haven't managed to get the indexing to complete! It seems to chug along for a while and then gets stuck. I stop and restart it and it seems like it's continue on but then gets stuck again moving nowhere, even after a whole weekend to work on it. Effectively we are getting no evaluation! Help!

[Neil Taylor]

You can open a ticket directly with Atlassian here: https://support.atlassian.com/customer/home

They might have some ideas. From experience with my Fisheye/Crucible which is tied in with Subversion, it can definitely take a long time (days/weeks) to index large repos. In the event it will take longer than the evaluation period, I'm sure talking to Atlassian they would understand your dilemma of not having any time to actually "evaluate" and might work with you on extending your evaluation license.

My experiences with Atlassian Support have always been positive. Best of luck!

[Andrew DeFaria]

It would seem to me that it would make a lot more sense to index new things before old things... Just a thought...

[JamieA]

It's quite possible that it really will take all the eval period to index, it just depends on the size and "shape" of your repository. To give some perspective, a full reindex would take several months at my instance, and the caches on disk are over 2 terabytes.

You could try filtering out branches, as it's probably slow at processing them, but bear in mind if you do that you need to reindex from scratch.

And no, I am no fan of fisheye/crucible to be honest. The future is git and stash.

[Andrew DeFaria]

Of git I say - it's a version control system that only a kernel engineer could love...

That said, months of reindexing and terabytes of storage requirements?!? Not looking good for the eval... Not being able to handle shelved commits is the other thing that will kill this (IMHO).

I've killed the indexing, remove the repository, symlinked the cache directory to a filer so I have enough space and then added the repository back pretty much eliminating any of the branches. I hope this will finish in time. I know, however, that if I let this loose for the local community to evaluate and play with I'm sure all of them will want to play with the work they are actively working on now, probably in branches that I have not indexed.

[JamieA]

FWIW you can renew the one-month eval on the website, up to 5 times, without speaking to anyone.

However, I suspect Swarm is a better option for you, particularly if your focus is code reviews and not "repository insight".

Something made by Perforce, for Perforce only, is probably going to work better that Fe/Cru. p4 is not one of the main platforms that their customers use, so probably gets less coverage, tuning.

This was a deal breaker for me in my perforce days anyway: https://jira.atlassian.com/browse/FE-226

[Andrew DeFaria]

That's good to know however time is of the essence as they say...

[JamieA]

my.atlassian.com

[Andrew DeFaria]

Care to tell me where this eval website is so I can renew the license? I couldn't find it...

[Andrew DeFaria]

Was there anything I said that was not true?

Before the disks filled it was indexing for quite literally a week! We recently updated our Perforce installsation. The entire database needed to be re-written for the upgrade. That took 20 hours IIRC. This took a week and didn't even finish. We also are evaluating Swarm. Swarm has no such indexing requirement. Install it and it just works - quickly. It also works with shelved submitts, something that Crucible/Fisheye does not support.

The product seems to have a lot more power though possibly a lot more confusion, and I'd like to recommend it. But if I can't get it to index by the time the evaluation license expires and if it will not worked with shelved submits I find it hard to recommend this solution to my client. Question is, if Atlassian wants a sale then I would think they would work with me and help me to answer such questions so I can give a positive recommendation.

Another little tip is to give some indication one when you will be finished with this indexing process. True you say processing changelist x of y and that's cool. But then you go one to "Processing labels" or some other such thing. Bottom line - I have no idea of when you will finish as I dont' know all the 'things" you need to "index". It's not like your are saying "Step x of y - processing label - a of b" so I have some indication of how many more things need to be processed.

Bottom line though is that if it did it in a few hours I wouldn't care. But as it it taking forever I do care.

[Radu Dumitriu]

Hm. I understand frustration, but believe me, I can be aggresive too. I even got negative points on this forum for showing my (long and sharp) tongue to people. What's not true?

Anyway, what's good is that more info is getting on the surface. Now we find that the disks are full. Can you imagine that it might be a problem?

[Andrew DeFaria]

You mean the Atlassian support guys do not bother to cruise this community and answer questions? Well that's dumb. Sorry but I've never really delt with Atlassian so I'm not even sure where to go to get support. I thought I'd get support here...

Gee https://confluence.atlassian.com/display/Support/Atlassian+Supportsays:

4. Join our Answers community

Share and discuss questions with the Atlassian community. Get your questions answered by community experts as well as Atlassian employees. 30% of questions are answered within the first hour!

Hmmm guess that's not true. And there are 3 steps before this! #1 is to check out their help page. Did that. Found nothing. #2 was to tour the knowledge base - lands me on a page that is missing. Redirects me by hand (again I'm doing all the work here...) seems to be lots of problems related to indexing. One mentions (https://confluence.atlassian.com/display/FISHKB/Repository+Stuck+at+Info:+Indexing+review+data) being stuck and suggests re-indexing. That'll take weeks and my evaluation period would be expired. Not good. Also, these indexes are filling my disks! We have only 18G on the root partitian of this VM and it's now 100% full! I can't even index our repository therefore I can't evaluate this product!

[Radu Dumitriu]

How about getting some support on it? I'm sure Atlassian guys are pretty friendly ...

My 2 cents would be a problem on the perforce side, but since I'm not a fisheye expert ...

Edit: to clarify, this is not a support site, but a forum.

[Andrew DeFaria]

It's now saying: Status message: Indexing content: Import/VSS/Devel/pandroid/u-boot/board/omap5912osk/config.mk (688001 of 1640927). It's been literally weeks of "indexing". If nobody can address this question then all I can say is I cannot recommend to my client to use this tool as it is not ready for prime time yet and you lose a sale...