Fisheye/Crucible - incremental indexing after pulling each commit from Mercurial repository.

[Tatiana Shambereva]

Fisheye/Crucible - incremental indexing after pulling each commit from Mercurial repository.
It lasts for two hours.

Why do we need this?
How can I disable reindexing?
Why don't we index only the files from the commit?

Fisheye/Crucible Version:4.4.1

[Grzegorz Lewandowski]

Incremental indexing and full reindex are two different things:

* incremental indexing is the thing you'd like to have -> pull newest changes and index new commits only

* reindex means dropping current repository indexes and indexing repository from scratch

In order to see updates on a repository in Fisheye you need incremental index. In order to determine why it takes 2h first of all determine if Fisheye commences incremental indexing or full reindex. You'll be able to find it in debug logs.

If it's really reindex then I suppose somebody is triggering that in some way (plugin, rest webhook etc) 

If it's incremental index then I suggest enabling debug logs and determining which part takes 2h. Is it connectivity issue and Fisheye is having hard time to pull changes up ? Is this a big changeset with massive changes therefore indexing takes its time ? Hard to tell at the moment.

The third option that comes to my mind is the fact that potentially you have repository polling set to every 2 hours in Fisheye. This means Fisheye will look for new changes in repository every 2h, effectively it seems like a lot of time for Fisheye to pick up changes but in fact it's not, it's jus the polling period. 

[Tatiana Shambereva]

Thank you, Grzegorz.

Mercurial - Non-CVS Repository.

Polling Interval - 60s

User-installed add-ons

No user-installed add-ons found in your FishEye instance.

logs:

2017-09-22 18:33:56,572 DEBUG [153:StdOutHandler DPC ] fisheye RepositoryStatus-setMessage - Status change [DPC]: pulling from http://....

...

2017-09-22 18:33:56,572 DEBUG [153:StdOutHandler DPC ] fisheye RepositoryStatus-setMessage - Status change [DPC]: added 1 changesets with 1 changes to 1 files
2017-09-22 18:33:56,772 DEBUG [153:StdOutHandler DPC ] fisheye RepositoryStatus-setMessage - Status change [DPC]: (run 'hg update' to get a working copy)
2017-09-22 18:33:56,888 DEBUG [IncrPing3 DPC ] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false pull http://..... took 569ms
2017-09-22 18:33:56,889 DEBUG [IncrPing3 DPC ] fisheye DvcsProcessMonitor-onBeforeStart - Starting process: /usr/bin/hg --config ui.verbose=false branches -c
2017-09-22 18:33:57,123 DEBUG [IncrPing3 DPC ] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false branches -c took 234ms
2017-09-22 18:33:57,124 DEBUG [IncrPing3 DPC ] fisheye RepositoryStatus-setMessage - Status change [DPC]: Updated from http://....
2017-09-22 18:33:57,124 INFO [IncrPing3 DPC ] fisheye BaseRepositoryScanner-ping - processing repository DPC (DPC)
2017-09-22 18:33:57,124 INFO [IncrPing3 DPC ] fisheye LifecycleRepositoryScanner-doSlurpTransaction - Starting slurp of DPC (DPC)
2017-09-22 18:33:57,124 DEBUG [IncrPing3 DPC ] fisheye RepositoryStatus-setMessage - Status change [DPC]: Indexing has commenced for repository DPC
2017-09-22 18:33:57,125 DEBUG [IncrPing3 DPC ] fisheye Timer-output - -> slurp of DPC mem_usage = 401219968 heapsize = 608698368 freemem = 207478400

...

...

and then a lot of lines...

2017-09-22 18:35:07,413 DEBUG [IncrPing3 DPC ] fisheye DvcsProcessMonitor-onAfterFinished - Finished process: /usr/bin/hg --config ui.verbose=false annotate -r 1b3fc7713b094f6fbe4df8d484752768ef7439e1 -c -f -l ....cs took 151ms
2017-09-22 18:35:07,439 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-3:22760, 3-4:22943, 4-11:22760, 11-14:22913, 14-15:22760, 15-18:32712, 18-23:22760, 23-24:22913, 24-28:22760, 28-29:22913, 29-30:22760, 30-31:25436, 31-33:22760, 33-34:22913, 34-35:22760, 35-36:25436, 36-37:22913, 37-38:25436, 38-40:22760, 40-41:22943, 41-42:22760, 42-46:22943, 46-47:25436, 47-70:22943, 70-71:22760, 71-75:22943, 75-76:22760, 76-78:22943, 78-79:22913, 79-81:22943, 81-82:22913, 82-83:22760, 83-91:22943, 91-103:22760, 103-104:23030, 104-116:23086, 116-117:23030, 117-118:32712, 118-121:23030, 121-122:32712, 122-124:23030, 124-126:22760], hunks=[<5:0, 5:1>, <30:1, 31:2>, <35:1, 37:5>, <37:1, 43:1>], hunksRevId=71856
2017-09-22 18:35:07,451 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-1:2072, 1-2:25438, 2-3:2072, 3-4:2088, 4-5:2057, 5-8:22224, 8-9:32717, 9-10:28228, 10-14:25438, 14-15:24257, 15-16:53075, 16-17:22224, 17-18:22765, 18-20:24257, 20-21:22765, 21-22:22947, 22-23:23653, 23-25:28228, 25-26:22224, 26-27:2057], hunks=[<11:1, 11:0>, <15:1, 14:0>, <17:1, 15:0>], hunksRevId=71857
2017-09-22 18:35:07,455 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-39:54472, 39-40:60038, 40-46:54472, 46-47:57607, 47-58:54472, 58-59:60038, 59-100:54472], hunks=[<96:1, 96:1>], hunksRevId=71858
2017-09-22 18:35:07,459 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-2:53538, 2-3:54490, 3-4:53538, 4-5:54490, 5-7:53538, 7-8:54490, 8-10:53538, 10-11:57053, 11-13:53538, 13-14:60046, 14-16:53538, 16-17:60148, 17-18:60248, 18-19:66655, 19-22:53538, 22-23:54490, 23-24:53538, 24-25:60046, 25-31:53538, 31-32:58192, 32-34:53538, 34-35:56582, 35-36:53538, 36-39:65913, 39-43:53538, 43-76:54490, 76-78:53538, 78-79:54490, 79-92:53538, 92-93:60046, 93-95:53538, 95-96:65970, 96-97:54336, 97-99:59471, 99-100:54336, 100-102:59471, 102-103:54336, 103-104:59471, 104-105:65970, 105-107:54336, 107-108:65970, 108-109:54336, 109-111:59471, 111-112:54336, 112-113:59471, 113-114:65970, 114-115:54336, 115-147:54464, 147-155:57053, 155-156:55516, 156-160:54464, 160-161:60242, 161-162:55516, 162-164:54464, 164-165:58192, 165-169:54464, 169-172:66655, 172-176:58192, 176-177:66655, 177-187:58192, 187-188:56582, 188-194:54464, 194-195:59477, 195-196:54464, 196-197:59477, 197-198:54464, 198-215:59477, 215-217:54464, 217-219:59477, 219-220:54464, 220-227:59477, 227-229:54464, 229-230:58192, 230-231:54464, 231-232:58192, 232-233:54464, 233-234:55516, 234-235:58192, 235-236:54464, 236-253:54490, 253-255:53538], hunks=[<10:0, 10:1>, <241:0, 242:1>, <242:1, 244:1>], hunksRevId=71859
2017-09-22 18:35:07,467 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-2:2098, 2-3:1900, 3-4:2098, 4-5:2111, 5-8:2098, 8-9:52626, 9-10:32832, 10-11:39831, 11-12:37875, 12-14:39831, 14-15:32832, 15-17:39831, 17-18:37875, 18-19:52626, 19-20:39831, 20-36:68903, 36-37:32832, 37-38:39831, 38-39:52626, 39-40:22773, 40-41:39831, 41-42:52626, 42-43:25218, 43-47:52626, 47-48:25218, 48-50:39831, 50-51:22773, 51-53:39831, 53-54:22773, 54-56:39831, 56-58:22773, 58-59:39831, 59-60:52626, 60-61:22773, 61-62:39831, 62-63:52626, 63-64:2485, 64-65:52626, 65-66:54325, 66-67:22773, 67-69:52626, 69-70:39831, 70-71:52626, 71-72:25218, 72-76:52626, 76-77:28241, 77-79:52626, 79-80:2499, 80-81:39831, 81-82:39869, 82-83:52626, 83-84:39017, 84-85:2557, 85-86:2499, 86-87:39831, 87-88:54325, 88-100:53643, 100-101:58313, 101-102:53080, 102-103:58201, 103-104:39831, 104-108:58201, 108-109:53080, 109-111:58201, 111-112:53080, 112-115:58201, 115-116:54325, 116-120:52626, 120-121:39831, 121-123:52626, 123-125:25218, 125-126:52626, 126-127:58313, 127-137:52626, 137-138:39869, 138-139:39831, 139-146:52626, 146-147:39869, 147-148:39831, 148-149:52626, 149-151:39831, 151-153:52626, 153-154:39831, 154-158:52626, 158-159:39869, 159-160:39831, 160-167:52626, 167-168:39869, 168-169:39831, 169-170:52626, 170-172:39831, 172-174:52626, 174-175:39831, 175-179:52626, 179-180:39869, 180-181:39831, 181-188:52626, 188-189:39869, 189-190:39831, 190-191:52626, 191-193:39831, 193-195:52626, 195-196:39831, 196-200:52626, 200-201:39869, 201-202:39831, 202-209:52626, 209-210:22231, 210-211:39831, 211-218:52626, 218-219:22773, 219-220:39831, 220-223:52626, 223-224:39831, 224-227:52626, 227-228:28241, 228-229:39831, 229-230:52626, 230-232:39831, 232-233:52626, 233-235:39831, 235-236:52626, 236-237:22773, 237-238:39869, 238-241:52626, 241-242:39831, 242-244:52626, 244-245:39831, 245-248:52626, 248-249:22773, 249-250:39869, 250-251:52626, 251-252:39869, 252-254:52626, 254-255:39831, 255-257:52626, 257-258:39831, 258-267:52626, 267-268:39869, 268-269:39831, 269-276:52626, 276-277:23661, 277-278:39869, 278-281:52626, 281-282:39831, 282-285:52626, 285-286:30122, 286-287:39869, 287-288:52626, 288-289:39831, 289-290:39869, 290-291:52626, 291-292:39831, 292-293:39869, 293-294:52626, 294-295:23661, 295-296:39869, 296-297:52626, 297-298:39831, 298-300:52626, 300-301:39831, 301-303:52626, 303-304:39831, 304-313:52626, 313-314:39869, 314-315:39831, 315-322:52626, 322-324:39869, 324-325:52626, 325-326:39831, 326-327:39869, 327-328:52626, 328-329:39831, 329-330:39869, 330-333:52626, 333-334:58313, 334-340:52626, 340-341:28821, 341-347:58297, 347-348:58313, 348-349:39831, 349-350:52626, 350-351:22773, 351-353:52626, 353-354:39831, 354-355:52626, 355-356:53634, 356-357:39831, 357-358:53643, 358-359:53634, 359-360:39831, 360-361:52626, 361-362:53634, 362-363:22773, 363-364:52626, 364-365:25218, 365-370:52626, 370-372:25218, 372-373:52626, 373-374:22773, 374-376:52626, 376-377:22773, 377-378:52626, 378-379:25447, 379-384:52626, 384-386:25447, 386-387:52626, 387-388:25447, 388-393:52626, 393-394:25447, 394-395:22773, 395-396:52626, 396-397:22950, 397-398:39831, 398-399:52626, 399-400:22987, 400-401:39831, 401-402:52626, 402-403:22987, 403-404:39831, 404-405:52626, 405-406:22987, 406-407:39831, 407-408:52626, 408-409:22987, 409-410:22950, 410-411:52626, 411-412:22987, 412-413:39831, 413-414:52626, 414-415:22987, 415-416:39831, 416-417:52626, 417-418:30116, 418-419:23052, 419-420:52626, 420-421:22773, 421-422:22950, 422-423:52626, 423-424:22927, 424-425:39831, 425-426:39869, 426-427:22927, 427-428:39831, 428-429:39869, 429-430:22927, 430-431:39831, 431-432:39869, 432-434:22927, 434-435:22950, 435-436:52626, 436-437:23661, 437-438:39869, 438-439:22950, 439-440:39831, 440-441:39869, 441-442:24264, 442-443:39831, 443-444:39869, 444-445:23052, 445-446:23661, 446-447:52626, 447-448:22773, 448-449:52626, 449-450:22773, 450-463:66269, 463-465:2098, 465-467:1900, 467-468:2098, 468-475:24138, 475-478:1900, 478-479:24138, 479-480:64464, 480-481:28241, 481-482:1900, 482-483:2098, 483-484:1900], hunks=[<142:1, 142:0>], hunksRevId=71860
2017-09-22 18:35:07,473 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-13:44880, 13-14:62369, 14-18:44880, 18-19:45186, 19-20:62704, 20-23:45186, 23-29:44880, 29-43:45186, 43-44:62369, 44-48:44880, 48-49:45186, 49-50:62704, 50-53:45186, 53-60:44880, 60-73:45186, 73-79:45692, 79-80:62704, 80-81:45692, 81-82:45229, 82-93:45692, 93-94:45229, 94-96:44880], hunks=[<29:2, 29:0>], hunksRevId=71861
2017-09-22 18:35:07,480 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-3:29250, 3-4:33159, 4-5:29250, 5-6:71677, 6-9:44459, 9-10:29250, 10-11:29851, 11-12:29250, 12-13:71677, 13-14:31838, 14-19:29250, 19-20:71677, 20-23:29851, 23-24:62705, 24-26:29851, 26-27:62705, 27-41:29851, 41-46:31281, 46-48:29851, 48-49:71677, 49-61:29851, 61-62:62705, 62-68:29851, 68-69:31838, 69-70:60461, 70-71:29250, 71-73:31838, 73-74:34505, 74-76:31838, 76-80:34505, 80-83:43666, 83-84:69440, 84-86:43666, 86-87:69440, 87-88:43666, 88-89:65517, 89-90:43666, 90-91:62705, 91-94:69440, 94-96:43666, 96-97:34505, 97-98:71677, 98-104:34505, 104-107:31838, 107-108:29851, 108-109:71677, 109-110:29250, 110-111:29421, 111-112:71697, 112-114:31838, 114-115:71677, 115-119:71697, 119-120:71677, 120-121:31281, 121-123:29421, 123-124:71697, 124-125:31838, 125-126:29421, 126-127:71677, 127-131:71697, 131-132:71677, 132-135:31838, 135-136:71677, 136-138:31838, 138-139:71677, 139-141:71634, 141-142:71677, 142-146:71634, 146-148:31838, 148-151:71634, 151-152:71677, 152-153:71634, 153-155:31838, 155-157:71634, 157-161:31838, 161-172:71677, 172-173:71697, 173-175:31838, 175-176:71677, 176-178:31838, 178-182:71697, 182-184:31838, 184-186:71634, 186-188:31838, 188-190:29250, 190-193:34090, 193-194:71677, 194-196:34090, 196-197:62705, 197-199:44249, 199-201:34090, 201-204:60461, 204-209:71677, 209-214:60461, 214-224:37483, 224-232:37585, 232-233:44459, 233-234:71677, 234-235:44459, 235-236:71677, 236-240:44459, 240-243:71677, 243-250:44459, 250-255:71677, 255-261:44459, 261-262:71677, 262-265:44459, 265-266:62705, 266-271:44459, 271-272:71677, 272-280:44459, 280-281:62705, 281-295:44459, 295-296:62705, 296-310:44459, 310-311:62705, 311-316:44459, 316-317:71677, 317-325:44459, 325-326:62705, 326-340:44459, 340-341:62705, 341-346:44459, 346-347:71677, 347-348:44459, 348-349:29250, 349-350:71677], hunks=[<6:0, 6:1>, <172:0, 173:11>], hunksRevId=71862
2017-09-22 18:35:07,533 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-3:21752, 3-4:43049, 4-5:21866, 5-6:21752, 6-7:33160, 7-8:21752, 8-9:38163, 9-10:43049, 10-13:21752, 13-15:70491, 15-16:43049, 16-17:50824, 17-18:43049, 18-21:21752, 21-22:71609, 22-24:21752, 24-27:51734, 27-35:21752, 35-36:21866, 36-39:70491, 39-40:43049, 40-45:70491, 45-65:43049, 65-68:43563, 68-82:43049, 82-84:43563, 84-85:43049, 85-101:43563, 101-106:43049, 106-111:34506, 111-120:23600, 120-121:32671, 121-122:28282, 122-128:23600, 128-129:23550, 129-132:28427, 132-135:55024, 135-136:28427, 136-141:55024, 141-142:28427, 142-143:50824, 143-144:29852, 144-146:70491, 146-148:28427, 148-160:50824, 160-162:51734, 162-166:50824, 166-180:51734, 180-183:50824, 183-192:32671, 192-194:65754, 194-208:32671, 208-210:65754, 210-216:32671, 216-217:38163, 217-222:32671, 222-224:38163, 224-235:32671, 235-237:65754, 237-247:32671, 247-249:65754, 249-265:32671, 265-267:65754, 267-273:32671, 273-275:65754, 275-286:32671, 286-313:50824, 313-318:70960, 318-320:50824, 320-325:70960, 325-333:50824, 333-360:70491, 360-362:21752], hunks=[<8:0, 8:1>], hunksRevId=71863
2017-09-22 18:35:07,535 DEBUG [IncrPing3 DPC ] fisheye DefaultBlameChunksCalculator-applyHunks - Applying hunks; oldBlame=[0-2:71683, 2-3:71610, 3-5:71683, 5-12:71610, 12-14:71683, 14-16:71610], hunks=[<5:0, 5:1>, <14:0, 15:1>], hunksRevId=71864

[Tatiana Shambereva]

Grzegorz, how to fix it?

[Grzegorz Lewandowski]

Sorry, I'm not sure what's there to fix. 

Name of the thread IncrPing3 DPC indicates it's incremental indexing, so I think Fisheye indexes only new changes. 

I don't know how much repository slurp takes on your instance. There should be a INFO logs indicating how much time it's take for Fisheye to index a single set of changes (it may be multiple commits at once) 

fisheye BaseRepositoryScanner-ping - done slurp for repository repository (repository) time = 8732 ms

The logs you've linked (DefaultBlameChunksCalculator)  are related to a blame calculations. You can turn them off if you really want to but it will impact some of the Fisheyes features.