Confluence Stops Indexing New Content after Several Days

[Veracode]

We are running Confluence 5.5.2, and it appears that Confluence stops indexing new content after a few days.

1. Indexing had stopped.
2. We ran a manual "rebuild" (by clicking the "Rebuild" button), and then indexing appeared to be working.
3. Three days later it stopped.
4. We ran another manual "rebuild" and then indexing appeared to be working again.
5. Three days later it stopped again.
6. Now, instead of running a "rebuild" we are clicking on the "Flush Queue" link periodically.

There are a few errors that I see only on the days when the automatic indexing stopped running.

YYYY-MM-DD HH:MM:DD,123 WARN [http-bio-NNNN-exec-NNNNN] [com.benryan.conversion.PPTSlideServlet] doGet Conversion task timed out for attachment : FILENAME.pdf with page number 0, waited : 300, now executing in background.

These entries (on those days) are the only rows displayed when I search the logs for lines containing the two strings "PPT" and "pdf".  That makes me think that it might be related to this question: https://answers.atlassian.com/questions/180082

There are also errors related to Excel files that I see during the manual "rebuild" as well as on the days when I believe the automatic indexing stopped running.  I don't know if these are relevant.

YYYY-MM-DD HH:MM:SS,123 ERROR [http-bio-NNNN-exec-NNNNN] [officeconnector.index.excel.ExcelXMLTextExtractor] endDocument expected [ 3560 ] entries but read [ 3563 ]

Note: I checked these questions, and I do not think they apply.

• https://answers.atlassian.com/questions/137105
  • This is for confluence 4.1.4.  We are running 5.5.2
• https://answers.atlassian.com/questions/217583
  • This is regarding a problem with PDF files.  We do not appear to have that problem, but I could be mistaken about that.

-Vik Solem

[Rodrigo Adami]

Hi Veracode,

There are multiple causes for indexing malfunctioning as you stated. There's one that causes most issues: Performance.

By performance I mean JVM not working as expected (low memory, GC running a lot) and database connections (Number of allocated connections (pool), networking). These are the most common causes of indexing not working as expected.

So I suggest to review the memory size of confluence and make sure you are running with 1 or 2 GB more than the actual usage (set XMX and XMS the same size to avoid running into GC issue) and for the database, I recommend checking the database connection pool with some external thread dumps at peak usage time and update the connection pool accordingly.

I hope that helps

Cheers,

Rodrigo

[Veracode]

Hi Rodrigo. 

I'm pretty sure we don't have a memory problem, and we have no reports of any performance problems.  I find it odd that the auto-indexing dies, yet I am able to flush the index queue successfully, and when I manually re-build the indexes (with no confluence restart) the auto-indexing then works for several days. In your experience have you seen that those match either a memory or a database problem? 

I will confirm the memory and database settings and post those here.

-Vik

[Veracode]

Am I reading the JVM stats correctly? It appears to say that we have 1,400 Meg free or more than a gig free. This appears to satisfy your statement about free memory. Since we do not appear to have any performance issues, I don't know when we should be checking the database connections. Are there any other suggestions? I think it might be time to open a ticket...

[Veracode]

Due to a problem with an XL spreadsheet this was in the log today. 

2014-11-21 15:33

[atlassian.confluence.status.SystemErrorInformationLogger] writeToLog

--------------------------

JVM Stats

--------------------------

allocatedHeap = 2772107264

usedMemory = 1205552336

availablePermGen = 790840112

totalMemory = 2772107264

freeMemory = 1566554928

maxPermGen = 1073741824

usedPermGen = 282901712

freeMemoryInMegabytes = 1493

freeAllocatedHeap = 1566554928

availableHeap = 1657781040

usedMemoryInMegabytes = 1149

maxHeap = 2863333376

usedHeap = 1205552336

totalMemoryInMegabytes = 2643

--------------------------

Reformatted:

--------------------------

JVM Stats

--------------------------

allocatedHeap =          2,772,107,264

usedMemory =             1,205,552,336

availablePermGen =         790,840,112

totalMemory =            2,772,107,264

freeMemory =             1,566,554,928

maxPermGen =             1,073,741,824

usedPermGen =              282,901,712

freeMemoryInMegabytes =          1,493

freeAllocatedHeap =      1,566,554,928

availableHeap =          1,657,781,040

usedMemoryInMegabytes =          1,149

maxHeap =                2,863,333,376

usedHeap =               1,205,552,336

totalMemoryInMegabytes =         2,643

--------------------------