nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

What is the difference between a system level and project level reindex?

Chris Volker

My question is what is the best practice, or when should I be using the project level re-index as opposed to the full system reindex. Clearly, time is a factor. Does the project level re-index only affect fields etc, that are used or added to a given project?

Accepted answers

MattS

Chris, you've got me wondering if I could just index projects that are in use and ignore inactive ones to make the Lucene index smaller and faster smile

All comments

Jobin Kuruvilla [Adaptavist]

What do you mean by project level reindex? That is not available by default in JIRA.

crf

Look at the "Actions" drop-down on the main project page. We added it in 6.2.

crf

There are times when a reindex is needed due to a very specific configuration change, such as adding a new custom field. When the configuration change only applies to certain projects (say a "Purchase Order Number" for a PURCHASING project), reindexing everything just to get that one field to show up properly in the one project that's using it is wasteful.

JIRA cannot currently tell when the change only matters for a subset of the projects (this is an improvement that the JIRA Enterprise team is looking into), but presumably you can. Re-indexing just the affected projects is a way to get the new field to show up where it is needed with less immediate impact to the system. I believe you still have to eventually do a full (background is okay!) reindex to make the warning message go away, but at least you can get up and running immediately.

Another example would be that if you know that something is wrong with the index (say you had manually taken an index backup and restored it and know that it is out of date, for example), you could selectively reindex your highest priority projects first and leave lower priority projects until later.

We haven't really thought out formal "best practices" around this – it is just another tool for knowledgable administrators to work with.

crf

As to questions about specific fields, etc. -- There is no such thing as partially reindexing an issue's fields at this time. At the code level, the specific affected indexes can be selected (meaning issues, comments, or change history, with worklogs slated to be indexed as well in 6.4). But the indexing technology (Lucene) doesn't understand the concept of modifying just one field of an indexed document; it only knows how delete the previous document and create a complete replacement for it. Although I could see how we might be able to gain quite a bit of speed by creating the new document with just that one field changed, it really is much more complicated than this, because the field's value is generally not the only thing that changes. For example, to distinguish between a field being present-but-empty and it being irrelevant due to field configurations, there is also a field that tracks all of the visible fields, and that value would change, too.

Chris Volker

This is great information, thanks very much for taking the time to reply and fill in the gaps on this.

MattS

Chris, you've got me wondering if I could just index projects that are in use and ignore inactive ones to make the Lucene index smaller and faster smile

Chris Volker

That is another application as well, initially I thought my users would not have a whole environment denied while one update was made. P.S. Thank you for your work on your Practical Jira Admin book. That has been my handbook, road map, and life saver in travelling through the land of Jira in getting started with this application.

crf

What we've generally found in our performance testing and work on support cases is that Lucene activity normally only becomes a problem when you have especially slow hardware or significantly (say 10-20x) higher write activity than a typical instance would get. Either one is *usually* traceable back to some kind of environmental problem. That said, the JIRA Enterprise team has discussed the possibilities of how we might improve the indexing problem. With the obligatory disclaimer that I'm not promising any of these will ever actually be delivered, much less giving timelines or anything of the sort, some of the things we've looked at include: * Finding ways to avoid the full reindex when we have a more reliable way to identify which issues are affected. If you just modified a field configuration scheme, then only the projects that actually use that scheme need to be reindexed, not all issues in the system. * Making upgrade tasks that "require" a reindex but do not seriously cripple the system in the meantime wait until the administrator says to do that reindex instead of doing so immediately * Sharding the indexes by "active" vs. "archived". This would be a kind of on-server version of archiving with the advantage that it is much simpler to implement, but the disadvantage that all the field configurations, permission schemes, etc. that *really* cause the slowdowns would still be there, so there is far less benefit. * Sharding the indexes by project, which would give the ability to do things like reindex just that one project without issues from other projects being involved in the segment merge activity. Similarly, searches that are limited to a single project would have a much smaller index to work with more naturally this way. * Taking better advantage of Near Real Time search capabilities, which allow searches to see up to date information before it has actually been committed to disk. The advantage is that your writers no longer block the readers. The disadvantage is uncertain but might, for example, be a larger window for index inconsistencies on unclean shutdowns. * Looking at other ways we could do this besides using Lucene, such as indexing directly into the database or into another technology (related or not) like elasticsearch or solr. This has the disadvantage of breaking the API but is almost certainly going to be necessary, regardless. If you (meaning Chris, Matt, or anyone else) have your own ideas to contribute to the list of possible improvements we could make to this picture, then we would definitely love to hear more.

MattS

Chris, is the project level reindex for jira admins or system admins? We found it is not for project admins (which is good)

crf

From system-view-project-operations-sections.xml: <web-item key="reindex_project" name="Reindex Project" section="system.view.project.operations" i18n-name-key="webfragments.view.project.operations.item.reindex.project.name" weight="40"> <label key="admin.projects.reindex.project" /> <link linkId="reindex_project">/secure/admin/IndexProject.jspa?pid=$helper.project.id</link> <condition class="com.atlassian.jira.plugin.webfragment.conditions.HasSelectedProjectCondition" /> <condition class="com.atlassian.jira.plugin.webfragment.conditions.UserIsAdminCondition" /> </web-item> From actions.xml: <action name="project.IndexProject" alias="IndexProject" roles-required="admin"> <view name="success">/secure/project/views/projectindex.jsp</view> So it should require normal JIRA admin access, not system admin, meaning that customer admins in Cloud have enough permission to do this, as well.

MattS

Chris, interesting ideas, thanks for sharing them. There is a test search app written to showcase some newer parts of Lucene at http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html It made me think about having a main Lucene index for non-archived issues and a separate Lucene index for archived issues. Mind you, Lucene could probably be configured differently in JIRA for large indexes and that might be a good start.

Steven F Behnke

For the record, it seems that if I make changes that ONLY affect ONE project, indexing THAT Project gets rid of the Index Needed warning.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

jira

jira-administration