nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Is there a way to limit the number of active repositories?

Chad Barnes1

Fisheye has a concept of repository passivation. As with many things Fisheye, it is a mystery to us. There is a MBean called "activeCount" that seems to indicate how many "active" repositories at any given time. We've noted significant impact to our JVM heap when the "activeCount" approaches our actual number of repositories (~700). The active count will happily stay in the 600-700 range even when the heap is full and the GC is struggling to free memory (1 minute pauses, or more).

However, sometimes, when the system is running relatively calm... the "activeCount" shoots down to 0 and 6-8 GiB of heap is freed-up. Then, until the activeCount gradually rises again to several hundred, the system runs much better and the GC is able to keep up without the huge STW collections.

Is activeCount related to repository passivation?

If so, is there a way to limit the number of active repositories such that we can limit the amount of heap the active repositories consume?

Fisheye 3.7.1

indexing_vs_cache_size.png

2016-04-20_17-02-15.png

2016-04-20_8-17-04.png

2016-04-20_9-07-33.png

2016-05-04_13-20-27.png

2016-04-20_8-47-35.png

Accepted answers

All comments

Marek Parfianowicz

Hello Chad,

Yes, the activeCount is related to repository passivation. FishEye starts to passivate repositories when it detects that time spent on GC exceeds 15% of CPU time. It will usually passivate up to 5 repositories at once. If you enable debug logging, you shall find messages "Passivating NN repos due to GC load" in FishEye log.

FishEye has one parameter named maxTotalCacheSize . It's the total of cachesizes for all active (non-passivated) repositories. By default it is 33% of Runtime.maxMemory(), it can be also set using a property "repcache.memcache.total_size" in the config.xml.

&lt;config&gt;
  &lt;properties&gt;
    &lt;property name="repcache.memcache.total_size" value="(size in bytes)"/&gt;
  &lt;/properties&gt;
&lt;/config&gt;

Another parameter is the maximum number of active repositories, which is calculated as follows: maxActive = maxTotalCacheSize / 5MB. It can be also set using a "repcache.passivate.soft_max_active" property in the config.xml.

However, before tuning these parameters I would suggest to have a look at this problem:

... and the GC is struggling to free memory (1 minute pauses, or more)

Having 1-minute pauses suggests that you have wrong memory settings. Could you tell what JVM options you use ( -Xmx, -Xmx, garbage collector type etc)? I'm asking, because it's not uncommon to use a wrong garbage collector or wrong GC settings.

References:

How FishEye uses memory - passivation

Cheers
Marek

Grzegorz Lewandowski

There is a MBean called "activeCount" that seems to indicate how many "active" repositories at any given time.

That is correct. Passivation manager should passivate open repository handles, but it can't simply force close if some other threads are using it. Passivation manager should try to passivate repositories on three conditions:

Maximum active repositories this number is calculated using following formula.
```
0.3 * Runtime.getRuntime().maxMemory() / (5 * 1024 * 1024)
```
Look at Marek comment, it can be adjusted by setting repcache.memcache.total_size
On timely manner. There's timer task which runs every minute to:
1. check which repositories can be passivated
2. look for high GC spikes and tries to passivate some of the repositories, look for following debug logs:
```
Passivating 3 repos due to GC load
```

But as I said before passivation mechanism can only request repository release and can't force it. Those are closed on log entries:

DB close on REPO_NAME

And it can only happen when all threads release the handle, look for:

acquire engine on REPO_NAME, count= HOW_MANY_CALLERS_HOLDS_THE_REPO
release engine on REPO_NAME, count= HOW_MANY_CALLERS_STILL_HOLDS_THE_REPO
(it has to equal 0 in order to release repo)

You can verify how many repositories awaits for closing in needingPassivationCount counter. You can reference it from MBean you've pointed.

the active count will happily stay in the 600-700 range

I wonder why there's so many open repositories, is it due to index polling, or most of those repositories are used by HTTP threads? If it's the former maybe you should probably think about moving to SCM Hooks and disable polling which will cause to index repository only if there's something new to index.

Note: In order to see any of log entires I provided here you have to enable debug logs.

Chad Barnes1

Thanks for the response.

I've included an image of some JMX metrics for a 24-hour period. In it there are some interesting points:

1) Overnight (very low end-user usage) full GCs continue to hit in groups. Mostly just hourly polling happening overnight. Some of our nasty repos may still be processing very large or complex change sets. These full GC groupings essentially deny our users service for minutes.

2) The active repositories metric stays high even though it seems to be struggling with memory. But, at about 11:00, they all get passivated. Memory loosens-up and the full GCs go away for a while.

2016-04-20_7-25-38.png

-XX:-CreateMinidumpOnCrash
-XX:ReservedCodeCacheSize=350m
-XX:+UseG1GC
-XX:+DisableExplicitGC
-XX:-UseBiasedLocking
-XX:InitiatingHeapOccupancyPercent=25
-XX:MonitorBound=32768
-XX:+PerfDisableSharedMem
-XX:+AlwaysPreTouch
-XX:-UseAdaptiveSizePolicy
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xms16384m
-Xmx16384m

These arguments have evolved from our trials and errors starting with no arguments, moving to G1GC (from throughput), and then incorporating some options that have really helped us with other heavy Java web apps.

We've got close to 700 repositories. They range from small, unused repos. Properly formed repos (red bean trunks,tags,branches) to not-so-properly formed. To huge repos with millions of files and very heavy branching and renaming. Our total Fisheye repository cache is approaching 2TB.

All repositories are set to poll each hour as a fail-safe in the case a specific SCM trigger didn't run correctly.

Thanks for your assistance.

Chad Barnes1

Thanks for all that good insight. I'll pass it on to my system administrators.

Regarding the SCM Hooks, we do have those in place. We keep an hourly polling in Fisheye in the event the hooks don't fire correctly or they haven't been setup on a repository yet. We have considered moving them to daily.

I've also responded with some additional information about our current state to Marek's post.

Chad Barnes1

For reference, here's a current snapshot of the MBeans.

#activeCacheSize 3181457210
#activeCount 370
#cacheUsage 0.5555555475034958
#gcPassivations 253
#maxActivePassivations 0
#maxActiveRepos 1092
#maxCacheSize 5726623061
#needingPassivationCacheSize 0
#needingPassivationCount 0
#perRepoCacheSize 8598533
#runnableCount 666

Kevin Radke

Some additional details.

Our largest Fisheye index file is 145GB for a 55GB sized repository. I frequently see Fisheye creating 100GB+ of temp files in var/tmp while processing branch operations. Disk I/O has hit 8Gb/s during these operations. Memory allocation has been over 4GB/s during the same timeframe.

For a single repository I'll see Fisheye requesting nearly 100 requests per second of the Subversion server. This value doesn't limit even if the throttle connections value is set to 1 per second.

The gc.log shows that Fisheye has gone through 26TB of memory in the last 44 hours.

Eventually we would want to scale this to over 3,000 repositories. Our largest repository is around 650GB, but Fisheye is not configured to index it (yet).

2016-04-20_8-17-04.png

Kevin Radke

Forgot to attach the GC count table. Most GCs are less than 7s...

2016-04-20_8-47-35.png

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

fisheye-crucible

fisheye-crucible-administration