Huge size of auditmail folders

[Simon Gao]

Hi Andy,

While looking for the reason why disk space has been used so fast, I found that the auditmail folders took a huge amount of disk space. Here is the sumary usage of the folders under /dstore/atlassian/application-data/jira/data/jemh/auditmail/2013/0 (total usage 153GB in size):

4.9M 1
9.4M 10
6.8M 11
10M 12
5.6G 13
6.1G 14
25G 15
41G 16
41G 17
36G 18
4.4M 2
8.1M 3
6.0M 4
4.9M 5
4.3M 6
9.2M 7
6.9M 8
8.7M 9

I assume the folder name corresponding to a date of January. For 16, 17, 28, the folder sizes are so large. Regular "ls" command failed with following error:

# ls 18/* | wc -l
-bash: /bin/ls: Argument list too long

This means there are more than hundreds of thousands of files in one directory. This WILL cause serious performance problem!

The number of files for 15 - 18:

15: 361592

16: 614279

17: 618852

18: 565786

Given the number of files in one directory, JEMH is driving the system on the edge all the time. Any further load would knock down JIRA service. For 1- 12, the directories are empty, but the size of directory itself is huge. This usage scenario is not scalable.

Currently I set JEMH to clean up audit trail history at 1AM PST. But It seemed not working since Jan 13, 2013.

Please help resolve this problem.

Thanks,

Simon

[Andy Brook [Plugin People]]

Hi Simon,

Please log a feature request in JEMH JIRA for this, thanks.

[Simon Gao]

It seems JEMH 1.3 only compatible with JIRA 5.2 and above. We are running 5.1.7. Is it possible you can provide a version of 1.3 that works with JIRA 5.1.7?

Simon

[Andy Brook [Plugin People]]

Update, autoPurge of successfully created mail files and db data has now been implemented, in 1.3 RC-2 that I have just uploaded. Can you determine if this addresses the major part of your space issues?

[Andy Brook [Plugin People]]

Hi Simon, 17million emails, thats a lot of traffic!

Are you retaining failures? Doing this will mean mails that fail to be processed for some reason would be retained through a purge.

Yes, retention period currently has a lower limit of 6hrs, but the clean up period only fires once a day (to reduce load on the server).

Though you haven't indicated what percentage of mails are successful vs not, assuming most of them are successful, a further performance means for high load environments would be to allow disabling archiving of successfully processed emails. This would have some minor feature impact:

- Any forensic analysis of success emails would not be possible

- Test Cases cant be created

Im going to work on this now so it will be in the 1.3 release coming very soon. Any further insight into how the above will work for you would be useful.

[Simon Gao]

As of this morning, this is the total messages being processed 17119245. For Jan 21 (as of 10:16AM), I have 248103 files (total 22G) in the audit directory. I believe this number will go up to 60G or more by the end of today.

We have 6 JEMH profiles, all have catchmail addresses. The auto delete is set to last 6 hours. Does that mean we have a rention of 6 hours?

[Andy Brook [Plugin People]]

Hi Simon,

Thats quite a lot of email history. Do you monitor the daily audit history, what are your normal volumes? If the numbers you see are vastly outside your norm, I wonder if there is an email notification loop? Do you have a catchmail address or jemhAddresseeRegexp address? This is how inbound mail loops are stopped.

The audit history should indeed be cleared (subject to the retention range you specified), could you tell me what that is?

Regarding the 13th Jan time, has there been any JEMH update at that point?

A short term fix to resolve space issues would be to:

1. suspend mail processing by uninstalling the plugin

2. remove all the JIRA_HOME/jemh/auditmail/ files. They are only required in some scenarios just after issue creation (for non-jira user attachment detection/provision). The scheduled mop-up should remove these.

3. dealing with the database, you can also manually drop the related database tables, see https://studio.plugins.atlassian.com/wiki/display/JEMH/Common+Problems#CommonProblems-AuditHistoryistoolarge

4. reinstall the plugin, re-enabling mail processing.

5. Check your audting settings, set an initial retention period to 1day

6. Enable logging and trap content at the expiry time for review

As to the cause of this, I will certainly do some checking. My JIRA will be migrated to OnDemand https://javahollic.atlassian.net tomorrow, happy to track via email javahollic @ gmail.com