JEMH errors

[Simon Gao]

I saw following errors with JEMH. What caused the failure to write audit log?

We are running JIRA 5.1.7 with JEMH 1.2.82.

Simon

2013-04-08 10:15:24,316 QuartzWorker-0 ERROR [jira.emh.service.DefaultJEMHAuditTrailManager] IOException: Failed to write message to audit log, processing is unlikely to succeed: null

2013-04-08 10:15:24,322 QuartzWorker-0 ERROR [jira.emh.service.EnterpriseMessageHandlerImpl] Unhandled Exception (bug?): null (NPE?): MessageRemovedException: null:

2013-04-08 10:15:24,325 QuartzWorker-0 ERROR [jira.emh.service.EnterpriseMessageHandlerImpl] Unhandled Exception occurred during processing (mail will be deleted (to stop re-processing) but forwarded), type=javax.mail.MessageRemovedException), location=com.sun.mail.imap.IMAPMessage.checkExpunged(IMAPMessage.java:205). This may be a BUG: null

[Andy Brook [Plugin People]]

My view is that means that for some reason, the Mail Server was unable to provide the message to JIRA. From the exception 'MessageRemovedException' it could suggest that there are more than one clients pulling email, perhaps another got there first?

The first thing JEMH does in processing messages is to read the message fully by storing it to disk locally, as this then forms the basis of the audit event. If the mail can't even be streamed, processing would have failed at some point very sooner after.

A quick fix is to change the authentication password on the mailbox, if you are the only one that knows it you should be done, though, from experience, JIRA caches open connections to mailservers, so this is not guaranteed to work with immediate effect, but will rule out such interference eventually.

[Simon Gao]

Thanks Andy. This answered my question. I believe what happened is this.

We have 8 profiles, running through all 8 profiles takes more than 5-6 minutes due to large amount of email accumulated in JIra inbox. But JEMH is set to run every 1 minute. This not only caused contention problem (competing for run slots), but also possibly read the mailbox at the same time. Is there any way to check whether there are existing JEMH instances running before it launch new JEMH tasks? If there are, the new tasks should simply quit.

Simon

[Andy Brook [another me]]

Hey Simon, thats a formiddable configuration, yes if you are sharing mailboxes it could certainly happen. I'm not sure to what degree JIRA isolates mail retrieval, from what I know, I would guess that a connection is shared, which may be the cause of the problems you see.

The solution isn't quite as simple as just quitting though, with a single resource (the mailbox) between two or more contending Profiles, you have a race condition for each Profile to use the mailbox through (I think) a shared connection. If a mailhandler detected an instance was currently running, and quit, then that Profile would be denied access to the resource on that occasion and possibly many more.

It does seem that the orchestration of the mailhandler invocations should be able to manage this better, and specifically, to not share the connection.

Im torn as to how to address this, I could:

a) completely reimplement the inbound mail server configuration within JEMH, implement my own retrieveal service that ran every minute but executed all configured profiles in a sequential manner.

PRO: solves the problem, implements some code Im going need anyway

CON: takes time to implement, rewrites more of JIRA's mail handling features, dupllicates core features, extends the envelope of JEMH to cover all mail interaction not just the mail processing (has support issues).

b) rasie a bug report...

PRO: fast, easy

CON: not going to fix the problem any time soon

c) implement as you describe a suicide feature for mail handlers using a common mailbox so that 'there can be only one'.

PRO: achieves a solution with minimal effort, despite being a bit iffy with providing fair access to some Profiles.

CON: possibly starving a Profile for extended periods (through repeated bad luck) may cause delays in processing of some email.

Hmm. This is going to have to go into a feature request in any case, I've logged https://thepluginpeople.atlassian.net/browse/JEMH-1318

Have pinged a contact in Atlassian about this.

[Andy Brook [Plugin People]]

Hey Simon, thats a formiddable configuration, yes if you are sharing mailboxes it could certainly happen. I'm not sure to what degree JIRA isolates mail retrieval, from what I know, I would guess that a connection is shared, which may be the cause of the problems you see.

The solution isn't quite as simple as just quitting though, with a single resource (the mailbox) between two or more contending Profiles, you have a race condition for each Profile to use the mailbox through (I think) a shared connection. If a mailhandler detected an instance was currently running, and quit, then that Profile would be denied access to the resource on that occasion and possibly many more.

It does seem that the orchestration of the mailhandler invocations should be able to manage this better, and specifically, to not share the connection.

Im torn as to how to address this, I could:

a) completely reimplement the inbound mail server configuration within JEMH, implement my own retrieveal service that ran every minute but executed all configured profiles in a sequential manner.

PRO: solves the problem, implements some code Im going need anyway

CON: takes time to implement, rewrites more of JIRA's mail handling features, dupllicates core features, extends the envelope of JEMH to cover all mail interaction not just the mail processing (has support issues).

b) rasie a bug report...

PRO: fast, easy

CON: not going to fix the problem any time soon

c) implement as you describe a suicide feature for mail handlers using a common mailbox so that 'there can be only one'.

PRO: achieves a solution with minimal effort, despite being a bit iffy with providing fair access to some Profiles.

CON: possibly starving a Profile for extended periods (through repeated bad luck) may cause delays in processing of some email.

Hmm. This is going to have to go into a feature request in any case, I've logged https://thepluginpeople.atlassian.net/browse/JEMH-1318

Have pinged a contact in Atlassian about this.