Service Desk queue creating thousands issues

[Jessica]

Jira 6.1 using service desk. Issues are created via a mail handler. Something has created hundreds of thousands of issues. The emails are from an automated monitoring system and at one point on Monday there was a brief flood of emails caused by an outage, but since Monday evening there have only been 38 emails. Until I deleted the mail handler this morning the queue was jumping by 100s every few seconds. Anyone seen anything like this?

[darylchuah · Accepted solution]

Hi Jessica

I suspect it could be due to the automated monitoring system that causing the mail spamming activity, the 38 emails you are seeing is those emails which has not converted into issues by the JIRA mail handler. Therefore, you might want to check with the automated system whether is generating the correct and right amount of email and also if there is any mail spamming activity going on.

[Jessica]

not the monitoring system - currenly that system still sends the emails to my inbox as well as Jira so I can QA my processes. out of the 260,000+ issues in my instance, there are 585 unique ones.

[Jessica]

any help with how to go about analyzing what happened would be appreciated. particular log file to review perhaps?

[Aseem Parikh]

If you have incoming mail logging turned on in your Logging & Profiling screen, you should be able to see some activity in atlassian-jira-incoming-mail.log.

[edit] If you don't have that turned on, you'll have to rely on your SMTP server logs.

[Jessica]

thanks - the incoming mail logging is enabled (debugging is off) but i don't have that log file on the server

[Jessica]

Can someone tell me the path for the mail log file? Should it be program files\atlassian\JIRA\.... ?

[Teck-En]

Hey Jessica, it should be in your <JIRA_HOME>/logs/ and some files name incomingmail.

If the contents is duplicate, there might be mail loop issue. Check out the following KB:

https://confluence.atlassian.com/display/JIRAKB/Logs+Contain+%27Received+Message+Carrying+this+JIRA+Instance+Fingerprint%27+Due+to+Mail+Loop+Detected

[Jessica]

I found the logs - thank you. Not sure if the loop issue is what's occurring. I see a lot of instances :The folder has been closed on us, stop processing any more emails: java.io.EOFException: Read timed out

Then repeated: Exception: Folder is not Open
java.lang.IllegalStateException: Folder is not Open

Exception: Folder is not Open
java.lang.IllegalStateException: Folder is not Open
at com.sun.mail.pop3.POP3Folder.checkOpen(POP3Folder.java:547)
at com.sun.mail.pop3.POP3Folder.getProtocol(POP3Folder.java:577)
at com.sun.mail.pop3.POP3Message.loadHeaders(POP3Message.java:606)
at com.sun.mail.pop3.POP3Message.getHeader(POP3Message.java:364)
at com.atlassian.jira.service.services.mail.MailFetcherService

BUT - they're throughout the log, not just during the time this issue was happening.

I don't see anything else outstanding in the log at the time this started.

[Jessica]

I think - the messages flooded into the mailbox, because of the limits on the pop mail server it the connection closed and since the folder could not be opened, the messages did not get deleted and the cycle started of building issues in jira then erroring out again.

That's a very non-technical thought process I realize. In my environment this type of outage will occur again where I need several hundred emails processed quickly. Is it settings on the mail servers that need to be changed? Within Jira config?

[Teck-En]

JIRA side mail configuration options are quite limited as JIRA just fired the connection to the pop box and pull the mail for issue creation. The fix should be something to be venture in the mail server side.

As for the folder not open, it might be related to other mail handlers(check out the mail handler ID) which it might possible to cause issue not to be created in certain projects: https://confluence.atlassian.com/display/JIRAKB/JIRA+stops+creating+tickets+from+emails