Missing avatars and attachment issues when migrating from Cloud to Server

[Steven Hamblin]

Hello,

I've just migrated from Jira Software Cloud to Server on a self-hosted instance.  Installation went fine, and the migration (following the Atlassian directions) went okay - issues are there, projects are fine, users migrated, etc.  However, two persistent issues are bothering me, one minor and one less minor:

Minor:  Attachment healthcheck seems to be throwing a spurious error.

In the Support tools (JIRA Administration -> System -> Support Tools), the Attachment Healthcheck reports an error ("The attachment (XXXX.jpg) for issue (XXXXX) could not be read.").  In the server logs, the healthcheck for attachments is reporting that the file it's looking for is not there:

2017-11-11 13:11:38,521 HealthCheck:thread-5 WARN anonymous 785x35x1 218r8q 88.98.211.66,127.0.0.1 /secure/SetupImport.jspa [c.a.j.p.healthcheck.support.AttachmentHealthCheck] There was a RuntimeException when testing the attachment {0}

com.atlassian.jira.issue.attachment.NoAttachmentDataException: Attachment does not exist in filesystem /var/atlassian/application-data/jira/data/attachments/XXXXX/10000/XXXX-17/27560

 at com.atlassian.jira.issue.attachment.store.DefaultFileSystemAttachmentStore.getAttachmentData(DefaultFileSystemAttachmentStore.java:143)

 at com.atlassian.jira.issue.attachment.store.DefaultFileSystemAttachmentStore.getAttachment(DefaultFileSystemAttachmentStore.java:125)

 at com.atlassian.jira.issue.attachment.BackwardCompatibleAttachmentStore.getAttachment(BackwardCompatibleAttachmentStore.java:160)

 at com.atlassian.jira.issue.attachment.BackwardCompatibleAttachmentStore.getAttachment(BackwardCompatibleAttachmentStore.java:206)

 at com.atlassian.jira.issue.managers.DefaultAttachmentManager.streamAttachmentContent(DefaultAttachmentManager.java:727)

[...]

However, the files are definitely there on the filesystem, they have correct permissions (jira:jira), they're readable - and they actually show up on the issue in JIRA itself!  (I can pull up the issue in the JIRA GUI and the attachment referenced is there and I can download / open it).  The database seems to have the correct information as well.

Question:  Why is the healthcheck failing when the issue has the attachment correctly linked?

Less minor: All of the avatars are gone.

I'm reporting this together as I think it might be related, so:  all of the avatars are missing.  In the server logs, I get similar errors as above:  

2017-11-11 14:40:42,911 http-nio-8090-exec-16 ERROR      [o.a.c.c.C.[.[.0.0.1].[/].[viewUniversalAvatarServlet]] Servlet.service() for servlet [viewUniversalAvatarServlet] in context with path [] threw exception

java.io.FileNotFoundException: /var/atlassian/application-data/jira/data/avatars/filenamehere.png (No such file or directory)

 at java.io.FileInputStream.open0(Native Method)

 at java.io.FileInputStream.open(FileInputStream.java:195)

 at java.io.FileInputStream.<init>(FileInputStream.java:138)

--

     at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.6]

     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]

     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]

     at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.6]

     at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]

    , referer=http://35.189.112.121/projects/PROJID/issues/ISSUEID?filter=allopenissues, servletErrorMessage=}

Again, the files are correctly named and exist on the filesystem (under /var/atlassian/application-data/jira/data/avatars).  They are readable and owned by the jira user.

The import did not copy over either the avatar or attachment directories, so I had to do a manual move on the server from the import directory, but I don't see why JIRA refuses to see the files on the filesystem and use them.

Question: Why are the avatars not being read from disk?

Can anyone give me a steer on these issues?

[Andy Heinzer]

Hi Steven,

The Cloud backups of Jira contain the XML backup zip file and in addition to those the contents of both the /data/attachments/ and /data/avatars/ folders.  However Jira Server when it creates backups, does not bundle these two subdirectories in.   You have to migrate them separately.

As a result, it might be necessary in some cases to restore the /data/ directories flat files BEFORE you restore the XML backup.   I say might because this kind of a migration, from Cloud to Server, can cause Jira Server to have to execute different upgrade/downgrade tasks.   It's possible that the filenaming convention and/or file subdirectory name convention can change between versions of Jira.   We tend to see this with Jira Server upgrades that jump a large number of revisions at once.   But I have not yet encountered this same kind of problem with a Cloud migration.  

What can happen is that Jira has references in the database for each attachment, when an upgrade/downgrade occurs, these filename references in the database get renamed, and if the files are on the disk in the correct path at that time, they too get renamed.  But if those files do not exist in the correct path at the time of the upgrade, the references get broken.   Adding the files in after the fact does not help because we don't have a means to re-trigger that upgrade task to run again to rename the files on disk correctly.  But if this is happening, then the filename in the error is what Jira is expecting, but the filenames on the disk might actually be different.

I'd like to better understand your specific situation here.  The healthcheck warnings are there to help you know if something is wrong, but they are not 100% infallible.  If you're attachments are working correctly, but the logs still generate this error, it might be safe to ignore, but I'd want to check on the specific files in order to make sure they are working and located in the specific directory.   A restart of Jira might help reset this healthcheck warning too.

As for the avatars, I would want to scrutinize the specific filepath Jira is looking for these, and compare it against the filesystem to make sure that file exists there as expected.

Adding these in after the fact, while Jira is running might not be enough to stop this warning.  It's possible you might have to restart Jira in order to reset this healthcheck warning as well.

I'd also want to know exactly which document you followed for this migration process.   We have a couple of different guides out there and I'm concerned if we don't have the exact process to follow documented correctly we might be causing other users to have this same problem.  But also migrations between server and cloud can change due to recent changes in the Cloud platform.  So if we have incorrect docs out there, I want to know so we can take the steps to correct them as soon as possible.

It might be worth spinning up a brand new Jira instance, restoring those files in place before you import the XML backup to see if this is one way to avoid this problem.

Regards,
Andy