Unchanged files appear under Unstaged Files and disappear when staged

[Deleted user]

This quite annoying and has bothered me for some time now (years). In combination with Unity it seems that every now and then, files pop up in the Unstaged files list. When selecting any of those they show as not containing any changes. When staging them, they disappear. I don't know where this comes from, but it unnecessarily clutters the view all the time. It also keeps me from switching branches unless I remove those by staging them.
It might have to do with branch switching. I have yet to find out when exactly that happens, but even then I would be glad to know what to look out for