I'm afraid you won't be able to go just so granular.
There are certain permissions you can set to add or remove attachments but not really one that blocks the opening/downloading of attachments.
The question might be if the user is even allowed to view the issue itself?
Can you maybe elaborate on your use case a bit more?
I have some planning files and there are only certain people that should view them. Maybe I have to create a page with specific people only having view access?
I'd suggest indeed to make another page and restrict access to it.
Thanks for your help. This is what I am going to do.