Agile boards permissions are based on the permissions set to the filter used to create the board. If you don't have access to the filter, you will not be able to see the board. If you don't have transition rights to the issue, you will not be able to move it around the board. In agile itself not much can be set, except who has admin rights on the board.
Had this issue as well. Filter was the culprit..
It looks like you're new here. Sign in or register to get started.