Mirror can't sync SOME repositories

[yago]

We've setup a Data Center 8.9.9 with several mirrors.

One of the newly added mirrors has the situation that it can sync some repositories from the upstream while others get stuck and show the following errors:

2024-05-18 20:56:56,114 DEBUG [initial-sync-thread-pool:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=12798}
com.atlassian.bitbucket.scm.NotAuthorizedScmException: The requested remote repository does not exist, or you do not have permission to access it
at com.atlassian.stash.internal.scm.git.command.fetch.MirrorSynchronizeCommand.doFetch(MirrorSynchronizeCommand.java:133)
at com.atlassian.stash.internal.scm.git.command.fetch.MirrorSynchronizeCommand.call(MirrorSynchronizeCommand.java:94)
at com.atlassian.stash.internal.scm.git.command.fetch.MirrorSynchronizeCommand.call(MirrorSynchronizeCommand.java:33)
at com.atlassian.bitbucket.internal.mirroring.mirror.command.MirroringCommandFactory$CredentialRefreshingCommandDecorator.call(MirroringCommandFactory.java:150)
at com.atlassian.bitbucket.internal.mirroring.mirror.command.MirroringCommandFactory$CredentialRefreshingCommandDecorator.call(MirroringCommandFactory.java:126)
at com.atlassian.bitbucket.internal.mirroring.mirror.farm.synchronization.InitialSyncHelper.maybeDoFullSync(InitialSyncHelper.java:200)
at com.atlassian.bitbucket.internal.mirroring.mirror.farm.synchronization.InitialSyncHelper.syncRepository(InitialSyncHelper.java:274)
at com.atlassian.bitbucket.internal.mirroring.mirror.farm.synchronization.InitialSyncHelper.lambda$submitRepositorySync$3(InitialSyncHelper.java:261)
at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.lang.Thread.run(Thread.java:748)
... 1 frame trimmed
Caused by: com.atlassian.bitbucket.scm.git.command.fetch.GitFetchAuthorisationException: The requested remote repository does not exist, or you do not have permission to access it
at com.atlassian.stash.internal.scm.git.command.fetch.MirrorFetchCommandHandler$RemoteFetchExitHandler.evaluateStdErr(MirrorFetchCommandHandler.java:329)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:204)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)
at com.atlassian.bitbucket.scm.BaseCommand$CommandFuture.internalGet(BaseCommand.java:348)
at com.atlassian.bitbucket.scm.BaseCommand$CommandFuture.get(BaseCommand.java:312)
at com.atlassian.bitbucket.scm.BaseCommand.call(BaseCommand.java:144)
at com.atlassian.stash.internal.scm.git.command.SimpleGitCommand.configureAndCall(SimpleGitCommand.java:84)
at com.atlassian.stash.internal.scm.git.command.fetch.MirrorSynchronizeCommand.doFetch(MirrorSynchronizeCommand.java:129)
... 13 common frames omitted
2024-05-18 20:56:56,114 DEBUG [initial-sync-thread-pool:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Request: SyncRequest{externalRepositoryId=12798} giving up: 3/3 attempts failed
2024-05-18 20:56:56,119 WARN [AtlassianEvent::thread-2] c.a.b.i.m.m.DefaultUpstreamService BITBUCKET/PROJECT/repo[855]: fetch from upstream failed 4 times in a row. Further fetches will not be attempted until the repository is next updated.

• This seems to happen randomly with some repositories
• The failing repositories exist.
• The message 'you do not have permission to access it' makes little sense to us, as the mirror is authorized  on the upstream and it successfully syncs other repositories. In some cases, several repositories in a project are synced and some in the same project fail.

The error message "fetch from upstream failed 4 times in a row. Further fetches will not be attempted until the repository is next updated" is not clear to us: when would it be reattempted,  when a change is made through the central instance? It is now impossible to push a change to the mirror for these repositories.

We've tried deleting the mirror and re-adding , but each time, some repositories fail randomly and are never synced because of the "will not be attempted until the repository is next updated" behavior.

Please advice.

Note: The mirror was working correctly for years. We tried to update to a newer version and this problem started happening. We then reverted to the old version and re-added, but same problem. We followed the documentation and removed shared/data and shared/config.  Again, the problem happens for some repositories but not all.