Reduce of rebuilding Automerge commits

[Alexey_Efimov]

In our company we setup CI integration server to montoring of 'refs/pull-requests/*/merge-clean' references. This cause incremental updates on everyone pull request is merged.

So, does Stash have ability to build Pull Requests merge queue for Automerge commits. For example, you have 10 pull requests, first build Automerge commit #1 on master, second build Automerge commit #2 on top of Automerge commit #1, and so on.

If all pull requests is green on CI server, then all of them can be merged by simple fast forwarding without rebuilding automerge commits.

If someone pull requests is failed, then you remove it from queue and append into tail. Then rebuild entire queue once again.

This will be reduced number of rebuilding Automerge commits. And as results builds on CI servers.

How do you think, is it possible to write such requests to JIRA of Stash? :)

[cofarrell · Accepted solution]

Hi Alexey,

Let me see if I understand what you're asking for - you basically want Stash to use our 'merge-clean' commit when you actually merge, instead of creating an entirely new changeset? For that specifically the best place to ask is as a feature request. I don't beileve there's any technical reason why we don't do that - it's just the way it is.

Cheers

Charles

PS. Just as a heads-up, we plan on changing/reducing the number of exposed pull-request refs. In particular 'merge-clean' is going to replace 'merge' in 2.9 (which is not for at least a few weeks yet).

[cofarrell]

As Bryan just pointed out to me, there is a good reason why we don't do this - Stash doesn't know which author to use. Anyone (with permission) could potentially merge the pull request and the Git author really needs to reflect that fact.

Sorry Alexey.

[Alexey_Efimov]

Hello Charles,

Basically my question about what: https://jira.atlassian.com/browse/STASH-3903

How Stash rebuld Automerge (merge, merge-clean refs) if 'to' reference is changed. This can be improved if you will use merge queue, and build Automerge commits on top of previous PR Automerge commit and so on.

[cofarrell]

Hi Alexey,

I'm not 100% sure what you mean, but I get the get the general idea. I suspect Stash's automerge is never going to do exactly what you want. Honestly, they're really an internal detail for us to make pull request diffs accurate and fast, nothing more. It's unlikely (I can't speak as authority though) that we will change them in strange ways to support the workflow you're describing. As mentioned - we want the final merge to be done again when someone finally clicks the button - that will never work with your approach.

My suggestion is manually make your own merges where you need them, and bulild those instead of merge-clean. It sounds like you might have a plugin that does something like that - and that will always be a safer approach, rather than relying on Stash's implementation details.

Sorry.

Charels

[Alexey_Efimov]

4. Next, follow by 'waterfall' of updates to 'merge-clean' reference, you get all these updates on CI server, and you have dozens of rebuilds after there updates to make pull requests are green (to have ability to be merged).

5. How to reduce builds? We need to reduce targed branch updates. Or we need to build 'merge-clean' smarter. Not for 'target' branch directly, but throught merging queue. If you have pull requests 1, 2, 3, then you can build 'merge-clean' refernce for 'to_1' + 'from_1' => 'merge-clean_1'. But for second pull request, you need to get not 'to_2' refernce. You need to get instead of 'to_2' refernce 'merge-clean_1'. Then you build 'merge-clean_1' + 'from_2' => 'merge-clean_2', then 'merge-clean_2' + 'from_3' => 'merge-clean_3'. You get then priority queue of merging pull request. If all pull request completelly green, all of them can be merged by simple fast-forwarding. And you no need to rebuild 'merge-clean' reference, then pull request #1 is merged. All these references remain valid already.

So, i hope now, what i'm describe our problem successfully :)

[Alexey_Efimov]

Charles, thanks for describing. Let me to point on problem, and maybe you suggest a solution by another way. We a discussed it already for several times and this aproach for today is look like good for our side. But maybe we a wrong.

1. Stash rebuild merge-clean reference in background. But you are wrong about what Stash build it then someone pressed Merge button. It rebuild it then someone accessed pull request page for example, even it is just look through, without merge button pressing.

2. We have also Automerge Plugin, this plugin look on everyone Pull Request in background service and try to merge them automatically if all hooks accepted what merge. All builds are green, all reviewers approved and so on.

3. This automerging have a side effect. Each of not merged yet pull requests requested by this plugin every 10 minutes. It means, that reference 'merge-clean' will be always rebuilded automaticaly, everytime then target branch (for instance -- master) is updated. If you have 60 pull requests to master, after each next merge Stash will rebuild 59, then 58, then 57 etc pull request 'merge-clean' references. This is means -- waterfall of 'merge-clean' references updates.

[Bryan Turner]

What you describe for building "merge-clean" "smarter" is not workable, and wouldn't save you any builds even if it was. It requires substantial assumptions about the contents of pull requests (For the purpose of displaying a diff, "merge_clean_1" and "to_2" are in no way interchangeable; one includes all the changes for "from_1" and one does not) and the order in which they'll be merged (1 merged before 2, before 3, etc). Stash cannot make assumptions about either of those two things; it is a general purpose tool. Even if the assumptions were viable, every "merge-clean" still ends up referencing a different hash, whether the merge is built on "to_2" or ("to_1" + "from_1"). Either way, the hash for "merge_clean_1" does not match the hash for "merge_clean_2" and your CI will still build them individually because they are not the same commit.

Whether the system performs the merge directly from to two hashes or on top of an existing merge doesn't help your situation. Either way produces a new merge with a new hash, which will trigger any CI you have monitoring it. There is no avoiding that. I have warned you repeatedly about these build storms. You're making the situaiton worse through your polling with your auto-merging plugin, which will become a third trigger for forcing Stash to rebuild merges, and you have been warned multiple times against that as well. We will not be making changes to Stash's internals to support your setup.

Best regards,
Bryan Turner
Atlassian Stash

[Bryan Turner]

Alexey,

Stash does not rebuild the merge commit when "someone accessed pull request page, even if is just look through". It rebuilds the merge when it detects the parents are out of date, and only then. So long as the "from" and "to" hashes stored in the repository match what is on the pull request, Stash will not re-perform the merge unless a user clicks the "Merge" button.

Stash performs its merges lazily. That means the pull request may sit quietly with its "from" or "to" ref out-of-date, without its merge being updated, until a trigger causes the merge to be needed. There are currently only two such triggers shipped in Stash:

1. Someone views the pull request. This includes the overview or diff. On the overview page, the auto-merge is used to know whether or not to enable the "Merge" button, and on the diff page it is used to provide the tree and diff.
2. A pull request is rescoped (the "from" or "to" ref is updated) and the pull request has active (not outdated) comments. In order to move the comments to their correct lines when the diff changes, the merge must be performed.

[Alexey_Efimov]

Bryan, you are talking about the same things what i have described in previous comment. But you also tell what i'm wrong, and then you put the same thoughts :) Okay. Stash will sit and wait until someone open Pull Request page and then rebuild it. Or until Automerge plugin try to merge it. No matters.

I totally not understand your second comment. I know what you know Git :) So, mergin from_1 + to_1 makes a branch where presented commits from both branches, and then if you will use merge_1 as to_2, it will be work just the same way as Stash works now.

The only one difference. Now Stash merges all pull requests parallely, my suggestion is option to make Stash merging process through queue — serially. In small repos with a few pull requests parallel did not force build storm, but in large repos with many pull requests active queue will have a profit. Because you only need to rebuild tail of queue, not entire queue if some build is fail in middle of queue.

[Alexey_Efimov]

Bryan, you are talking about the same things what i have described in previous comment. But you also tell what i'm wrong, and then you put the same thoughts :) Okay. Stash will sit and wait until someone open Pull Request page and then rebuild it. Or until Automerge plugin try to merge it. No matters.

I totally not understand your second comment. I know what you know Git :) So, mergin from_1 + to_1 makes a branch where presented commits from both branches, and then if you will use merge_1 as to_2, it will be work just the same way as Stash works now.

The only one difference. Now Stash merges all pull requests parallely, my suggestion is option to make Stash merging process through queue — serially. In small repos with a few pull requests parallel did not force build storm, but in large repos with many pull requests active queue will have a profit. Because you only need to rebuild tail of queue, not entire queue if some build is fail in middle of queue.