Stash: using / managing multiple repos simultaneously?

Brandon Barlow April 25, 2013

If I usually need to work with multiple Stash repos across multiple projects, what's the best way to manage this? For example, I need Jenkins to be able to clone 12 different repos to do a full system integration test. I came across Google's git-repo : http://code.google.com/p/git-repo/ , but was wondering what the "Atlassian" solution is.

1 answer

0 votes
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 26, 2013

Hi Brandon,

That's a tricky question, and it depends entirely on how you version your repositories.

Atlassian generally doesn't have this specific problem because we use Maven, which handles dependency management at the binary level. You would build one project and release it (when ready) to Maven, and another project would depend on that exact version. All of the required artifacts would then be downloaded as part of a build. The downside to this approach is that making sweaping changes across multiple projects can require a little more co-ordination.

Another option that is specific to Git would be to use submodules. Personally, I'm not a huge fan of them, but the nice thing is they are supported in Bamboo. When you configure a build you have the option of downloading all the submodules before building, which might be exactly what you want. Submodules aren't all that different from Maven in that you (usually) depend on a specific commit from a specific repository (with a hard-coded URL, which is the part I don't like). Every time you make a change you have to update the parent repository with the new commit. This is actually a good thing, which means your builds will be identical every time they run, but could be considered tedious. As of Git 1.8.2 they added support for submodules tracking branches.

Finally you could just use artifact passing in Bamboo. So one build generates an artifact from one repository, and is passed to the next build which uses another repository, etc. Or alternatively just add the 12 repositories in a build and have all the source in one place. The downsides to this are that there is no explicit versioning and so you will never be able to rebuild to get the exact same result.

I'm afraid I haven't used git-repo so I couldn't comment about whether it would be better than one of these options, although I believe it is similar to submodules.

Just to summarize, my personal recommandation is to consider using a proper dependency management tool. This is likely to be lanuage specific, so Maven in the case of Java, or Bundler/RubyGems for Ruby, Pip/Eggs for Python, etc. The list goes on. If this isn't an option, submodules might be an easy way to achieve a similar outcome.

I hope this helps?

Charles

Brandon Barlow April 28, 2013

We are working towards using Gradle & Nexus for builds, dependency management, artifact publishing, etc. However, for mostly non-technical reasons, we won't be able to migrate our large legacy project to this stack and/or use artifact delivery instead of source delivery. Our plan was once we had our Stash repos set up, we could work on refactoring each repo to support artifact-based development, while still supporting requirements for managing lots of parallel repos clones in a giant build area.

cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 29, 2013

Hi Brandon,

Sounds like you're heading in the right direction regarding your migration plan. For now you'll either have to:

a. Leave the files in a single repository. This is the easiest approach, and is only a problem if the size of the actual git objects is too large. Git can handle fairly large histories, it's large binary files that are the bigger problem. You can then use something like git filter-branch to extract subrepositories when you move to Gradle. This is what I've done in the past, but it really depends on the size of the repository.

b. Use something like submodules and/or git-repo from above. Using the new branches feature in submodules may make your life easier to avoid having to bump the parent module on every change, but I have never used it in anger.

c. Just use bamboo to clone all the repositorires from a specific branch. That may make it interesting for developers to build on their local machine.

I hope I haven't missed anything.

Charles

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events