nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Is there an option to setup artifact storage for step artifacts?

Alexey Vishnyakov_xf0mf9

Good day

Is there any possibility to specify some custom server for artifact storing? That could significantly increase build process.

Other nice-to-have option is artifact download specification, could be extremely useful in parallel builds — now following steps download all artifacts from all previous steps, which causes not necessary network and io overhead

Thank you for your product, hope it will grow over time!

Accepted answers

Theodora Boudale

Hi Alexey, welcome to the community!

It's good to hear you're enjoying our product and thank you for the feedback.

I'm afraid that it is not possible to set up a custom server for artifact storage for files defined as artifacts in the yaml file. You could upload any such files to another server during the step that generates them, and then download them from that server in subsequent steps. However, this approach would not involve defining the files as artifacts in the yaml file.

May I ask why you're looking for this feature? What would be the benefit of having a custom server for artifacts? I'm just trying to understand your use case a bit better.

It is possible to prevent a step from downloading artifacts by adding the following in the yaml file for this step:

artifacts:
  download: false

You can see an example on this documentation page. Is this what you are looking for?

Kind regards,
Theodora

All comments

Theodora Boudale

Hi Alexey, welcome to the community!

It's good to hear you're enjoying our product and thank you for the feedback.

May I ask why you're looking for this feature? What would be the benefit of having a custom server for artifacts? I'm just trying to understand your use case a bit better.

It is possible to prevent a step from downloading artifacts by adding the following in the yaml file for this step:

artifacts:
  download: false

You can see an example on this documentation page. Is this what you are looking for?

Kind regards,
Theodora

Alexey Vishnyakov_xf0mf9

Hi Theodora!

Alas, solution you offer is inappropriate for me since it can't be used effectively on parallel steps. I'd like to build several docker containers with individual services, then push them to registry. I thought of creating parallel builds for each of these containers, and then publish each artifact to registry separately. This could potentially speed up build

Such scenario requires specifying artifacts needed to be acquired from storage to avoid downloading all artifacts from all parallel steps at once.

Maybe you have already guessed why I am asking about designated artifact storage in intranet: all this byte-sending activity may be quite time- and traffic-consuming. Actually I am pretty sure no one needs these artifacts outside of our intranet.

Another downside with parallel builds is that they are all started at once and all are trying to get same data from internet. This is quite ineffective since all these threads are clashing with each other and are fighting for bandwidth. It would be useful to start them not at once but sequentially with 10-30 sec delay, that could help with bandwidth and processor utilization.

I thought about manual artifact upload, but I hoped that is already solved problem with some minor setup like docker service in intranet

Theodora Boudale

Hi Alexey,

I'm afraid that your use case is not very clear to me.

When you say 'artifacts', are you referring to files that are generated during your build, and that you have defined as artifacts in your yaml file as per this documentation?
Or are you referring instead to files you download from a third-party server during your build?

It would be useful to have an example of the yaml file you are using (you can mask any private/sensitive info) and an explanation of which artifacts you are referring to in this yaml file, so we can better help you.

Kind regards,
Theodora

Alexey Vishnyakov_xf0mf9

Good evening Theodora

I do not download any file during by builds except for nuget/npm packages

When I refer to 'artifacts' I mean something that was produced by pipeline and that may require transfer to another build step. Standard way to do this is mark build result as artifact in 'artifact' section of yaml file.

But tricky thing is, some artifacts are required on one steps and other are required on another. E.g., I may have several pipelines inside one file. First one is pull-request-triggered build. Second is master-merge-triggered build. Third is custom step which performs not only build, but publish as well.

To avoid duplicating code, I made step definitions, which are used in all three scenarios. First step is setup, second step for build, third publishes containers built on 2nd step to registry, and forth deploys containers published on third step on environment.

In this scenario I need transfer for some setup (.bashrc file, to be precise) from 1st step to all other steps, tar-red containers from 2nd step to 3rd, and some deploy script from 1st step to 4th.

Now all files are marked as artifacts and retain incrementally between steps (everything produced on 1st steps transfers to 2nd, then everything from 2nd transfers to 3rd and so on), thus increasing load on artifact storage and network. And since tar-red containers are very heavy (I have 5 artifacts total size of 500Mib) and they are transferring among steps where I do not need them (but need other artifacts), I'd like to have an option to turn off artifact downloading for some files. Or, perhaps, move artifact storage more closely to build servers.

That was my intention.

Does that makes sense for you now?

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket-pipelines

artefact

step