Hello, I am building some pipelines to build and deploy Big Data jobs to spark. I was happy to see that there is an option of Run Customized for a build plan, but I was really surprised when that option is not available where I actually need it: In the deployment project.
My idea is to version the data, pass input and ouptut directories for the job as parameters, so I can run the same artifacts with different input and output to process different versions of the data.
When I have a chain of jobs (A -> B) I don't want to re-build B when A outputs some version of the data, just to tell B that it has to run with that version. I would like to be able to specify to B that the latest deployment from A was version a1, so it has to take a1 as input parameter to deploy version b1.
I understand that this automatic linking may be a special use case for jobs, so I thought to make it manual: I will specify the version in a bamboo variable, that I can modify at deployment time. But I didn't find the option in bamboo to do that!, is there already a feature request to implement this? Am I missing something?
P.S. We are defining the pipelines as code with specs, so changing the code where we define the bamboo variables just to deploy with a different input is not what we really want. We need exactly the same pipeline, but with the ability to change the variable at deploy time, so we can say where to read and where to write.
Thanks for your help!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.