We have about 400 end to end tests that take just over an hour to run, and they can fail for reasons other than functional failure, even with retries. I'd like to give our build runners the option to rerun only the failed tests, but writing a 400-step pipeline against 20 possible environments will be problematic and nigh-impossible to maintain manually.
I'm thinking I could write an autogen tool to recreate the pipeline for each build, but it occurs to me that this could be something that already exists in whole or in part. Has anyone else found and worked with something like that?