Our use case
In our system anybody can start a pipeline and approve a deployment. A deployment to production can be manually approved after a series of robust tests and build phases across multiple branches.
We have a continuous deployment system running in Bitbucket Pipelines, and the last step is performed by Terraform. Terraform provisions our infrastructure as it is defined in code across our repositories, and programmatically makes changes to our infrastructure if there are any differences in configuration.
Problem statement
Terraform, along with many other deployment systems, can only operate one instance/session per environment at a time. This is facilitated by a remote locking mechanism which is read/written to by all deployers and pipelines.
If a Bitbucket Pipelines pipeline step is "stopped", it does not gracefully shut-down. Ordinarily Terraform will be gracefully stopped (for example if Ctrl+C is hit in terminal) and the remote lock is released.
With Bitbucket Pipelines, anybody can force immediate termination of the pipeline script which results in our infrastructure half-updated and the state remains locked.
Solutions
This problem is not limited to Terraform, it applies to any deployment system which relies on a locking mechanism. Below are my suggestions for resolving this problem:
- Repository Settings -> Pipelines -> Toggle: "Stop" will issue a shutdown command to the process, consuming build minutes until the process terminates.
- Repository Settings -> Pipelines -> Toggle: Only admins can stop a pipeline.
If anybody else is experiencing a similar issue, has a workaround, or otherwise has any feedback, then please contribute to this discussion :)
