Our use case
In our system anybody can start a pipeline and approve a deployment. A deployment to production can be manually approved after a series of robust tests and build phases across multiple branches.
We have a continuous deployment system running in Bitbucket Pipelines, and the last step is performed by Terraform. Terraform provisions our infrastructure as it is defined in code across our repositories, and programmatically makes changes to our infrastructure if there are any differences in configuration.
Terraform, along with many other deployment systems, can only operate one instance/session per environment at a time. This is facilitated by a remote locking mechanism which is read/written to by all deployers and pipelines.
If a Bitbucket Pipelines pipeline step is "stopped", it does not gracefully shut-down. Ordinarily Terraform will be gracefully stopped (for example if Ctrl+C is hit in terminal) and the remote lock is released.
With Bitbucket Pipelines, anybody can force immediate termination of the pipeline script which results in our infrastructure half-updated and the state remains locked.
This problem is not limited to Terraform, it applies to any deployment system which relies on a locking mechanism. Below are my suggestions for resolving this problem:
If anybody else is experiencing a similar issue, has a workaround, or otherwise has any feedback, then please contribute to this discussion :)
Beginning on April 4th, we will be implementing push limits. This means that your push cannot be completed if it is over 3.5 GB. If you do attempt to complete a push that is over 3.5 GB, it will fail...