Simple Rails upgrade breaks Bitbucket pipeline

[Jack Phantom]

I've done the most minimum viable change on a working project, simply changing the version of Rails to 6 from 5.

A project that has been working fine for years now fails bitbucket builds with 

** Execute db:create

could not connect to server: No such file or directory

Is the server running locally and accepting

connections on Unix domain socket 

How does one debug this? Works on developer machines, was working fine on pipelines until the gem change. Can confirm that postgresql is up and running. What documentation exists to debug the internals? Is it possible to log in to bitbucket's pipeline at some point? There's a doc on how to "test your pipeline locally" but it's not failing locally.

[Theodora Boudale]

Hi Jack and welcome to the community.

The guide we have on debugging pipelines locally is for debugging pipelines locally with Docker, by using the same Docker image you use in Pipelines. Pipelines builds run in Docker containers, so running a build on your local machine does not simulate the environment where Pipelines builds run.

The following guide includes the steps you can take to debug the build locally with Docker:

• https://support.atlassian.com/bitbucket-cloud/docs/debug-pipelines-locally-with-docker/

It is important to use the same Docker image and image tag you use in your bitbucket-pipelines.yml file, and also to do a git reset --hard in the clone to the commit of the build you are debugging.

If your build uses service containers, there is a section on the doc titled "Testing with build services" which includes more details on how to debug locally a build with services. It is important to use the same Docker image that you use in bitbucket-pipelines.yml for the service.

If the only change you made was the version of Rails, it is possible that it is not compatible with any other tools you may be using.

I would suggest as a first step debugging this locally with Docker and see if you get the same error or not.

Kind regards,
Theodora

[Jack Phantom]

That's all well and good, but the issue isn't with my code, it's with the interaction with a service.

I don't know how bitbucket sets this up, so I don't know if the replication is alright. In either way, I need to debug why it can't connect to the database on bitbucket.

Is there no way to get a console into the process as it is running on your machine and debug what's running where?

[Theodora Boudale]

Hi Jack,

It is not possible to interact with a Pipelines build while it is running. If you'd like to debug this in Pipelines, you can add any debugging commands in the bitbucket-pipelines.yml file and inspect the output after the build is finished. Any services defined for your build are additional Docker containers that share a network adapter with your build container:

• https://support.atlassian.com/bitbucket-cloud/docs/databases-and-service-containers/

This is similar to the instructions we have for debugging locally a build that uses services:

• https://support.atlassian.com/bitbucket-cloud/docs/debug-pipelines-locally-with-docker/#Testing-with-build-services

Kind regards,
Theodora

[Jack Phantom]

Unfortunately none of this is any help. "Just reproduce our services on your machine" doesn't contain any instructions as to how to do so.

There's any number of ways to get something started and working, so a simple "to get postgres working on your machine to test this, do

docker run -it flange bloop foo bar baz" 

would be nice. 

"Is the server running and accepting TCP/IP connections on port 5432?" is something that gets CONSTANTLY asked on Stack Overflow and here because I haven't found any docs apart from "just define a service like so" and people "just define a service like so" and it doesn't work.

[Theodora Boudale]

Hi Jack,

Without knowing the content of your bitbucket-pipelines.yml file, I cannot give you the exact arguments you need to use in the docker commands. The documentation provides the commands you need to run, you just need to replace any image names in the commands with the Docker images you are using and the tag name with the tag you create.

In Step 2 of the documentation, when creating the Dockerfile, you need to replace python:2.7 with the Docker image name and tag you are using as a build container in your bitbucket-pipelines.yml file for the step that fails.

When building the image with the docker command provided in the doc, account/imageName:tag is a tag you provide.

The section "Testing with build services" provides an example of a Docker command with a MySQL image for a build that uses a MySQL service. In this command, you need to replace mysql:<tag> with the name and tag of the Docker image you use for the postgres service and also replace the environment variables with the ones you use in your bitbucket-pipelines.yml file (if you don't use any, you can omit them).

The Docker command at the end of this section that starts the build container with --network=host (in order to link it to the service container) uses the account/imageName:tag you defined in step 2.

When the build container is up and running, you can run individual commands from your bitbucket-pipelines.yml file to test them and configure tools inside the container.

You mentioned that the build started failing after a change you made in the version of Rails. Another thing you can do is rerun the last successful build in pipelines (prior to the change you made) and see if it's still successful or if it's failing. If the rerun is also successful, then the failures you see are most likely related to the change you made.

Kind regards,
Theodora

[Jack Phantom]

I have done exactly this and on my machine, I can't even get the "successful" commit to work. 

So in trying to figure out which change has broken Atlassian, I now have the incredible confusion of not seeing any version of the code work on my machine following these instructions. All the time with the "Is the server running on your machine and accepting connections?"

[Jack Phantom]

Let me flip this script a moment:

if I had this stanza in the bitbucket-pipelines.yml

definitions:

  services:

    postgres:

         image: postgres

         environment: 

               POSTGRES_DB: foo

               POSTGRES_USER: bar 

               POSTGRES_PASSWORD: baz 

and I wanted to set up to debug on my machine what Bitbucket pipelines runs based on that:

docker run --network=host --name mypostgres -p 5432:5432 -e POSTGRES_DB='foo' -e POSTGRES_USER='bar' POSTGRES_PASSWORD='baz' -d postgres?

Something like this?

Trying to guess the specific way you run postgres is the thing concerning me right now. "Oh no, it doesn't map ports" so I could get it to run on my machine fine but it not work on your pipline and I'm NONE the wiser as to why, is my point.

"How do you guys run databases on your pipeline, so we can try and debug" has basically been answered by "well, where we have "python" just put the name of what you're using". 

I hope this isn't landing as rude, but "please guess what we do" isn't really optimal :(

[Theodora Boudale]

Hi Jack,

I'm sorry for the confusion. I'll provide an example based on a bitbucket-pipelines.yml file like the following. I am trying to make it as concise as possible (which means I am omitting some info mentioned in the docs).

image: node:10.15.0

pipelines: 
  default: 
    - step: 
        script: 
          - npm install
          - npm test
        services: 
          - postgres

definitions: 
  services: 
    postgres: 
      image: postgres  
      variables: 
        POSTGRES_DB: 'pipelines' 
        POSTGRES_USER: 'test_user'
        POSTGRES_PASSWORD: 'test_user_password'

When you run a Pipelines build, for every step in that file (in this case it's only one step), a Docker container will start based on the Docker image you have defined (in this case node:10.15.0), this is the build container. The repository will get cloned in that container.

Since the step has a service, an additional Docker container will start for the service based on the Docker image you have defined for the service (in this case postgres). This will share a network adapter with your build container, and for Postgres, your tests just connect to port 5432 on localhost (no extra configuration needed).

After the build container starts, the commands of the step's script will get executed.

In order to debug this locally with Docker, as per our doc:

1) Create an empty directory on your machine, named for example DockerTests

2) Inside the DockerTests directory, we want to clone the repo.

If the Pipelines build that failed runs on branches, tags, if it's a custom pipeline, or if it's the default pipeline: we need the clone to have the commit of the failed build (in case you have builds running on a pull-requests definition in your bitbucket-pipelines.yml file, please let me know as steps 2 and 3 will be slightly different).

If this commit belongs to a branch other than the main one, then clone this branch with

git clone -b <branch_name> <bitbucket_repo_url> localDebugRepo

The repo will get cloned in a directory named localDebugRepo

3) Check the Pipelines build log of the failed build, specifically the section "Build setup". If you expand it, you will see a command like

git reset --hard 58ab3379d12cd394d0cca78d165d3b42625b0750

This is the commit where your build is running. Navigate to the directory of your clone, localDebugRepo, and execute the command above (replace the hash with the commit hash found in your build).

4) Navigate one directory up from the clone localDebugRepo, inside the DockerTests directory.

Create a Dockerfile named e.g. my.dockerfile

touch my.dockerfile

Open the file, create and save the following content

FROM node:10.15.0
WORKDIR /localDebugRepo
COPY ./localDebugRepo /localDebugRepo

I am using node:10.15.0 because this is the image I have defined as a build container in the example bitbucket-pipelines.yml file I shared above. Replace that with the image you use as a build container. localDebugRepo is the name I have given to the directory of the clone in step 2.

5) After you save the file, and while you are in the DockerTests directory, run the following command to build the Docker image that you will use for your build container locally:

docker build --memory=1g --memory-swap=1g -t jack/test:1 -f my.dockerfile .

my.dockerfile is the file created in Step 4.
jack/test:1 is a random name and tag I used, you can use a different one.

You have now built the image that you will use as a build container locally, and that image includes also the directory of the repo at the commit you specified in Step 3.

6) The next step is starting the Docker container for the postgres service:

docker run --network=host --name my-postgres -e POSTGRES_DB='pipelines' -e POSTGRES_USER='test_user' -e POSTGRES_PASSWORD='test_user_password' -d postgres

The name my-postgres can be any other name you want to give.
The argument postgres after -d, at the end of the command, is the name of the Docker image used for the service.

7) The final step is to start the build container (based on the image you built in step 5) and link it to the service container (postgres) of step 6.

We do this with the following command:

docker run -it --network=host --memory=4g --memory-swap=4g --memory-swappiness=0 --cpus=4 --entrypoint=/bin/bash jack/test:1

jack/test:1 is the tag you define in Step 5.
Since we're using --network=host, the build container will be linked to the postgres service container from step 6.

After you run this command, you are inside the build container and the code of your repo is there.
You can then start executing commands from your bitbucket-pipelines.yml file, or install any other tools you want.

I hope this helps, if you have any questions, please feel free to let me know.

Kind regards,
Theodora

[Jack Phantom]

Just one final one: you've been very helpful so far, thanks

I can connect to the PG by doing the docker-run it stuff directly into that container, but from my first container, it doesn't work.

One of the bits inside our code looks like

export DATABASE_URL=postgresql://user:password@localhost/pipelines

It would appear that running the container as my-postgres would have to have us already out of sync with what's running on your side, since that would imply local host, e.g. a service running locally.

The posts I've seen have said "you have to have this string" but never document or explain why or even what it's about.

[Theodora Boudale]

Hi Jack,

Another suggestion I made earlier was to rerun the last successful build in pipelines (prior to the change you made) and see if it's still successful or if it's failing.

Could you please give it a try and let me know if the rerun is successful or not?

Kind regards,
Theodora

[Jack Phantom]

Hi, yes: to reiterate, I did try this.

Nov 07, 2022

I have done exactly this and on my machine, I can't even get the "successful" commit to work. 

I am aware that the breakage is because of a change made. The change in question was to simply say "use Rails 6" rather than 5.

[Theodora Boudale]

Hi Jack,

I saw your comment and you mention trying this on your machine. My question concerned rerunning the last successful build in Pipelines (not locally), to confirm if it's still working or not (if not, perhaps something on our side may have caused the issue).

If you open a Pipelines build on Bitbucket website, there is an option Rerun next to the build number. Could you try that for the last successful build and let me know if the rerun is also successful or not?

If the repo belongs to a workspace on a paid billing plan, you can also create a ticket with the support team via https://support.atlassian.com/contact/#/ to request guidance about debugging.

Kind regards,
Theodora

[Jack Phantom]

It works for all branches except a new branch which uses Rails 6.

I have confirmed it attempts to connect EXACTLY the same way.

I'm stumped. I've tried to figure out what the heck is going on. 

[Theodora Boudale]

Hi Jack,

Apologies for the late reply, I was out of office. As I mentioned in my previous reply, if the repo belongs to a workspace on a paid billing plan, you can create a ticket with the support team via https://support.atlassian.com/contact/#/ to request guidance about debugging. In "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. If the repo belongs to a workspace on the free plan, I would suggest upgrading to a paid billing plan so you can create a ticket.

If we have an open support ticket, the engineer working on your case will be able to access the build logs. The ticket will be visible only to you and Atlassian staff, so you can share more details (like the commands you use to debug locally with Docker and their output) and they won't be publicly visible.

The support I can provide here is limited without access to your yml file, and the build logs and without more details on your attempt to debug this locally with Docker.

Kind regards,
Theodora