How to debug Docker in Pipelines?

[Ricardo Sanz]

I have a Docker image that works fine in my local, but when I use it in Pipelines the behaviour is different.

I've tested locally following instructions here: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html

Still working on my local. 

So, how to debug a Docker image in Pipelines when it works locally but not in Pipelines?

While my actual problem is not relevant to the answer (I would like to know how to debug it in general) I understand it's worth to explain my issue. The problem I've is related to Drupal. I'm trying to test a Drupal app. For that, I use Drush, the Drupal Shell. Drush it's unable to get the MySQL server connection details. The process is complex. The installation script generates a file with that information. That generated file is the same in my local and in Pipelines. What I would do is to debug the Pipelines docker image, but to do that I'd have to create patches for drush with print debug statements, make them publicly available, apply them with composer (drupal and drush are downloaded using composer) and check the output. This is feasible but slow and far from practical.

[kierenpitts]

Sorry to post on this old thread but I have a similar problem.

I have a Python/Django app with a test that checks Unicode Case Mapping Collisions. The test passes locally with identical Docker images and setup but fails in my Bitbucket Pipeline. I've run out of things to try in working out what the difference is. I also created a separate Vagrant Virtual box setup and the test works there too - it only fails in BBP.

[Victoria Catterson]

Hi Ricardo,

I heard back from Atlassian support that they use m4 instances on AWS for running pipelines.

I used an m4.xlarge instance running the CoreOS AMI ami-36497653 (Linux kernel version 4.14). This matches almost exactly the test setup for Bitbucket Pipelines as reported through logging that I added to my test suite. Yet the behaviour on this instance matches the behaviour I am seeing locally (which is different from the behaviour on BBP).

Just to be entirely clear, the differences in floating point results are in the order of 1e-3 to 1e3. I am using Python's `assertAlmostEqual()` test method with an epsilon of 1e-5. Floating point arithmetic is not precise, but it shouldn't be giving this large a divergence!

Sorry this isn't a happier result, but maybe replicating the instance type and host OS will help with your specific problem.

[Ricardo Sanz]

Thanks for the information Victoria. The error you have seems clearly related to the hardware, and although the divergence is big enough there's a reasonable explanation. In my case, it's hard to explain by a hardware issue. Anyway, I may try with that setup just to be sure. Thanks again!

[Victoria Catterson]

They are all running on x86_64 architecture with SSE and SSE2 instructions (locally, BBP, and AWS m4.xlarge), so I don't think it's as simple as a hardware issue. But I'm really stuck for what the difference is! Maybe BBP run on a custom AMI, which is somehow behaving differently despite running on the same hardware, with the same instruction set, same kernel version, same host OS, and same Docker image as my test on the m4.xlarge instance.

[Victoria Catterson]

I have this problem as well.

I run a Python test suite on a Docker container locally using the instructions Ricardo linked to above. Locally, all the tests pass. When I run the same test suite on the same Docker image using Bitbucket Pipelines (BBP), I get failing tests.

I have confirmed:

• BBP and my local machine are running the same Docker image (verified by comparing image IDs)
• BBP and my local machine are running the same versions of all relevant libraries (in my case, NumPy and SciPy, verified by printing the versions)

I think it must be a hardware or very low level software issue.

@atlassian can you confirm which type of EC2 instance pipelines run on and the AMI (or base AMI if you use a custom one)? These are the only differences in set-up that I can think of. If I know what type of instance BBP is running on, I can exactly replicate the hardware for the pipeline run.

More detail of my specific problem:

For one failing test I narrowed the problem to a floating point precision problem. Both BBP and local runs were reporting the same level of precision (details below), but calculated values differed in the order of 1e-17. This difference was percolating through later calculations to result in significant differences in test results.

To verify the floating point precision reported by the node, I added the following to one of my failing tests (where mydata_array is a NumPy array with differing values on BBP and my local run):

import numpy as np
print np.finfo(mydata_array.dtype)

The following is printed by both BBP and my local machine:

Machine parameters for float64
---------------------------------------------------------------
precision = 15 resolution = 1.0000000000000001e-15
machep = -52 eps = 2.2204460492503131e-16
negep = -53 epsneg = 1.1102230246251565e-16
minexp = -1022 tiny = 2.2250738585072014e-308
maxexp = 1024 max = 1.7976931348623157e+308
nexp = 11 min = -max
---------------------------------------------------------------

Therefore, both hardwares are reporting that they (a) are using the same type for my data and (b) are using the same precision and resolution for my data.

If I manually round my data (I chose 8 decimal places), both BBP and my local machine return the same results (all tests pass).

Without knowing more details of the host machine that the pipeline is running on, I can't take any further steps to replicate the environment, so I can't reproduce the test failures. I don't really want to manually round all my data! Any suggestions would be most welcome.