Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
Level
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to debug Docker in Pipelines?

I have a Docker image that works fine in my local, but when I use it in Pipelines the behaviour is different.

I've tested locally following instructions here: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html

Still working on my local. 

So, how to debug a Docker image in Pipelines when it works locally but not in Pipelines?

 

While my actual problem is not relevant to the answer (I would like to know how to debug it in general) I understand it's worth to explain my issue. The problem I've is related to Drupal. I'm trying to test a Drupal app. For that, I use Drush, the Drupal Shell. Drush it's unable to get the MySQL server connection details. The process is complex. The installation script generates a file with that information. That generated file is the same in my local and in Pipelines. What I would do is to debug the Pipelines docker image, but to do that I'd have to create patches for drush with print debug statements, make them publicly available, apply them with composer (drupal and drush are downloaded using composer) and check the output. This is feasible but slow and far from practical.

 

3 answers

I have this problem as well.

I run a Python test suite on a Docker container locally using the instructions Ricardo linked to above. Locally, all the tests pass. When I run the same test suite on the same Docker image using Bitbucket Pipelines (BBP), I get failing tests.

I have confirmed:

  • BBP and my local machine are running the same Docker image (verified by comparing image IDs)
  • BBP and my local machine are running the same versions of all relevant libraries (in my case, NumPy and SciPy, verified by printing the versions)

I think it must be a hardware or very low level software issue.

@atlassian can you confirm which type of EC2 instance pipelines run on and the AMI (or base AMI if you use a custom one)? These are the only differences in set-up that I can think of. If I know what type of instance BBP is running on, I can exactly replicate the hardware for the pipeline run.

 

More detail of my specific problem:

For one failing test I narrowed the problem to a floating point precision problem. Both BBP and local runs were reporting the same level of precision (details below), but calculated values differed in the order of 1e-17. This difference was percolating through later calculations to result in significant differences in test results.

To verify the floating point precision reported by the node, I added the following to one of my failing tests (where mydata_array is a NumPy array with differing values on BBP and my local run):

import numpy as np
print np.finfo(mydata_array.dtype)

The following is printed by both BBP and my local machine:

Machine parameters for float64
---------------------------------------------------------------
precision = 15 resolution = 1.0000000000000001e-15
machep = -52 eps = 2.2204460492503131e-16
negep = -53 epsneg = 1.1102230246251565e-16
minexp = -1022 tiny = 2.2250738585072014e-308
maxexp = 1024 max = 1.7976931348623157e+308
nexp = 11 min = -max
---------------------------------------------------------------

Therefore, both hardwares are reporting that they (a) are using the same type for my data and (b) are using the same precision and resolution for my data.

If I manually round my data (I chose 8 decimal places), both BBP and my local machine return the same results (all tests pass).

Without knowing more details of the host machine that the pipeline is running on, I can't take any further steps to replicate the environment, so I can't reproduce the test failures. I don't really want to manually round all my data! Any suggestions would be most welcome.

Hi Ricardo,

I heard back from Atlassian support that they use m4 instances on AWS for running pipelines.

I used an m4.xlarge instance running the CoreOS AMI ami-36497653 (Linux kernel version 4.14). This matches almost exactly the test setup for Bitbucket Pipelines as reported through logging that I added to my test suite. Yet the behaviour on this instance matches the behaviour I am seeing locally (which is different from the behaviour on BBP).

Just to be entirely clear, the differences in floating point results are in the order of 1e-3 to 1e3. I am using Python's `assertAlmostEqual()` test method with an epsilon of 1e-5. Floating point arithmetic is not precise, but it shouldn't be giving this large a divergence!

Sorry this isn't a happier result, but maybe replicating the instance type and host OS will help with your specific problem.

Thanks for the information Victoria. The error you have seems clearly related to the hardware, and although the divergence is big enough there's a reasonable explanation. In my case, it's hard to explain by a hardware issue. Anyway, I may try with that setup just to be sure. Thanks again!

They are all running on x86_64 architecture with SSE and SSE2 instructions (locally, BBP, and AWS m4.xlarge), so I don't think it's as simple as a hardware issue. But I'm really stuck for what the difference is! Maybe BBP run on a custom AMI, which is somehow behaving differently despite running on the same hardware, with the same instruction set, same kernel version, same host OS, and same Docker image as my test on the m4.xlarge instance.

Sorry to post on this old thread but I have a similar problem.

I have a Python/Django app with a test that checks Unicode Case Mapping Collisions. The test passes locally with identical Docker images and setup but fails in my Bitbucket Pipeline. I've run out of things to try in working out what the difference is. I also created a separate Vagrant Virtual box setup and the test works there too - it only fails in BBP.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

538 views 6 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you