Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Can a docker image with maven and Hadoop work in pipelines?

Struggling to get a docker image to work in bitbucket pipelines where we want Hadoop HDFS on the image (required for the unit tests)

The simplest start for us was to test we could build a custom image with maven 3.6.3 and JDK 8 (match our dev maven / prod JDK) and see if we could get pipelines to run the build with that - it worked fine.

Then we added HDFS to the image, single node configuration (just for unit testing 1 node is all it needs).   The image performs the basic install of HDFS, then the namenode format and start-dfs commands are running from the pipeline step.

It all seems to go well, then after the dfs is started we can see (using JPS) the namenode isn't running like we would expect, then the build starts and the first test fails when it can't see the hdfs server at localhost:9000.

There are no memory warnings, network warnings or anything else we might expect.  The docker image runs fine on local machines.

We have experimented with moving the start command from the pipelines.yml to the image, but get the same result.

Open to ideas of deploying HDFS in different ways - we started with these at it seemed (at the time) the simplest way of achieving what we want.

Any assistance greatly appreciated, especially pointers to relevant documentation!

0 answers

Suggest an answer

Log in or Sign up to answer
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,784 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you