pipeline memory limit - elasticsearch exceeded memory limit.

vincent-jensen January 12, 2018

 

Hello,

I am trying to run my functionnal tests in our bitbucket pipeline with elasticsearch an mysql as services.

But everytime, the pipeline launches, I get this error : elasticsearch exceeded memory limit.

Here is the content of my bitbucket-pipelines.yml : 

 

image:  php:7.1.1
options:
  size: 2x
pipelines:
  default:
    - step:
        caches:
          - composer
        script:
          - apt-get update && apt-get install -y unzip
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer install
          - bin/bootstrap
          - ./vendor/bin/simple-phpunit
        services:
          - elasticsearch
          - database
definitions:
  services:
    elasticsearch:
      image: elasticsearch:alpine
    database:
      image: mysql
      environment:
        MYSQL_ROOT_PASSWORD: root

 

I would be very gratefull of some help or some pointers.

Thx in advance,

Vincent.

 

 

2 answers

3 votes
Igor Chepurnoy June 27, 2018

Hi,

The following configuration works for me

definitions:
redis:
image: redis:4.0
memory: 512
elasticsearch:
image: elasticsearch:5.6
environment:
ES_JAVA_OPTS: '-Xms512m -Xmx512m'
jlien88 February 26, 2019

Agree with this. The elasticsearch docker config takes up most of the 1024m of memory. Specifying the ES_JAVA_OPTS and reducing the redis memory requirements are key. 

ES_JAVA_OPTS: '-Xms512m -Xmx512m'

In my config I also have chrome webdriver to run browser tests so ended up with a services definition like so:

services:
redis:
image: redis
memory: 512
postgres:
image: postgres
chromedriver:
image: selenium/standalone-chrome
memory: 512
ports:
- "4444:4444"
- "9515:9515"
elasticsearch:
image: elasticsearch:6.5.4
environment:
ES_JAVA_OPTS: -Xms512m -Xmx512m

This 4096 MB limitation isn't great for applications that get anymore complex with dependencies.

0 votes
Motoki Wu January 24, 2018

I'm having memory issues as well. A pipeline that worked last Friday hits the memory limit. I even added the `2x` option but it still hits the memory limit.

The build should be pretty similar to each other but all of a sudden a 2x memory is not enough?

vincent-jensen January 25, 2018

For me, it feels like the pipeline can not even build the elasticsearch docker image !!

Matthew Hillier January 26, 2018

I'm having a similar issue with an instance of the "microsoft/mssql-server-linux" SQL Server service.

It looks like the additional memory options don't apply to services, they're only shared with the pipelines, so when I start my pipeline, the mouseover on the log tabs in pipelines shows:

  • Build 7GB
  • mssql - 1GB

I can't see any obvious way to re-balance this for my needs.

vincent-jensen February 8, 2018

Hi,

You guys found a way to solve this or will I have to install my own CI tool ?

John Stites February 24, 2018

I got around it for now using a jvm.options file for elastic to limit the memory. I created a new dockerfile that was based on the elastic image we need, added in a jvm.options file with:

-Xms512m
-Xmx512m

This keeps it under the 1GB service size.

They've recently added service size limitations that presumably you could just bump your elastic container to 1.5 or so (https://confluence.atlassian.com/bitbucket/use-services-and-databases-in-bitbucket-pipelines-874786688.html#UseservicesanddatabasesinBitbucketPipelines-Definingaservicewithcustomresourcelimits) but I can't get it to work.

If I follow their example with a "memory: 1536m" the online validator tool says it's expecting an integer, but if I put an integer it ignores it. Perhaps it's so new that their documentation is ahead of their implementation so maybe you'll have luck in the next days/weeks.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events