OOM when running pipelines

[Vladimir Grigor]

Hello all,

as one (sometimes failing) step of our pipeline we

•  setup environment via installation of docker which runs 3 containers through own docker-compose. Unfortunately we could not figure out how to impose any memory limits on individual containers in docker-compose v3 config.
• run tests in `sbt` with  `SBT_OPTS="-Xmx1500m"`
• every second to third time step is failing with OOM

definitions:
  services:
    docker:
      memory: 1024

Explanation on the memory usage is not clear on this case.

Could somebody please explain what's going on with the memory allocation to containers, docker host, and step env?

Interesting is that running the same tests, same containers for multiple times usually gives different results. Understandable that real load on the physical server changes over time and running times can differ - but memory?

Puzzled...

[Kenny MacLeod · Accepted solution]

The "memory: 1024" setting you refer to is the total memory made available to the docker daemon running as part of your step. All containers started by docker as part of your step will share that memory, so if you're running 3 containers you could easily hit that limit.

As the documentation that you linked to says, you can increase that up to a maximum of 3GB (leaving 1GB for the rest of your step), or up to 7GB if you're using 2x step size.

[Vladimir Grigor]

Thank you Kenny for confirming the behaviour!