Using elasticsearch docker image as service, requires periods in environment variable names

[Lyndon Howie]

Hi,

I'm trying to run elasticsearch docker image as a service alongside my pipelines CI, however some of the configuration options we would like to specify as environment vars for elasticsearch have periods in the name eg. discovery.type, and pipelines does not seem to allow variable names with a period in them.
I'm wondering what options there are to work around this, is it possible to bind the es config file to one in our repository, or some other way to get such named options through? Or would creating a custom docker image with our desired config be the only way to do this for now?

Thanks,

Lyndon

[dlurth]

This also works:

script:
- docker run -td --name elasticsearch -p 9200:9200 -e discovery.type="single-node" -e ES_JAVA_OPTS="-Xms512m -Xmx512m" docker.elastic.co/elasticsearch/elasticsearch:7.9.2
- docker ps
- sleep 15s
- docker logs elasticsearch
- curl -is 127.0.0.1:9200

[Alex Seliviorstov]

That's a shame and really confusing that environment format is not documented.

As an example this yml file passes in the validator https://bitbucket-pipelines.prod.public.atl-paas.net/validator

image: php:7.3-alpine

pipelines:
  default:
    - step:
        caches:
          - composer
        services:
          - elasticsearch
        script:
          - apk update && apk add git openssh
          - sed -i 's/<env name="LOCATION_SEARCH_ELASTICSEARCH_SERVER" value="localhost:9999"\/>/<env name="LOCATION_SEARCH_ELASTICSEARCH_SERVER" value="localhost:9200"\/>/' phpunit.xml
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer install
          - cp .env.example .env
          - php artisan key:generate
          - vendor/bin/phpunit

definitions:
  services:
    elasticsearch:
      image: elasticsearch:7.3.0
      environment:
        - discovery.type=single-node
        - "ES_JAVA_OPTS=-Xmx512m -Xms512m"

But pipelines keep saying to use a different format.

 

However nothing is mentioned about it here

https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html#Configurebitbucket-pipelines.yml-ci_services

Or

https://confluence.atlassian.com/bitbucket/use-services-and-databases-in-bitbucket-pipelines-874786688.html

So basically if docker image uses not underscore but period in env variables there is no way around it in bitbucket pipelines:)

[Nathan_Rzepecki]

You find a way around this?

[Nathan_Rzepecki]

might help https://bitbucket.org/site/master/issues/14892/there-is-no-way-to-set-an-environment

[Alex Seliviorstov]

@Nathan_Rzepecki well, solution is to create your own image, e.g.:

FROM elasticsearch:7.3.0

RUN chown -R :elasticsearch /usr/share/elasticsearch/data
ENV discovery.type single-node


EXPOSE 9200

And use it in pipeline yml as: 

# This is a sample build configuration for PHP.
# Check our guides at https://confluence.atlassian.com/x/e8YWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: php:7.3-cli-alpine

pipelines:
  default:
    - step:
        services:
          - elasticsearch
        script:
          - composer install
          - vendor/bin/phpunit

definitions:
  services:
    elasticsearch:
      image: user/package-name:7.3
      environment:
        ES_JAVA_OPTS: '-Xms512m -Xmx512m'

I intentionally don't use cache for composer as it starts everything simultaneously. So for elastisearch it takes around 10-15 sec and composer install ~20 sec. Not sure what would be the best approach here. Other potential option is to use bash script with while.

[Nathan_Rzepecki]

Thats what I ended up doing https://cloud.docker.com/u/lionslair/repository/docker/lionslair/elasticsearch-bitbucket-pipelines

I have the vars set in the docker image so no need to pass any

[Ivan Takarlikov]

You can use my docker image: https://hub.docker.com/r/xiting/elasticsearch-bitbucket-pipeline

It is based on fresh Elasticsearch version: 7.3.2

Add service to your pipeline as below:

definitions:
  steps:
    - step: &run-tests
        name: Run tests
        script:
          - sleep 30 # Waiting elasticsearch. In your real pipeline you can not use it.
          - curl -XGET localhost:9250/_cat/health
        services:
          - elasticsearch
  services:
    elasticsearch:
      image: xiting/elasticsearch-bitbucket-pipeline
      variables:
        ES_JAVA_OPTS: '-Xms512m -Xmx512m'
    docker:
      memory: 2048

pipelines:
  pull-requests:
    '**':
      - step: *run-tests

[ericsantosnasc]

@Andrii Soldatenko I managed to but I had to build a custom docker image with my variables in the docker compose, like discovery.type

https://stackoverflow.com/questions/47286989/possible-solution-for-bitbucket-pipeline-docker-run-limitation/47304667#47304667

[Andrii Soldatenko]

any updates?

[ericsantosnasc]

Hello @Lyndon Howie did you managed to do this? I need same thing :D

[Lyndon Howie]

Hey,

Sorry for the extra late reply, the email notification went to my junk mail and I just saw it!

I had to build a custom docker image with the variables i needed based on the public ES docker image. I see you've done something similar below.

Cheers