Pipelines node 8 broken

[Ergun Polat]

Hi, 

I am trying to use node version 8.1.2, but I am getting the following erorr 

gyp WARN EACCES user "nobody" does not have permission to access the dev dir "/usr/local/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp/8.1.2"

Here is my config 

image: node:8.1.2

pipelines:
  default:
    - step:
        script:
          - cd pipeline
          - apt-get update && apt-get install -y npm libcairo2-dev libjpeg62-turbo-dev libpango1.0-dev libgif-dev build-essential g++
          - npm install --global node-pre-gyp
          - npm install --global @angular/cli
          - npm install && npm cache clean
          - export BRANCH="$BITBUCKET_COMMIT"
          - npm run build-prod
          - chmod +x install.sh && ./install.sh

  branches:
    master:
      - step:
          script:
            - cd pipeline
            - apt-get update && apt-get install -y npm libcairo2-dev libjpeg62-turbo-dev libpango1.0-dev libgif-dev build-essential g++
            - npm install --global node-pre-gyp
            - npm install --global @angular/cli
            - npm install && npm cache clean
            - export BRANCH="latest"
            - npm run build-prod
            - chmod +x install.sh && ./install.sh

options:
  docker: true

If anyone can help that would be much appricated.

[Philip Hodder · Accepted solution]

There was a bug introduced to npm 5. @Ergun Polat I've found three potential work arounds for you.

1. Use a node:7.* image. (This uses npm 4.x)

2. Running "npm install @angular/cli" (without --global) before running it again with the global flag seems to also work.

3. Upgrade the npm version to at least 5.1.0.

The issue is with npm version 5.0.x. This is the issue to follow:

https://github.com/npm/npm/issues/16766

Which is referenced by:

https://github.com/nodejs/docker-node/issues/423

https://github.com/nodejs/docker-node/issues/424

This issue should be fixed in npm 5.1.0 (the node docker image is still on npm 5.0.3). :)

Fix: https://github.com/npm/npm/commit/58be2ec596dfb0353ad2570e6750e408339f1478

Release: https://github.com/npm/npm/releases/tag/v5.1.0

[Ergun Polat]

@Philip Hodder Thanks for providing me these solution. Option 1 is our current solution. We will wait for the node docker image release of 5.1.0 before moving to node8. Thanks

[Ryan OConnell]

@Philip Hodder 

We are experiencing the same issue but can not downgrade to node 7.*
Do you have instructions on how to use npm 5.1.0 with pipelines
I tried adding the line below at the start of my script but that did not work
- npm install -g npm@5.1.0 

We use image: node:8.11.1

Thanks

Ryan

[Philip Hodder]

Hi Ryan

When did it start breaking, and what's the error message you're getting?

Thanks,

Phil

[Ryan OConnell]

It started breaking about 2hrs ago.

Here is the error message
> node-sass@4.9.2 install /opt/atlassian/pipelines/agent/build/node_modules/node-sass> node scripts/install.jsUnable to save binary /opt/atlassian/pipelines/agent/build/node_modules/node-sass/vendor/linux-x64-57 : { Error: EACCES: permission denied, mkdir '/opt/atlassian/pipelines/agent/build/node_modules/node-sass/vendor' at Object.fs.mkdirSync (fs.js:885:18) at sync (/opt/atlassian/pipelines/agent/build/node_modules/mkdirp/index.js:71:13) at Function.sync (/opt/atlassian/pipelines/agent/build/node_modules/mkdirp/index.js:77:24) at checkAndDownloadBinary (/opt/atlassian/pipelines/agent/build/node_modules/node-sass/scripts/install.js:114:11) at Object.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/node-sass/scripts/install.js:157:1) at Module._compile (module.js:652:30) at Object.Module._extensions..js (module.js:663:10) at Module.load (module.js:565:32) at tryModuleLoad (module.js:505:12) at Function.Module._load (module.js:497:3) errno: -13, code: 'EACCES', syscall: 'mkdir', path: '/opt/atlassian/pipelines/agent/build/node_modules/node-sass/vendor' }> node-sass@4.9.2 postinstall /opt/atlassian/pipelines/agent/build/node_modules/node-sass> node scripts/build.jsBuilding: /usr/local/bin/node /opt/atlassian/pipelines/agent/build/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=gyp info it worked if it ends with okgyp verb cli [ '/usr/local/bin/node',gyp verb cli '/opt/atlassian/pipelines/agent/build/node_modules/node-gyp/bin/node-gyp.js',gyp verb cli 'rebuild',gyp verb cli '--verbose',gyp verb cli '--libsass_ext=',gyp verb cli '--libsass_cflags=',gyp verb cli '--libsass_ldflags=',gyp verb cli '--libsass_library=' ]gyp info using node-gyp@3.7.0gyp info using node@8.11.1 | linux | x64gyp verb command rebuild []gyp verb command clean []gyp verb clean removing "build" directorygyp verb command configure []gyp verb check python checking for Python executable "python2" in the PATHgyp verb `which` succeeded python2 /usr/bin/python2gyp verb check python version `/usr/bin/python2 -c "import platform; print(platform.python_version());"` returned: "2.7.9\n"gyp verb get node dir no --target version specified, falling back to host node version: 8.11.1gyp verb command install [ '8.11.1' ]gyp verb install input version string "8.11.1"gyp verb install installing version: 8.11.1gyp verb install --ensure was passed, so won't reinstall if already installedgyp WARN EACCES user "undefined" does not have permission to access the dev dir "/root/.node-gyp/8.11.1"gyp WARN EACCES attempting to reinstall using temporary dev dir "/opt/atlassian/pipelines/agent/build/node_modules/node-sass/.node-gyp"gyp verb tmpdir == cwd automatically will remove dev files after to save disk spacegyp verb command install [ '--node_gyp_internal_noretry', '8.11.1' ]gyp verb install input version string "8.11.1"gyp verb install installing version: 8.11.1gyp verb install --ensure was passed, so won't reinstall if already installedgyp verb install version not already installed, continuing with install 8.11.1gyp verb ensuring nodedir is created /opt/atlassian/pipelines/agent/build/node_modules/node-sass/.node-gyp/8.11.1gyp WARN install got an error, rolling back installgyp verb command remove [ '8.11.1' ]gyp verb remove using node-gyp dir: /opt/atlassian/pipelines/agent/build/node_modules/node-sass/.node-gypgyp verb remove removing target version: 8.11.1gyp verb remove removing development files for version: 8.11.1gyp WARN install got an error, rolling back install

[Philip Hodder]

Can you try adding

unset NPM_CONFIG_USER

to the start of your build?

We did some infrastructure changes that seem to be affecting a subset of npm builds. We're currently working on a proper fix for this, but for the time being you can try the above workaround.

[Ryan OConnell]

That worked. Thanks!

[Philip Hodder]

Fix is deployed for this now, so you can remove the command now if you need. :)

[Gil Michael Regalado]

@Philip Hodder Where can we track this so that we can remove once this is fixed? We had to do this to make our pipeline work agian.

[Philip Hodder]

It's fixed now. Let me know if it works as expected now.

[Dan Dombrowski]

Not working for us yet. When you say "to the start of your build" what do you mean? First line in our script section? If we have multiple steps, should it be in each script section of each step?

[Dan Dombrowski]

...I think the temp fix had to be in each step? Seems to to work after doing that.

[Philip Hodder]

Yup, that's right. The temp fix shouldn't be required any more though. Let us know if that's not the case.

[Jirka Doubek]

We have the issue still in our builds, the temp hack works.

[Máté Hajzso]

We too still have this issue.

[nanomosfet]

I still have a build error on node-sass

[nopuck4you]

I've been having issues all day today as well. Whats the best way to get on a notification list when changes to the pipeline process are deployed by Atlassian and when issues are reported? (Or did you sell that to Slack as well? :grin: ) I'd like to not pull out what few hairs I have on my head. :) 

[Philip Hodder]

If the 'unset' command workaround doesn't work. You can try use a docker image that isn't based off of an Alpine image. For example, if you're using node:8, you can try node:8-jessie.

You can check if you're using an alpine image, by running:

$ cat /etc/os-release

There's a downstream bug affecting us that we'll have more details on soon. I'll link you information here when it's out.

Sorry for the inconvenience. 

[Philip Hodder]

Hello,

You can track the progress of the issue here: https://bitbucket.org/site/master/issues/16846/npm-error-eacces-permission-denied

We'll update the ticket as we get more information on the required fixes.

Thanks,

Phil

[GC]

Thanks @Philip Hodder! I was also trying node build today and getting similar error while using node:8.11-apline image :

EACCES: permission denied, mkdir '/opt/atlassian/pipelines/agent/build/node_modules/cwebp-bin/vendor'

Switching to node:8.11-jessie resolved it.

[Yoga M]

We were using node 10.0.0 and was facing this issue. We fixed it by replacing node v10 with v8.11.1.

[xtjhin]

At what command does your build fail at?

[Ergun Polat]

I've added the log 

Building: /usr/local/bin/node /usr/local/lib/node_modules/@angular/cli/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
gyp info it worked if it ends with ok
gyp verb cli [ '/usr/local/bin/node',
gyp verb cli   '/usr/local/lib/node_modules/@angular/cli/node_modules/node-gyp/bin/node-gyp.js',
gyp verb cli   'rebuild',
gyp verb cli   '--verbose',
gyp verb cli   '--libsass_ext=',
gyp verb cli   '--libsass_cflags=',
gyp verb cli   '--libsass_ldflags=',
gyp verb cli   '--libsass_library=' ]
gyp info using node-gyp@3.6.2
gyp info using node@8.1.2 | linux | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` succeeded python2 /usr/bin/python2
gyp verb check python version `/usr/bin/python2 -c "import platform; print(platform.python_version());"` returned: "2.7.9\n"
gyp verb get node dir no --target version specified, falling back to host node version: 8.1.2
gyp verb command install [ '8.1.2' ]
gyp verb install input version string "8.1.2"
gyp verb install installing version: 8.1.2
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp WARN EACCES user "undefined" does not have permission to access the dev dir "/root/.node-gyp/8.1.2"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp"
gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '8.1.2' ]
gyp verb install input version string "8.1.2"
gyp verb install installing version: 8.1.2
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 8.1.2
gyp verb ensuring nodedir is created /usr/local/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp/8.1.2
gyp WARN EACCES user "undefined" does not have permission to access the dev dir "/usr/local/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp/8.1.2"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/@angular/cli/node_modules/node-sass/.node-gyp"
gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '8.1.2' ]