Bitbucket pipeline deployment File not found

[Nir Haimov]

Hi All,

I'm not very expert with yml files.
Tried to create the pipeline with some help over the internet.

So far i manage to built my project in Bitbucket with atlas-package.

And i want to use the pipeline deploment to upload my project jar to the "deployment" section of the repository, i tried using Bitbucket built in feature:

script:
- pipe: atlassian/bitbucket-upload-file:0.1.4
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: 'package.json'

So my complete yml file is:

image: quantify/atlassian-sdk:latest

pipelines:
branches:
master:
- step:
caches:
- maven
script:
#- atlas-clean
- atlas-package
- pipe: atlassian/bitbucket-upload-file:0.1.4
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: --form files=@"target/$(cd ./target/ && ls -1 myproject*.jar)"

 And in the "pipe: atlassian/bitbucket-upload-file:0.1.4" stage, i get this error:

File --form files=@"target/myproject.jar" doesn't exist.

Can anyone please help me solve this?

[Nir Haimov · Accepted solution]

@Halyna Berezovska 
Just now i was also managed to accomplish this by using curl command

script: - atlas-clean - atlas-package - curl -X POST --user "${BB_AUTH_STRING}" "https://api.bitbucket.org/2.0/repositories/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}/downloads" --form files=@"target/$(cd ./target/ && ls -1 select2*.jar)"

[Halyna Berezovska]

Yes, you can do like that with curl. But I guess right now it is not supported within our pipe. Since we are not using bash, but python, it can not be used like that. But I'll propose some improvements for this. I notify once it is approved

[Nir Haimov]

Actually it does :)
I use it now and it works perfect.

Of course it will be much easier if you will add it as a feature as you mentioned before :)

[Vladislav Babin]

@Nir Haimov , thanks for sharing the solution! It works

[Halyna Berezovska · Accepted solution]

Hello,@Nir Haimov , we do not support such declaration of FILENAME variable, since it is only a part of curl  command and we do specific checks for the file existence. So, FILENAME should be the local relative / absolute path to the file.

For example, "target/myProject.jar" locating at "$working_dir_of_container/target/myProject.jar" or sth like 

"/opt/project/target/myproject.jar"

You also may tell us the path where you have this file and we will help to construct the FILENAME variable for you.

Thanks for using our pipe.

[Nir Haimov]

Hi @Halyna Berezovska 

Thank you for your answer!

This is the output of the "atlas-package" stage:

+ atlas-package[INFO] Project POM foundExecuting: /usr/share/atlassian-plugin-sdk-8.0.16/apache-maven-3.5.4/bin/mvn package -gs /usr/share/atlassian-plugin-sdk-8.0.16/apache-maven-3.5.4/conf/settings.xml[INFO] Scanning for projects...[INFO] [INFO] ---------------------< com.pitronote.jira:select2 >---------------------[INFO] Building select2 1.3.3[INFO] --------------------------[ atlassian-plugin ]--------------------------[INFO] [INFO] --- jira-maven-plugin:8.0.0:compress-resources (default-compress-resources) @ select2 ---[INFO] Compiling javascript using YUI[INFO] 4 Javascript file(s) were minified into target directory /opt/atlassian/pipelines/agent/build/target/classes[INFO] 1 CSS file(s) were minified into target directory /opt/atlassian/pipelines/agent/build/target/classes[INFO] Compressing XML files[INFO] 3 XML file(s) were minified into target directory /opt/atlassian/pipelines/agent/build/target/classes[INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ select2 ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Copying 13 resources[INFO] [INFO] --- jira-maven-plugin:8.0.0:filter-plugin-descriptor (default-filter-plugin-descriptor) @ select2 ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Copying 1 resource[INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ select2 ---[INFO] No sources to compile[INFO] [INFO] --- jira-maven-plugin:8.0.0:generate-rest-docs (default-generate-rest-docs) @ select2 ---[INFO] [INFO] --- atlassian-spring-scanner-maven-plugin:1.2.13:atlassian-spring-scanner (default) @ select2 ---[INFO] Starting Atlassian Spring Byte Code Scanner...[INFO] [INFO] [INFO]  Analysis ran in 54 ms.[INFO]  Encountered 0 total classes[INFO]  Processed 0 annotated classes[INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ select2 ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] skip non existing resourceDirectory /opt/atlassian/pipelines/agent/build/src/test/resources[INFO] [INFO] --- jira-maven-plugin:8.0.0:filter-test-plugin-descriptor (default-filter-test-plugin-descriptor) @ select2 ---[INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ select2 ---[INFO] No sources to compile[INFO] [INFO] --- jira-maven-plugin:8.0.0:unit-test (default-unit-test) @ select2 ---[INFO] Surefire 2.22.1 test configuration:[INFO] <?xml version="1.0" encoding="UTF-8"?><configuration>  <systemPropertyVariables/>  <excludes>    <exclude>%regex[it[/\\].*]</exclude>    <exclude>**/*$*</exclude>  </excludes>  <excludedGroups/></configuration>[INFO] No tests to run.[INFO] [INFO] --- jira-maven-plugin:8.0.0:copy-bundled-dependencies (default-copy-bundled-dependencies) @ select2 ---[INFO] Unpacking /root/.m2/repository/com/google/code/gson/gson/2.2.2-atlassian-1/gson-2.2.2-atlassian-1.jar to /opt/atlassian/pipelines/agent/build/target/classes with includes "" and excludes "atlassian-plugin.xml,META-INF/MANIFEST.MF,META-INF/*.DSA,META-INF/*.SF"[INFO] Unpacking /root/.m2/repository/com/atlassian/plugin/atlassian-spring-scanner-runtime/1.2.13/atlassian-spring-scanner-runtime-1.2.13.jar to /opt/atlassian/pipelines/agent/build/target/classes with includes "" and excludes "atlassian-plugin.xml,META-INF/MANIFEST.MF,META-INF/*.DSA,META-INF/*.SF"[INFO] Unpacking /root/.m2/repository/com/atlassian/plugin/atlassian-spring-scanner-annotation/1.2.13/atlassian-spring-scanner-annotation-1.2.13.jar to /opt/atlassian/pipelines/agent/build/target/classes with includes "" and excludes "atlassian-plugin.xml,META-INF/MANIFEST.MF,META-INF/*.DSA,META-INF/*.SF"[INFO] [INFO] --- jira-maven-plugin:8.0.0:copy-test-bundled-dependencies (default-copy-test-bundled-dependencies) @ select2 ---[INFO] [INFO] --- jira-maven-plugin:8.0.0:generate-manifest (default-generate-manifest) @ select2 ---[INFO] Generating a manifest for this plugin[INFO] using maven-bundle-plugin v3.5.0[INFO] [INFO] --- jira-maven-plugin:8.0.0:generate-test-manifest (default-generate-test-manifest) @ select2 ---[INFO] [INFO] --- jira-maven-plugin:8.0.0:validate-manifest (default-validate-manifest) @ select2 ---[INFO] Manifest found, validating...[INFO] Manifest validated[INFO] [INFO] --- jira-maven-plugin:8.0.0:jar (default-jar) @ select2 ---[INFO] Building jar: /opt/atlassian/pipelines/agent/build/target/select2-1.3.3.jar[INFO] [INFO] --- jira-maven-plugin:8.0.0:generate-obr-artifact (default-generate-obr-artifact) @ select2 ---[INFO] using maven-bundle-plugin v3.5.0[INFO] Writing OBR metadata[INFO] Installing select2-1.3.3.jar[INFO] Writing OBR metadata[INFO] Building jar: /opt/atlassian/pipelines/agent/build/target/select2-1.3.3.obr[INFO] [INFO] --- jira-maven-plugin:8.0.0:validate-test-manifest (default-validate-test-manifest) @ select2 ---[INFO] [INFO] --- jira-maven-plugin:8.0.0:test-jar (default-test-jar) @ select2 ---[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 4.524 s[INFO] Finished at: 2020-04-30T17:56:01Z[INFO] ------------------------------------------------------------------------

According to this, if i'm not mistake, the output folder is:

/opt/atlassian/pipelines/agent/build/target/select2-1.3.3.jar

as stated in this line:

[INFO] Building jar: /opt/atlassian/pipelines/agent/build/target/select2-1.3.3.jar

And this is the output of the deployment stage:

+ docker container run \   --volume=/opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build \   --volume=/opt/atlassian/pipelines/agent/ssh:/opt/atlassian/pipelines/agent/ssh:ro \   --volume=/usr/local/bin/docker:/usr/local/bin/docker:ro \   --volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes \   --volume=/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/bitbucket-upload-file:/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/bitbucket-upload-file \   --workdir=$(pwd) \   --label=org.bitbucket.pipelines.system=true \   --env=BITBUCKET_STEP_TRIGGERER_UUID="$BITBUCKET_STEP_TRIGGERER_UUID" \   --env=BITBUCKET_REPO_FULL_NAME="$BITBUCKET_REPO_FULL_NAME" \   --env=BITBUCKET_GIT_HTTP_ORIGIN="$BITBUCKET_GIT_HTTP_ORIGIN" \   --env=BITBUCKET_REPO_SLUG="$BITBUCKET_REPO_SLUG" \   --env=BITBUCKET_PROJECT_UUID="$BITBUCKET_PROJECT_UUID" \   --env=CI="$CI" \   --env=BITBUCKET_REPO_OWNER="$BITBUCKET_REPO_OWNER" \   --env=BITBUCKET_REPO_IS_PRIVATE="$BITBUCKET_REPO_IS_PRIVATE" \   --env=BITBUCKET_WORKSPACE="$BITBUCKET_WORKSPACE" \   --env=BITBUCKET_REPO_OWNER_UUID="$BITBUCKET_REPO_OWNER_UUID" \   --env=BITBUCKET_STEP_RUN_NUMBER="$BITBUCKET_STEP_RUN_NUMBER" \   --env=BITBUCKET_BUILD_NUMBER="$BITBUCKET_BUILD_NUMBER" \   --env=BITBUCKET_BRANCH="$BITBUCKET_BRANCH" \   --env=BITBUCKET_GIT_SSH_ORIGIN="$BITBUCKET_GIT_SSH_ORIGIN" \   --env=BITBUCKET_COMMIT="$BITBUCKET_COMMIT" \   --env=BITBUCKET_REPO_UUID="$BITBUCKET_REPO_UUID" \   --env=BITBUCKET_CLONE_DIR="$BITBUCKET_CLONE_DIR" \   --env=BITBUCKET_PROJECT_KEY="$BITBUCKET_PROJECT_KEY" \   --env=PIPELINES_JWT_TOKEN="$PIPELINES_JWT_TOKEN" \   --env=BITBUCKET_DOCKER_HOST_INTERNAL="$BITBUCKET_DOCKER_HOST_INTERNAL" \   --env=DOCKER_HOST="tcp://host.docker.internal:2375" \   --env=BITBUCKET_PIPE_SHARED_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes" \   --env=BITBUCKET_PIPE_STORAGE_DIR="/opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/atlassian/bitbucket-upload-file" \   --env=BITBUCKET_APP_PASSWORD="$BITBUCKET_APP_PASSWORD" \   --env=BITBUCKET_USERNAME="$BITBUCKET_USERNAME" \   --env=FILENAME="--form files=@\"target/$(cd ./target/ && ls -1 select2*.jar)\"" \   --add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \   bitbucketpipelines/bitbucket-upload-file:0.1.4Unable to find image 'bitbucketpipelines/bitbucket-upload-file:0.1.4' locally0.1.4: Pulling from bitbucketpipelines/bitbucket-upload-filec499e6d256d6: Pulling fs layer62b0f1bf7919: Pulling fs layer30b7b8e6b50d: Pulling fs layerf4947adccc3c: Pulling fs layer219c07faa520: Pulling fs layer1fdd33e2f7a9: Pulling fs layerceb102cff1e0: Pulling fs layer8c00eb12c498: Pulling fs layer5c3f857c8dd5: Pulling fs layerceb102cff1e0: Waiting1fdd33e2f7a9: Waitingf4947adccc3c: Waiting8c00eb12c498: Waiting219c07faa520: Waiting5c3f857c8dd5: Waiting62b0f1bf7919: Verifying Checksum62b0f1bf7919: Download completef4947adccc3c: Verifying Checksumf4947adccc3c: Download completec499e6d256d6: Verifying Checksumc499e6d256d6: Download complete1fdd33e2f7a9: Download complete219c07faa520: Download complete30b7b8e6b50d: Verifying Checksum30b7b8e6b50d: Download complete8c00eb12c498: Verifying Checksum8c00eb12c498: Download completeceb102cff1e0: Download complete5c3f857c8dd5: Download completec499e6d256d6: Pull complete62b0f1bf7919: Pull complete30b7b8e6b50d: Pull completef4947adccc3c: Pull complete219c07faa520: Pull complete1fdd33e2f7a9: Pull completeceb102cff1e0: Pull complete8c00eb12c498: Pull complete5c3f857c8dd5: Pull completeDigest: sha256:dd2f61c460cef13dede75048c7727c6767856055e3cbeb865a08a17d1456c42dStatus: Downloaded newer image for bitbucketpipelines/bitbucket-upload-file:0.1.4INFO: Executing the pipe...✖ File --form files=@"target/select2-1.3.3.jar" doesn't exist.

Hope you can help me figure out how to fix this.

Thanks,

Nir.

[Halyna Berezovska]

@Nir Haimov  , I believe, this corrected command should help you

docker run --volume .... --env ..... \
--env=FILENAME="/opt/atlassian/pipelines/agent/build/target/select2-1.3.3.jar" ...

Unfortunately, we do not support paths by pattern like you likely wanted:

/opt/atlassian/pipelines/agent/build/target/select2-*.jar

Right now I have some solutions (workarounds) for you to propose:

• write script to identify the EXACT path of your jar file, so that you could run docker with env variable I wrote above
• identify the version on the file that select2 file will be generated with since you know another path as "/opt/atlassian/pipelines/agent/build/target/" (and it never changes) and file will be like ${TARGET_PATH}/select2-${version_of_jar_generated-1.3.3}.jar

And then you can run docker with appropriate absolute path you defined and it should work.

In the meantime, I will discuss the possibility to accept not only full path, but patterns of file also, like it is in your situation. If it is approved, we will implement this as the feature in the future release

[Nir Haimov]

Thank you @Halyna Berezovska 

Will appreciate if you can keep me posted if the featured approved or not.
and if yes, when it's gonna be released :)

[Halyna Berezovska]

@Nir Haimov sure, thank you for giving such an idea

[Halyna Berezovska]

@Nir Haimov this feature will be included in the next pipe releases. Thank you

[Nir Haimov]

@Halyna Berezovska thank you so much! :)

[Vladislav Babin]

Hi,

@Oleksandr Kyrdan , thanks for sharing that!

My own solution (based on @Nir Haimov 's):

- atlas-package
- source ./target/maven-archiver/pom.properties
- curl -X POST --user "${BITBUCKET_USERNAME}:${BITBUCKET_APP_PASSWORD}" "https://api.bitbucket.org/2.0/repositories/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}/downloads" --form files=@"target/${artifactId}-${version}.war"

[Oleksandr Kyrdan]

Hi @Nir Haimov @Vladislav Babin ,

new bitbucket-upload-file pipe's version that supports multiple files released:

script:
  - pipe: atlassian/bitbucket-upload-file:0.2.0
    variables:
      BITBUCKET_USERNAME: $BITBUCKET_USERNAME
      BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
      FILENAME: 'target/select2*.jar'

Welcome to share your feedback.

[Alex Bondarev]

`target` folder was created in the previous step and not present in the next step 

script:
  - pipe: atlassian/bitbucket-upload-file:0.3.3
    variables:
      BITBUCKET_USERNAME: $BITBUCKET_USERNAME
      BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
      FILENAME: 'target/select2*.jar'

by default. what should i do reuse build results - i.e. make target/*.* files available for upload?