Bamboo times out publishing artifact

I have a Job that (tries to) publish an artifact that other Stages and Jobs are dependant upon.  However, it seems to time out whenever it tries to publish the artifact.

If it matters, the artifact matches **/* so that I can copy the sources / compiled sources along the pipeline without re-checking out or re-compiling.  The artifact is only roughly 1mb.

I've attached the log below.  Although Bamboo will retry and repeat the process for ~30 minutes or so until it finally gives up.

2014-10-09 15:09:03,803 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [ExecuteBuildTask] Running post build plugin 'Artifact Copier'
2014-10-09 15:09:03,805 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [BuildArtifactPostProcessor] Copying the build artifacts for build: HS-WS-MCL-111
2014-10-09 15:09:03,823 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [AbstractArtifactManager] Publishing [HoS Sources + Compiled Sources] for HS-WS-MCL-111: 68 file(s) mat ching [**/*] in directory /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MCL
2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out
2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request
2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out
2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request
2014-10-09 15:11:37,472 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-343] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:168)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
	at java.io.InputStream.read(InputStream.java:82)
	at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

-----

It may be worth mentioning that the first job does succeed to publish the artifact if it's the first job run on a newly spun up instance.  However, subsequent jobs can not process the artifact. Logs below.

2014-10-09 17:55:12,544 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) taken from queue.
2014-10-09 17:55:12,545 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) assigned to agent 5013525
2014-10-09 17:55:12,618 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Changing context: null -> HS-WS-MUTL-113 on Elastic Agent on i-6be6a785/57319f76
2014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) started building on agent Elastic Agent on i-6be6a785
2014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.v2.build.task.InitializeBuild
2014-10-09 17:55:12,787 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) VCS sync started
2014-10-09 17:55:12,947 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [CheckoutDirectoriesSnapshotHelper] Build working directory is /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MUTL
2014-10-09 17:55:12,948 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask
2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Executing build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113)
2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Preparing artifact for use: HoS Sources + Compiled Sources
2014-10-09 17:55:39,205 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        at java.io.InputStream.read(InputStream.java:82)
        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2014-10-09 17:55:40,203 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        at java.io.InputStream.read(InputStream.java:82)
        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request

 

 

2 answers

1 accepted

This widget could not be displayed.

If you have EIP on elastic agent and it will change IP after agent started it may be a cause of a problem.

If that's the case setting it up before agent starts could help.

This one is a little outdated but take a look at comments section. 

https://confluence.atlassian.com/display/BAMKB/Automatically+associating+Elastic+IP+addresses+to+Elastic+Agents

This widget could not be displayed.

It's not very efficient to do it if there are many small files. How many files are there? If there are many could you try to tar/zip it before? 

Have in mind that every job can be run on separate agent and in parallel in one stage. Also jobs in another stages don't have to run on same agents. So it's copied through the network both ways. I think it will also index information of every single file on a server side to database (but I may be wrong here).

Usually you would get sources from repository and dependant binaries through some kind of repository (like artifactory). It's all case by case basis so it's hard to say without knowing your exact pipeline.

 

Hey Pawel, So I only have one agent running at the moment. There are 142 files for about 1.5mb. The pipeline looks like this. Maybe that gives you some insight as to why I'm sharing the artifact in this way. http://i.imgur.com/2XfnULY.png

If you have another suggestion, please let me know, but I would also like to solve this issue.

It's worth noting that I think this captures the spirit of what the Best Practices recommend: https://confluence.atlassian.com/display/BAMBOO/Bamboo+Best+Practice+-+Sharing+artifacts

You may have one agent but Bamboo does not have one agent detection and any optimization for it. It works general case. What if all those were in one job as consecutive tasks? You would have one stage and one job. Admittedly you are loosing parallel running unit tests and integration tests. But unit tests are usually very quick and having one agent you don't use it at the moment anyway. You probably have more use cases to cover (everyone does) but they are not revealed by this diagram.

Those best practices would better work with only few artifacts shared. Like if you build jar in stage one and them do some integration tests with it. (I personally would try the simplest possible thing that works so one stage, one job until you know you need more) Still 142 files doesn't seem that bad. You could try zip/make artifact/unzip only to see if it helps so it narrows down possible cause. There is still possibility that something else is wrong. Do you have OnDemand instance of server? What OS is Server and Agent running on? Any firewalls/proxies/other strange things worth mentioning like virtual machines/docker etc.?

I'll try the simple solution for now, which shouldn't require any artifact sharing at that level. I'm fairly positive it will work just fine. To fill you in, we are running OnDemand, we are behind a VPC and we do have EIP enabled. It is one of the default AMIs: ami-25f7ed4c. I see the comment at the bottom there - that seems like a potential solution. I'll try that and get back to you.

I'll try to get to author of best practices. I believe that unit tests are usually done with source code so this diagram could be made better.

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Published yesterday in Marketplace Apps

The 7 hacks of highly successful automation

...there's anything I've learnt from working, it's that people are lazy! No offense to anyone reading this, but it's true and we can all admit it. The easier you make something for someone, the more...

119 views 0 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you