Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

Bamboo times out publishing artifact

Brandon Vulaj1
Brandon Vulaj October 9, 2014

I have a Job that (tries to) publish an artifact that other Stages and Jobs are dependant upon.  However, it seems to time out whenever it tries to publish the artifact.

If it matters, the artifact matches **/* so that I can copy the sources / compiled sources along the pipeline without re-checking out or re-compiling.  The artifact is only roughly 1mb.

I've attached the log below.  Although Bamboo will retry and repeat the process for ~30 minutes or so until it finally gives up.

2014-10-09 15:09:03,803 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [ExecuteBuildTask] Running post build plugin 'Artifact Copier'
2014-10-09 15:09:03,805 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [BuildArtifactPostProcessor] Copying the build artifacts for build: HS-WS-MCL-111
2014-10-09 15:09:03,823 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [AbstractArtifactManager] Publishing [HoS Sources + Compiled Sources] for HS-WS-MCL-111: 68 file(s) mat ching [**/*] in directory /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MCL
2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out
2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request
2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out
2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request
2014-10-09 15:11:37,472 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-343] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:168)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
	at java.io.InputStream.read(InputStream.java:82)
	at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

-----

It may be worth mentioning that the first job does succeed to publish the artifact if it's the first job run on a newly spun up instance.  However, subsequent jobs can not process the artifact. Logs below.

2014-10-09 17:55:12,544 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) taken from queue.
2014-10-09 17:55:12,545 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) assigned to agent 5013525
2014-10-09 17:55:12,618 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Changing context: null -> HS-WS-MUTL-113 on Elastic Agent on i-6be6a785/57319f76
2014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) started building on agent Elastic Agent on i-6be6a785
2014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.v2.build.task.InitializeBuild
2014-10-09 17:55:12,787 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) VCS sync started
2014-10-09 17:55:12,947 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [CheckoutDirectoriesSnapshotHelper] Build working directory is /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MUTL
2014-10-09 17:55:12,948 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask
2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Executing build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113)
2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Preparing artifact for use: HoS Sources + Compiled Sources
2014-10-09 17:55:39,205 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        at java.io.InputStream.read(InputStream.java:82)
        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2014-10-09 17:55:40,203 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        at java.io.InputStream.read(InputStream.java:82)
        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request

 

 

Answer
Watch
Like Be the first to like this
1254 views

2 answers

1 accepted

0 votes
Answer accepted
Pawel Skierczynski
Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2014

If you have EIP on elastic agent and it will change IP after agent started it may be a cause of a problem.

If that's the case setting it up before agent starts could help.

This one is a little outdated but take a look at comments section. 

https://confluence.atlassian.com/display/BAMKB/Automatically+associating+Elastic+IP+addresses+to+Elastic+Agents

Reply
0 votes
Pawel Skierczynski
Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2014

It's not very efficient to do it if there are many small files. How many files are there? If there are many could you try to tar/zip it before? 

Have in mind that every job can be run on separate agent and in parallel in one stage. Also jobs in another stages don't have to run on same agents. So it's copied through the network both ways. I think it will also index information of every single file on a server side to database (but I may be wrong here).

Usually you would get sources from repository and dependant binaries through some kind of repository (like artifactory). It's all case by case basis so it's hard to say without knowing your exact pipeline.

 

View More Comments
Brandon Vulaj1
Brandon Vulaj October 9, 2014

Hey Pawel, So I only have one agent running at the moment. There are 142 files for about 1.5mb. The pipeline looks like this. Maybe that gives you some insight as to why I'm sharing the artifact in this way. http://i.imgur.com/2XfnULY.png

Like
Brandon Vulaj1
Brandon Vulaj October 9, 2014

If you have another suggestion, please let me know, but I would also like to solve this issue.

Like
Brandon Vulaj1
Brandon Vulaj October 9, 2014

It's worth noting that I think this captures the spirit of what the Best Practices recommend: https://confluence.atlassian.com/display/BAMBOO/Bamboo+Best+Practice+-+Sharing+artifacts

Like
Pawel Skierczynski
Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2014

You may have one agent but Bamboo does not have one agent detection and any optimization for it. It works general case. What if all those were in one job as consecutive tasks? You would have one stage and one job. Admittedly you are loosing parallel running unit tests and integration tests. But unit tests are usually very quick and having one agent you don't use it at the moment anyway. You probably have more use cases to cover (everyone does) but they are not revealed by this diagram.

Like
Pawel Skierczynski
Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2014

Those best practices would better work with only few artifacts shared. Like if you build jar in stage one and them do some integration tests with it. (I personally would try the simplest possible thing that works so one stage, one job until you know you need more) Still 142 files doesn't seem that bad. You could try zip/make artifact/unzip only to see if it helps so it narrows down possible cause. There is still possibility that something else is wrong. Do you have OnDemand instance of server? What OS is Server and Agent running on? Any firewalls/proxies/other strange things worth mentioning like virtual machines/docker etc.?

Like
Brandon Vulaj1
Brandon Vulaj October 9, 2014

I'll try the simple solution for now, which shouldn't require any artifact sharing at that level. I'm fairly positive it will work just fine. To fill you in, we are running OnDemand, we are behind a VPC and we do have EIP enabled. It is one of the default AMIs: ami-25f7ed4c. I see the comment at the bottom there - that seems like a potential solution. I'll try that and get back to you.

Like
Pawel Skierczynski
Pawel Skierczynski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 9, 2014

I'll try to get to author of best practices. I believe that unit tests are usually done with source code so this diagram could be made better.

Like
Reply

Suggest an answer

Log in or Sign up to answer
Bamboo
Bamboo
TAGS
AUG Leaders

Atlassian Community Events

    See all events