Elastic Experiment Executor

yorlov May 11, 2018

I am a developer of Awesome Graphs from the StiltSoft team.

We are in the process of making the app compatible with the new Data Center program. For this purpose, we used the Performance Testing Framework for Bitbucket to run an experiment in our own environment. We managed to set up the framework and run the workload, but we have some trouble with analyzing the data (the Analyzing Your Experiment Data step).

I am running e3 experiment, and having problems with parsing some of the metrics.

2018-02-15 13:08:48,991 - MainThread - INFO - Graphing node 'cluster-node-001' for stage 'stage-000' of thread 'thread-001' of run 'first-blood'
2018-02-15 13:08:49,719 - MainThread - INFO - Graphing node 'worker-node-001' for stage 'stage-000' of thread 'thread-001' of run 'first-blood'
2018-02-15 13:08:49,719 - MainThread - INFO - Graphing node 'file-server-001' for stage 'stage-000' of thread 'thread-001' of run 'first-blood'
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/gantt-thread-001-stage-000.gnuplot
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/times-thread-001-stage-000.gnuplot
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000.gnuplot
line 0: Non-numeric string found where a numeric expression was expected
cat: stdout: Broken pipe
line 0: Non-numeric string found where a numeric expression was expected
cat: stdout: Broken pipe
line 0: Non-numeric string found where a numeric expression was expected
cat: stdout: Broken pipe
line 0: Non-numeric string found where a numeric expression was expected
cat: stdout: Broken pipe
gnuplot> plot "<(cat /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/thread-001/cluster-node-001/collectd-csv/localhost/GenericJMX-com.atlassian.bitbucket_HostingTickets/gauge-QueuedRequests-????-??-??)" using ($1-start/1000):2 with filledcurves x1 fs transparent solid 0.5 noborder lc 1 title "cluster-node-001 GenericJMX-com.atlassian.bitbucket_HostingTickets/gauge-QueuedRequests"

line 0: all points y value undefined!

Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/throughput.gnuplot
Warning: empty y range [0:0], adjusting to [0:1]
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/histogram-thread-001.gnuplot
Traceback (most recent call last):
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 104, in <module>
calculate_histogram(grinder_files, workload_list, sys.stdout)
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 65, in calculate_histogram
tps = float(fields[6])
ValueError: invalid literal for float(): 10,62
line 0: warning: Skipping data file with no valid points
Traceback (most recent call last):
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 104, in <module>
calculate_histogram(grinder_files, workload_list, sys.stdout)
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 65, in calculate_histogram
tps = float(fields[6])
ValueError: invalid literal for float(): 10,62
line 0: warning: Skipping data file with no valid points
Traceback (most recent call last):
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 104, in <module>
calculate_histogram(grinder_files, workload_list, sys.stdout)
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 65, in calculate_histogram
tps = float(fields[6])
ValueError: invalid literal for float(): 10,62
line 0: warning: Skipping data file with no valid points
Traceback (most recent call last):
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 104, in <module>
calculate_histogram(grinder_files, workload_list, sys.stdout)
File "/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py", line 65, in calculate_histogram
tps = float(fields[6])
ValueError: invalid literal for float(): 10,62
line 0: warning: Skipping data file with no valid points
gnuplot> plot "<(/Users/yorlov/Projects/elastic-experiment-executor/e3/analysis/gnuplot/CalculateHistogram.py /Users/yorlov/Projects/elastic-experiment-executor/e3-home/workloads/bitbucket-mixed.json /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/thread-001/worker-node-001/stage-000/e3-worker-node1-0.log)" using 2:xtic(1), for [i=3:5] "" using i
^
line 0: x range is invalid
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/mean-test-time-001.gnuplot
Warning: empty y range [0:0], adjusting to [0:1]
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/mean-test-time-002.gnuplot
Warning: empty y range [0:0], adjusting to [0:1]
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/mean-test-time-003.gnuplot
Warning: empty y range [0:0], adjusting to [0:1]
Plotting /Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/mean-test-time-004.gnuplot
Warning: empty y range [0:0], adjusting to [0:1]
montage: improper image header `/Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000-00006.png' @ error/png.c/ReadPNGImage/4231.
montage: improper image header `/Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000-00004.png' @ error/png.c/ReadPNGImage/4231.
montage: improper image header `/Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000-00001.png' @ error/png.c/ReadPNGImage/4231.
montage: improper image header `/Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000-00002.png' @ error/png.c/ReadPNGImage/4231.
montage: improper image header `/Users/yorlov/Projects/elastic-experiment-executor/e3-home/runs/first-blood/analysis/collectd-thread-001-stage-000-00003.png' @ error/png.c/ReadPNGImage/4231.

Looking at the worker logs (e3-worker-node1-0.log) it's easy to see, that PushChangesSsh provisioning step is failing on all worker threads, so I supposedthat metrics parsing errors, at the end of the experiment, happen because of incomplete provisioning step (in other words, some metrics are just missed, because there is no data in BB).

Tests resulting in error only contribute to the Errors column.
Statistics for individual tests can be found in the data file, including
(possibly incomplete) statistics for erroneous tests. Composite tests
are marked with () and not included in the totals.

Looking at the exception stacktrace, it looks for me that worker just cannot find ssh users for BB instances (arrays with BB users is empty, so we have IndexOutOfBoundsExceptions)

Experiment configuration (first-blood.json) as well as node (e3-app-node-1.json) and worker (e3-worker-node-1.json) descriptions are attached.

Could you suggest what to do in this situation? 

1 answer

1 vote
Andrey Levchenko July 30, 2018

Hi Yuri

Were you able to resolve issues? I'm working for Mohami and we are going to use E3 framework for same purpose but we run into similar issues.

Andrey Levchenko August 10, 2018

I figured out that changing [logger_command] to DEBUG in e3/logging.conf may help with troubleshooting.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events