TeamCity results don't match JMeter's

Question

I\'m currently using JMeter to load test some application, I used the duration assertion and I set it to, let\'s say 200, when the load time exceeds 200, the Request in the Resu

Answer 1

You don't really specify a lot of information. How do you run the application? Do you have the TeamCity JMeter plugin available?

I run a maven project with the jmeter-maven-plugin and teamcity with the JMeter-plugin. Now you can specify in the pom.xml if you want the test to fail on test failures:

<ignoreResultFailures>false</ignoreResultFailures>

Please also read info on the TeamCity plugin as it uses a specific (non-standard) jtl format:

    <propertiesJMeter>
      <jmeter.save.saveservice.default_delimiter>\t</jmeter.save.saveservice.default_delimiter>
      <jmeter.save.saveservice.print_field_names>true</jmeter.save.saveservice.print_field_names>
    </propertiesJMeter>

For more info, see this page, which explains the configuration of the TeamCity JMeter plugin: https://www.blazemeter.com/blog/how-run-jmeter-tests-teamcity-continuous-integration

As a bonus: I invested some time to see how to pass parameters to the JMeter script from the maven build. I came to the conclusion that the maven plugin does not support the normal -J JMeter parameters and you should use -D instead:

    <jMeterProcessJVMSettings>
      <arguments>
        <argument>-Dimage_path=${teamcity.build.workingDir}/src/test/resources/images</argument>
        <argument>-Denvironment=TEST</argument>
      </arguments>
    </jMeterProcessJVMSettings>

Now in JMeter you can use the following variable value to get the value specified in the pom.xml:

${__groovy(System.getProperty("image_path"\,"/some/default/dir"))}

Answer 2

Take a look at the Performance Metrics Calculation under Build Features in Team City's project. You need to make sure that assertions is checked in order to make the build fail on assertions.

In addition to that, the newer versions of JMeter and Team City add more columns to the .jtl file that is output, so you need to make sure your "success" column is in the 5th spot in the file. I found this out after coming across this post in the TeamCity JMeter Plugin git repo: https://github.com/jtorgan/jmeter_plugin/issues/24#issuecomment-421016226

Instead of creating a script, I just made sure to remove some columns in the JMeter user.properties settings on the TeamCity server to make sure the success column was in the 5th spot.

Here were the columns I had to turn off under user.properties for JMeter:

jmeter.save.saveservice.response_message=false
jmeter.save.saveservice.thread_name=false
jmeter.save.saveservice.data_type=false

Answer 3

What you are looking at is the summary. You need to add a listener to your project, that saves results to a file. Then you need to have Teamcity read that file and assert the result.

Teamcity is just telling you "Ehhhhh the script you wanted me to run worked".

Here's a small guide of how: https://devblog.xero.com/run-jmeter-performance-tests-on-teamcity-8315f7ccffc1#.8ga4jso7k