Note: This is an updated (preferred) version of this original post about running JMeter tests in OpenShift using Jenkins. This version uses the more user-friendly JMeter reports dashboard for displaying test results.
I've recently been working with an app team to set up automated integration and performance testing with Apache JMeter for their suite of microservices hosted in OpenShift and orchestrated by a Jenkins pipeline. One of the challenges we faced was the integration of each of these tools into one seamless process, with all of the necessary OpenShift objects for JMeter and quick communication between Jenkins and the cluster. The eventual goal was to slip this into our main CI/CD pipeline.
JMeter dashboard view from Jenkins
The final result is a section of our Jenkins pipeline that:
- builds an image of the JMeter test suite in OpenShift
- spins up an OpenShift job with the latest image of the test suite
- runs the JMeter tests included in the container against all the services in the OpenShift project
- waits for the JMeter test pod to tell the pipeline it is finished with the Jenkins Webhook Step plugin
- grabs the final JMeter reports dashboard created by the tests and displays it using the Jenkins HTML Publisher plugin
This proved to be a bit of a hassle to coordinate, so here are the steps I took, both for my future self and anyone else who might benefit. :)
Setting up the Test Suite Repo
Required files in the repo:
- JMeter JMX files all in one folder
- a Dockerfile -- we're using a Docker build strategy in our test suite's BuildConfig, so this is required to produce the test suite image
- OpenShift objects in YAML form (a BuildConfig, ImageStream, and Job)
- a Jenkinsfile for coordinating the OpenShift objects and parsing the test results
- shell script as entrypoint for the test suite image
Creating the JMeter Base Image
Thanks to this awesome blog post, this part wasn't too complicated. This rhel-jmeter image contains Java and JMeter, and sets up the jmeter/tests and jmeter/results folders that will contain the application team's JMX test files. This image should be built and stored separately in whatever registry you're using, or included at the beginning of the test suite Dockerfile instead of using "FROM."
You need one extra file stored with the above Dockerfile: a custom user.properties file (below) that is used to create the pretty JMeter reports dashboard. This is copied over at the end of the Dockerfile, so make sure the user.properties file location matches the one in the Dockerfile.
Learn more about configurations in the user.properties file here.
Creating the Test Suite Image
The test suite image inherits the rhel-jmeter base image above and copies the JMeter test files and scripts. (In the test suite repo, all the JMX test files are stored in the jmeter folder.) When we build the integration test suite image in OpenShift, the container will have Java, JMeter, and the JMX files, as well as the runjob.sh script that coordinates the tests running.
Building the OpenShift Objects
To run these tests in OpenShift, we need a BuildConfig, an ImageStream, and a Job.
The BuildConfig pulls down the test suite repo and builds the image above with all the test files, using the repo's Dockerfile. The ImageStream organizes all of your test suite images over time, and gives you the most recent one with the "latest" tag. The Job runs a pod using the latest test suite image, with the runjob.sh script as its entrypoint.
We're using a Job here because it conveniently spins down after completing the runjob.sh script, but you could also use a DeploymentConfig with a single pod replica and then scale it down manually in the Jenkinsfile after receiving the test results.
The BuildConfig and ImageStream objects are saved in YAML form. To create/update these in OpenShift, you'll run oc apply -f file.yaml.
For building images from a secure Git repo, you'll also need a secret with your Git credentials saved in your OpenShift project. Run:
oc secrets new-basicauth gitsecret --username=<git-user> --password=<git-pass>
Then make sure your BuildConfig uses the source secret "gitsecret."
The Job is structured as a Template, so that we can input our own parameters when we create it. The Job will be created by processing the template and piping it into an "oc apply" command: oc process -f job-template.yaml | oc apply -f - -n <project>.
BuildConfig
ImageStream
Job Template
Running the Tests
The Job runs whatever is under the "command" line of the template. In this case, it's a script that we've included in the test suite image -- runjob.sh. This script runs the specified JMeter tests file in the container against the other services in the OpenShift project, creates a reports dashboard of the results (basically just a folder of HTML files), and then curls back to a Jenkins webhook to let the pipeline know that the tests finished running.
It takes two arguments (under the "args" line):
- Jenkins pipeline webhook URL -- the URL of the webhook step we create so we know when the tests have finished running
- file name -- name of the JMX file to run
Coordinating with Jenkins
Our setup of Jenkins uses the Kubernetes plugin to dynamically spin up agents inside OpenShift, labeled 'jenkins-agent' in the node line below. Each of these agents has the oc client and git installed.
Here's the gist of the Jenkins side of things:
- spin up a Jenkins agent and login to the OpenShift cluster
- checkout the test suite Git repo into the agent's workspace
- build the test suite image in your OpenShift project using the build config and image stream objects
- for every JMX file found in the test suite repo:
- kick off a job in OpenShift that runs the test
- create an webhook step in the pipeline that waits for the test pod to complete
- wait for the job to curl back to that webhook step
- once the tests have run, get the job's pod name and retrieve the reports dashboard folder
- use the HTML Publisher plugin to display the results in Jenkins
In our full CI/CD pipeline Jenkinsfile, most of these steps are in methods and reused elsewhere, but for the purpose of this outline I've separated them out and commented each line.
Output
The pipeline logs will look a little something like this:
While running this in Jenkins, you can view the test suite logs by hopping into OpenShift and looking for pods named jmeter-test-suite-* in your project. Once the jobs complete, you'll see a number of "_____ Report" links show up on the Jenkins build sidebar, and there are your test results!
Additional Features
Here are some other things you could add or change about the above Jenkins pipeline depending on your setup:
- If your tests take a long time to run and you don't want to use up a Jenkins executor waiting for the JMeter tests to finish, you could set up the webhook step to run outside of an agent and only spin up an agent to retrieve the reports dashboard files.
- If you want to add parameters to your JMeter tests, you can append "-JDuration=___" or other options to the end of the JMeter command in runjob.sh and pass in additional parameters through the Jenkinsfile.
- If you want to run all of your tests in parallel, you can move everything from the for-loop into a method and have the jobs use the test files as their name, then create a parallel step for each test file.
- If your JMeter pod is taking forever or getting an out-of-memory error, you can increase the container resources or add min/max heap sizes with the JAVA_OPTS or JAVA_ARGS environment variables.
- If you need a really high thread count for your JMeter tests and would like to run them with a JMeter master and multiple JMeter slaves in parallel, you can create an additional JMeter slave image and spin up as many of those with the master as you need.
Feel free to email me if you'd like to see any of these in action.