Skip to main content

Improving the execution time of CI pipelines


Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages


Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.


Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.


Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming.  If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.
Labels:

Comments

Post a Comment

Popular posts from this blog

Projects Ownership

Developers should be informed transparently about the next things that they are going to do since it impacts their mentality as well as their accountabilities. A view of Vung Tau beach First thing first, the Product Owner plays a very important role in understanding the business of the projects clearly and planning the User Stories for Development Teams. For Development Teams, we have two roles related to a project including Owner and Maintainer. The following are the definition and responsibilities for each role: Project Owner The Owner is normally assigned to the team that created the project at the beginning. It is known as the long-term project of the team. Therefore, the members of the Owner have almost knowledge of the projects including the history (i.e decisions made), business logic, and technologies used.  - Maintain a good system design (code) of the projects. Therefore, they are mandatory to review code (pull requests) from other teams. - Maintain the related documents ...
Post a Comment
Read more

Building Axon.ivy Projects on Bitbucket Pipelines

Read me  if you don't know what Axon.ivy (Ivy) is. Motivation -  Ivy projects are designed to be built on a continuous integration (CI) server like Jenkins - Today, Bitbucket supports for CI with Bitbucket Pipelines - We're using Bitbucket. Then, why not? It must be very cool and convenient for us if we can centralize our CI and VCS (version control system) tools in one place. Here is an approach We have to use a maven plugin called project-build-plugin  to build ivy projects. This plugin requires an instance of Ivy engine during building time. Bitbucket Pipelines allows us to specify our own docker image as a build environment. What we need to do  is to prepare our docker image with needed stuffs such as JDK, Maven, Ivy engine, etc. Step 1. Prepare Docker images For testing purpose, I already created two docker images: Maven and Axon.ivy engine. They are now available on Docker Hub This image for Maven using Oracle JDK 8 This image for Axon.iv...
Post a Comment
Read more

My must-have apps for daily work

There is no doubt that cool apps can help us be more productive and enjoyable at work. For the time being, I really love the following apps which are used by me almost every day. 1. A personal Kanban In fact, a personal kanban is the most useful app for me. Why does it matter? It is not just a to-do list, but it keeps me motivated every day because it helps me be able to know what my "big picture" is. I usually set up my plans together with a path to reach them.  KanbanFlow  is my preferred tool. KanbanFlow 2. A terminal Needless to say, a terminal is a must-have app for every developer, especially the ones use macOS/Linux. Due to its importance, I love to decorate and enhance it to be super exciting with various tools such as  iTerm ,  oh-my- zsh , and  thefuck . ;) iTerm + oh-my-zsh 3. A documentation "ecosystem" As a developer, I can not remember all things that I have experimented a day. Moreover, a document is really useful for sharing an...
Post a Comment
Read more

Tip for resolving the compile errors when importing a Maven project in Eclipse

Sometimes, I import a existing Maven project into Eclipse and I get some compile errors related to "pom.xml", "facet version of Web", and  so on. It was really frustrating me because I even cleaned, built and updated the project many times but the errors still occurred. I accidentally found the following steps to resolve this issue but honestly I still don't know why. Step 1 : Delete the project that met the compile errors in Eclipse, but don't delete it on disk. Step 2 : Open the project (by Windows Explorer if you use Windows) and remove  file/folder ".classpath", ".settings" and ".project" . Step 3 : Import the project back into Eclipse. ---^0^--- Maven is a build automation tool used primarily for Java projects. Eclipse is an integrated development environment (IDE).
Post a Comment
Read more