Skip to main content

Improving the execution time of CI pipelines


Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages


Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.


Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.


Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming.  If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.
Labels:

Comments

Post a Comment

Popular posts from this blog

Monday vhandit #1

This is the first post in a series of articles called "Monday vhandit". You can go to the original post by clicking on each head title. Character encodings: essential concepts This article points out why Unicode matters and differentiates among concepts: character sets, coded character set (unit is code points), and character encoding. Explained from First Principles: the Internet The very structural and well-explained article contains almost concepts of the Internet. I really appreciated the author's effort put on that post. Dev Community - DevDojo I just joined this community. DevDojo is really a developer-oriented platform. It is very nice, easy, fast, and sufficient to write (Markdown supported) and search for a post. The best of it, you also have a personal blog for your posts at "<your_username>.devdojo.com". My username is vnnvanhuong Everything you need to know about HTTP security headers A good collection of HTTP security-related headers. Moreover...
Post a Comment
Read more

Projects Ownership

Developers should be informed transparently about the next things that they are going to do since it impacts their mentality as well as their accountabilities. A view of Vung Tau beach First thing first, the Product Owner plays a very important role in understanding the business of the projects clearly and planning the User Stories for Development Teams. For Development Teams, we have two roles related to a project including Owner and Maintainer. The following are the definition and responsibilities for each role: Project Owner The Owner is normally assigned to the team that created the project at the beginning. It is known as the long-term project of the team. Therefore, the members of the Owner have almost knowledge of the projects including the history (i.e decisions made), business logic, and technologies used.  - Maintain a good system design (code) of the projects. Therefore, they are mandatory to review code (pull requests) from other teams. - Maintain the related documents ...
Post a Comment
Read more

Monday vhandit #2

  Introduction to OpenLDAP directory service "A directory is a specialized database specially designed for searching and browsing, in addition to supporting basic lookup and update functions" A directory service can be local, providing a restricted context; or global, providing service to a much broader context. Curlie is a good example of a directory service. LDAP (Lightweight Directory Access Protocol) is a protocol for accessing directory services , specifically X.500-based directory services. OpenLDAP is an open-source implementation of LDAP. Writing system software: code comments I have read Clean Code (by Uncle Bob) and I thought that I should void comments since the code it explains its implementation itself. That is right but not always true. In this post, the author categorized the comments into 9 types. Only "trivial comments" and "backup comments" are the ones that should be avoided. I myself agree with "writing good comments is harder tha...
Post a Comment
Read more

My 2017 Review

Passion for System Design After finishing a one year project, my longest stable team (lasted for 3 years) was separated into two smaller teams. Sadly, but that was a good chance for me to become a key member in my new team. My preferred skills were about system architectures; therefore most of the tasks of building the application structures were handled by me. In order to enhance my design system skills, I have spent much my time for reading books closely after work. These following books help me a lot. - Object-Oriented Thought Process | Matt Weisfeld - Head First Design Pattern  | Elisabeth Freeman and Kathy Sierra - Java 8 in Action: Lambdas, Streams, and Functional-style Programming | Alan Mycroft and Mario Fusco Junior Technical Architect I was requested to join a technical architect team (aka Team. Alpha) where I actually had gained experiences almost on interviewing candidates for my company (lol). Besides, I noticed myself must improve the skills of convinci...
1 comment
Read more