Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Junit - Test fails on French or German string assertion

In my previous post about building a regex to check a text without special characters but allow German and French . I met a problem that the unit test works fine on my machine using Eclipse, but it was fail when running on Jenkins' build job. Here is my test: @Test public void shouldAllowFrenchAndGermanCharacters(){ String source = "ÄäÖöÜüß áÁàÀâÂéÉèÈêÊîÎçÇ"; assertFalse(SpecialCharactersUtils.isExistSpecialCharater(source)); } Production code: public static boolean isExistNotAllowedCharacters(String source){ Pattern regex = Pattern.compile("^[a-zA-Z_0-9_ÄäÖöÜüß áÁàÀâÂéÉèÈêÊîÎçÇ]*$"); Matcher matcher = regex.matcher(source); return !matcher.matches(); } The result likes the following: Failed tests: SpecialCharactersUtilsTest.shouldAllowFrenchAndGermanCharacters:32 null A guy from stackoverflow.com says: "This is probably due to the default encoding used for your Java source files. The ö in the string literal in the J...

Gzip upload on browsers

Today, I faced a problem that I could not upload my archive file with gzip format on Firefox, even it worked on Chrome. I was using macOS. My application had a setting to whitelist accepted files. I’ve already added "application/gzip" to that list. "It’s strange!", I thought. I finally figured out that my uploaded file's type actually was "application/x-gzip" on Firefox. I also asked my colleagues to check their uploaded files on Window and Ubuntu. Hmm… they were totally different! It was "application/x-compressed" on Window, and was "application/x-compressed-tar" on Ubuntu. In fact, gzip is already standardized by IANA. There is a note in RFC-6713 as below: "Some applications have informally used media types such as application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, and gzip/document to describe data compressed with gzip. The media types defin...

Java 8 - Persistent data structure

The following is a series of posts about "functional programming in Java" which is the result of my understanding by reading the book " Java 8 in Action: Lambdas, Streams, and Functional-style Programming, by Alan Mycroft and Mario Fusco ". 1. Why functional programming? 2. Functional programming in Java 8 3. Java 8 - Using Functions as Values 4. Java 8 - Persistent data structure Persistent data structure is also known as a simple technique but it's very important. Its other names are functional data structure and immutable data structure. Why is it "persistent"? Their values persist and are isolated from changes happening elsewhere. That's it! This technique is described as below: If you need a data structure to represent the result of a computation, you should make a new one and not mutable an existing data structure. Destructive updates version public static A doSomething(A a){ a.setProp1("new value"); return...

DevOps Toolchain Enhancement

Historically, our company ubitec had started with a customer project. Agile/Scrum was our proposal for working with customers. Time by time, Agile/Scrum also became our culture for software development. To be successful with this development approach, we somehow needed to have a fast release for customers (i.e. every one week). Back then, we had a build tool Jenkins which was responsible for having sprint release packages for our customers. The build job pipelines contain some steps such as gathering the artifacts, checking the code convention, running the tests, building docker images, and packaging an archived file (a zip file). The set of tools involved in a pipeline is roughly called a toolchain. It is just a part of a bigger process called the DevOps toolchain. Source: https://www.ibm.com/blogs/cloud-archive/2016/11/devops-architecture-available-on-bluemix-garage-method-site/ DevOps is a proven method that fits Agile. Today, it is even treated as a mandatory factor...

Fulfilling Your Contribution Needs

Human resource management motivation Managing human today is quite different from the industrial age which treats people as just "chickens". Rather than people now are very important to the success of an organization. People are an organization's special resource. They should be encouraged to grow to contribute their effort and creativeness to their beloved working environment because the contribution is one of their most needs in life. Training people: getting rid of the ineffective model and adopting the new one The ineffective model of training people: Hiring new people --> giving them a crash course once --> expecting them working effectively. That somehow makes sense but you're about to expect a luck because you do not really spend your effort for mentoring them. If they can work effectively, well...lucky you! Otherwise, you will blame that these people are ineffective and you let them go and hire the new ones. What a waste of time! The new effe...

vhandit

Search This Blog