Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Avoiding Time-Wasting Pitfalls in Agile Estimation

If you do Scrum at work, you might be very familiar to the estimation in Planning 1 . My PO has once complained to my team that why it took too long for estimating just a story. Wasting time results in the planning timebox is violated. I give you some advice from my experience: Estimation is estimation, not measure. When you read some requirements, you see some risks but you actually don't know how complicated it will be. Don't try to influence the others by explaining how to do it in too detail. Just keep in mind that you know the business domain pertaining to customer needs and estimate how much effort you will spend for it. The effort should be compared to your baseline one that you use for a simple requirement. The bottom line is we do "relative estimation", not absolute estimation. For example, you are asked to estimate the height of a building. Basically, you just need to answer "how many times higher is the build than your height"; you do...

Math fundamentals and Katex

It was really tough for me to understand many articles about data science due to the requirements of understanding mathematics (especially linear algebra). I’ve started to gain some basic knowledges about Math by reading a book first. The great tool Typora and stackedit with supporting Katex syntax simply helps me to display Math-related symbols. Let’s start! The fundamental ideas of mathematics: “doing math” with numbers and functions. Linear algebra: “doing math” with vectors and linear transformations. 1. Solving equations Solving equations means finding the value of the unknown in the equation. To find the solution, we must break the problem down into simpler steps. E.g: x 2 − 4 = 4 5 x 2 − 4 + 4 = 4 5 + 4 x 2 = 4 9 x = 4 9 ∣ x ∣ = 7 x = 7 or x = − 7 \begin{aligned} x^2 - 4 &= 45\\ x^2 - 4 + 4 &= 45 + 4\\ x^2 &= 49\\ \sqrt{x}&=\sqrt{49}\\ |x| &= 7\\ x=7 &\text{ or } x=-7 \end{aligned} x 2 − 4 x 2 − 4 + 4 x 2 x ∣ x ∣ x = 7 = 4 5 = 4 ...

How I did customize "rasa-nlu-trainer" as my own tool

Check out my implementation here Background I wanted to have a tool for human beings to classify intents and extract entities of texts which were obtained from a raw dataset such as Rocket.chat's conversation, Maluuba Frames or here . Then, the output (labeled texts) could be consumed by an NLU tool such as Rasa NLU. rasa-nlu-trainer was a potential one which I didn't need to build an app from scratch. However, I needed to add more of my own features to fulfill my needs. They were: 1. Loading/displaying raw texts stored by a database such as MongoDB 2. Manually labeling intents and entities for the loaded texts 3. Persisting labeled texts into the database I firstly did look up what rasa-nlu-trainer 's technologies were used in order to see how to implement my mentioned features. At first glance rasa-nlu-trainer was bootstrapped with Create React App. Create React App is a tool to create a React app with no build configuration, as it said. This too...

The HelloWorld example of JSF 2.2 with Myfaces

I just did by myself create a very simple app "HelloWorld" of JSF 2.2 with a concrete implementation Myfaces that we can use it later on for our further JSF trying out. I attached the source code link at the end part. Just follow these steps below: 1. Create a Maven project in Eclipse (Kepler) with a simple Java web application archetype "maven-archetype-webapp". Maven should be the best choice for managing the dependencies , so far. JSF is a web framework that is the reason why I chose the mentioned archetype for my example. 2. Import dependencies for JSF implementation - Myfaces (v2.2.10) into file pom.xml . The following code that is easy to find from http://mvnrepository.com/ with key words "myfaces". <dependency> <groupId>org.apache.myfaces.core</groupId> <artifactId>myfaces-api</artifactId> <version>2.2.10</version> </dependency> <dependency> <groupId>org.apache.myfaces.core<...

Strategy Design Pattern

For example, I have a program with an Animal abstract class and two sub-classes Dog and Bird. I want to add new behavior for the class Animal, this is "fly". Now, I face two approaches to solve this issue: 1. Adding an abstract method "fly" into the class Animal. Then, I force the sub-classes should be implemented this method, something like: public abstract class Animal{ //bla bla public abstract void fly(); } public class Bird extends Animal{ //bla bla public void fly(){ System.out.println("Fly high"); } } public class Dog extends Animal{ //bla bla public void fly(){ System.out.println("Cant fly"); } } 2. Creating an interface with method "fly" inside. The same issue to an abstract class, I force the classes these implement this interface should have a method "fly" inside: public interface Flyable{ public void fly(); } public class Bird implements Flyable{ //bla bla public void fly(){ System.out.pr...

vhandit

Search This Blog