Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Git Feature Branch Workflow

Motivator It's important for a team to have an agreement on how the changes of source code should be applied. According to projects and teams size, we will define a workflow or select one from recommended workflows ; the "Feature Branch Workflow" is a candidate. What is it? - One branch "master" for main codebase - Several separated branches for features development Why should we care? - Be super simple and allow each developer works on a particular feature. - A stable codebase (master) benefits for continuous integration (CI) environment - Leverage "Pull request" for Code review How it works? A lifecyle of a feature branch (usually created by a story) 1. Creator creates a new branch from a story. For example: "ABC-1-setup-projects" 2. Creator checkouts the created branch and works on the branch (commits, pushes) 3. Creator has done the feature, he uses "pull request" to merge his branch into branch "master...

Monday vhandit #2

Introduction to OpenLDAP directory service "A directory is a specialized database specially designed for searching and browsing, in addition to supporting basic lookup and update functions" A directory service can be local, providing a restricted context; or global, providing service to a much broader context. Curlie is a good example of a directory service. LDAP (Lightweight Directory Access Protocol) is a protocol for accessing directory services , specifically X.500-based directory services. OpenLDAP is an open-source implementation of LDAP. Writing system software: code comments I have read Clean Code (by Uncle Bob) and I thought that I should void comments since the code it explains its implementation itself. That is right but not always true. In this post, the author categorized the comments into 9 types. Only "trivial comments" and "backup comments" are the ones that should be avoided. I myself agree with "writing good comments is harder tha...

Managing JAR files with Apache Maven

Apache Maven (Maven) is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information. Why Maven? Let’s learn about the Maven’s history here https://maven.apache.org/background/history-of-maven.html to understand why Maven is created. There are a lot of useful things that we should use Maven. In my opinion, the most important thing is that Maven solves the problems with managing jar files. It centralizes these files in one place and it is easy to use by declaring dependencies in a xml file (pom.xml). Using Maven to manage JAR files Firstly, I would like to give you an example about using some JAR files without Maven. In my project, I need some JAR files such as SNMP4J.jar, jta26.jar, jgraphx.jar, etc… so I had to search them on the Internet with visiting a lot of websites, download and add these files into my proje...

Building Axon.ivy Projects on Bitbucket Pipelines

Read me if you don't know what Axon.ivy (Ivy) is. Motivation - Ivy projects are designed to be built on a continuous integration (CI) server like Jenkins - Today, Bitbucket supports for CI with Bitbucket Pipelines - We're using Bitbucket. Then, why not? It must be very cool and convenient for us if we can centralize our CI and VCS (version control system) tools in one place. Here is an approach We have to use a maven plugin called project-build-plugin to build ivy projects. This plugin requires an instance of Ivy engine during building time. Bitbucket Pipelines allows us to specify our own docker image as a build environment. What we need to do is to prepare our docker image with needed stuffs such as JDK, Maven, Ivy engine, etc. Step 1. Prepare Docker images For testing purpose, I already created two docker images: Maven and Axon.ivy engine. They are now available on Docker Hub This image for Maven using Oracle JDK 8 This image for Axon.iv...

My must-have apps for daily work

There is no doubt that cool apps can help us be more productive and enjoyable at work. For the time being, I really love the following apps which are used by me almost every day. 1. A personal Kanban In fact, a personal kanban is the most useful app for me. Why does it matter? It is not just a to-do list, but it keeps me motivated every day because it helps me be able to know what my "big picture" is. I usually set up my plans together with a path to reach them. KanbanFlow is my preferred tool. KanbanFlow 2. A terminal Needless to say, a terminal is a must-have app for every developer, especially the ones use macOS/Linux. Due to its importance, I love to decorate and enhance it to be super exciting with various tools such as iTerm , oh-my- zsh , and thefuck . ;) iTerm + oh-my-zsh 3. A documentation "ecosystem" As a developer, I can not remember all things that I have experimented a day. Moreover, a document is really useful for sharing an...

vhandit

Search This Blog