Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Gzip upload on browsers

Today, I faced a problem that I could not upload my archive file with gzip format on Firefox, even it worked on Chrome. I was using macOS. My application had a setting to whitelist accepted files. I’ve already added "application/gzip" to that list. "It’s strange!", I thought. I finally figured out that my uploaded file's type actually was "application/x-gzip" on Firefox. I also asked my colleagues to check their uploaded files on Window and Ubuntu. Hmm… they were totally different! It was "application/x-compressed" on Window, and was "application/x-compressed-tar" on Ubuntu. In fact, gzip is already standardized by IANA. There is a note in RFC-6713 as below: "Some applications have informally used media types such as application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, and gzip/document to describe data compressed with gzip. The media types defin...

My must-have apps for daily work

There is no doubt that cool apps can help us be more productive and enjoyable at work. For the time being, I really love the following apps which are used by me almost every day. 1. A personal Kanban In fact, a personal kanban is the most useful app for me. Why does it matter? It is not just a to-do list, but it keeps me motivated every day because it helps me be able to know what my "big picture" is. I usually set up my plans together with a path to reach them. KanbanFlow is my preferred tool. KanbanFlow 2. A terminal Needless to say, a terminal is a must-have app for every developer, especially the ones use macOS/Linux. Due to its importance, I love to decorate and enhance it to be super exciting with various tools such as iTerm , oh-my- zsh , and thefuck . ;) iTerm + oh-my-zsh 3. A documentation "ecosystem" As a developer, I can not remember all things that I have experimented a day. Moreover, a document is really useful for sharing an...

BIRT - Fix the size of an image

I use a dynamic image as a logo my report in pdf. At the beginning, I use table to align the logo in left or right. I meet a problem with some images with a large width or height. My customer requires that the logo should be displayed in original size. These following steps solves my problem: 1. Use Grid instead of Table 2. Set Grid "Height" is 100% and "Width" is blank 3. Set "Fit to container" for images are "true". Download the the template here .

Automating deployment and managing apps on OpenShift

Previously, we maintained OpenShift templates for deploying apps in development environments as well as delivering these templates to our customers for their on-prem deployment. Customers who refer to our templates (as well as documentation) have their own configuration management tools to automate the deployment such as ArgoCD and FluxCD. My son's buildings Our developers usually modify templates (YAML) directly on OpenShift for testing and then adjust the corresponding templates stored in the Git repository in Bitbucket. This sometimes causes an issue that delivered templates are incorrect because: - Developers forget to update the templates in Git repositories. - Developers don’t test the templates Therefore, our goal was to integrate a tool into our CI/CD that can automate and manage the configuration of OpenShift apps. The delivered templates should be the ones that are able to run on our OpenShift with the following purposes: - Automate deployment from templated in Git repos...

DevOps Toolchain Enhancement

Historically, our company ubitec had started with a customer project. Agile/Scrum was our proposal for working with customers. Time by time, Agile/Scrum also became our culture for software development. To be successful with this development approach, we somehow needed to have a fast release for customers (i.e. every one week). Back then, we had a build tool Jenkins which was responsible for having sprint release packages for our customers. The build job pipelines contain some steps such as gathering the artifacts, checking the code convention, running the tests, building docker images, and packaging an archived file (a zip file). The set of tools involved in a pipeline is roughly called a toolchain. It is just a part of a bigger process called the DevOps toolchain. Source: https://www.ibm.com/blogs/cloud-archive/2016/11/devops-architecture-available-on-bluemix-garage-method-site/ DevOps is a proven method that fits Agile. Today, it is even treated as a mandatory factor...

vhandit

Search This Blog