Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Selenium - Override javascript functions to ignore downloading process

I have got an issue with downloading process on IE 8. This browser blocks my automatic-download functionality on my app so that I could not work with my test case any more after that. In my case, I didn't care about the file is downloaded or not, I just focus on the result after downloading process finished successfully. Therefore, I found a solution to ignore this process so that I can work normally. I use Primefaces, here is the remote command to trigger the download process <p:remoteCommand name="cmdGenerateDocument" actionListener="#{logic.onGenerateDocument}" update="xrflDocumentCreationPanel" oncomplete="clickDownloadButton();"/> The following is my test case: @Test public void shouldUpdateStep6ToWarningIfStep1IsValidAfterFinished(){ MainPage mainPage = new MainPage(); waitForLoading(mainPage.getDriver()); EmployeeDetailPage empDetailPage = new EmployeeDetailPage(); waitForLoading(empDetailPage.getDriver()); e...

[Snippet] CSS - Child element overlap parent

I searched from somewhere and found that a lot of people says a basic concept for implementing this feature looks like below: HTML code: <div id="parent"> <div id="child"> </div> </div> And, CSS: #parent{ position: relative; overflow:hidden; } #child{ position: absolute; top: -1; right: -1px; } However, I had a lot of grand-parents in my case and the above code didn't work. Therefore, I needed an alternative. I presumed that my app uses Boostrap and AngularJs, maybe some CSS from them affects mine. I didn't know exactly the problem, but I believed when all CSS is loaded into my browser, I could completely handle it. www.tom-collinson.com I tried to create an example to investigated this problem by Fiddle . Accidentally, I just changed: position: parent; to position: static; for one of parents -> the problem is solved. Look at my code: <div class="modal-body dn-placeholder-parent-positi...

Generating PDF/A From HTML in Meteor

My live-chat app was a folk of project Rocket.Chat which was built with Meteor. The app had a feature that administrative users were able to export the conversations into PDF files. And, they wanted to archive these files for a long time. I happened to know that PDF/A documents were good for this purpose. It was really frustrated to find a solution with free libraries. Actually, it took me more than two weeks to find a possible approach. TL, DR; Using Puppeteer to generate a normal PDF and using PDFBox to load and converting the generated PDF into PDF/A compliance. What is PDF/A? Here is a definition from Wikipedia: PDF/A is an ISO -standardized version of the Portable Document Format (PDF) specialized for use in the archiving and long-term preservation of electronic documents . PDF/A differs from PDF by prohibiting features unsuitable for long-term archiving, such as font linking (as opposed to font em...

PSMDB - A MongoDB alternative for having Encryption At Rest

Encryption is the most popular tool for securing data both in transit and at rest. - For protecting data in transit, we can configure to use the TLS connection - For protecting data at rest, we can use Percona Server for MongoDB (PSMDB), an open-source alternative for MongoDB Enterprise. License PSMDB Docker images follow the SSPL license. Therefore, it is not a problem when I only have my containers deployed in on-premises environments. Running MongoDB Replication on OpenShift I have successfully installed the replication by following the guide Install Percona Server for MongoDB on OpenShift . In order to make it work properly with my needs, I disabled some features from the default deployment. See the detail in this change Basically, I needed to create a CRD (Custom Resource Definition) to let OpenShift/Kubernetes what PSMDB is. Then, I deployed the Operator pod. Finally, I deployed the PSMDB StatefulSet. I used NFS shares for Persistent Volumes. Create CRD for PSMDB 2 git clone http...

Why Functional Programming Matter

What issues do we concern when implementing and maintaining systems? One of the most concern is debugging during maintenance: "this code crashed because it observed some unexpected value." Then, it turns out that the ideas of no side effects and immutability , which functional programming promotes, can help. Shared mutable data is the root cause Shared mutable data are read and updated by more than one of the methods. Share mutable data structures make it harder to track changes in different parts of your program. An immutable object is an object that can't change its state after it's instantiated so it can't be affected by the actions of a function. It would be a dream to maintain because we wouldn't have any bad surprises about some object somewhere that unexpectedly modifies a data structure. A new thinking: Declarative programming There are two ways thinking about implementing a system by writing a program. - Imperative programming: has...

vhandit

Search This Blog