Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

My impression of Google I/O 2016

I have not yet had a chance to attend this event but I just watched it on Youtube as usual. :) Google Assistant You might be very familiar to Google Now if you're Android fan. I saw that Google Assistant is like an upgrade to Google Now. It looks smarter because we are able to make a conversation with Google rather than just saying "Ok Google" and getting the tasks done as Google Now. That means Google Assistant can understand our personal words in our contexts. For example: I : Who is Google CEO? Google : Sundar Pichai I : Show me his awards => Here I don't need to say "Show me Sundar Pichai awards", because I don't know to pronounce his name correctly. :) I am thinking about that we can compare Google Assistant to Siri of Apple. Google Home At Google I/O 2015 last year, Sundar mentioned about working on a exciting project with Internet of Things and I think Google Home is a result. Working with Google Assistant, Google ho...

Selenium - Override javascript functions to ignore downloading process

I have got an issue with downloading process on IE 8. This browser blocks my automatic-download functionality on my app so that I could not work with my test case any more after that. In my case, I didn't care about the file is downloaded or not, I just focus on the result after downloading process finished successfully. Therefore, I found a solution to ignore this process so that I can work normally. I use Primefaces, here is the remote command to trigger the download process <p:remoteCommand name="cmdGenerateDocument" actionListener="#{logic.onGenerateDocument}" update="xrflDocumentCreationPanel" oncomplete="clickDownloadButton();"/> The following is my test case: @Test public void shouldUpdateStep6ToWarningIfStep1IsValidAfterFinished(){ MainPage mainPage = new MainPage(); waitForLoading(mainPage.getDriver()); EmployeeDetailPage empDetailPage = new EmployeeDetailPage(); waitForLoading(empDetailPage.getDriver()); e...

Monday vhandit #2

Introduction to OpenLDAP directory service "A directory is a specialized database specially designed for searching and browsing, in addition to supporting basic lookup and update functions" A directory service can be local, providing a restricted context; or global, providing service to a much broader context. Curlie is a good example of a directory service. LDAP (Lightweight Directory Access Protocol) is a protocol for accessing directory services , specifically X.500-based directory services. OpenLDAP is an open-source implementation of LDAP. Writing system software: code comments I have read Clean Code (by Uncle Bob) and I thought that I should void comments since the code it explains its implementation itself. That is right but not always true. In this post, the author categorized the comments into 9 types. Only "trivial comments" and "backup comments" are the ones that should be avoided. I myself agree with "writing good comments is harder tha...

Performance of a Data Structure

Why data structures matter The fact is that programs are all about processing data. Data structures are referred to how data is organized which affects the time of executing a program. How to measure the performance of a data structure In order to measure "how fast"/efficiency/performance of a data structure, we measure the performance of its operations. There are four basic operations including reading , searching , insertion , and deletion . A pure time consuming is not used for the measuring because it is not reliable depending on the hardware that it is run on. But instead, we use the term time complexity which refers to how many steps an operation takes. An example of how a single rule can affect efficiency Let's compare two data structures: Array and Set (with N elements). 1. Array - Reading : 1 step (because the computer has the ability to jump to any particular index in the array) - Searching : N steps (the worst case with linear search) - Inserti...

How I did customize "rasa-nlu-trainer" as my own tool

Check out my implementation here Background I wanted to have a tool for human beings to classify intents and extract entities of texts which were obtained from a raw dataset such as Rocket.chat's conversation, Maluuba Frames or here . Then, the output (labeled texts) could be consumed by an NLU tool such as Rasa NLU. rasa-nlu-trainer was a potential one which I didn't need to build an app from scratch. However, I needed to add more of my own features to fulfill my needs. They were: 1. Loading/displaying raw texts stored by a database such as MongoDB 2. Manually labeling intents and entities for the loaded texts 3. Persisting labeled texts into the database I firstly did look up what rasa-nlu-trainer 's technologies were used in order to see how to implement my mentioned features. At first glance rasa-nlu-trainer was bootstrapped with Create React App. Create React App is a tool to create a React app with no build configuration, as it said. This too...

vhandit

Search This Blog