Improving the execution time of CI pipelines

Executing a large number of tests, especially integration tests, takes a lot of time. For instance, the pipeline of one of our projects for each Pull Request previously took nearly 30 minutes, including over 1 thousand test cases. This article guides you through several good techniques that we have discovered and applied to improve the time-consuming process.

Parallel stages

Analyze the current phases in your pipeline and categorize them in parallel. For example, we can separate the build and verify code of Node.js and Maven modules simultaneously in our Jenkins pipelines. Please mind using the setting failFast whether you want to abort the pipeline immediately.

Parallel test execution

If you use Maven, the plugin maven-failsafe-plugin is used to execute integration tests during phases integration-test and verify the build lifecycle. It allows us to execute tests in parallel. There are many settings related to parallel test execution, so make sure to configure them correctly for your environment. For example, we can execute tests in parallel using the classes mode and set threadCount=7 for the Jenkins Agent with 8 CPU cores.

Application profiles

There are certain integration tests that don't require the use of a database. These tests focus on scenarios like error handling, authentication filters, or loading configuration. For example, initializing the database with Testcontainers is time-consuming. If you use Spring Boot, you can utilize a different profile and use the H2 in-memory database for these tests. The test class will be annotated with the @ActiveProfiles annotation to run with the specified profile.

Comments

Only allow input number value with autoNumeric.js

autoNumeric is a jQuery plugin that automatically formats currency and numbers as you type on form inputs. I used autoNumeric 1.9.21 for demo code. 1. Dowload autoNumeric.js file from https://github.com/BobKnothe/autoNumeric 2. Import to project <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script> <script type="text/javascript" src="js/autoNumeric.js"></script> 3. Define a function to use it <script type="text/javascript"> /* only number is accepted */ function txtNumberOnly_Mask() { var inputOrgNumber = $("#numberTxt"); inputOrgNumber.each(function() { $(this).autoNumeric({ aSep : '', aDec: '.', vMin : '0.00' }); }); } </script> 4. Call the function by event <form> <input type="text" value="" id="numberTxt"/>(only number) </form> <script ty...

Why Business Rules Engine Matter

"A good DSL minimizes the 'communication gap' between a domain concept and the code that implements it" - Robert C. Martin A Use Case It looks like adopting a new technology is always driven by a business need. The organizations, such as banks, have their own business processes (such as data gathering and document management). These processes are different from others but usually the differences are not big. How do we build a product which can be reused the similar features but still be adaptable with the specific requirements of each bank? Then, the answer is we should have a library/framework. There is an idea that we can implement this library/framework by using a business rules engine. Which has the following benefits: Be able to modify implementation of business domain at runtime (such as XML, CSV) S ource code is more readable for both developers and domain experts. Therefore, source code can be consider it as a document Since Business Rules Engine...

The Evolution of Team Collaboration

I n my point of view, if a team has a good collaboration, team members will achieve the following: To be more effective To make work more enjoyable I have been working for a company for nearly four years as an software developer. Working on various projects from maintaining existing systems to developing a substantial product resulted in me moving to new teams three times. Actually, my most stable team lasted only around three years. Every time I've moved to a new team, I have a chance to work with new members and a new team culture again. Indeed, I realize that there is a process of developing the team collaboration which gets better time by time. I think it is an evolution ! Phase 1: Poorly collaborate For example, that is when the team members have the following issues: Only work on his/her area of expertise Don't communicate to others Be not confident to take on new challages Don't listen to other members. Subsequently, the team ha...

Building Axon.ivy Projects on Bitbucket Pipelines

Read me if you don't know what Axon.ivy (Ivy) is. Motivation - Ivy projects are designed to be built on a continuous integration (CI) server like Jenkins - Today, Bitbucket supports for CI with Bitbucket Pipelines - We're using Bitbucket. Then, why not? It must be very cool and convenient for us if we can centralize our CI and VCS (version control system) tools in one place. Here is an approach We have to use a maven plugin called project-build-plugin to build ivy projects. This plugin requires an instance of Ivy engine during building time. Bitbucket Pipelines allows us to specify our own docker image as a build environment. What we need to do is to prepare our docker image with needed stuffs such as JDK, Maven, Ivy engine, etc. Step 1. Prepare Docker images For testing purpose, I already created two docker images: Maven and Axon.ivy engine. They are now available on Docker Hub This image for Maven using Oracle JDK 8 This image for Axon.iv...

Applying pipeline “tensorflow_embedding” of Rasa NLU

According to this nice article , there was a new pipeline released using a different approach from the standard one ( spacy_sklearn ). I wanted to give it a try to see whether it can help with improving bot’s accuracy. After applying done, I gave an evaluation of “tensorflow_embedding”. It seemed to work better a bit. For example, I defined intents “greet” and “goodbye” with some following messages in my training data. ## intent:greet - Hey! How are you? - Hi! How can I help you? - Good to see you! - Nice to see you! - Hi - Hello - Hi there ## intent:goodbye - Bye - Bye Bye - See you later - Take care - Peace In order to play around with Rasa NLU, I created a project here . You can have a look at this change from this pull request . Yay! When I entered message “hi bot”, then bot with “tensorflow_embedding” could detect intent “greet” with better confidence scores rather than bot with “spacy_sklearn”. The following are responses after executing curl -X POST loc...

vhandit

Search This Blog