Abstract
Continuous Integration and Continuous Delivery — also known as CI/CD — plays a big role in the current day-to-day life of developers. Proponents of the CI/CD pipeline claim it greatly reduces development cycle times, and increases software quality reliability. Critics, however, claim and show that within the CI/CD deployment a multitude of problems can arise when the tooling is misused. An analysis of the development cycle, both manual and with a CI/CD pipeline, shows that CI/CD requires an initial time sacrifice to set up, but greatly reduces development cycle time compared to manually building, testing, and deploying. Users of the technology should be wary that they are sufficiently trained and able enough to set up a CI/CD pipeline to make sure no problems arise.
Introduction
During the project “Project Everyware”, I have been working on a significant part of the web-app and API. A significant part of the time that was spent on this project consisted of manually testing the application and API. This testing consisted of making sure all the API routes worked properly, that the code for both the API and the web-app compiled correctly, and that none of the existing codebase was conflicting with the new features. Manual testing is not only slow consuming, it can also become unreliable, as it greatly relies on the testers efforts to ensure full coverage of all affected elements of the code. In older software development teams, developers would write the code, and a group of people referred to as ‘IT Operations’ would build and test new features. On top of that, they are also responsible for successful deployment of the website or application, and making sure everything functions as expected. In recent developments, to combat long development cycles, the developer and IT operation roles have been merged and are now largely in the hands of the developers. The merging of the two practices is commonly referred to as DevOps, a combination of ‘Development’ and ‘Operations. DevOps uses something called the Continuous Integration/Continuous Delivery pipeline (hereinafter referred to as CI/CD) to quickly and automatically build, test, and deploy new features and code.
While modern software development teams make heavy use of DevOps in their day-to-day coding work, critics have noted that since DevOps is a relatively new practice, many DevOps integrations are plagued with challenges, barriers, and practices facing it’s adoption.
Integrating a CI/CD pipeline takes time and skill, and needs to be done in a correct manner to ensure the time spent on implementing is worth it. To decide whether it is worth it to use DevOps and construct a CI/CD pipeline to automate all building, test, and deploying of the project, this paper will focus on the related work that has already been done, and supply an analysis on my own experience with DevOps and the CI/CD pipeline.
Related works
In his research paper ‘Understanding and Improving Continuous Integration and Delivery Practice using Data from the Wild’, Massimiliano Di Penta writes that a CI/CD pipeline introduces numerous benefits in software development, including the claims that CI/CD increases software quality reliability, and decreases release cycle deployment time. However, Di Penta also noted that challenges, barriers, and bad practices plague CI/CD deployments. To demonstrate this, Di Penta references several articles that have shown the bad side of CI/CD deployments. (Di Penta, 2020)
In one of the articles referenced by Di Penta, ‘Use and Misuse of Continuous Integration Features: An Empirical Study of Projects That (Mis)Use Travis CI’, it’s author, Shane McIntosh, analyzes over nine thousand open-source project that make use of Travis CI, a common CI/CD tool. To quantify the misuse of Travis CI, McIntosh makes use of a tool called HANSEL, a piece of software that detects so-called ‘anti-patterns’. An anti-pattern is a structure or pattern within a piece of software, that despite initially appearing to be an appropriate and effective response to a problem, has more bad consequences than good ones. (Gamma, 1995) Using HANSEL, McIntosh findings report that 83% of the 9,312 open-source projects using Travis CI include anti-patterns in Travis CI deployment, suggesting developers generally do not have sufficient knowledge to correctly implement Travis CI. (McIntosh, 2018)
Analysis
According to both Di Penta and McIntosh, and almost all articles referenced in their work, Continuous Integration and Continuous Delivery increases software quality reliability and decreases release cycle time. However, they also state that insufficient knowledge of CI/CD implementation causes high numbers of misuse of CI/CD tooling.
To determine whether the upsides are worth the downsides, I have conducted several experiments relating to the effectivity and efficiency of building, testing, and deploying software. After implementing two small new features in the project, I tested the projected manually as well as through a CI/CD pipeline for each of the features. The results can be seen in the tables below.
The results might seem strange at first glance; for the first feature, manual seems to be quicker than automated. For the second feature, the results are reversed, and automated CI/CD is a lot quicker. The part that took the longest for feature 1 was the ‘Setup Time’. The setup time consisted of writing the CI/CD scripts in GitHub Actions. The scripts cover the whole deployment cycle: building the project, testing the code and running the separate unit tests, and deploying to Google Firebase. For the manual cycle, there was no setup time. Testing all the features manually took the longest, and it took up to 7 minutes to test every correct and incorrect way the feature could be used. Deploying to Firebase manually took almost three times as long compared to the automated cycle.
For the second feature I took the same approach. However, because the CI/CD scripts had already been written, the setup time was 0 minutes. This greatly reduced our total cycle time, as building, testing, and deploying only took 1 minute and 52 seconds. Our manual cycle however, clocked in at 14 minutes and 31 seconds, as testing took up a big chunk of our time.
Using this data we can show that while initial setup time for automated CI/CD is higher than when manually testing, every subsequent automated development cycle is up to seven times faster than it’s manual counterpart. However, in my own experience using CI/CD I have found that some errors, bugs, and other flaws can slip by a CI/CD script. These bugs require manual testing or a more thorough CI/CD pipeline to find. Besides that, I have also ran into problems when setting up a CI/CD pipeline, especially when it was relatively new. My own experiences and test results seem to support the conclusions that both Di Penta and McIntosh lay out in their articles.
Conclusions
Developers can benefit greatly from an automated CI/CD pipeline to assist them and relieve them of certain parts of building, testing, and deploying their software. While initial setup time of a CI/CD pipeline is marginally longer than doing it manually, the subsequent deployment cycles are significantly faster. However, developers should realize DevOps and CI/CD is a job that requires them to invest time in learning how to use their tooling, as misuse can cause anti-patterns to emerge.
Recommendations
The benefits of CI/CD and DevOps outweigh the negative effects it can produce. The time that is saved by having an automated CI/CD pipeline in place will greatly increase the time developers have to spend on coding and developing instead of building, testing, and deploying. However, developers and employers should take great care in making sure they — or their employees — are trained in DevOps before setting up infrastructure.
Bibliography
Di Penta, M. (2020). Understanding and Improving Continuous Integration and Delivery Practice using Data from the Wild. Proceedings of the 13th Innovations in Software Engineering Conference on Formerly known as India Software Engineering Conference, 1, 1. 10.1145/3385032.3385059
Gamma, E. (1995). Design Patterns. Pearson Education.
McIntosh, S. (2018). Use and Misuse of Continuous Integration Features: An Empirical Study of Projects That (Mis)Use Travis CI. IEEE, 46(1), 33–50. 10.1109/TSE.2018.2838131
