Jenkins is a fabulous piece of software program. As an execution and automation engine, it is among the finest you are going to discover. Jenkins serves as a key part in numerous steady integration (CI) techniques, and this can be a testomony to the worth of what its group has constructed over time. But that is what it’s—a part. Jenkins will not be a CI system itself; it simply runs issues for you. It does that basically effectively and has a wide range of built-ins and a vibrant ecosystem of plugins that will help you inform it what to run, when, and the place.
CI is, on the most basic stage, about integrating the work of a number of software program improvement streams right into a coherent complete with as a lot frequency and as little friction as doable. Jenkins, by itself, does not find out about your supply code or how you can merge it collectively, nor does it know how you can give constructive suggestions to you and your colleagues. You can, in fact, glue it along with different software program that may carry out these actions, and that is what number of CI techniques incorporate Jenkins.
It’s what we did for OpenStack, too, at the very least at first.
If it isn’t examined, it is damaged
In 2010, an open supply group of initiatives referred to as OpenStack was forming. Some of the builders introduced in to help with the collaboration infrastructure additionally labored on a free database mission referred to as Drizzle, and a key philosophy inside that group was the concept “if it’s not tested, it’s broken.” So OpenStack, on day one, required all proposed modifications of its software program to be reviewed and examined for regressions earlier than they may very well be accredited to merge into the trunk of any supply code repositories. To do that, Hudson (which later forked to kind the Jenkins mission) was configured to run exams exercising each change.
A plugin was put in to interface with the Gerrit code assessment system, mechanically triggering jobs when new modifications had been proposed and reporting again with assessment feedback indicating whether or not they succeeded or failed. This might sound rudimentary by in the present day’s requirements, however on the time, it was a revolutionary development for an open supply collaboration. No developer on OpenStack was particular within the eyes of CI, and everybody’s modifications needed to cross this rising battery of exams earlier than they might merge—an idea the mission referred to as “project gating.”
There was, nonetheless, an rising flaw with this gating concept: To assure two unrelated modifications did not alter a chunk of software program in functionally incompatible methods, they needed to be examined one after the other in sequence earlier than they might merge. OpenStack was sophisticated to put in and check, even again then, and shortly grew in reputation. The rising quantity of developer contributions coupled with rising check protection meant that, throughout busy durations, there was merely not sufficient time to check each change that handed assessment. Some longer-running jobs took almost an hour to finish, so the higher sure for what might get by means of the gate was roughly two dozen modifications in a day. The ensuing merge backlog confirmed a brand new resolution was required.
Enter Zuul
During an OpenStack CI assembly in May 2012, one of many CI staff members, James Blair, announced that he’d “been working on speculative execution of Jenkins jobs.” Speculative execution is an optimization mostly discovered within the pipelines of contemporary microprocessors. Much just like the analogy with processor hardware, the idea was that by optimistically predicting optimistic gating outcomes for modifications lately accredited however that had not but accomplished their exams, subsequently accredited modifications may very well be examined concurrently after which conditionally merged so long as their predecessors additionally handed exams and merged. James stated he had a reputation for this clever scheduler: Zuul.
Within this timeframe, challenges from making an attempt to carry out higher revision management for Jenkins’ XML job configuration led to the creation of the human-readable YAML-based Jenkins Job Builder templating engine. Limited success with the JClouds plugin for Jenkins and cumbersome makes an attempt to make use of jobs for refreshing cloud pictures of single-use Jenkins slaves ended with the creation of the Nodepool service. Limited log-storage capabilities resulted within the staff including separate exterior options for organizing, serving, and indexing job logs and assuming maintainership of an deserted safe copy protocol (SCP) plugin changing the less-secure FTP possibility that Jenkins offered out of the field. The OpenStack infrastructure staff was slowly constructing a fleet of companies and utilities round Jenkins however started to bump up towards a efficiency limitation.
Multiplying Jenkins
By mid-2013, Nodepool was always recycling as many as 100 digital machines registered with Jenkins as slaves, however this was now not sufficient to maintain up with the rising workload. Thread competition for international locks in Jenkins thwarted all makes an attempt to push previous this threshold, regardless of how a lot processor energy and reminiscence was thrown on the grasp server. The mission had gives to donate extra capability for Jenkins slaves to assist relieve the frequent job backlog, however this could require a further Jenkins grasp. The environment friendly division of labor between a number of masters wanted a brand new channel of communication for dispatch and coordination of jobs. Zuul’s maintainers recognized the Gearman job server protocol as an excellent match, in order that they outfitted Zuul with a brand new geard service and prolonged Jenkins with a customized Gearman shopper plugin.
Now that jobs had been unfold throughout a rising meeting of Jenkins masters, there was now not any single dashboard with an entire view of job exercise and outcomes. In order to facilitate this new multi-master world, Zuul grew its personal standing API and WebUI, in addition to a characteristic to emit metrics by means of the StatsD protocol. Over the following few years, Zuul steadily subsumed extra of the CI options its customers relied on, whereas Jenkins’ place within the system waned accordingly, and it was changing into a legal responsibility. OpenStack made an early option to standardize on the Python programming language; this was mirrored in Zuul’s improvement, but Jenkins and its plugins had been carried out in Java. Zuul’s configuration was maintained in the identical YAML serialization format that OpenStack used to template its personal Jenkins jobs, whereas Jenkins saved every little thing in baroque XML. These variations sophisticated ongoing upkeep and led to an unnecessarily steep studying curve for brand new directors from associated communities that had began making an attempt to run Zuuls.
The time was proper for one more revolution.
The rise of Ansible
In early 2016, Zuul’s maintainers launched into an formidable year-long overhaul of their rising fleet of companies with the objective of eliminating Jenkins from the general system design. By this time, Jenkins was serving solely as a conduit for working jobs consisting principally of shell scripts on slave nodes over SSH, offering real-time streaming of job output and copying ensuing artifacts to longer-term storage. Ansible was discovered to be a fantastic match for that first want; purpose-built to run instructions remotely over SSH, it was written in Python, identical to Zuul, and likewise used YAML to outline its duties. It even had built-in modules for options the staff had beforehand carried out as bespoke Jenkins plugins. Ansible offered true multi-node help proper out of the field, so the identical playbooks may very well be used for each simulating and performing advanced manufacturing deployments. An ever-expanding ecosystem of third-party modules stuffed in any gaps, in a lot the identical means because the Jenkins group’s plugins had earlier than.
A brand new Zuul executor service stuffed the prior position of the Jenkins grasp: it acted on pending requests within the scheduler’s geard, dispatched them by way of Ansible to ephemeral servers managed by Nodepool, then collected outcomes and artifacts for publication. It additionally uncovered in-progress construct output over the basic RFC 742 Name/Finger protocol, streamed in actual time from an extension of Ansible’s command output module. Once it was now not essential to restrict jobs to what Jenkins’ parser might comprehend, Zuul was free to develop new options like distributed in-repository job definitions, shareable between initiatives with inheritance and safe dealing with of secrets and techniques, in addition to the flexibility to test-drive proposed modifications for the roles themselves. Jenkins served its goal admirably, however at the very least for Zuul, its usefulness was lastly at an finish.
Testing the longer term
Zuul’s group likes to say that it “tests the future” by means of its novel software of speculative execution. Gone are the harrowing days of questioning whether or not the advance you need to make to an present job will render it non-functional as soon as it is utilized in manufacturing. Overloaded assessment groups for an enormous central job repository are a factor of the previous. Jobs are handled as part of the software program and shipped proper alongside the remainder of the supply code, benefiting from Zuul’s different options like cross-repository dependencies in order that your change to a part of a job in a single mission might be exercised with a proposed job change in one other mission. It will even remark in your job modifications, highlighting particular traces with syntax issues as if it had been one other code reviewer providing you with recommendation.
These had been options Zuul solely dreamed of earlier than, however which required freedom from Jenkins in order that it might take job parsing into its personal arms. This is the way forward for CI, and Zuul’s customers reside it.
As of early 2019, the OpenStack Foundation acknowledged Zuul as an unbiased, brazenly ruled mission with its personal id and flourishing group. If you are into open supply CI, think about looking. Development on the following evolution of Zuul is all the time underway, and also you’re welcome to assist. Find out extra on Zuul’s website.