commit ba407c2fbedc023e22bd4a9e8dccb3fefcee9d7e Author: Max Rottenkolber Date: Mon Nov 30 20:03:21 2015 +0100 Typos, capitalization, grammar, commas in “Continuous Integration for Snabb Switch”. diff --git a/blog/snabb-ci.mk2 b/blog/snabb-ci.mk2 index c42b78d..3f72029 100644 --- a/blog/snabb-ci.mk2 +++ b/blog/snabb-ci.mk2 @@ -2,7 +2,7 @@ For over year now, I have been working on [Snabb Switch](https://github.com/SnabbCo/snabbswitch), an open source software Ethernet networking stack. One of my ongoing responsibilities has been to devise, develop and maintain a _continuous integration_ (CI) -tools for the project. Wait, why would anyone spend time “developing” a +tool for the project. Wait, why would anyone spend time “developing” a new continuous integration system with _Jenkins_ and _Travis CI_ being around? @@ -18,8 +18,8 @@ around? are recommended to merge with. The release itself consists of commonly beneficial contributions pulled in from individual forks during the course of a month. This work flow enables Snabb Switch development to - take place in a loosely decentralized fashion: Contributors can do - _their thing_¹ and propose bug fixes and new features that they feel + take place in a loosely decentralized fashion: contributors can do + _their thing_,¹ and propose bug fixes and new features that they feel benefit a wider audience to be merged into upstream while receiving everyone else's contributions too. Snabb Switch development is coordinated on _GitHub_. @@ -28,13 +28,13 @@ around? in mind. Often, a fork will develop a Snabb Switch application that itself has no place in the upstream repository because its too specific to a certain problem or product.² These applications usually have their - own test suites and benchmarks in addition to the upstream suite as - well, and the testing infrastructure provided by Snabb GmbH supports that - use case too.³ + own test suites and benchmarks in addition to the upstream test suite as + well, and the testing infrastructure provided by Snabb GmbH supports + that use case too.³ Testing a high performance Ethernet networking stack requires high performance networking hardware. Further than that, a big part of our - test suite are _performance regressions tests_. Performance regression + test suite are _performance regression tests_. Performance regression tests are benchmarks in which Snabb Switch may not score below a certain threshold. In order to run a representative benchmark you need to run it in a deterministic environment. Specifically, we need a dedicated @@ -46,19 +46,19 @@ around? with the right hardware as well as a broad range of different hardware and software platforms. Some parts of the test suite even spin up virtual machines using _QEMU_ to test virtualized networking. Different - CPU models, different Operating Systems and different user land software + CPU models, different operating systems and different user land software versions: all these are factors that can impact the performance of Snabb Switch and thus need to be covered by the test infrastructure. Another important factor to effectively cooperating on a peer developed software system is _reproducibility_. Whenever somebody produces a test - result for review it needs to be reproducible so others can confirm the - result. Traditionally, truthfully reproducing a given test result, which - may depend on the specific hardware and/or software in use, can be a - tedious and time consuming task. Ideally, that process should be simple - and fast. Especially for Snabb Switch, which supports a broad range of - different platforms, there is demand for reducing the overhead inherent - to reproducing different test environments. + result for review it needs to be reproducible so that others can confirm + the result. Traditionally, truthfully reproducing a given test result, + which may depend on the specific hardware and/or software in use, can be + a tedious and time consuming task. Ideally, that process should be + simple and fast. Especially for Snabb Switch, which supports a broad + range of different platforms, there is demand for reducing the overhead + inherent to reproducing different test environments. + 1. The different parties working on Snabb Switch indeed have different goals, needs and specialties. @@ -81,7 +81,7 @@ around? possible solutions just to throw most of them away afterward. _Big dependencies_ get in the way of that process: first you need to - learn how to use a complex system and then it will be hard to adapt to + learn how to use a complex system, and then it will be hard to adapt to new requirements. We prefer a 200 line _Bash_ script that does exactly what we _think_ we want over a complex Java application that probably only does a subset of what we _think_ we want. Small, orthogonal @@ -101,8 +101,8 @@ around? _selftest_ which is invoked to run a module's unit tests. (This is a convention imposed by Snabb Switch.) + Additionally, any file named _selftest.sh_ is considered a test and - run as such. This enables “free form” testing entailing things such as - spinning up virtual machines. + run as such. This mechanism allows for “free form” testing that + entails things such as spinning up virtual machines. + Some tests can or must be configured via _environment variables_ and the result of a test (success, failure or “skipped”) is indicated by its return value. @@ -137,7 +137,7 @@ around? lines, _Bash_) is provided to seamlessly run the test suite of your Snabb Switch working tree within such a container. That way, test results depending on a specific software environment can be quickly - reproduced on any machine that runs _Docker_. The creation of our tests + reproduced on any machine that runs _Docker_. The creation of our test environment and the containing _Docker_ image is another complex hassle that I only want to touch briefly. For instance, we need to repeatably build virtual machine images containing specific versions of some commit 36ec80cc11d1c75bf81df2586d3637869d0f54c4 Author: Max Rottenkolber Date: Mon Nov 30 13:28:39 2015 +0100 Add screen shot of GitHub statuses to “Continuous Integration for Snabb Switch”. diff --git a/blog/snabb-ci.mk2 b/blog/snabb-ci.mk2 index 63e9f8b..c42b78d 100644 --- a/blog/snabb-ci.mk2 +++ b/blog/snabb-ci.mk2 @@ -144,6 +144,9 @@ around? software. For that, we [make use of Docker](build-vm-image-with-docker.html) too. + #media# + snabb-ci.png + Finally, there is _SnabbBot_ (200 lines, _Bash_): Snabb Switch's own tailor made continuous integration service. _SnabbBot_ is a _cron job_ that launches automated test runs for incoming contributions: commit e03a8def8b4cbf3f04b9936551c8ad51af927ba8 Author: Max Rottenkolber Date: Sun Nov 29 23:40:29 2015 +0100 Add “Continuous Integration for Snabb Switch”. diff --git a/blog/snabb-ci.mk2 b/blog/snabb-ci.mk2 new file mode 100644 index 0000000..63e9f8b --- /dev/null +++ b/blog/snabb-ci.mk2 @@ -0,0 +1,177 @@ +For over year now, I have been working on +[Snabb Switch](https://github.com/SnabbCo/snabbswitch), an open source +software Ethernet networking stack. One of my ongoing responsibilities +has been to devise, develop and maintain a _continuous integration_ (CI) +tools for the project. Wait, why would anyone spend time “developing” a +new continuous integration system with _Jenkins_ and _Travis CI_ being +around? + +< Snabb Switch is Different + + Snabb Switch is being developed as an open source project by a diverse + community of interested parties. The “upstream” repository, through + which the consensus of development is mediated, is maintained by + [Snabb GmbH](http://snabb.co). Other interested parties, be it + individuals or companies, maintain their own “forks” of Snabb Switch in + which they develop features specific to their interest. Every month, + upstream releases a new version of Snabb Switch, with which the forks + are recommended to merge with. The release itself consists of commonly + beneficial contributions pulled in from individual forks during the + course of a month. This work flow enables Snabb Switch development to + take place in a loosely decentralized fashion: Contributors can do + _their thing_¹ and propose bug fixes and new features that they feel + benefit a wider audience to be merged into upstream while receiving + everyone else's contributions too. Snabb Switch development is + coordinated on _GitHub_. + + Our testing infrastructure is built with this decentralized organization + in mind. Often, a fork will develop a Snabb Switch application that + itself has no place in the upstream repository because its too specific + to a certain problem or product.² These applications usually have their + own test suites and benchmarks in addition to the upstream suite as + well, and the testing infrastructure provided by Snabb GmbH supports that + use case too.³ + + Testing a high performance Ethernet networking stack requires high + performance networking hardware. Further than that, a big part of our + test suite are _performance regressions tests_. Performance regression + tests are benchmarks in which Snabb Switch may not score below a certain + threshold. In order to run a representative benchmark you need to run it + in a deterministic environment. Specifically, we need a dedicated + “Network Interface Card” (NIC) and an idle CPU core on the same [NUMA + node](https://en.wikipedia.org/wiki/Non-uniform_memory_access). These + requirements already exclude us from using most hosted solutions. + + Even further, we support multiple NIC drivers that have to be tested + with the right hardware as well as a broad range of different hardware + and software platforms. Some parts of the test suite even spin up + virtual machines using _QEMU_ to test virtualized networking. Different + CPU models, different Operating Systems and different user land software + versions: all these are factors that can impact the performance of + Snabb Switch and thus need to be covered by the test infrastructure. + + Another important factor to effectively cooperating on a peer developed + software system is _reproducibility_. Whenever somebody produces a test + result for review it needs to be reproducible so others can confirm the + result. Traditionally, truthfully reproducing a given test result, which + may depend on the specific hardware and/or software in use, can be a + tedious and time consuming task. Ideally, that process should be simple + and fast. Especially for Snabb Switch, which supports a broad range of + different platforms, there is demand for reducing the overhead inherent + to reproducing different test environments. + + + 1. The different parties working on Snabb Switch indeed have different + goals, needs and specialties. + + 2. There might even be active closed source forks tracking upstream + development. Hard to tell. + + 3. Snabb GmbH sponsors the development of our custom continuous + integration tools as well as the “Snabb Lab”: a flock of + development machines that sport the right networking hardware, + accessible to anyone who wants to tinker with Snabb Switch. + +> + +< Rewritable Software + + The Snabb Switch community encourages its members to produce [rewritable + software](https://github.com/lukego/blog/issues/12). The term + _rewritable software_ describes the ideal of a simple and easy to + understand program distilled from prolonged experimentation and + iterative design. Distilling simple programs requires us to try out many + possible solutions just to throw most of them away afterward. + + _Big dependencies_ get in the way of that process: first you need to + learn how to use a complex system and then it will be hard to adapt to + new requirements. We prefer a 200 line _Bash_ script that does exactly + what we _think_ we want over a complex Java application that probably + only does a subset of what we _think_ we want. Small, orthogonal + programs on the other hand fuel the process. In fact, we reuse _tons_ of + Unix command line tools in our test suite: be it _iperf_ to benchmark + throughput or _awk_ to compute the median and standard deviation of a + benchmark's results. + +> + +< Continuous Integration as Part of the System + + At the core of the Snabb Switch test suite is a _Makefile_. It has a + target _test_ that runs “all the tests”: + + + Every Lua module can optionally provide a function aptly named + _selftest_ which is invoked to run a module's unit tests. (This is a + convention imposed by Snabb Switch.) + + Additionally, any file named _selftest.sh_ is considered a test and + run as such. This enables “free form” testing entailing things such as + spinning up virtual machines. + + Some tests can or must be configured via _environment variables_ and + the result of a test (success, failure or “skipped”) is indicated by + its return value. + + Some of the individual tests are small and independent, some require a + specific piece of hardware, some even require a couple gigabytes of + assets such as virtual machine images to run. A good example of a + complex test script is the _Snabb NFV_ test suite: it comes with its own + abstraction layer (150 lines, _Bash_) to facilitate spinning up virtual + machines connected to each other via _Snabb NFV_ and running commands on + the individual guests such as _ping_, _iperf_ and _nc_. This abstraction + is used by some benchmarks as well. + + A Snabb Switch benchmark is just a shell script that prints a floating + point number indicating a score. These scripts can again be arbitrarily + complex. The target _make benchmarks_ (20 lines, _Bash_) runs all + benchmarks and prints their names followed by their score. The sample + size of _make benchmarks_ can be configured through an environment + variable in order to compute median and standard deviation values. + + #code Possible output of _make benchmarks_: The _basic1-100e6_ benchmark + ran five times with a median score of 17.9 and a standard deviation of + 0.352136.# + $ SNABB_PERF_SAMPLESIZE=5 make benchmarks + basic1-100e6 17.9 0.352136 + [...] + # + + As for the reproducibility of test results, we maintain _Docker_ images + publicly hosted on _DockerHub_ that contain all the external, variable + assets required to run the full Snabb Switch test suite. A script (20 + lines, _Bash_) is provided to seamlessly run the test suite of your + Snabb Switch working tree within such a container. That way, test + results depending on a specific software environment can be quickly + reproduced on any machine that runs _Docker_. The creation of our tests + environment and the containing _Docker_ image is another complex hassle + that I only want to touch briefly. For instance, we need to repeatably + build virtual machine images containing specific versions of some + software. For that, we [make use of + Docker](build-vm-image-with-docker.html) too. + + Finally, there is _SnabbBot_ (200 lines, _Bash_): Snabb Switch's own + tailor made continuous integration service. _SnabbBot_ is a _cron job_ + that launches automated test runs for incoming contributions: + + + It uses the [GitHub API](https://developer.github.com/) to fetch new + _pull requests_ and post a status with a log attached, which is again + posted as a _gist_. + + It merges the _pull request_ branch with the _master_ branch. + + It uses _Docker_ to build and run the test suite within a container. + + It uses the _Makefile_ target _test_ to run the test suite and, on + failure, attaches the test output. + + It uses the _Makefile_ target _benchmarks_ to compare performance + between the _pull request_ branch and the _master_ branch and detects + performance regressions. + + Multiple instances of _SnabbBot_ can provide results for a single + Snabb Switch repository using different test environments and it is + [trivial](https://github.com/SnabbCo/snabbswitch/blob/master/src/doc/testing.md#running-a-snabbbot-ci-instance) + to run _SnabbBot_ on a forked repository. + + A lesson learned along the way: continuous integration should not be an + external tool, but part of the program it tests in the same way a + program's test suite is part of that program. Every piece of the + continuous integration puzzle is part of the mainline Snabb Switch + source tree (with exception of the variable test assets and more + importantly, the scripts that build them). Continuous integration as + part of the system. Thanks for reading! + +> + +_Interested in Snabb Switch? I offer Snabb Switch consulting! Feel free +to contact me at service@mr.gy._ \ No newline at end of file