Visualization of Regression and Performance

Know when something went wrong

When you try a new software version, something may be wrong or slow. After updating OpenBSD you might see that something does not work as it used to be. To simplify debugging it is helpful to determine the point in time when the change was introduced and search for the relevant commit.

Making statements about performance is more difficult than for functional regressions. The requirements for measurements may change, new test programs are needed, test hardware is only available for a limited time. So it is not sufficient to store historic data on a daily basis. You want to change granularity or look back into the past.

For that purpose I have created a system that can create a performance history of the OpenBSD kernel. The kernel is compiled from certain CVS checkouts. That may sound easier than it is as the OpenBSD kernel is not self contained. It belongs to a base system, there may be incompatibilities with userland. The performance also depends on the compiler version that changes over an OpenBSD development cycle. My framework addresses this in an automated way.

An overview of all tests results is here: https://bluhm.genua.de/perform/results/perform.html

This is a follow up to my talk at BSDCan 2019. I will add new findings and conclusions which were detected during the last months.