Continuous Discussion: Agile, DevOps and Continuous Delivery

Two days ago I participated in a online panel discussing Agile, DevOps and Continuous Delivery. The panel consisted of 11 panelists and was organized by Electric Cloud. Topics discussed include:

  • What are/were your big Agile obstacles?
  • What does DevOps mean to you?
  • What is Continous Delivery? What does it take to get there?

You can watch a recording of the session at Electric Cloud's blog. There is also information about the next online panel that will be arranged.


Java 8 Functional Interface Naming Guide

People have been complaining about the naming of Java 8's functional interfaces in the java.util.function package. Depending on the types of arguments and return values a function may instead be called a consumer, supplier, predicate or operator, which is further complicated by two-argument functions. The number of interfaces explodes because of specialized interfaces for some primitive types. In total there are 43 interfaces in that package. Compare this to Scala where just three interfaces (with compiler optimizations) cover all the same use cases: Function0, Function1 and Function2.

To make some sense out of this all, I wrote a program that generates the interface names based on the function's argument types and return type. The names it generates match those in the Java library. For this article I also made visualization which you can see below.


Click to make it bigger.


The interface name is determined by the following algorithm. The names are described as regular expressions; the generic interfaces have the shortest names, but specialized interfaces for functions taking or returning primitives include the primitive types in the interface name.

  1. Does it return void?
    • Arity 0: Runnable
    • Arity 1: (|Int|Long|Double)Consumer
    • Arity 2:
      • Both arguments are generic: BiConsumer
      • First argument is generic: Obj(Int|Long|Double)Consumer
  2. Does it take no arguments?
    • Arity 0: (|Int|Long|Double|Boolean)Supplier
  3. Does it return boolean?
    • Arity 1: (|Int|Long|Double)Predicate
    • Arity 2: BiPredicate
  4. Do all arguments have the same type as the return value?
    • Arity 1: (|Int|Long|Double)UnaryOperator
    • Arity 2: (|Int|Long|Double)BinaryOperator
  5. Otherwise:
    • Arity 1: (|Int|Long|Double)(|ToInt|ToLong|ToDouble)Function
    • Arity 2: (|ToInt|ToLong|ToDouble)Function

The method names follow. When the return type is a primitive, the method name is a bit longer (this is apparently because the JVM supports method overloading also based on the method return type, but the Java language does not).

  • Runnable: run
  • Consumers: accept
  • Suppliers: get(|AsInt|AsLong|AsDouble|AsBoolean)
  • Predicates: test
  • Functions and operators: apply(|AsInt|AsLong|AsDouble)

I hope that helps some of you in remembering what interface to look for when browsing the API.


Lambda Expressions Backported to Java 7, 6 and 5

Do you want to use lambda expressions already today, but you are forced to use Java and a stable JRE in production? Now that's possible with Retrolambda, which will take bytecode compiled with Java 8 and convert it to run on Java 7, 6 and 5 runtimes, letting you use lambda expressions and method references on those platforms. It won't give you the improved Java 8 Collections API, but fortunately there are multiple alternative libraries which will benefit from lambda expressions.

Behind the Scenes

A couple of days ago in a café it popped into my head to find out whether somebody had made this already, but after speaking into the air, I did it myself over a weekend.

The original plan of copying the classes from OpenJDK didn't work (LambdaMetafactory depends on some package-private classes and would have required modifications), but I figured out a better way to do it without additional runtime dependencies.

Retrolambda uses a Java agent to find out what bytecode LambdaMetafactory generates dynamically, and saves it as class files, after which it replaces the invokedynamic instructions to instantiate those classes directly. It also changes some private synthetic methods to be package-private, so that normal bytecode can access them without method handles.

After the conversion you'll have just a bunch of normal .class files - but with less typing.

P.S. If you hear about experiences of using Retrolambda for Android development, please leave a comment.


Refactoring Primitive Obsession

Primitive Obsession means using a programming language's generic type instead of an application-specific domain object. Some examples are using an integer for an ID, a string for an address, a list for an address book etc. Others have explained why to fix it - this article is about how to fix it.

You can see an example of refactoring Primitive Obsession in James Shore's Let's Play TDD episodes 13-18. For a quick overview, you may watch episode #14 at 10-12 min and episode #15 at 0-3 min, to see him plugging in the TaxRate class.

The sooner the Primitive Obsession is fixed, the easier it is. In the above videos it takes just a couple of minutes to plug in the TaxRate class, but the Dollars class takes over half an hour. James does the code changes manually, without automated refactorings. For a big project with rampant Primitive Obsession it will easily take many hours, even days, to fix the problem of a missing core domain type.

Here I'm presenting some tips of using fully automated refactorings to solve Primitive Obsession. I'm using IntelliJ IDEA's Java refactorings, but the ideas should, to some extent, be applicable also to IDEs with inferior refactoring support.

The Example

Let's assume that we have a project that uses lots of "thingies" which are saved in a database. The thingies each have an ID that at the moment is just an integer. To avoid the thingy IDs getting mixed with other kinds of IDs, we create the following value object:

public final class ThingyId {

    private final int id;

    public ThingyId(int id) {
        this.id = id;

    public int toInt() {
        return id;

    public boolean equals(Object obj) {
        if (!(obj instanceof ThingyId)) {
            return false;
        ThingyId that = (ThingyId) obj;
        return this.id == that.id;

    public int hashCode() {
        return id;

    public String toString() {
        return getClass().getSimpleName() + "(" + id + ")";

Creating such a class is easy, but putting it to use is not so when the primitive ID is used in a couple of hundred places…

Starting Small

Refactoring Primitive Obsession is quite mechanical, but because it requires cascading changes, it's very easy to mess things up. So it's best to start small and proceed in small steps.

It makes sense to start from a central place from where the change can be propagated to the whole application. For example by starting to use ThingyId inside this one class, without changing its public interface:

(Cannot see the video? Watch it as GIF)

This example refactoring had to be done manually, because the field was mutable (we must update all reads and writes to the field in one step), but the following refactorings can be done with the help of automatic refactorings.

Pushing Arguments Out

When there is a method which wraps one of its arguments into ThingyId, we can propagate it by pushing the act of wrapping outside the method. In IntelliJ IDEA this can be done with the Extract Parameter (Ctrl+Alt+P) refactoring:

(Cannot see the video? Watch it as GIF)

Pushing Return Values

When there is a method which unwraps its return value from ThingyId to int, we can propagate the unwrapping outside the method. There is no built-in refactoring for that, but it can be accomplished by combining Extract Method (Ctrl+Alt+M) and Inline (Ctrl+Alt+N).

First extract a method that does the same as the old method, but does not unwrap ThingyId. Then inline the original method and rename the new method to be the same as the original method.

(Cannot see the video? Watch it as GIF)

Pushing Return Values of Interface Methods

A variation of the previous refactoring is required when the method is part of an interface. IntelliJ IDEA 12 does not support inlining abstract methods (I would like it to ask that which of the implementations to inline), but since IDEA can refactor code that doesn't compile, we can copy and paste the implementation into the interface and then inline it:

(Cannot see the video? Watch it as GIF)

Pushing Arguments In

Instead of trying to refactor a method's arguments from the method caller's side, it's better to go inside the method and use Extract Parameter (Ctrl+Alt+P) as described earlier. Likewise for a method's return value. This leaves us with some redundant code, as can be seen in this example. We'll handle that next.

(Cannot see the video? Watch it as GIF)

Removing Redundancy

By following the above tips you will probably end up with some redundant wrapping and unwrapping such as new ThingyId(thingyId.toInt()) which is the same as thingyId. Changing one such thing manually would be easy, but the problem is that there are potentially tens or hundreds of places to change. In IntelliJ IDEA those can be fixed with one command: Replace Structurally (Ctrl+Shift+M).

In the following example we use the search template "new ThingyId($x$.toInt())" and replacement template "$x$". For extra type safety, the $x$ variable can be defined (under the Edit Variables menu) to be an expression of type ThingyId.

(Cannot see the video? Watch it as GIF)

Updating Constants

When there are constants (or final fields) of the old type, as is common in tests, those can be updated by extracting a new constant of the new ThingyId type, redefining the old constant to be an unwrapping of the new constant, and finally inlining the old constant:

(Cannot see the video? Watch it as GIF)

Finding the Loose Ends

The aforementioned refactorings must be repeated many times until the whole codebase has been migrated. To find out what refactoring to do next, search for the usages of the new type's constructor and its unwrapping method (e.g. ThingyId.toInt()). Use an appropriate refactoring to push that usage one step further. Repeat until all the usages are at the edges of the application (e.g. saving ThingyId to database) and cannot be pushed any further.

And as always, run all your tests after every step. If the tests fail and you cannot fix them within one minute, you're about to enter Refactoring Hell and it's the fastest that you revert your changes to the last time when all tests passed. Reverting is the easiest with IntelliJ IDEA's Local History which shows every time that you ran your tests and whether they passed or failed, letting you revert all your files to that time. The other option is to commit frequently, after every successful change (preferably rebased before pushing), and revert using git reset --hard.

This article was originally published at my company's blog.


Faster JUnit Tests with Jumi Test Runner and Class Loader Caching

Jumi test runner version 0.4 adds JUnit backward compatibility, so that Jumi can run existing JUnit tests out-of-the-box. All testing frameworks that can be run with JUnit may also be run on Jumi, so there is a low barrier of entry.

One advantage for JUnit users to try out Jumi is faster test execution. In this article I'm showing some benchmark results of the current (non-optimized) Jumi version, and estimates of how much more faster it will at least get once I implement some performance optimizations.

Class Loader Overhead

According to earlier experiences of parallelizing JUnit, "for a fairly optimized unit-test set, expect little or no gain - maybe 15-20%." That's because of the overhead of class loading. With Java 7 the class loaders are at least parallel capable, but I'm still expecting it to affect the test run times considerably.

That's why in this benchmark I'm using a project that has exactly such CPU-bound unit tests. It'll be much more interesting than looking at slow or IO-bound integration tests that scale much more easily to multiple CPU cores. ;)

Benchmark Setup

As a test subject I'm using the unit tests from Dimdwarf's core module (mixed Java and Scala). It has over 800 unit tests, they all are CPU-bound and take just a few of seconds to run. Over half of the tests have been written using JDave, a Java testing framework that runs on JUnit. The rest have been written using Specsy, a testing framework for Java, Scala, Groovy and with little effort any other JVM based language. Specsy 1 ran on JUnit, but Specsy 2 runs on Jumi, which allows test method level parallelism and solves a bunch of issues Specsy had with JUnit's limited execution model. Running JUnit tests on Jumi is limited to test class level parallelism (until JUnit itself implements Jumi support).

All measurements were run on Core 2 Quad Q6600 @ 3.0 GHz, 4 GB RAM, jdk1.7.0_07 64bit, Windows 7. The measurements were repeated 11 times and the median run times are reported. The program versions used were: Jumi 0.4.317, JUnit 4.8.2, IntelliJ IDEA 12.0.3 UE, Maven Surefire Plugin 2.13.

For those measurements which were started from IntelliJ IDEA, the Java compiler and code coverage were disabled to avoid their latency and overhead. The measurement was started when IDEA's Run Tests button was clicked, and stopped when IDEA showed all tests finished. The time was measured at 1/30 second accuracy using a screen recorder (that recorded just a small screen area - barely noticable on CPU usage).

For those measurements which were started from Maven, the time was measured starting from when the text "maven-surefire-plugin:2.13:test" shows up, until the "Results" line shows up. The time was measured using a screen recorder, same as above.

For the synthetic Jumi benchmarks which estimate future optimizations, the time was measured using System.currentTimeMillis() calls inside the Jumi test runner daemon process. Also Jumi was modified to run the test suite inside the same JVM multiple times (because I haven't yet implemented connecting to an existing test runner daemon process). It's safe to assume that the inter-process communication latency is much smaller than 100 ms, so the results should be fairly accurate.

The Results

Click to see them bigger.

Benchmark 1: JUnit, IDEA-JUnit, 1 thread

As a baseline the tests were run with the JUnit test runner from within IDEA. JUnit doesn't support parallel execution, so this test run was single threaded. The result was 5.8 seconds.

Benchmark 2: JUnit, Maven Surefire, 1 thread

As another baseline the Maven Surefire Plugin was used, which gave 5.0 seconds. Surprisingly Maven was much faster than IDEA's JUnit integration. This is probably due to the initialization that IDEA does in the beginning of a test run, presumably to discover the test classes and create a list of the tests to be run, or then related to the real-time test reporting.

Though Surefire supports running tests in parallel, I wasn't able to make it work - it threw a NullPointerException inside the Surefire plugin, probably due to an incompatibility with the Specsy 1 testing framework. Since JUnit was not originally designed to run tests in parallel and JUnit is very relaxed in what kinds of events it accepts from testing frameworks, this kinds of incompatibilities are to be expected.

Benchmark 3: Jumi, IDEA-JUnit, 1 thread

This was run with the Jumi 0.4 test runner, but since it doesn't yet have IDE integration, the test run was bootstrapped from a JUnit test which was started using IDEA. So it has the overhead of one extra JVM startup and IDEA's test initialization.

The result was one second slower than running JUnit tests directly with IDEA. Based on my experiences, a bit over half a second of that is due to the time it takes to start a second JVM process for the Jumi daemon. The rest is probably due to the class loading on the Jumi launcher side - for example at the moment it uses for communication Netty, which is quite a big library (about 0.5 MB of class files), so loading its classes takes hundreds of milliseconds.

Benchmark 4: Jumi, IDEA-JUnit, 4 threads

As expected, running on multiple threads does not make unit tests much faster. Adding threads takes Jumi from 6.8 seconds to 5.1 seconds, only 25% less time, barely cancelling out the overhead of the extra JVM creation.

As we'll see, the majority of the time is spent in class loading. Also this Jumi version has not yet been optimized at all, so the class loading overhead is probably even more severe than it needs to be (one idea I have is to run in parallel threads tests that use a different subset of classes - that way they shouldn't be blocked on loading the same classes that much).

Benchmark 5: Jumi, IDE integration (estimated), 1 thread

This benchmark measures how fast Jumi would be with IDE integration. This was measured around the code that launches Jumi and shows the test results, as it would be done by an IDE. Since this code would run in the same process as the Java IDE, the measurement was done in a loop inside one JVM instance, to eliminate the overhead of class loading on the Jumi launcher side and to let the JIT compiler warm up.

This gives about the same results as running JUnit tests directly in IDEA. Though it's not as fast as Surefire, maybe because Surefire's test reporting is more minimal, or because Jumi's daemon side has more class loading overhead for its own internal classes (as mentioned abobe when discussing benchmark 3).

Benchmark 6: Jumi, IDE integration (estimated), 4 threads

The absolute speedup over 1 thread is about the same as before (under 2 seconds), but with the JVM startup overhead away we are now on the winning side.

Benchmark 7: Jumi, persistent daemon process (estimated), 1 thread

Jumi's daemon process, which runs the tests, will eventually be reused for multiple test suite runs. The benefits of that feature were estimated by modifying Jumi to run the same tests multiple times in the same JVM.

This avoids the JVM startup overhead completely on subsequent test suite runs, and the JIT compiler starts kicking in for Jumi's internal classes (the first suite run is about 1 second slower). At 4.5 seconds we are now better than JUnit also in single-threaded performance.

Benchmark 8: Jumi, persistent daemon process (estimated), 4 threads

With 4 threads we see the same about 2 second speedup over a single thread as before. The tests are still dominated by the same amount of class loader overhead as before, but now we are anyways down by 50%, or 2× faster than JUnit.

Benchmark 9: Jumi, class loader caching of dependencies (estimated), 1 thread

For this benchmark Jumi was modified further to reuse the class loader that loads all libraries used by the system under test. In this particular project this includes big libraries such as the Scala standard library, Google Guice, Apache MINA and CGLIB. The project's own production and test classes, as well as some testing libraries (that didn't work well with multiple class loaders), were not cached, but their class loader was re-created for each test suite run.

We see a 40% improvement over just the persistent daemon process, the same speedup as if we had used multiple threads. The JIT compiler starts now kicking in, so that the full speed is reached only on the third or fourth test suite run (1st run 5.2s, 2nd run 3.1s, 3rd run 3.0s, 4th run 2.7s) after it has had some time to optimize the library dependencies' code.

Benchmark 10: Jumi, class loader caching of dependencies (estimated), 4 threads

With 4 threads we are down to 1.4 seconds, an improvement of 75%, already 4× faster than JUnit!

Benchmark 11: Jumi, class loader caching of all classes (estimated), 1 thread

This is an estimate of the ideal situation of having no class loader overhead. In this benchmark Jumi was modified to create just a single class loader for both the dependencies and application classes, and then reuse that for all test suite runs. After the first run there is no more class loading to be done, and on the third or fourth run the JIT compiler has optimized it.

We see that under ideal circumstances, it takes only 1.6 seconds to run all the tests single-threadedly. It's arguable whether the class loading overhead can be eliminated this much with techniques such as reloading classes without hurting reliability.

Benchmark 12: Jumi, class loader caching of all classes (estimated), 4 threads

With 4 threads we get down to 0.9 seconds, which is about 50% less time than the single-threaded benchmark. The ideal speedup on 4 cores would have been 75%, down to 0.4 seconds. This shows us that there is some contention that needs to be optimized inside Jumi. One possible area of improvement is the message queues - now all test threads write to the same queue, which violates the single writer principle, and also it doesn't take advantage of batching.


In this benchmark we only looked at speeding up a fast unit test suite, which are notoriously hard to speed up on the JVM. Slow integration tests should get much better speed improvements when run on multiple CPU cores. In the Jumi wiki there are some tips on how to make integrations tests isolated, so that they can be run in parallel.

Jumi's JUnit compatibility gives the ability to run JUnit tests in parallel at the test class level. For test method level parallelism the testing frameworks must implement the Jumi Driver API. Right now you can get that full parallelism with Specsy for all JVM based languages (at the moment Scala/Groovy/Java, but creating a wrapper for a new languages is simple) and hopefully other testing frameworks will follow suit.

Once somebody* implements IDE integration for Jumi, its single-threaded speed will be on par with older test runners, and its native parallel test execution will push it ahead. In near future, when Jumi implements the persistent daemon process, class loader caching, test order priorization (run the most-likely-to-fail tests first, similar to JUnit Max) and other optimizations, it will be the fastest test runner ever. :)

* I'll need to rely on the expertise of others who have IDE plugin development experience. Three big IDEs is too much for one small developer... :(

You can get started on using Jumi through Jumi's documentation. You're welcome to ask questions on the Jumi mailing list. Please come there to tell us that which features you would like to see implemented first. Testing framework, IDE and build tool developers are especially encouraged to get in touch, to tell about their needs and to develop tool integration.


New Testing Tools: Jumi 0.2 & Specsy 2

New versions of two of my projects have just been released. They both are testing tools for Java/JVM-based languages.

Jumi 0.2

Jumi is a new test runner for the JVM, which overcomes the JUnit test runner's limitations to better support all testing frameworks, and offers better performance and usability to end-users.

This release is ready for early adopters to start using Jumi. It adds support for running multiple test classes (though they must be listed manually). The next releases will add automatic test discovery (so you can identify all tests with e.g. "*Test") and JUnit backward compatibility (so that Jumi will run also JUnit tests).

The Specsy testing framework has been upgraded to run using Jumi. We hope that more testing frameworks will implement Jumi support in near future - please come to the Jumi mailing list if you're a tool vendor interested in implementing Jumi support.

Specsy 2

The Specsy testing framework supports now more languages than ever (Specsy 1 was Scala-only). For now it supports Scala (2.7.7 and higher), Groovy (any version) and Java (7 or higher; lambdas strongly recommended), but it's only a matter of adding one wrapper class to add support for a new JVM-based language.

Specsy 2 runs using the new Jumi test runner, fixing a bunch of issues that Specsy 1.x had with the JUnit test runner's limited expressiveness. Actually Specsy 2 was released already in September, but Jumi wasn't then ready for general use, but now it is.


Continuous Delivery with Maven and Go into Maven Central

In continuous delivery the idea is to have the capability to release the software to production at the press of a button, as often as it makes sense from a business point of view, even on every commit (which would then be called continuous deployment). This means that every binary built could potentially be released to production in a matter of minutes or seconds, if the powers that be so wish.

Maven's ideology is opposed to continuous delivery, because with Maven it must be decided before building whether the next artifact will be a release or a snapshot version, whereas with continuous delivery it will be known only long after building the binaries (e.g. after it has passed all automated and manual testing) whether the binary is fit for release.

In this article I'll explain how I tamed Maven to support continuous delivery, and how to do continuous delivery into the Maven Central repository. For continuous integration and release management I'm using Go (not to be confused with Go, go or other uses of go). Git's distributed nature comes also into use, when creating new commits and tags during each build.

The project in question is Jumi, a new test runner for Java to replace JUnit's test runner. It's an open source library and framework which will be used by testing frameworks, build tools, IDEs and such, so the method of distribution is publishing it to Maven Central through Sonatype OSSRH.

Version numbering with Maven to support continuous delivery

Maven Release Plugin is diametrically opposed to continuous delivery, as is the use of snapshot version numbers in Maven, so I defined the project's version numbering scheme to use the MAJOR.MINOR.BUILD format. In the source repository the version numbers are snapshot versions, but before building the binaries the CI server will use the following script to read the current version from pom.xml and append the build number to it. For example, the latest (non-published) build is build number 134 and the version number in pom.xml is 0.1-SNAPSHOT, so the resulting release version number will be 0.1.134.

In the project's build script I'm then using Versions Maven Plugin to change the version numbers in all modules of this multi-module Maven project. The changed POM files are not checked in, because in continuous delivery each binary should anyways be built only once. To make the sources traceable I'm tagging every build, but I publish those tags only after releasing the build - more about that later.

P.S. versions:set requires the root aggregate POM to extend the parent POM, or else you will need to run it separately for both the root and the parent POMs by giving Maven the --file option. Also, to keep all plugin version numbers in the parent POM's dependencyManagement section, the root should extend parent, or else you will need to define the plugin version numbers on the command line and can't use the shorthand commands for running Maven plugins (unless you're OK with Maven deciding itself that which version to use, which can produce unrepeatable builds).

Managing the deployment pipeline with Go

Before going into further topics, I'll need to explain a bit about Go's artifact management and give an overview of Jumi's deployment pipeline.

Go is more than just a continuous integration server - it is designed for release and deployment management. (I find that Go fits that purpose better than for example Jenkins.) In Go the builds are organized into a deployment pipeline, as described in the Continuous Delivery book. All pipeline runs are logged forever* and configuration changes are version controlled (Go uses internally Git), so it also provides full auditability especially when it's used for deployment. The commercial version has additional deployment environment and access permission features, but the free version has been adequate for me for now (though being an open source project I could get the enterprise edition for free). Update 2012-09-20: Since Go 12.3 the free version has the same features as the commercial version - only the maximum numbers of users and remote agents differ. Update 2014-02-25: Go is now open source and fully free!

* For experimenting I recommend cloning a pipeline with a temporary name, to avoid the official pipeline's history from being filled with experimental runs. You can't remove things from the history (except by removing or hacking the database), but you can hide them by deleting the pipeline and not reusing the pipeline's name (if you create a new pipeline with the same name then the old history will become visible). Only build artifacts of old builds can be removed, and there's actually a feature for that to avoid the disk getting too full.

Go's central concepts are pipelines, stages and jobs. Each pipeline consists of one or more sequentially executed steps, and each step consists of one or more jobs (which can be executed in parallel if you have multiple build agents). A pipeline can depend on the stages of other pipelines and a stage can be triggered automatically or manually with a button press, letting you manage complex builds by chaining multiple pipelines together.

A pipeline may even depend on multiple pipelines, for example if the system is composed of multiple separately built applications, in which case one pipeline could be used to select that which versions to deploy together. Then further downstream pipelines or stages can be used to deploy the selected versions together into development, testing and finally into the production environment.

You can save artifacts produced by a job on the Go server and then access those artifacts in downstream jobs by fetching the artifacts into the working directory before running your scripts. Go uses environment variables to tell the build scripts about the build number, source control revision, identifiers of previous stages, custom parameters etc.

Here you can see two dependent pipelines from Jumi's deployment pipeline. Clicking the button in jumi-publish would trigger that pipeline using the artifacts from this particular build of the jumi pipeline. I can trigger the downstream pipeline using any previous build - it doesn't have to be the latest build.

Running a shell command in Go requires more configuration than in Jenkins (which has a single text area for inputting multiple commands), which has the positive side-effect that it drives you to store all build scripts in version control. I have one shell script for each Go job which in turn call a bunch of Ruby scripts and Maven commands.

Below is a diagram showing Jumi's deployment pipeline at the time of writing. The names of pipelines are in bold, stages underscored, and jobs are links to their respective shell scripts. For more details see the pipeline configuration.

 |--> jumi (polling automatically)
      build         --> analyze
      build-release     coverage-report
         |--> jumi-publish (manually triggered)
              pre-check           --> ossrh           --> github
              check-release-notes     promote-staging     push-staging

The jumi/build stage builds the Maven artifacts and saves them on the Go server for use in later stages. It also tags the release and updates the release notes, but those commits and tags are not yet published, but they are saved on the Go server.

The jumi/analyze stage runs PIT mutation testing and produces line coverage and mutation coverage reports. They can be viewed in Go on their own tab.

The jumi-publish/pre-check stage makes sure that release notes for the release have been filled in (no "TBD" line items), or else it will fail the pipeline and prevent the release.

The jumi-publish/ossrh stage uploads the build artifacts from Go into OSSRH. It doesn't yet run the last Nexus command for promoting the artifacts from the OSSRH staging repository into Maven Central (I need to log in to OSSRH and click a button), because I haven't yet written smoke tests which would make sure that all artifacts were uploaded correctly.

The jumi-publish/github stage pushes to the official Git repository the tags and commits which were created in jumi/build. It will merge automatically if somebody has pushed there commits after this build was made.

Future plans for improving this pipeline include adding a new jumi-integration pipeline between jumi and jumi-publish. It will run consumer contract tests of programs using Jumi, against multiple versions of those programs to notice any backward incompatibility issues. This stage might eventually take hours to execute, in which case I may break it into multiple jobs and run them in parallel. I will also reuse a subset of those tests as smoke tests in the jumi-publish pipeline, after which I can automate the final step to promote from OSSRH to Maven Central.

Staging repository for Maven artifacts in Go

Creating publishable Maven artifacts happens with the Maven Deploy Plugin and the location of the Maven repository can be configured with altDeploymentRepository. I'm using -DaltDeploymentRepository="staging::default::file:staging" to create a staging repository with only this build's artifacts, which I then save on the Go server. (The file:staging path means the same as file://$PWD/staging but works also on Windows.)

That staging repository can be accessed from the Go server using HTTP, so it would be quite simple to let beta users access them (optionally using HTTP basic authentication). For example the URL to a build's staging repository could be http://build-server/go/files/jumi/134/build/1/build-release/staging/ Though inside Go jobs it's the easiest to fetch the staging repository to the working directory. That avoids the need to configure Maven's HTTP authentication settings.

When it is decided that a build can be published, I trigger the jumi-publish pipeline which uses the following script to upload the staging repository from a directory into a remote Maven repository. It uses curl to do HTTP PUT commands with HTTP basic authentication. (I wasn't able to find any documentation about the protocol of a Maven repository like Nexus, but was able to sniff it using Wireshark.)

In addition to the above uploading, the publish script uses Nexus Maven Plugin to close the OSSRH repository to which the artifacts were uploaded. It could also promote it to Maven Central, but I want to first create some smoke tests to make sure that all the necessary artifacts were uploaded. Until then I'll do a manual check before clicking the button in OSSRH to promote the artifacts to Maven Central.

Publishing to Maven Central puts some additional requirements on the artifacts. Since I'm not using Maven Release Plugin, I need to manually enable the sonatype-oss-release profile in Sonatype OSS Parent POM to generate all the required artifacts and to sign them. If you don't need to publish artifacts to Maven Central, then you might not need to do this signing. But if you do, it's good to know that the Maven GPG Plugin accepts as parameters the name and passphrase of the GPG key to use. They can be configured in the Go pipeline using secure environment variables which are automatically replaced with ******** in the console output. (For more security, don't save the passphrases on the Go server, but manually enter them when triggering the pipeline. Otherwise somebody with root access to the Go server could get the Go server's private key and decrypt the passphrase in Go's configuration files. Though using passphraseless SSH and GPG keys on the CI server is much simpler.)

Tagging the release and updating release notes with Git

When I do a release, I want it to be tagged and the version and date of the release added to release notes, which are in a text file in the root of the project. In order to get the release notes included in the tagged revision and for the tag to be on the revision which was built, that commit needs to be done before building (an additional benefit is that all GPG signing - the build artifacts and the tag - will be done in the build stage). Since I'm using Git, I can avoid the infinite loop, which would otherwise ensue from committing on every build, by pushing the commits only if the build is released.

The release notes for the next release can be read from the release notes file with a little regular expression (get-release-notes.rb). Writing the version number and date of the release into release notes is also solvable using regular expressions (prepare-release-notes.rb), as is preparing for the next release iteration by adding a placeholder for the future release notes (bump-release-notes.rb).

With the help of those helper scripts the build script, shown below, will be able to create a commit that contains the finalized release notes and tag it with a GPG signed tag (I'm including the release notes also in the tag message). It saves the release metadata into files, so that later stages of the pipeline would not need to recalculate them (for example in promote-staging.sh), and so that I could see them in a custom tab in Go (build-summary.html). Then the script does the build with Maven and after that does another commit which prepares the release notes for a future release.

At the end of the above script you will see what lets me get away with doing commits on build. I'm creating a new repository to the directory staging.git and saving that on the Go server the same way as all build artifacts.

Then when a release is published, the following script is used to merge those commits to the master branch and push them to the official Git repository:

Hopefully this article has given you some ideas for implementing continuous delivery using Maven.