Pinning protoc and protobuf-java version with Maven

There is one thing about Protocol Buffers in Java projects that can get wrong, and it bites silently: the protoc compiler on your machine is a different version than the protobuf-java library your project depends on.

It usually works. Until it doesn’t.

TLDR: Don’t install protoc on your machine or CI/CD runner. Let Maven download the exact protoc binary matching your protobuf-java dependency, keeping generated code in sync with model consumers. Whole thing as one pom.xml, no manual install, identical on your laptop and in CI/CD, works everywhere. Example project: protobuf-model-example

If you are interested how, continue reading

The Problem

You want to generate Java classes from a .proto file. So you do the obvious thing:

# macOS
brew install protobuf
# Debian/Ubuntu
apt-get install -y protobuf-compiler

Or following the Protocol Buffer Compiler Installation and getting latest version.

You get whatever version the package manager hands you - usually the newest one. Today that might be protoc 33.x, tomorrow 34.x, and on the CI runner, whatever was baked into the image six months ago.

Meanwhile, your pom.xml (and those using the model) depends on a fixed library version:

<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>4.33.5</version>
</dependency>

And here is the catch: the protoc compiler and the protobuf-java runtime are released together, as a matched pair. The generated code emitted by protoc calls into the runtime, and the contract between them changes between versions. When protoc is newer than protobuf-java, the generated sources can reference runtime methods that simply do not exist in your dependency yet, and you get a compilation failure - or worse, something that compiles but misbehaves at runtime.

So the version of protoc is rarely exactly the same as the protobuf-java your project imports. On your machine, on your colleague’s machine, and on the CI/CD pipeline - three different versions, three different results from the same source tree. It might work. Sometimes. Until it breaks with parsing errors on production and version mismatch is the last thing that comes to mind.

In this article, we go through steps needed to pin version of the protoc compiler and protobuf-java, reliably, reproducibly.

Setting up the project

Let’s build the smallest possible project that does it right. The structure is plain Maven:

model/
├── pom.xml
└── src/
    └── main/
        └── resources/
            └── proto/
                └── sail_event.proto

We keep the .proto files under src/main/resources/proto so they ship inside the JAR as well - consumers who want the original schema get it for free.

Here is the example message, src/main/resources/proto/sail_event.proto:

syntax = "proto3";

package com.michalklempa.model;

option java_package = "com.michalklempa.model";
option java_outer_classname = "SailEventProto";

message SailEvent {
  double windSpeed = 2;
  double speedOverGround = 4;
  string name = 7;
}

To turn that into Java, naive approach is to call protoc from the build. We can do exactly that with maven-antrun-plugin, binding it to the generate-sources phase:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-antrun-plugin</artifactId>
    <version>3.1.0</version>
    <executions>
        <execution>
            <id>protobuf-compile</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>run</goal>
            </goals>
            <configuration>
                <target>
                    <mkdir dir="${project.build.directory}/generated-sources"/>
                    <apply executable="protoc" failonerror="true">
                        <arg value="--java_out=${project.build.directory}/generated-sources"/>
                        <arg value="--proto_path=${project.basedir}/src/main/resources/proto"/>
                        <fileset dir="${project.basedir}/src/main/resources/proto" includes="*.proto"/>
                    </apply>
                </target>
            </configuration>
        </execution>
    </executions>
</plugin>

And because the generated .java files land in a directory Maven doesn’t know about, we tell it to compile them too, with build-helper-maven-plugin:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <version>3.6.1</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${project.build.directory}/generated-sources</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>

It works. But why is it fragile?

Look at the executable="protoc" line. It runs whatever protoc is first on the PATH. That is exactly the unpinned version from the previous section. We moved the problem into the build, but we did not solve it. We need a protoc that is guaranteed to match 4.33.5.

Downloading the exact protoc with Maven

Here is the part most people don’t know: Google publishes the protoc binary itself to Maven Central, as an artifact with the same coordinates and the same version as the library. It is packaged with <type>exe</type> and you can find it in Maven Central.

The link for example for my x86-64 architecture with Linux, would be (https://repo1.maven.org/maven2/com/google/protobuf/protoc/4.33.5/protoc-4.33.5-linux-x86_64.exe).

We can ask Maven to fetch the binary that belongs to our dependency version, using maven-dependency-plugin:

<build>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.8.1</version>
    <execution>
        <id>copy-dependencies</id>
        <phase>validate</phase>
        <goals>
            <goal>copy</goal>
        </goals>
        <configuration>
            <artifactItems>
                <artifactItem>
                    <groupId>com.google.protobuf</groupId>
                    <artifactId>protoc</artifactId>
                    <version>${protobuf.version}</version>
                    <type>exe</type>
                    <classifier>linux-x86_64</classifier>
                    <overWrite>false</overWrite>
                    <destFileName>protoc.exe</destFileName>
                </artifactItem>
            </artifactItems>
            <outputDirectory>${project.build.directory}</outputDirectory>
        </configuration>
    </execution>
...

Note the classifier as linux-x86_64 to get the right binary. To make this download right binary on another machine (e.g. CI/CD) this must be changed, we will solve this later, keep reading.

We define protobuf.version once, in <properties>, and both the library dependency and the compiler read from it:

<properties>
    <protobuf.version>4.33.5</protobuf.version>
</properties>

One property, one version, no drift. The compiler and the runtime can no longer disagree, because they are literally the same number.

Then we point antrun at the downloaded binary instead of the PATH one:

<apply executable="${project.build.directory}/protoc.exe" failonerror="true">

But wait - my OS is Linux. What is this .exe?

Good question, and this is where it gets confusing.

The artifact type is exe and the file we save is called protoc.exe. On Windows that is honest. On Linux or macOS it looks absurd - there are no .exe files there.

The trick: the .exe is just the artifact’s extension on Maven Central, not the actual binary format. The real binary differs per operating system and CPU architecture - an ELF executable on Linux, a Mach-O on macOS, a PE on Windows. Maven Central stores all of them, distinguished by a classifier like linux-x86_64, osx-aarch_64, or windows-x86_64. So on Linux, the file named protoc.exe is in fact a perfectly normal ELF binary that runs fine - the name is cosmetic.

We don’t want to hardcode that classifier, otherwise the build breaks the moment someone runs it on a different OS - exactly the CI/CD problem we set out to kill.

Enter os-maven-plugin, a Maven extension that detects the current OS and architecture and exposes it as ${os.detected.classifier}:

<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.7.1</version>
        </extension>
    </extensions>
    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.8.1</version>
    <execution>
        <id>copy-dependencies</id>
        <phase>validate</phase>
        <goals>
            <goal>copy</goal>
        </goals>
        <configuration>
            <artifactItems>
                <artifactItem>
                    <groupId>com.google.protobuf</groupId>
                    <artifactId>protoc</artifactId>
                    <version>${protobuf.version}</version>
                    <type>exe</type>
                    <classifier>${os.detected.classifier}</classifier>
                    <overWrite>false</overWrite>
                    <destFileName>protoc.exe</destFileName>
                </artifactItem>
            </artifactItems>
            <outputDirectory>${project.build.directory}</outputDirectory>
        </configuration>
    </execution>
    ...
</build>

That single property resolves to linux-x86_64 on your CI runner, osx-aarch_64 on a Mac laptop, and windows-x86_64 on Windows - each downloading the correct native binary, all pinned to the same 4.33.5. The build no longer cares where it runs.

Extra bit: well-known types

If you also need the well-known types (google/protobuf/timestamp.proto and friends), unpack them from the protobuf-java artifact using maven-dependency-plugin and add them to the proto path - the same matched version guarantees they line up too (even though they never change):

<execution>
    <id>unpack</id>
    <phase>validate</phase>
    <goals>
        <goal>unpack</goal>
    </goals>
    <configuration>
        <artifactItems>
            <artifactItem>
                <groupId>com.google.protobuf</groupId>
                <artifactId>protobuf-java</artifactId>
                <version>${protobuf.version}</version>
                <overWrite>true</overWrite>
                <outputDirectory>${project.build.directory}/include</outputDirectory>
            </artifactItem>
        </artifactItems>
    </configuration>
</execution>

Attaching sources - don’t make your consumers guess

There is one last step that people skip, and it quietly hurts everyone downstream.

Your model JAR contains generated .class files. The Java source that produced them was generated into target/generated-sources during the build - it is not in src, it does not live in git (which I find being advantage), and it is gone the moment someone consumes your artifact.

So a developer who pulls your model as a dependency opens SailEventProto in their IDE and sees… decompiled bytecode.

No comments, no field names you intended, no chance to step through it while debugging.

Attach them with maven-source-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-source-plugin</artifactId>
    <version>3.3.1</version>
    <executions>
        <execution>
            <id>attach-sources</id>
            <phase>verify</phase>
            <goals>
                <goal>jar-no-fork</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Because we registered the generated directory with build-helper-maven-plugin earlier, the source plugin picks it up automatically and publishes a model-1-SNAPSHOT-sources.jar alongside the main JAR. Now when a consumer navigates into SailEvent, their IDE shows the real generated Java, comments and all.

How it all fits together

The phases line up like this:

Build it:

mvn clean verify

No protoc installed anywhere (except the ./target directory). Same result on every machine.

When to use this

• You ship a model / schema artifact that other projects depend on.
• Your build runs in more than one place - laptops, CI, release pipeline - and you are tired of “works on my machine”.
• You want the protoc version to be reviewable in git, pinned next to the dependency it must match.

Advantages

• One version, zero drift - compiler and runtime can never disagree, they read the same property.
• No manual install - nobody has to brew install protobuf, and the CI image needs nothing extra.
• OS-agnostic - the same pom.xml produces the right native binary on Linux, macOS and Windows.
• Consumer-friendly - generated sources travel with the artifact.

Disadvantages

• More moving parts in the pom.xml than a single protobuf-maven-plugin line - this is the explicit, transparent version of what that plugin does for you.
• The protoc.exe naming is cosmetically odd on non-Windows systems (but harmless).
• First build downloads the binary; offline builds need it cached in the local repository.

Comparison to other solutions

There are dedicated plugins wrap the same idea:

• protoc-jar-maven-plugin - protoc is also downloaded dynamically (if artifact sroperty is set). But the plugin is unmaintained.
• xolstice protobuf-maven-plugin - pulls protoc from Maven Central via protocArtifact, same trick as here. No longer maintained, and you still add maven-source-plugin yourself.
• ascopes protobuf-maven-plugin - looks like the modern, maintained successor; same Maven-Central protoc, plus gRPC/Kotlin support. Documentation sparse and I haven’t tried it, yet. Let me know if you use it.

Example Source Code

As always the example repository is available at protobuf-model-example