Repository navigation
Description
Describe the Bug
Python scripts complain about protobuf mismatch between gencode(protoc) and runtime(protobuf) versions.
Steps to Reproduce
docker run -it --rm -w /tmp ubuntu:22.04 bash
#Within the container
apt -y update; apt -y upgrade
apt -y install coreutils wget vim git make cmake python3.10 python3-pip
pip install --upgrade pip
git clone https://github.com/mlcommons/chakra.git
cd chakra/
pip install .
chakra_generator
#Error message
root@28df32360644:/tmp/chakra# chakra_generator
Traceback (most recent call last):
File "/usr/local/bin/chakra_generator", line 5, in <module>
from chakra.src.generator.generator import main
File "/usr/local/lib/python3.10/dist-packages/chakra/src/generator/generator.py", line 3, in <module>
from ...schema.protobuf.et_def_pb2 import (
File "/usr/local/lib/python3.10/dist-packages/chakra/schema/protobuf/et_def_pb2.py", line 12, in <module>
_runtime_version.ValidateProtobufRuntimeVersion(
File "/usr/local/lib/python3.10/dist-packages/google/protobuf/runtime_version.py", line 106, in ValidateProtobufRuntimeVersion
_ReportVersionError(
File "/usr/local/lib/python3.10/dist-packages/google/protobuf/runtime_version.py", line 50, in _ReportVersionError
raise VersionError(msg)
google.protobuf.runtime_version.VersionError: Detected mismatched Protobuf Gencode/Runtime major versions when loading et_def.proto: gencode 6.31.0 runtime 5.29.5. Same major version is required. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.Possible Solutions
The two possible fixes are:
- Edit
pyproject.tomlto accept protobuf==6.* - Find a way to configure
setuptools-grpcto accept a specific protoc version (Very much unlikely from my impression of the source code, but still)
Detailed explanation:
Copy & Pasted from astra-sim/astra-sim#314 .
TL, DR: As long as Chakra uses
setuptools-grpcin the build process to compile.protofiles, regardless of what is written inpyproject.toml, the generatedpb2.pyfile will always be compiled with whateverprotocthegrpcrepo points to (which is v.6.31.0 as of today)
gencode 6.31.0refers to theprotocversion used to convertet_def.protointoet_def_pb2.py.runtime 5.29.0refers to the pipprotobufpackage, that is triggered when running python scripts likepython gen_chakra_traces.py.
We know that pip install . installs protobuf v5.29.0 and uses it, as it is the dependency defined in Chakra's pyproject.toml file.
Then where does the 6.31.0 come from?
When running pip install ., Chakra's pyproject.toml declares that it will use setuptools, specifically a plugin called setuptools-grpc whose repo is available (here)[https://github.com/CZ-NIC/setuptools-grpc]. setuptools-grpc imports and uses grpc_tools.protoc. That is, setuptools-grpc does not use the system installed, or even the pip installed protobuf/protoc to compile et_def.proto into et_def_pb2.py. This can be verified by trying to run pip install . without installing protoc in either the system (apt-get -y ...) or pip. You can see that the pb2.py files are created nontheless.
Then what protoc version does setuptools-grpc use? If you go into the grpc repo which holds grpc_tools, you can see that it imports the source code of the protocol buffer as a submodule, and compiles protoc from source. That is, any code calling grpc_tools.protoc is fixed to whichever protoc version is submoduled in the grpc repo.
Looking into the code, we see that currently (2025-JUN-12), the grpc repo points to commit 3d4adad of the protobuf repo. This commit has a tag of v31.0, which corresponds to v6.31.0 in Python. That's where the gencode 6.31.0 comes from.
More notes: Why does v31.0 correspond to v6.31.0 "in Python"?
Ref: https://protobuf.dev/support/version-support/
The protobuf repo has a repo-wide version, which consists only of minor/patch numbers (v31.0, v29.5, etc).
For each repo-wide version, each language may use different major versions. For example, for the repo-wide version v31.0, C++ and Python calls it v6.31.0, while Ruby calls it v4.31.0. This is because the same repo-wide update may have a groundbreaking change in one language, while it has minimal effect in another language. That is, C++ v6.31.0 and Ruby v4.31.0 points to the same commit in the protobuf repo.
So, once we know the repo-wide version v31.0, we look up the table in the link above to find the major number for Python
Activity
theodorbadea commentedon Jun 13, 2025
@jinsun-yoo thanks for raising this, yesterday I struggled a lot with this issue and I really did not know if it's something on my end or not. As you have highlighted, changing the proto compiler version embedded in setuptools-grpc seems a nightmare. Ultimately, I started using 6.31.0, so basically your proposed solution 1. and I did not face any issue.
sanalcc1 commentedon Jun 17, 2025
The gencode version is picked up based on the version of python you are using. Python 3.8 worked fine without any errors for me. (3.8.20 to be exact)
jjasoncool commentedon Jun 19, 2025
I have a question.
If I want to build my own Dockerfile, where should I place
pip install protobuf==6.31.0?Below is my Dockerfile.
I’ve tried installing different versions, but I still get the same error.
jjasoncool commentedon Jun 19, 2025
I found the solution: I moved the
RUN pip install protobuf==6.31.0command to after the Chakra installation in the Dockerfile, and it now works fine.theodorbadea commentedon Sep 26, 2025
@jinsun-yoo I noticed you've addressed the protobuf version in PR#202. Can this issue be closed?
theodorbadea commentedon Oct 22, 2026
Follow-up on this: I think we should pin grpcio-tools to 1.80.0 and protobuf>=6.31.1,<8. Latest is 7.36.2. All the constraints seem to originate from user environments and this combo seems the best compromise...
AI summary of version checks:
Latest grpcio-tools (1.84.0) requires protobuf>=7.35.1,<1 so we circle back to this issue, 1.71.0 wants protobuf>=5.29,<7 which works on 6.x with warning and on 7.x by accident so we would rely on unsupported behavior. If kept unpinned as it is today, will get the latest with a protobuf unbounded so I feel like we would circle back to this issue. grpcio-tools==1.80.0 wants protobuf>=6.31.1,<8 so this seems like the happy combo.