Skip to content

Commit

Permalink
Only support protoc toolchainization for >= v29.0
Browse files Browse the repository at this point in the history 
Updates `README.md` to explain why this requirement exists, and why
Bazel 6.5.0 is capped at `protobuf` v29 in general, even without
protocol compiler toolchainization. Removes the `v28.2` and `v28.3`
entries from `scripts/update_protoc_integrity.py` and
`protoc/private/protoc_integrity.bzl`.

I tried several things to get it to work with `protobuf` v28.2, as
described below. However, each path only led to the same suffering
described in the new "Why this requires `protobuf` v29 or later" section
of the README.

Without `--incompatible_enable_proto_toolchain_resolution`, everything
still worked (i.e., `bazel build`  compiled `protoc` and finished
successfully). But there's no point in keeping any of the new patches
or flags if `--incompatible_enable_proto_toolchain_resolution` still
breaks.

I discovered along the way that `protobuf` v30 isn't compatible with
Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0
compatibility" section of `README.md`.

---

I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and
`rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2
with the following statements:

```py
load("//bazel/common:proto_common.bzl", "proto_common")
load("@rules_proto//proto:proto_common.bzl", "toolchains")

_PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type"
_PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION"
_PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN)

def _protoc_files_to_run(ctx):
    if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False):
```

I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`.

I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to
fix a "no such package: //proto" breakage:

```diff
 6.0.2 patch for `proto_toolchain()`:

```diff
diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl
index a091b80..def2699 100644
--- i/proto/private/rules/proto_toolchain.bzl
+++ w/proto/private/rules/proto_toolchain.bzl
@@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []):

     native.toolchain(
         name = name + "_toolchain",
-        toolchain_type = "//proto:toolchain_type",
+        toolchain_type = Label("//proto:toolchain_type"),
         exec_compatible_with = exec_compatible_with,
         target_compatible_with = [],
         toolchain = name,

```

I tried adding combinations of the following
`--incompatible_autoload_externally` flag values to .bazelrc`:

```txt
common --incompatible_autoload_externally=+@protobuf,+@rules_java
```

Nothing worked.
  • Loading branch information
@mbland
mbland committed Mar 8, 2025
1 parent d7352d6 commit 116dd19
Show file tree
Hide file tree
Showing 3 changed files with 91 additions and 39 deletions.
108 changes: 91 additions & 17 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -196,16 +196,17 @@ load(

### <a id="protoc"></a>Using a precompiled protocol compiler

`rules_scala` now supports
[`--incompatible_enable_proto_toolchain_resolution`][]. When using this flag
with the `MODULE.bazel` or `WORKSPACE` configurations below, `rules_scala` will
use a precompiled protocol compiler binary by default.
`rules_scala` now supports the
[`--incompatible_enable_proto_toolchain_resolution`][] flag when using
[`protobuf` v29 or later](#why-proto-v29). When using this flag with the
`MODULE.bazel` or `WORKSPACE` configurations below, `rules_scala` will use a
precompiled protocol compiler binary by default.

[`--incompatible_enable_proto_toolchain_resolution`]: https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution

__Windows builds now require the precompiled protocol compiler toolchain.__ See
the [Windows MSVC builds of protobuf broken by default](#protoc-msvc) section
below for details.
__Windows builds now require using `protobuf` v29 or later with the precompiled
protocol compiler toolchain.__ See the [Windows MSVC builds of protobuf broken
by default](#protoc-msvc) section below for details.

#### Common setup

Expand All @@ -226,12 +227,11 @@ register_toolchains("@rules_scala//protoc:all")

#### Temporary required `protobuf` patch

As of `protobuf` v29.3, enabling protocol compiler toolchainization requires
applying [protoc/0001-protobuf-19679-rm-protoc-dep.patch][]. It is the `git
diff` output from the branch used to create protocolbuffers/protobuf#19679.
Without it, there remains a transitive dependency on
`@com_google_protobuf//:protoc`, causing it to recompile even with the
precompiled toolchain registered first.
At the moment, enabling protocol compiler toolchainization requires applying
[protoc/0001-protobuf-19679-rm-protoc-dep.patch][]. It is the `git diff` output
from the branch used to create protocolbuffers/protobuf#19679. Without it, a
transitive dependency on `@com_google_protobuf//:protoc` remains, causing
`protoc` to recompile even with the precompiled toolchain registered first.

[protoc/0001-protobuf-19679-rm-protoc-dep.patch]: ./protoc/0001-protobuf-19679-rm-protoc-dep.patch

Expand Down Expand Up @@ -289,6 +289,55 @@ load("@platforms//host:extension.bzl", "host_platform_repo")
host_platform_repo(name = "host_platform")
```

#### <a id="why-proto-v29"></a>Why this requires `protobuf` v29 or later

Using `--incompatible_enable_proto_toolchain_resolution` with versions of
`protobuf` before v29 causes build failures due to a missing internal Bazel
dependency.

Bazel's builtin `bazel_java_proto_aspect` transitively depends on a toolchain
with a [`toolchain_type`][] of `@rules_java//java/proto:toolchain_type`.
Experimentation with `protobuf` v28.2 using both Bazel 6.5.0 and 7.5.0 led to
the following error:

```txt
ERROR: .../external/bazel_tools/src/main/protobuf/BUILD:28:15:
in @@_builtins//:common/java/proto/java_proto_library.bzl%bazel_java_proto_aspect
aspect on proto_library rule
@@bazel_tools//src/main/protobuf:worker_protocol_proto:
Traceback (most recent call last):
File "/virtual_builtins_bzl/common/java/proto/java_proto_library.bzl",
line 53, column 53, in _bazel_java_proto_aspect_impl
File "/virtual_builtins_bzl/common/proto/proto_common.bzl",
line 364, column 17, in _find_toolchain
Error in fail: No toolchains registered for
'@rules_java//java/proto:toolchain_type'.
ERROR: Analysis of target
'@@bazel_tools//src/main/protobuf:worker_protocol_proto' failed
```

See the commit "Only support protoc toolchainization for >= v29.0" from
bazelbuild/rules_scala#1710 for details of the experiment.

For `protobuf` v29.0, protocolbuffers/protobuf#18308 added the
[`@protobuf//bazel/private/toolchains`][proto-private-tc] package and updated
`protobuf_deps()` from `@protobuf//:protobuf_deps.bzl` to register it:

```py
native.register_toolchains("//bazel/private/toolchains:all")
```

[`toolchain_type`]: https://bazel.build/extending/toolchains#writing-rules-toolchains
[proto-private-tc]: https://github.com/protocolbuffers/protobuf/blob/v29.0/bazel/private/toolchains/BUILD.bazel

protocolbuffers/protobuf#18435 then introduced
[`java_source_toolchain_bazel7`][java-proto-tc] with the required
`toolchain_type`.

[java-proto-tc]: https://github.com/protocolbuffers/protobuf/blob/v29.0/bazel/private/toolchains/BUILD.bazel#L50-L74

#### More background on protocol compiler toolchainization

- [Proto Toolchainisation Design Doc](
Expand Down Expand Up @@ -905,11 +954,36 @@ At the moment, `WORKSPACE` builds mostly continue to work with Bazel 6.5.0, but
not out of the box, and may break at any time. You will have to choose one of
the following approaches to resolve `protobuf` compatibility issues.

First, you may choose to use protocol compiler toolchainization. See the [Using
a precompiled protocol compiler](#protoc) section for details.
First of all, you _must_ use `protobuf` v29 or earlier. `rules_scala` now uses
v30 by default, which removes `py_proto_library` and other symbols that Bazel
6.5.0 requires:

```txt
ERROR: Traceback (most recent call last):
File ".../external/bazel_tools/src/main/protobuf/BUILD",
line 1, column 46, in <toplevel>
load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")
Error: file '@com_google_protobuf//:protobuf.bzl'
does not contain symbol 'py_proto_library'
ERROR: .../src/java/io/bazel/rulesscala/worker/BUILD:3:13:
no such target '@bazel_tools//src/main/protobuf:worker_protocol_java_proto':
target 'worker_protocol_java_proto'
not declared in package 'src/main/protobuf'
defined by .../external/bazel_tools/src/main/protobuf/BUILD
(Tip: use `query "@bazel_tools//src/main/protobuf:*"`
to see all the targets in that package)
and referenced by '//src/java/io/bazel/rulesscala/worker:worker'
```

You may use protocol compiler toolchainization with `protobuf` v29 to avoid
recompiling `protoc`. See the [Using a precompiled protocol compiler](#protoc)
section for details.

Otherwise, per bazelbuild/rules_scala#1647, you must add the following flags to
`.bazelrc`, required by the newer `abseil-cpp` version used by `protobuf`:
Otherwise, per bazelbuild/rules_scala#1647, add the following flags to
`.bazelrc` to compile the newer `abseil-cpp` versions used by newer `protobuf`
versions:

```txt
common --enable_platform_specific_config
Expand Down
20 changes: 0 additions & 20 deletions protoc/private/protoc_integrity.bzl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ PROTOC_VERSIONS = [
"29.2",
"29.1",
"29.0",
"28.3",
"28.2",
]

PROTOC_BUILDS = {
Expand All @@ -31,8 +29,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-Kc9IPi+yGCfl+sSWTjXq5HKiOOKMdi8C+xfc2T/4uJ8=",
"29.1": "sha256-H3Sj8zVd58Bma8ElYRwTUywlmPhTUh0NPmIaWwnyR5k=",
"29.0": "sha256-MF8b5a57LzlFGHCzErRcHguiaZAcg7oW2F+fnRRBs0g=",
"28.3": "sha256-HeUiAyqLGUAC/jXKuG10eEgji15N5PmWSDcgefW0b5o=",
"28.2": "sha256-kdglPNwPDw/FHCtpyAZ3mWYy9SWthFBL+ltO44rT5Jw=",
},
},
"linux-ppcle_64": {
Expand All @@ -46,8 +42,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-uiCJWht/NKb/ql5Rw0ExbErrxMFEN3hjQITpn262f/k=",
"29.1": "sha256-B1vWZq1B60BKkjv6+pCkr6IHSiyaqLnHfERF5hbo75s=",
"29.0": "sha256-EJAnjNB1e3AsNrA+6t9KvTYCAnm7B7DfX4C9iW8+IDM=",
"28.3": "sha256-dSKdPN5z5wYZcXgU9R+m9K16NiwzUe5ZGXuxV8sAgsY=",
"28.2": "sha256-xcFrR2f/iGYJDuEOhwkDABBSAbXbHs7zgO3XPCM03Ng=",
},
},
"linux-s390_64": {
Expand All @@ -61,8 +55,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-LwpVmdprgpMqCNQz+nkTux1oWIl8zTxrwyhvAYt2xOA=",
"29.1": "sha256-J5fNVlyn/7/ALaVsygE7/0+X5oMkizuZ3PfUuxf27GE=",
"29.0": "sha256-LhXZqwaFbCXKbeYi4RR4XZGHHb25HCwQPrYouH8m3Ew=",
"28.3": "sha256-jhtvqCX7CVlqiS5d6bgRLXoJtL3ftxzLv1+GgKT1aKc=",
"28.2": "sha256-ESIsQ4+G6Hsv2vaKKmzB2ytiKBP9btiRUJ4vOw4F7hs=",
},
},
"linux-x86_32": {
Expand All @@ -76,8 +68,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-FU+NR+6YO8bxoa4tsUl+53E8Qt7vY1EXDxhmG/vNouw=",
"29.1": "sha256-nd/EAbEqC4dHHg4POg0MKpByRj8EFDUUnpSfiiCCu+s=",
"29.0": "sha256-tKyBCfKrSGLV5WuH4/cVMPug46YeyLnZQOKUdskGAQE=",
"28.3": "sha256-DJ6zLLnl06rHLGfPc38In+Mr8kVssBc+EpWQfNGIXhI=",
"28.2": "sha256-ucjToo5Lq5WcwQ5smjjxfFlGc3Npv+AT9RqG19Ns//E=",
},
},
"linux-x86_64": {
Expand All @@ -91,8 +81,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-Uunn7OVcfjDn6LvSVLSyG0CKUwm8qCZ2PHEktpahMuk=",
"29.1": "sha256-AMg/6XIthelsgblBsp8Xp0SzO0zmbg8YAJ/Yk33iLGA=",
"29.0": "sha256-PFEGWvO5pgbZ4Yob9igUNzT/S55pcl1kWYV0MLp6eN8=",
"28.3": "sha256-CtlJ8EpqF02oPNy9s23uCkklJypbbYP3mmv5hSB21T8=",
"28.2": "sha256-L+v9QrWc6Too63iQGaRwo90ESWGbwE+E2tEzPaJh3sE=",
},
},
"osx-aarch_64": {
Expand All @@ -106,8 +94,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-DhU6ONbaGVlMmA5/fNPqDd1SydoQaMA8DYUzNp+/6yA=",
"29.1": "sha256-uP1ZdpJhmKfE6lxutL94lZ1frtJ7/GGCVMqhBD93BEU=",
"29.0": "sha256-srWfA7AwyKdIYj1oKotbycwJnkvP0GuJZM6J7AZbMQM=",
"28.3": "sha256-ks7v2mpyk+wBTm7KyC1kcZNXFFy2/ChlutreteYsBDE=",
"28.2": "sha256-e7BI9ShBeJ2exhmDvgzkyeT7O9mhQ0YoILqaO+CgN5c=",
},
},
"osx-x86_64": {
Expand All @@ -121,8 +107,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-uivZg7XwbsONZjtgKISll96jmQpDgD1+FT7Y98VCaeE=",
"29.1": "sha256-2wK0uG3k1MztPqmTQ0faKNyV5/OIY//EzjzCYoMCjaY=",
"29.0": "sha256-56HP/ILiHapngzARRJxw3f8eujsRWTQ4fm6BQe+rCS8=",
"28.3": "sha256-l/5dRCCQtNvCPNE4T7m0RPodxuZ9FbteH+TeDadjiyA=",
"28.2": "sha256-Iy8H0Sv0gGIHp57CxzeDAcUuby9+/dIcDdQW8L2hA+w=",
},
},
"win32": {
Expand All @@ -136,8 +120,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-73CfcaUbOompsm3meCBc7zyV4h0IGLFn+/WwOackr9E=",
"29.1": "sha256-EQXg+mRFnwsa9e5NWHfauGTy4Q2K6wRhjytpxsOm7QM=",
"29.0": "sha256-154nzOTEAXRUERc8XraBFsPiBACxzEwt0stHeneUGP4=",
"28.3": "sha256-sI/m/M9DE+LMxv1ybchV7okFM+JH9MVDyYf/2YDP1bI=",
"28.2": "sha256-V6hpbqvtUgl19PpGkUGwuC+iEIYj0YtyOqi+yae0u1g=",
},
},
"win64": {
Expand All @@ -151,8 +133,6 @@ PROTOC_BUILDS = {
"29.2": "sha256-Weph77JLnYohQXHiyj/sVcPxUX7/BnZWyHXYoc0Gzk8=",
"29.1": "sha256-fqSCJYV//BIkWIwzXCsa+deKGK+dV8BSjMoxk+M26c4=",
"29.0": "sha256-0DuSGYWLikyogGO3i/Clzec7UYCLkwxLZvBuhILDq+Y=",
"28.3": "sha256-zmT0m97d70nOS9MTqPWbz5L89ntYMe+/ZhcDhtLmaUg=",
"28.2": "sha256-S94ZJx7XyrkANXDyjG5MTXGWPq8SEahr87sl2biVF3o=",
},
},
}
2 changes: 0 additions & 2 deletions scripts/update_protoc_integrity.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@
"29.2",
"29.1",
"29.0",
"28.3",
"28.2",
]

PROTOC_RELEASES_URL = "https://github.com/protocolbuffers/protobuf/releases"
Expand Down

0 comments on commit 116dd19

Please sign in to comment.