Add precompiled protoc toolchain

Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazelbuild#1482 and bazelbuild#1652. Stops `protoc` recompilation, and fixes the build breakage in bazelbuild#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazelbuild#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0
mbland · Mar 1, 2025 · 7b7ab2a · 7b7ab2a
1 parent b55ab59
commit 7b7ab2a
Show file tree

Hide file tree

Showing 23 changed files with 595 additions and 9 deletions.
diff --git a/.bazelrc b/.bazelrc
@@ -2,6 +2,11 @@
 # https://github.com/bazelbuild/rules_scala/issues/1482
 common --enable_workspace --noenable_bzlmod
 
+# Remove once proto toolchainization becomes the default
+# - https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution
+# - https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit
+common --incompatible_enable_proto_toolchain_resolution
+
 build --enable_platform_specific_config
 
 #Windows needs --worker_quit_after_build due to workers not being shut down when the compiler tools need to be rebuilt (resulting in 'file in use' errors). See Bazel Issue#10498.

diff --git a/README.md b/README.md
@@ -62,6 +62,13 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+# The next two calls instantiate the `@host_platform` repo to work around:
+# - https://github.com/bazelbuild/bazel/issues/22558
+# Only required if using `--incompatible_enable_proto_toolchain_resolution`.
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()
@@ -168,6 +175,40 @@ load(
 )
 ```
 
+### <a id="protoc"></a>Using a precompiled protocol compiler
+
+`rules_scala` now supports
+[`--incompatible_enable_proto_toolchain_resolution`][]. When using this flag,
+`rules_scala` will use a precompiled protocol compiler binary by default. To set
+it in your `.bazelrc` file:
+
+[`--incompatible_enable_proto_toolchain_resolution`]: https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution
+
+```txt
+common --incompatible_enable_proto_toolchain_resolution
+```
+
+Set the `protoc_platforms` attribute of `scala_toolchains()` if you need to
+configure protocol compilers for platforms other than the host platform.
+
+Windows builds now require the precompiled protocol compiler toolchain. See the
+[Windows MSVC builds of protobuf broken by default](#protoc-msvc) section
+below for details.
+
+More background on proto toolchainization:
+
+- [Proto Toolchainisation Design Doc](
+    https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit)
+
+- [bazelbuild/bazel#7095: Protobuf repo recompilation sensitivity](
+    https://github.com/bazelbuild/bazel/issues/7095)
+
+- [bazelbuild/rules_proto#179: Implement proto toolchainisation](
+    https://github.com/bazelbuild/rules_proto/issues/179)
+
+- [rules_proto 6.0.0 release notes mentioning Protobuf Toolchainization](
+    https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0)
+
 ### Persistent workers
 
 To run with a persistent worker (much faster), add the following to
@@ -567,6 +608,25 @@ http_archive(
 )
 ```
 
+### <a id="protoc-msvc"></a>Windows MSVC builds of `protobuf` broken by default
+
+MSVC builds of recent `protobuf` versions started failing, as first noted in
+bazelbuild/rules_scala#1710. On top of that, `protobuf` is planning to stop
+supporting Bazel + MSVC builds per:
+
+- [protocolbuffers/protobuf#12947: src build on windows not working](
+    https://github.com/protocolbuffers/protobuf/issues/12947)
+
+- [protobuf.dev News Announcements for Version 30.x:Poison MSVC + Bazel](
+    https://protobuf.dev/news/v30/#poison-msvc--bazel)
+
+- [protocolbuffers/protobuf#20085: Breaking Change: Dropping support for
+    Bazel+MSVC](https://github.com/protocolbuffers/protobuf/issues/20085)
+
+Enable [protocol compiler toolchainization](#protoc) to avoid recompiling
+`@com_google_protobuf//:protoc` on Windows platforms and fix broken Windows
+builds.
+
 ### Bzlmod configuration (coming soon!)
 
 The upcoming Bzlmod implementation will funnel through the `scala_toolchains()`

diff --git a/WORKSPACE b/WORKSPACE
@@ -5,6 +5,13 @@ load("//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+# The next two calls instantiate the `@host_platform` repo to work around:
+# - https://github.com/bazelbuild/bazel/issues/22558
+# Only required if using `--incompatible_enable_proto_toolchain_resolution`.
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/dt_patches/test_dt_patches/WORKSPACE b/dt_patches/test_dt_patches/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/dt_patches/test_dt_patches_user_srcjar/WORKSPACE b/dt_patches/test_dt_patches_user_srcjar/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/crossbuild/WORKSPACE b/examples/crossbuild/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/overridden_artifacts/WORKSPACE b/examples/overridden_artifacts/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/scala3/WORKSPACE b/examples/scala3/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/semanticdb/WORKSPACE b/examples/semanticdb/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/testing/multi_frameworks_toolchain/WORKSPACE b/examples/testing/multi_frameworks_toolchain/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/testing/scalatest_repositories/WORKSPACE b/examples/testing/scalatest_repositories/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/examples/testing/specs2_junit_repositories/WORKSPACE b/examples/testing/specs2_junit_repositories/WORKSPACE
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
 
 rules_scala_dependencies()
 
+load("@platforms//host:extension.bzl", "host_platform_repo")
+
+host_platform_repo(name = "host_platform")
+
 load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
 
 rules_java_dependencies()

diff --git a/scala/deps.bzl b/scala/deps.bzl
@@ -15,6 +15,16 @@ def rules_scala_dependencies():
         ],
     )
 
+    maybe(
+        http_archive,
+        name = "platforms",
+        urls = [
+            "https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.11/platforms-0.0.11.tar.gz",
+            "https://github.com/bazelbuild/platforms/releases/download/0.0.11/platforms-0.0.11.tar.gz",
+        ],
+        sha256 = "29742e87275809b5e598dc2f04d86960cc7a55b3067d97221c9abbc9926bff0f",
+    )
+
     maybe(
         http_archive,
         name = "rules_cc",

diff --git a/scala/private/protoc/protoc_integrity.bzl b/scala/private/protoc/protoc_integrity.bzl
@@ -0,0 +1,148 @@
+"""Protocol compiler build and integrity metadata.
+
+Generated and updated by scripts/update_protoc_integrity.py.
+"""
+
+PROTOC_RELEASES_URL = "https://github.com/protocolbuffers/protobuf/releases"
+PROTOC_DOWNLOAD_URL = (
+    PROTOC_RELEASES_URL +
+    "/download/v{version}/protoc-{version}-{platform}.zip"
+)
+
+PROTOC_VERSIONS = [
+    "29.3",
+    "29.2",
+    "29.1",
+    "29.0",
+    "28.3",
+    "28.2",
+]
+
+PROTOC_BUILDS = {
+    "linux-aarch_64": {
+        "exec_compat": [
+            "@platforms//os:linux",
+            "@platforms//cpu:aarch64",
+        ],
+        "integrity": {
+            "29.3": "sha256-ZCc0kUDgHwbgSecHpYcJpPIhrnOrmgQlvEoAyNDhqzI=",
+            "29.2": "sha256-Kc9IPi+yGCfl+sSWTjXq5HKiOOKMdi8C+xfc2T/4uJ8=",
+            "29.1": "sha256-H3Sj8zVd58Bma8ElYRwTUywlmPhTUh0NPmIaWwnyR5k=",
+            "29.0": "sha256-MF8b5a57LzlFGHCzErRcHguiaZAcg7oW2F+fnRRBs0g=",
+            "28.3": "sha256-HeUiAyqLGUAC/jXKuG10eEgji15N5PmWSDcgefW0b5o=",
+            "28.2": "sha256-kdglPNwPDw/FHCtpyAZ3mWYy9SWthFBL+ltO44rT5Jw=",
+        },
+    },
+    "linux-ppcle_64": {
+        "exec_compat": [
+            "@platforms//os:linux",
+            "@platforms//cpu:ppc64le",
+        ],
+        "integrity": {
+            "29.3": "sha256-DpiU7C45krFNGD586sFkZdam7nPh0jRpXYDm0elHAUw=",
+            "29.2": "sha256-uiCJWht/NKb/ql5Rw0ExbErrxMFEN3hjQITpn262f/k=",
+            "29.1": "sha256-B1vWZq1B60BKkjv6+pCkr6IHSiyaqLnHfERF5hbo75s=",
+            "29.0": "sha256-EJAnjNB1e3AsNrA+6t9KvTYCAnm7B7DfX4C9iW8+IDM=",
+            "28.3": "sha256-dSKdPN5z5wYZcXgU9R+m9K16NiwzUe5ZGXuxV8sAgsY=",
+            "28.2": "sha256-xcFrR2f/iGYJDuEOhwkDABBSAbXbHs7zgO3XPCM03Ng=",
+        },
+    },
+    "linux-s390_64": {
+        "exec_compat": [
+            "@platforms//os:linux",
+            "@platforms//cpu:s390x",
+        ],
+        "integrity": {
+            "29.3": "sha256-Y3hX/bqwsTNL2ysIcz8L5JaF5pMBG2EEgJSRrGL71NU=",
+            "29.2": "sha256-LwpVmdprgpMqCNQz+nkTux1oWIl8zTxrwyhvAYt2xOA=",
+            "29.1": "sha256-J5fNVlyn/7/ALaVsygE7/0+X5oMkizuZ3PfUuxf27GE=",
+            "29.0": "sha256-LhXZqwaFbCXKbeYi4RR4XZGHHb25HCwQPrYouH8m3Ew=",
+            "28.3": "sha256-jhtvqCX7CVlqiS5d6bgRLXoJtL3ftxzLv1+GgKT1aKc=",
+            "28.2": "sha256-ESIsQ4+G6Hsv2vaKKmzB2ytiKBP9btiRUJ4vOw4F7hs=",
+        },
+    },
+    "linux-x86_32": {
+        "exec_compat": [
+            "@platforms//os:linux",
+            "@platforms//cpu:x86_32",
+        ],
+        "integrity": {
+            "29.3": "sha256-VGzx5pHOc/ZuKZzCLN1aF8YaEf5LbV34UHUHg4nx/Ck=",
+            "29.2": "sha256-FU+NR+6YO8bxoa4tsUl+53E8Qt7vY1EXDxhmG/vNouw=",
+            "29.1": "sha256-nd/EAbEqC4dHHg4POg0MKpByRj8EFDUUnpSfiiCCu+s=",
+            "29.0": "sha256-tKyBCfKrSGLV5WuH4/cVMPug46YeyLnZQOKUdskGAQE=",
+            "28.3": "sha256-DJ6zLLnl06rHLGfPc38In+Mr8kVssBc+EpWQfNGIXhI=",
+            "28.2": "sha256-ucjToo5Lq5WcwQ5smjjxfFlGc3Npv+AT9RqG19Ns//E=",
+        },
+    },
+    "linux-x86_64": {
+        "exec_compat": [
+            "@platforms//os:linux",
+            "@platforms//cpu:x86_64",
+        ],
+        "integrity": {
+            "29.3": "sha256-PoZmIMW+J2ZPPS+i1la18+CbUVK0Lxvtv0J7Mz6QAho=",
+            "29.2": "sha256-Uunn7OVcfjDn6LvSVLSyG0CKUwm8qCZ2PHEktpahMuk=",
+            "29.1": "sha256-AMg/6XIthelsgblBsp8Xp0SzO0zmbg8YAJ/Yk33iLGA=",
+            "29.0": "sha256-PFEGWvO5pgbZ4Yob9igUNzT/S55pcl1kWYV0MLp6eN8=",
+            "28.3": "sha256-CtlJ8EpqF02oPNy9s23uCkklJypbbYP3mmv5hSB21T8=",
+            "28.2": "sha256-L+v9QrWc6Too63iQGaRwo90ESWGbwE+E2tEzPaJh3sE=",
+        },
+    },
+    "osx-aarch_64": {
+        "exec_compat": [
+            "@platforms//os:osx",
+            "@platforms//cpu:aarch64",
+        ],
+        "integrity": {
+            "29.3": "sha256-K4o0A80Jf5XzumVuFLdscytrJtfxgzMLEeNu8rwCh2U=",
+            "29.2": "sha256-DhU6ONbaGVlMmA5/fNPqDd1SydoQaMA8DYUzNp+/6yA=",
+            "29.1": "sha256-uP1ZdpJhmKfE6lxutL94lZ1frtJ7/GGCVMqhBD93BEU=",
+            "29.0": "sha256-srWfA7AwyKdIYj1oKotbycwJnkvP0GuJZM6J7AZbMQM=",
+            "28.3": "sha256-ks7v2mpyk+wBTm7KyC1kcZNXFFy2/ChlutreteYsBDE=",
+            "28.2": "sha256-e7BI9ShBeJ2exhmDvgzkyeT7O9mhQ0YoILqaO+CgN5c=",
+        },
+    },
+    "osx-x86_64": {
+        "exec_compat": [
+            "@platforms//os:osx",
+            "@platforms//cpu:x86_64",
+        ],
+        "integrity": {
+            "29.3": "sha256-mniANtj5hU97A8MF30d3zw5U5bCB4lvxUlLah+DpCHU=",
+            "29.2": "sha256-uivZg7XwbsONZjtgKISll96jmQpDgD1+FT7Y98VCaeE=",
+            "29.1": "sha256-2wK0uG3k1MztPqmTQ0faKNyV5/OIY//EzjzCYoMCjaY=",
+            "29.0": "sha256-56HP/ILiHapngzARRJxw3f8eujsRWTQ4fm6BQe+rCS8=",
+            "28.3": "sha256-l/5dRCCQtNvCPNE4T7m0RPodxuZ9FbteH+TeDadjiyA=",
+            "28.2": "sha256-Iy8H0Sv0gGIHp57CxzeDAcUuby9+/dIcDdQW8L2hA+w=",
+        },
+    },
+    "win32": {
+        "exec_compat": [
+            "@platforms//os:windows",
+            "@platforms//cpu:x86_32",
+        ],
+        "integrity": {
+            "29.3": "sha256-x8gCjBxNgBxTYCkg8shokgVAhr2WW2sjpLqV0hHcsdQ=",
+            "29.2": "sha256-73CfcaUbOompsm3meCBc7zyV4h0IGLFn+/WwOackr9E=",
+            "29.1": "sha256-EQXg+mRFnwsa9e5NWHfauGTy4Q2K6wRhjytpxsOm7QM=",
+            "29.0": "sha256-154nzOTEAXRUERc8XraBFsPiBACxzEwt0stHeneUGP4=",
+            "28.3": "sha256-sI/m/M9DE+LMxv1ybchV7okFM+JH9MVDyYf/2YDP1bI=",
+            "28.2": "sha256-V6hpbqvtUgl19PpGkUGwuC+iEIYj0YtyOqi+yae0u1g=",
+        },
+    },
+    "win64": {
+        "exec_compat": [
+            "@platforms//os:windows",
+            "@platforms//cpu:x86_64",
+        ],
+        "integrity": {
+            "29.3": "sha256-V+pZ6fVRrY1x/6qbXPvgyh9Ocglyodt+wtEqtEv/k4M=",
+            "29.2": "sha256-Weph77JLnYohQXHiyj/sVcPxUX7/BnZWyHXYoc0Gzk8=",
+            "29.1": "sha256-fqSCJYV//BIkWIwzXCsa+deKGK+dV8BSjMoxk+M26c4=",
+            "29.0": "sha256-0DuSGYWLikyogGO3i/Clzec7UYCLkwxLZvBuhILDq+Y=",
+            "28.3": "sha256-zmT0m97d70nOS9MTqPWbz5L89ntYMe+/ZhcDhtLmaUg=",
+            "28.2": "sha256-S94ZJx7XyrkANXDyjG5MTXGWPq8SEahr87sl2biVF3o=",
+        },
+    },
+}