Create an alternate SwiftPM build system called Swift Build #8271

cmcgee1024 · 2025-02-01T13:55:08Z

Add basic test coverage for the new build system for the supported platforms: Linux, macOS, and Windows. Provide basic progress reporting, and an option for "very verbose" mode so that that individual build tasks and the arguments can be inspected for correctness.

Build a package with the new build system like this:

swift build --build-system=swiftbuild

Enable very verbose task outputs with their flags, options, and arguments:

swift build --build-system=swiftbuild --vv

It was only active on macOS and Linux but not other platforms due to existing dependency complexities, and just isn't that useful overall. This will help unblock swiftlang/swift-package-manager#8271 without having to add new dependencies to the Swift Docker images.

rauhul · 2025-02-01T14:54:31Z

Not a review comment, but I can't wait for this to land :D this is awesome!

It was only active on macOS and Linux but not other platforms due to existing dependency complexities, and just isn't that useful overall. This will help unblock swiftlang/swift-package-manager#8271 without having to add new dependencies to the Swift Docker images.

dschaefer2 · 2025-02-01T18:34:34Z

I just tried this out and get:

error: Dependencies could not be resolved because root depends on 'swift-argument-parser' 1.4.0..<1.5.0 and 'swift-driver' depends on 'swift-argument-parser' 1.2.2..<1.3.0.

dschaefer2 · 2025-02-01T18:36:53Z

Never mind, had a really old version of the deps. Did a swift package update and it's going now.

dschaefer2 · 2025-02-01T19:02:32Z

Build works on my Mac. A few weird warnings. Also looks like swift-test has hardcoded the location of the xctest bundle to the old scratch dir (.build) layout. Lots of work to do indeed :).

Thanks @cmcgee1024. This is a great start!

bkhouri

I have not reviewed this extensively, however there are a few minor comments.

bkhouri · 2025-02-03T18:36:57Z

Sources/_InternalTestSupport/misc.swift

+        Xswiftc: Xswiftc,
+        buildSystem: buildSystem
+    )
+    let buildArgs = getBuildSystemArgs(for: buildSystem)


issue (non-blocking): this call is not needed as the build system argument is handled in swiftArgs()

bkhouri · 2025-02-04T15:13:40Z

Sources/XCBuildSupport/XcodeBuildSystem.swift

@@ -230,6 +230,8 @@ public final class XcodeBuildSystem: SPMBuildCore.BuildSystem {

            throw Diagnostics.fatalError
        }
+
+        print("Build complete!")


question: are changes to this file necessary?

Also, this should use existing logging facilities that respect -v/-q settings.

I've converted this over to use an observability scope emit. One problem though, is that we probably want this output to be emitted in all log levels, but I don't want to indicate that it is an error.

bkhouri · 2025-02-04T15:15:18Z

Sources/SwiftBuildSupport/SwiftBuildSystem.swift

+                        if report {
+                            progressAnimation.complete(success: false)
+                            // FIXME: a whole range of pretty printed location, context, and fixups
+                            print("\(info.kind): \(info.location) \(info.message) \(info.fixIts)")


suggestion (possibly-blocking): Instead of a print call, can we use the observabilityScope.emit(...) calls?

This comment applies to everywhere we are using print calls.

I agree, we should avoid using print and rely on existing logging facilities instead, especially since those can propagate diagnostic information like message level and location already.

I've converted the print calls to observability scopes. See my note above about what to do about certain kinds of messages like "Build complete!"

MaxDesiatov

I am concerned by uses of print here. We have commands like swift-run that are supposed to only print freshly-built product's output. The rest of diagnostic messages even successful ones like "Build complete" should go to stderr, since there is an existing body of scripts and utilities that read from swift-run invocations stdout via a pipe. Stray print polluting such output with diagnostics will break those scripts.

Tests/CommandsTests/BuildCommandTests.swift

cmcgee1024 · 2025-02-05T15:36:14Z

@swift-ci please test

Tests/CommandsTests/BuildCommandTests.swift

MaxDesiatov

IMO removing existing expectations for cross-compilation for existing build systems is a significant regression in the test suite. I'm totally ok with merging Swift Build support if it can't cross-compile yet, but it shouldn't come at the expense of reducing the test coverage for other build systems.

cmcgee1024 · 2025-02-05T18:08:15Z

IMO removing existing expectations for cross-compilation for existing build systems is a significant regression in the test suite. I'm totally ok with merging Swift Build support if it can't cross-compile yet, but it shouldn't come at the expense of reducing the test coverage for other build systems.

This is an issue with the CI infrastructure using an old Xcode that can't handle Swift 6, which is required by the SwiftBuild code. I will ask about getting the CI updated with a newer version.

cmcgee1024 · 2025-02-13T22:44:55Z

Please test with following pull request:
apple/swift-crypto#326
apple/swift-asn1#88

@swift-ci test windows

There are a few places that check the BuildSystemProvider.Kind is Xcode to determine the build directory structure. With the upcoming Swift Build integration, which uses the "Xcode path" structure, we would need to update all instances to check the two build system provider kinds. Add an extension on the BuildSystemProvider.Kind that create a boolean variable `useXcodeBuildSystemPath`, which determines whether the build system should use the xcode path structure or not. In addition, update the code that requires a "isXcodeBuildSystemEnabled" to return this build system provider variable. This will hopefully help address swiftlang#8272 when swiftlang#8271, where it adds `case .swiftbuild: return true` to the `useXcodeBuildSystemPath`

There are a few places that check the BuildSystemProvider.Kind is Xcode to determine the build directory structure. With the upcoming Swift Build integration, which uses the "Xcode path" structure, we would need to update all instances to check the two build system provider kinds. Add an extension on the BuildSystemProvider.Kind that create a boolean variable `useXcodeBuildEngine`, which determines whether the build system should use the xcode path structure or not. In addition, update the code that requires a "isXcodeBuildSystemEnabled" to return this build system provider variable. This will hopefully help address swiftlang#8272 when swiftlang#8271, where it adds `case .swiftbuild: return true` to the `useXcodeBuildEngine`

cmcgee1024 · 2025-02-20T11:49:25Z

Please test with following pull request:
apple/swift-crypto#326

@swift-ci please test

There are a few places that check the BuildSystemProvider.Kind is Xcode to determine the build directory structure. With the upcoming Swift Build integration, which uses the "Xcode path" structure, we would need to update all instances to check the two build system provider kinds. Add an extension on the BuildSystemProvider.Kind that create a boolean variable `useXcodeBuildEngine`, which determines whether the build system should use the xcode path structure or not. In addition, update the code that requires a "isXcodeBuildSystemEnabled" to return this build system provider variable. This will hopefully help address swiftlang#8272 when swiftlang#8271, where it adds `case .swiftbuild: return true` to the `useXcodeBuildEngine`

dschaefer2 · 2025-02-26T04:24:09Z

@swift-ci please test

There are a few places that check the BuildSystemProvider.Kind is Xcode to determine the build directory structure. With the upcoming Swift Build integration, which uses the "Xcode path" structure, we would need to update all instances to check the two build system provider kinds. Add an extension on the BuildSystemProvider.Kind that create a boolean variable `useXcodeBuildEngine`, which determines whether the build system should use the xcode path structure or not. In addition, update the code that requires a "isXcodeBuildSystemEnabled" to return this build system provider variable. This will hopefully help address #8272 when #8271 is merged and added `case .swiftbuild: return true` to the `useXcodeBuildEngine ` Fixes #8272 rdar://144023142

dschaefer2 · 2025-02-27T03:05:47Z

DriverSupport needs swift-driver always independent of swift-build.

owenv · 2025-02-27T18:12:50Z

Sources/PackageGraph/ResolvedPackagesStore.swift

@@ -560,7 +560,12 @@ extension ResolvedPackagesStore.ResolvedPackage {
        let location = mirrors.effective(for: pin.location)
        switch pin.kind {
        case .localSourceControl:
-            packageRef = try .localSourceControl(identity: identity, path: AbsolutePath(validating: location))
+            // FIXME investigate why local source control packages resolve to a non-local URL location


Is this change intended to be part of the PR?

This could be an artifact of one of the previous approaches to getting the CI working. I will try disable it and see if everything continues to succeed.

Package.swift

dschaefer2 · 2025-02-27T19:03:36Z

Package.swift

        .package(url: "https://github.com/apple/swift-crypto.git", .upToNextMinor(from: "3.0.0")),
        .package(url: "https://github.com/swiftlang/swift-syntax.git", branch: relatedDependenciesBranch),
        .package(url: "https://github.com/apple/swift-system.git", from: "1.1.1"),
        .package(url: "https://github.com/apple/swift-collections.git", "1.0.1" ..< "1.2.0"),
        .package(url: "https://github.com/apple/swift-certificates.git", "1.0.1" ..< "1.6.0"),
        .package(url: "https://github.com/swiftlang/swift-toolchain-sqlite.git", from: "1.0.0"),
+        .package(url: "https://github.com/apple/swift-driver.git", branch: relatedDependenciesBranch),


This should come from swiftlang.

I've basically undone this diff, so it is the same as what is currently in main.

cmcgee1024 · 2025-02-27T20:08:21Z

@swift-ci test windows

cmcgee1024 · 2025-02-27T20:55:33Z

@swift-ci test macOS

owenv · 2025-02-28T01:12:50Z

@swift-ci build toolchain

Add basic test coverage for the new build system for the supported platforms: Linux, macOS, and Windows. Provide basic progress reporting, and an option for "very verbose" mode so that that individual build tasks and the arguments can be inspected for correctness. Build a package with the new build system like this: swift build --build-system=swiftbuild Enable very verbose task outputs with their flags, options, and arguments: swift build --build-system=swiftbuild --vv

cmcgee1024 · 2025-02-28T14:57:34Z

@swift-ci please test

bkhouri · 2025-02-28T15:44:59Z

Utilities/build-using-self

@@ -138,6 +138,9 @@ def main() -> None:
            shlex.split("swift --version"),
        )

+        call(


Out of curiosity, why was this added it? Are previous CI builds impacting the current PR build?

This was added because the package may have been built earlier using the bootstrap with the environment variable set to exclude swift-build as a dependency. This is one of the issues with using environment variables in packages. If you flip them then SwiftPM may not be aware of the change and produce inconsistent build results since its build graph is not obviously out of sync with the current package state.

I believe that adding this will force the package to be entirely reset, and cleaned before building it again with swift-build support.

…lang#8298) There are a few places that check the BuildSystemProvider.Kind is Xcode to determine the build directory structure. With the upcoming Swift Build integration, which uses the "Xcode path" structure, we would need to update all instances to check the two build system provider kinds. Add an extension on the BuildSystemProvider.Kind that create a boolean variable `useXcodeBuildEngine`, which determines whether the build system should use the xcode path structure or not. In addition, update the code that requires a "isXcodeBuildSystemEnabled" to return this build system provider variable. This will hopefully help address swiftlang#8272 when swiftlang#8271 is merged and added `case .swiftbuild: return true` to the `useXcodeBuildEngine ` Fixes swiftlang#8272 rdar://144023142

cmcgee1024 · 2025-02-28T23:47:37Z

@swift-ci please test

cmcgee1024 · 2025-03-01T13:46:43Z

@swift-ci test Linux

cmcgee1024 · 2025-03-01T13:46:54Z

@swift-ci test Windows

bkhouri · 2025-03-02T14:39:30Z

Sources/SPMBuildCore/BuildSystem/BuildSystem.swift

@@ -172,6 +173,7 @@ extension BuildSystemProvider.Kind {
    public var usesXcodeBuildEngine: Bool {
        switch self {
            case .native: return false
+            case .swiftbuild: return false


Chore (blocking): this should return "true"

SwiftBuild isn't using any Xcode or its build engine. It's unclear what this check is for.

The variable name may be incorrect, but this is meant to determine if the underlying build engine uses the same "build output directory structure" as Xcode.

Setting this to true will likely resolve #8272

SwiftBuild doesn't use the same directory structure.

bkhouri · 2025-03-02T14:41:15Z

Sources/SwiftBuildSupport/BuildSystem.swift

+extension BuildConfiguration {
+    public var swiftbuildName: String {
+        switch self {
+        case .debug: "Debug"


Question: can we have the build configuration confirm to "ConvertibleString" ?

We can fix the swift test in a subsequent change

cmcgee1024 requested review from bnbarham, MaxDesiatov, jakepetroules, xedin, dschaefer2, shawnhyam, bripeticca, plemarquand, owenv and bkhouri as code owners February 1, 2025 13:55

jakepetroules approved these changes Feb 1, 2025

View reviewed changes

jakepetroules mentioned this pull request Feb 1, 2025

Remove the dependency on libreadline swiftlang/swift-build#34

Merged

dschaefer2 approved these changes Feb 1, 2025

View reviewed changes

swiftlang deleted a comment from jakepetroules Feb 1, 2025

bkhouri mentioned this pull request Feb 4, 2025

Tests: Augment Command Tests #8276

Draft

bkhouri reviewed Feb 4, 2025

View reviewed changes

MaxDesiatov reviewed Feb 5, 2025

View reviewed changes

bkhouri reviewed Feb 5, 2025

View reviewed changes

Tests/CommandsTests/BuildCommandTests.swift Show resolved Hide resolved

MaxDesiatov reviewed Feb 5, 2025

View reviewed changes

Tests/CommandsTests/BuildCommandTests.swift Outdated Show resolved Hide resolved

MaxDesiatov previously requested changes Feb 5, 2025

View reviewed changes

bkhouri mentioned this pull request Feb 19, 2025

build: define build path structure in BuildSystemProvider.Kind #8298

Merged

owenv reviewed Feb 27, 2025

View reviewed changes

owenv approved these changes Feb 27, 2025

View reviewed changes

dschaefer2 reviewed Feb 27, 2025

View reviewed changes

Package.swift Show resolved Hide resolved

dschaefer2 reviewed Feb 27, 2025

View reviewed changes

owenv mentioned this pull request Feb 28, 2025

DNM - toolchain testing swiftlang/swift#79684

Open

cmcgee1024 force-pushed the swiftbuild_integration branch from e069aa4 to 60b1702 Compare February 28, 2025 14:56

bkhouri reviewed Feb 28, 2025

View reviewed changes

Restore local source control package resolution

1ff2e2a

Remove fixme

2712e9e

bkhouri previously requested changes Mar 2, 2025

View reviewed changes

dschaefer2 merged commit af93a3c into swiftlang:main Mar 3, 2025
5 checks passed

Create an alternate SwiftPM build system called Swift Build #8271

Create an alternate SwiftPM build system called Swift Build #8271

Conversation

cmcgee1024 commented Feb 1, 2025

rauhul commented Feb 1, 2025

dschaefer2 commented Feb 1, 2025

dschaefer2 commented Feb 1, 2025

dschaefer2 commented Feb 1, 2025

bkhouri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxDesiatov Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxDesiatov Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxDesiatov left a comment • edited Loading

Choose a reason for hiding this comment

cmcgee1024 commented Feb 5, 2025

MaxDesiatov left a comment • edited Loading

Choose a reason for hiding this comment

cmcgee1024 commented Feb 5, 2025

cmcgee1024 commented Feb 13, 2025

cmcgee1024 commented Feb 20, 2025

dschaefer2 commented Feb 26, 2025

dschaefer2 commented Feb 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmcgee1024 commented Feb 27, 2025

cmcgee1024 commented Feb 27, 2025

owenv commented Feb 28, 2025

cmcgee1024 commented Feb 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmcgee1024 commented Feb 28, 2025

cmcgee1024 commented Mar 1, 2025

cmcgee1024 commented Mar 1, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkhouri Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxDesiatov Feb 5, 2025 •

edited

Loading

MaxDesiatov Feb 5, 2025 •

edited

Loading

MaxDesiatov left a comment •

edited

Loading

MaxDesiatov left a comment •

edited

Loading

bkhouri Mar 3, 2025 •

edited

Loading