Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build ksh 1.0.10 with MSYS2/MinGW #801

Open
cedricblancher opened this issue Nov 25, 2024 · 15 comments
Open

Cannot build ksh 1.0.10 with MSYS2/MinGW #801

cedricblancher opened this issue Nov 25, 2024 · 15 comments
Labels
buildfail Does not compile

Comments

@cedricblancher
Copy link

We cannot build ksh 1.0.10 with MSYS2/MinGW, bin/package fails like this:
$ (SHELL=/bin/dash ; $SHELL ./bin/package make)
package: /home/ced/mingw_ksh93/ksh: must be in the package root directory tree

@cedricblancher
Copy link
Author

Still does not work:

$ export SHELL=/bin/bash.exe ; $SHELL ./bin/package make CC="/usr/bin/gcc -m64 -std=gnu17" PACKAGEROOT="$PWD" HOSTTYPE="msys.i386-64"
package: initialize the /home/ced/work/mingw_ksh93/ksh/arch/msys.i386-64 view
package: update /home/ced/work/mingw_ksh93/ksh/arch/msys.i386-64/bin/mamprobe
package: update /home/ced/work/mingw_ksh93/ksh/arch/msys.i386-64/lib/probe/C/make/probe
package: update /home/ced/work/mingw_ksh93/ksh/arch/msys.i386-64/bin/mamake
cc: fatal error: cannot execute 'cc1': spawn: No such file or directory
compilation terminated.

@McDutchie
Copy link

Thanks for the report. MSYS2/MinGW is not a system I've ever used. I generally try to avoid Windows as much as possible, but I'm going to have to figure out a way to test it on this system and make it build.

FWIW, Cygwin should work, it's tested every now and then.

@McDutchie McDutchie added the buildfail Does not compile label Nov 25, 2024
@gisburn
Copy link

gisburn commented Nov 26, 2024

@McDutchie Quick setup via Cygwin shell:
---- snip ----

wget 'https://github.com/msys2/msys2-installer/releases/download/2024-11-16/msys2-x86_64-20241116.exe'
chmod a+x msys2-x86_64-20241116.exe 
./msys2-x86_64-20241116 

---- snip ----
... then in MSYS2 UCRT terminal:
---- snip ----

pacman -S --noconfirm gcc emacs gdb make gettext gettext-devel git subversion flex bison unzip pax tar libiconv-devel ncurses-devel gmp-devel mpfr-devel mpc-devel isl-devel  isl-devel procps-ng libiconv-devel sed time coreutils util-linux grep sed clang base-devel

---- snip ----

Does that help ?

@cedricblancher
Copy link
Author

@McDutchie Any updates?

@McDutchie
Copy link

No one is currently getting paid for developing ksh, and I don't know when I'll have the time, opportunity and inclination to sit down with a Windows 10 box and figure this out. I don't have one in the house. So, unless someone else figures this out before me, you're going to have to be patient.

@xeiavicaflashrepository
Copy link

@cedricblancher
Hi, I'm trying to build ksh93 for msys2 myself. I got a kinda hacky workaround, which fixes the cc1 issue, only to run into a new issue with forking the process to build libsum after libast is done.
You can see it here:

msys2/MSYS2-packages#5243

@gisburn
Copy link

gisburn commented Mar 9, 2025

I'm also interested. Where can I find your script you used to try to build ksh93 ?

@xeiavicaflashrepository

No matter what, it fails on windows because of fork(). Does mamake like any other make, have ability to resume if it failed during compilation? Asking because as I understand, this is a series of shell scripts. Same error too (minus pid number):
0 [main] mamake 1763 dofork: child -1 - forked process 11484 died unexpectedly, retry 0, exit code 0xC0000142, errno 11

@xeiavicaflashrepository

@McDutchie
Good news, I managed to successfully build for msys2 with my PKGBUILD script in the linked merge request. It can build. You just have to run the build command twice, because about half way through it WILL fail because of fork issue. Should this issue be closed now?

@cedricblancher
Copy link
Author

@McDutchie Good news, I managed to successfully build for msys2 with my PKGBUILD script in the linked merge request. It can build. You just have to run the build command twice, because about half way through it WILL fail because of fork issue. Should this issue be closed now?

Could you please send a stand alone script which can be used to build ksh93 on msys2? Just a shell script, without the whole PKGBUILD machinery.

@xeiavicaflashrepository

@McDutchie Good news, I managed to successfully build for msys2 with my PKGBUILD script in the linked merge request. It can build. You just have to run the build command twice, because about half way through it WILL fail because of fork issue. Should this issue be closed now?

Could you please send a stand alone script which can be used to build ksh93 on msys2? Just a shell script, without the whole PKGBUILD machinery.

Sure. Though it is optional, you might want to patch in this patch I used from gisburn's repo linked.

ksh-unc-paths.patch

I wasn't sure where you wanted it to be installed, so I left that part out. If you have problems with the script, let me know.

#!/bin/sh

# This script assumes you're using gcc.
# Optional: Apply the supplied patch file. Your choice.
#
# A few environmental variables need to be set:
export GCC_DIR="$(gcc -print-search-dirs | sed -ne '/^install: /s///p')" # To fix the cc1.exe not found issue.
export PATH="$PATH:$GCC_DIR"

# A few tweaks to the other environmental variables. To fix other issues that will arise.
export CFLAGS="-I${GCC_DIR}/include -L${GCC_DIR} -L/usr/lib"
export CCFLAGS="${CFLAGS} -fno-use-linker-plugin"
export LIBRARY_PATH="${GCC_DIR}"

# These next two sed commands are necessary to ensure it builds right.
# First one, is to add a shebang to a file because windows can't run it properly without it.
sed -i '1s/^/#!\/usr\/bin\/bash\n/' src/cmd/INIT/mamprobe.sh
sed -i '6s/^/#undef _lib_iconv_open\n/' src/lib/libast/features/iconv # Without this, fails with libiconv installed

# This will fail the first time, this is to be expected and will then run again.
CC="/usr/bin/gcc" /usr/bin/bash bin/package make PACKAGEROOT="$PWD" || true

# Run it again.
CC="/usr/bin/gcc" /usr/bin/bash bin/package make PACKAGEROOT="$PWD" || true

@xeiavicaflashrepository

Though I'm wondering: Why did you not want makepkg? You just run makepkg as long as the very few dependencies are installed like gcc and makepkg itself.

@cedricblancher
Copy link
Author

Though I'm wondering: Why did you not want makepkg? You just run makepkg as long as the very few dependencies are installed like gcc and makepkg itself.

  1. Because I want to do ksh93 development on MSYS2
  2. I've been bugging Roland Mainz to add MSYS2 support to one of the ms-nfs41-client CI test scripts (https://github.com/kofemann/ms-nfs41-client/blob/master/tests/nfsbuildtest/nfsbuildtest.ksh93), which simply runs builds of bash, gcc, ksh93 on Windows NFSv4.2 filesystem mount. ksh93 didn't work so far, and Roland did not had time. If your script works, then we have a solution.

@jeremyd2019
Copy link

jeremyd2019 commented Mar 13, 2025

There was a bit going on, so I'll try to summarize some of the craziness. First, MSYS2 essentially is Cygwin (at the lowest level, it's a fork of Cygwin with some additional patches). Now, on to what's necessary to get ksh 1.0.10 to build on it

  1. /bin and /usr/bin are the same directory. Think of it as a bind mount in Linux terms. Gcc tries to be "relocatable" by finding its files relative to its binary. This usually works OK, /usr/bin/gcc -> /usr/bin/../lib/gcc/. However, the bin/package script decided to put /bin before /usr/bin in the PATH. Now /bin/gcc -> /bin/../lib/gcc doesn't work, because /lib is not a bind mount to /usr/lib. This is the cause of the cc1 error.
    • To work around this, I added a hack:
      @@ -1792,6 +1793,12 @@
              esac
       fi
      
      +if test /bin -ef /usr/bin; then
      +       # /bin and /usr/bin are the same - but MSYS2 gcc dies badly if invoked
      +       # as /bin/cc, so make sure /usr/bin is first
      +       DEFPATH="/usr/bin:$DEFPATH"
      +fi
      +
       export DEFPATH  # for iffe, etc.
      
       PATH=$(sanitize_PATH "/opt/ast/bin:$DEFPATH:$PATH")
    • /bin to /usr/bin questions msys2/msys2-runtime#266
  2. The package script is opinionated about settings of the CYGWIN environment variable. On MSYS2, that variable has been renamed to MSYS. (there has been some discussion about switching this back). Regardless, even on upstream Cygwin, the options this script insists on are obsolete and no longer have any effect. I set CYGWIN=ntsec in the build script just to shut it up, but I'd recommend removing that check from the package script here.
  3. The feature that ntsec used to control is now a per-mount acl/noacl option. MSYS2 has that set to noacl. This means the issues that made it want ntsec in the first place are back in play. Most importantly, in noacl mode Cygwin uses a heuristic to determine if a file should be considered executable or not. Various scripts in this repository fail that heuristic, because they are missing shebang lines. In order to get a successful build, we needed to add a shebang to bin/package and src/cmd/INIT/mamprobe.sh. This is also why setting CC to /usr/bin/gcc didn't work around the first issue: when you set CC to anything other than cc, package writes a wrapper script into $INSTALLROOT/bin which it puts first on the PATH. Unfortunately, it doesn't include a shebang, so MSYS2 considers it not executable, and it ends up finding /bin/cc next on the PATH instead. Another, less critical fallout of the lack of real permission bits is that src/cmd/ksh93/tests/shtests fails due to mkdir -m failing. I patched the -m arguments out to at least run some tests.
  4. Due to how fork is emulated in Cygwin, fork will fail if the executable for a running process is replaced and it subsequently tries to fork. This happens to mamake. There's already a workaround present for this for Cygwin, but due to different defaults in MSYS2 this fails to help. The following patch makes it work there too
    --- a/bin/package	Thu Aug 01 18:54:10 2024
    +++ b/bin/package	Mon Mar 10 20:42:45 2025
    @@ -2464,7 +2464,7 @@
     				# To allow relinking the binary against libast later, add a _bootstrap suffix and create
     				# a symlink to it. The symlink can be removed/renamed instead of the _bootstrap binary.
    				$exec $CC -O $CCFLAGS $LDFLAGS -o "$INSTALLROOT/bin/${i}_bootstrap" "$INITROOT/$i.c" || return
    -				$exec ln -sf "${i}_bootstrap" "$INSTALLROOT/bin/$i" || return
    +				MSYS="$MSYS winsymlinks:sys" $exec ln -sf "${i}_bootstrap" "$INSTALLROOT/bin/$i" || return
    				;;
     			*)	$exec $CC -O $CCFLAGS $LDFLAGS -o "$INSTALLROOT/bin/$i" "$INITROOT/$i.c" || return
     				;;

Lastly, there is a difference of opinion on the pull request to add ksh93 as a package in MSYS2 on whether we should override a bunch of defaults (such as changing /opt/ast/bin to /usr/ast/bin, and shipping along a custom header enabling & disabling different builtins). No Linux distribution I've checked does this, and I'm of the opinion that we should follow upstream here. Not least of which, scripts and tests in this repo hard-code /opt/ast/bin.

@xeiavicaflashrepository

@cedricblancher
I have very good news, If my MR gets approved, you won't even need my older shell script anymore. I'm aiming to make it a single command to compile ksh93 on msys2.
When this gets merged, you can just run bash bin/package PACKAGEROOT="$PWD" and you're good to go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
buildfail Does not compile
Projects
None yet
Development

No branches or pull requests

5 participants