See Cython and C++ compiler console output in editable install

[joshayers]

I have a project that uses Cython and C++. Is it possible to see the console output from Cython and the C++ compiler? It often contains helpful warning messages. It's not shown in the console (even in verbose mode) and I couldn't find it in any of the logs in the build directory.

For reference, I removed the types from a loop variable in a Cython file. I know Cython issues a warning in that case, but this image is all the output that is shown.

This is similar to discussion 505. In that case, the person was looking for the compiler flags, while I would like the output.

Thanks in advance for any help!

[rgommers]

Hi @joshayers, the actual compilation is done by ninja, so you want to execute that in verbose mode. To do so, you need the -Ccompile-args=-v argument, e.g. to build a wheel in verbose mode with build:

python -m build -wnx -Ccompile-args=-v

This should show you output like this:

[304/305] arm64-apple-darwin20.0.0-clang -Inumpy/random/_generator.cpython-39-darwin.so.p -Inumpy/random -I../numpy/random -I../numpy/random/src -Inumpy/_core -I../numpy/_core -Inumpy/_core/include -I../numpy/_core/include -I../numpy/_core/src/common -Inumpy -I/Users/rgommers/mambaforge/envs/numpy-dev/include/python3.9 -I/Users/rgommers/code/numpy/.mesonpy-t8cm41h1/meson_cpu -fvisibility=hidden -fcolor-diagnostics -DNDEBUG -w -std=c99 -O3 -fno-strict-aliasing -ftrapping-math -Wno-unsupported-floating-point-opt -DNPY_HAVE_NEON_VFPV4 -DNPY_HAVE_NEON_FP16 -DNPY_HAVE_NEON -DNPY_HAVE_ASIMD -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DNPY_NO_DEPRECATED_API=0 -MD -MQ numpy/random/_generator.cpython-39-darwin.so.p/meson-generated_numpy_random__generator.pyx.c.o -MF numpy/random/_generator.cpython-39-darwin.so.p/meson-generated_numpy_random__generator.pyx.c.o.d -o numpy/random/_generator.cpython-39-darwin.so.p/meson-generated_numpy_random__generator.pyx.c.o -c numpy/random/_generator.cpython-39-darwin.so.p/numpy/random/_generator.pyx.c
[305/305] arm64-apple-darwin20.0.0-clang++  -o numpy/random/_generator.cpython-39-darwin.so numpy/random/_generator.cpython-39-darwin.so.p/meson-generated_numpy_random__generator.pyx.c.o -L/Users/rgommers/mambaforge/envs/numpy-dev/lib -Wl,-dead_strip_dylibs -Wl,-headerpad_max_install_names -Wl,-undefined,dynamic_lookup -bundle -Wl,-undefined,dynamic_lookup -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/rgommers/mambaforge/envs/numpy-dev/lib -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/rgommers/mambaforge/envs/numpy-dev/include numpy/random/libnpyrandom.a numpy/_core/libnpymath.a
ld: warning: -pie being ignored. It is only used when linking a main executable

[joshayers]

No, I don't see the warnings with either of those commands. When I build with setuptools I see them, but not with meson.

Let me know if there's any other info that would help diagnose.

[joshayers]

I tried running ninja directly, using the command that's created by meson. Here's the only output I get, even with the -v flag. No compiler warnings. Note that I ran it twice, first time without the -v flag and the second time with it.

When the compilation fails completely, those errors are shown, but warnings are being swallowed for some reason. It seems to be an issue in ninja, so I'll investigate there further. If you have any suggestions, that'd be appreciated. Thanks again!

[joshayers]

I found the problem. The build.ninja file included "-w" as an argument for all the compiler calls. That disables all warnings. I replaced it with "/Wall" and I get an entire screen-full of warnings.

I tried setting a different warning level in my top level meson.build file but it didn't change anything. What is the correct way to change this warning level?

project(
  'rs_example_lib', 'cython', 'cpp',
  license: 'Apache-2.0 or LGPL-2.1-or-later',
  meson_version: '>= 1.3.0',
  default_options: ['warning_level=2'],
)`

[rgommers]

Ah, that makes sense. Did you have it set to warning_level=0 explicitly then? The default is 1, and that has always worked well enough that I never found a reason to adjust it.

[joshayers]

No, I didn't have it set at all. Setting the warning level in the project section doesn't change anything. The -w suppress all warnings argument is still there.

I tried this to override it:

project(
  'rs_example_lib', 'cython', 'cpp',
  license: 'Apache-2.0 or LGPL-2.1-or-later',
  meson_version: '>= 1.3.0',
)

if meson.get_compiler('cpp').get_id() == 'msvc'
  warn_arg = '/W3'
else
  warn_arg = '-Wall'
endif
add_global_arguments(warn_arg, language: 'cpp')

It correctly adds the additional argument in the build.ninja file, but -w is still there. It works in MSVC (/W3 takes precedence over -w) but with gcc -w seems to take precedence over -Wall and warnings are still suppressed.

Here are the compiler command line arguments copied from build.ninja. Where do the default compiler args come from? Is there any method in meson.build to explicitly remove them?

MSVC:
ARGS = "-Irs_example_lib\frf.cp311-win_amd64.pyd.p" "-Irs_example_lib" "-I..\..\rs_example_lib" "-I..\..\.venv\Lib\site-packages\numpy\core\include" "-IC:\Program$ Files\Python311\Include" "-DNDEBUG" "/MD" "/nologo" "/showIncludes" "/utf-8" "/Zc:__cplusplus" "-w" "/EHsc" "/O2" "/Gw" "/W3" "-DMS_WIN64=" "-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION" "/Fdrs_example_lib\frf.cp311-win_amd64.pyd.p\meson-generated_rs_example_lib_frf.pyx.cpp.pdb"

GCC:
ARGS = -Irs_example_lib/frf.cpython-311-x86_64-linux-gnu.so.p -Irs_example_lib -I../../rs_example_lib -I../../.venv/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -I/usr/include/x86_64-linux-gnu/python3.11 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -w -O3 -Wall -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION

[dnicolodi]

I suspect that the -w comes from the compilation flags injected by one of the dependencies of your project. Without having access to a reproducer for the problem, I don't think we cannot help you further.

[joshayers]

@dnicolodi @rgommers Here's a minimal example that reproduces the problem. The only dependencies are NumPy, Cython, build, and Meson.

https://github.com/joshayers/meson_test

I tested it using Windows/MSVC and Linux/gcc. The build.ninja file included the "-w" flag that suppresses all compiler warnings in both cases.

Thanks again for your help. Let me know if you're not able to reproduce the problem or need more info.

[dnicolodi]

Turns out that Meson disables warnings when compiling the C or C++ code generated by Cython: https://github.com/mesonbuild/meson/blob/30184a48a032e6a2a6dbf0a76e437aa676c3e4aa/mesonbuild/backend/ninjabackend.py#L2830-L2832 The conditional there checks for Vala generated source code, but the Cython support is bolted on the side of the Vala support and the code does not distinguish which transpiler emitted the C code.

I am not sure whether getting the C compiler warnings for generated code makes sense or not.

[joshayers]

Cython produces a minimal number of warnings in my experience, at least at the default warning level.

A common use case for it is wrapping C or C++ code. I've always done that by compiling the Cython-generated code and the manually-written C or C++ into the same module. Here's one example of the same thing in NumPy: https://github.com/numpy/numpy/blob/023c65180cccd53108e1f2dc19fd44dd397ff655/numpy/random/meson.build#L59. It seems undesirable to hide warnings in that case.

Would it be possible to add a config option for this? Or detect that the compiler is Cython instead of Vala and not disable warnings in that case?

Thanks again.

[dnicolodi]

By the way, your test project specifies meson-python, meson, cython, and build as runtime dependencies. Most likely, you don't want to do that. The first three are only build time dependencies. build should not be listed as a dependency, not even as a build time one: users could use a different Python package build front-end, such as pip.

Also, you explicitly add the Python (dep_py = py.dependency()) a as dependency for the Python extension modules. This is not necessary since at least Meson 0.63.

[dnicolodi]

A common use case for it is wrapping C or C++ code. I've always done that by compiling the Cython-generated code and the manually-written C or C++ into the same module. Here's one example of the same thing in NumPy: https://github.com/numpy/numpy/blob/023c65180cccd53108e1f2dc19fd44dd397ff655/numpy/random/meson.build#L59. It seems undesirable to hide warnings in that case.

This use case is not affected. The warnings are suppressed only in the compilation of the generated C or C++ source code, not during the compilation of all the source code involved in producing a build artifact.

Would it be possible to add a config option for this?

I don't know how such an option would look like. Meson does not expose any facility to set compilation options per source file.

Or detect that the compiler is Cython instead of Vala and not disable warnings in that case?

I thought that this could be the solution. However, this would not be a backward compatible change: project that use -Werror would start to fail to compile if their Cython code emits a warning. In general, users do not have a way to resolve warnings emitted by the C or C++ compiler when compiling generated code. Therefore, it is of little use to emit warnings for that code, as they cannot be acted upon. As explained, emitting warnings for generated code is not compatible with using -Werror to catch warnings in parts of the code base on which the user has direct control.

@dcbaker @eli-schwartz opinions?

[eli-schwartz]

It's even more important to stop using -w in Vala targets than it is to stop using it in cython targets.

The number of extremely dangerous UB issues in Vala that are going unreported is much higher than in cython. It is not meson's job to lie to users and pretend that vala is a safe programming language (by any definition of language safety) when compilers are defaulting to telling you that the generated code violates all the semantic laws of the C programming language.

It is harmful to users.

[dnicolodi]

mesonbuild/meson#12596 Is a minimally invasive fix. It removes the warning suppression for Cython generated source code, while keeping it for Vala. I don't know enough about Vala to say whether suppressing the warnings is a good idea or not.

[eli-schwartz]

I'm already working on this via eli-schwartz/meson@2e8bb87 but I missed reverting a bit.

[eli-schwartz]

mesonbuild/meson#12597

[rgommers]

Thanks for fixing this!

Now that the fun is over, I realized why I do see Cython warnings in SciPy: we're always using generators to invoke cython, and then pass the resulting C/C++ files as regular source targets to py.extension_module, rather than passing that the .pyx sources. So we sidestepped this -w issue by accident.

How many cython users do we have, realistically? I think that number might be small enough that even if it's a breaking change in theory as long as it doesn't break those projects I would be fine with it.

Hard to say, but growing at a steady pace. Many meson-python users use Cython, and we crossed the 0.5M downloads/month threshold. I've seen ~15 significant packages adopt it, and more smaller/personal projects.