conda-build includes files from dependencies when using multiple outputs

[beenje]

Actual Behavior

I'm creating a recipe with 2 outputs using files to specify the files to be included in each package:

outputs:
  - name: dummy-libs
    requirements:
      host:
        - openssl
    files:
      - include
      - lib/dummy*
    test:
      commands:
        - test -f ${PREFIX}/include/dummy.h
        - test -f ${PREFIX}/lib/dummy.so
  - name: dummy
    requirements:
      host:
        - openssl
      run:
        - {{ pin_subpackage('dummy-libs', exact=True) }}
    files:
      - bin

In dummy, only bin/dummy.sh is included as expected. bin/openssl isn't part of the package:

$ tar tvfj /opt/conda/conda-bld/linux-64/dummy-0.1.0-h3e2b116_0.tar.bz2 | grep -v info
-rwxrwxr-x 0/0              32 2022-03-24 20:42 bin/dummy.sh

In dummy-libs, include/openssl is included in the package:

tar tvfj /opt/conda/conda-bld/linux-64/dummy-libs-0.1.0-h3e2b116_0.tar.bz2 | grep -v info
-rw-rw-r-- 0/0               0 2022-03-24 20:42 lib/dummy.so
-rw-rw-r-- 0/0             358 2022-03-24 08:45 include/openssl/ecdh.h
...
-rw-rw-r-- 0/0           76828 2022-03-24 08:45 include/openssl/evp.h
-rw-rw-r-- 0/0               0 2022-03-24 20:42 include/dummy.h

Expected Behavior

The dummy-libs package should only include:

• lib/dummy.so
• include/dummy.h

include/openssl shouldn't be added to the package.

Steps to Reproduce

Dummy recipe to reproduce the issue: https://github.com/beenje/dummy-package/tree/main/recipe

git clone https://github.com/beenje/dummy-package.git
cd dummy-package
conda build recipe

Output of conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 4.11.0
    conda-build version : 3.21.8
         python version : 3.9.7.final.0
       virtual packages : __linux=5.10.76=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.26.0 CPython/3.9.7 Linux/5.10.76-linuxkit centos/7.9.2009 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False

[jakirkham]

Am a little torn. See where Benjamin is coming from that conda-build typically excludes duplicate files to avoid clobbering and it is odd that it doesn't do this here. However can see why one might want to clobber anyways. Maybe there is a middle ground where clobbering requires explicitly listing the file to clobber?

[beenje]

One strange thing for me: if include is in the dummy-libs output, openssl headers are included. But they aren't when include is in the dummy output.

With:

outputs:
  - name: dummy-libs
    requirements:
      host:
        - openssl
    files:
      - lib/dummy*
    test:
      commands:
        - test -f ${PREFIX}/lib/dummy.so
  - name: dummy
    requirements:
      host:
        - openssl
      run:
        - {{ pin_subpackage('dummy-libs', exact=True) }}
    files:
      - bin
      - include

We get:

$ tar tvfj /opt/conda/conda-bld/linux-64/dummy-0.1.0-h3e2b116_0.tar.bz2 | grep -v info
-rw-rw-r-- 0/0               0 2022-03-25 07:50 include/dummy.h
-rwxrwxr-x 0/0              32 2022-03-25 07:50 bin/dummy.sh
$ tar tvfj /opt/conda/conda-bld/linux-64/dummy-libs-0.1.0-h3e2b116_0.tar.bz2 | grep -v info
-rw-rw-r-- 0/0               0 2022-03-25 07:50 lib/dummy.so

That's not consistent and it makes it easy to create invalid packages.

It'd be nice if the behaviour was the same for all outputs.
There might be technical reason why it doesn't. In that case, it should be well documented and an alternative recommended (install scripts?).

[jakirkham]

What happens if the split package order is flipped? IOW if dummy is first and dummy-libs is second?

[beenje]

I I remove:

      run:
        - {{ pin_subpackage('dummy-libs', exact=True) }}

in the previous example, then dummy is created first and it includes the files from openssl (both include and bin).
But if I moved back include to dummy-libs, it's not filtered either...

So the order seems to have an impact but not only.
Last output might need to have the main package name for filtering to work?

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[carterbox]

I agree that when using file lists to select which files to include in each output, files in the prefix that were added by host dependencies should be automatically excluded. This should be the expected behavior because conda-build already excludes files from host dependencies when using the script method.