Conan workspaces dynamic editables

[memsharded]

I am trying to apply conan workspace install approach to our monorepo, having Conan installed from 2.13.0dev at a21c0cf .
Not all packages from our dependency tree have been adapted vor v2, so self.load_conanfileing everything from glob("*/conanfile.py") yields all sorts of errors - from invalid imports to 3rdparty deps missing in our conanv2 remote.
I expected this and was going to filter out through dependency tree manually, somewhat like this:

def editables(self):
  editables_result = {}
  todos = [TARGET_PROJECT]
  while todos:
    cf = Path(todos.pop()).as_posix()
    conanfile = self.load_conanfile(cf)
    editables_result[f"{conanfile.name}/editable"] = {"path": f}
    todos.extend(__something_to_get_direct_dependencies_of(conanfile))

...But the only two interfaces I could think of seem to be unavailable during editables() evaluation?

• conanfile.dependencies() raises an exception, which is kinda expected since the graph is being populated right now:

  File "/home/igor/git/KdMonoRepo/conanws.py", line 44, in editables
    print(conanfile.dependencies)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/internal/model/conan_file.py", line 200, in dependencies
    self._conan_dependencies = ConanFileDependencies.from_node(self._conan_node)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/internal/model/dependencies.py", line 93, in from_node
    for require, transitive in node.transitive_deps.items())
AttributeError: 'NoneType' object has no attribute 'transitive_deps'

• conanfile.requires is constructed but empty, since apparently def requirements() has not been called by this time:

            print(type(conanfile.requires))
            print(conanfile.requires)

<class 'conan.internal.model.requires.Requirements'>
odict_values([])

I could probably call conanfile.requirements() manually to populate the requirements dictionary, but seeing how requires.py: class Requirements: def __call__ contains a check for if self._requires.get(req): raise ConanException, I do not think this is a good idea.

What are my options @memsharded ?

Originally posted by @Artalus in #15992

[memsharded]

Hi @Artalus.

Thanks for your question.

todos.extend(__something_to_get_direct_dependencies_of(conanfile))

I am afraid I didn't fully get what you are trying to achieve.
It shouldn't be necessary any kind of analysis of dependencies to define the editables of a workspace. The editables of a workspace should happen based on what subprojects are in the current projects, so basically loading local files in the subfolders. It is expected that requires/self.dependencies is not available in the Workspace.editables() method, there is nothing that can be done here, there is no graph available at this point in time, it is impossible to be, first it is needed to define which references are "editable" before being able to even compute the graph and dependencies.

[Artalus]

I am afraid I didn't fully get what you are trying to achieve.

In global sense, I am still trying to mitigate #17549, so switching out to a yet another new Github Issue feels weird 😁

Just to clarify: the def editables(self) method of a workspace works just fine. Things start to fall apart at conan workspace install.

In #17675 we briefly discussed that conan workspace install <single subproject> is a bad thing, and it is the editables() that should be filtering out "unusable" project. So I am exploring what filtering is available for me at all.

• Currently I am trying to filter out anything that is not a dependency for some Foo project that I am trying to build. Because if I only need to build Foo with its 3-5 dependencies, it makes no sense for conan workspace install to install countless gigabytes for all the 100+ deps from the remaining projects in the monorepo.
  Cannot do that, see the starting post.
  (No, regular conan install / conan build are not an option too, see [bug] Conan 2.X generates invalid transitive dependency info when building project with editable packages #17549 (comment) )
• At the very least, I might want to filter out things not available for the current settings. There are subprojects that make sense only on Windows, there are others only for cross-compiling to ARM arch, etc. In comments of Workspace install with meta-project and monolithic build #17675 I gave this exact example.
  I cannot use conanfile.settings either, as at the point where editables() executes, the .settings has not yet resolved from a tuple into an object:

  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/api/subapi/workspace.py", line 203, in info
    "editables": self._ws.editables()}
  File "/home/igor/git/KdMonoRepo/conanws.py", line 45, in editables
    if conanfile.settings.os == 'Linux':
AttributeError: 'tuple' object has no attribute 'os'

Granted, getting values for this from conanfile settings is probably wrong altogether... Should I be looking at some profile object instead?

[memsharded]

Granted, getting values for this from conanfile settings is probably wrong altogether... Should I be looking at some profile object instead?

I am afraid that settings (or profiles) are not available either when the editables() are being evaluated. Having some definition of editables that depends on profiles or the graph expansion itself is what is very challenging, it is kind of a chicken-and-egg problem.

The dynamic definition of editables is only possible from filesystem information. No information related to the dependency graph or the resolution of the dependency graph like the profiles can be available at that point. The load_conanfile() can only load the basic class information, with things like name and version, but it is impossible to have profiles there, at that point in time they do not exists. Note the workspace editables have to be resolved for many commands that do not even have a profile as input.

It is also possible to define the subsets of dependencies you want to work on, either in command line with conan workspace add <folder> or adding them to the conanws.yml file. It is also very possible to define subsets of the subprojects with different naming, for example you can have subprojects like group1_subproj1, group1_subproj2, group2_subproj3, which can allow to define the subprojects you want as editables by filtering by groupX_* name. You can also drop files inside your subprojects that define the belonging to one or more groups, so the editables() method can read those files.

This doesn't look like a limitation of conan workspace install, this command is intended to install the external dependencies of a monolith. It seems that you are looking for a way to define different subsets of a large monolith, but up to my knowledge this is not possible with common build systems like CMake, unless some pre-definitions of the groups of dependencies to work on is done and stored in the project files.

[Artalus]

The load_conanfile() can only load the basic class information, with things like name and version, but it is impossible to have profiles there, at that point in time they do not exists.

This is weird at first glance, but I guess makes some sense -- considering that profiles can have various package specifications, from requesting options to adding more packages to the mix via tool_requires...
I think at the current state looking simply at import os / import platform system variables might be enough, but I believe it would grow out of control very fast.
Wish profile could be partially-initialized with "static" data (os=Linux, compiler=clang - like it is with conanfile's name=pkg, version=somedefaultvalue)...

It is also possible to define the subsets of dependencies you want to work on, either in command line with conan workspace add <folder> or adding them to the conanws.yml file.

conan workspace add complains about package names not being lowercase.
I added couple of projects in conanws.yml in the similar layout as conan new workspace does, but am not sure what to do with it next. Both conan workspace build Foo (what is specified in products:) and conan workspace build Foo/version@user/channel (what is specified in editables:.

You can also drop files inside your subprojects that define the belonging to one or more groups, so the editables() method can read those files.

This sounds extremely hacky. If we go all the way to reading custom files instead of relying on conanfile.py, then I might as well sprinkle conanfiles with additional variables, enforce "never use requires=(...)" rule, and build my own dependency graph by parsing conanfiles themselves via regexes like ^\s{8,}self.requires 🫤

It is also very possible to define subsets of the subprojects with different naming, for example you can have subprojects like group1_subproj1, group1_subproj2, group2_subproj3, which can allow to define the subprojects you want as editables by filtering by groupX_* name.

Can you elaborate about this grouping a bit? Or would it be done in some external files in a similar manner?

This doesn't look like a limitation of conan workspace install, this command is intended to install the external dependencies of a monolith. It seems that you are looking for a way to define different subsets of a large monolith, but up to my knowledge this is not possible with common build systems like CMake

Then the whole thing should have been named conan monolith =/
We don't have a monolith, if anything it's more like forest with a bunch of products, and the deps that are connected tightly but not entirely.

As I mentioned before somewhere, we had it working with Conan-1 by using add_subdirectory() in each of the "modules", not only in the root projects.

[memsharded]

Ok, I think I see a bit better the issue.

I think it might be possible to add a conan workspace install --requires=xxxx filter that allows to define the subset of editables that you want to process and not all. The part of the super-project would be on your side, but it seems that this is not an issue for your use case. I'll give it a try.

[memsharded]

#17887 has been merged for next Conan 2.14.
It allows to define the subprojects to conan workspace install subfolder1 subfolder2... to avoid installing all the workspace editables, but only a subset.

[Artalus]

@memsharded it does not seem to work when packages in workspace use a python_requires:

❯ conan workspace install `pwd`/KdCommon
...
-- workspace processing /home/igor/git/KdMonoRepo/MonoRepo-Ecosystem/conan/python_requires
-- workspace processing /home/igor/git/KdMonoRepo/KdCommon
...
Filtering and installing only selected editable packages
Filtered references: [KdCommon/0.1.0@kudan/testing]

======== Computing dependency graph ========
Graph root
    cli
Requirements
    KdCommon/0.1.0@kudan/testing - Editable
    catch2/2.13.6#9e6c4d70fc592df854beed54640d875c - Cache
    fmt/10.0.0#69d97b04119d23d14739af45a611abfa - Cache
    inih/53#569c0aad47de924212eff3fe956fa3aa - Cache
    rttr/0.9.6#e314c9dc3d27f18cb97709b848ff02a7 - Cache
    spdlog/1.11.0#d68a762715fdb25d1f3b0029c244a2cc - Cache
Python requires
    pyreq/0.1@kudan/stable - Editable

======== Collapsing workspace editables ========

-------- Collapsed graph --------
ERROR: Traceback (most recent call last):
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/cli.py", line 307, in main
    cli.run(args)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/cli.py", line 192, in run
    command.run(self._conan_api, args[0][1:])
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/command.py", line 195, in run
    sub.run(conan_api, parser, *args)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/command.py", line 213, in run
    info = self._method(conan_api, parent_parser, self._parser, *args)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/commands/workspace.py", line 208, in workspace_install
    print_graph_basic(ws_graph)
  File "/home/igor/git/KdMonoRepo/.me/venv/lib/python3.8/site-packages/conan/cli/printers/graph.py", line 21, in print_graph_basic
    for r in node.conanfile.python_requires._pyrequires.values():  # TODO: improve interface
AttributeError: 'str' object has no attribute '_pyrequires'

Additionally, it appears that paths provided on CLI has to match paths provided in conanws exactly. In my conanws.py I do this:

    def editables(self) -> dict:
...
        for f in folder.glob("./Kd*/conanfile.py"):
            f = f.parent.as_posix()
            self._editables_result[f"{conanfile.name}/0.1.0@kudan/testing"] = {"path": f}

, but conan workspace install KdCommon errors:

❯ conan workspace install ./KdCommon
...
-- workspace processing /home/igor/git/KdMonoRepo/MonoRepo-Ecosystem/conan/python_requires
-- workspace processing /home/igor/git/KdMonoRepo/KdCommon
...
Filtering and installing only selected editable packages
Filtered references: []
ERROR: This workspace cannot be installed, it doesn't have any editable

, so I have to use conan workspace install pwd/KdCommon instead - so that the absolute path from CLI matches the absolute path from Python.