Load tarball - IPM look for dependency from repo instead

[jinlouISC]

Environment:
IPM v0.10.beta-4 on an I4H instance with no repo configured.

Background:
The module has dependencies. The tarball includes all the dependency modules.

Behavior:
When I do zpm "load ", it stopped at the first dependency hs.rest, "Could not find satisfactory version of hs.rest in any repositories. Required by: healthshare.hp.pas: ^2.1.0". I think it is looking for the dependency from the repository, not from the tarball.

In contrast, on a instance with a repo configured, when I load the tarball, IPM would install the dependencies from the repo, not from the tarball.

@isc-shuliu

[isc-shuliu]

Minimum reproducible example

/tmp/load-export-deps/module.xml

<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25">
  <Document name="load-export-deps.ZPM">
    <Module>
      <Name>load-export-deps</Name>
      <Version>0.0.1</Version>
      <Packaging>module</Packaging>
    </Module>
  </Document>
</Export>

/tmp/load-export/module.xml

<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25">
  <Document name="load-export.ZPM">
    <Module>
      <Name>load-export</Name>
      <Version>0.0.1</Version>
      <Packaging>module</Packaging>
      <Dependencies>
        <ModuleReference>
          <Name>load-export-deps</Name>
          <Version>0.0.1</Version>
        </ModuleReference>
      </Dependencies>
    </Module>
  </Document>
</Export>

Then run,

zpm "repo -fs -name local -path /tmp"
zpm "install load-export"
zpm "load-export package -export-deps 1 -DPath=/tmp/package -v"
zpm "repo -delete-all"
zpm "uninstall load-export-deps"
zpm "uninstall load-export"
zpm "load /tmp/package.tgz"

Also verified that /tmp/package contains a hidden path.modules/load-export-deps/module.xml

@isc-tleavitt Has this worked before in IPM v0.7? It's strange that such a problem has gone unnoticed for so long.

[isc-shuliu]

@jinlouISC A v0.10.0-beta.12 release artifact is triggered by merging #726 should be generated in a few minutes. Can you try that and verify this bug is fixed?

[jinlouISC]

@isc-shuliu thank you for letting me know, yes will do and will keep you updated!

[jinlouISC]

I tested on v0.10.0-beta.12 where IPM was able to locate the dependencies in the tarball. But I ran into another issue where repository 'ipm-temp-dot-modules-1' is not being created, so none of the dependencies were installed because IPM shell would complain "ERROR! Repository 'ipm-temp-dot-modules-1' is unavailable."

[isc-eneil]

@isc-shuliu @jinlouISC I was not able to reproduce this error with a smaller module with dependencies.

Environment:
IPM v0.10.0-beta.15 installed on a 2024.2 I4H instance with python.

# do $System.OBJ.Load("<ipm root dir>\preload\cls\IPM\Installer.cls","ck")
# do ##class(IPM.Installer).setup("<ipm root dir>", 3)

Windows OS.

Steps:

1. Created new namespace and database via CPF merge.
2. Enabled IPM in all namespaces from the namespace that had v0.10.0-beta.15 installed via CPF merge.
3. Output of zpm "version" at this
   zpm:TEST-211-2>version

HSLIB> zpm 0.9.0+snapshot (DeveloperMode)
TEST-211-1> zpm 0.10.0-SNAPSHOT (DeveloperMode)
TEST-NS1> zpm 0.10.0-beta.4 (DeveloperMode)
USER> zpm 0.9.1 (DeveloperMode)
%SYS> zpm 0.9.1 (DeveloperMode)
Installed In: USER
HSCUSTOM> zpm 0.9.1 (DeveloperMode)
Installed In: USER
HSSYS> zpm 0.9.1 (DeveloperMode)
Installed In: USER
HSSYSLOCALTEMP> zpm 0.9.1 (DeveloperMode)
Installed In: USER
TEST-211-2> zpm 0.10.0-SNAPSHOT (DeveloperMode)
Installed In: TEST-211-1

4. Loaded a test module with dependencies (ipm\tests\integration_tests\Test\PM\Integration_data\load-export) into the new namespace via CPF merge:
   Execute:Namespace="TEST-211-2",ClassName="%IPM.Main",MethodName="Shell",Arg1="load C:\Users\eneil\ipm\tests\integration_tests\Test\PM\Integration\_data\load-export\ -verbose -DNoLock=1 -DNoTransaction=1 -DNoMapping=1"
5. Output of zpm "list-installed":

zpm:TEST-211-2>list-installed
load-export 0.0.1 (DeveloperMode)
load-export-deps 0.0.1

[isc-shuliu]

After debugging with @isc-eneil and @jinlouISC, it turns out to be a macOS specific issue (doesn't impact docker containers on macOS). When using work queue manager to load dependencies in other processes, the call to %Manager.CheckServiceCache in %IPM.Utils.Module:LoadModuleReference fails to detect the newly-setup temp repo and sets Output pAvailable to 0.

The quickest workaround is to support loading with single processing (by setting pSynchronous to 1) in %IPM.Main:Load(), but there's definitely an underlying issue here.

[isc-shuliu]

This may very well be related to #736 (comment), since both Jin and I also use multiprocessing (which according to Tim, runs as irisowner) on our macs, and our IRIS instances are set up in such a way that the owner of iris is our default OS user (instead of irisowner)

This can well explain the ERROR! Repository 'ipm-temp-dot-modules-1' is unavailable error because the folder permission may be too restrictive for irisowner to access.

[jinlouISC]

@isc-shuliu @isc-eneil I think that is a good theory, and I gave it a try on this instance whose owner is irisusr to load tarball. Unfortunately I am still getting the ERROR #5001: Repository 'ipm-temp-dot-modules-1' is unavailable. error. I still need to use the workaround to load with single processing to make it work.

A good finding is that the 3 flags mentioned in #717 -DNoLock=1 -DNoTransaction=1 -DNoMapping=1 don't seem to be needed anymore, which makes sense because those modifiers are used to load modules snychronously (I think?)

[isc-shuliu]

I'm not sure about NoMapping but yes, I think NoLock and NoTransaction are not needed when we are loading synchronously

[jinlouISC]

Hi @isc-shuliu ! Thank you for putting in the fix. To load module synchronously on macOS with this change, would the command just be zpm "load package-name -synchronous"? Thanks in advance!

[isc-shuliu]

Hi @isc-shuliu ! Thank you for putting in the fix. To load module synchronously on macOS with this change, would the command just be zpm "load package-name -synchronous"? Thanks in advance!

Yes, exactly. And it works for other platforms as well (although macOS seems to be the only platform that needs synchronous loading to avoid the bug).