Tracking develop branch issues:

[tacaswell]

• ~~ pixi run install-editable does not place compiled programs convcell & LATTIC into .pixi/envs//bin; (N.B. pixi run install does do this)~~ (moved below)

• comment in GSASIIphsGUI.py: "XXX I suspect this is Unix-specific -- need porting help!". Is this still relevant?

• Best way to read & write configuration info to an external file?

  • will use configparser. Has r/w capabilities in stdlib; Don't need more sophisticated features of toml, etc.

• running tests/test_scripting.py fails due to "path hacking"; problem seems to be within pytest, but is not seen with the other self-tests.

  • Tom will make path-hacking test "opt in" via a PR (done in ca6402f, BHT)

• can pixi install & install-editable run the GSAS-II/install/save_versions.py script (before copying files) or perhaps this should be made into a git smudge activity?

  • use setuptools-scm or put command into the pixi.toml file
  

Originally posted by @briantoby in #134 (comment)

[tacaswell]

I made a new issue for this rather than using a comment is a merged PR.

[briantoby]

Installation tasks @briantoby:

• Build old-style binary .tar files (BHT)

• Build gsas2full style self-updating self-installer w/develop branch (BHT)

• confirm that switching branches does not break gsas2full installation (BHT)

  • win
  • Mac
  • linux

• Implement tradition version numbering (X.Y.Z type) for GSAS-II; saving version #s as tags (BHT)

  • read https://jacobtomlinson.dev/effver/ and think about versioning concepts
  • linear # tagging script revised to also increment minor version # & new script written to increment mini #.
  • X.Y.Z version now in the importable file (though not accessed yet by code).

• Test the binary self-installers (gitstrap & in main code) with develop & 3.12 or 3.13 & new binaries.

  • win
  • Mac

Self-tests @briantoby:

• scriptref.py not producing same results on Windows as in MacOS & Linux
  • this is due to the use of flang in the windows build, which breaks pypowder.
• should tests be made faster?
• Add test for texture

[briantoby]

@tacaswell Here is stuff that I'd like your help with:

• pixi run install-editable does not place compiled programs convcell & LATTIC into path (.pixi/envs/bin) could that be fixed?

• Implement a conda package for GSAS-II develop branch; N.B. at some future point this will be renamed to main

More long range, but for people who have a traditionally-installed (non self-updating) version of GSAS-II, I'd like still generate a message that shows how far the installed version is behind from the current version. Any thoughts on how to find this out?

Low priority: It is not clear to me how pixi would use one python environment for building and installing GSAS-II and another for running it, but I have left some comments on how different conda environments might be deployed in the TODO comment in pixi.toml,

[briantoby]

@tacaswell

• Change needed for pixi install: Need to build windows binaries with gfortran

  I now understand why the smoke test fails on Windows (only). The pixi install on windows is using clang (perhaps OK) and flang-new (llvm-flang 19.1.6 -- not OK). We have only used gfortran with GSAS-II. The binaries produced by clang/flang are definitely more convenient, in that no non-system .dlls are needed, but they fail spectacularly when computing a powder pattern. I really don't want to debug that.

  There is an example for how I build the GSAS-II binaries with gfortran with this messy GH action (which is called by this), but I really don't see where the build configuration for windows is set. Your help is needed.

• pyproject.toml change: add PyCifRW

  The develop branch now requires module PyCifRW (imported as CifFile) for all use (scripting & GUI). This is required in pixi.toml, but I think is needed in pyproject.toml

[tacaswell]

for the programs I see a couple paths:

• add an entrypoint to a python shim that calls the actual program. There might be a path to leverage meson's auto rebuild-if-you-need-it logic on this, but if those programs do not change very often, probably not worth it.
• in the path hacking logic also update os.environ['PATH'] so all of the special casing is in one place
• in the pixi install-editable tasks we can just copy them over to the right place in the environment

for flang:

We need to use flang to compile on windows as that is the only toolchain that can be distributed on conda-forge (see https://labs.quansight.org/blog/building-scipy-with-flang ) on windows. It is not clear to me where gfortran is coming from in that GHA. I suspect that debugging the fortran will be simpler than getting the gfortran toolchain to run on windows for CF.

I tried tracking back through the code to figure out which fortran was being called, but got lost. Is there a more direct way to call the code that is misbehaving?

[briantoby]

• add an entrypoint to a python shim that calls the actual program. There might be a path to leverage meson's auto rebuild-if-you-need-it logic on this, but if those programs do not change very often, probably not worth it.

I prefer to have Python run externally-compiled programs (particularly Fortran ones) rather than interface them as .pyd/.so libraries. This could be done for several of the routines (diffax and pyspg come to mind as they are not called that often and the amount of data transfer is small compared to execution time). How would it help if there are more .exe's and fewer .pyd's?

For pypowder, histogram2d and maybe pytexture, they need to run fast and do in-memory data xfer, so they really need to be wrapped for python modules. Perhaps ideal would be to recode them in C, but pypowder needs lots of subroutines.

pack_f and unpack_f are used only for MAR images. That could be retired IMHO.

• in the path hacking logic also update os.environ['PATH'] so all of the special casing is in one place

not understanding this

• in the pixi install-editable tasks we can just copy them over to the right place in the environment

please do

We need to use flang to compile on windows as that is the only toolchain that can be distributed on conda-forge (see https://labs.quansight.org/blog/building-scipy-with-flang ) on windows. It is not clear to me where gfortran is coming from in that GHA. I suspect that debugging the fortran will be simpler than getting the gfortran toolchain to run on windows for CF.

Oh dear. At least it will be possible to install GSAS-II from flang side-by-side with GSAS-II from gfortran so we can see where it fails.

I tried tracking back through the code to figure out which fortran was being called, but got lost. Is there a more direct way to call the code that is misbehaving?

not sure what you want here

[tacaswell]

I assumed the programs were stand-alone because users also directly call them, if they are only used internally having them available as so/pyds seems better (even if the data transfer is slow, launching new processes are not the most performant thing).

However, that is not what I meant to suggest. Instead to write functions like:

def really_call_prog(*args):
    prog = find_the_prog()
    subprocess.run([prog, *args]).check()

and then registering that function as a entry point so the editable wheel install would automatically drop those in if we put entries in [project.scripts] in pyproject.toml

Doing the copies is probably the easiest path (I may need some help sorting out how to get the right path on Windows).

---

[tacaswell]

not sure what you want here

I started from

GSAS-II/tests/test_scriptref.py

Line 45 in 3ab4088

testR('Before fitting',96.681098,99.748994)

and was trying to understand where the calls out to fortran extensions are happening. You said it is failing trying to generate a powder pattern, is there a short bit of code that will just generate the pattern (without the rest of the fitting / validation / state management work)?

[briantoby]

I will see if I can write something that just does a single call to pypowder that tests it.

[briantoby]

Here is a bit of code that exercises pypowder:

# compute a single peak with pypowder
import os
import sys
import importlib.util
import numpy as np

home = os.path.dirname(__file__)
G2loc = None
try: 
    G2loc = importlib.util.find_spec('GSASII.GSASIIscriptable')
except ModuleNotFoundError:
    print('ModuleNotFound for GSASII.GSASIIscriptable')

if G2loc is None: # fixup path if GSASII not installed into Python
    print('GSAS-II not installed in Python; Hacking sys.path')
    sys.path.append(os.path.dirname(home))

from GSASII import GSASIIpath
GSASIIpath.SetBinaryPath()

if GSASIIpath.binaryPath:
    import  pypowder as pyd
else:
    from . import pypowder as pyd

# these values generate a nearly symmetric peak
xdata = np.arange(18,22,.05)
pos = 20.
sig = 525.9994695543994
gam = 2.1444606947716025
shl = 0.002
ydata = pyd.pypsvfcjo(len(xdata),xdata-pos,pos,sig,gam,shl)

# these values generate a peak with significant low-angle broadening
xdata = np.arange(0.1,5,.05)
pos = 2
sig = 525.9994695543994
gam = 2.1444606947716025
shl = 0.035
ydata = pyd.pypsvfcjo(len(xdata),xdata-pos,pos,sig,gam,shl)

#import matplotlib.pyplot as plt
#plt.plot(xdata,ydata)
#plt.show()

[briantoby]

Above computed with gfortran on mac. When I use gfortran-compiled code on Windows I get plots that look the same.

Below is what I get on windows for the flang-compiled code for the 1st peak:

The low-angle peak seems to compute OK with flang.

[briantoby]

Now that there are a lot more optional packages used in GSAS-II, it might be a good idea to have a "stub" package so that something like conda install ... gsas2opts will pull in all the packages needed by GSAS-II optionally avoiding things that cause problems like the latest zarr release. Might be helpful for use like this conda install gsas2pkg gsas2opts too.

[briantoby]

@tacaswell for next meeting: are there some basic protection rules I should have on the master branch?

[tacaswell]

When building a the conda-package you can have multiple outputs from the same recipe (e.g. for Matplotlib we have matplotlib-base (which does not depend on a GUI) and matplotlib which depends on a Qt binding). We should do the same thing for gsas-ii (e.g. gsas2-base / gsas2 or gsas2 / gsas2-full depending on what you want the "normal" thing to be).

[tacaswell]

I've tweaked the reproducer code a bit to:

# compute a single peak with pypowder
import GSASII.pypowder as pyd
import matplotlib.pyplot as plt
import numpy as np

fig, ax1 = plt.subplots()

# these values generate a nearly symmetric peak
xdata = np.arange(15, 25, 0.05)
pos = 20.0
sig = 525.9994695543994
gam = 2.1444606947716025
shl = 2.0


for shl in [2, 0.2, 0.002, 0]:
    ydata = pyd.pypsvfcjo(len(xdata), xdata - pos, pos, sig, gam, shl)
    ax1.plot(xdata, ydata, label=f"{shl=}")
ax1.legend()
plt.show()

which works correctly in the shl==0 case so the bug is in the asymmetric code path (which is unfortunately most of the function at hand).