[Feature request] Make apio installation a single step process

[zapta]

Currently apio installation includes multiple steps

1. Python instllation
2. Apio pip package installation
3. Apio packages installation.
4. Install homebrew (on Mac osx, for ftdi drivers)

This feature request is for making it a single step process. For example, by running a script or program that will do the rest of the installation.

BTW, we have feature request #517 to eliminate #2 using automatic on-demand apio package instllation.

[zapta]

Note to self. One option is to use pyinstaller one-folder mode (the one-file mode would be too slow for the fequent invocations of apio).

https://pyinstaller.org/en/stable/operating-mode.html#how-the-one-folder-program-works

We should check also what installer icestudio does. @cavearr, any thoughts on an installer for apio that includes the python interpreter and all the other requirements?

[cavearr]

Hello @zapta, I tell you how I see it, from Icestudio it is not that much of a problem to install Apio because since the installation is managed from Icestudio it does not matter the number of steps to be carried out since everything is simplified for the user.

In Icestudio I am changing things that I will soon make operational during the installation, for example I want the user to be able to use their own python environment if they have it installed or in the development version I want to eliminate the need to have git installed (possibly I will create preleases or directly packages that can be downloaded and decompressed from Apio).

For these reasons, at Icestudio it is not critical to further simplify the installation.

Looking at Apio independently, I would not abandon the pip format since in the end the problem with bundlers is that they take up a lot, if for each python application, all python was packaged... I don't know in that sense I'm not very pro bundlers. But it's my way of seeing it and for newer users it may be attractive to install in a single step.

I also tell you that an installer could be made via script per operating system that would do the entire process and could be called from the command line with a curl call to a url that would download and execute the script.

For example, I don't like the Icestudio bundler system for Linux since it happens a bit the same as before, all the bundlers like snap, flatimage, etc. for me are an overload of resources and I know that today the price of the mega hard drive is very cheap but at least it bothers me to feel that I have the same thing installed in multiple packages.

If I had to choose a package manager that seems very interesting to me in the long term, it is Nix, it is cross-platform and works really well.

I was helping a group of Nix packagers this summer to package Icestudio (for now only for Linux and Windows), I didn't know Nix well until then and I thought it was amazing.

[zh4ngx]

NixOS user here and I second the idea of using Nix Packages :)

I ran into an issue where I can't use an FTDI device due to having to modify udev rules, considered immutable in Nix.

Filed the issue here: #534

[zapta]

I see two terms here, NixOS and Nix Packages.  

1. In apio's terms, is NixOS a new platform in addition to these ones?  https://github.com/FPGAwars/apio/blob/develop/apio/resources/platforms.json

2. Does the nix packager support all the other platforms on that list?

3. What will be the steps of installing apio using Nix packages on, let's say, a vanilla Windows machine?

I made a quick experiment with pyinstaller on darwin arm 64 and it seems promising, it created a 20MB zip file that includes an apio binary and a directory called _internal that includes the support files, including the python interpreter and packages I think (?)

Anybody want to give it a try? Zip below, unzip on a Mac OSX apple silicon and add the apio binary to your path.

apio-release.zip

EDIT: (updated zip file) After unzipping, read README.txt in the unzipped directory and run source activate. Then, set your PATH to include the directory and use the apio command. The first run may be slower than usual.

[cavearr]

Hi @zapta! i'm using nix for some packages(not nixos as operating system, nix as package manager) and this is great, is multiplatform and very powerful.

With Icestudio we have nix packages for linux , in osx we have problems (almost for the official packages in nix) i have a PR open but goes slowly and windows i don't know the state of this. Last summer i was in contact with the maintainers and try to help them with it: https://mynixos.com/nixpkgs/package/icestudio

I'm not an expert in nix but we could research around it and how we could create a repository for our nix packages and could be our future packager for Apio and Icestudio

I'll try your package in osx and works well, for windows or linux we need to create that packagers.

If you want i could create the architecture for our own nix channel , i think could be a very great idea. If you want i invest time in it.

[zh4ngx]

@zapta

The answers to your Qs to the best of my understanding:

1. No, NixOS is more like a Linux distro (eg Debian, Fedora) but does not use the Unix FHS.
2. Nix packages (nixpkgs) does run on all of those platforms, but definitely double-check my claim here. Packages built in nixpkgs try to follow NixOS conventions, adapted to the particular platform. For example, on non-NixOS Linux, you can run nixpkgs alongside the FHS, but nixpkgs will will never make any FHS assumptions out-of-the-box.
3. I'm not super familiar with nixpkgs on Windows, but I set up nixpkgs, both on Fedora and NixOS, using this guide. It has a WSL section, which is what I assume you're using for apio.
   If you use the Determinate installer:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

You should be able to run a nix shell with apio installed like so:

nix shell nixpkgs#apio nixpkgs#libftdi nixpkgs#gtkwave

Let me know if you run into any issues. You may see something related to "nix flakes," although the Determinate installer is supposed to set that up for you.

[zapta]

Thanks Any, I will look into it. My concern is that we will require the user to install an intrusive package management system just to use an app. It's kind of "the tail wagging the dog" situation.

[zh4ngx]

Totally get it. The practice I've seen some packages adopt is to make nix the gold standard and then adapt it to other package managers, but those packages often start as nix-first. In your situation, you would likely do the opposite; see what is working in other package managers and try do simplify / do it better in nixpkgs. Then you can backport some of those practices to other package managers. nixpkgs/NixOS is often more restrictive than other package managers / distros. Most backports involve sandboxing changes made to the environment. For example, the nix shell ... you see there doesn't actually install anything in the global env.

It doesn't look like apio is too far off from being able to accomplish this. The main issues I see are udev rules and the assumption that services exists - more discussion in the issue.

[zapta]

@zh4ngx, if you would like to give it a try, I created here self contain apio releases for testing. They don't have any prerequisites, not even python.

https://github.com/zapta/apio_dev/releases/tag/2025-02-13-17-33

You can also try creating release on your systems, once this is accepted. #570

[zapta]

@cavearr, I moved the experimental installer scripts to a separate repo and and added 'real' InstallBuilder installers. The script and the release are in this repo https://github.com/zapta/apio-installers/releases/tag/0.9.6

Please give it a try. You can also try rebuilding the installers. First you need to create the pyinstall packages (need to be done on each platform) and then build the InstallBuilder installers. Please let me know if you have any questions.

[zapta]

@cavearr, per out discussion in the pull request, we will not use InstallBuilder installers, that makes sense.

Do you know of an open source installer that can install the pyinstaller generated apio zip files? It requires to unzip the pyinstaller files, set a path to the apio binary, and handle whatever permissions issues each system has.

You can find the zip files here https://github.com/zapta/apio-installers/releases/tag/0.9.6 and you can rebuild them using the build-package.sh script here https://github.com/zapta/apio-installers

[cavearr]

@zapta ,I love that you ask this question because I'm looking at it, also for icestudio, I'm reviewing how to distribute it and it is very aligned with this. This week I will try to make progress on it, I will tell you okay?

[zapta]

Very good. Thanks @cavearr.

[zapta]

@cavearr, a question for you, when I create the apio pyinstaller packages I need to build it on each of the platforms to create it's package. For example, I need to build the windows package on windows.

Is it possible, with github automation, to create all of them automatically using vertual machines or anything else that github supports?

[cavearr]

Depends how do you create the package. for example for icestudio i create packages i github actionas as is:

Package	Github action runner
Linux x86	Linux x86 runner
Linux arm	Linux x86 runner with cross tools for arm
Windows x86	Linux x86 runner with cross tools for windows
OSX	Osx runner

The key is for example if you want to build the installer for windows in linux, you need the cross building tools in linux for this builder. Witch are you trying?

[zapta]

@cavearr, I don't use cross building tools. When pyinstaller runs on windows, it can create only for windows. When it runs on macosx, it can create only for macosx, and so on.

What is the 'osx runner' in the table you sent me, does github run some automation tools on osx?

[cavearr]

yes, cross building tools is for create only platform binary but from other system that the target.
for example the icestudio windows package is a native windows binary , only works on windows but we build in linux with a cross toolchain for windows.

And yes you could select some runners OSs

https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#choosing-github-hosted-runners

And this is the workflows most useful documentation page (i use a lot for reference):
https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions

if you need help with concise task tell me and i'll try to help you.

[zapta]

Thanks @cavearr, I also asked chatgpt and it says its possible to run virtual machines with github automation. I will play with it and see how it goes. I will probably have more questions.

[cavearr]

yes! you could check this icestudio files if you neede .github/workflows we implement very basic actions but you could view how upload artifacts, etc

If you need help tell me!

[zapta]

@Obijuan, @cavearr, I have been playing recently with installers, especially for osx and managed to have working installers with both commercial (e.g InstallBuilder) and free (e.g. pkgbuilder and productbuilder) tools. However, to have a clean user experience, the installer need to be signed and this required a subscription in the apple development program.

Have you considered getting a membership?

https://developer.apple.com/programs/enroll/

[cavearr]

Hi @zapta i'm paying a membership for this (in icestudio for mac is the same, if you want i could generate the app and signing capabilities that you need.

If you prefer pay your own go ahead, but i think if only is for this i'm paying now. Tell me what do you need and i'm preparing it.

[cavearr]

@zapta In any case, I insist on not using any commercial solution in which the source code is not available. I am considering integrating this installer into the icestudio wips to test it with users and I do not want to take a path that will be truncated at some point by a commercial turn.

[zapta]

@cavearr, I created a osx installer with pyinstaller, pkgbuild, and productbuild, I believe they are all free with no restrictions.

I don't want to have a membership myself, you and Juan are a better choice. What do you need to sign an osx installer, can I send you a working .pkg installer and you will sign it or is the signing operation need to be part of the .pkg construction?

[cavearr]

to try it ,send me or give me an url to download and today i'll try ti sign in.

Have you automatize the pkg creation flow in github actions?

[zapta]

@cavearr, can you try signing this osx package? https://github.com/zapta/apio-installers/tree/main/darwin-native-installer/release

I created it using pyinstaller, pkgbuild and productbuild. Scripts here:

https://github.com/zapta/apio-installers/blob/main/pyinstaller/build-package.py
https://github.com/zapta/apio-installers/blob/main/darwin-native-installer/build-installer.sh

For icestudio, if you want include apio in the icestudio package, you can just include the files that pyinstaller generates in the icestudio package. For example, unzip this https://github.com/zapta/apio-installers/blob/main/pyinstaller/release/apio-darwin-arm64-0.9.6-pyinstaller-package.zip . It includes everything apio needs to run, including python.

I haven't started yet with the github automation, still working on the manual scripts.

[cavearr]

i'll try between today and tomorrow and tell you the results!.

I'm asking you about the github automation because i have configured the signing process for icestudio in it. And when you have done i could configure for apio.

[zapta]

@cavearr, you don't need to sign now.

I will setup automation first so we will have better security and paper trail. It's safer this way.

[zh4ngx]

@zh4ngx, if you would like to give it a try, I created here self contain apio releases for testing. They don't have any prerequisites, not even python.

https://github.com/zapta/apio_dev/releases/tag/2025-02-13-17-33

You can also try creating release on your systems, once this is accepted. #570

Thanks for setting this up and sorry for the delay. What do you think is the best way to ensure the udev rules are in the right place, either «apio»/etc/udev/rules.d or «apio»/lib/udev/rules.d?

[zh4ngx]

@zh4ngx, if you would like to give it a try, I created here self contain apio releases for testing. They don't have any prerequisites, not even python.
https://github.com/zapta/apio_dev/releases/tag/2025-02-13-17-33
You can also try creating release on your systems, once this is accepted. #570

Thanks for setting this up and sorry for the delay. What do you think is the best way to ensure the udev rules are in the right place, either «apio»/etc/udev/rules.d or «apio»/lib/udev/rules.d?

Looking good so far; able to replicate the same outcome with FTDI:

$ apio system --lsftdi
Number of FTDI devices found: 1
Checking device: 0
Manufacturer: Alchitry, Description: Alchitry Cu

[zapta]

@zh4ngx, we set up experimental build automation, so you can get the pyinstaller package from here https://github.com/zapta/apio_dev/actions/runs/13537686815

Unzip it twice (github artifacts add one level of zip) and you will get a directory called apio that includes a binary called apio and a resources directory called _internal. The apio binary is the apio command you want to run.

This is the latest dev version, so the command structure is different, for example to list devices run `apio drivers list, or for help run apio -h``, ``apio drivers -h``, ``apio drivers list -h``, etc.

Please let me know if you have any questions.

EDIT: Updated the installer link above. The original one didn't have the rule files.