PlanktoScope OS (Sunray) v2024.0.0-beta.1

[ethanjli]

The v2024.0.0 software release, which is part of the "Sunray" series of software releases started with v2023.9.0, includes major fixes for various problems and improvements to existing functionality with the PlanktoScope software.

About this pre-release

Starting with v2024.0.0, we are calling the software we provide in our SD card images the "PlanktoScope OS" (because it is an operating system for the PlanktoScope).

PlanktoScope OS v2024.0.0-beta.1 is a pre-release version for everyone to test out. It may have undiscovered software bugs, breakage, and other problems. If you decide to try out this pre-release, please file bug reports on GitHub Issues and/or let us know about any bugs you discover in the #6-dev-software channel on the PlanktoScope Slack workspace! Here are quick links to the SD card images - you should download the image specific to your PlanktoScope hardware version:

• fairscope-latest: use this if your PlanktoScope hardware is v2.6.

• planktoscopehat: use this if your PlanktoScope hardware is v2.5, v2.3, or some other nonstandard design using the PlanktoScope HAT. Note: you can also use this SD card image on the v2.6 PlanktoScope hardware, it'll just have a setup step on the Node-RED dashboard's homepage which you wouldn't need to do if you used the fairscope-latest image.

• adafruithat: use this if your PlanktoScope hardware is v2.1 or some other nonstandard design using the Adafruit Stepper HAT.

Contributors

The PlanktoScope OS is a community effort, and the v2024.0.0 release would not have been possible without the PlanktoScope community:

• Thanks to Wassim Chakroun (@W7CH) for code contributions in the new picamera2-based image-acquisition module!
• Thanks to (in alphabetical order) Ana Fernandez Carrera, Adam Larson (@babo989), Fabien Lombard (@fabienlombard), Kevin Du Clos (@kevin-duclos), Satoshi Kitajima (@kitazy52), Pierre Kostyrka (@pkostyrka), Tanguy Cebron, and Thibaut Pollina (@tpollina), for contributing feature requests, bug reports, and software testing efforts for this release!

Changelog since v2023.9.0

This changelog only lists items we think you are likely to notice or to need to know about; for a full list of changes, please refer to software/CHANGELOG.md.

Added:

• A fairscope-latest SD card image is now built for FairScope customers; its only difference from the planktoscopehat SD card image is that the default hardware settings are set for the v2.6 hardware, instead of the v2.5 hardware.
• The PlanktoScope now shares internet access over its Wi-Fi hotspot and Ethernet port (and a USB-to-Ethernet adapter, if one is attached) from a phone connected to it in USB tethering mode.
• On the Node-RED dashboard, the System Monitoring page will now prompt you to reset the PlanktoScope's system clock to match the time from your web browser, if the two times are very different. This enables you to temporarily fix an inaccurate system clock on your PlanktoScope, at least until the next time you unplug the PlanktoScope from power.

Changed:

• The pscopehat version of the SD card image has been renamed to planktoscopehat for more consistent naming.
• The first time you boot up the planktoscopehat SD card image, the Node-RED dashboard's homepage will ask you to select the hardware version of your PlanktoScope - instead of assuming that you are using the v2.5 hardware (and providing default hardware settings for the v2.5 hardware).
• The hardware controller's camera driver has been fully rewritten to fix various problems we had with raspimjpeg (which has now been replaced with a different camera control mechanism). This change may require you to use different ISO and white balance gains than before; accordingly, the Node-RED dashboard's ISO selector has been changed to a slider to enable finer adjustment of the ISO setting.
• Previously, the segmenter's default behavior was to subtract consecutive masks to try to mitigate image-processing issues with objects which get stuck to the flowcell during imaging. However, when different objects occupied the same space in consecutive frames, the subtraction behavior would subtract one object's mask from the mask of the other object in the following frame, which would produce clearly incorrect masks. This behavior is no longer enabled by default.
• The PlanktoScope OS is now built as a 64-bit OS, instead of a 32-bit OS. This change speeds up the segmenter, potentially by a factor of 2.
• On the Node-RED dashboard, the System Monitoring page’s monitoring panel has been replaced with an embedded Grafana dashboard, with additional graphs available in an expandable subpanel.
• SD card images are now provided as .img.xz files instead of .img.gz files; they will still work with the Raspberry Pi Imager software.

Deprecated:

• The current Node-RED dashboard (both the version for the Adafruit HAT and the version for the PlanktoScope HAT) is transitioning to maintenance mode: no new features will be added, and any bugs will be only be fixed if someone volunteers to fix them. The current Node-RED dashboard will be completely replaced by a fully-rewritten Node-RED dashboard, though there is no timeline for completion of that new dashboard. Currently, our plan for deprecating and eventually removing the current Node-RED dashboard is as follows: maintenance mode (no new features, only some bugfixes), then deprecation (no maintenance; not enabled by default, but still installed), then removal (not installed by default, but anyone is free to install it and see if it still works); deprecation will not occur before the rewritten Node-RED dashboard is stable for general-purpose usage. If you have concerns, please share your feedback on GitHub or on the PlanktoScope Slack.
• The "USB backup" functionality of the Node-RED dashboard will be removed in v2024.1.0 (the next release after v2024.0.0). Instead, you should use the datasets file browser (which you can access by a link on the PlanktoScope's landing page) for backing up and deleting dataset files on your PlanktoScope.
• The raspimjpeg-based imaging module in the Python hardware controller has not yet been deleted, so that you can change the Python hardware controller code to switch back from the new picamera2-based imaging module if picamera2 ends up causing big problems for you. However, we are deprecating the raspimjpeg-based imaging module, and we will fully delete it in a future release after we get sufficient information about the picamera2-based imaging module's reliability for PlanktoScope users.
• Portainer will no longer be installed/provided by default after v2024.0.0, because it takes up lots of disk space and it is difficult to set up and its functionality overlaps with the much simpler Dozzle. After we remove Portainer from the default SD card image, you can use Forklift to install Portainer.
• Now that the PlanktoScope project is building 64-bit versions of the PlanktoScope OS, 32-bit versions of the PlanktoScope OS are no longer built or officially supported by the project. However, for v2024.0.0 you can still use the installer scripts (with the PlanktoScope OS's non-standard installation process) to set up the PlanktoScope OS on 32-bit base images of the Raspberry Pi OS. After v2024.0.0 we may stop testing whether the installer scripts work on 32-bit Raspberry Pi OS images.

Fixed:

• The Node-RED dashboard’s “Sample” page’s Filtered Volume field is now exported as the sample_total_volume metadata field for all sample types; previously it was only exported for horizontal plankton tow samples.
• The Node-RED dashboard should now correctly load the flowcell setting from the config.json settings file.
• Startup is now faster.

Changelog since v2024.0.0-beta.0

Added

• A fairscope-latest SD card image is now built for FairScope customers; its only difference from the planktoscopehat SD card image is that the default hardware settings are set for the v2.6 hardware, instead of the v2.5 hardware.

Changed

• The first time you boot up the planktoscopehat SD card image, the Node-RED dashboard's homepage will ask you to select the hardware version of your PlanktoScope - instead of assuming that you are using the v2.5 hardware (and providing default hardware settings for the v2.5 hardware).

• SD card images are now provided as .img.xz files instead of .img.gz files; they will still work with the Raspberry Pi Imager software.

Fixed

• The Node-RED dashboard should now correctly load the flowcell setting from the config.json settings file.

Known Problems

• (Known as of v2023.9.0-beta.1) You may encounter weird behavior with the system time if you try to use Cockpit to manually adjust the system clock. If you experience this problem, please report it and your testing circumstances (e.g. your GPS situation, and the time you tried to change the system clock to), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) In certain cases, some computers connected to both a Wi-Fi network providing internet access and to a PlanktoScope may be unable to access anything on the internet. If you experience this problem, please report it and your testing circumstances (e.g. your OS, your network configuration for internet access, etc.), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) If you restart the Node-RED dashboard (e.g. via Cockpit), you will also need to manually restart the hardware controller, or else the Node-RED dashboard won’t know the type of camera installed in the PlanktoScope. We are working on a solution to this problem, for a future release after v2024.0.0.

Planning, management, and maintenance of the v2024.0.0 release of PlanktoScope OS is led by Ethan Li (@ethanjli), with input and oversight from attendees of the weekly PlanktoScope software development meetings. If this release causes new problems for how you are using PlanktoScope, Ethan will take responsibility for addressing any new issues caused by this release - just tag them in a new Github Discussions post or in the #6-dev-software channel on the PlanktoScope Slack workspace!

Pull Requests

• Refactor GitHub Actions pre-caching of container images by @ethanjli in Refactor GitHub Actions pre-caching of container images #427
• Load container images into containerd, without QEMU VM by @ethanjli in Load container images into containerd, without QEMU VM #428
• Pin initial pallet for use in factory-reset process by @ethanjli in Pin initial pallet for use in factory-reset process #429
• Try to fix regression in starting hardware controller by @ethanjli in Try to fix regression in starting hardware controller #431
• Bump urllib3 from 2.2.1 to 2.2.2 in /documentation by @dependabot in Bump urllib3 from 2.2.1 to 2.2.2 in /documentation #433
• Bump ws, engine.io and socket.io-adapter in /software/node-red-dashboard by @dependabot in Bump ws, engine.io and socket.io-adapter in /software/node-red-dashboard #434
• Fix OS builds on PRs from external fork repos by @ethanjli in Fix OS builds on PRs from external fork repos #435
• Move v2.6 default hardware version to a FairScope-specific OS image by @ethanjli in Move v2.6 default hardware version to a FairScope-specific OS image #430
• Add an automatic timeout to the QEMU VM setup step by @ethanjli in Add an automatic timeout to the QEMU VM setup step #436
• Un-deprecate the planktoscope.local mDNS name by @ethanjli in Un-deprecate the planktoscope.local mDNS name #438
• Show dialog on homepage when config.json's hw version is null by @ethanjli in Show dialog on homepage when config.json's hw version is null #432
• Fix loading of default flowcell type into Fluidic Acquisition page by @ethanjli in Fix loading of default flowcell type into Fluidic Acquisition page #439
• Try to move repo checkout to start of OS image build workflow by @ethanjli in Try to move repo checkout to start of OS image build workflow #440
• Add CI workflows to fast-forward release branches upon tag push by @ethanjli in Add CI workflows to fast-forward release branches upon tag push #441
• Bump version numbers for v2024.0.0-beta.1 by @ethanjli in Bump version numbers for v2024.0.0-beta.1 #442
• Fix OS image filename generation from pushed release tags by @ethanjli in Fix OS image filename generation from pushed release tags #443

Full Changelog: software/v2024.0.0-beta.0...software/v2024.0.0-beta.1

---

This discussion was created from the release PlanktoScope OS (Sunray) v2024.0.0-beta.1.