SIMOC Live planning / meta-issue

[ezio-melotti]

RPi image(s)

Ideally we should only have a base image for the RPi 4/5s and one for the 0s.
The base image should:

• have unnecessary dependencies removed to keep the size small
• have additional dependencies and utils installed
• have additional drivers (e.g. screens, WiFi dongles, etc.) installed
• have everything needed to run completely offline (i.e. without downloading anything)

These steps should also be automated as simoc-sam.py commands, so that:

• we can reuse them for creating new updated base images with more recent Linux versions
• they can be applied on different images too (default RPi images, custom images, etc.)

We should also investigate network boot, especially for the RPi 0s. The RPi 4/5 could provide images for the 0s.

RPi 4/5

We currently don't have an image for the RPi 4/5. We should create a new one:

• start with a default RPi 4/5 image
• remove unneeded dependencies to make it smaller (this might include X too, TBD)
  • possibly automate it as a simoc-sam.py command
• install missing dependencies
  • Add an initial-setup command to simoc-sam.py #131
• perform initial configuration (e.g. set up /etc/hosts, .bash_aliases)
  • Add an initial-setup command to simoc-sam.py #131

RPi 0

We already have a base image and a few derived images for different setups (mesh node, hotspot, etc.). Ideally we should finish automating the setups, stick to the base image, and use simoc-sam.py to switch between setups (see next section). See also:

• Update RPi 0 images #95
• Recreate RPi 0 image #65

Networking

NetworkManager

We decided to use NetworkManager and already implemented the hotspot and WiFi setups using it. The mesh setup still uses batman and other tools. It might be possible to reimplement the mesh setup using NM too. If this works, we should be able to get rid of other networking sub-systems and tools and do everything using NM:

• Support and automate different (network) configurations #98
• Reimplement the mesh setup using NetworkManager #107
  • Automate mesh network set up #106
• Add a simoc-sam.py command to connect to a WiFi network. #105
• Add a simoc-sam.py command to setup a hotspot. #101

autossh

Recently we configured autossh on the RPI 4 to ensure the SSH tunnel we set up stays up. This was done manually, but it should be automated as a simoc-sam.py command:

• automate autossh setup

Handle RPi 4 disconnections

We observed regular disconnections on the RPi 4. These seem to be happen when the DHCP lease expires and the RPi 4 doesn't request a renewal. This could either be a default behavior of the RPi 4, a power-saving feature, or something that we accidentally disabled either while (un)installing deps or while configuring the mesh network.

Once the images are recreated and we switched to NetworkManager, we should test if we still experience the same issue, and if so, fix it.

• Investigate and fix RPi 4 disconnections

Time synchronization

Since the time in the RPis tend to drift quite a bit (several seconds per week), in order to maintain consistency in the sensor readings, all devices should be synchronized. The RPi 0s should get the time from the RPi 4/5, and the RPi 4/5 should get it from the Internet -- if available.

• Set up NTP on the RPi 4/5 (server) and RPi 0s (clients)

systemd-timesyncd should be used to do this.

Remote command execution

It might be convenient to write some utility to execute a certain command on all the sensor arrays. This could also be used to push updates to the 0s.

Naming conventions

To support the deployment on multiple sites, we should:

• Define hostnames and passwords for the RPi 0s and 4/5s
• Define static IP addresses/names (in /etc/hosts)
• Define names for the MQTT topics

Both hostnames and passwords can be easily set directly from the imager while burning the image on the SD card. This could be exploited to get the name for the MQTT topic from the hostname: if we use a naming scheme like <location><type><id> (e.g. sambridge1), the location (e.g. sam) can be used as prefix for the MQTT topics.

Another option is to omit the location from the hostnames and just use <type><id> everywhere (e.g. bridge1). This will make scripts simpler, but it will require a separate config file that will need to be edited after burning the images to specify the location.

In order to access the devices by name (e.g. while using ping or ssh), we should configure /etc/hosts to map the names with the corresponding (static) IPs. Since these are only relevant within the mesh network, we should be able to hardcode them, with two caveats: first, the actual number of devices might change; second, if we use <location><type><id> the /etc/hosts files will have different content depending on the location (can be automated though).

As for the passwords, security within the mesh network shouldn't be a big concern, but the bridges should use a strong and unique password, since they are exposed to the Internet.

Dashboards

Once the data are collected, we have a few options to visualize them.

SIMOC Live dashboard

This is what we have been using, and it's fine while running on PCs, despite some of its limitations.

Thingboard

We have been looking into Thingboard, that seems to be an easier and more flexible solution. Due to its flexibility, it might be a better choice, especially while working with new sensors and setups, since it allows rapid deployment of new dashboards and panels.

TUI-based interface

A TUI-based interface on top of simoc-sam.py should be developed, and it will offer three main advantages:

• it will make using simoc-sam.py easier
• it can be used to inspect data through a text-only SSH connection
• it can be used on the RPi 4/5 3.5" screen

I'm looking into developing one using textualize:

• Create a basic TUI for sensor status and readings #94

Data collection/transmission

There are a few issues and PRs that still need to be reviewed and finalized:

• Add cvs writer and replayer #77
• Create mqttproxy.py #113
• Create mqttproxy.service #114

Data logging

The RPi 0s should keep a local log of the readings as a backup. This is currently done by writing each reading in a .jsonl file. This log shouldn't need to be accessed except for debugging or in case we need to recover lost data.

Currently we are generating about 2MB of data per array per day. With a 64GB SD, it would take over 80 years of continuous recording (every 10s) to fill up the SD. With smaller delays and more sensors it will still take a few years, but eventually the SD will fill up.

One option to avoid this could be using journald to log data, since it already provides settings to limit the max file size / retention period / etc.. These settings can be configured globally or on specific custom namespaces and journalctl can be used for log visualization/filtering (including specific time ranges). The simplest option would be printing the readings on stdout and only journal stdout, but there is also a Python API that provides more control.

The RPi 4 aggregates the readings received from the 0s and write them to a CSV file. This file is consumed by human, and there should be an easy way to restart it (e.g. at the beginning of a new mission). The simplest solution is probably to just move the existing CSV file into an archived dir, so that the CSV writer starts writing on a new file automatically. Moving the file could be done manually or through a simoc-sam.py command.

Scripts

We should create a dir with misc stand-alone scripts, including:

• script to convert data from .jsonl to .csv
• script to plot data
• script to install/setup things, invoked by simoc-sam.py
• Add a script to check for and re-establish network connectivity. #117

CI/CD

In order to keep everything running smoothly we should:

• Set up dependabot and pin dependencies #104
  • Pin dependencies #123
  • Enable dependabot for this repo #124

Currently we are testing with Python 3.10, but we are running this code on a number of devices with different versions of Python (the RPis, the beta server, etc.):

• Check which versions of Python we have
  • 3.9 on the 0s, 3.11 on the 4
• Test with multiple Python versions #129

We should also try to add tests for all the features we add and run them automatically.

• Add a GitHub workflow that runs tests. #16

We might want to consider publishing this on PyPI too.

Hardware

Sensors

We currently support a few Adafruit and Vernier sensors:

• Add driver for the VEML7700 sensor #103
• Add driver for the TSL2591 sensor #102
• Add driver for the SCD-4X sensor #74
• BME-688 driver #46
• SGP-30 driver #34
• Add vernier-WTHR sensor driver. #51
• Vernier sensor drivers #35

We have other Adafruit sensors that we should test, and add drivers for them.

• write drivers for the new sensors

Screen

We tested the GeeekPi 3.5" touchscreen and it works. The driver should be installed in the base RPi 4/5 image. A TUI should be developed for it (see above).

• Add a command to install the touchscreen. #130

Even though the screen supposedly has a resolution of 480x320, I was able to set it to 720x480 and 960x640 by editing this line at the end of /boot/config.txt:

-hdmi_cvt 480 320 60 6 0 0 0
+hdmi_cvt 960 640 60 6 0 0 0

Editing this file requires a reboot, but the resolution can additionally be changed directly using fbset -xres 960 -yres 640.

Since everything seems to work fine and the screen feels much less cramped, we might want to update the install-touchscreen command to update the resolution. This could be done by using sudo sed -i -e '/hdmi_cvt / s/480 320/ 720 480/' ./MHS35-show as described in goodtft/LCD-show#369 (comment).

• Set 960x640 as default res for the 3.5" screen #133
• Add a simoc-sam.py command to change the resolution (and possibly rotate the screen too)

WiFi dongles

We tested a couple of WiFi dongles to be used with the RPi 4/5. One was the TP-Link TL-WN823N. We should check again if it works out of the box or if it needs drivers. I found some notes that included the following commands, but they might only be necessary for a different dongle or maybe for the RPi 0s:

sudo apt-get install git raspberrypi-kernel-headers build-essential dkms
git clone https://github.com/clnhub/rtl8192eu-linux
cd rtl8192eu-linux/
vim Makefile
    CONFIG_PLATFORM_I386_PC = n
    ...
    CONFIG_PLATFORM_ARM_RPI = y
./install_wifi.sh

Inputs

We are looking at different input solutions for the RPI 4/5, including:

• Macro keyboards (USB)
• ANO Rotary encoder
• Touchscreen

Batteries/UPSs

We are also looking at different solutions for powering the RPis. Power banks work, but the ones we tested are unable to switch between pass-through and battery power without causing a reboot in the Raspberry. UPS units solve this problems, but they are generally more limited capacity-wise.

Circuit Playground Express

We have a few Circuit Playground Express. They include a few sensors (temperature, light, motion), 10 LEDs, microphone, speaker, etc.

LEDs

We also have a few LEDs that can be used for basic reporting (e.g. alerting about values that are outside range).