OTA Update node to install firmware updates

[phoddie]

I've been working on an OTA Update node to allow firmware updates to be installed on remote devices.

The OTA Update Node sources are part of this repository at $NODEREDMCU/nodes/ota-update. It is available via npm as @moddable-node-red/ota-update.

The OTA Update node works in both push and pull modes. In pull mode, the firmware update is downloaded by the microcontroller and installed. The download is triggered by a message sent to the OTA Update with the URL of the firmware image to install. In push mode, the firmware image is uploaded to the microcontroller, for example using curl. Push mode is enabled by specifying the path of the http end-point to receive the firmware upload.

When the firmware has been received, the OTA Update node sends a message. The message can be routed to the MCU Restart node to reboot the MCU to begin using the new firmware. The OTA Update does not automatically restart the microcontroller so that the flow controls the timing of the restart.

There are two example flows, one that demonstrates push / upload mode and the other pull / download.

The node's documentation includes important details, including where to locate the firmware image created by a build.

It could be interesting for the MCU Plugin to use push mode to directly deploy updates to remote devices. I'm not sure how that might fit, but if it could be done, it would simplify the process of managing the firmware on devices.

Important notes:

• OTA Update currently supports the ESP32 family of microcontrollers
• The ESP32 must have OTA partitions configured in the build (as required by the ESP-IDF). This is not normally the case with Moddable SDK builds. With an update to the Moddable SDK in the coming days, any firmware build that contains the OTA Update node will automatically have the required OTA partitions. I'll update this note when that's available. (The Moddable SDK support is live as of April 12, 2023)
• Once the OTA Update node is considered stable and the Moddable SDK updates are available, this node will added to the Node-RED Library so it can be installed and managed from the Node-RED Palette.

[colinl]

This looks great

• OTA Update currently supports the ESP32 family of microcontrollers

I haven't got any ESP32, but the price they are now there doesn't seem much point using esp8266 (though if one wanted to run off a battery perhaps the consumption is lower). I have ordered 6 off Wemos S2 Mini from Ali Express, total cost £14.08, which is ridiculous. I hope those are ok with node-red-mcu.

[phoddie]

Great price on the S2! From what's been explained to me, the ESP32-S2 is kind of the successor to the ESP8266 - in the ESP32 family but single core and Wi-Fi only (no BLE).

[phoddie]

In the initial notes above, I forgot to mention the biggest design choice. As you see, the OTA feature is implemented by nodes in the flow. The alternative would be to make OTA updates a feature of the Node-RED MCU runtime host itself. That would not require adding any nodes to the flow to support OTA. But, it would need some way to be configured, triggered, etc. That might be less work to implement (for me!) but would offer less control for developers building flows. I think the approach of having an OTA Update node (with a supporting MCU Restart node) is the more powerful approach. For example, it makes it very straightforward to initiate an OTA Update by posting the firmware URL to an MQTT topic, which would allow for batch updates of devices. It does mean adding a couple nodes to a project that wants to use OTA, but that seems manageable. I'm curious to see how that plays out for people.

[bartbutenaers]

@phoddie,
Nice work!!

Some remarks from first quick look:

1. Could it be that the Github version is more recent compared to the npm version? Because on Github I see two example flows, and in Nore-RED only one:

   

2. I would find it easier to understand if the node's info panel would have a separate explanation of push and pull. And that the example flow names also contain 'push' and 'pull', because that makes it easier to understand compared to PUT and download.

3. Although you have written very nice documenation, the both mechanisms are not exactly clear to me yet. I mean with the endpoints, url's, curl, ... Will need to draw a quick sketch as soon as I have time.

4. The documentation shows a curl command. Is the same possible with a http-request node? Because that would be the standard way for Node-RED user to execute an url.

5. Did I understand correctly that you need to build a bin (e.g. using Ralph's sidebar), and the deploy the bin via OTA. So you have a bin file that contains everything: not only the moddable SDK stuff and the ESP related stuff (from Espressiv), but also the Node-RED MCU flow. If so, it might be useful if you could describe the process in a short ordered list in the docs. Thanks!

6. I totally agree with you that such nodes are incredible powerfull. This way you allow people to build very easily their own deploy logic via other Node-RED nodes. You provide some example nodes, and they can make it as complex as they want. Brilliant!!

7. Just wondering how this could be used. Suppose I have N MCU's, and for each sensor I have a tabsheet in my flow editor. Now I want to update all my MCU's. So I need to build all the flows manually one by one. But I need to deploy each flow via OTA after building it, because otherwise my build would be overwritten by the next build? Because all builds currently end up in the same folder, or am I mistaken? Or is it the intention to build a Build-node in front of the OTA-update node? Just wondering how this could work for companies having a large number of sensors running.

8. At first sight I would have guessed that I should use the xs_esp32.bin file located in this path:

   

   But in your readme you show another path, in this case for Moddable Two. Personally I think it is easier to understand if you just say that they to use the path specified in the output of the build command.

9. And a very minor issue. For some reason - when I import your example flow - I see empty tabsheets:

   

   Never seen that before...

BTW I will be computerless this week, but I will try to follow this discussion and test it next weekend. So unfortunately I can't test it now...

Bart

[phoddie]

@bartbutenaers – thank you for the carefully considered feedback. I have tried to follow the model of the built-in nodes when preparing the documentation for the nodes I implement to respect the community conventions. From your comments, it seems that is not required. So.... I reworked the documentation following your thoughts. I also updated names to use Push and Pull consistently throughout. The OTA Update node in the Node-RED MCU repository and the npm package have both been updated.

Could it be that the Github version is more recent compared to the npm version? Because on Github I see two example flows, and in Nore-RED only one:

You are looking at the mcu-restart node, which only has one example. The OTA Update node really does have two. ;)

I would find it easier to understand if the node's info panel would have a separate explanation of push and pull. And that the example flow names also contain 'push' and 'pull', because that makes it easier to understand compared to PUT and download.

Done!

Although you have written very nice documenation, the both mechanisms are not exactly clear to me yet. I mean with the endpoints, url's, curl, ... Will need to draw a quick sketch as soon as I have time.

I'm afraid I feel that way when reading the documentation many of the Node-RED nodes. ;) Since embedded firmware updates are new to you, it will take a little time and effort to understand. That said, the documentation could be more clear. I hope the updated version helps a bit here.

The documentation shows a curl command. Is the same possible with a http-request node? Because that would be the standard way for Node-RED user to execute a url.

I've kept curl as an example for testing. It is pretty widely used. I don't know that the HTTP Request node will work here, because it does a multi-part MIME upload which is not the same as HTTP PUT. I'm happy to integrate an example using the HTTP Request node if someone can help me understand how to do that.

Did I understand correctly that you need to build a bin (e.g. using Ralph's sidebar), and the deploy the bin via OTA. So you have a bin file that contains everything: not only the moddable SDK stuff and the ESP related stuff (from Espressiv), but also the Node-RED MCU flow.

That's correct. There's nothing special to do though - that file is already generated by every build you perform.

Just wondering how this could be used. Suppose I have N MCU's, and for each sensor I have a tabsheet in my flow editor. Now I want to update all my MCU's. So I need to build all the flows manually one by one.

Good question. It is a little out of scope here. The OTA Update node would be in the flow on the MCU, but the tooling to build and apply the updates is something else. The MCU plug-in might be able to provide this kind of functionality, though its current design seems focused on working with one device at a time (quite reasonably!).

Because all builds currently end up in the same folder, or am I mistaken?

I'm not sure how the MCU plug-in manages that. I defer to @ralphwetzel!

Or is it the intention to build a Build-node in front of the OTA-update node

No. The OTA Update node runs on the MCU, not on your computer. It is responsible for applying the firmware update to the MCU. Building and managing firmware images is something else. One could build nodes that run on the computer to help manage that, but that's really something else.

Just wondering how this could work for companies having a large number of sensors running.

There are entire businesses dedicated to OTA strategies for fleets of microcontroller powered products. Let's not try to recreate that here. ;)

As background, for those scenarios it is common to use Pull mode where an MQTT message received by the MCU contains the URL of the firmware update to download and install. Those firmware images would normally be created through an automated batch file invoking the Moddable SDK build, not through a visual development front-end like the MCU plugin.

At first sight I would have guessed that I should use the xs_esp32.bin file located in this path:

But in your readme you show another path, in this case for Moddable Two. Personally I think it is easier to understand if you just say that they to use the path specified in the output of the build command.

In fact... the MCU plug-in generates a random project name, I believe to avoid conflicts. That's what you are seeing. I added a note about this to the docs. Not all uses of Node-RED MCU Edition go through the MCU plug-in, so the documentation needs to cover the generate case to be complete.

And a very minor issue. For some reason - when I import your example flow - I see empty tabsheets

The flows didn't have labels set. I added those. Perhaps it helps?

[bartbutenaers]

@phoddie: thanks for the extensive feedback! Will get back to you next weekend, when I have tested it.
BTW your documentation was already good, but due to my lack of knowledge about the topic, my digest interval is a bit longer as e.g. for @colinl :-)

[mtoko]

Thanks for the great work
I have been trying the OTA and restart examples and have added these to a simple flow (already working) that is only an inject node and an MQTT out node. The build fails and the error is
Error: 'undefined/nodes/ota-update/manifest.json': manifest not found!

I have
1.) Pulled the latest moddable sdk and built it
2.) Pulled the latest node-red-mcu and installed it (the OTA and restart nodes are in the palette)

I can see from the error this is a path issue, but not sure how to fix it
Any advice appreciated

[phoddie]

@mtoko – thanks for taking a look! As noted above, there are some changes to the Moddable SDK required to use the OTA Node. One of those changes is required to build flows with the OTA node. That's why you are getting the missing file error. Apologies for the trouble.

Fortunately, we finished testing the Moddable SDK changes yesterday so they should be live later today. I'll post an update here when they are.

[phoddie]

The Moddable SDK changes are live. You'll need to update your Moddable SDK install and rebuild tools.

Error: 'undefined/nodes/ota-update/manifest.json': manifest not found!

This error suggests that your Node-RED MCU installation is not up-to-date. Would you check your node-red-mcu/manifest_runtime.json file to make sure it has this line:

		"NODEREDMCU": "./"

[bartbutenaers]

Hi @phoddie,
I'm back ...

The OTA Update node runs on the MCU, not on your computer.

Aaaah, I fell into this trap again. After years of doing Node-RED, my brain is so used that all nodes run on the NodeJs backend. So after I had a quick look at your push and pull flow, it made no sense to me at all. However when I look again (knowing that the Ota-Update node runs on the MCU), it indeed looks very logical ;-)

Allthough it might look very obvious (from your point of view) that it runs on the MCU, it might perhaps be helpfull if you put in bold letters (at the start of the docs in the info panel) that it runs on the MCU...

This error suggests that your Node-RED MCU installation is not up-to-date. Would you check your node-red-mcu/manifest_runtime.json file to make sure it has this line

I have the same error as @mtoko, although I think everything is up-to-date:

1. I have added your push demo flow to my MCU flow.

2. I have updated the Moddable SDK on my Raspberry. As you can see I am on the latest commit (d77e86):

   MCU Build system check: p1.3.1 + #94677d2 @ m3.8.0-20-gd77e86
   

3. I have updated the ESP32 tools. Not sure if that is required every time, because it says after the pull that everything is up to date already?

4. I have reinstalled Ralph's sidebar to make sure I have the latest node-red-mcu.

5. I have checked the file, and your change is available:

   pi@raspberrypi:~/.node-red/node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu $ cat manifest_runtime.json
   {
        "build": {
                "NODEREDMCU": "./"
        },
        "include": [
                "./manifest_core.json",
   

6. I tried to run the build with "clean" target, but that fails due to the same error.

If you have any tips, please let us know!

[bartbutenaers]

Just to confirm, I also get the same error as @mtoko currently has:

Error: './/nodes/ota-update/manifest.json': manifest not found!

[phoddie]

@bartbutenaers, I know you are using the MCU plug-in. @mtoko, are you also using the MCU plug-in?

[mtoko]

Yes

[phoddie]

Thanks. I suspect something about that environment is confusing things. I'll take a look.

[phoddie]

Good news, I think. I made a bad assumption about how environment variables are resolved in the build section of manifests. It worked in my test environment but not in the MCU plug-in environment. I've updated mcconfig to provide the behavior I expected. With that in place, the OTA Update node builds and runs as expected with the MCU plug-in. It even shows status in the Node-RED Editor as the firmware is installed.

The only required changes are in the Moddable SDK -- update and rebuild tools.

I also pushed an update to the OTA Update node. It updates the docs based on feedback from @bartbutenaers and fixes a bug in error reporting when an invalid firmware image is used.

[bartbutenaers]

BTW thanks for implementing some of my feedback!!!

Some other remarks:

1. I think there is somewhere a typo in the push demo, because there are 3 different paths to the endpoint:

   
   
   

2. Based on your experience, are there any (dis)advantages of the pull versus push mechanisms? Things like security or some other things? If so, it might be usefull if these could added to the readme page, to assist people in choosing one of those two.

3. Am I correct that a disadvantage of the pull model is that there needs to be an endpoint started in Node-RED (preferrably), which makes the firmware available?

4. For troubleshooting afterwards, what is the best (read easiest) way to check if the endpoint on the MCU is healthy? I could not test it yet, but I assume it returns some kind of http status code?

5. I think it should be possible to push the firmware using a http-request node (which is located on a non-MCU tabsheet ;-) ), after reading this stackoverflow discussion.

[phoddie]

Allthough it might look very obvious (from your point of view) that it runs on the MCU, it might perhaps be helpfull if you put in bold letters (at the start of the docs in the info panel) that it runs on the MCU...

In fact, there is something like that quite near the top. ;)

It isn't bold, but it is very blue. I can make that even stronger.

I think there is somewhere a typo in the push demo, because there are 3 different paths to the endpoint:

I noticed that too. It should be consistent in version 0.8.2 which was pushed late last night.

Based on your experience, are there any (dis)advantages of the pull versus push mechanisms? Things like security or some other things? If so, it might be usefull if these could added to the readme page, to assist people in choosing one of those two.

There are good reasons for both. Certainly having an open http listener provides no security, but if you trust your local network, it isn't an issue either. I'll add a few notes, but it definitely starts to move into opinion territory.

Am I correct that a disadvantage of the pull model is that there needs to be an endpoint started in Node-RED (preferrably), which makes the firmware available?

It is true that there needs to be an endpoint somewhere with the firmware. I'm not sure I would characterize that as a disadvantage or advantage. It depends on your project needs.

For troubleshooting afterwards, what is the best (read easiest) way to check if the endpoint on the MCU is healthy?

The output message from the OTA Update node indicates successful completion of both Push and Pull OTA updates. Details in the docs.

I think it should be possible to push the firmware using a http-request node (which is located on a non-MCU tabsheet ;-) ), after reading this stackoverflow discussion.

Thanks for the reference. From the docs of the HTTP Request node, I think it could work too. You would do something like:

• Use a Read File node to load the firmware into a buffer
• Use the HTTP Request node to upload the buffer with method set to PUT

[bartbutenaers]

In fact, there is something like that quite near the top. ;)

Ah yes that should be clear enough. Will then be a problem with my brain, which seems to be not accepting that some nodes don't run at the server side :-) So ignore my question...

From the docs of the HTTP Request node, I think it could work too

Will try it as soon as I have the Ota-Update node running.

it definitely starts to move into opinion territory

Yes true. But I take into account the opinion of people from a limited subset of humanity. And you are certainly within that subset ;-)

[phoddie]

I'd 'like to share a quick update on the OTA Node. @mshioji did some experiments with Node-RED desktop to PUSH a firmware update to Node-RED MCU using the HTTP Request. It works!

You can try it yourself. On the Node-RED MCU, use the example PUSH flow; on Node-RED desktop, it is a simple flow:

Node-RED desktop push OTA flow

[
    {
        "id": "17e0510a0d9b5ade",
        "type": "tab",
        "label": "ota push server",
        "disabled": false,
        "info": "",
        "env": [],
        "_mcu": {
            "mcu": false
        }
    },
    {
        "id": "b2e2dcf7e0c4ce27",
        "type": "http request",
        "z": "17e0510a0d9b5ade",
        "name": "",
        "method": "PUT",
        "ret": "bin",
        "paytoqs": "ignore",
        "url": "http://10.0.0.31/ota/firmware",
        "tls": "",
        "persist": false,
        "proxy": "",
        "insecureHTTPParser": false,
        "authType": "",
        "senderr": false,
        "headers": [],
        "_mcu": {
            "mcu": false
        },
        "x": 490,
        "y": 60,
        "wires": [
            [
                "b4504790f64ed12e"
            ]
        ]
    },
    {
        "id": "f859529f6c01f1d2",
        "type": "file in",
        "z": "17e0510a0d9b5ade",
        "name": "xs_esp32.bin",
        "filename": "/Users/user_name/Projects/moddable/build/bin/esp32/moddable_two/debug/balls/xs_esp32.bin",
        "filenameType": "str",
        "format": "",
        "chunk": false,
        "sendError": false,
        "encoding": "none",
        "allProps": false,
        "_mcu": {
            "mcu": false
        },
        "x": 310,
        "y": 60,
        "wires": [
            [
                "b2e2dcf7e0c4ce27"
            ]
        ]
    },
    {
        "id": "bc56d7cb09a560df",
        "type": "inject",
        "z": "17e0510a0d9b5ade",
        "name": "start update",
        "props": [
            {
                "p": "payload"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "_mcu": {
            "mcu": false
        },
        "x": 150,
        "y": 60,
        "wires": [
            [
                "f859529f6c01f1d2"
            ]
        ]
    },
    {
        "id": "b4504790f64ed12e",
        "type": "debug",
        "z": "17e0510a0d9b5ade",
        "name": "log",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": {
            "mcu": false
        },
        "x": 650,
        "y": 60,
        "wires": []
    }
]

I've committed a couple of small updates to the OTA and HTTP nodes in Node-RED MCU Edition to improve error handling when things go wrong during an OTA (incorrect path, invalid firmware image, etc).

By the way, @mshioji is one of the authors of this great new book about Node-RED MCU Edition.

[colinl]

My S2 Minis have arrived. What device should I specify for mcconfig? There doesn't seem to be an specific device type for it.

[phoddie]

@colinl – there's support for three ESP32-S2 development boards - esp32/kaluga, esp32/saola_wroom, and esp32/saola_wrover. The S2 Mini is bare bones, so the esp32/saola_wroom is probably closest -- the only feature the Saola Wroom has over a bare board is a single Neopixel.

But... the S2 Mini is only programmable over USB. There's no serial bridge. That can take some tweaking to get working. I'm not sure you want to wrestle with that, so I ordered an S2 Mini so I can try that out.

[colinl]

Googling suggests it is put into serial programming mode by holding down the 0 button and hitting Reset, and indeed I can see it connecting as /dev/ttyACM0 when I do that. When it tries to download however I get

esptool.py v3.3.2
Serial port /dev/ttyACM0
Connecting...Traceback (most recent call last):
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 5482, in <module>

    _main()
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 5475, in _main
    main()
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 4782, in main
    esp = esp or get_default_connected_device(ser_list, port=args.port, connect_attempts=args.connect_attempts,
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 116, in get_default_connected_device
    _esp.connect(before, connect_attempts)
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 689, in connect
    last_error = self._connect_attempt(mode=mode, usb_jtag_serial=usb_jtag_serial, extra_delay=extra_delay)
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 639, in _connect_attempt
    self.bootloader_reset(usb_jtag_serial, extra_delay)
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 623, in bootloader_reset
    self._setDTR(False)  # IO0=HIGH, done
  File "/home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py", line 557, in _setDTR
    self._port.setDTR(state)
  File "/home/colinl/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages/serial/serialutil.py", line 603, in setDTR
    self.dtr = value
  File "/home/colinl/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages/serial/serialutil.py", line 473, in dtr
    self._update_dtr_state()
  File "/home/colinl/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages/serial/serialposix.py", line 715, in _update_dtr_state
    fcntl.ioctl(self.fd, TIOCMBIC, TIOCM_DTR_str)
OSError: [Errno 71] Protocol error
CMake Error at run_serial_tool.cmake:56 (message):
  /home/colinl/.espressif/python_env/idf4.4_py3.10_env/bin/python
  /home/colinl/esp32/esp-idf/components/esptool_py/esptool/esptool.py --chip
  esp32s2 failed

Perhaps that is what you meant by having to tweak things. If so then I will just have to be patient while we wait for yours to arrive.

[phoddie]

Yes, that's the area where I would expect a challenge. It looks like the S2 Mini I ordered will arrive over the weekend, so I should be able to try this out soon.

[phoddie]

As we are a little far from the original topic, I've opened a separate discussion about the S2 Mini.