Design decision: RED interface

[ralphwetzel]

As already stated, there've been created currently > 4000 nodes for node-red. For this "spin-off" we need - in my eyes & at this early stage - a fundamental design decision: Is there the intension to re-use these existing nodes?

• If yes, node-red-mcu has to mimic the interface of class RED (& a few others) of standard node-red & to follow in-principle the given design definitions. ESP might still lack relevant capabilities ... but at least the already existing nodes (as long as they do not rely on special additional libraries) should be 'runnable'.
• If no, node-red-mcu is free in its decisions & interface definitions ... yet no available node will be usable without relevant (additional) work.

I strongly vote to follow the yes path - despite i know, that this means to re-do a relevant part of this project. I'm yet convinced, it's not too late to adjust direction ... for a better of this project in the long run.

[phoddie]

@ralphwetzel – you bring up an important. I think we are mostly in agreement. Let me share my current thoughts.

Please keep in mind that I come to this effort with zero background in Node-RED. I'm happy to work with you and others on this. While I don't know Node-RED well yet, I do have experience creating efficient embedded JavaScript. Both sets of knowledge will be required.

As much as practical, I would like the MCU runtime to match the full RED runtime. That's why, for example, the function to clone a message in the MCU runtime is accessed via RED.util.cloneMessage() as in the full runtime.

It will be interesting to see how much re-use of existing module code is possible. Even if code cannot be used exactly as-is, we should aim to minimized the changes required. Beyond the Node-RED MCU runtime itself there are other differences that impact re-use:

• The Moddable SDK runtime is not Node.js. While we can provide some of the same services, fundamentally it is a different runtime because there are different goals. Code that depends directly on Node.js will need to adapt.
• Some Node package dependencies (eg. Mustache.js, JSONata, JSON Schema) may just be too big to be practical, though meaningful subsets, however, may be possible.
• MCUs have much less memory and CPU power. Code that uses lots of either of those will be problematic.

All that said, what is very cool is that flows can be re-used. Even if nodes need to be reimplemented (which is what I've done so far), the fact that flows can work on an MCU without a change is kind of amazing. The work we do to make it easier to bring more nodes to the MCU runtime will grow the number of flows that will work.

I'm happy to rework the current implementation as necessary to provide better API / runtime compatibility with the full Node-RED. At under 1300 lines, it isn't that big yet. While quite functional, I view the current effort as a learning project and proof-of-concept, not the final approach. It will need to evolve. For example, I intended to rework when and how the exported JSON of the flows is processed to be able to generate much more efficient code.

[ralphwetzel]

For example, I intended to rework when and how the exported JSON of the flows is processed to be able to generate much more efficient code.

From my perspective many - if not all - of these topics circle around the question of "How to integrate node-red-mcu with legacy node-red?" - to make life easy for everyone & generate a positive user experience from the first moment on. I can see two paths to follow:

• Setup -mcu as a seperate(d) application: This leads to the situation, that - when intending to re-use existing nodes - this environment has to be created, in the sense of 'there's the need to install those node definitions another time'. On top, -mcu demands a significant amount of infrastructure e.g. to handle the links to node-red.
• Integrate -mcu into the node-red editor ... and run the dedicated stuff, e.g. the build process at the right time on demand. One infrastructure, all information accessible by the diffenrent APIs, the full power of node.js at hand e.g. to support the build process, the option to work with dedicated nodes for the -mcu hardware, the opportunity to patch existing nodes for -mcu, create a new build triggered by a deploy, and so on, and so on.

That said, I propose to create a plugin for the editor as a starting point for integration. I'll raise a PR as soon as I have an initial prototype up & running.

Final comment: To be fair, this requires node-red & node-red-mcu to be installed on the same system, e.g. locally.

[phoddie]

@ralphwetzel, thanks for the detailed thoughts.

I agree that integrating MCU support into the full Node-RED editor is optimal. That is more convenient for users, and it sets a clear direction that the MCU support is intended to be as compatible as practical with the full version.

FWIW – I think that Nodes can be grouped into the following categories:

1. Works everywhere. Supported by both the full and MCU runtimes.
2. Works everywhere with some limits. For example, the Change node works on the MCU but doesn't support JSONata expressions.
3. Works only on the full version.
4. Works only on the MCU Edition (features unique to the embedded devices)5.

For (1) and (2), there are a few possibilities for the MCU runtime implementation:

1. The implementation is built into Node-RED MCU. This is basically the code nodes.2.
2. The implementation for Node-RED MCU and full Node-RED are identical. For example, nodes are implemented only in pure JavaScript.
3. Separate implementations for Node-RED MCU and full Node-RED that provide equivalent functionality. One example of this is "rpi - gpio in" and "rpi - gpio out". These are supported on the MCU using the Ecma-419 Digital Class Pattern, rather than Raspberry Pi IO.

The Node-RED editor could help manage this so that, for example, the correct version of the runtime is selected.

I propose to create a plugin for the editor as a starting point for integration. I'll raise a PR as soon as I have an initial prototype up & running.

That sounds great. We'll learn a lot from that. Please let me know if I can help with information about the MCU work.

[phoddie]

@ralphwetzel, it looks like the page about the "JavaScript file" for creating nodes (which @Steve-Mcl helpfully pointed me to) is part of what you are talking about. You'd like the MCU Edition to support nodes that work in this same way, yes? That seems possible.

FWIW – the use of events registered at runtime (e.g. this.on('input'...) is a heavier mechanism than the current implementation uses. It will require more RAM and more time to dispatch. Still, in the interest of compatibility and familiarity, it is probably the right thing to do.

[phoddie]

FWIW – I implemented a Compatibility Node to try to run modules written for the full Node-RED. It is able to run the example lower-case node as-is.

There is some additional overhead (memory and CPU) compared to the built-in nodes. It seems a acceptable price for compatibility. Nodes can still be written optimized for the MCU runtime too.

[ralphwetzel]

A first teaser of the UI...

[mtoko]

Looks great!!

[ralphwetzel]

After some reconsiderations I decided to change from a plugin to a node.
The main reason is that, by going with a node, I can utilize the whole, already existing node-red infrastructure - whereas a plugin has to recreate most of the stuff.
Further details in the PR.

Teaser: