Debug MCU via node-red editor

[ralphwetzel]

@phoddie
When running a flow, the NR runtime sends messages back to the editor to display display debug data.
With reference to xsbug, I see - for this feature - a significant overlap. It seems to me, that the MCU (Simulator) send messages back to xsbug, that are displayed there.
If there was a documentation of the protocol, I could create a logic that connects to the MCU (Simulator) & catches this information - to display it in the editor. This could eliminate the need to operate with an additional tool & make life easier for people...
Of course xsbug provides additional information & options - which yet might only be necessary for experienced operators.

[phoddie]

@ralphwetzel – Having a debugger for JavaScript is a feature. I personally cannot imagine developing a project of any complexity without one. We have that feature. I certainly wouldn't want to take it away.

I understand that Node-RED generally runs without a JavaScript debugger so people working with Node-RED are accustomed to that. To maintain that experience, an option to run without xsbug would be fine.

The Node-RED editor provides feedback about running projects. As I understand it, that is done using the status set on nodes. The Node-RED MCU runtime has that information. It could relay that back to the editor. That's not trivial. The communication path used by xsbug varies by device and simulator. It can be serial (ESP32, for example) or a TCP socket (simulator, for example).

I don't know that JavaScript debugging and display of status in the Node-RED editor need to be mutually exclusive. It seems possible that they could co-exist and that it would be valuable if they did.

Would you consider inverting your suggestion? Rather than starting with the xsbug protocol, let's start by understanding the data that get sent to the Node-RED Editor for status and how that is communicated to Node-RED. Since we are not running inside the Node runtime, that's going to be quite different.

[ralphwetzel]

@phoddie
You intension to begin with implementing the functionality of the .status() feature looks like a good idea. I briefly checked the NR source - and both topics, .status() & debug, seem to be implemented quite similar. Could you please provide a small flow that we use as test bed? I anticipate that there will be work to be done on both ends of the interface...

[ralphwetzel]

Ok. Here's the proposal of how to implement the .status() functionality:

MCU end:
The Node status is an object of three properties (fill, shape, text). In addition, I need the id of the node that is sending it's status; this info is present in the flows.js JSON:

{
  "id": "e101f979cfca0e99",
  "status": {
    "fill": "green",
    "shape": "dot",
    "text": "I feel good..."
  }
}

This object shall be sent (e.g. JSONized) to a server running in NR that is awaiting incoming calls. I propose to send this initially via websockets - if feasable on MCU side. Host & port data could be (transparently) provided to MCU as part of the build process. When this is operational w/ the sim, we need to find a way of how to integrate a real hardware. Do you have a proposal for a setup?

NR end:
As part of the build process, the mentioned server will be opened. Incoming status data will then be injected into the NR system.
I would prefer to be able to kill the instances of mcsim & xsbug spawned as part of the build process - to ensure there's always a clean test environment. Do you see an option to get their PID's returned somehow?

[phoddie]

I wouldn't recommend using the Wi-Fi connection to carry the status information. There are some problems:

1. Not all MCUs have a network connection. The Raspberry Pi Pico and nrf52 are both examples of projects that could run Node-RED MCU but don't have Wi-Fi.
2. Buffering. When sending data over Wi-Fi, eventually the internal send buffers of the MCU fill and no more data can be sent until they empty. Managing that is complex and potentially uses quite a bit of memory (which is limited).

Carrying the status (and debug) data over the xsbug connection avoids both of these problems because the debugging communication with an MCU usually runs over serial.

Once the status / debug data is available over the xsbug connection, then it can be relayed to Node-RED as you describe.

[ralphwetzel]

It's a very fair point to consider that MCUs (usually) use a serial connection for debugging purposes. In that case, we should concentrate on a serial protocol rather than demanding a "man in the middle" setup. If an operator yet likes to use xsbug, relaying status & (later) debug back to NR should keep a consistent experience.

[phoddie]

In that case, we should concentrate on a serial protocol...

But the simulator connects to xsbug using a TCP connection. ;) A general solution needs to support both. The xsbug connection is the one place the Moddable SDK guarantees that a connection is available. From a compatibility and maintenance perspective, it is the most direct path. It also ensures that the JavaScript debugger is available when desired.

[phoddie]

FWIW – I have an idea for how to implement this which I think will work well for both of our goals. I need to think about it a bit more before trying to explain.

[ralphwetzel]

👍

[phoddie]

@ralphwetzel, I've committed the skeleton of a possible solution.

• Works with the simulator
• Works with all MCUs
• Re-uses existing xsbug communication
• Allows Node-RED to receive messages (status, debug, etc)
• Allows xsbug to operate as usual for debugging JavaScript
• Works without xsbug

The implementation is at tools/xsbugproxy. As the name suggests, it is a proxy for the xsbug connection. It is written in Node. The Moddable SDK project connects to the proxy. The proxy optionally connects to xsbug. I've tested the proxy running stand-alone. I assume you would update it so it relays messages to the Node-RED runtime.

The example has comments that attempt to explain the details. It is a little rough, but hopefully you can work through it.

N.B. The proxy does not currently extract any messages (status or otherwise). There is a comment where that should be done. Note that message may be fragmented, especially when arriving from an MCU, so the message parsing will also need to potentially combine fragments to create a complete message.

[ralphwetzel]

@phoddie, this sounds like a promising concept.
Let me check how to integrate it...

[ralphwetzel]

node-red-mcu/tools/xsbugproxy/index.js

Lines 5 to 14 in a9aebf7

	/*
	Moddable SDK prrojects connect to xsbug at loaclhost:5002.
	This proxy listens on loaclhost:5004. To get the Noddable SDK project to connect to the proxy
	requires a manual source code change. On macOS, in $MODDABLE/xs/platforms/mac_xs.c, change
	address.sin_port = htons(5002);
	to
	address.sin_port = htons(5004);
	Linux and Windows have similar changes. Eventuall We can add an option to mcconfig to set the xsbug
	port.
	*/

There's already a logic implemented to alter the port from 5002 to 5003.
Would it be possible to trigger this change on demand & "reuse" this logic ... eliminating the need to patch the code to support the proxy?

[phoddie]

@ralphwetzel - Good find. It is there to allow debugging of xsbug (which is written in JavaScript using the Moddable SDK) with xsbug. The trigger is the name of the executable. My feeling is we need something more general but I'm not sure yet. Either way it should be too difficult.

Also, to use this proxy with an MCU requires changing the port used by serial2xsbug similar to the change for macOS. I'll track that down later.

[ralphwetzel]

node-red-mcu/tools/xsbugproxy/index.js

Lines 65 to 68 in a9aebf7

	// parse messages here
	// each message is an XML document
	// status messages are sent in a bubble right message of the form:
	// <xsbug><bubble name="" value="2" path="/Users/hoddie/Projects/moddable/examples/helloworld/main.js" line="18">JSON STATUS MESSAGE HERE</bubble></xsbug>

Unfortunately the info sent is not sufficient to display the status next to the "emitting" node in the editor. What I need (in the message sent) is the ID that identifies this node within NR. As said, this ID is available in the JSON that is provided as flows.js. I propose to hand this over as a standard property when creating the instance of the class / node ... and then have it at hand when necessary.

Sorry. I was listening to debug... status() already sends the right data!

[phoddie]

I think it would help me to have an example of a simple flow that generates status and the JSON message it generates in Node-RED for status. That would give something to emulate and ensure we are looking at the same thing. Would you provide that?

By the way, were you able to run the proxy successfully?

[phoddie]

OK, great! status is correct. How about debug?

[ralphwetzel]

@phoddie:

By the way, were you able to run the proxy successfully?

Yes. Great idea...

OK, great! status is correct. How about debug?

This is how status looks like:

   {
      xsbug: {
        bubble: {
          _name: '',
          _value: '1',
          _path: '/Users/ralph/Projekte/node-red-mcu-test/nodered.js',
          _line: '237',
          __text: '{"status":{"text":1658081729238,"source":{"id":"799b7e8fcf64e1fa","type":"debug","name":"debug 4"}}}',
          toString: [Function: toString]
        }
      }
    }

And here's an example for debug:

    {
      xsbug: {
        bubble: {
          _name: '',
          _value: '2',
          _path: '/Users/ralph/Projekte/node-red-mcu-test/nodered.js',
          _line: '325',
          __text: '{"payload":1658081681239,"topic":"","test":"tester","xxx":true}',
          toString: [Function: toString]
        }
      }
    }

As you can see, status comes with source - which holds the info necessary to get the "emitting" node.

[phoddie]

I've update the debug node so it includes a source property on the JSON that goes to the sidebar. That should give you what you need.

[ralphwetzel]

Proof of concept - that still needs a lot of polishing:

mcustatus.mov

Would it be possible to send a message through the proxy to the MCU ... to e.g. forward a button press on an inject node to the flow running on the MCU?

[phoddie]

Great work! That's a lot of different pieces to have working together.

Would it be possible to send a message through the proxy to the MCU ... to e.g. forward a button press on an inject node to the flow running on the MCU?

Yes, it should be possible to relay a message over the xsbug connection to the MCU. Once the message gets to the MCU, there is some additional runtime work needed to dispatch it to the node, and implement the expected behavior. The details will require some experimentation.

[phoddie]

@ralphwetzel – There's an update that supports triggering an inject node over the xsbug connection.

• The implementation is easily extensible to send other triggers as needed
• xsbugproxy is updated to show how to send the inject trigger
• Works in the simulator. It will work on MCUs too, with a small Moddable SDK update

[ralphwetzel]

👍 I'll check how to utilize this...

[ralphwetzel]

@phoddie
It lasted a bit longer than I expected - but here it is: I've created a plugin to integrate this great add-on into the node-red world.
Be aware that it's currently in the state of a PoC & not at all stable.
You'll find some words of what it does & how to install it in the repo: node-red-mcu-plugin.
The PR I raised some minutes ago is necessary to enable the status output of the debug node (of the node-red-mcu side).

[phoddie]

Any reason why this is necessary for the sim? The proxy operation should have no impact on the sim - to my understanding... ?

@ralphwetzel – I'm not certain I understand. Node-RED MCU project run on your computer (macOS, Windows, Linux) use the simulator (mcsim process). When use your MCU Build tool inside the Node-RED editor, it launches the mcsim to run the flows and that connects to the proxy.

[ralphwetzel]

I'm not certain I understand.

Never mind. I just didn't get why the sim needs to know anything about xsbug...

[phoddie]

No problem. If there's something that still doesn't make sense, let me know. I'm happy to try to explain.

[ralphwetzel]

Be sure to rebuild tools after updating. This new capability is implemented for the following:

• macOS simulator
• Linux simulator
• ESP8266 builds on macOS and Linux
• ESP32 builds on macOS and Linux
• Raspberry Pico builds on macOS and Linux

Some notes:

• Windows support is not yet ready. We'll get there.

@phoddie -- Did you find the time to implement this for Windows as well? I managed to build a flow there successfully ... but it's lacking the features to interact with the editor. I already spent some time checking for errors ... and then remembered this exchange from the early days of our journey! 😄

[phoddie]

No progress here, but the notes above aren't quite correct either. Some details:

• For embedded build target (ESP8266, ESP32, Raspberry Pi Pico) Node-RED MCU Edition supports everything that the Moddable SDK supports:
  • ESP8266 - macOS, Windows, Linux
  • ESP32 - macOS, Windows, Linux
  • Raspberry Pi Pico - macOS, Linux
• The Moddable SDK simulator (mcsim) works on macOS, Windows, and Linux. However, the ECMA-419 network classes (TCP, Listener, and UDP) are not yet implemented on Windows. That's very do-able but it is some work.

For any further discussion about this, let's open a new topic, since this one is pretty far from where it started.

[ralphwetzel]

@phoddie Is there a way to enumerate (all) the possible platform identifiers (like esp32/moddable_two)?

[phoddie]

Is there a way to verify this - checking e.g. the presence of a file/folder?

Unfortunately, no. The tool set-up for the embedded targets is complex, and each one is different (silicon manufacturer specific!) There's no easy way to verify that they are set-up correctly. xs_dev might be able to provide an indication when it is used for set-up of the embedded tools.

[phoddie]

I hoped for a built-in function ...

The best I think we could do is to automate what I described in a Moddable SDK tool. That would be more convenient for you and xs-dev. In the future, that could be updated if other mechanisms become available.

[ralphwetzel]

👍

[HipsterBrown]

In xs-dev, I opted for the quickest solution of manually creating the list of available platforms: https://github.com/HipsterBrown/xs-dev/blob/main/src/commands/run.ts#L33-L70

There is no environment validation when providing those options within xs-dev at the moment; however, this should not be too difficult to implement given some knowledge of the tooling required for various targets, i.e. making sure the Espressif SDK is available for ESP8266.

Dynamically creating the list of available device / simulator targets could also be accomplished in a similar manner as the list of Moddable examples: https://github.com/HipsterBrown/xs-dev/blob/main/src/commands/run.ts#L89-L97 🤔

[phoddie]

I opted for the quickest solution of manually creating the list of available platforms

That's very pragmatic. The list doesn't change very often, so that's manageable.

There is no environment validation when providing those options within xs-dev at the moment; however, this should not be too difficult to implement given some knowledge of the tooling required for various targets, i.e. making sure the Espressif SDK is available for ESP8266.

This will always be a bit fragile but as a first-order filter it should work pretty well.

[ralphwetzel]

@phoddie

I'm working on the xsbug port support and look forward to the UI improvements.

The UI update is available now. I'm not sure how this plays with your latest changes though; going to check this as a next step...

[phoddie]

Your latest works well! It has become a very big project (maybe more total lines of code than the core of Node-RED MCU itself?!?!)

The configuration editor is quite an accomplishment. It also reminds me that it is getting to be time to build some nodes that draw to the screen.

A few comments:

• It appears ot be required to Deploy in Node-RED before building for the MCU. If not, a stale flow is used. It took a little while to figure that out. Is it possible to build the current editor state instead of the most recent deploy?
• The Instrumentation pop-up is a little confusing with two checkboxes to control three options. Maybe it can just be a pop-up with "debug", "release", and "instrumented" with the label "build type"
• When there are several flows, the last flow is not shown in the "Flows to build for MCU" list

Having the MCU build available so easily let me quickly run a bunch of different flows in different environments. That uncovered a few bugs in my code, so I committed a few fixes. ;)

[ralphwetzel]

Is it possible to build the current editor state instead of the most recent deploy?

No, that's not possible. We need to inform the runtime that MCU is in control for the selected flows - and do some modifications behind the scene. That's why all nodes (in the selected flows) get flagged as changed (blue ball) & the deploy button illuminates as well. Yet this should only be necessary once...

• When there are several flows, the last flow is not shown in the "Flows to build for MCU" list

The panel is resizable. Could you please confirm, if it's really missing - or just hidden as overflown?

[phoddie]

The panel is resizable. Could you please confirm, if it's really missing - or just hidden as overflown?

You are right. It is there if I resize the panel enough. There's some bug with the scrolling.

We need to inform the runtime that MCU is in control for the selected flows - and do some modifications behind the scene. That's why all nodes (in the selected flows) get flagged as changed (blue ball) & the deploy button illuminates as well. Yet this should only be necessary once...

That's no problem for now. Maybe it should be shown in the UI somehow, to help people? Eventually we should get some support from the Node-RED Editor to make that smoother.

[ralphwetzel]

There's some bug with the scrolling.

Confirmed. I'll check...

Maybe it should be shown in the UI somehow, to help people?

Well, it is - as described. In addition to that: The Build button should only be active (and trigger-able) if there's at least one flow selected...

[ralphwetzel]

There's some bug with the scrolling.

Fixed now.