Testing against the Moddable Two board

[rei-vilo]

I ordered the Moddable Two board and rushed to test it when I received it. Very nice board with a crisp display.

I followed the procedure at Moddable Two Developer Guide to install the Moddable SDK and the ESP32 tools.

Tests with mcconfig succeed

I am using the minimal example.

cd $MODDABLE/examples/piu/balls
mcconfig -d -m -p esp32/moddable_two

The example builds and links, uploads and runs fine.

Tests with node-red-mcu-plugin fail

I am using the minimal example.

Code

[
    {
        "id": "9255b5caec02d914",
        "type": "inject",
        "z": "104f181135f2ef9c",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "_mcu": {
            "mcu": true
        },
        "x": 180,
        "y": 80,
        "wires": [
            [
                "6c7b846e3025a90b"
            ]
        ]
    },
    {
        "id": "6c7b846e3025a90b",
        "type": "debug",
        "z": "104f181135f2ef9c",
        "name": "Debug from MCU",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "_mcu": {
            "mcu": true
        },
        "x": 510,
        "y": 200,
        "wires": []
    }
]

Selected board is esp32/moddable-two. Build fails with

/Users/USER/esp32/esp-idf/export.sh: line 99: python: command not found

Log

Starting Build process...
Creating build environment for platform esp32/moddable_two.
Working directory: /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
$IDF_PATH identified: /Users/USER/esp32/esp-idf
> cd /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
> source $IDF_PATH/export.sh && mcconfig -d -x localhost:5004 -m -p esp32/moddable_two -t build ssid="LAN-0A61FC7B" password="MikNycmJdqAeNsmHsryA6x3t"
Detecting the Python interpreter
Checking "python" ...
/Users/USER/esp32/esp-idf/toUSER/detect_python.sh: line 16: python: command not found
Checking "python3" ...
Python 3.9.6
"python3" has been detected
Adding ESP-IDF toUSER to PATH...
Using Python interpreter in 
Checking if Python packages are up to date...
/Users/USER/esp32/esp-idf/export.sh: line 99: python: command not found

Python 3 has been detected but line 99 is hard-coded and calls python instead.

Changing line /Users/USER/esp32/esp-idf/export.sh: line 99 from python to python3

python3 "${IDF_PATH}/tools/check_python_dependencies.py" || return 1

Build goes further with fails anew.

IDF_PYTHON_ENV_PATH: (not set)

Log

Starting Build process...
Creating build environment for platform esp32/moddable_two.
Working directory: /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
$IDF_PATH identified: /Users/USER/esp32/esp-idf
> cd /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
> source $IDF_PATH/export.sh && mcconfig -d -x localhost:5004 -m -p esp32/moddable_two -t build ssid="LAN-0A61FC7B" password="MikNycmJdqAeNsmHsryA6x3t"
Detecting the Python interpreter
Checking "python" ...
/Users/USER/esp32/esp-idf/toUSER/detect_python.sh: line 16: python: command not found
Checking "python3" ...
Python 3.9.6
"python3" has been detected
Adding ESP-IDF toUSER to PATH...
Using Python interpreter in 
Checking if Python packages are up to date...
The following Python requirements are not satisfied:
click>=7.0
pyserial>=3.3
cryptography>=2.1.4
pyparsing>=2.0.3,<2.4.0
pyelftoUSER>=0.22
idf-component-manager~=1.0
gdbgui==0.13.2.0
pygdbmi<=0.9.0.2
python-socketio<5
jinja2<3.1  # See https://github.com/espressif/esp-idf/issues/8760
itsdangerous<2.1
kconfiglib==13.7.1
reedsolo>=1.5.3,<=1.5.4
bitstring>=3.1.6
ecdsa>=0.16.0
construct==2.10.54
Please follow the instructions found in the "Set up the toUSER" section of ESP-IDF Getting Started Guide
Diagnostic information:
    IDF_PYTHON_ENV_PATH: (not set)
    Python interpreter used: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
    Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
    PATH: /Users/USER/.espressif/toUSER/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin:/Users/USER/.espressif/toUSER/xtensa-esp32s2-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32s2-elf/bin:/Users/USER/.espressif/toUSER/xtensa-esp32s3-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32s3-elf/bin:/Users/USER/.espressif/toUSER/riscv32-esp-elf/esp-2021r2-patch3-8.4.0/riscv32-esp-elf/bin:/Users/USER/.espressif/toUSER/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/USER/.espressif/toUSER/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/USER/.espressif/toUSER/openocd-esp32/v0.11.0-esp32-20220411/openocd-esp32/bin:/Users/USER/esp32/esp-idf/toUSER:/usr/bin:/bin:/usr/sbin:/sbin:/Users/USER/Projects/moddable/build/bin/mac/release

I went to espressif/esp-idf#8760 and applied the recommend fix, to no avail.

% pip install "jinja2<3.1"  "itsdangerous<2.1"

Requirement already satisfied: jinja2<3.1 in /Users/USER/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages (3.0.3)
Requirement already satisfied: itsdangerous<2.1 in /Users/USER/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages (2.0.1)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/USER/.espressif/python_env/idf4.4_py3.10_env/lib/python3.10/site-packages (from jinja2<3.1) (2.1.1)

I checked IDF_PYTHON_ENV_PATH

% echo $IDF_PYTHON_ENV_PATH
/Users/USER/.espressif/python_env/idf4.4_py3.10_env

Questions

I do not understand why mcconfig works and not node-red-mcu-plugin.

Is there something wrong with my configuration?

Thank you for your help!

[ralphwetzel]

Thank you for reporting your challenges. Let's try to sort things out step by step:

Initial question: What's the platform you're building on? macOS or Linux or ... ?

First observation:

$IDF_PATH identified: /Users/USER/esp32/esp-idf

The statement, that $IDF_PATH had to be identified, could indicate that it's not defined. Could you please check what % echo $IDF_PATH tells you?

[rei-vilo]

I am using MacOS 13.0.1 Ventura.

IDF_PATHgives

% echo $IDF_PATH
/Users/USER/.local/share/esp32/esp-idf

[ralphwetzel]

Selected board is esp32/moddable-two.

Have you been able to run the minimal example successfully on a simulator?

[phoddie]

I ordered the Moddable Two board and rushed to test it when I received it. Very nice board with a crisp display.

Thank you! A lot of work went into finding that display.

Good idea to check that mcconfig is working correctly. That means all the pieces are present and working.

@ralphwetzel is the expert on the MCU Plug-in. Since this looks like it might be an IDF configuration issue, I want to ask one question. The IDF needs to be configured in each new terminal window. On macOS, that is usually done by this line in your .profile (and something similar on Linux).

source $IDF_PATH/export.sh

Is it possible that isn't happening in the terminal created by the MCU plugin?

[rei-vilo]

Thank you for the answers and hints.

New test: I opened a new Terminal window, ran and then called node-red. So Node-RED is running from the terminal.

Build goes further but fails due to missing "URL" module.

### ReferenceError: module "url" not found!

Log

Starting Build process...
Creating build environment for platform esp32/moddable_two.
Working directory: /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
$IDF_PATH is defined: /Users/USER/.local/share/esp32/esp-idf
> cd /Users/USER/.node-red/mcu-plugin-cache/configtgrufvjolkj
> source $IDF_PATH/export.sh && mcconfig -d -x localhost:5004 -m -p esp32/moddable_two -t build
Detecting the Python interpreter
Checking "python" ...
Python 3.10.8
"python" has been detected
Adding ESP-IDF toUSER to PATH...
Using Python interpreter in /Users/USER/.espressif/python_env/idf4.4_py3.10_env/bin/python
Checking if Python packages are up to date...
/Users/USER/.local/share/esp32/esp-idf
/Users/USER/.local/share/esp32/esp-idf/toUSER/check_python_dependencies.py
Python requirements from /Users/USER/.local/share/esp32/esp-idf/requirements.txt are satisfied.
Added the following directories to PATH:
/Users/USER/.local/share/esp32/esp-idf/components/esptool_py/esptool
  /Users/USER/.local/share/esp32/esp-idf/components/espcoredump
  /Users/USER/.local/share/esp32/esp-idf/components/partition_table
  /Users/USER/.local/share/esp32/esp-idf/components/app_update
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:
idf.py build
WARNING: Failed to load shell autocompletion for bash version: 3.2.57(1)-release!
# nodered2mcu flows
# xsc main.xsb
# mcrez resources
# xsc flows.xsb
# xsl modules
Total resource size: 0 bytes
# cc mc.resources.c (slots in flash)
ReferenceError: module "url" not found
### ReferenceError: module "url" not found!
make: *** [/Users/USER/Projects/moddable/build/tmp/esp32/moddable_two/debug/configtgrufvjolkj/mc.xs.c] Error 22

So I performed a full refresh of the Moddable SDK and the Node-RED MCU, to no avail.

Where should I look at?

[rei-vilo]

Have you been able to run the minimal example successfully on a simulator?

Selecting simulator with Simulator / Moddable Two throws the same error.

ReferenceError: (host): module "url" not found

Log

Starting Build process...
Creating build environment for platform sim/moddable_two.
Working directory: /Users/OlS/.node-red/mcu-plugin-cache/configtgrufvjolkj
> cd /Users/OlS/.node-red/mcu-plugin-cache/configtgrufvjolkj
> mcconfig -d -x localhost:5004 -m -p sim/moddable_two -t build
# nodered2mcu flows
# xsc main.xsb
# mcrez resources
Total resource size: 0 bytes
# xsc flows.xsb
# xsl modules
# cc mc.resources.c
ReferenceError: (host): module "url" not found
### ReferenceError: (host): module "url" not found!
# exception: (host): module "url" not found!
make: *** [/Users/OlS/Projects/moddable/build/tmp/mac/moddable_two/debug/configtgrufvjolkj/mc.xs.c] Error 22

[ralphwetzel]

Python 3 has been detected but line 99 is hard-coded and calls python instead.

This should have been fixed some time ago in the Espressif toolchain. I propose to update that as well to the latest version...

[phoddie]

Now, building on Node-RED MCU no longer complains about missing URL but

What changed?

[rei-vilo]

I did a clean install of everything: Moddable and ESP32 SDKs.

Compared to previous iterations, the difference is the build folder of Moddable has been deleted.

But Node-RED MCU still needs some attention, even with the official Moddable-Two board.

[phoddie]

Compared to previous iterations, the difference is the build folder of Moddable has been deleted.

I see. That's good to know. The update instructions do note that you need to remove the bin and tmp folders and rebuild tools (step 3 on Linux updates).

But Node-RED MCU still needs some attention, even with the official Moddable-Two board.

Are you referring to the install issue about about flashing from the MCU Plug-in in the Node-RED Editor?

[rei-vilo]

Compared to previous iterations, the difference is the build folder of Moddable has been deleted.

I see. That's good to know. The update instructions do note that you need to remove the bin and tmp folders and rebuild tools (step 3 on Linux updates).

So those specific instructions are valid for macOS as well :)

But Node-RED MCU still needs some attention, even with the official Moddable-Two board.

Are you referring to the install issue about about flashing from the MCU Plug-in in the Node-RED Editor?

I am referring to the flashing step from Node-RED MCU, which works fine against the Raspberry Pi Pico but not against the ESP32.

Actually, I was expecting that using the official hardware would bring some ease of use to explore Moddable + Node-RED MCU.

[ralphwetzel]

Some people at Moddable have been running internally with ESP-IDF 4.4.3 for over two weeks now without a problem. We expect to make that the recommended ESP-IDF release shortly.

After several attempts I can confirm meanwhile that v4.4.3 works w/ the Moddable SDK. That's good news! 👍

[rei-vilo]

Building succeeds but ends with

Project build complete. To flash, run this command:
/Users/USER/.espressif/python_env/idf4.4_py3.10_env/bin/python ../../../../../../../../../.local/share/esp32/esp-idf/components/esptool_py/esptool/esptool.py -p (PORT) -b 460800 --before default_reset --after hard_reset --chip esp32  write_flash --flash_mode dio --flash_size detect --flash_freq 80m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/xs_esp32.bin
or run 'idf.py -p (PORT) flash'
#
# Built files at /Users/USER/Projects/moddable/build/bin/esp32/moddable_two/debug/configtgrufvjolkj
#

Why the port mentioned on the configuration pane isn't taken into account for (PORT)? It works fine against the Raspberry Pi Pico.

The suggested command is not valid and should be

/Users/OlS/.local/share/esp32/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/cu.usbserial-0001 --baud 921600 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 4MB 0x1000 /Users/OlS/Projects/moddable/build/bin/esp32/moddable_two/debug/configtgrufvjolkj/bootloader.bin 0x10000 /Users/OlS/Projects/moddable/build/bin/esp32/moddable_two/debug/configtgrufvjolkj/xs_esp32.bin 0x8000 /Users/OlS/Projects/moddable/build/bin/esp32/moddable_two/debug/configtgrufvjolkj/partition-table.bin

The mentioned folder contains

% ls /Users/USER/Projects/moddable/build/bin/esp32/moddable_two/debug/configtgrufvjolkj 
bootloader.bin		xs_esp32.a		xs_esp32.map
partition-table.bin	xs_esp32.bin		xs_esp32.sym

Still with the same example, sending a timestamp does print anything on the debug node.

To be continued...

[rei-vilo]

There is still a major confusing issue on macOS.

If Node-RED is started at boot, build against the ESP32 fails.

Diagnostic information:
IDF_PYTHON_ENV_PATH: (not set)
Python interpreter used: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Warning: python interpreter not running from IDF_PYTHON_ENV_PATH

If Node-RED is started from a Terminal window, build succeeds.

% echo $IDF_PYTHON_ENV_PATH
/Users/USER/.espressif/python_env/idf4.4_py3.10_env

• Could Node-RED MCU populate IDF_PYTHON_ENV_PATH when Node-RED is started at boot?

[ralphwetzel]

I'll check...

[ralphwetzel]

What did you setup to start Node-RED at boot?

[rei-vilo]

The file /Users/USER/Library/LaunchAgents/org.node-red.agent.plist launches Node-RED at boot.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>org.node-red.agent</string>
	<key>LimitLoadToSessionType</key>
	<string>Aqua</string>
	<key>ProgramArguments</key>
	<array>
	  <string>/usr/local/bin/node</string>
	  <string>/usr/local/lib/node_modules/node-red/red.js</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>StartInterval</key>
	<integer>3623</integer>
	<key>StandardErrorPath</key>
  <string>/dev/null</string>
	<key>StandardOutPath</key>
  <string>/dev/null</string>
</dict>
</plist>

[rei-vilo]

Summary

1. Starting Node-RED at boot does not works now with release 3.5.0.
2. Node-RED needs to be launched from a Terminal window in order to have IDF_PYTHON_ENV_PATH defined.
3. The temporary folder /Users/USER/Projects/moddable/build/tmp/esp32/moddable_two/debug/ needs to be deleted to get a clean environment.

UPDATED 13 Dec 2023 with release 3.5.0

[rei-vilo]

I added to ~/.node-red/settings.js

process.env.IDF_PYTHON_ENV_PATH = "/Users/USER/.espressif/python_env/idf4.4_py3.10_env"

to no avail.

[phoddie]

To check that I understand you correctly:

• If you launch Node-RED at start-up, ESP32 builds do work with the plugin
• But... there is a warning about IDF_PYTHON_ENV_PATH not being set... and so a warning that an unexpected version of Python is used

[ralphwetzel]

IDF_PYTHON_ENV_PATH should be defined correctly by idf_tools.py, which is always invoked at the beginning of the build process.
I've nevertheless implemented a small change to take over a predefined IDF_PYTHON_ENV_PATH value into the build environment; don't know yet if this is a good decision. We'll see...

[ralphwetzel]

I've nevertheless implemented a small change to take over a predefined IDF_PYTHON_ENV_PATH value into the build environment; don't know yet if this is a good decision. We'll see...

As a matter of documentation: I've today removed this "fix" (taking over IDF_PYTHON_ENV_PATH) again, as it broke the build run w/ ESP-IDF 5.1.1.

[phoddie]

Thanks for the note. The ESP-IDF is, unfortunately, a moving target. Updating to IDF v5 was a substantial effort for us, even though the result to most developers using the Moddable SDK looks nearly transparent.