Merge pull request #572 from TEN-framework/feat/esp32

feat: add esp32 client code

esp32-client/CMakeLists.txt

@@ -0,0 +1,12 @@
+# The following lines of boilerplate have to be in your project's CMakeLists
+# in this exact order for cmake to work correctly
+cmake_minimum_required(VERSION 3.5)
+
+# (Not part of the boilerplate)
+# This example uses an extra component for common functions such as Wi-Fi and Ethernet connection.
+# set(EXTRA_COMPONENT_DIRS $ENV{IDF_PATH}/examples/common_components/protocol_examples_common)
+
+include($ENV{ADF_PATH}/CMakeLists.txt)
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+
+project(videodoorbell_esp32)

esp32-client/README.cn.md

@@ -0,0 +1,185 @@
+# 声网 ESP32 大模型智能对话
+
+*简体中文| [English](README.md)*
+
+## 例程简介
+
+本例程演示了如何通过乐鑫 ESP32-S3 Korvo V3 开发板，模拟一个典型的大模型智能对话场景，可以演示进入大模型进行实时智能对话。
+
+### 文件结构
+```
+├── CMakeLists.txt
+├── components                                  Agora iot sdk component
+│   ├── agora_iot_sdk
+│   │   ├── CMakeLists.txt
+│   │   ├── include                             Agora iot sdk header files
+│   │   │   ├── agora_rtc_api.h
+│   │   └── libs                                Agora iot sdk libraries                      
+│   │       ├── libagora-cjson.a
+│   │       ├── libahpl.a
+│   │       ├── librtsa.a
+├── main                                        LLM Demo code
+│   ├── ai_agent.h
+│   ├── app_config.h
+│   ├── common.h
+│   ├── audio_proc.h
+│   ├── rtc_proc.h
+│   ├── CMakeLists.txt
+│   ├── Kconfig.projbuild
+|   ├── ai_agent.c
+|   ├── audio_proc.c
+|   ├── rtc_proc.c
+│   └── llm_main.c
+├── partitions.csv                              partition table
+├── README.en.md
+├── README.md
+├── sdkconfig.defaults
+└── sdkconfig.defaults.esp32s3
+```
+
+## 环境配置
+
+### 硬件要求
+
+本例程目前仅支持`ESP32-S3-Korvo-2 V3`开发板。
+
+## 编译和下载
+
+### Agora IOT SDK
+
+编译运行本示例需要 Agora IoT SDK。Agora IoT SDK 可以在 [这里](https://rte-store.s3.amazonaws.com/agora_iot_sdk.tar) 下载。
+将 `agora_iot_sdk.tar` 放到 `esp32-client/components` 目录下，并运行如下命令：
+
+```bash
+cd esp32-client/components
+tar -xvf agora_iot_sdk.tar
+```
+
+### Linux 操作系统
+
+#### 默认 IDF 分支
+
+本例程支持 IDF tag v[5.2.3] 及以后的，例程默认使用 IDF tag v[5.2.3] (commit id: c9763f62dd00c887a1a8fafe388db868a7e44069)。
+
+选择 IDF 分支的方法，如下所示：
+```bash
+cd $IDF_PATH
+git checkout v5.2.3
+git pull
+git submodule update --init --recursive
+```
+
+本例程支持 ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052)
+
+#### 打上 IDF 补丁
+
+本例程还需给 IDF 合入1个 patch， 合入命令如下：
+```bash
+export ADF_PATH=~/esp/esp-adf
+cd $IDF_PATH
+git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch
+```
+
+#### 编译固件
+
+将本例程(esp32-client)目录拷贝至 ~/esp 目录下。请运行如下命令：
+```bash
+$ . $HOME/esp/esp-idf/export.sh
+$ cd ~/esp/esp32-client
+$ idf.py set-target esp32s3
+$ idf.py menuconfig	--> Agora Demo for ESP32 --> (配置 WIFI SSID 和 Password)
+$ idf.py build
+```
+配置freertos的前向兼容能力
+在menuconfig中Component config --> FreeRTOS --> Kernel设置configENABLE_BACKWARD_COMPATIBILITY
+
+### Windows 操作系统
+
+#### 默认 IDF 分支
+
+下载IDF，选择v5.2.3 offline版本下载，例程默认使用 IDF tag v[5.2.3]
+https://docs.espressif.com/projects/esp-idf/zh_CN/v5.2.3/esp32/get-started/windows-setup.html
+
+下载ADF，ADF目录Espressif/frameworks，为支持ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052)
+https://docs.espressif.com/projects/esp-adf/zh_CN/latest/get-started/index.html#step-2-get-esp-adf
+
+
+#### 打上 IDF 补丁
+
+方法一.系统设置中将ADF_PATH添加到环境变量
+E:\esp32s3\Espressif\frameworks\esp-adf
+方法二.命令行中将ADF_PATH添加到环境变量
+```bash
+$ setx ADF_PATH Espressif/frameworks/esp-adf
+```
+
+注意：ADF_PATH环境变量设置后，重启ESP-IDF 5.2 PowerShell生效
+
+本例程还需给 IDF 合入1个 patch， 合入命令如下：
+```bash
+cd $IDF_PATH
+git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch
+```
+
+#### 编译固件
+
+将本例程(esp32-client)目录拷贝至 Espressif/frameworks 目录下。请运行如下命令：
+```bash
+$ cd ../esp32-client
+$ idf.py set-target esp32s3
+$ idf.py menuconfig	--> Agora Demo for ESP32 --> (配置 WIFI SSID 和 Password)
+$ idf.py build
+```
+配置freertos的前向兼容能力
+在menuconfig中Component config --> FreeRTOS --> Kernel设置configENABLE_BACKWARD_COMPATIBILITY
+
+
+### 下载固件
+
+请运行如下命令：
+```bash
+$ idf.py -p /dev/ttyUSB0 flash monitor
+```
+注意：Linux系统中可能会遇到 /dev/ttyUSB0 权限问题，请执行 sudo usermod -aG dialout $USER
+
+下载成功后，本例程会自动运行，待设备端加入RTC频道完成后，可以看到串口打印："Agora: Press [SET] key to Join the Ai Agent ..."。
+
+
+## 如何使用例程
+
+### 五分钟快速体验
+
+注意：
+
+1. 请确认开发板上已至少接入一个扬声器。
+
+### 配置你自己的 AI Agent
+
+1. 请在 `app_config.h` 文件中配置你自己的 AI Agent。
+2. 修改 `TENAI_AGENT_URL` 为你自己的 TEN-Agent 服务器 URL (一般为你通过 `task run` 启动的8080服务)。
+3. 修改 `AI_AGENT_CHANNEL_NAME` 为你自己的 AI Agent Channel 名称。
+4. 如果你之前就配置过 `openai_v2v` graph，那在配置好服务器URL后，你就可以开始体验了。
+5. 如果你没有配置过 `openai_v2v` graph 或者想使用其他 graph，你需要在 `ai_agent.c` 的 `_build_start_json` 函数中修改启动的相关参数。
+6. 重新编译后烧录到芯片上。
+
+#### Demo：大模型AiAgent实时语音对话
+
+1. 按键 `SET` 表示启动大模型
+2. 按键 `MUTE` 表示停止大模型
+3. 按键 `VOL+` 表示增大音量，一次增大10，最大到100
+4. 按键 `VOL-` 表示减小音量，一次减小10，最小到0
+5. 设备开机后，设备自动连接到服务器生成APPID对应的Channel频道内。按下 `SET` 键，开始启动大模型，开始进行实时语音对话；按下 `MUTE` 键，停止大模型。
+
+
+## 关于声网
+
+声网音视频物联网平台方案，依托声网自建的底层实时传输网络 Agora SD-RTN™ (Software Defined Real-time Network)，为所有支持网络功能的 Linux/RTOS 设备提供音视频码流在互联网实时传输的能力。该方案充分利用了声网全球全网节点和智能动态路由算法，与此同时支持了前向纠错、智能重传、带宽预测、码流平滑等多种组合抗弱网的策略，可以在设备所处的各种不确定网络环境下，仍然交付高连通、高实时和高稳定的最佳音视频网络体验。此外，该方案具有极小的包体积和内存占用，适合运行在任何资源受限的 IoT 设备上，包括乐鑫 ESP32 全系列产品。
+
+## 技术支持
+
+请按照下面的链接获取技术支持：
+
+- 如果发现了示例代码的 bug 和有其他疑问，可以直接联系社区负责人
+
+我们会尽快回复。
+

esp32-client/README.md

@@ -0,0 +1,195 @@
+# Agora ESP32 Large Model Intelligent Dialogue
+
+*[简体中文](./README.cn.md) | English*
+
+## Overview
+
+This is an RTC Client SDK & Demo running on Espressif ESP32-S3 Korvo V3 development board. This example demonstrates how to make TEN-Agent work with it.
+
+### File Structure
+```
+├── CMakeLists.txt
+├── components                                  Agora IoT SDK component
+│   ├── agora_iot_sdk
+│   │   ├── CMakeLists.txt
+│   │   ├── include                             Agora IoT SDK header files
+│   │   │   ├── agora_rtc_api.h
+│   │   └── libs                                Agora IoT SDK libraries                      
+│   │       ├── libagora-cjson.a
+│   │       ├── libahpl.a
+│   │       ├── librtsa.a
+├── main                                        LLM Demo code
+│   ├── ai_agent.h
+│   ├── app_config.h
+│   ├── common.h
+│   ├── audio_proc.h
+│   ├── rtc_proc.h
+│   ├── CMakeLists.txt
+│   ├── Kconfig.projbuild
+|   ├── ai_agent.c
+|   ├── audio_proc.c
+|   ├── rtc_proc.c
+│   └── llm_main.c
+├── partitions.csv                              Partition table
+├── README.en.md
+├── README.md
+├── sdkconfig.defaults
+└── sdkconfig.defaults.esp32s3
+```
+
+## Environment Setup
+
+### Hardware Requirements
+
+This example currently supports the `ESP32-S3-Korvo-2 V3` development board only.
+
+## Compilation and Download
+
+### Agora IOT SDK
+
+To compile and run this example, you need the Agora IoT SDK.
+The SDK can be downloaded at [here](https://rte-store.s3.amazonaws.com/agora_iot_sdk.tar)
+Put `agora_iot_sdk.tar` to `esp32-client/components` directory and run the following command:
+
+```bash
+cd esp32-client/components
+tar -xvf agora_iot_sdk.tar
+```
+
+### Linux Operating System
+
+#### Default IDF Branch
+
+This example supports IDF tag v[5.2.3] and later versions, with the default set to IDF tag v[5.2.3] (commit id: c9763f62dd00c887a1a8fafe388db868a7e44069).
+
+To select the correct IDF branch, run the following commands:
+```bash
+cd $IDF_PATH
+git checkout v5.2.3
+git pull
+git submodule update --init --recursive
+```
+
+This example supports ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052).
+
+#### Applying the IDF Patch
+
+A patch must be applied to IDF. Use the following command:
+```bash
+export ADF_PATH=~/esp/esp-adf
+cd $IDF_PATH
+git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch
+```
+
+#### Compiling the Firmware
+
+Copy the example project directory (esp32-client) to the `~/esp` directory and run the following commands:
+```bash
+$ . $HOME/esp/esp-idf/export.sh
+$ cd ~/esp/esp32-client
+$ idf.py set-target esp32s3
+$ idf.py menuconfig	--> Agora Demo for ESP32 --> (Configure WIFI SSID and Password)
+$ idf.py build
+```
+
+To configure FreeRTOS backward compatibility:
+In `menuconfig`, navigate to `Component config` --> `FreeRTOS` --> `Kernel` and enable `configENABLE_BACKWARD_COMPATIBILITY`.
+
+### Windows Operating System
+
+#### Default IDF Branch
+
+Download IDF, selecting version v5.2.3 (offline version) from the following link:  
+[ESP-IDF Windows Setup](https://docs.espressif.com/projects/esp-idf/zh_CN/v5.2.3/esp32/get-started/windows-setup.html)
+
+Download ADF to the `Espressif/frameworks` directory to support ADF v2.7 tag (commit id: 9cf556de500019bb79f3bb84c821fda37668c052):  
+[ESP-ADF Setup](https://docs.espressif.com/projects/esp-adf/zh_CN/latest/get-started/index.html#step-2-get-esp-adf)
+
+#### Applying the IDF Patch
+
+Method 1: Add `ADF_PATH` to the environment variables in system settings:
+```
+E:\esp32s3\Espressif\frameworks\esp-adf
+```
+
+Method 2: Add `ADF_PATH` via the command line:
+```bash
+$ setx ADF_PATH Espressif/frameworks/esp-adf
+```
+
+**Note:** After setting the `ADF_PATH` environment variable, restart ESP-IDF 5.2 PowerShell for changes to take effect.
+
+Apply the required patch to IDF using:
+```bash
+cd $IDF_PATH
+git apply $ADF_PATH/idf_patches/idf_v5.2_freertos.patch
+```
+
+#### Compiling the Firmware
+
+Copy the example project directory (esp32-client) to the `Espressif/frameworks` directory and run the following commands:
+```bash
+$ cd ../esp32-client
+$ idf.py set-target esp32s3
+$ idf.py menuconfig	--> Agora Demo for ESP32 --> (Configure WIFI SSID and Password)
+$ idf.py build
+```
+
+Configure FreeRTOS backward compatibility:  
+In `menuconfig`, navigate to `Component config` --> `FreeRTOS` --> `Kernel` and enable `configENABLE_BACKWARD_COMPATIBILITY`.
+
+### Flashing the Firmware
+
+Run the following command:
+```bash
+$ idf.py -p /dev/ttyUSB0 flash monitor
+```
+**Note:** On Linux, you might encounter permission issues with `/dev/ttyUSB0`. Run the following command to fix it:
+```bash
+sudo usermod -aG dialout $USER
+```
+
+Once flashing is complete, the example will run automatically. After the device joins the RTC channel, the serial output will display:  
+**"Agora: Press [SET] key to Join the Ai Agent ..."**
+
+## How to Use the Example
+
+### Quick Start in 5 Minutes
+
+**Note:**  
+Ensure at least one speaker is connected to the development board.
+
+
+### Configure your own agent
+
+Let's walk you through creating your own user and device accounts.
+
+#### Update app_config.h
+
+1. Open the `app_config.h` file in the project.
+2. Modify `TENAI_AGENT_URL` to the URL of your TEN-Agent server (usually the 8080 port service).
+3. Modify `AI_AGENT_CHANNEL_NAME` to a unique channel name.
+4. If you have your Ten-Agent configured for `openai_v2v` graph working, you should be good to go.
+5. If you want to use other graphs, you can modify how request is constructed in `ai_agent.c` file the `_build_start_json` method.
+6. Recompile and flash the firmware.
+
+#### Demo: Real-time Voice Dialogue with Large Model AiAgent
+
+1. Press the `SET` button to start the large model.
+2. Press the `MUTE` button to stop the large model.
+3. Press the `VOL+` button to increase volume (increments of 10, up to a maximum of 100).
+4. Press the `VOL-` button to decrease volume (decrements of 10, down to a minimum of 0).
+5. After the device boots up, it will automatically connect to the RTC channel associated with the generated APPID. Press the `SET` button to initiate real-time voice dialogue; press the `MUTE` button to stop it.
+
+
+## About Agora
+
+Agora’s audio and video IoT platform leverages its proprietary real-time transmission network, **Agora SD-RTN™ (Software Defined Real-time Network)**, to provide real-time audio and video streaming capabilities for Linux/RTOS devices with networking capabilities. The solution ensures high connectivity, real-time performance, and stability even under uncertain network conditions using advanced techniques such as forward error correction, intelligent retransmission, bandwidth prediction, and stream smoothing. Additionally, it offers a minimal memory footprint, making it ideal for resource-constrained IoT devices, including the entire Espressif ESP32 series.
+
+## Technical Support
+
+For technical support, follow the links below:
+
+- Report bugs and inquiries directly to the community representatives.
+
+We will respond as soon as possible.

esp32-client/main/CMakeLists.txt

@@ -0,0 +1,3 @@
+idf_component_register(SRCS llm_main.c ai_agent.c rtc_proc.c audio_proc.c
+					REQUIRES audio_hal audio_pipeline audio_stream audio_board esp_peripherals esp-adf-libs 
+							 input_key_service esp_wifi nvs_flash agora_iot_sdk mbedtls)