Audio support added, optimization issues

[fommike]

Hello,

I have implemented the audio support for the SensorTag CC2650. The core functionality is in the file cc2650.js. I also added an example for using audio (file two_tags.js). I tested the whole setting with Raspberry Pis (3B and Zero W) and several SensorTags.

There is a couple of important things to bear in mind. First, the BLE connection parameters. For reliable audio transmission they should be set to appropriate values, both on the side of the nodejs (i.e. Raspberry Pi) and on the side of the SensorTag itself (i.e. in the audio example see the file sensortag.c, parameters “DEFAUTL_DESIRED_MIN_CONN_INTERVAL”, “DEFAUTL_DESIRED_MAX_CONN_INTERVAL” and “DEFAUTL_DESIRED_SLAVE_LATENCY”). For my experiments I used the minimal possible value for connection intervals, i.e. 6 and 0 for latency. For details on the BLE connection see the BLE developer’s guide from TI.

To set those parameters on the nodejs side there are two steps: 1) set up the Linux kernel parameters for BLE connection (see scripts ble_setup.sh) 2) in your sensortag js directory adjust the parameters “min_interval”, “max_interval” and “latency” in hci.js (function Hci.prototype.createLeConn) which should be in the local branch of node_modules (e.g. for me “/home/pi/sensortag-nodejs/node_modules/noble/lib/hci-socket/hci.js”). Actually, taking both steps is kind of overkill, because they essentially do the same, but while googling around I found that the kernel BLE parameters may not be set up properly from the application so I am setting both just for reliability. A side note here: having an API which would allow setting BLE connection parameters would be very handy in sensortag nodejs. To set those parameters on the SensorTag side adjust the constants mentioned above in the sensortag.c file.

Second, after running the ble_setup.sh and manually setting the BLE parameters in hci.js the audio transmission can be done. For that I made the example two_tags.js which actually allows streaming audio from two tags simultaneously. I tested it on Raspberry Pi 3B and managed to obtain two audio streams from two tags simultaneously without any errors. Note: setting up connection parameters to appropriate values is essential to get reliable audio transmission.

Third, to decode the audio I modified the python script provided by TI.

Fourth, current problem: for Raspberry Pi 3B I managed to obtain streaming from only two tags simultaneously in a reliable way, when I tested it with 3 tags I started to lose frames. Moreover, I tested my setting on Raspberry Pi Zero W and noticed that I am losing frames even for a single tag. I monitored the system resources and it turns out while transmitting audio the CPU load is 100% which is probably the reason why the frames get lost.

I am not an expert in Javascript and it is very possible that the code I wrote is not optimal (maybe saving to a file like this -> cc2650.js is overkill or something smarter can be done in two_tags.js). Anyhow, I think it would be great if people with more expertise can have a look at the code and try optimizing it e.g. finding a better way to save/store audio data.

I am looking forward to your comments and suggestions.

Sources:

1. For “ble_setup.sh” courtesy goes to here: Is there a way to configure the connection interval, slave latency and supervisor timeout? (hcitool lecup) IanHarvey/bluepy#117
2. “Audio.py” is a modified version from here: https://github.com/ti-simplelink/ble_examples/blob/master/tools/scripts/audio/audio_frame_serial_print.py

[jarvis-hal]

Great job! @fommike, do you need to change the firmware on cc2650stk? Or the stock firmware on TI BLE Stack SDK 2.2.1 will work.

[fommike]

Thanks @jarvis-hal, no you do need to change the firmware on the tag (though I did some improvements). It should work with the off-the-shelf audio example, provided by TI (C:\ti\simplelink\ble_sdk_2_02_01_18\examples\cc2650stk\sensortag_audio). Bear in mind that for this example you are going to have to push and hold the left button to enable the streaming. And of course, the BLE connection parameters are vital for reliable audio transmission.

[fommike]

Feel free to try the code out and let me know if it works for you.

[jarvis-hal]

Nice. I have a quick question. Does it mean that once cc2650stk is programmed for audio, it can not be used to stream other sensor data, like temperature?Also, have you found a way in the firmware to change from press button to more like something we can trigger through an API?

[fommike]

Does it mean that once cc2650stk is programmed for audio, it can not be used to stream other sensor data, like temperature?
-> That is tricky, but in principle it can be done. This is the part I am currently investigating. That is the short answer, because the long answer is really long;)

-> Have you found a way in the firmware to change from press button to more like something we can trigger through an API?

There is no such thing there of course from TI, but I implemented it. Currently, my setting works as follows: once the tag receives two audio notifications from the Pi (i.e. "notifyAudioConfig" and "notifyAudioStream" in cc2650.js, example in two_tags.js) it starts recording the audio during the time interval that can be set on the tag itself (for me 30 sec). Once 30 second elapses the tag stops streaming and sends 0x00 command to the Pi indicating that audio streaming is over (see "saveAudioData" in cc2650.js). I am using this event to send the de-notifications to the tag so that it could gracefully finalize all the audio stuff and be ready for new transmission upon the request.

[jarvis-hal]

I am trying to implement something continuous streaming audio from sensortag. I will try it out when I have time, and report back how it works.

[sandeepmistry]

Hi @fommike,

That's for taking the time to submit this PR. It's going to need a bit more work before it can be merged.

Overal, a few questions:

1. How can I get my SensorTag flashed with the right firmware to test

2. What's the use case for this feature?

3. Why is data saved to a hard coded filename?

4. Why is there a Python script to convert the data? Should the library do this instead?

5. What's the end to end developer flow to use this feature?

Apologies for all the questions, I'm just trying piece everything together ...

[sandeepmistry]

Why is the tagId object needed in the library, isn't every SensorTag object self contained?

It's also a bit weird that the file name is hard coded. What's the current experience/steps for using this?

[sandeepmistry]

What is setAudioFlag suppose to do?

[sandeepmistry]

Couldn't the logic in this file be converted to JavaScript?

[fommike]

Hi, @sandeepmistry, I added some changes to the code so tagId and setAudioFlag were indeed unnecessary. The answers to the rest of your questions are below:

1. How can I get my SensorTag flashed with the right firmware to test?

-> for audio support you can flash the SensorTag using the TI's audio example, on Windows it should be at C:\ti\simplelink\ble_sdk_2_02_01_18\examples\cc2650stk\sensortag_audio.

2. What's the use case for this feature?

-> one can stream audio in real-time from the SensorTag, the quality of audio is ok and can be used for, e.g. voice recognition solutions (see here: http://processors.wiki.ti.com/index.php/BLESDK-2.2.x-CC2650RC_Developers_Guide#TI_Audio_Profile).

3. Why is data saved to a hard coded filename?

-> The file name should not be hardcoded of course, it is kind of a workaround. Unfortunately, you have to write the audio data to a file inside the function convertAudioData. Otherwise, if you just convert it to a buffer and return as a callback parameter and do the writing inside your "main working code" (e.g. two_tags.js) then it will result in the frame loss (was the case for me). As I mentioned initially being able to process audio fast is a prerequisite to the correct audio reception.

I am not an expert in javascript, perhaps you can review my code to make it more efficient.

4. Why is there a Python script to convert the data? Should the library do this instead?

-> The Python script just shows how to do the stuff. No doubt, this functionality can be implemented in nodejs as well, I just did not do it. However, if this goes to the library it should be a standalone module and it should not be called real-time, e.g. when audio is being recorded, otherwise it would result in frame loss as well (see my point above).

Of course, I had the setting SensorTags/Raspberry Pis, if you have a powerful laptop as a receiving device then it should not be a problem to do fancy processing real-time.

5. What's the end to end developer flow to use this feature?

-> Did not quite get what you meant? Can you elaborate a bit on that?

[sandeepmistry]

I think we should hold off merging this until TI's SensorTag firmware supports it, as it seems a bit complicated to setup at the moment ...

but we can continue the discussions here...

[OpenEarthLab]

Hi, @sandeepmistry @fommike . Thanks for all your work on SensorTag, I am doing experiment and collecting data with SensorTag now. With your help, I could connect my SensorTag to my MAC and save their sensed data of multi sensors to my computer for further analyzing. Thanks a lot.
Currently, I am also looking forward to get the audio data together with other sensed data, not only audio data. Keep looking at the progress of this work.