Voice to Text & Text to Voice Pipelines

[samsullivandelgobbo]

The new voice to text and text to voice functionality on the ChatGPT ios app should be easily replicable with something like whisper for voice to text, and offerings from eleven labs and other OS alternatives could easily be implemented for text to voice.

The main question is if a pipeline for text to voice and voice to text would be in the scope of this project?

[dosstx]

I am interested in this topic because I am currently using ElevenLab's text-to-speech API with the useChat() utility. However, I am not sure how the useChat() can make it easier to work with ElevenLabs's API. We just need to watch the messages and then send that to the speech API, right? Right now I have it working by using the onFinish callback from useChat, but my problem at the moment is doing this logic using edge function environments (I use Netlify) due to size limitations. Thus, I will try to see if I can stream the messages to the text to speech API and then use Elevenlab's stream API endpoint. Not sure if I need websocket connection on front-end. Currently researching how to do it in Nuxt (Vue Framework).

Here is an example from ElevenLabs on how to use text to speech websocket in Javascript for streaming audio from ChatGPT API: https://docs.elevenlabs.io/api-reference/text-to-speech-websockets#other-examples

[mcwwwwww]

I need to know how the useChat() utility could handle voice also.

[d-ivashchuk]

hey @dosstx! I am actually looking into a similar flow that you are doing right now, did you manage to solve it with websockets?

[dosstx]

What JS framework ?

I put it on back burner . I am just streaming short stories for now and the Elevenlabs Turbo API is fast enough for me at the moment (8sec latency) after story is generated. I'd love to get real time audio at some point.

[lgrammel]

RFC PR for TTS: #922

[haydenbleasel]

I made an AI toolkit inspired by this project for TTS and STT — https://www.orate.dev/