Audio to web

[luisesn]

This is not intended to merge to main.

Its a work in progress so the app can receive and play audio sent from the radio via websockets (raw pcm…).

For now it works with an standalone example (soundWs/index.html) from your local computer, and I have started to implement it in the vue app but for now it only works locally, not from the device (but as I dont have any idea of how to program in Vue... it'll have to do for today)

(and, of course, it needs to have a loooooot of optimizations, as I said: very WIP but W)

[kamilsss655]

That's an interesting concept. Will take a look!

[kamilsss655]

Can you provide some higher level overview on how this works?
Is it just sending audio samples over websocket or is there some audio encoding happening in between?
I see some sum of squares stuff etc

[luisesn]

At the esp level I've just copied your recording function (and structure/task/way of activating) so it sends the data over ws.
(I've removed the filtering, can be readded, just wanted to see it working before I complicated things...)

I have created a version of your ws send to send binary data (but there is some error somewhere because sometimes the packets are corrupt and ws connection closes [on the client side], didn't have time to check it better but I suspect it has to do with the way the buffers are treated.
)
I have modified the audio data block size, audio rate and converted data from 16 bit to 8 bit because I dont need the quality for now and preferred the speed and less data (256kbps to 64 kbps reduction aprox.) (maybe if we find some way of compressing the data).

At the web side its only a websocket that receives the data and buffers it so it can be played (with some "apaños" because it wasn't detecting the binary data right).

The sum of squares its only to show the volume in a progress bar, the only needed operation its converting from int values to -1.0/1.0 ones so audiocontext can understand (the /128 in the current version).

[kamilsss655]

Thanks for the explanation. Makes a lot of sense.

I haven't tried it yet, but will do soon!

[kamilsss655]

I tested it and it looks good so far. Hope you can make it work from the device.

The audio was glitching for me, but I guess that is part of the WIP.

Really like the idea and hope you can make it work. It would make a fine addition to the feature set of ESPRI.

Good luck!

[luisesn]

The glitching is caused by the websocket closing, I'm hoping to correct it...
I'll keep working on it

[kamilsss655]

This might be irrelevant, but I noticed a lot of log messages being output over serial during audio listening. Perhaps disabling these might help?

[luisesn]

it should not affect as the sound is processed in background in another thread

the sttutering is mostly caused by the websocket closing (maybe 1% of the ocassions is network speed/lag/cuts, TCP its not the best transport for this but... mostly the closing)

[kamilsss655]

Hey I know this is old, but I just watched this and at the end there is a audio to websocket shown just like with your application. The author mentioned that you need a browser plugin in order to get more speed via HTTP over local connection. Maybe this helps?

Here is the video: https://www.youtube.com/watch?v=0jR-QNTfydA