Batching / Concurrent audio streams for different clients

[JamesDeAntonis]

Is it possible to handle multiple audio streams at once? I think this comment is about how to finish one stream and then start another. I'm curious about whether it's possible to start processing a new stream while still processing existing ones. I think there's two levels two this:

Level 1. handle concurrent audio streams in a non-batching fashion, like be able to handle two different states and perhaps alternate inference calls to each in a round robin fashion
Level 2. handle concurrent audio streams in the same model call, optimized with batching, and dynamically push and pop different streams as they start and finish

It seems like the asr model itself isn't stateful (looks like KV caching happens outside the model). So, maybe a modest refactor could allow for sharing the same model instance across multiple active online asr instances. Unless I'm missing something, this would accomplish level 1 which would be good

Level 2 seems like a more significant lift

Have you guys thought about implementing either of these things?

Thanks!

[Gldkslfmsd]

hi,
the code doesn't able multiclient. It's a demo code, proof of concept.
Yes, refactoring would be needed.
Also, I'm afraid there is no way of multiclient implementation that would be as fast as single client. Imagine that one client has 99% size of voiced chunk, so it waits, the other client has full chunk so it blocks encoder-decoder, and then the first client will have the chunked much longer.

Also, implementing ASR and translation at once would be nice.

[Gldkslfmsd]

so, to conclude:

• refactoring for stateless PaddedAlignAttWhisper is appreciated. The states should be tracked within SimulWhisperASR (which in fact is not only for ASR but also for translation).
• multi-targeting ASR + translation via batching would be appreciated.
• level 2 is a bigger project. I prefer it in another repo that uses SimulStreaming as a module. Then I'm happy to link it from README, the same way as with WhisperLiveKit

[JamesDeAntonis]

Ok looks good. I'll probably get to level 1 at some point, and maybe stomach my way through level 2 if I'm not satisfied with the performance of level 1. Idk if level 2 is neatly possible with SimulStreaming as a module, as I think it requires pretty major surgery. But I haven't fully thought it through yet so not sure the extent of it