CRITICAL | WSServerHandshakeError: 429 message='Invalid response status'

[usmanbinimran-octopus]

Experiencing the following error in logs after deploying on Azure Cloud using "az up" command in GitHub Codespace.

`2024-11-20T20:08:25.88930 Connecting to the container 'main'...

2024-11-20T20:08:25.94451 Successfully Connected to container: 'main' [Revision: 'capps-backend-xxx--azd-1732091844-6c9995f49b-fnl55', Replica: 'capps-backend-xxx--azd-1732091844']
Closing OpenAI's realtime socket connection.
[2024-11-20 20:07:55 +0000] [7] [ERROR] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
resp = await handler(request)

       ^^^^^^^^^^^^^^^^^^^^^^

File "/app/rtmt.py", line 224, in _websocket_handler
await self._forward_messages(ws)
File "/app/rtmt.py", line 191, in _forward_messages
async with session.ws_connect("/openai/realtime", headers=headers, params=params) as target_ws:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/client.py", line 1194, in aenter
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/client.py", line 848, in _ws_connect
raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status', url=URL('https://cog-xxx.openai.azure.com/openai/realtime?api-version=2024-10-01-preview&deployment=gpt-4o-realtime-preview')
`

[pamelafox]

429 is a rate-limit error generally - can you make sure you've closed your tabs using the real-time API and don't have any colleagues using that deployment? With only 1K TPM, it's easy to exceed it.

[usmanbinimran-octopus]

Connecting...

2024-11-20T20:25:35.16089 Connecting to the container 'main'...

2024-11-20T20:25:35.23117 Successfully Connected to container: 'main' [Revision: 'capps-backend-xxxxxxxxxxx--azd-1732091844-6c9995f49b-fnl55', Replica: 'capps-backend-xxxxxxxxxxx--azd-1732091844']

{"title":"PodrPodOrContainerNotFound","status":404,"detail":"Pod or container main not found.","loggerContext":{"centurionRoleId":"lgn-hostrole-northeurope-001_1406","hostExternalIpv4Address":"172.24.0.171","legionStampName":"northeurope-001","nestedRoleId":"lgn-hostrole-northeurope-001_1406-nestedvm_2","nestedRoleIpv4Address":"169.254.0.5","podName":"capps-backend-xxxxxxxxxxxxx--azd-1732091844-6c9995f49b-fnl55","resourceNamespace":"mangowater-e1781041","vmPlatformFaultDomain":"0","vmZone":"2"},"genevaTraceId":"74470fd706e16e30af7d08336e5445fb","callerMemberName":"GetContainerLog","callerFilePath":"C:\__w\1\s\src\Legion\Common.Podr\Services\PodrK8sService.cs"}

[usmanbinimran-octopus]

@pamelafox, if you can provide assistance.

Really appreciate it.

[usmanbinimran-octopus]

Yes, No one is using the app.

No error on frontend console.

Mic button is accessible, means front end loading smoothly. Cannot hear any response from OpenAI Realtime API. Indeed 429 suggests rate-limit. But since, no one is accessing the app, I think there's more to look into.

[pamelafox]

And there are no other tabs with it open? Can you open up the deployment in Azure OpenAI studio and see if it works in the real-time playground? Sometimes there are intermittent issues with the servers being over capacity.
Another thing you can try is to make a deployment in the other supported region, in case that region is over capacity.

[usmanbinimran-octopus]

@pamelafox, I cannot click the "Start Listening" button: