Invalid response from WLED API in Home Assistant when simply turning on the LED strip

[FGOD1983]

What happened?

Many times when one of my automations in Home Assistant is just simply turning on the LED strip (60 LEDs, no segments, so not really a big thing), the Home Assistant automation fails to do so because it received a Invalid response from WLED API error response from the device.

1 minute later it would just turn on fine as it would normally do without issues.

This started to happen since version 0.15.0 and as some people reported on this Home Assistant issue (home-assistant/core#132284), the issues are gone when you downgrade back to 0.14.4.

To Reproduce Bug

It is hard to reproduce as it seems to be just randomly happening and a few seconds/minutes later there is no issue at all when turning on the device. It also seems to be correctly working when you manually turn on the device in Home Assistant.

It could be that in the automation I let it set a preset and then turn it on which could cause a API timeout if that happens to fast, but then again, it does work a minute later and there are no issues with this on 0.14.4

Expected Behavior

just normally turn on the LED strip without errors or timeouts

Install Method

Binary from WLED.me

What version of WLED?

0.15.0

Which microcontroller/board are you seeing the problem on?

ESP8266

Relevant log/trace output

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[julianfs]

It's very intermittent and I can't reproduce it on demand, but here's a log entry from when activating a scene in Home Assistant.

Logger: homeassistant.core
Source: core.py:2814
First occurred: 09:07:52 (1 occurrences)
Last logged: 09:07:52

Error executing service: <ServiceCall scene.turn_on (c:01JKT47KP8F6HGM9636HYRD2T1): entity_id=['scene.kitchen_blue']>
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/wled/helpers.py", line 26, in handler
    await func(self, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/wled/light.py", line 276, in async_turn_on
    await self.coordinator.wled.segment(**data)
  File "/usr/local/lib/python3.13/site-packages/wled/wled.py", line 450, in segment
    await self.request("/json/state", method="POST", data=state)
  File "/usr/local/lib/python3.13/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/wled/wled.py", line 201, in request
    raise WLEDError(
    ...<2 lines>...
    )
wled.exceptions.WLEDError: (503, {'error': 3})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/core.py", line 2814, in _run_service_call_catch_exceptions
    await coro_or_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 2837, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
        hass, entity, func, data, call.context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/scene/__init__.py", line 118, in _async_activate
    await self.async_activate(**kwargs)
  File "/usr/src/homeassistant/homeassistant/components/homeassistant/scene.py", line 366, in async_activate
    await async_reproduce_state(
    ...<4 lines>...
    )
  File "/usr/src/homeassistant/homeassistant/helpers/state.py", line 69, in async_reproduce_state
    await asyncio.gather(
        *(worker(domain, data) for domain, data in to_call.items())
    )
  File "/usr/src/homeassistant/homeassistant/helpers/state.py", line 63, in worker
    await platform.async_reproduce_states(
        hass, states_by_domain, context=context, reproduce_options=reproduce_options
    )
  File "/usr/src/homeassistant/homeassistant/components/light/reproduce_state.py", line 178, in async_reproduce_states
    await asyncio.gather(
    ...<6 lines>...
    )
  File "/usr/src/homeassistant/homeassistant/components/light/reproduce_state.py", line 165, in _async_reproduce_state
    await hass.services.async_call(
        DOMAIN, service, service_data, context=context, blocking=True
    )
  File "/usr/src/homeassistant/homeassistant/core.py", line 2794, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2837, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
        hass, entity, func, data, call.context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 644, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 179, in async_turn_on
    await self.hass.services.async_call(
    ...<5 lines>...
    )
  File "/usr/src/homeassistant/homeassistant/core.py", line 2794, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2837, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
        hass, entity, func, data, call.context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 644, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/wled/helpers.py", line 35, in handler
    raise HomeAssistantError("Invalid response from WLED API") from error
homeassistant.exceptions.HomeAssistantError: Invalid response from WLED API

[DedeHai]

to offer any help, we'd need to know what ther expected response is and what the actual response is WLED sends.
The next question then is why it works in 0.14 and if this is a bug in WLED or just something not (yet) fixed in HA.

[Aircoookie]

Thank you for the log!
I'm seeing a (503, {'error': 3}) error, which would imply this is a problem on the WLED side indeed and indicates this is a failure to allocate the JSON buffer, which also explains it's intermittency.
Now to figure out whether there is a bug with releasing the buffer somewhere, or a case of too many things trying to acquire the buffer at the same time...

[FGOD1983]

@Aircoookie anything we can do to figure this out? Personally I think it has to do with the ESP8266 module that has less memory then the ESP32, but still weird though because it works on 0.14.4 and it is a fairly short ledstrip (although long data cable to the ledstrip itself of 3m.

Just thinking out loud here as my knowledge about this is far from many of you here, but could it be that the cable length is causing a delay between the command send by WLED and the response from the strip itself which in turn causes these issues?

I can't speak for the others noticing this same bug, but I can imagine something like this happens. And it could be that added features in the 0.15 release causes the node to use more memory and therefor have more issues with things like this on lower end node like the ESP8266.

[julianfs]

If it helps, my log above is from an ESP8266 device running WLED.

[DedeHai]

@FGOD1983 can you try if the latest alpha (nightly build) changes anything? recently a new way of incoming request-queue handling was added but I am not familiar with what exactly changed.

[FGOD1983]

yeah sure, will give that a try the coming days. Let's hope it fixes the issues

[FGOD1983]

Do I need to use the normal 8266 bin file or the compat one?

ESP8266 binaries also include _compat binaries which use 0.14.0 style networking stack for situations where network connectivity was an issue.

[DedeHai]

try the normal one, I am not sure.
the new code does add this, maybe it fixes the issue:

[FGOD1983]

updated to the latest nightly (0.16.0 alpha). Let's see if that helps...

[blazoncek]

FYI handling of 503 (standard HTTP error code for internal server error) should be correctly implemented in Home Assistant.

As @DedeHai pointed out in the code, 503 is issued when JSON buffer cannot be locked as it is already locked by another request. There is a work by @willmmiles that will implement queueing in AWS but is not yet used. Unfortunately it increases binary size considerably.

If you want immediate resolution, switch to ESP32 or reduce number of HTTP/MQTT/IR/playlist JSON requests (and their cadence).

[FGOD1983]

@blazoncek so basically, my assumption that the setting of the preset by the automation and then turning the ledstrip on, goes to fast sometimes resulting in these errors and putting a second wait between them could resolve this issue?

[willmmiles]

Sorry I'm just catching up. The queuing logic is all merged, but it looks like something has gone wrong with that process - we shouldn't be generating a 503 and then calling deferResponse(); those are mutually exclusive operations. deferResponse() is supposed to be a replacement for the 503 that will delay handling the request until the lock can be acquired. I'm taking a further look.

[willmmiles]

Arg, I'm just misinterpreting the image of a diff above -- I'm used to a line-oriented diff display. The 0.16 code is fine.

So to summarize: a 503 HTTP response ("Service Unavailable") is WLED telling the client to "come back later", and no, HA doesn't respect them and faults instead. The switch to using the new AWS deferResponse() feature in 0.16 should "correct" this by queuing the request in WLED until it's ready to be serviced.

[FGOD1983]

so with 0.16 there is no need for HA to adjust something in this matter?

So far the ledstrip has been working correctly, but I didn't go outside enough in the evening/morning yet to test it enough.

[willmmiles]

so with 0.16 there is no need for HA to adjust something in this matter?

"Need" is a strong word .. ;) Yes, 0.16 attempts to accommodate clients that don't properly handle HTTP retries.

[wjb178]

I'm on 0.15.0
I think disabling wifi sleep on my esp8266 devices worked for me.
I will note that most of my errors came when an automation was turning off the light after it had been on for many hours.
I haven't seen errors in a few days.
What I did:
[]home-assistant/core#132284 (comment)