Timeouts to api.sensor.community/v1/push-sensor-data/

I am using Home Assistant to push my sensor values to your great project by submitting rest data. This has been working fine over the last 9 months or so, but since a few days I am receiving timeout errors on the API calls to the api endpoint:

2021-04-13 14:36:44 WARNING (MainThread) [homeassistant.components.rest_command] Timeout call https://api.sensor.community/v1/push-sensor-data/

The actual Home Assistant automation triggering the sending can be found here, which triggers a rest_command which can be found here.

I am not suspecting anything to be wrong with the Home Assistant part - as many data uploads go through just fine between these errors and most data is visible on the map; it is just that in the last few days I see a lot more timeout errors returning. It used to be in the numbers of 1-2 errors a day, now I am seeing errors every few minutes.

While investigating, I noticed that a GET to the endpoint also provides slow results - it took 22 seconds to return with a 405 - Method not allowed error.

Is there anything we can do on our side to improve this behavior?

Could you please check this again? If possible please check this also without https (http only). There is a significant overhead if SSL is used.

1 Like

The 14:36 error was the last one today. I’ll keep an eye on it.

I’ll update one of the commands to use http instead of https, to see if it makes any difference in number of timeout errors.

The http endpoints also timeout once in a while:

2021-04-14 08:44:55 WARNING (MainThread) [homeassistant.components.rest_command] Timeout call http://api.sensor.community/v1/push-sensor-data/
2021-04-14 08:52:55 WARNING (MainThread) [homeassistant.components.rest_command] Timeout call http://api.sensor.community/v1/push-sensor-data/

Timeout is the default 10sec.

Changing to http and/of increasing timeout seem to improve matters, suggesting sensor.community might need faster/bigger servers. So i made another small donation :slight_smile:

Hi, I never sent data to sensor.community before, so not sure if this some temporary degradation or this is “normal” behavior, or there is something wrong on my side. I do have occasional periods when my internet connection feels a bit slow connecting to some servers (but not all, maybe depends on location). To understand if that’s something on my side, I also sent data to madavi.de, which is (as I understand) located somewhere near.

So what I see, is sending data to api.sensor.community often takes quite a lot of time. I don’t know exact distribution, as I have only 1 second timeout in my ESPHome device. But trying to send requests from my PC, I see that it could take 5-30 seconds. Below you can see a screenshot with detailed timing. However, seems like it varies quite a lot. Even in “slow periods”, some requests complete quite fast <1s. I haven’t run my sensor for long time yet, but it seems that there are some “slow periods” when requests timeout 95% of time (in my case 1s timeout as I said), and “fast periods” when most of requests complete successfully within 1s timeout.

Also, you can see that in screenshot below total request time is mainly a sum of TCP Handshake + Transfer start. I wouldn’t say TCP Handshake takes always that much, usually more often time is spent somewhere after connection was established, but still it occurs quite often, it seems like some server/web server issue rather than app code or database. And note, that I’m not using HTTPS, just plain HTTP.

For comparison, I also send data to madavi.de. I’m not sure how it relates to sensor.community and whether they use similar backend/db, but usually requests to it complete much faster.

Also I have a question, if it possible to send data to sensor.community in one request (temperature + PM) like it is possible with madavi.de? It is especially could be useful if requests take long time. Having a single 5-30s request is better than 2 of them.

Why I’m setting such small timeout =1s? I’m using ESPHome and by default it has 5s watchdog, which is a background timer that checks that no thread blocks loop for long time. If it does - it reboots the device. So I’m considering 2 options:

  1. Increasing this timeout. However, as I’ve seen some requests could take ~30sec and if you send 2 of them sequentially, I’m not sure what timeout should be “good enough”, and I believe blocking the loop for long time is not a good practice, as there are some other background tasks that ESP (like wifi) might want to execute.
  2. Just ignoring timeouts. Seems like despite receiving timeouts the data still passes through most of the time, I’m just not getting confirmation whether the request was successful or not. However I’m not sure if that the case when timeout happens on TCP handshake stage.

I understand that you have quite a number of sensors (like ~13K, if that’s the correct number on the home page) that are sending you data every few minutes, but still it feels a bit strange that inserting 3 values into DB might take this long.

Today I increased timeout to 5 seconds and gathered some statistics. I also used 2 other servers for comparison.

From the first chart you can see individual response times:

Second chart displays what percentage of requests complete within what time:

A few observations from these charts:

  1. Best case (and most frequent) response time for both madavi.de and sensor.community is about 300-400 ms. About ~80% of all requests to madavi.de complete within this time, however for sensor.community this percentage is significantly lower ~45%
  2. While madavi.de performs quite well overall, there are still ~15% of request that take >1s. This might be due to network latency and not server’s issue. There more experiments needed to dissect it further. But if sensor.community endpoint had same response times, I’d be pretty happy.
  3. sensor.community clearly performs worst of all. Even with 5s timeout only 80% requests complete within it. And you can’t really see what’s the distribution >5s, it could take 10-20-30 seconds.
  4. There are periods (like in the middle of timeline) when sensor.community performs quite well.

Note: Why if timeout is set to 5s, there are some datapoints larger than 5s. Actually there are 2 timeouts: connect timeout and response timeout. Each of them is set to 5s, so in total request could complete successfully, even if total time (connect + response) is greater than 5s, but less than 10s.

1 Like

Hey,

I retrieve my SDS011 values via an mqtt connection.
I would be willing to upload those data, but where can I find a description on the API?

Br, Osven

I believe official API documentation is here:

There is also unofficial one, which is also quite helpful (you just need to ignore chrome warnings about expired certificate):
https://api-sensor-community.bessarabov.com/

Also you might try searching github for examples of API usage in C++ or Python:

1 Like