Maybe I overlooked this in the documentation. Since I read my sensors (SDS011, SPS30) locally in Home Assistant (HA, home automation system that I use), I regularly receive sensor data that falls far outside the usual measuring range. I don’t see those outliers in the data sent to the Sensor Community site. They may be filtered out there. That’s what I’m doing myself in HA now.
I wonder if these are special data values that indicate a (error) condition. Is there documentation about that?
I doubt that there is any type of filtering. Are you referring to the sensor community or to madavi? If it is the former, there are regular errors for many users when uploading to the server, so momentary spikes would be less likely to appear in this case.
There is no filter. Please send me the values from archive, the graphics and the Sensor ID
HA produces maybe byte shifting in the transmission which produces Bad values.
I am sending the requested information:
- the sensor data from the past 10 days stored in the Home Assistant database;
- the graphs as shown by Home Assistant for the past 10 days;
- the sensor ID: 14990217
It took some work (and a steep learning curve) to get the data from the HA database. I did this with TablePlus, with which the HA database can be read properly.
Note: The time is in UTC.
It seems I was able to locate part of the problem. If the outliers occur, the data has been swapped. For example, it seems that the PM2.5 then gets the value of the air pressure. I have done some more tests, the data as HA shows it, corresponds well with the data that the sensor shows as a web server.
For the sake of completeness: I use the following template to read in the sensor values:
- platform: command_line
name: "Air Quality opzij PM10 (SDS)"
command: 'curl http://192.168.178.20/data.json'
value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
unit_of_measurement: "µg/m³"
- platform: command_line
name: "Air Quality Aside PM2.5 (SDS)"
command: 'curl http://192.168.178.20/data.json'
value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
unit_of_measurement: "µg/m³"
Outliers are filtered out by:
- platform: template
sensors:
air_quality_opzij_pm25:
friendly_name: Air quality opzij PM 2.5 correctie
unit_of_measurement: "µg/m³"
value_template: >-
{% if states('sensor.luchtkwaliteit_opzij_pm2_5_sds') | float < 10000 %} {{ states('sensor.luchtkwaliteit_opzij_pm2_5_sds') }}
{% else %} 0
{% endif %}
Data can be found here:
– Link to data removed by admin after issue was solved –
By the way, import by Home Assistant is based on the approach described in "Alarm" for values - #4 by RuudvanMunsterZTM
Do you mean by “shifting in the transmission” that some values are missing in the JSON file? After some investigations I found that it happens regularly that the lines with the SDS sensor data in the file below are missing.
> {"software_version": "NRZ-2020-133", "age":"118",
> "sensordatavalues":[
> {"value_type":"SDS_P1","value":"17.35"},
> {"value_type":"SDS_P2","value":"9.60"},
> {"value_type":"BME280_temperature","value":"7.12"},
> {"value_type":"BME280_pressure","value":"103936.44"},
> {"value_type":"BME280_humidity","value":"62.25"},
> {"value_type":"samples","value":"5060693"},
> {"value_type":"min_micro","value":"28"},
> {"value_type":"max_micro","value":"20128"},
> {"value_type":"interval","value":"145000"},
> {"value_type":"signal","value":"-40"}
> ]}
Because the code I have been using to import the local sensor data into Home Assistant sensor assigns the data to the sensor based upon the position in the JSON data area, the BME280_temperature is assigned to sensor SDS_P1 and the BME280_pressure value is assigned to SDS_P2. Especially the BME280_pressure values are seen in the graphs, becaus of their higher values.
I solved the problem by changing the following code to obtain the values locally:
- platform: command_line
name: "Luchtkwaliteit Opzij PM10 (SDS)"
command: 'curl http://192.168.178.20/data.json'
value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
unit_of_measurement: "µg/m³"
- platform: command_line
name: "Luchtkwaliteit Opzij PM2.5 (SDS)"
command: 'curl http://192.168.178.20/data.json'
value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
unit_of_measurement: "µg/m³"
by:
- platform: rest
name: JSON airquality
unique_id: airquality_opzij
json_attributes:
- software_version
- age
- sensordatavalues
resource: http://192.168.178.20/data.json
value_template: "{{ value_json.software_version }}"
- platform: template
sensors:
airquality_opzij_software_version:
unique_id: airquality_opzij_software_version
friendly_name: "Airquality Opzij Software version"
value_template: "{{ state_attr('sensor.airquality', 'software_version') }}"
airquality_opzij_age:
unique_id: airquality_opzij_data_age
friendly_name: "Airquality Opzij Data Age"
value_template: "{{ state_attr('sensor.airquality', 'age') }}"
- platform: template
sensors:
air_quality_opzij_pm10_new:
unique_id: airquality_opzij_PM10_REST
friendly_name: "Air quality opzij PM10 (REST)"
unit_of_measurement: "µg/m³"
value_template: >-
{% if state_attr('sensor.airquality', 'sensordatavalues')[0].value_type == 'SDS_P1' %}
{{ state_attr('sensor.airquality', 'sensordatavalues')[0].value | round(1) }}
{% else %} 0
{% endif %}
- platform: template
sensors:
air_quality_opzij_pm25_new:
unique_id: airquality_opzij_PM25_REST
friendly_name: "Air quality opzij PM2.5 (REST)"
unit_of_measurement: "µg/m³"
value_template: >-
{% if state_attr('sensor.airquality', 'sensordatavalues')[1].value_type == 'SDS_P2' %}
{{ state_attr('sensor.airquality', 'sensordatavalues')[1].value | round(1) }}
{% endif %}
This solution checks whether the sensors are at the right position in the JSON file. It compares the value_type of the sensor data against the expected value_type. The problem I described above was because some of the data are simply missing in the datafile if there is a sensor error (e.g. if the measurment failed) (I am still searching for the cause of that)
The new template ignores the data if it is at the wrong position in the JSON file.
You see two variations on this theme:
- The first sensor template adds zero’s for the missing data. Which is convenient for testing purposes.
- The second sensor simply ignores the wrong data. Which is beter for operational use, I think.
There must be a better way to code this, using a list construction which links the data_value to the right sensor based upon the value_type. I am still a novice in this kind of programming, in a steep learning curve. So suggestions here are very welcome.
The new solution is working now for about a week, together with the former solution, which enables me to check how it works. It looks very stable now.
Based upon my experience of the last months, I think we should prefer the following approach:
- Have your sensor send the data directly to the Sensor Community site, for sharing data and research;
- Use a local data-exchange for your own operational purposes. As far as I see now, that is far more reliable. And it fits better in the (Home Assistant) philosophy to avoid the cloud for real-time operational purposes.
As a side-effect there is less load on the Community servers. Their intended use is probably just for sharing data and research not for being “polled” every few minutes by a lot of users.
And of course, I am still wondewring why these data are missing. Are there transmission problems? Is the sensor defective or has it to be cleaned up? Now having more control about the incoming data, I can do some investigation on that.
I mean byte shifting in the Serial transmission from the sensor to the board. It can produce huge values.
Your cables are defective 
Check the connections and put some Kontakt60 everywhere (not a lot, just the right amount…).
Thanks! I’ll check that.
Did you solve it?
I’ve just learn by reading this thread the possibility to get data from “http://…local IP…/data.json”… I did similar thing by taking in another manner (someone could consider this “not efficient”, but it was the only one that I was able to do… I’m not so skilled):
- Install in Home Assistant the MULTISCRAPE integration from HACS (GitHub - danieldotnl/ha-multiscrape: Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.), I already was using it for other purpose (get hourly price of energy from an institutional website).
- Add to the configuration.yaml the following code (I’m reporting just for the first sensor)
multiscrape:
- name: AQ Local
resource: http://...local IP.../values
scan_interval: 60
sensor:
- unique_id: AQ Local PM2.5
name: AQ Local PM2.5
select: 'tr:nth-child(2) > td.r'
value_template: '{{ value[:-6] | float }}'
unit_of_measurement: 'µg/m³'
icon: 'mdi:molecule'
I’ll test my way, and in case I’ll try with the “http://…local IP…/data.json”.
Thank you