Codes indicating sensor errors

Maybe I overlooked this in the documentation. Since I read my sensors (SDS011, SPS30) locally in Home Assistant (HA, home automation system that I use), I regularly receive sensor data that falls far outside the usual measuring range. I don’t see those outliers in the data sent to the Sensor Community site. They may be filtered out there. That’s what I’m doing myself in HA now.
I wonder if these are special data values that indicate a (error) condition. Is there documentation about that?

I doubt that there is any type of filtering. Are you referring to the sensor community or to madavi? If it is the former, there are regular errors for many users when uploading to the server, so momentary spikes would be less likely to appear in this case.

There is no filter. Please send me the values from archive, the graphics and the Sensor ID

HA produces maybe byte shifting in the transmission which produces Bad values.

I am sending the requested information:

  • the sensor data from the past 10 days stored in the Home Assistant database;
  • the graphs as shown by Home Assistant for the past 10 days;
  • the sensor ID: 14990217
    It took some work (and a steep learning curve) to get the data from the HA database. I did this with TablePlus, with which the HA database can be read properly.
    Note: The time is in UTC.
    It seems I was able to locate part of the problem. If the outliers occur, the data has been swapped. For example, it seems that the PM2.5 then gets the value of the air pressure. I have done some more tests, the data as HA shows it, corresponds well with the data that the sensor shows as a web server.
    For the sake of completeness: I use the following template to read in the sensor values:
- platform: command_line
   name: "Air Quality opzij PM10 (SDS)"
   command: 'curl'
   value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
   unit_of_measurement: "µg/m³"
- platform: command_line
   name: "Air Quality Aside PM2.5 (SDS)"
   command: 'curl'
   value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
   unit_of_measurement: "µg/m³"

Outliers are filtered out by:

- platform: template
      friendly_name: Air quality opzij PM 2.5 correctie
      unit_of_measurement: "µg/m³"
      value_template: >-
        {% if states('sensor.luchtkwaliteit_opzij_pm2_5_sds') | float < 10000 %} {{ states('sensor.luchtkwaliteit_opzij_pm2_5_sds') }}
        {% else %} 0
        {% endif %}

Data can be found here:

– Link to data removed by admin after issue was solved –

By the way, import by Home Assistant is based on the approach described in "Alarm" for values - #4 by RuudvanMunsterZTM

Do you mean by “shifting in the transmission” that some values are missing in the JSON file? After some investigations I found that it happens regularly that the lines with the SDS sensor data in the file below are missing.

> {"software_version": "NRZ-2020-133", "age":"118",
> "sensordatavalues":[
> {"value_type":"SDS_P1","value":"17.35"},
> {"value_type":"SDS_P2","value":"9.60"},
> {"value_type":"BME280_temperature","value":"7.12"},
> {"value_type":"BME280_pressure","value":"103936.44"},
> {"value_type":"BME280_humidity","value":"62.25"},
> {"value_type":"samples","value":"5060693"},
> {"value_type":"min_micro","value":"28"},
> {"value_type":"max_micro","value":"20128"},
> {"value_type":"interval","value":"145000"},
> {"value_type":"signal","value":"-40"}
> ]}

Because the code I have been using to import the local sensor data into Home Assistant sensor assigns the data to the sensor based upon the position in the JSON data area, the BME280_temperature is assigned to sensor SDS_P1 and the BME280_pressure value is assigned to SDS_P2. Especially the BME280_pressure values are seen in the graphs, becaus of their higher values.

I solved the problem by changing the following code to obtain the values locally:

- platform: command_line
  name: "Luchtkwaliteit Opzij PM10 (SDS)"
  command: 'curl'
  value_template: "{{ value_json.sensordatavalues[0].value | round(1) }}"
  unit_of_measurement: "µg/m³"
- platform: command_line
  name: "Luchtkwaliteit Opzij PM2.5 (SDS)"
  command: 'curl'
  value_template: "{{ value_json.sensordatavalues[1].value | round(1) }}"
  unit_of_measurement: "µg/m³"


- platform: rest
  name: JSON airquality
  unique_id: airquality_opzij
    - software_version
    - age
    - sensordatavalues
  value_template: "{{ value_json.software_version }}"

- platform: template
      unique_id: airquality_opzij_software_version
      friendly_name: "Airquality Opzij Software version"
      value_template: "{{ state_attr('sensor.airquality', 'software_version') }}"
      unique_id: airquality_opzij_data_age
      friendly_name: "Airquality Opzij Data Age"
      value_template: "{{ state_attr('sensor.airquality', 'age') }}"

- platform: template
      unique_id: airquality_opzij_PM10_REST
      friendly_name: "Air quality opzij PM10 (REST)"
      unit_of_measurement: "µg/m³"
      value_template: >-
        {% if state_attr('sensor.airquality', 'sensordatavalues')[0].value_type == 'SDS_P1' %}
          {{ state_attr('sensor.airquality', 'sensordatavalues')[0].value  | round(1) }}
        {% else %} 0 
        {% endif %}

- platform: template
      unique_id: airquality_opzij_PM25_REST
      friendly_name: "Air quality opzij PM2.5 (REST)"
      unit_of_measurement: "µg/m³"
      value_template: >-
        {% if state_attr('sensor.airquality', 'sensordatavalues')[1].value_type == 'SDS_P2' %}
          {{ state_attr('sensor.airquality', 'sensordatavalues')[1].value  | round(1) }}
        {% endif %}

This solution checks whether the sensors are at the right position in the JSON file. It compares the value_type of the sensor data against the expected value_type. The problem I described above was because some of the data are simply missing in the datafile if there is a sensor error (e.g. if the measurment failed) (I am still searching for the cause of that)

The new template ignores the data if it is at the wrong position in the JSON file.
You see two variations on this theme:

  • The first sensor template adds zero’s for the missing data. Which is convenient for testing purposes.
  • The second sensor simply ignores the wrong data. Which is beter for operational use, I think.

There must be a better way to code this, using a list construction which links the data_value to the right sensor based upon the value_type. I am still a novice in this kind of programming, in a steep learning curve. So suggestions here are very welcome.

The new solution is working now for about a week, together with the former solution, which enables me to check how it works. It looks very stable now.

Based upon my experience of the last months, I think we should prefer the following approach:

  • Have your sensor send the data directly to the Sensor Community site, for sharing data and research;
  • Use a local data-exchange for your own operational purposes. As far as I see now, that is far more reliable. And it fits better in the (Home Assistant) philosophy to avoid the cloud for real-time operational purposes.

As a side-effect there is less load on the Community servers. Their intended use is probably just for sharing data and research not for being “polled” every few minutes by a lot of users.

And of course, I am still wondewring why these data are missing. Are there transmission problems? Is the sensor defective or has it to be cleaned up? Now having more control about the incoming data, I can do some investigation on that.

I mean byte shifting in the Serial transmission from the sensor to the board. It can produce huge values.

Your cables are defective :slight_smile:
Check the connections and put some Kontakt60 everywhere (not a lot, just the right amount…).

Thanks! I’ll check that.