Coming back to the recent topics of the utility of the SDS011 data above 70% humidity plus the possibility of the BME280 sensors saturating and inaccurately reporting the humidity value. I was wondering how the community handles the data that is recorded at above 70% humidity? Is there a recommended algorithm and, if so, is this used?
As we have already explained, there is no data correction for humidity. The data produced by the SDS are sent the same under and above 70%. A few algorithms exist (in the litteratur), feel free to make a pull request with a correction in our Github repo sensor-software and of course document well the kind of correction you implemented. We will then check how it works and maybe merge it.
Thank you.
I guess I was wondering if anyone had already done this. Has anyone manipulated the higher humidity data for their own purposes or their own local website. Just wondered if there was a best practice that already existed.
Most of the algorithm which were developped for now by scientists were not stisfactory I think. But the FAIREMODE Group CT6 of the JRC of the EU Comission works currently on something. They can use data of sensor installed in a radius f 250 m around an official station.
It is the reason why we did such a map for people to place sensors next to the official stations in their city.:
https://sensors2ref.maps.sensor.community/#2/0.0/0.0
Ahh, thank you! Interesting and useful information.
Merci pour cette carte, qui me confirme que, concernant les stations “officielles”, je suis dans le désert…
Thank you for this map, which confirms that, as far as the “official” stations are concerned, I am in the desert…
Bernd Laquai worked on that. Correction of the raw data is possible.
https://www.researchgate.net/profile/Bernd-Laquai
Comparisons of HR compensation equations can be found here.
https://www.researchgate.net/publication/263289183_Humidity_and_Gravimetric_Equivalency_Adjustments_for_Nephelometer-Based_Particulate_Matter_Measurements_of_Emissions_from_Solid_Biomass_Fuel_Use_in_Cookstoves
Thank you for the links. Over the year I have become familiar with Bernd Laquai’s work and have found his papers very useful. The information from the other group of workers is very useful too.
Part of my reason for asking the question originally was to understand if anything was being done by others in the Sensor Community to correct for humidity etc. and determine best practice etc.
Now that I am more familiar with the research and I have investigated the output from some of the more expensive sensors that are used in the UK, I now realise that there is considerable variation in these. This is due to a number of factors including limits of detection.
Therefore, I am starting to think that::
-
Unless the humidity is persistent then any errors associated with overcounting due to high humidity are no higher than the difference between higher cost instruments that use different technologies (Eg TEOMs and FIDAS see link below.)
https://www.scottishairquality.scot/sites/default/files/publications/2021-09/Pilot_Research_Study_to_Investigate_PM_Monitorng_Techniques_in_Scotland_issue_1_0.pdf -
As the aim for most countries is to reduce this pollutant based on yearly averages, assuming the weather patterns over a year do not vary year on year, then any reduction seen using the low cost sensors could demonstrate that measures to reduce PM2.5 are effective.
-
For realtime sensing, it is better for the counts to be higher than expected than to not count potential hazards. Therefore any overcounting due to high humidity is acceptable and probably falls within the variation of other equipment. Equally, correcting the value using a simple equation such as SDS011 humidity corrected? - #3 by MichaelLazan might be preferred.
These above thoughts are mine only and I am happy for them to be debated, I also need to revisit the FAIREMODE work suggested by Pierre-Jean Guéno to see how they have progressed.
4 Likes
- “then any errors associated with overcounting due to high humidity are no higher than the difference between higher cost instruments that use different technologies”… are you sure ? I see huge discrepancies due to RH in low cost sensors (ref to https://pudi.lubw.de/detailseite/-/publication/90536-Ein_Vergleich_mit_einem_eignungsgeprüften_Feinstaubanalysator.pdf ), mainly due to RH, but not only.
- “assuming the weather patterns over a year do not vary year on year,”… with on-going weather changes, I’ll not do this assumption. Average yearly temperature is likely to change and RH at he same time. I’ll never state that “any reduction seen using the low cost sensors could demonstrate that measures to reduce PM2.5 are effective”. By the way, such statement may reduce the pressure on the “politics” on the subject.
- “Therefore any overcounting due to high humidity is acceptable and probably falls within the variation of other equipment” … not agree on that. “falls within variation”… refer to 1., and “overcounting due to high humidity is acceptable” is also difficult to support in front of politics with some know how on the subject because they will argue that the measure is not reliable and exceed the reality.
Because all of that, I fully agree with your last statement that a correction of the measurement for RH needs to be done and several equations can be use.
I can imagine a typical procedure as follow:
- plot the PM vs RH over 24 h (selecting 24 h while RH changes substantially). This will be the reference curve to adjust the parameters of the selected compensation curve. Maybe select several 24h samples and check that the parameters are almost constant.
- Correct the PM measures accordingly.
- recheck the parameters on a regular basis ( every week ?..). Change to new parameters as required (not sure how it can be automated to take such decision…)
Unfortunately, I do not have skill to implement such procedure.
Hope it will help.
Thank you for commenting on my thoughts, and I must really apologise that I was not specific enough in my response. I should have made it clear that I was only talking about the PM2.5 measurements. I also apologise for a lengthy response.
Your thoughts are totally reasonable, but currently it is still my belief that we should look beyond the need to be absolutely accurate with our measurements and play to the strengths of the counters, which in my opinion is to identify changes rather than give absolute values. Indeed, I think this does tie in with the first paragraph of section 6 of the LUBW document you linked where they discuss being qualitative rather than quantitative. I also ask why the sensor community does not use a correction for humidity. Is it because it is too difficult? Or not worth it?
On the subject of the LUBW document. It is a good preliminary study. However, it only gives results for PM10. Over the time I have owned my SDS011s I have learned to abandon using the PM10 data due to its inaccuracies. In calculating my PM2.5 values I average and normalise the data from more than one low cost system to be more accurate.
I agree with your point that yearly weather patterns might differ. Ideally variation over a number of years should be used to take account of this. I accept that using low cost systems to do this is not really ideal (see last bullet below). I believe that all European Governments are required to do something to reduce the PM2.5 levels (although the UK has not legislated for this yet). I therefore assume that if their measures are effective this will be observed in the low cost sensor data. Equally, if they are not delivering the required outcome, then the low cost systems will also show this. It is our way of keeping a check that the authorities are dealing with this issue and not a means of letting them off the hook.
For point 3 and the most difficult point. There are so many factors that influence the counts at high humidity that there isn’t really a fit all solution. As I said above, if there was an easy solution I think the sensor community would have already implemented a correction factor. These factors include:
- The hygroscopic nature of the particles (variable throughout the year and from location to location).
- The variability of the number of smaller particles in the environment that become more detectable as the humidity increases (effectively this improves the limit of detection and could be telling us something useful).
- How much of the higher count is associated solely with water droplets.
- The accuracy of the BME280 systems. These tend to saturate and read too high, so if this value is used to correct for particle count this could be in error.
- The SDS011s age and give different results with time.
I believe Bernt Laquai discusses all of this in his papers.
Because of all of these variables, rather than use a correction factor, I have been investigating the differences in the PM2.5 concentrations associated with heated and unheated sensors at my location. I have already reported some of my findings (see SDS011 datasets with- & without dryer - #4 by sensorsalnorth12 and Preliminary comparison of heated and unheated systems). There is clearly a difference. At the time of posting, I estimated a difference of around 20% between the heated and unheated systems. However, the heated system doesn’t correct for all circumstances e.g. fog.
My next step is (or was) to determine a way of calibrating the heated low cost system against a calibrated sensor that is already accepted by authorities such as a TEOM or FIDAS (assuming I could get agreement from my local authority). This is when I discovered that there appears to be a similar discrepancy between these systems!
So this is why I am currently thinking that this is far, far more complex than just correcting using a simple formula. The difference in the accepted technologies is a concern. At the moment I am trying to understand which technology was used by the WHO for their guidelines as this should be the one for calibration.
From my perspective the more I find out and think I know about this subject, the more I realise that I do not know.
One final thought is that I believe that some of the above bulleted factors are less significant over 12 months than 24 hours. I recently calculated the annual average PM2.5 concentration for my location and found that it agreed reasonably well with that of the modelled data supplied to UK’s DEFRA.
I hope this explains my reasoning better.
5 Likes
RIVM, the Dutch national institute for public health and environmental hygiene (literally translated) facilitates a project called Samen Meten (measuring together). Long story short: they’re working on an experimental re-calibration of the SDS011’s PM values against their own gravimetric measuring stations. Assuming PM10/PM2.5 do not change much within a certain urban area, overestimated SDS011 in that area, can be corrected for against gravimetric measurements in the same area.
More here, in Dutch. Google Translate is your friend.
A formula to correct overestimation as a function of humidity and temperature does not seem to exist yet. However, they do provide an API that delivers raw data and re-calibrated data of individual sensors. From what I’ve seen, this is hour by hour rolling averages.
There’s also openSenseMap from which you can download even more raw data, taken every minute or so.
1 Like
A few stuff I made in the last weeks:
Actually, the official station is higher than the sensors (no correction).
Moving average 24h:
Mesures:
1 Like
I have found that TEOMs read higher than FIDAS systems. I think is to do with the limit of detection. I have found that our local council sensor reads higher than our sensor community sensors it is currently a TEOM.
Hi everyone! I’m promoting Sensor Community in my country, it is quite foggy and humid in winter (and most polluted due to residental gas/coal/wood heating), so the question about the accuracy will arise.
Our govt uses these stations as “official” https://click.clarity.io/hubfs/Marketing%20Assets%20-%20PDFs/Product%20and%20Specification%20Sheets/Node-S%202%20Specifications%20Sheet%20(October%202021).pdf
I don’t understand do they correct humidity? Are they heated? Are they TEOM or FIDAS?
It is said that they have Correlation (R2) with USEPA FEM instrument > 0.8
I thought about using these stations as a source for correcting the humidity issue.
Unfortunately, the AQMD site is down. If even the US .gov websites do not work when I need them…
The sensor in this system is probably a chinese stuff as well and probably not RH corrected.
The accuracy issue with RH is mostly a merchant of doubt technique…
SC produce the raw data.
We could/should implement an humidity correction on the map but it will start long debates about which one etc.
The RIVM in Holland is developing a correction algorithm which spreads among the low cost sensors but it is still not public. But actully it can be seen on their map.
2 Likes
Have you seen any of the resulting FAIREMODE papers on this?
The technique used is light scattering so it is probably most like FIDAS rather than TEOM which is gravimetric. There is no mention of heating the air but they state that the operational range is up to 98% rel. humidity. I cannot comment on its accuracy at the higher humidities.