Getting noise data into sensor.community

bertrik · September 5, 2021, 12:56pm

Marcel Meek has developed an environmental noise sensor that picks up audio, analyses the audio into intensities per frequency band and sends the data over LoRaWAN. His project can be found here:

Some interesting features:

Measures a spectrum instead of a single LAeq value (+ min/max), so you have more details about the type of noise, can apply more fine-grained correction factors.
Uses LoRaWAN instead of WiFi for telemetry. Some cities have pretty good coverage now from TheThingsNetwork. There is no dependency on a master particulate matter sensor node.
Uses off-the-shelf components, an ESP32 board with integrated LoRa transceiver, like the TTGO LoRa32 v1 or the Heltec LoRa32 v2

I think that is a fantastic idea for a citizen science project, so I have written a kind of forwarder that picks up his data and can forward it to sensor.community. Currently my forwarder emulates a sensor.community node with a DNMS module. My code can be found at:

It sends data like this:

2021-09-05 14:00:42,258 INFO  SensComUploader:57 - Sending for esp32-216015141040444 to pin 15: '{"software_version":"https://github.com/bertrik/noiseforwarder","sensordatavalues":[{"value_type":"noise_LAeq","value":"32.4"},{"value_type":"noise_LA_min","value":"30.5"},{"value_type":"noise_LA_max","value":"42.8"}]}'
2021-09-05 14:00:42,760 WARN  SensComUploader:62 - Request failed: Forbidden

The data is currently refused because the node is not registered. I could of course register it, but I would not want to send data before verifying with sensor.community first if this is OK.

Our Dutch National Institute for Public Health and the Environment (National Institute for Public Health and the Environment | RIVM) picks up the sensor.community data and combines it with other sources to display on their national citizen science map. So if we can get the data into sensor.community, that would be the ideal situation.

Who should I talk to about this within sensor.community?
I am having trouble getting a reply by e-mail, my plan is to join the weekly online video call next wednesday.

Lodie · September 6, 2021, 10:16am

Very interesting. I would love to see this or something similar being integrated as Noise seems to cause illness.

ricki-z · September 6, 2021, 5:00pm

@bertrik I have seen that the project is also measuring min, max, and average levels for dB(A) for an interval. So we could easily integrate these values. Could you suggest a short name for this sensor (instead of DNMS)? This way we could include the data in a short time.
We are only measuring dB(A) as this is the value all regulations are based on…

bertrik · September 9, 2021, 12:38pm

@ricki-z Thanks for your reply. I don’t know a nice short name for the sensor yet, I’d like input from Marcel too, but he is on holiday for a week. We’ll come back to that, OK?
We’re also thinking about the best way to present the measurements done against reference microphones.

What I’ve done so far, is emulate the interface used for DNMS, so indeed we’re sending min,max,avg(Leq) levels in dB(A). The sensor was registered at devices.sensor.community as DNMS for now. I’ve marked it as ‘indoor’ for now (even though it is outdoor) to indicate the experimental state.

I see RIVM is already picking up the data, it’s the one in the city of Apeldoorn, Netherlands on their map at Samen Meten - Dataportaal
(select “Geluid/geluidniveau” on the left)

ricki-z · September 9, 2021, 8:37pm

I will try to add the other dB values to the list of possible values. The names should be similar to the db(A) values.
But I’m unsure about the frenquency values. We may be able to save the JSONs as a string. But this is a large amount of data that needs to be saved. This would result in a faster growing database and archive …

Roland · November 8, 2023, 1:45pm

I have just completed my ESP32 noise sensor based on LoRaSoundkit. I forward the TTN data to my own server und from there via logstash pipeline to the API. The tricky think was to find the logstash commands to build the JSON structure.

input { pipeline { address => "sensor.community-noise" } }

filter {
  # process data for sensor.community
    mutate {
      add_field => [ "[@metadata][app_id]", "%{[end_device_ids][application_ids][application_id]}" ]
      add_field => [ "[@metadata][device_id]", "%{[end_device_ids][device_id]}" ]
    } # mutate
  #

  # BEGIN device specific configuration section
  # each TNN device (sensor)  requires one configuration section
  if ([@metadata][device_id] == "YOUR-TTN-DEVICENAME") {
    mutate {
      add_field => { "[@metadata][send_data]" => "true" }
      add_field => { "[@metadata][sensor]" => "YOUR-NODE-ID" }
    }
  }
  # END device specific configuration section

  if ([@metadata][send_data] == "true") {
    mutate {
      add_field => { "software_version" => "1.0" }
    }

    ruby {
        code => '
          event.set("sensordatavalues", [
            {"value" => event.get("[uplink_message][decoded_payload][la][avg]"),"value_type" => "noise_LAeq"},
            {"value" => event.get("[uplink_message][decoded_payload][la][min]"),"value_type" => "noise_LA_min"},
            {"value" => event.get("[uplink_message][decoded_payload][la][max]"),"value_type" => "noise_LA_max"}
          ] )
        '
    }

  } #if

  mutate { remove_field => ["app_id","@version","@timestamp","host","received_at","_source","headers","correlation_ids","uplink_message","end_device_ids"] }
} # filter

output {
# curl test command to upload parameters
# curl --location --request POST 'https://api.sensor.community/v1/push-sensor-data/' --header 'Content-Type:application/json' --header 'X-Pin:15' --header 'X-Sensor:YOUR-NODE-ID' --data-raw '{"software_version": "1.0", "sensordatavalues":[{"value_type":"noise_LAeq","value":"21.21"},{"value_type":"noise_LA_max","value":"22.22"}, {"value_type":"noise_LA_min","value":"23.23"}]}'

  if ([@metadata][app_id] == "YOUR-TTN-APPLICATION-ID" and [@metadata][send_data] == "true") {
    http {
     url => "https://api.sensor.community/v1/push-sensor-data/"
     #url => "http://localhost:1099"
     http_method => "post"
     format => "json"
     headers => {
       "X-Pin" => "15"
       "X-Sensor" => "%{[@metadata][sensor]}"
     }
    } # http

   # for DEBUG only
   # file {
   #  path => "/tmp/sensor.community-%{[@metadata][device_id]}.log"
   #
    #}

  #stdout { 
  # codec => rubydebug { metadata => true } 
  #}
 } # if [@metadata][app_id] 
} # output

The community version of Logstash is available at Download Logstash Free | Get Started Now | Elastic.

The ESP32 consumes 35 mA in the middle and now I will connect the device to my solar based PM sensor.

I send the noise data also to opensensemap with a second logstash pipeline.

ricki-z · November 10, 2023, 12:34am

Hello @Roland ,
you should clarify what people should use for ‘[@metadata][sensor]’ .
This should have a prefix ‘TTN-’ (the sensor ‘board’ at devices.sensor.community) and then a unique ID. We prefer the chipID or the TTN device ID for the second part. With these people should be able to register their device.

Roland · November 10, 2023, 10:37am

Hi @ricki-z, I changed the nodename to TTN-XXXXXX. Is there any different functional behavior regarded to the prefix TTN or ESP32?

ricki-z · November 10, 2023, 6:03pm

It’s only to differntiate between direct transmission i.e. wifi and TTN. If there is a problem with transmitting data then it’s easier for us to find possible problems. Wifi is a local problem in most cases, while with TTN there are much more steps included until the data is received by our servers.

bertrik · December 23, 2023, 7:28pm

I heard there is another API now for noise, which should also be ready for high-update-rate noise data. Is there any more information about his?