Upload from influx db

Tanja · February 25, 2025, 4:36pm

is there already a solution for just pushing the data from influx to your servers? we have also a lot of historical data, would it be also possible to send that to you?

pjg · February 25, 2025, 6:57pm

What kind of data do you have ?
Which sensor(s)?
Which time span ?

Tanja · February 25, 2025, 7:53pm

i have a SDS011 with a bme280, i fed them directly into my influxdb, i started logging 2023-02-15, still doing so. these are the fields i have:

time BME280_humidity BME280_pressure BME280_temperature SDS_P1 SDS_P2 interval max_micro min_micro node samples signal

pjg · February 26, 2025, 5:42am

For collecting the data from the beginning I don’t know.
But for current data, you just have to forward from influx db to SC API with POST requests:

Prepare everything for the SC API and also the Madavi API (2 requests) and then I will help you to finalize the requests because there is a small thing to take care of.

Tanja · February 27, 2025, 2:52pm

hmmm, thank you very much. i found this page in english. i was hoping someone already built such a forwarder tool. but i guess, i have to do it then. will take some time, being swamped with other stuff. but i will have come back to this.

pjg · February 27, 2025, 4:35pm

Hi,
These are normal POST requests with headers and data as json.

Tanja · February 27, 2025, 4:39pm

sure, sounds easy enough, i just hoped someone already had this done.

pjg · February 27, 2025, 4:45pm

A lot of people are doing it currently. Actually I found everything in my notes:

SC server:

url: https://api.sensor.community/v1/push-sensor-data/
method: POST
content_type: ‘application/json’
headers:
X-Pin: 1
X-Sensor: Board ID

{
  "software_version": "<WHAT YOU WANT>", 
   "sensordatavalues":[
    {"value_type":"P1","value":"<PM10>"},
    {"value_type":"P2","value":"<PM2.5>"}
  ]
}

url: https://api.sensor.community/v1/push-sensor-data/
method: POST
content_type: ‘application/json’
headers:
X-Pin: 11
X-Sensor: Board ID

{
  "software_version": "<WHAT YOU WANT>", 
   "sensordatavalues":[
     {"value_type":"temperature","value":"<TEMP>"},
     {"value_type":"pressure","value":"<PRESS>"},
     {"value_type":"humidity","value":"<HUMI>"}
  ]
}

Madavi server:
url: https://api-rrd.madavi.de/data.php
method: POST
content_type: ‘application/json’
headers:
X-Pin: 1
X-Sensor: Board ID

{
  "software_version": "<WHAT YOU WANT>", 
   "sensordatavalues":[
    {"value_type":"SDS_P1","value":"<PM10>"},
    {"value_type":"SDS_P2","value":"<PM2.5>"}
  ]
}

url: https://api-rrd.madavi.de/data.php
method: POST
content_type: ‘application/json’
headers:
X-Pin: 11
X-Sensor: Board ID

{
  "software_version": "<WHAT YOU WANT>", 
   "sensordatavalues":[
     {"value_type":"BME280_temperature","value":"<TEMP>"},
     {"value_type":"BME280_pressure","value":"<PRESS>"},
     {"value_type":"BME280_humidity","value":"<HUMI>"}
  ]
}

The problem is the Board ID.

There is no InfluxDB-…
@ricki-z An idea ?

Tanja · February 28, 2025, 7:17pm

thanks for sharing your awesome notes! <3
why cannot i use the original board id from the device that dumps to my influx?

pjg · February 28, 2025, 9:40pm

Yes you can! Is it an esp ?

Tanja · March 7, 2025, 2:02am

yes. padding to 20 chars.

Tanja · March 7, 2025, 2:44am

three questions:

the “values” in the json payload, do they have to be strings, or can they be floats?
the software_version, don’t really know, is N/A, null, or something else acceptable?
the boardid, would that be ''esp8266-1234567" or only the numbered part “1234567”?

apart from these details i think i have a first draft that might work (the boardid needs to be set correctly, and the request.posts uncommented, but otherwise i’d give it a try):

#!/usr/bin/env python

from influxdb import InfluxDBClient
import time, requests, datetime

boardid = 'esp8266-1234567'

lastseen=None
def push(d):
   global lastseen
   # [{'time': '2025-03-07T02:00:54.099740Z', 'BME280_humidity': 73.43, 'BME280_pressure': 101565.53, 'BME280_temperature': 6.63, 'SDS_P1': 13.2, 'SDS_P2': 8.0, 'interval': 10000.0, 'max_micro': 1137.0, 'min_micro': 23.0, 'node': 'esp8266-1234567', 'samples': 416113.0, 'signal': -93.0}]
   ts = datetime.datetime.fromisoformat(d['time'][:-1]).replace(tzinfo=datetime.timezone.utc)
   if ts == lastseen: return
   now = datetime.datetime.now(datetime.timezone.utc)
   if(ts<now-datetime.timedelta(seconds=15)):
       print("last record is too old:", now-ts)
       return
   print(d)
   lastseen = ts

   data = { "software_version": "<WHAT YOU WANT>",
           "sensordatavalues":[
               {"value_type":"P1","value":d['SDS_P1']},
               {"value_type":"P2","value":d['SDS_P2']}]}
   headers = { 'X-Pin': 1, 'X-Sensor': boardid }
   #requests.post('https://api.sensor.community/v1/push-sensor-data/', json=data, headers = headers)
   data = { "software_version": "<WHAT YOU WANT>",
           "sensordatavalues":[
               {"value_type":"SDS_P1","value":d['SDS_P1']},
               {"value_type":"SDS_P2","value":d['SDS_P1']}]}
   #requests.post('https://api-rrd.madavi.de/data.php', json=data, headers=headers)
   #
   headers = { 'X-Pin': 11, 'X-Sensor': boardid }
   data = { "software_version": "<WHAT YOU WANT>",
           "sensordatavalues":[
               {"value_type":"temperature","value":d['BME280_temperature']},
               {"value_type":"pressure","value":d['BME280_pressure']},
               {"value_type":"humidity","value":d['BME280_humidity']}]}
   #requests.post('https://api.sensor.community/v1/push-sensor-data/', json=data, headers = headers)
   data = { "software_version": "<WHAT YOU WANT>",
           "sensordatavalues":[
               {"value_type":"BME280_temperature","value":d['BME280_temperature']},
               {"value_type":"BME280_pressure","value":d['BME280_pressure']},
               {"value_type":"BME280_humidity","value":d['BME280_humidity']}]}
   #requests.post('https://api-rrd.madavi.de/data.php', json=data, headers=headers)

# Create an InfluxDB client
client = InfluxDBClient(database="sensors")

# Monitor new values in a loop
try:
    while True:
        rs = client.query('select * from feinstaub ORDER BY time DESC LIMIT 1')
        last = list(rs.get_points())[0]
        push(last)
        time.sleep(10)  # Adjust the sleep time as needed
except KeyboardInterrupt:
    print("Monitoring stopped.")

# Close the client
client.close()

pjg · March 7, 2025, 7:18am

the “values” in the json payload, do they have to be strings, or can they be floats?
String mit . in “”

the software_version, don’t really know, is N/A, null, or something else acceptable?
You can put what you want

the boardid, would that be ''esp8266-1234567" or only the numbered part “1234567”?
@ricki-z I can’t remember…

Tanja · March 7, 2025, 2:26pm

looks like it is ‘eps8266-1234567’ looking at the source:

./airrohr-firmware/airrohr-firmware.ino:                http.addHeader(F("X-Sensor"), String(F(SENSOR_BASENAME)) + esp_chipid);

Tanja · March 7, 2025, 3:44pm

so got this running, and on api-rrd.madavi i see data being processed.

thank you for all the help. if anyone needs it, this is my final script

#!/usr/bin/env python

from influxdb import InfluxDBClient
import time, requests, datetime, json

boardid = 'esp8266-1234567'

lastseen=None
def push(d):
   global lastseen
   # [{'time': '2025-03-07T02:00:54.099740Z', 'BME280_humidity': 73.43, 'BME280_pressure': 101565.53, 'BME280_temperature': 6.63, 'SDS_P1': 13.2, 'SDS_P2': 8.0, 'interval': 10000.0, 'max_micro': 1137.0, 'min_micro': 23.0, 'node': 'esp8266-1234567', 'samples': 416113.0, 'signal': -93.0}]
   if d['node'] != boardid: return
   ts = datetime.datetime.fromisoformat(d['time'][:-1]).replace(tzinfo=datetime.timezone.utc)
   if ts == lastseen: return
   now = datetime.datetime.now(datetime.timezone.utc)
   if(ts<now-datetime.timedelta(seconds=15)):
       print("last record is too old:", now-ts)
       return
   print(d)
   lastseen = ts

   data = { "software_version": "null",
           "sensordatavalues":[
               {"value_type":"P1","value":str(d['SDS_P1'])},
               {"value_type":"P2","value":str(d['SDS_P2'])}]}
   headers = { 'X-Pin': "1", 'X-Sensor': boardid }
   requests.post('https://api.sensor.community/v1/push-sensor-data/', json=data, headers = headers)
   #print(json.dumps(data), json.dumps(headers))
   data = { "software_version": "null",
           "sensordatavalues":[
               {"value_type":"SDS_P1","value":str(d['SDS_P1'])},
               {"value_type":"SDS_P2","value":str(d['SDS_P1'])}]}
   requests.post('https://api-rrd.madavi.de/data.php', json=data, headers=headers)
   #print(json.dumps(data), json.dumps(headers))
   headers = { 'X-Pin': "11", 'X-Sensor': boardid }
   data = { "software_version": "null",
           "sensordatavalues":[
               {"value_type":"temperature","value":str(d['BME280_temperature'])},
               {"value_type":"pressure","value":str(d['BME280_pressure'])},
               {"value_type":"humidity","value":str(d['BME280_humidity'])}]}
   requests.post('https://api.sensor.community/v1/push-sensor-data/', json=data, headers = headers)
   #print(json.dumps(data), json.dumps(headers))
   data = { "software_version": "null",
           "sensordatavalues":[
               {"value_type":"BME280_temperature","value":str(d['BME280_temperature'])},
               {"value_type":"BME280_pressure","value":str(d['BME280_pressure'])},
               {"value_type":"BME280_humidity","value":str(d['BME280_humidity'])}]}
   requests.post('https://api-rrd.madavi.de/data.php', json=data, headers=headers)
   #print(json.dumps(data), json.dumps(headers))

# Create an InfluxDB client
client = InfluxDBClient(database="sensors")

# Monitor new values in a loop
try:
    while True:
        rs = client.query('select * from feinstaub ORDER BY time DESC LIMIT 1')
        last = list(rs.get_points())[0]
        push(last)
        time.sleep(4)  # Adjust the sleep time as needed
except KeyboardInterrupt:
    print("Monitoring stopped.")

# Close the client
client.close()