Link to sensor data for ESP32

FAQ
1

I’ve got an airRohr sensor.

For my website i retrieve my data as follows…
$jsonurl = “http://data.sensor.community/airrohr/v1/sensor/”. $tempid."/";
$json = file_get_contents($jsonurl);
where $tempid is my sensorid. This works fine.

Now I want this data on my ESP32 too:

I used an existing example for weatherdata
// String serverPath = “http://api.openweathermap.org/data/2.5/weather?q=” + city + “,” + countryCode + “&APPID=” + openWeatherMapApiKey;

I received the json-file form openweather correctly.

So I changed the call to
String serverPath = “http://data.sensor.community/airrohr/v1/sensor/53656/”;

but now I receive an undocumented error:: HTTP Error code: -11

The call works correct when called in a browser.

So mine ESP32 can receive json-files, I can get a json-file with my sensor data from a browser and with my PHP-code.

I assume something wrong with the serverPath, but I can’t figure out what the error is,

2

I Check a code of mine soon.

3

There is another thing I discovered:
data.sensor.community changes to https even if you specify only http, when I retrieve my data from my browser Chrome of Firefox. ( is that correct for ‘open data’?)

maybe that the cause of my trouble. That should implicate that my site “veldbies.nl” is considered a trusted party, whereas my ESP8266 is not considered as a trusted party.

That should explain a lot.

4

Yes, I have just seen this:

static const char URL_API_SENSORCOMMUNITY[] PROGMEM = "https://data.sensor.community/airrohr/v1/sensor/";

then:

strcpy(url_ok, URL_API_SENSORCOMMUNITY);
strcat(url_ok, sensorNr);
strcat(url_ok, "/");

then:

float getData(const char *url, unsigned int pm_type)
{

  String pm="";

  switch (pm_type)
  {
  case 0:
    pm = "P0";
    break;
  case 1:
    pm = "P1";
    break;
  case 2:
    pm = "P2";
    break;
  }

  String reponseAPI;
  DynamicJsonDocument doc(JSON_BUFFER_SIZE);
  char reponseJSON[JSON_BUFFER_SIZE];
  if (!client.connect("data.sensor.community", httpPort))
  {
    Debug.println("connection failed");
    display.setSegments(oups);
    delay(1000);
    doc.garbageCollect();
    return -1.0;
  }

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + "data.sensor.community" + "\r\n" +
               "Connection: close\r\n\r\n");

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + "data.sensor.community" + "\r\n" +
               "Connection: close\r\n\r\n");
  unsigned long timeout = millis();
  while (client.available() == 0)
  {
    if (millis() - timeout > 30000) //30s call to be sure the API answers
    {
      Debug.println("Client Timeout !");
      //client.flush();
      client.stop();
      display.setSegments(oups);
      delay(1000);
      doc.garbageCollect();
      return -1.0;
    }
  }
  // Read all the lines of the reply from server and print them to Serial
  while (client.available())
  {
    reponseAPI = client.readStringUntil('\r');
    strcpy(reponseJSON, reponseAPI.c_str());
    DeserializationError error = deserializeJson(doc, reponseJSON);
      if(strcmp (error.c_str(),"Ok") == 0) //revérifier
      {
        JsonArray JSONarray = doc.as<JsonArray>();
        serializeJsonPretty(doc, Debug);
        Debug.println("");
        JsonArray::iterator it = JSONarray.begin();
        JsonObject object = it->as<JsonObject>();
        JsonObject sensorAPI = object["sensor"].as<JsonObject>();
        serializeJsonPretty(sensorAPI, Debug);
        Debug.println("");
        for (JsonObject sensor : sensors_list)
        {
          if (sensorAPI["id"].as<String>() == sensor["sensor"])
          {
            sensor["time"] = object["timestamp"].as<String>();
            break;
          }
        }

        JsonArray dataarray = object["sensordatavalues"].as<JsonArray>();

        for (JsonObject dataobj : dataarray)
        {
          if (dataobj["value_type"].as<String>() == pm)
          {
            Debug.print("PM: ");
            Debug.println(dataobj["value"].as<String>());
            client.stop(); //OBLIGATOIRE?
            for (JsonObject sensor : sensors_list)
            {
              if (sensorAPI["id"].as<String>() == sensor["sensor"])
              {
                sensor["value"] = dataobj["value"].as<String>();
                break;
              }
            }
          //  client.flush();
          //  client.stop();
            doc.garbageCollect(); //empty buffer of JSONdocument
            //sensors_json.garbageCollect(); //empty buffer of JSONdocument
            return dataobj["value"].as<String>().toFloat();
          }
        }

        if (pm_type == 0)
        {
          display.setSegments(nop1);
          delay(1000);
          //client.flush();
         // client.stop();
          doc.garbageCollect();
          return -1.0;
        }
    }
    else
    {
      Debug.print(F("deserializeJson() failed: "));
      Debug.println(error.c_str());
    }
  }
  display.setSegments(oups);
  delay(1000);
  //client.flush();
  //client.stop();
  doc.garbageCollect();
  return -1.0;
}

But actually my code works on an NodeMCU 0.9.

That works perfectly in my PM Lamp.

5

The definition of open data is about the data itself, not about the way you get it … But beside this, I have checked the server configs and none of the data servers should redirect http to https.
Some browsers try to switch to the https version of a page if the server is answering on port 443. And they will also save the information that https is possible for a site if you called one time this way.

6

Thx for your reply and parts of your code.

Is it correct you use “https” and not “http” in your URL?

I didnt specify a httPort and you do, which port do you use?

As I understand from rick-z you can use “http” as well as “https” in the URL. May be I should use a specific port.

7

Yes, I programmed like this.

const int httpPort = 80;

I use these libraries as in the main firmware:

#include <ESP8266HTTPClient.h>
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>
8

This is part of the reply my ESP32 received:

Apache Server at srv02.sensor.community Port 80

Parsing input failed!
HTTP Response code: 400
jsonbuffer

400 Bad Request

Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

I used port 80 standard, but it seems you want me to speak HTTPS. At least that’s what it’s looks like.

9

I’m using the same system to read data for my display.

I’m calling “http://data.sensor.community/airrohr/v1/sensor/86564/

Normally the response is the Json message containing the sensor data.
But from time to time i get a “” answer, hours on end, while my sensor is still active, and can be seen with Grafana, and on the map.
I think the problem arises due to too frequent polling. After a while the server starts to respond with error code 500, and only recovers a day or more later.
Only the sensor that was frequently polled is locked out. Other sensors respond with data.
Which are the rules for polling this url with an API-ID? What is the frequency that can safely be used, without being locked out, as that seems to be the problem ?

10

There is no automatic blocking based on the frequency of access. And you would get a “403 Forbidden” error if your IP is blocked. But this blocking is done manually and only for IPs with more than 10 requests per minute.
The data file contains the data of the last 5 minutes. Most sensors will send new data every 2 1/2 minutes. So it should be okay to request new data every 2-3 minutes.

The problem: Most computers are time synced. And so most requests are done at the start of each minute. This may cause time outs or empty results for some requests.

11

Specifically the API 86564 & 86565 are blocked now, as i’m testing my display with my own brand new outdoor sensor with SPS30/BME280. Can you check & release the blocking ? I’m normally accessing the data once per 5 minutes, but the frequency can be higher at certain moments during testing, when a new beta is loaded …
The blocking also happened once while giving a demo with 10 displays, on all 10 simultaniously, showing different sensors, embarrasing … It took a full day to recover then.
Since then a limit was build in, only reading the data once per 5 minutes.
But as I already told you this might shorter during testing …
Right now my indoor sensor’s reading at API 27786 & 27787 (an old one, with SDS011/DHT22) is not blocked, while my outdoor sensor’s reading at API 86564 & 86565 is blocked, while they are accessed at the same frequency ? Both sensors produce data …
How can a user ununlock a sensor in blocked state ? Not accessing it for how many hours, or what else?

12

Getting empty results hours on end, while testing software, is very decouraging, and hampers software development. How to go around these empty Json results (), or get out of that locked situation ? My only SPS30/BME280 sensor (86564/86565) is at this moment producing these empty results, even after not being in use for the last 2 hours. I need this special sensor with SPS30 since its Json message contains unique PM04, PM01 & pressure results, and precisely this modified Json decoding logic is in test. The situation might be self-locking, as receiving empty messages as a response to frequent readings, might automatically enforce new, more frequent readings, to desperately get the needed data.

13

THERE IS NO BLOCKING if you get an empty result array!
If an IP address is blocked you will get a HTTP answer “403 - Forbidden”.
If your sensor is “not in use” (i think that means it’s not sending data), then the result for the last 5 minutes is an empty array. Or what should we send as a valid JSON result in this case?
And I will correct my last answer. If there is too much load on the server then it should respond with an error “500 - Internal server error”.

14

It looks like my recent problems were due to a faulty WiFi router - an Orange Flybox 4G+ that i took out of service just now, and replaced with a hotspot on my PC with same SSID & password. “Magically” the Json message is back.
But since you mention the “500 - internal server error” i’ve seen this a lot too the last two months, and when that error comes, it won’t go away for ar least a day, even without sending requests. Is that normal ?
Anyway my software debugging can continue unhampered. Thanks for the suggestions.

15

If you mean the “500 - internal server error” on the status page of the sensor itself then ist is the last error(!) code. It’s not the result of the last transmission. The other numbers are error counts, that you can compare to the number of measurements.

16

The 500 - server error is not on the status page of the sensor, but is the error response on the failed requests by my display. A totally empty Json message is the result, not even brackets. This error response after data request, when received, won’t go away for the next thousands of requests. Requests have to be halted for more than a day, to restore the servers ability to respond normally. That is what i mean. But this was 10 weeks ago, during the demo, and my faulty router now made me think “there it is again”.