R-Script or tips to download and use sensor data

[cmbbiggs]

@pjg

list.of.packages <- c("lubridate","tidyverse", "tidyr","RCurl", "httr","readr","ggplot2","stringi")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)

library(lubridate)
library(tidyr)
library(tidyverse)
library(RCurl)
library(httr)
library(readr)
library(stringi)
library(ggplot2)
set_config(config(ssl_verifypeer = 0L))

setwd("/Users/PJ/CODES/R_VGA")

url_dir<-"http://archive.sensor.community" 
inicio <- "2023-02-01"
fin <- "2023-07-28"

#sensor_id=c("76689", "76699", "76701", "76703", "76735", "76737", "77531", "77535", "80342")
#sensor_type=c("sds011", "sds011", "sds011", "sds011", "sds011", "sds011", "sds011", "sds011", "sds011")

sensor_id=c("76560", "76685")
sensor_type=c("sds011", "sds011")


inicio_date <- ymd(inicio)
actual<-inicio_date
fin_date <- ymd(fin)

dates <- seq(inicio_date, by = "day", length.out = (fin_date-inicio_date)+1)
print(dates)

url_test<-paste(url_dir,"/",actual,"/",actual,"_",sensor_type[1],"_","sensor","_",sensor_id[1],".csv",sep="")
print(url_test)

x<-GET(url_test)
http_status(x)
if(x$status_code==200){
  print(url_test[1])
  luft_data_fin <- read_delim(url(url_test[1]),delim=";")
  print("ok")
}else{
  print("error")
}

print(luft_data_fin)

luft_data_fin <- luft_data_fin[0,]
print(luft_data_fin)

for (i in 1:length(sensor_id)){
  print(sensor_id[i])
  for (j in 1:length(dates)) {
    print(dates[[j]])
    print(sensor_id[i])
    url_day<-paste(url_dir,"/",dates[[j]],"/",dates[[j]],"_",sensor_type[i],"_","sensor","_",sensor_id[i],".csv",sep="")
    print(url_day)
    x<-GET(url_day)
    http_status(x)
    if(x$status_code==200){
      luft_data <- read_delim(url(url_day[1]),delim=";")
      luft_data_fin<-rbind(luft_data_fin,luft_data)
    }else {
      print("error")
      print(dates[[j]])
    }
  }
}


luft_data_fin <- luft_data_fin[,-c(2,3,4,5,8,9,11,12)]

write_csv(luft_data_fin,file=paste("datasensor_suite2.csv"),na="NA",append=FALSE,col_names=TRUE)

Thanks so much, that worked perfectly. But when I tried to use the code for sensors local to me, I think the sensor type must have been different to yours. How do I find sensor type? When I click on a sensor on the main map I see only the sensor ID not the sensor type.

Tina

[pjg]

Please give me the sensor ID and I will show you how to find out the sensor type.

[cmbbiggs]

Sure - it’s #78434 and #78947 - they re both in Redland in Bristol.

Tina

[pjg]

On this page:
https://archive.sensor.community/2023-10-02/

2023-10-02_pms5003_sensor_78434.csv
2023-10-02_sds011_sensor_78947.csv

78947 is sds011
78434 is pms5003

Just set this in the code. It should work

[cmbbiggs]

@pjg

2023-10-02_pms5003_sensor_78434.csv
2023-10-02_sds011_sensor_78947.csv

78947 is sds011
78434 is pms5003

Thanks - so I have inserted my own values for sensor id and sensor type (see below) but I only get as far as http_status(x) when it returns “Clent error (404): Not found.”

Code to import from Sensor Community

list.of.packages ← c(“lubridate”,“tidyverse”, “tidyr”,“RCurl”, “httr”,“readr”,“ggplot2”,“stringi”)
new.packages ← list.of.packages[!(list.of.packages %in% installed.packages()[,“Package”])]
if(length(new.packages)) install.packages(new.packages)

library(lubridate)
library(tidyr)
library(tidyverse)
library(RCurl)
library(httr)
library(readr)
library(stringi)
library(ggplot2)
set_config(config(ssl_verifypeer = 0L))

url_dir<-“http://archive.sensor.community”
inicio ← “2023-02-01”
fin ← “2023-07-28”

#sensor_id=c(“78434”, “78947”)
#sensor_type=c(“sds011”, “sds011”)

sensor_id=c(“78434”, “78947”)
sensor_type=c(“pms5003”, “sds011”)

inicio_date ← ymd(inicio)
actual<-inicio_date
fin_date ← ymd(fin)

dates ← seq(inicio_date, by = “day”, length.out = (fin_date-inicio_date)+1)
print(dates)

url_test<-paste(url_dir,“/”,actual,“/”,actual,““,sensor_type[1],””,“sensor”,“_”,sensor_id[1],“.csv”,sep=“”)
print(url_test)

x<-GET(url_test)
http_status(x)

[pjg]

For all the dates ?
I don’t understand.

But there are not a lot of PMS5003. Use the csvgrep method.

[cmbbiggs]

Sorry, yes, you are right. I was running it line by line and stopped at the first error message! It’s now running smoothly - thanks so much for your help.

Tina