在文件夹中的多个 .txt 文件上应用 R 脚本

Question

I am extremely new to building functions and loops. I have looked at previous questions that are similar to my issue but I can't seem to find the solution for my problem. My goal is to extract climate data from a webpage like this:

https://mesonet.agron.iastate.edu/cgi-bin/request/coop.py?network=NECLIMATE&stations=NE3065&year1=2020&month1=1&day1=1&year2=2020&month2=12&day2=31&vars%5B%5D=gdd_50_86&model=apsim&what=view&delim=comma&gis=no&scenario_year=2019

where I will use this data to calculate growing degree days for a crop growth model. I have had success pulling data using a for loop.

uticaNE <- "https://mesonet.agron.iastate.edu/cgi-bin/request/coop.py?network=NECLIMATE&stations=NE8745&year1=2020&month1=1&day1=1&year2=2020&month2=12&day2=31&vars%5B%5D=gdd_50_86&model=apsim&what=view&delim=comma&gis=no&scenario_year=2019"

friendNE <- "https://mesonet.agron.iastate.edu/cgi-bin/request/coop.py?network=NECLIMATE&stations=NE3065&year1=2020&month1=1&day1=1&year2=2020&month2=12&day2=31&vars%5B%5D=gdd_50_86&model=apsim&what=view&delim=comma&gis=no&scenario_year=2019"

location.urls <- c(uticaNE, friendNE)
location.meso.files <- c("uticaNe.txt", "friendNE.txt")

for(i in seq_along(location.urls)){
  download.file(location.urls[i], location.meso.files[i], method="libcurl")
} 

I will have around 20 locations I will be pulling data in daily. What I want to do is apply a task where I calculate fahrenheit, GDD, etc. to each file and save the output of each file separately.

This is the following code I have currently.

files <- list.files(pattern="*.txt", full.names=TRUE, recursive=FALSE)

  func <- for (i in 1:length(files)){
  df <- read.table(files[i], skip=10, stringsAsFactors = 
  FALSE)
  colnames(df) <- c("year", "day", "solrad", "maxC", 
  "minC", "precipmm")
  df$year <- as.f(df$year)
  df$day <- as.factor(df$day)
  df$maxF <- (df$maxC * (9/5) + 32)
  df$minF <- (df$minC * (9/5) + 32)
  df$GDD <- (((df$maxF + df$minF)/2)-50)
  df$GDD[df$GDD <= 0] <- 0
  df$GDD.cumulateive <- cumsum(df$GDD)
  df$precipmm.cumulative <- cumsum(df$precipmm)
  return(df)
  write.table(df, path="./output", quote=FALSE, 
  row.names=FALSE, col.names=TRUE)
}

data <- apply(files, func)

Any help would be greatly appreciated.

-ML

Answer 1

Instead of using base R which ,you can install tidyverse library. https://www.tidyverse.org/ In which you can use load the link into data frame as tsv(tab separated value) using read_tsv function.

dataframe<-read_tsv(url("http://some.where.net/"))

Then create a loop in R and do calculations

something<-c('link1','link2') #vector in R
for(i in someting){
 #make sure to indent with one space
}

At the end, you save data frame to a file using

write_csv(dataframe, file = "c:\\myname\\yourfile.csv")

Answer 2

Here is an approach using base R, and lapply() with an anonymous function to download the data, read it into a data frame, add the conversions to fahrenheit and cumulative precipitation, and write to output files.

First, we create the list of weather stations for which we will download data

# list of 10 stations
stationList <- c("NE3065","NE8745","NE0030","NE0050","NE0130",
                 "NE0245","NE0320","NE0355","NE0375","NE0420")

Here we create two URL fragments, one for the URL content prior to the station identifier, and another one for the URL content after the station identifier.

urlFragment1 <- "https://mesonet.agron.iastate.edu/cgi-bin/request/coop.py?network=NECLIMATE&stations="
urlFragment2 <- "&year1=2020&month1=1&day1=1&year2=2020&month2=12&day2=31&vars%5B%5D=gdd_50_86&model=apsim&what=view&delim=comma&gis=no&scenario_year"

Next, we create input and output directories, one to store the downloaded climate input files, and another for the output files.

# create input and output file directories if they do not already exist 
if(!dir.exists("./data")) dir.create("./data")
if(!dir.exists("./data/output")) dir.create("./data/output")

The lapply() function uses paste0() to add the station names to the URL fragments we created above, enabling us to automate the download and subsequent operations against each input file.

stationData <- lapply(stationList,function(x){
     theURL <-paste0(urlFragment1,x,urlFragment2)
     download.file(theURL,
                   paste0("./data/",x,".txt"),method="libcurl")
     df <- read.table(paste0("./data/",x,".txt"), skip=11, stringsAsFactors = 
                           FALSE)
     colnames(df) <- c("year", "day", "solrad", "maxC", 
                       "minC", "precipmm")
     df$year <- as.factor(df$year)
     df$day <- as.factor(df$day)
     df$maxF <- (df$maxC * (9/5) + 32)
     df$minF <- (df$minC * (9/5) + 32)
     df$GDD <- (((df$maxF + df$minF)/2)-50)
     df$GDD[df$GDD <= 0] <- 0
     df$GDD.cumulative <- cumsum(df$GDD)
     df$precipmm.cumulative <- cumsum(df$precipmm)
     df$station <- x
     write.table(df,file=paste0("./data/output/",x,".txt"), quote=FALSE, 
                 row.names=FALSE, col.names=TRUE)
     df
})
# add names to the data frames returned by lapply()
names(stationData) <- stationList

...and the output, a directory containing one file for each station listed in the stationList object.

Finally, here is the data that has been written to the ./data/output/NE3065.txt file.

year day solrad maxC minC precipmm maxF minF GDD GDD.cumulateive precipmm.cumulative station
2020 1 8.992 2.2 -5 0 35.96 23 0 0 0 NE3065
2020 2 9.604 5.6 -3.9 0 42.08 24.98 0 0 0 NE3065
2020 3 4.933 5.6 -3.9 0 42.08 24.98 0 0 0 NE3065
2020 4 8.699 3.9 -7.2 0 39.02 19.04 0 0 0 NE3065
2020 5 9.859 6.1 -7.8 0 42.98 17.96 0 0 0 NE3065
2020 6 10.137 7.2 -5 0 44.96 23 0 0 0 NE3065
2020 7 8.754 6.1 -4.4 0 42.98 24.08 0 0 0 NE3065
2020 8 10.121 7.8 -5 0 46.04 23 0 0 0 NE3065
2020 9 9.953 7.2 -5 0 44.96 23 0 0 0 NE3065
2020 10 8.905 7.2 -5 0 44.96 23 0 0 0 NE3065
2020 11 0.416 -3.9 -15.6 2.29 24.98 3.92 0 0 2.29 NE3065
2020 12 10.694 -4.4 -16.1 0 24.08 3.02 0 0 2.29 NE3065
2020 13 1.896 -4.4 -11.1 0.51 24.08 12.02 0 0 2.8 NE3065
2020 14 0.851 0 -7.8 0 32 17.96 0 0 2.8 NE3065
2020 15 11.043 -1.1 -8.9 0 30.02 15.98 0 0 2.8 NE3065
2020 16 10.144 -2.8 -17.2 0 26.96 1.04 0 0 2.8 NE3065
2020 17 10.75 -5.6 -17.2 3.05 21.92 1.04 0 0 5.85 NE3065

Note that there are 11 rows of header data in the input files, so one must set the skip= argument in read.table() to 11, not 10 as was used in the OP.

Enhancing the code

The last line in the anonymous function returns the data frame to the parent environment, resulting in a list of 10 data frames stored in the stationData object. Since we assigned the station name to a column in each data frame, we can combine the data frames into a single data frame for subsequent analysis, using do.call() with rbind() as follows.

combinedData <- do.call(rbind,stationData)

Since this code was run on January 17th, the resulting data frame contains 170 observations, or 17 observations for each of the 10 stations whose data we downloaded.

At this point the data can be analyzed by station, such as finding the average year to date precipitation by station.

> aggregate(precipmm ~ station,combinedData,mean)
   station   precipmm
1   NE0030 0.01470588
2   NE0050 0.56764706
3   NE0130 0.32882353
4   NE0245 0.25411765
5   NE0320 0.28411765
6   NE0355 1.49411765
7   NE0375 0.55235294
8   NE0420 0.13411765
9   NE3065 0.34411765
10  NE8745 0.47823529
>