Accessing netcdf files from a URL linkwith ncdf4 in R
Question
Are there any workarounds to the problem with ncdf4::nc_open not being able to access some .nc files from a URL? I would like to avoid having to download the file first since this is for a shiny app deployed on a server and so I want to avoid users being able to download files to the server.
Some URLs work eg this OPeNDAP URL from a THREDDS server:
library(ncdf4)
nc <- nc_open("https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc")
But others do not eg this NetCDF Subset Service URL from a THREDDS server:
nc <- nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1")
# Error in nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1") :
# Error in nc_open trying to open file https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1
or this file directly from a website:
nc <- nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc")
# syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
# context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p></body></html>
# Error in R_nc4_open: NetCDF: file not found
# Error in nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc") :
# Error in nc_open trying to open file https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc
What could be the reason why some methods work and some don't, and are there any ways to fix this?
1 answers
solution1
1 2021-12-15 17:46:21
I ran into the same issue and contacted the creator of ncdf4 . Here is his reply:
The ncdf4 R package is an interface to the underlying netcdf library on your machine. The R package cannot avoid limitations or problems with the netcdf library itself. You can always see how the underlying netcdf library handles a URL by just trying to use "ncdump" with the URL. For example,
% ncdump "https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc"
Gives:
netcdf Test_pixel_count {
dimensions:
lat = 283 ;
lon = 451 ;
variables:
byte crs ;
crs:_Unsigned = "false" ;
crs:grid_mapping_name = "latitude_longitude" ;
crs:long_name = "CRS definition" ;
crs:longitude_of_prime_meridian = 0. ;
crs:semi_major_axis = 6378137. ;
crs:inverse_flattening = 298.257222101 ;
crs:spatial_ref = "GEOGCS[\"GEOCENTRIC DATUM of AUSTRALIA\",DATUM[\"GDA94\",SPHEROID[\"GRS80\",6378137,298.257222101]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433],AXIS[\"Longitude\",EAST],AXIS[\"Latitude\",NORTH]]" ;
crs:GeoTransform = "141.4607205456306 0.0075 0 -22.95189554231917 0 -0.0075 " ;
etc. Ie, the netcdf library itself can read this URL/file.
However when I do this:
% ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"
I get this error:
ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"
ncdump: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1:
NetCDF: Malformed or unexpected Constraint
So the netcdf library itself cannot access that URL/file, and therefore ncdf4 cannot do so. There's not anything I would be able to do to the ncdf4 library to make up for this.
You might try contacting the netcdf library folks and ask if this is expected behavior or a bug.
solution2
-2 2021-12-09 13:09:33
I'm having the same issue and would be very interested to find a workaround, or at least better understand what is causing this. Calling the nc_open function in debug mode just reveals that the error occurs here:
.C("R_nc4_open", as.character(filename), as.integer(rv$cmode),
id = as.integer(rv$id), error = as.integer(rv$error),
PACKAGE = "ncdf4")
So, the function seems to compile some C code and debugging that is beyond me...
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.