netCDF files in R

Question

I have netCDF file obtained from here with name precip.mon.total.v6.nc. I am using ncdf package in R to open and analyse the file.

 new <- open.ncdf("precip.mon.total.v6.nc")
    > new
    [1] "file precip.mon.total.v6.nc has 4 dimensions:"
    [1] "lat   Size: 360"
    [1] "lon   Size: 720"
    [1] "nbnds   Size: 2"
    [1] "time   Size: 1320"
    [1] "------------------------"
    [1] "file precip.mon.total.v6.nc has 1 variables:"
    [1] "float precip[lon,lat,time]  Longname:GPCC Monthly total of     precipitation Missval:-9.96920996838687e+36"

But when I extract the variable, I got the error

      > get.var.ncdf(new, "precip")
Error: cannot allocate vector of size 2.5 Gb
In addition: Warning messages:
1: In double(totvarsize) :
  Reached total allocation of 2047Mb: see help(memory.size)
2: In double(totvarsize) :
  Reached total allocation of 2047Mb: see help(memory.size)
3: In double(totvarsize) :
  Reached total allocation of 2047Mb: see help(memory.size)
4: In double(totvarsize) :
  Reached total allocation of 2047Mb: see help(memory.size)

My queries are: (a) How to handle memory issue? (b) How can I change the resolution of this netCDF file from 0.5*0.5 to 0.25*0.25 data? I have tried the similar problem in MATLAB. It can tackle memory issue better than R for netCDF files. But changing resolution is still a problem as I am not good at MATLAB. I will be very thankful for any help in this direction.

Answer 1

When you extract your variable, you need to specify which dimensions you want. Currently you're asking R to get everything and so I suspect it's creating a 3D array which will likely be enormous.

The ncdf4 package generally supersedes ncdf, you should try using that instead. You need to decide if you want to read data by location for time or by time step for location. This is easier to envisage on a plain 2D grid:

• Single cell at all time steps
• All locations single time step

Yours is a 3D grid through time (albeit with the 3rd dimension only two bands), however it looks like your variable isn't using the bands dimension. Here's a 2D workflow based on ncdf4, ignoring your bands:

Package:

install.packages("ncdf4")
library(ncdf4)

Open connection:

nc = nc_open("~/dir/dir/file.nc")

For a grid at one time step

Read dimensions:

precip = list()
precip$x = ncvar_get(nc, "lon")
precip$y = ncvar_get(nc, "lat")

Read data (note start is the index in dimensions to begin and count is how many observations from that point, so here we read the whole grid at the first time step):

precip$z = ncvar_get(nc, "precip", start=c(1, 1, 1), count=c(-1, -1, 1))
# Convert to a raster if required
precip.r = raster(precip)

To read a single cell at all time steps

You need to find your cell index, precip$x and precip$y will help. Once you have it (e.g. cell x=5 and y=10):

precip.cell = ncvar_get(nc, "precip", start=c(5, 10, 1), count=c(1, 1, -1))

Answer 2

(a) memory:

If on a linux box [sudo apt-get install cdo] (or also windows with cygwin installed) you can use cdo to help you out.

For example, if you are only interested in a specific date you can select that first to keep the file size down:

cdo seldate,date in.nc out.nc 

or you might have wanted to view the time mean:

cdo timmean in.nc out.nc 

That will keep the file size down, and you can then open it in R to make your plot (or use ncview for a quickview investigation).

(b) remapping

cdo can also interpolate the file to 0.25 degrees, (although I am not sure why you would want to do this, as you are not adding any information and you are making the files four times larger!!!)

cdo remapcon,r1440x720 in.nc out.nc

or

cdo remapnn,r1440x720 in.nc out.nc

But as I said, if you want to interpolate to compare to another 0.25 degree product (e.g. TRMM), better to go the other way and interpolate the finer dataset to 0.5 degree.

By the way, in 2015, v7 was released of GPCC, still at 0.5 degree.