How Best to Download Climate Data?

I’ll be working on gathering this data for the Dakotas (bounding box: 49 N, 42 S, -96 E, -105 W), and then summarizing it to the level of sample blocks they created earlier in the project. I used NetcdfSubset but the request is too big for their on-line retrieval system. Instead, they suggested I use nccopy with the URI they generate. This is the URI of the climate data I need, broken up for readability in this blog post (i.e. I put the new lines there, and they shouldn’t be in the real URI you pass to nccopy):

http://cida.usgs.gov/thredds/dodsC/dcp/conus_t?lon[163:1:230],time[0:1:51099],lat[137:1:192],
ccsm-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230]

To get the file(s) you pass…

nccopy -u <URI> whatever.nc

I copied the big URI (broken up to separate lines as above) and pasted it into a text file called “climate_structure.” Then, I deleted the first line, and got the names of the climate models into a new file, “climate_models” like so:

cat climate_structure | cut -d [ -f1 > climate_models

Then, I setup a simple bash script to loop over the related models and variables now in “climate_models” and download each into a separate *.nc file, and give each the name of the model and variable combination.

#!/bin/bash
#loop over list of climate model names

while read climate_model; do
 echo "Now downloading $climate_model"
 nccopy -u "http://cida.usgs.gov/thredds/dodsC/dcp/conus_t?lon[163:1:230],time[0:1:51099],lat[137:1:192],${climate_model}[0:1:51099][137:1:192][163:1:230]" ${climate_model}.nc
done < climate_models

Next I’ll need to get the precipitation.

My friend suggested instead I try GrADS to handle the data request, and had many friends that used it successfully and often. It can be scripted so I’m looking into that now! In the meantime, please comment if you’ve had any experiences with big climate data and gathering lots of NetCDF files!

Leave a Reply Cancel reply