MN Ornithologists’ Union Young Birders’ Committee

I’m pleased to have been asked to serve on the MOU Young Birders’ Committee! We’re generously defining “young” as 18-40, but not without precedent (Erikson 1963). Our overarching goal is to improve participation of those of us in this age group in conservation-related organizations, which in our specific case is MOU. Conveniently, my friend who chairs this committee did his thesis on the topic! He studied how young birder participation in clubs differs from that of previous generations, and I resonate with his core findings. With this growing demographic in birding, it will be important to address the needs of young birders within organizations in order to fully engage the community of interest.

How to Handle Climate NetCDF Files

I’ve started working with my climate NetCDF’s in CDO because there’s a pretty easy function that does exactly what I need for my 1st step: it averages a daily time step file into a monthly averaged (or whatever metric you choose) file. So this time, now that I have the *.nc files, I got a list of the base file names as such:

cat climate_structure | cut -d . -f1 > climate_models

Then  I wrote this simple bash script to loop over them all and write out the monthly averages:

#!/bin/bash
while read climate_model; do
 echo "Now averaging $climate_model"
 cdo monmean ${climate_model}.nc ${climate_model}_monthly.nc
done < climate_models

Some of my closest colleagues, Brooke Bateman and Andy Allstadt, had some good advice for how to work with the netCDF files in R once I get them:

  • ncdf4 package
  • raster package: can open netCDF either as…
    • single layer (with the raster function)
    • the entire thing (brick function)
  • SDMTools
    • sample x,y points from raster using extract data function
    • convert the raster to ASCII

NetCDF: Wading through the Wreckage

There were apparently a few people last week (and maybe this week) who were downloading tons of data, so it was “clogging the pipe” on our connection as our IT people put it. However, a few issues still remain, but they may not be of high importance to us…

Now downloading cnrm-a1b-tmin-NAm-grid
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>500 Proxy Error</title></head><body><h1>Proxy Error</h1>The proxy server could not handle the request <em><a href="/thredds/dodsC/dcp/conus_t.dods">GET&nbsp;/thredds/dodsC/dcp/conus_t.dods</a></em>.<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /></body></html>
NetCDF: DAP server error
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1044
Now downloading cnrm-a2-tmax-NAm-grid
Now downloading cnrm-a2-tmin-NAm-grid
NetCDF: DAP server error
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1044
Now downloading cnrm-b1-tmax-NAm-grid
syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
NetCDF: DAP server error
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1355
Now downloading cnrm-b1-tmin-NAm-grid
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>500 Proxy Error</title></head><body><h1>Proxy Error</h1>The proxy server could not handle the request <em><a href="/thredds/dodsC/dcp/conus_t.dds">GET&nbsp;/thredds/dodsC/dcp/conus_t.dds</a></em>.<p>Reason: <strong>Error during SSL Handshake with remote server</strong></p><p /></body></html>
NetCDF: DAP server error
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1355
...
Now downloading echam5-a1b-tmin-NAm-grid
CURL Error: Couldn't connect to server
curl error details: 
NetCDF: I/O failure
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1044

How Best to Download Climate Data?

I’ll be working on gathering this data for the Dakotas (bounding box: 49 N, 42 S, -96 E, -105 W), and then summarizing it to the level of sample blocks they created earlier in the project. I tried using OPeNDAP but I’m not quite sure how to use it. So, I tried NetcdfSubset instead, which seems like it would do what I want, but the request is too big for their on-line retrieval system. Instead, they suggested I use nccopy.

This is the giant URI of the climate data I need, broken up for readability in this blog post. In other words, I put the new lines there, and they shouldn’t be in the real URI you pass to nccopy.

http://cida.usgs.gov/thredds/dodsC/dcp/conus_t?lon[163:1:230],time[0:1:51099],lat[137:1:192],
ccsm-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
ccsm-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t47-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cgcm3_t63-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
cnrm-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
csiro-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echam5-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
echo-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-0-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-0-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-0-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-0-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
gfdl_2-1-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
giss_aom-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
giss_aom-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
giss_aom-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
giss_aom-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadcm3-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadgem-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadgem-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadgem-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
hadgem-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_hi-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_hi-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_hi-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_hi-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
miroc_med-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
mri_cgcm2-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1b-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1b-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1fi-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a1fi-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a2-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-a2-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-b1-tmax-NAm-grid[0:1:51099][137:1:192][163:1:230],
pcm-b1-tmin-NAm-grid[0:1:51099][137:1:192][163:1:230]

To get the file(s) you pass…

nccopy -u <URI> whatever.nc

My request is so big that perhaps it comes across a missing file somewhere in the list, if I try to just pass the whole URI? I get this error from nccopy (running v. 4.4.0 on Xenial Xerus):

NetCDF: file not found
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/nccopy.c; line 1355

I’m thinking that probably the best thing to do will be to script this request, so it handles each URI separately. The friendly data portal manager has been much help, in that it looks like from his example to write a standalone URI for each model, it would be something of the form (an example of the first dataset):

So here’s what I did: I copied the big URI broken up and pasted it into a text file called “climate_structure.” Then, I deleted the first line, and got the names of the climate models into a new file, “climate_models” like so:
cat climate_structure | cut -d [ -f1 > climate_models

Then, I setup a simple bash script to loop over the related models and variables now in “climate_models” and download each into a separate *.nc file, and give each the name of the model and variable combination.

#!/bin/bash
#loop over list of climate model names

while read climate_model; do
 echo "Now downloading $climate_model"
 nccopy -u "http://cida.usgs.gov/thredds/dodsC/dcp/conus_t?lon[163:1:230],time[0:1:51099],lat[137:1:192],${climate_model}[0:1:51099][137:1:192][163:1:230]" ${climate_model}.nc
done < climate_models

Next I’ll need to get the precipitation.

My friend suggested instead I try GrADS to handle the data request, and had many friends that used it successfully and often. It can be scripted so I’m looking into that now! In the meantime, please comment if you’ve had any experiences with big climate data and gathering lots of NetCDF files!

NSF Macro-systems Grant Meeting

We had a meeting here today of some of the co-PI’s from other universities, so it was nice to finally meet some of the people who have been involved with the grant. Besides our WETLANDSCAPE group, I had never met anyone outside our institute on the grant before today. It helped me get a feel for what others had been doing on the project, and what needs to be done. I also forged some nice new connections, and I look forward to collaborating further to carry out the grant responsibilities. Today we outlined our plans for what needs to be done, in the form of planned papers. We also pinpointed tasks needed to get re-analyses done.