Month: June 2017

How I’ve Been Processing Climate Data

Posted on by Jess

From a THREDDS server, I’m using the NetcdfSubset portal to obtain a spatial subset of my climate data set of interest. Since the files I need are too big to download from their HTTP server, they instead give me an OPeNDAP URI with the spatial subset info. I then pass that to nccopy from the NetCDF library.

Then, I installed CDO and wrote a script to do the monthly means for all the files: it averages a daily time step file into a monthly averaged (or whatever metric you choose) file.

cdo monmean foo_hourly_or_daily.nc  foo_monthly_mean.nc

Then, I installed nco to flip the axes.

ncpdq -a lat,lon in.nc out.nc

Then, I imported the correctly-oriented files into R.

rm(list = ls())
library(raster)
library(rgdal)
library(ncdf4)
setwd("wherever your files are")
climate <- brick("filename.nc",varname="whatever your climate variable is")

In my case, as I think is common for climate files, longitude was on a 0-360 scale, instead of -180/180.

Spatially Manipulating NetCDF Files

Posted on by Jess

I ended up getting a bunch of climate NetCDF files from a colleague for each combination of climate model, climate change scenario and variable. So, what I have is a list of 3-D files consisting of observations/predictions of a given weather variable over latitude, longitude and time (you can picture them as cubes, if you like). I will need to spatially adjust the files, and then subset the data.

ncpdq -a lat,lon in.nc out.nc

I need to…

  • keep only the extent within a shape file we’re using
  • figure out how to summarize it by the polygons in the shape file
    • average?
    • keep only the center point?

Some of my closest colleagues, Brooke Bateman and Andy Allstadt, had some good advice for how to work with the netCDF files in R once I get them:

  • ncdf4 package
  • raster package: can open netCDF either as…
    • single layer (with the raster function)
    • the entire thing (brick function)
  • SDMTools
    • sample x,y points from raster using extract data function
    • convert the raster to ASCII

You can load those libraries, and then do a few things to take a look.

library(<span class="pl-smi">raster</span>)
print(raster(<span class="pl-s"><span class="pl-pds">"</span>afile.nc<span class="pl-pds">"</span></span>))  <span class="pl-c">## same as 'ncdump -h afile.nc</span>
b <- brick(<span class="pl-s"><span class="pl-pds">"</span>afile.nc<span class="pl-pds">"</span></span>, <span class="pl-v">varname</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>pr<span class="pl-pds">"</span></span>) #this is the variable name internal to the file

MN Ornithologists’ Union Young Birders’ Committee

Posted on by Jess

I’m pleased to have been asked to serve on the MOU Young Birders’ Committee! We’re generously defining “young” as 18-40, but not without precedent (Erikson 1963). Our overarching goal is to improve participation of those of us in this age group in conservation-related organizations, which in our specific case is MOU. Conveniently, my friend who chairs this committee did his thesis on the topic! He studied how young birder participation in clubs differs from that of previous generations, and I resonate with his core findings. With this growing demographic in birding, it will be important to address the needs of young birders within organizations in order to fully engage the community of interest.

Getting Started: Learning How to Observe Birds

Posted on by Jess

Meta-learning

As with any skill, there’s a certain element to “learning how to learn” to observe birds. You’ll see what I mean if you invest time in this new hobby. You start to learn what marks to look for and where to look, or how to listen. You pick up how to watch and anticipate movements, and what’s an important feature to try to gather information. Learning how to learn bird song was especially a notable process for me, because I’m not a natural “by ear” learner, and I’d venture to say most aren’t. Those with musical backgrounds might have more developed skills in this arena, but this is where the concept of “meta-learning” really stuck out to me as I tried to learn more about birds. I not only memorized songs (often via mnemonics), but eventually I learned how to listen.

Different Styles

  • Visual: Get a quality field guide
  • Aural: Check out an audio guide to learning how to listen to birds
  • Verbal: Go out on a local field trip led by an expert
  • Physical: Look for animal rehab centers or natural areas in your area that offer demonstrations that will allow you to see birds up close. It will be a rewarding experience if you’re able to attend an event where live birds are captured for research, that you can see in the hand before they’re released!
    • This one is a little harder to give general advice on, because only certain places do this, and often only certain times of the year. Start by searching for bird observatories in your area, and keep up with event calendars for nearby natural areas that, e.g., have welcome centers and related organizations.
  • Logical: consider first learning about birds (e.g. taxonomy) and how the topological characteristics follow
    • Certain field guides are organized by taxonomy (though schemes differ). It may help you to understand how birds are classified, along with perhaps theories of how they evolved to those places, and thus what characteristics belong to each to give you a base of information to which you can attach bird species IDs.
  • Social: join a local bird club
    • Most bird organizations have a social media presence, so start by searching for your state’s bird club online and follow along through posts for opportunities to get involved in the community!
  • Solitary: spend some quiet time outdoors looking and listening (not much more explanation needed!)

So, consider which strategies most appeal to you and get out there observing birds! It’s easy and fun to get started!

My Annoying Climate Download Process

Posted on by Jess

Here’s a summary of my week: I needed mass amounts of data from the NetcdfSubset portal, but it was too much for the HTTP server to handle (they set a cap) with just selecting the products and spatial extent to download. So, instead they returned to me a URI that needed to be passed through an external program, nccopy, to download the data. I wrote a script that separated the URI into separate files by model and scenario, and thus automated the download to save each combination of model, scenario and variable into separate NetCDF files.

The problem became that the download was really slow, owing to traffic here on my work network. Since there was no file size estimate given to me, I assumed maybe the files were huge. So, I did some internal compression to get them to download faster, but at the expense of read access speed for the files. Once I realized it wasn’t the file size, I redid the request for the files without chunking. I then had to kill that request to tailor the data acquisition for our needs, so I finally got all the temperature files today.

Then, I installed CDO and wrote a script to do the monthly means for all the files: it averages a daily time step file into a monthly averaged (or whatever metric you choose) file. I got a list of the base file names as such:

cat climate_structure | cut -d . -f1 > climate_models

Then  I wrote this simple bash script to loop over them all and write out the monthly averages:

#!/bin/bash
while read climate_model; do
 echo "Now averaging $climate_model"
 cdo monmean ${climate_model}.nc ${climate_model}_monthly.nc
done < climate_models