running WRF with NARR--an overview

updated slides from our WRF club/WRF party this week are attached as a movie (!!)

Comment here with questions or additional info. you find ...
because ain't no party like a W R F party 'cause a W R F party don't stop...

(don't want to squint-read or read enlarged blurry text? want to be able to click on the links on the slides? a pdf of slides is also on nitrate: /archive/shared/WRF_club.pdf )

How to compile CMAQ in parallel

Recently, Phil and I put together a parallel compilation of CMAQ on nitrate, the recently purchased Linux box. Direct comparison with a serial compilation on the same machine, with the same input data, reveals that CMAQ scales reasonably well on a small number of cores. We observed a 7x increase in run speed by running on all 8 cores as compared to a single core. However, some key modifications need to be made to the installation scripts for successful compilation. These are listed, to the best of my knowledge, below. Please note that this is NOT meant to be a comprehensive guide on how to install CMAQ, nor can I guarantee that I've caught all the necessary changes, since I'm writing this up about a week after the actual install. That said, it may serve as a useful resource for someone who already knows how to compile CMAQ in serial and wants to get it running successfully in parallel.

1. Before installing
Some libraries that work well for serial compilations don't play nicely with the parallel compilation. To save yourself grief later, link these libraries into the CMAQ libraries location:

libnetcdf.a - Make sure this is compiled without DAP support. I have no idea what DAP support is, but it breaks parallel compilations and we don't use it. There is a flag that can be passed to the configure script when installing netcdf that turns it off.

libioapi.a - Surprisingly, a standard version of IOAPI will work just fine with a parallel compilation. IOAPI has a bunch of parallel IO options that you can set when compiling, but CMAQ doesn't use them. CMAQ (at least 4.7.0) is only parallelized for processing, not for file IO, so just use whatever library you used for the serial compilation. Of course, make sure you've properly included the fixed_src folder as you'll need the contents throughout.

libmpich.a - This doesn't have an explicit folder the way IOAPI and netCDF do in the CMAQ installation, but you'll need it for a parallel installation. If it isn't on your system, download and install it.

2. pario
Installing the parallel IO library (pario) is not necessary for a serial installation, but it is necessary to build CMAQ in parallel. Install it as you would any other component (bcon, icon, etc.) by modifying the library paths and the compiler path/flags.

3. stenex
The stencil exchange library has both parallel and serial components. You can get away with just installing sef90_noop for a serial build (built from bldit.se_noop), but for parallel you'll also need to run bldit.se to generate se_snl. It may be possible to skip the installation of sef90_noop if you want to run strictly in parallel, but I haven't tried. In any case, the only difference in installing these two files is that se_snl needs the mpich header file location.

4. Other components
To the best of my knowledge, the installation for m3bld, jproc, icon, and bcon is all unchanged from serial installation. Build these as you normally would.

5. cctm
This is probably where the largest numbers of changes need to occur. Let's break it down into two categories: building and running

Building cctm:

Make the following changes to the bldit.cctm script:

• Uncomment the line reading "set ParOpt"
• appropriately set location of mpich in the MPICH variable. Note that this is the top-level directory, and should have include, bin, and lib directories underneath it
• Change FC from whatever compiler you were using before to mpif90 (provided it is installed on your system). mpif90 is a wrapper compiler that adds in extra flags as needed for compiling parallel programs. Note that this may not be available for MPI implementations other that MPICH
• Add a flag to F_FLAGS reading -f90=oldCompiler where "oldCompiler" is the compiler you were using before. This makes sure mpif90 wraps the correct compiler.
• find the line where the script sets the COMP variable. Comment it out and replace with

  set COMP = "intel"

Running cctm:

Make the following changes to the run.cctm script

• Change the variables NPCOL_NPROW and NPROCS to reflect the number of processors you would like to use and their organization. There should be an example commented out in the file already. Note that the two values for NPCOL_NPROW should multiply to give NPROCS
• At the very bottom of the file, comment out the line "time $BASE/$EXEC"
• uncomment the four lines beginning "set MPIRUN", "set TASKMAP", "cat $TASKMAP", "time $MPIRUN"
• Change the location of MPIRUN to reflect the actual path to the executable on your system (at the command line, run "which mpirun" to find the executable if you don't know where it is)

Make the following changes BEFORE RUNNING

• There should be a file in the cctm directory labeled "machines8". Open up this file, erase the contents (they are meaningless) and enter in "sysname:num" on each line, where sysname is the name of the system you're working on, and num is a number starting at 1, and 1 larger each line. Put this string on each line, continuing until you've reached the max number of processors. IE for nitrate, the machines8 file looks something like

  nitrate:1
  nitrate:2
  nitrate:3
  nitrate:4
  nitrate:5
  nitrate:6
  nitrate:7
  nitrate:8

You should now be ready to run CMAQ in parallel!

How to find my stuff. Chapter 1.

Please let me know if there are any permissions issues or if this info seems incomplete. I tried to make everything public but may have missed things.

The output from SMOKE/emissions input to CMAQ are all on mercury in:
/Volumes/archive/luedke/data/emis/
There is a separate subfolder for each scenario I did (baseline, no HDDV, no onroad, no PTIPM)

The output from CMAQ is on mercury in:
/Volumes/archive/luedke/data/cctm/
Again there's a subfolder for each scenario.

On SOX I have all the data and scripts I used to make plots, and the plots themselves. They are in folders generally pertaining to the type of plot:

/Users/luedke/8hr_max : getting the average monthly 8hr max ozone

/Users/luedke/extreme_plotting : plotting extreme events. some scripts are to plot the frequency and location of these events, and others are for percentage contribution only during those events. what constitutes an "extreme event" can be defined by you.

/Users/luedke/making_averages : here is where i made averages of emissions from the SMOKE output to compare with NEI attributes listed on the EPA website

/Users/luedke/nitrate_plotting : nitrate pm2.5

/Users/luedke/no2_plotting : NO2 plots across the whole CONUS

/Users/luedke/nox_plotting_in_the_east : NO and NO2 across just east US

/Users/luedke/ozone_plotting : monthly average ozone. not as useful as 8hr max so this didn't make it to my thesis.

/Users/luedke/pm25_plotting : total PM2.5, calculated by just adding all the PM2.5 species together (see previous blog post about getting PM2.5 from CMAQ. this is the result of the simplest method from the CMAS presentation in 2010)

/Users/luedke/sulfate_plotting : sulfate pm2.5

On NOX I have all the SMOKE goodies. In the future our group will hopefully be using an updated version of SMOKE and this may not be useful (I used 2.4), but here it is:

/home/luedke/assigns : where my assigns files are. some sectors needed different scripts which was dumb but that's why there are some extras

/home/luedke/go : where my runscripts are for each sector and for merging them

/home/luedke/intermed/2002ac : intermediate data from each sector script. the results of merging in the right combination were moved to mercury.

thank you and good day.

Notes on data, scripts, and externals

I am attaching several word documents that will give you all a little more information on my scripts, data, and externals.

In general, all my externals can be found in room 287 on one of the bookshelves. This document describes what is on each external. You might have to do a little "digging" to find exactly what you want, but most of it is explained in the documentation

http://www.scribd.com/doc/63671067/Externals?secret_password=ttur6t1wu39esfd00at

The next two documents are not so detailed. I made a quick "fact" sheet - called DATA- of where important data can be found - this will help instead of "digging" around the external sheet. Mostly it will help for Caitlin's old runs (SMOKE, WRF, CMAQ) and my new CMAQ runs

http://www.scribd.com/doc/63671344/Data?secret_password=2b3o23k974ub33mvi4xb

The last just states where you can find my early NCL scripts, my newer ones for my MS thesis, ioapi scripts, and where I ran CMAQ and CHEMMECH.

http://www.scribd.com/doc/63671304/Script-Notes?secret_password=2e1lrlfyupye3dm1t2xk

If anyone needs to contact me with questions you can contact me at jamorton74@gmail.com
Thanks everyone and it has been my pleasure to work with you all!

J

Getting PM2.5 from CMAQ

PM2.5 is a big deal. But it is quite a beast to model, and CMAQ does so in a speciated way. This means that CMAQ keeps track of all the different types of PM (from nitrate, sulfate, whatever) and keeps them separate.

That sounds cool, because I mean that would let you look at a specific one if all you actually care about is sulfate for example. Makes sense.

But! It is way less cool for you if you really only care about PM2.5 as a total count, or if you want to use the total count as a frame of reference to a specific kind. The EPA NAAQS regulate PM2.5 as a total, so you really should care about PM2.5 as a whole too.

The reason it is less cool for you is because there is no "total PM2.5" output from CMAQ at all. You have to make it yourself. This sounds easy but can be VERY misleading, so I want to try and help you do this the right way. I did it the wrong way and it caused me more work so let's avoid that. If this confuses you even more I apologize...

There are conflicting options for calculating PM2.5. The way I did it was according to a presentation given by CMAS in 2010. That powerpoint is here: http://arset.gsfc.nasa.gov/umbc/files/Session1Day3/CMAQ-Introduction-for-ARSET.ppt

On slide 10, it says that PM2.5 is equal to the sum of a bunch of different species. So that's what I did. Is that right?

I once thought it was. Then things changed.

Instead, you can follow the approach in this document here:
www.epa.gov/CAIR/pdfs/CMAQ_Evaluation.pdf

This says on page 3 that you did the right thing when you picked your species to count in the mix, but you really should be scaling two of them by 1.167. Is that right?

Possibly? Maybe?

Next we check "Evaluation of the community multiscale air quality (CMAQ) model version 4.5: Sensitivities impacting model performance; Part II—particulate matter" by K. Wyat Appel. Note that this is for version 4.5 and you shouldn't be using that version, but in any case it says not only should you be doing that scaling bit, but also you should be looking at sodium and chlorine species too.

This is also in "A multi–pollutant,risk–based approach to air quality management:Case study for Detroit" by Karen Wesson. Is that right?

Could be?

Sometimes in life and in modeling, you have to make your own answers.

Mercury Paper Figure Documentation

I am attaching my word document describing the figures in the mercury paper that Tracey and Caitlin wrote. Each page is describing a new figure and what data was used to create it. Most of the figures were made using excel, but some where created with NCL.

http://www.scribd.com/full/61868877?access_key=key-1zae00432g0xpzpcp93a

If figures were made in excel the pathway is bolded in purple.
If figures were made with NCL the pathway to the script is bolded in black. The data for the NCL plots are bolded in blue.

I will posting more documentation on other figures I have made for my thesis work.

Jami

CMAQ Output: ACONC vs. CONC

At last week's group meeting, I discussed the difference between 2 of CMAQ's output files: ACONC and CONC.

CONC = instantaneous pollutant concentration at each output timestep (each hour)

ACONC = average pollutant concentration for each model hour (average throughout hour)

The CONC file automatically includes all pollutants at all vertical model layers, whereas the pollutants and vertical layers in the ACONC are set by the user in the CCTM run script. In general, use ACONC for comparison with ground-based measurements and CONC for comparison with satellite data.

Overall, I found that ACONC and CONC concentrations compare very well over both the CONUS and GL domains for secondary (formed in atmosphere) pollutants. A much bigger difference is seen between ACONC and CONC for primary (directly emitted) pollutants. This difference is expected - subhourly variations in wind speed and other meteorological factors will affect the local concentrations of primary pollutants emitted from point sources (e.g. SO2, CO) much more that the concentrations of the more well-mixed secondary pollutants (e.g. O3, ASO4).

Nothing earth-shattering, but might be helpful to someone running CMAQ for the first time.

Steve