Using NCO with CCR netCDF Files
- What is NCO and why should I use it?
- NCEA: Ensemble Averager
- NCRCAT: File Concatenator
- NCKS: Kitchen Sink
- NCDIFF: File Differencer
- NCRENAME: Rename
Other NCO Documentation
What is NCO and why should I use it?
NCO is a group of programs that allows you to manipulate netCDF files. You can edit file attributes, rename variables, extract variables for certain dimensions (such as a given time slice or geographical area), and create ensembles and averages. You can do much of the same things with NCL, but NCO is much easier to use. The commands are given at the unix command prompt (in our examples, plum-%), or they can be strung together in a shell script, depending on the amount of manipulation that needs to be done. The official documentation for NCO can be found here. We've put together some examples here using some CCR files. In all cases, the original data file is available for download.
How to use NCO and these examples: first, make a back-up copy of any files you wish to process. NCO will usually give you an error message if something goes wrong, but it's always best to have a copy of your unaltered data. On this page, all commands are shown in red type.
On dave (dave.ucar.edu) at NCAR, you should use a newer version of NCO. The path is /usr/local/apps/nco-2.7.1/bin/
Ncea creates ensemble averages. It is generally used when you have several years of data, and you want to create an ensemble, or average, of those years (i.e., going from individual year files to mone files).
Example: you have five files, each of which have twelve months of data in them (Jan-Dec). You want to create an ensemble (average) of these five years.
If the files are: (click on them to download):
Then the command would be:
plum-% ncea -n 5,5,1 ha.F1.PBC_TPBOX.00510.nc ha.F1.PBC_TPBOX.510_514.nc
- 5 is the number of files in the ensemble
- 5 is the digit_number, the fixed number of numeric digits comprising the numeric_suffix (00510, 00511, 00512, etc.)
- 1 is the numeric_increment, the constant, integer-valued difference between the numeric_suffix of any two consecutive files
- ha.F1.PBC_TPBOX.00510.nc is the first input file in the series
- ha.F1.PBC_TPBOX.510_514.nc is the output file name
NCEA can also be used to create seasonal files from monthly files. For example, if you have three files representing March (moni03.nc), April (moni04.nc), and May (moni05.nc), you can create a file that is the average of those three files - a spring average (March-April-May).
The command would be:
plum-% ncea -n 3,2,1 moni03.nc mami.nc
As before, 3 is the number of files, 2 is the number of digits in the file name, and 1 is the increment.
Ncrcat concatenates (glues together) files across the record dimension, which is usually time. It can be used to create files that contain multiple years of the same month or season, such as "anni" or "juli" files. It may also be used to glue individual year files into a single file, thus creating a "moni" file.
Example: you have five files, each of which have twelve months of data in them (Jan-Dec). You want to glue them together to create a single file that will contain all the months for five years - a "moni" file. Therefore instead of having five files with twelve months each, you will end up with a file with 60 time steps (5 years of 12 months = 60 months).
Using the files from the previous example, the command would be:
plum-% ncrcat -n 5,5,1 ha.F1.PBC_TPBOX.00510.nc ha.F1.PBC_TPBOX.moni.nc
Example: you have five files, each of which have twelve months of data in them (Jan-Dec). You want to create files that contain 5 years of January, 5 years of February ... 5 years of December (i.e., jani,febi,...deci files). You also want to create an "anni" file (5 years of annual average). First we create the individual months files, then the anni file. Using the same files as the above example, the command to create the january individual year file (jani-we call it "moni01") is:
ncrcat -F -d time,1,,12 -n 5,5,1 ha.F1.PBC_TPBOX.00510.nc moni01.nc
To create the febi ("moni02") file, you would change the increment of time to 2
plum-% ncrcat -F -d time,2,,12 -n 5,5,1 ha.F1.PBC_TPBOX.00510.nc moni02.nc
And so on. Once you have made the twelve individual files, (moni01.nc through moni12.nc), you can create an anni file using the ncea command.
ncea -p -n 12,2,1 moni01.nc anni.nc
Notice that the file input command "-n 12,2,1" has changed, since we now have 12 input files (moni01-mon12), and each file name has two digits (moni01.nc-moni12.nc).
Ncks - or nc kitchen sink - is probably the most useful of the NCO operators. You can use it to extract a variable from a file, or to extract multiple variables, or to get variables at certain times or geographical areas. It's especially helpful for when you are moving files from NCAR to CCR. You can extract just the variables you want, making the file sizes much smaller and more manageable.
Example: extract a single variable from a netCDF file
plum-% ncks -v PRECTMM GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-MONE0509.prectmm.nc
Example: extract multiple variables from a netCDF file
plum% ncks -v PRECTMM,PSLEVMB,CLOUD GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-SEAE0509.3vars.nc
Example: extract a single variable from a netCDF file for the month of June from a mone file.
Note: NCO counts from zero. To count from one, use the -F flag in any operation.
plum-% ncks -d time,5 -v PRECTMM GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-MONE0509.prectmm.may.nc
You would get the same result using the following:
plum-% ncks -F -d time,6 -v PRECTMM GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-MONE0509.prectmm.may.nc
Ncdiff allows you to two subtract one netCDF file from another and create a new file with the resulting deltas.
Example: Create a difference file from two yearly FOAM files.
plum-% ncdiff ha.F1.PBC_TPBOX.00511.nc ha.F1.PBC_TPBOX.00510.nc ha.F1.PBC_TPBOX.00511-00510.nc
In the example, we are subtracting the file ha.F1.PBC_TPBOX.00510.nc from the file ha.F1.PBC_TPBOX.00511.nc and creating a new file that will contain the differences, ha.F1.PBC_TPBOX.00511-00510.nc.
You can download the three files and examine the output for yourself:
Ncrename is used to rename variables in a netCDF file.
Example: In many CCR files, precipitation-evaporation is named P-E. However, GrADS and many other software packages will not recognize variables with a minus sign in them. So in order to look at P-E, it must be renamed.
The following command will change the name P-E to PME. The file GEN2AGCM-CONB-MONE0509.rename.nc will be identical to GEN2AGCM-CONB-MONE0509.nc except that P-E will be named PME. You can download the sample file here: GEN2AGCM-CONB-SEAE0509.nc. Here is the file metadata/header information (retrieved via ncdump -h): GEN2AGCM-CONB-SEAE0509.ncdump.txt.
plum-% ncrename -v P-E,PME GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-MONE0509.rename.nc
If you wanted to, you could write over the original file and not create a new file.
plum-% ncrename -v P-E,PME GEN2AGCM-CONB-MONE0509.nc
Ncrename can also be used to rename coordinate variables such as latitude and longitude. This is a little trickier because these latitude and longitude are both dimensions and variables within a file. Therefore, you have to rename both the dimension and and the variable.
Example: rename latitude and longitude to lat and lon, respectively, using the same file as above, GEN2AGCM-CONB-MONE0509.nc.
plum-% ncrename -d longitude,lon -d latitude,lat -v longitude,lon -v latitude,lat GEN2AGCM-CONB-MONE0509.nc GEN2AGCM-CONB-SEAE0509.rename2.nc