National Oceanic and
Atmospheric Administration
United States Department of Commerce

Using data that spans multiple files.

Using data that spans multiple files.

Question:

How can I plot the time series of a variable which spans multiple data files?

Example:

> I have some netcdf-files with a lot of data. Each file represents the
> status of a certain model time. Now I want to plot one variable (say the
> instantanous mean) as a function of time. How can I do it?
>
> And can I save the calculated time series in a seperate (binary or netcdf)
> file from Ferret?

Explanation:

The answer to this depends on how your existing NetCDF files are arranged. You say that each file has a single time step. Let us suppose there are 3 spatial dimensions for the variables in your NetCDF files. There are two ways that this could be encoded: either as 3D (time-independent) variables or as 4D variables on a single point time axis. "ncdump -h mynetcdf.cdf" will show if the variables have a 1-point time axis.

Solution:

==> If the files have a 1 point time axis there are three solutions:

  1. If you have a THREDDS server installed, use THREDDS aggregation to make the data into a single virtual dataset. THREDDS aggregation may be used to combine files that contain different timesteps of the same variable(s). It also includes other powerful tools for modifying and combining datasets.
     
  2. Define a virtual dataset in the Ferret session (Ferret v7 and higher) using DEFINE DATA/AGGREGATE/T.  (Virtual dataset aggregation may also be done in the E direction, to define an ensemble, or the F direction, defining a forecast collection.)
    yes? let file_list = SPAWN("ls -1 tmp/step_*.nc")
yes? TSERIES all_times = file_list
  3. Use ferret to concatenate the files:
     use file1
 save/file=concat.cdf var1, var2, var3
 use file2
 save/append/file=concat.cdf var1,var2,var3
.
.
.

    note that the names "var1", "var2",... refer to the "current" data set since no explicit data set was specified as in "var1[d=file1]".

    4.  

  4.  
  5.  
  6.  
  7.  
  8. Create an "MC" (multi-cdf) data set by writing a descriptor file that ties the files into a single data set. An example of such a descriptor is below.

==> If your files do not have a time word,

 

  • then you must use Ferret to impose atime word. Suppose you have 10 files, each separated by one model day, running 1-jan-1980 to 10-jan-1980. Then you might use this technique:
     ! *** set up
 define axis/t=1-jan-1980:10-jan-1980:1/unit=days t10days
 define grid/t=t10days gg

 let var1_with_T = t[g=gg]*0 + var1
 set variable/title="xxx"/units="uuu" var1_with_T

 ! *** loop over input files
 use file1.cdf
 save/file=concat.cdf var1_with_T[l=1]

 use file2.cdf
 save/file=concat.cdf/append var1_with_T[l=2]

 ...

 

--------------------------------------------------
Here is an example "MC" descriptorfile: my_mc_descriptor.des:

 

 $FORMAT_RECORD
 D_TYPE = ' MC',
 D_FORMAT = ' 1A',
 D_SOURCE_CLASS = 'MODEL OUTPUT',
 D_SOURCE = 'PHIL/SIEG',
 D_SUBSOURCE = 'TROPICAL PACIFIC',
 $END
 $BACKGROUND_RECORD
 D_EXPNUM = '0068',
 D_MODNUM = ' AA',
 D_TITLE = 'MOM model output; Hindcast, Double Ramp; NCEP 79-95 twice daily winds'
 D_FORCING_SET = 'NCEP 79-95 daily',
 D_T0TIME = '14-JAN-1978 14:00:00',
 D_TIME_UNIT = 3600.0,
 D_TIME_MODULO = .TRUE.,
 D_ADD_PARM = 15*' ',
 $END
 $MESSAGE_RECORD
 D_MESSAGE = ' ',
 D_ALERT_ON_OPEN = F,
 D_ALERT_ON_OUTPUT = F,
 $END
*************************************************
*************************************************
 $EXTRA_RECORD
 $END

 $STEPFILE_RECORD
 s_filename = 'mtaa068-nc.001',
 S_AUX_SET_NUM = 0,
 S_START = 87623.0,
 S_END = 183599.0,
 S_DELTA = 24.0,
 S_NUM_OF_FILES = 400,
 S_REGVARFLAG = ' ',
 $END
 **************************************************
 $STEPFILE_RECORD
 s_filename = '**END OF STEPFILES**'
 $END
 **************************************************