National Oceanic and
Atmospheric Administration
United States Department of Commerce

The Fregrid Service

Last modified: Mon, 04/03/2017 - 17:29
LAS can be used to transform data on unusual grids to regular lat/lon grids and to deliver the result. This service is implemented on top of Fregrid from GFDL which handles the transformations.

To use the Fregrid service you must first configure the data sets and variables that you want users to be able to transform and download. Internally, LAS can handle data defined on many different grids, but the user interface only allows users to make selections over a rectilinear lat/lon grid. So when defining these variables in LAS you should assign them LAS <grid> and <axis> descriptions that cover the region for the target grids which will be the result of the transformation (rather than describing the data on their native grids).

For example, if a file contains data which can be transformed into a global rectilinear grid then use a similar grid in the LAS configuration definitions as show here:

<grids>
 <global_grid>
 <link match="/lasdata/axes/global_x" />
 <link match="/lasdata/axes/global_y" />
 <link match="/lasdata/axes/level" />
 </global_grid>
</grids>
<axes>
 <global_x type="x" units="degrees_east">
 <arange start="0." step="1" size="360" />
 </global_x>
 <global_y type="y" units="degrees_north">
 <arange start="-89.5" step="1" size="180" />
 </global_y>
 <level type="z" units="mb">
 <v>3.650293</v>
 <v>19.0884</v>
 <v>52.34019</v>
 <v>99.12992</v>
 <v>157.381</v>
 <v>224.7492</v>
 <v>298.9444</v>
 <v>378.0866</v>
 <v>458.3251</v>
 <v>533.7864</v>
 <v>601.7172</v>
 <v>662.5931</v>
 <v>716.8593</v>
 <v>764.9444</v>
 <v>807.2677</v>
 <v>844.2424</v>
 <v>876.2772</v>
 <v>903.7783</v>
 <v>927.1495</v>
 <v>946.7935</v>
 <v>963.1113</v>
 <v>976.5034</v>
 <v>987.3693</v>
 <v>996.1084</v>
 </level>
</axes>

The Fregrid software will be looking for a collection of files which contain different sections of the whole grid definition so the url definition for the data set should describe the base name of the pattern, not the individual files. In order to use the Fregrid syntax of specifying an input directory in one argument and the file name pattern in another argument, I added a url_base attribute to each variable. This will be used in the template that builds the Fregrid command when a Fregrid request is processed. For example, for these files, the data set line would look like this:

<datasets>
 <fregrid_ex name="Fregrid Example Data Set" url="file:19800101.atmos_daily">
 <variables>
 <zsurf name="Surface Height" units="m"
 url_base="/home/porter/rhs/gridspec/tools_20071113/data/input/">
 <link match="/lasdata/grids/global_grid" />
 </zsurf>
 <temp name="Temperature" units="Deg K"
 url_base="/home/porter/rhs/gridspec/tools_20071113/data/input/">
 <link match="/lasdata/grids/global_grid" />
 </temp>
 <t_surf name="Surface temperature" units="Deg K"
 url_base="/home/porter/rhs/gridspec/tools_20071113/data/input/">
 <link match="/lasdata/grids/global_grid" />
 </t_surf>
 </variables>
 </fregrid_ex>
</datasets>

I have added a Fregrid operation and an example service_action script to the base LAS configuration. The operation definition specifies one netCDF output file, one debug file and a cancel file.


 <operation name="Regrid" ID="Fregrid" service_action="launch" output_template="fregrid">
 <response ID="FregridResponse">
 <result type="debug" ID="debug" file_suffix=".txt"/>
 <result type="netCDF" ID="netcdf" file_suffix=".nc"/>
 <result type="cancel" ID="cancel" file_suffix=".txt"/>
 </response>
 <properties>
 <property_group type="backend_request">
 <property>
 <name>exclude</name>
 <value>ferret</value>
 </property>
 </property_group>
 </properties>
 <service>fregrid</service>
 </operation>

In order to run Fregrid, you must assemble the correct command line arguments based on the information in the LAS request. This is done via a the service_action template. Here is an example:

#set($var0 = $las_backendrequest.getVariables().get(0))
#set($infile = $las_backendrequest.getDataAttribute($var0, "url"))
#set($base = $las_backendrequest.getDataAttribute($var0, "url_base"))
#set($vars = $las_backendrequest.getVariablesAsString())
#set($netcdf = $las_backendrequest.getResultAsFile("netcdf"))
--input_mosaic /home/porter/rhs/gridspec/tools_20071113/data/output/C48_mosaic.nc --input_dir $base --input_file $infile --scalar_field $vars --output_mosaic
 /home/porter/rhs/gridspec/tools_20071113/data/output/N45_mosaic.nc --interp_method conserve_order1 --output_file $netcdf --check_conserve

In this case, the input directory for the source files, the pattern for the individual file names and the variables to be dumped are all derived from the LAS request. The first line sets the value of $var0 the first variable in the request. Since all of the variables in this data set share the same input file directory and file names, this is sufficient for setting the values of $infile (the name pattern for the input files) and $base (the base directory name where the file are located). In this template these values are pulled from the attributes of the <data> element of the backend request XML. These attributes correspond to the attributes on the individual <variable> elements in the configuration after they have been merged with the attributes on the <dataset> element (which is how the "url" attribute's value is passed from the parent <dataset> to the child <variable> elements).

The values for the --input_mosaic and --output_mosaic are fixed in this template. In an actual LAS it might be necessary to configure this information into the data set as constraints in the UI and/or as data set properties.

Here is the LAS UI Request XML that was used to test this service.

<lasRequest href="file:las.xml" package="">
 <link match="/lasdata/operations/operation[@ID='Fregrid']"/>
 <args>
 <link match="/lasdata/datasets/fregrid_ex/variables/zsurf"/>
 <link match="/lasdata/datasets/fregrid_ex/variables/temp"/>
 <link match="/lasdata/datasets/fregrid_ex/variables/t_surf"/>
 <region>
 <range
 high="180.0"
 low="-180.0"
 type="x"/>
 <range
 high="89.0"
 low="-89.0"
 type="y"/>
 <point
 type="t"
 v="15-Jan"/>
 </region>
 </args>
</lasRequest>

Note:

  1. Both user interfaces limit the user to selecting only one variable. In order to regrid many variables at once the UI will have to be customized for this data set. The UI will have to allow more that one variable to be selected, additional variables must be passed in as constraints or groups of variables must be defined in the template and included in the output based on the single variable selected. In the example above, three variables are passed in and all three of the variables get included in the command line for Fregrid via the getVariablesAsString() method.
  2. The X, Y, and T limits of this request are ignored. There is likely to be some work required to pass these values to Fregrid so it can create sub-sets of whole grids.

When the template is processed using this input from this request this is the set of Fregrid arguments it produces:

--input_mosaic /home/porter/rhs/gridspec/tools_20071113/data/output/C48_mosaic.nc --input_dir /home/porter/rhs/gridspec/tools_20071113/data/input/ --input_fi
le 19800101.atmos_daily --scalar_field zsurf,temp,t_surf --output_mosaic /home/porter/rhs/gridspec/tools_20071113/data/output/N45_mosaic.nc --interp_method c
onserve_order1 --output_file /home/porter/rhs/tomcat/webapps/baker/output/18818164FEA9182CE5CC293931AB73A4_netcdf.nc --check_conserve

LAS will produce an debug file with the output from the Fregrid run. In this case, the file looks like this:


****fregrid: first order conservative scheme will be used for regridding.
NOTE: done calculating index and weight for conservative interpolation
the flux(data*area) sum of zsurf: input = 1.18137e+17, output = 1.18137e+17, diff = 784. 
the flux(data*area) sum of temp: input = 3.07006e+18, output = 3.07006e+18, diff = -38912. 
the flux(data*area) sum of t_surf: input = 1.44497e+17, output = 1.44497e+17, diff = 1632. 
the flux(data*area) sum of temp: input = 3.07389e+18, output = 3.07389e+18, diff = -5120. 
the flux(data*area) sum of t_surf: input = 1.43815e+17, output = 1.43815e+17, diff = -1072. 
the flux(data*area) sum of temp: input = 3.08072e+18, output = 3.08072e+18, diff = 124416. 
the flux(data*area) sum of t_surf: input = 1.4378e+17, output = 1.4378e+17, diff = -544. 
the flux(data*area) sum of temp: input = 3.08894e+18, output = 3.08894e+18, diff = -214016. 
the flux(data*area) sum of t_surf: input = 1.43837e+17, output = 1.43837e+17, diff = 624. 
the flux(data*area) sum of temp: input = 3.097e+18, output = 3.097e+18, diff = -4608. 
the flux(data*area) sum of t_surf: input = 1.43912e+17, output = 1.43912e+17, diff = 208. 
the flux(data*area) sum of temp: input = 3.10426e+18, output = 3.10426e+18, diff = -54272. 
the flux(data*area) sum of t_surf: input = 1.44004e+17, output = 1.44004e+17, diff = 544. 
the flux(data*area) sum of temp: input = 3.11089e+18, output = 3.11089e+18, diff = -210432. 
the flux(data*area) sum of t_surf: input = 1.44115e+17, output = 1.44115e+17, diff = 176. 
the flux(data*area) sum of temp: input = 3.11696e+18, output = 3.11696e+18, diff = -42496. 
the flux(data*area) sum of t_surf: input = 1.44231e+17, output = 1.44231e+17, diff = 1312. 
Successfully running fregrid and the following output file are generated.
****/home/porter/rhs/tomcat/webapps/baker/output/18818164FEA9182CE5CC293931AB73A4_netcdf.nc

I made a simple output template to display the resulting netCDF file.

<html>
<!-- LAS SERVER OK -->
<head>
<title>LAS Output -- $las_config.getTitle()</title>
</head>
<body>
Netcdf file is <a href="$las_response.getResult("netcdf")">here.</a>
</body>
</html>

It might be necessary to configure the output template to display the debug output information to the user if it is important for them to see it.

The actual binary that is executed and the environment it runs under and the error messages it returns is controlled by the FregridBackendConfig file. Here is an example:

<application>
 <invoker base_dir="resources/fregrid"
 executable="/home/porter/rhs/gridspec/tools_20071113/bin/fregrid"
 time_limit="200" template_dir="templates"
 output_dir="/home/rhs/tomcat/webapps/baker/output"
 http_base_url="http://strider.weathertopconsulting.com:8880/baker/output"
 iosp_script_dir="resources/iosp/scripts"
 iosp_temp_dir="/home/rhs/workspace/baker/conf/server/temp"
 iosp_data_dir="/home/rhs/workspace/baker/conf/server/data/dynamic"
 las_tfds_data_dir="/home/rhs/workspace/baker/conf/server/data"
 iosp_base_dir="resources/iosp"
 opendap_base_url="http://porter.pmel.noaa.gov:8920/thredds/dodsC/output" />
 <cacher dataset="true" dataset_queue_size="10000" dataset_size="40"
 subset="true" subset_queue_size="10000" meta="true"
 meta_queue_size="10000" />
 <messages>
 <message type="startsWith" level="error">
 <key>Error</key>
 <text>
 An error occurred in the service that was creating your
 product.
 </text>
 </message>
 <message type="startsWith" level="error">
 <key>REQUIRED:</key>
 <text>
 An error occurred in the service that was creating your
 product.
 </text>
 </message>
 <message type="contains" level="error">
 <key>unrecognized</key>
 <text>
 An error occurred in the service that was creating your
 product.
 </text>
 </message>
 <message type="startsWith" level="error">
 <key>FATAL</key>
 <text>
 An error occurred in the service that was creating your
 product.
 </text>
 </message>
 </messages>
 <environment>
 <variable>
 <name>NPES</name>
 <value>1</value>
 </variable>
 </environment>
</application>

NPES is the only environment variable I am aware of that is used by Fregrid. This controls the number of processors used by the Fregrid. All of the error messages that I know it produces are included in the messages section. These messages are hidden from the user and the <text> section of the <message> element is sent in its place. Submitting with debug turned on will reveal the actual error message.

Note: The FregridBackendConfig.xml file is not created automatically in the LAS configure process. You'll have to make your own in an editor. There is a base template included and a complete example in the source you can use to create your own.