Personal tools
You are here: Home Documentation Installer Documentation Installation Installing and integrating TDS with LAS.
Document Actions

Installing and integrating TDS with LAS.

by roland — last modified 2009-04-17 16:23

We use the THREDDS data server to provide some of the services LAS needs. To make this all work you'll have to install TDS and integrate it with LAS.

N.B.  We have noticed that our Tomcat environment is much more stable if we tell robots to not crawl our THREDDS catalog HTML interface.  We actually exclude robots from our entire Tomcat site with a robots.txt file in the ROOT webapp directory.  The robots.txt file we use contains:

User-agent: *
Disallow: /

LAS allows users to define new variables which are analytic transformations of existing variables. For example, a user can define a new variable that is the average over all depths from an existing LAS variable that is defined on latitude, longitude and depth. The implementation of the transformation is done via the integration of the analytical capabilities of Ferret and the OPeNDAP data services or the THREDDS Data Server. Installing and integrating TDS with your LAS installation is what make user defined variables and any automatic interpolation between grids when making comparison plots (like differences between variables on different grids) possible. A separate write-up has the full details on the integration between TDS and Ferret (as used by LAS or as an independent OPeNDAP server). So let's get started getting TDS ready to be used by LAS.

Important Note: The LAS configuration process assumes you have the same proxy setup for your TDS server as you do for your LAS server. If you have added proxy directives to an Apache server for LAS, you'll need to add similar proxy statements to the configure for your THREDDS server using. For example your LAS set up is as follows:

<Location /las>
ProxyPass http://myhost.gov:8080/las
ProxyPassReverse http://myhost.gov:8080/las
</Location>
<Proxy http://myhost.gov:8080/las>
AllowOverride None
Order allow,deny
Allow from All
</Proxy>
you'll need a similar setup for your THREDDS server:
<Location /thredds>
ProxyPass http://myhost.gov:8080/thredds
ProxyPassReverse http://myhost.gov:8080/thredds
</Location>
<Proxy http://myhost.gov:8080/thredds>
AllowOverride None
Order allow,deny
Allow from All
</Proxy>


Install TDS.

This is pretty straight forward and there are excellent instructions at the Unidata site.

Create or identify some disk areas for data and temporary files.

When LAS starts up it will automatically build the necessary Ferret journal files that will allow TDS to serve the data from your LAS via OPeNDAP. In order to do this, LAS needs to write a small journal file in a directory that is accessible by the tomcat host and allows read/write privileges by the tomcat user. The default directory selected by the configuration process is $LAS_HOME/conf/server/data. You'll need to remember the full path name of this directory and use it when you install LAS.

The process of building and serving the LAS data and new variables based on these data requires some temporary disk space. An ideal directory would be large scratch area configured on your server which is not part of the regular back ups, but is accessible to the tomcat host and grants read/write privileges to the tomcat user. Files written to this area are also reused by F-TDS in the event that the same data are requested more that once so it's nice if they can stay around for a while, but if a file is not found it will be recreated as needed. The default directory selected by the configuration process is $LAS_HOME/conf/server/temp. You'll need to remember the full path name of this directory and use it when you install LAS.

Create a <datascan> element where the LAS data hierarchy will appear in TDS.

Now that TDS is installed and you've figured out where you're going to store the data and temporary files you can finish the integration of TDS with your LAS installation. Find the THREDDS catalog that is being used to configure your TDS. It's $TOMCAT_HOME/content/thredds/catalog.xml. Edit this file and add an entry so that it will server your LAS data.

<datasetScan name="Data From LAS" path="las" location="/home/roland/data/las" serviceName="gridded">
<filter>
<include wildcard="*.nc"/>
<include wildcard="*.fds"/>
<include wildcard="*.jnl"/>
</filter>
</datasetScan>

The location is the directory you identified or created as your data directory in the previous step. The serviceName should point to a service of serviceType="OpenDAP". Or a compound service that includes an OpenDAP service. Since LAS will configure TDS to serve all of your LAS data defined on a grid (and currently only the data on grids) via TDS you might want to associate this datasetScan directory with a compound service that include the TDS WCS service so that GIS clients can access your LAS data. For example, I used a compound service that looks like this:

   <service name="gridded" serviceType="Compound" base="" >
<service name="opendap" serviceType="OpenDAP" base="/thredds/dodsC/"/>
<service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
</service>

If you want the WCS service you must turn on it on explicitly in your TDS configuration by adding something like this:

  <WCS>
<allow>true</allow>
<dir>/home/roland/temp/wcs</dir>
<scour>15 min</scour>
<maxAge>30 min</maxAge>
</WCS>
to your $TOMCAT_HOME/content/thredds/threddsConfig.xml file.

Configure in the FerretDataSource and FerretIOServiceProvider.

In order to use TDS with you LAS you have configure a couple of "plug-ins" into your TDS. These plug-ins allow TDS to serve data opened in a Ferret journal file via OPeNDAP and allow new data variables to be defined by passing TDS special a special URL syntax (make a link to F-TDS under "speical URL syntax"). To add these capabilities to LAS you'll need to add the following to your $TOMCAT_HOME/content/thredds/threddsConfig.xml file.

  <datasetSource>gov.noaa.pmel.tmap.iosp.FerretDataSource</datasetSource>

<nj22Config>
<ioServiceProvider class="gov.noaa.pmel.tmap.iosp.FerretIOServiceProvider"/>
</nj22Config>

For reference the full threddsConfig.xml file I use is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<threddsConfig>

<datasetSource>gov.noaa.pmel.tmap.iosp.FerretDataSource</datasetSource>

<nj22Config>
<ioServiceProvider class="gov.noaa.pmel.tmap.iosp.FerretIOServiceProvider"/>
</nj22Config>

<WCS>
<allow>true</allow>
<dir>/home/roland/temp/wcs</dir>
<scour>15 min</scour>
<maxAge>30 min</maxAge>
</WCS>

</threddsConfig>

Once you have made all of these changes you're ready to go on with your LAS installation. When you run the configuration script that comes with LAS you'll be asked for the directory paths of the temporary area and data areas you created. The configuration script will also compile and copy the FerretDataSource and FerretIOServiceProvider classes to the correct place in the TDS webapps directory.

When LAS starts it will write at least one .jnl file for each LAS data set into the data directory you created in this step. When you're done installing LAS and have restarted you tomcat server you should see your "Data From LAS" (or whatever you called it in your datasetScan element) in your THREDDS catalog. Go ahead and poke around in there and verify that you can manipulate those data sets via the THREDDS HTML interface.

Configure the logging for the FerretIOServiceProvider and the FerretDataSource (optional).

The TDS comes configured to write logging information into a special directory. The log messages are separated into different files based on the component of TDS that is logging the message. You add configuration information into the log4j.xml file so that message from the LAS components are written to their own file in the TDS logs directory.

Find the log4j.xml file in $TOMCAT_HOME/webapps/thredds/WEB-INF and copy it to $TOMCAT_HOME/content/thredds. This is necessary since the log configuration files is not one of the files that the THREDDS distribution will preserve when you up-grade your TDS.

Edit your copy in $TOMCAT_HOME/content/thredds to add an appender:

<appender name="ferretIOSP" class="org.apache.log4j.RollingFileAppender">
<param name="File" value="${logdir}/ferretIOSP.log"/>
<param name="MaxFileSize" value="10MB"/>
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd'T'HH:mm:ss.SSS Z} [%10r][%8X{ID}] %-5p - %c - %m%n"/>
</layout>
</appender>

and to add a logger:

  <logger name="gov.noaa.pmel.tmap.iosp" additivity="false">
<level value="DEBUG"/>
<appender-ref ref="ferretIOSP"/>
</logger>

Save the file and copy it back to $TOMCAT_HOME/webapps/thredds/WEB-INF/classes. You have to restart you tomcat server for these changes to take effect. After the restart any logging messages written by the LAS components installed in TDS will appear in the $TOMCAT_HOME/content/thredds/logs/ferretIOSP.log file.


Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: