National Oceanic and
Atmospheric Administration
United States Department of Commerce

Speeding up large calculations

Speeding up large calculations

Question:

Why does my script run so slowly? How might I improve the speed of the calculations?

Explanation:

SET MODE DIAGNOSTIC lets you see where resources are being used. Memory use can be adjusted, and using explicitly defined regions may allowFerret to able to take advantage of reusing cached results.

Solution:

Here's how you figure out where Ferret is spending its time during a largecalculation:

yes? SET MODE DIAGNOSTIC

On the commands that follow this Ferret will list diagnostic informationdescribing its internal actions. A fragment of the output with explanationis listed below. Try the following:

  • Watch for messages about "gathering". These indicate that Ferret felt thecalculation was too large to fit into memory and it has broken the calculationinto slices. This can slow down the IO considerably (but the calculationdoes get finished without any intervention.) You can influence how Ferretmakes its decision to use gathering -- see section SET MODE DESPERATE in the Ferret Users Guide,
  • Use SET MEMORY/SIZE=<value> to allocate large memory. (see SET MEMORYin the Users Guide.) In conjunction with the information in the UsersGuide on MODE DESPERATE this can greatly enhance performance by fully utilizingthe physical memory of your system.
  • For maximum performance be explicit in telling Ferret the region of interest(i.e. don't leave the limits of axes unspecified to imply the full domain).When limits are not explicit Ferret misses opportunities for reusing cachedresults -- slower processing and potentially more memory needed due tofragmentation may result.

Explicit limits are especially important when you see "gathering" (above).Watch the diagnostic output to see if Ferret is re-reading the same data.If so, consider using LOAD (and LOAD/PERMANENT) to read file variablesinto memory prior to requesting a result. This is especially useful whencalculations involve derivatives and shift operators. For example, usingMY_VAR[I=101:110] and MY_VAR[I=101:110@SHF:2] in the same calculation mayrequire two separate reading operations; pre-reading with LOAD MY_VAR[I=101:112]will cache a single block of data big enough for both.

Example:

Here's a fragment of MODE DIAGNOSTIC output and some explanation:

yes? SET MODE DIAGNOSTICyes? LOAD SST[l=1:400@AVE] getgrid EX#1 5 D: 2 I: 1 1 J: 1 1 K: 1 1 L: 1 1 eval EX#1 4 D: 2 I: -111 -111 J: -111 -111 K: -111 -111 L: -111 -111 strip gathering SST on Y axis: 1 90 strip -->; SST[T=01-JAN-1946:01-MAY-1979@AVE,D=2] reading SST 3 D: 2 I: 1 180 J: 1 4 K: -111 -111 L: 1 400 doing -->; SST[T=01-JAN-1946:01-MAY-1979@AVE,D=2] doing gathering SST on Y axis: 1 4

Explanation:

  • D: 2 tells us that the operations are from data set#2
  • the getgrid pass determines the grid of the result
  • eval EX#1 is the start of evaluation for the first expression (theremay be multiple comma-separated expressions)
  • - I: -111 -111 means that the limits on the I (X) axis are IMPLIED(unknown). Explicit limits may improve performance and memory management.
  • strip means that Ferret has detected the need for an operation andallocated space for it on a stack.
  • strip gathering SST on Y axis means this calculation is being brokeninto slices along the Y axis (to fit into memory)
  • reading SST ... means IO is in progress. The I,J,K,L limits areshown. You may see delays during large IO operations.
  • doing tells us the operation that was "stripped" above is beingdone
  • gathering SST on Y axis: 1 4 tells us that theJ=1:4 slice of the calculation has been completed. J=5:8 will probablyfollow.