The algorithm has three controlling parameters:
How shall we choose the values of these parameters? There are two competing constraints: reliability and efficiency. Does the algorithm find all maxima, and how much time does it take to do so? Both quantities are taken in comparison to a grid search on a dense regular grid, which is most reliable and least efficient.
An analytic treatment of the properties of this algorithm is out of bounds, so I ran a parameter study with random stair functions
The measured quantities were defined in comparison to a regular grid: The question is how good the algorithm performs with respect to the most reliable and least efficient algorithm. The true function could be used as reference for the determination of reliability, but efficiency is is more useful to have a comparison with another algorithm. The regular grid consisted of function samples in a distance equal to the minimum distance between samples of the importance sampling.
The algorithm properties to be determined are thus
A random step function is constructed from 10 function samples in the depth range 0-700km. The change of the function value at each of the samples is restricted to +1, 0 or -1 (or, in later test series, to integers from -n to +n, including 0) the function as a whole is restricted to positive values. All step heights are equally likely. Large function jumps are possible only at depth 0 and at depth 700. (It is assumed that real DOC functions will be relatively "smooth", e.g. have no large jumps). The distance between function samples is at least twice the smallest sampling distance allowed for the search algorithm to guarantee that the algorithm has a chance to find all maxima. There is no upper limit for sample distance, except the implicit limit that all samples have to be in the given depth range.
The maxima of the step function are then searched by the non-uniform algorithm with prescribes control parameters n, ε and δ (δ is not modified throughout the test). A uniform grid search is also performed, using the smallest sample distance of the non-uniform smapling as grid spacing. The number of maximal function values obtained in both methods is compared to obtain the reliability of the non-uniform search, and the number of function evaluations in both methods is counted to obtain its efficiency.
Each individual step function is evaluated only once: a repeated evaluation is not necessary, since the algorithm is deterministic.For each combination of n and ε, a number N of random step function is generated and optimized to obtain a statistics of the performance in the light of varying objective functions. Thus, For every combination of n and ε, a list of N reliability and efficiency values if obtained that can be evaluated statistically.
Such lists can be drawn individually as distribution densities, and statistical parameters of the lists can be plotted as functions of the algorithm's controlling parameters. Such functions are used to decide for the most favorable values of the control parameters.
The relevant LOCSMITH routines are:
Inspection of individual distribution densities of reliability and efficiency (individual means: for a given pair of n and ε) shows that reliability is generally very high: in almost any case, the algorithm finds all maxima. Efficiency, on the other hand, is a real statistic quantity and the distribution covers a wide range.
To make several runs comparable, the median efficiency, the mean reliability (in this special case, the median does not make very much sense: usually, the median guarantees that 50% of all values are smaller than the median and 50% are larger. But this is not the case if the distribution is like shown here: the median of {10 20 30 100 100 100 100 100} is 100, but less than 50% of the values are smaller than 100. The median is therefore misleading. The mean of this list is, on the other hand, 70 and gets closer to 100 if the number of "100"s increases), and the standard error (standrad deviation divided by sqrt(N)) of both. The difference of mean and median is also computed for both efficiency and reliability: if this difference becomes large, the mean is not a meaningful descriptor for the distribution.
The reliability turns out to be almost perfect and independent of the control parameters. The only area of the mean reliability function where it is clearly below 100% is also an area, where the mean can be identified as sub-optimal descriptor for the statistics, since the distance between median and mean becomes larger for the corresponding control parameters. This means that the reduction of the realiability is only an artefact of the statistics. The median reliability is 100% in all cases. In other words: the reliability for "smooth" step functions is almost perfect and control parameters can be chosen with respect to efficiency only.
(You may ask yourself why I didn't plot the median reliability when it's so fantastic. The reason is that the median does not show that there is some variation in the distributions: the number of undiscovered function maxima does increase and the reliability gets slightly worse, but the median is too robust to notice it)
The tradeoff between reliability and efficiency is shown in the following plot.
But as shown in the previous figures, the mean reliability becomes a poor measure especially at those places where it becomes smaller, and, in fact, the median reliability is 100% in all cases! This means that the reliability was 100% in more than 50% of all tests, independent of the acceptance threshold or the number of initial intervals. The question, therefore, is not how reliable the algorithm is, but in how many more than 50% of all cases it was perfect.
This has been measured independently for the two cases in which the reliability changes most (stairs with 10 samples of max. height 10) and least (5 samples, max height 1) with control parameters.
Obviously, the failure rate (the number of runs, in which not all maxima have been found, or in which the mapping of found maxima was incomplete) is less than about 1/3 of all runs and essentially independent of both the number of initial intervals (or their relative length) and the acceptance threshold. Even the outliers do not show a clear picture in the sense that outliers are furthest out for a certain control parameter value. The only dependency is that on the complexity of the step function, but from the two test scenarios used here it is difficult to understand how this dependency works.
The conclusion from these tests is that acceptnce threshold and number of initial intervals cna be chosen with respect to efficiency alone.
Considered as function of the relative length of initial intervals, the scatter of efficiency values is much greater than the decrease which follows from the obvious observation that with many initial intervals, the algorithms becomes closer and closer to a uniform search.
The dependence of efficiency on the acceptqance threshold is much clearer. The higher this threshold, the higher the efficiency. Only for matchthresh values below 20% and above 80%, the gradient gets smaller.Taking the scattering into account, the best efficiency is obtained if the acceptance threshold is set to 100%, which is also easy to understand, since smaller thresholds mean that more new points are generated that need time to be evaluated.
The default value for lsopt.iter_vertical_matchthresh is set to 100%.
To define the number of initial depth samples, several lsopt-values have to be set. In the case of triangular grids, these are:
The efficiency of the algorithm can be made arbitrary good by setting δ, the vertical resoultion used as stop criterion, to very small values. This is, however, not a very good idea, since this maximizes only the efficiency compared to uniform sampling with sample distance δ, which is very poor. The non-uniform sampling with small δ will have very high relative efficiency, but nevertheless it will be expensive in terms of CPU time. Parameter δ should not be mad unnecessarily small, depending on the depth resolution expected from the data.