Signal/noise Analysis In Perimetry

Signal/noise analysis to compare tests for measuring visual field loss, and its progression. short title Signal/noise ratios in perimetry words, figures, tables 3,650 words, 11 figures, 3 tables. IOVS proofs signed off Aug.’09 section & previous Section code GL, previously shown at the International Perimetric Society presentations (IPS) meeting in Portland, Oregon in 2006 keywords perimetry, visual field, sensitivity, progression, glaucoma, FDT, frequency doubling, Humphrey-Matrix

www.wordle.net (courtesy of J Feinstein)

authors & affiliation Paul H Artes & Balwantray C Chauhan Ophthalmology and Visual Sciences Dalhousie University, Rm 2035, West Victoria, 1276 South Park St, Halifax, Nova Scotia B3H 2Y9, Canada [email protected] support No commercial relationships, support from the E A Baker Foundation (PHA), and the Canadian Institutes of Health Research (BCC, MOP-11357)

1

Artes & Chauhan, Signal/noise ratios in perimetry

Abstract Purpose To describe a methodology for establishing signal/noise ratios (SNRs) for different perimetric techniques, and to compare SNRs of Frequency-Doubling Technology (FDT2) perimetry and standard automated perimetry (SAP).

Methods Fifteen patients with open-angle glaucoma (median MD, -2.6 dB, range +0.2 to -16.1 dB) were tested 6 times with FDT2 and SAP (SITA Standard program 24-2) within a 4 week period. Signals were estimated from the average superior-inferior difference between the Mean Deviation (MD) values in 5 mirror-pair sectors of the Glaucoma Hemifield Test, and noise from the dispersion of these differences over the 6 repeated tests. SNRs of FDT2 and SAP were compared by mixed effects modelling.

Results There was a moderate correlation between the signals of FDT2 and SAP (r2=0.68, p<0.001) but no correlation of the noise (r2=0.01, p=0.16). Although both signal as well as noise estimates were larger with FDT2 compared to SAP, 60-70% of sector pairs showed higher SNRs with FDT2. The SNRs of FDT2 were between 20% and 40% larger than those of SAP (p=0.01). There were no meaningful differences between parametric and non-parametric estimates of signal, noise, or SNR.

Conclusion The larger SNRs of FDT2 suggest that this technique is at least as efficient as SAP at detecting localised visual field losses. Signal/noise analyses may provide a useful approach for comparing visual field tests independent of their dB-scales and may provide an initial indication of sensitivity to visual field change over time.

2

Artes & Chauhan, Signal/noise ratios in perimetry

Introduction Visual field loss and its progression are hallmarks in glaucoma, optic neuritis, and other diseases.1 However, clinically important “signals” in the visual field are often small compared to the variability between successive tests (“noise”). This applies to the detection of abnormalities with single examinations as well as to the measurement of change over time with serial examinations. Consequently, tests may have to be repeated several times before one can be certain that damage is either present or absent, and at least 5 tests are required to detect change with any confidence.2-7 With typical variability, at least 3 years of 6-monthly tests are needed to detect sight-threatening rates of visual field progression.8, 9 Previous investigators have aimed to reduce variability through optimised threshold strategies,10-14 closer control over factors such as response bias, fatigue, and attention,15-19 and most notably through new types of stimuli.20-24 Studies on retest variability, and on response variability estimated by psychometric functions, have shown that several of the newer techniques do not suffer from the large increase in variability in damaged fields seen with SAP but have nearly uniform variability across their dynamic range.25-29 However, it is challenging to compare visual field data between one technique and another. It is difficult, for example, to comparing threshold estimates from SAP to those of motion perimetry, and although the technique’s scales can be made to appear similar by empirical “correction factors”, this is more likely to conceal the problem rather than to solve it. Second, when visual fields change over time, a 10 dB change with one technique may translate into a smaller or larger change with another technique, and this relationship may not be constant but vary with the degree of damage. Moreover, techniques differ in their measurement ranges – in a damaged area of the visual field one technique may still provide useful threshold estimates while another only measures absolute losses (0 dB) that are not informative for determining change.28 In combination, these issues limit the usefulness of current methods for comparing different techniques. Direct evidence that one technique performs better than another can only be obtained through comparative studies with substantial numbers of patients, but even these studies do not always give conclusive results. Because normal reference data for different techniques are usually obtained from different samples of healthy controls, it can be difficult to compare probability maps from different techniques.30, 31 Longitudinal studies are needed to compare effectiveness in measuring change over time, but such studies are costly and often take several years.32-34 They may also be difficult to interpret because no single technique provides an ideal reference standard.35 A method is therefore needed that can provide clues to the potential merit of a new technique within a relatively short period of time.

3

Artes & Chauhan, Signal/noise ratios in perimetry

In this work we propose a simple extension for analyses of retest data and demonstrate how perimetric techniques may be compared even if they use different types of stimuli that do not share the same dBscale. The underlying rationale of this analysis is to compare the ability of the techniques to measure systematic differences within a visual field, for example the asymmetries between the superior and inferior areas that are characteristic of glaucomatous visual field loss.36 Differences in sensitivity, or deviation from normal, between the superior and inferior mirror-pairs of sectors within a visual field can be interpreted as a signal, and the variability of these differences from test to test can be interpreted as noise. By estimating the ratio between signal and noise (signal/noise ratio, SNR), the ability of the technique to identify localised visual field loss may be expressed independent of the dB-scale of the instrument, enabling a paired comparison to be made between perimetric techniques.

Methods Data Fifteen patients with open-angle glaucoma (mean age, 66.3 years, range, 56.1-80.6 years) who had early to moderately advanced visual field loss with SAP (median MD, -2.6 dB, range +0.2 to -16.1 dB) were recruited from the clinics of the QEII Health Sciences Centre (Halifax, Nova Scotia, Canada). Inclusion criteria were a clinical diagnosis of open-angle glaucoma, refractive error within 5 D equivalent sphere or 3 D astigmatism, visual acuity better than or equal to 6/12 (+0.3 logMAR), and prior experience with FDT1 perimetry and SAP. During a period of 4 weeks, one eye of each patient was tested 6 times with FDT2 (24-2 threshold test) and 6 times with SAP (SITA Standard; 24-2 test), in randomized order. The protocol was approved by the Queen Elizabeth II Health Science Centre Research Ethics Committee, and all participants had given written informed consent. Full details of this dataset are described in a previous paper on threshold and variability properties of the Humphrey-Matrix (FDT2) perimeter.26

Analysis The threshold values of FDT2 and SAP were transformed to total deviation values using reference data from healthy volunteers.37 For each test, we then averaged the total deviation values within the 10 visual field sectors of the Glaucoma Hemifield Test 36 to obtain 10 sectoral Mean Deviation (sMD) values (Fig. 1). Reduced major axis (RMA) regression 38 was used to estimate the relationship between the sMDs of FDT2 and SAP. Unlike ordinary-least-squares regression, which assumes that the x-values are from an “independent” variable measured without error, the line fitted by RMA regression minimizes the residuals in both vertical as well as horizontal directions; it is therefore a more appropriate method for establishing the slope of the relationship between FDT2 and SAP.

4

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 1) Stimulus locations of FDT2 (grey squares) and SAP (black dots). The five visual field sectors in the superior hemifield which are compared with their mirror images in the lower hemifield are outlined in red. The location of the blind spot is shown by the black square and white dot.

Signal / Noise analysis Because each patient had been examined 6 times, there were 6 sMD values, one for each sector, for both SAP and FDT2. For each patient, we calculated the superior-inferior difference in sMD between the mirror sectors in each test, and in all 30 combinations of the 6 tests, so that 36 differences were obtained for each patient, each sector, and each of the two techniques (SAP and FDT2). Estimates of signal and noise were then derived from the mean and the population standard deviation (SD) of the distribution of differences (n=36), respectively (Fig. 2). Because a single outlying data point can have an unduly large influence on both the mean and the SD, we also computed non-parametric estimates by replacing the mean with the median, and the SD with the Median Absolute Deviation (MAD)39 of the differences. This is described in the Appendix, along with Bland-Altman comparisons of parametric and non-parametric estimates. Signal, noise, and signal/noise ratios (SNRs) were then compared between SAP and FDT2, in each of the 5 pairs of sectors and each of the 15 patients. All analyses were performed in the freely available open-source environment R.40, 41 The nlme library was used to estimate statistical significance and confidence intervals; patients were treated as random factors to adjust for the non-independence of the 5 sector pairs within each individual patient.42

5

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 2) Example illustrating how signal and noise were derived. a) In the upper and lower arcuate sectors, total deviation values of each test were averaged to sector MDs. b) From the 6 repeated tests, 6 sMDs were obtained in each of the sectors. c) By enumerating all possible combinations of the 6 test, a distribution of 36 differences between the upper and lower sectors was obtained , and d) estimates for signal (black arrow head) and noise (curly bracket) were derived from the average and the dispersion of this distribution, respectively.

6

Artes & Chauhan, Signal/noise ratios in perimetry

Results The sectoral MD values in the 10 visual field sectors ranged from -27.8 dB to +1.5 dB (median, -2.3 dB) with SAP, and from -26.2 dB to +3.3 dB (median, -2.4 dB) with FDT2. The relationship between SAP and FDT accounted for 69% of the variance in the data (Spearman rank correlation, p<0.001, Fig. 3), and, for sectors with SAP sMD better than -10 dB, the relationship between SAP and FDT2 appeared linear (Tukey’s test for additivity,43 p=0.22) with a slope of 2.1 (RMA regression, 95% C.I, 1.9-2.3). For sectors with more advanced damage, however, the relationship between the sMDs of both techniques became progressively weaker and deviated significantly from that observed at less damaged sectors (p<0.001, Tukey’s test). Despite the averaging of 6 repeated tests with both techniques, the data exhibited a large degree of scatter (Fig. 3). Fig. 3) Relationship between sectoral MD values obtained with SAP and FDT2. Each data point represents the mean of 6 repeated tests with FDT2 and SAP. For sectors with SAP MDs better than -10 dB (filled dots), RMA regression estimated a slope of 2.1 (95% C.I., 1.9, 2.3). The 1:1 line is shown for comparison.

Signal and noise estimates were derived from the mean and the SD of the superior-inferior differences (Fig. 2), with both FDT2 and SAP. Three selected examples are shown in Figs. 4-6.

7

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 4) Example A In this visual field with mild damage, a 2 dB asymmetry between the nasal superior and inferior sectors was measured with FDT2 but not with SAP. The dispersion of the differences (noise) was similar with SAP and FDT2 (0.9 and 0.7 dB, respectively), and FDT2 had a larger SNR (3.0 compared to 0.2 with SAP). The vertical grey bar (noise) represents ±2 SD to indicate if the mean of the differences is significantly different from zero (dashed grey line).

8

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 5) Example B In this extensively damaged visual field, both SAP and FDT2 revealed a small difference (2.8 and 4.2 dB, respectively) between the superior and inferior paracentral sectors. With SAP, the large variability of both sectors (SD, 3.7 dB) made it difficult to distinguish this signal (SNR, 0.8). With FDT2, the asymmetry was more clearly apparent (SNR, 3.1), chiefly because of lower variability (1.3 dB). The vertical grey bar (noise) represents ±2 SD to indicate if the mean of the differences is significantly different from zero (dashed grey line).

9

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 6) Example C In this visual field, both SAP and FDT indicated substantial damage. With SAP, an asymmetry of 6 dB was clearly apparent in the nasal sectors (SNR > 2), despite large variability in the more damaged inferior sector. Despite lower variability (1.3 dB), there was no detectable signal with FDT2 (SNR, 0.3). The vertical grey bar (noise) represents ±2 SD to indicate if the mean of the differences is significantly different from zero (dashed grey line).

10

Artes & Chauhan, Signal/noise ratios in perimetry

There was a moderately close relationship between the signals of both techniques (Fig 7a, r2 = 0.52, p<0.001), but no such relationship between the noise estimates (r2=0.01, p=0.16, Fig. 7b). Fig. 7) Relationship between superior/inferior differences (signal) of SAP and FDT2 (a) and noise (b). Axes are drawn on a squareroot scale to emphasize mid-range values.

Although both signal and noise appeared numerically larger with FDT2 compared to SAP (Table 1), a direct comparison between the estimates is difficult to interpret because they are expressed in instrument-specific dB-units.

Table 1. Distribution of signal and noise estimates from FDT2 and SAP. P-values and confidence intervals were established by mixed effects modelling because each patient contributed 5 estimates.

Table 1

FDT2 (dB)

SAP (dB)

signal mean, median [range]

3.5, 1.9 [0.1 – 15.4]

51/75 (68%) 1.40 [1.06, 1.85] 2.8, 1.2 [0.0 – 16.6]

0.02

noise

1.7, 1.6 [0.6 – 3.0]

1.5, 1.3 [0.5 – 5.2]

48/75 (64%) 1.17 [0.97, 1.42]

0.10

mean, median [range]

FDT2>SAP

FDT2/SAP

p-value

However, by calculating the ratio between signal and noise, the instrument-specific units in numerator and denominator cancel each other such that the resulting signal/noise ratio (SNR) is independent of the dB-scale of the instrument. There was a moderately close association between the SNRs of SAP and FDT2 (r2=0.68, p<0.001, Fig. 8). Of the 75 sector pairs, 46 (61%) had a larger SNR with FDT2. On average, the SNRs of FDT2 were 19% larger (95% C.I., -6%, 52%, p=0.14) than the corresponding SNRs of SAP.

11

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 8) Relationship between signal-to-noise ratios with SAP and FDT2. The points labelled A, B, C correspond to the examples in Figs. 4, 5, and 6. Axes are drawn on a square-root scale to emphasize mid-range values.

Sector pairs within which there were no differences in damage contribute little information, but they may dilute genuine differences between the techniques if they are included in an overall comparison. To reduce this dilution while avoiding any selection bias, we compared the SNRs of SAP and FDT2 for all those pairs in which either SAP or FDT2 had an SNR greater than 0.5, 1.0 and 2.0 (Table 2). Table 2: Comparison between SNRs of FDT2 and SAP. A ratio >1 in column 2 means that FDT2 provided a larger SNR than SAP. Column 3 gives the number of pairs in which the SNR of FDT2 was greater than that of SAP. P-values and confidence intervals were established by mixed effects modelling because each patient contributed 5 estimates.

SNR

Ratio of SNR (FDT2/SAP)

SNR FDT2 > SAP / total

p-value

>0.5

1.20 [0.93 - 1.56]

66/71 (93%)

p=0.15

>1.0

1.43 [1.05 – 1.94]

34/50 (68%)

p=0.03

>2.0

1.39 [1.04 – 10.6]

22/30 (73%)

p=0.02

Of the sector pairs with SNRs > 0.5 with either FDT2 or SAP, between 68-93% had larger SNRs with FDT2 compared to SAP (Table 2, column 3). However, these findings were statistically significant (p<0.05) only for the subset of SNRs which were greater than 1.0. In these sector-pairs, the SNRs of FDT2 were approximately 40% larger than the corresponding SNRs of SAP (Table 2, column 2).

12

Artes & Chauhan, Signal/noise ratios in perimetry

Discussion The aim of this paper was to develop an approach for comparing perimetric techniques independent of their measurement scales, and to apply this methodology to visual field data from the Humphrey-Matrix (FDT2) perimeter. In a previous study,26 we demonstrated that the variability characteristics of both techniques were qualitatively different – threshold estimates from FDT2 had nearly uniform variability across the measurement range of the instrument, while those of SAP showed an exponential increase in variability with decreasing sensitivity. Similar results were obtained in other studies, both with FDT127, 29and more recently with FDT2.28 We were then, however, unable to make a quantitative comparison of the variability, because both instruments use different types of stimuli and different definitions of the dBscales. The lack of a general method for comparing threshold data from different perimetric tests motivated the signal/noise analyses performed in this paper. By relating systematic differences within a visual field (signal) to the precision with which such differences can be measured (noise), techniques can be compared independent of their underlying measurement units. Our data showed a substantial correlation between the signals of the two techniques (r2=0.52), but no such correlation for the noise. The lack of a relationship between the noise estimates of FDT2 and SAP clarifies why, in some patients, either technique may have systematic advantages for measuring losses that are less detectable with the other.26, 31, 44 While our dataset of 15 patients is too small to carry out a meaningful subgroup analysis, we suggest that the signal/noise methodology proposed in this paper provides a useful framework for studying systematic differences between different perimetric techniques. It is particularly important to establish factors that contribute to the large scatter apparent in Fig. 3. Because 6 tests had been averaged for each data point, it is unlikely that this scatter can be explained solely by measurement variability of FDT2 and SAP. For sectors with SAP MDs better than -10 dB, the slope of the relationship between the sectoral MD values of FDT2 and SAP was 2.1 (Fig. 3). This closely mirrors the findings reported in our earlier paper in which we compared threshold estimates from individual test locations, and it is also in agreement with the slope of 2.0 expected from the different definitions of the dB-scale of both techniques.45 With FDT2, a dB is defined as -20 log10 of Michelson contrast such that a change of 20 dB corresponds to a 1 log unit change in contrast, whereas with SAP, a dB is defined as -10 log10 of Weber contrast, such that a change of 20 dB corresponds to a change of 2 log units. Importantly, the empirically determined slope ~2 would suggest that the magnitude (in dB) of early and moderate visual field losses, and changes over time, could be up to twice as large with FDT2 as compared to SAP. In our data, the signals (superiorinferior differences between the mirror-pairs of visual field sectors), were on average only 40% larger

13

Artes & Chauhan, Signal/noise ratios in perimetry

with FDT2 compared to SAP, but this average would have been reduced by sector pairs between which there were no meaningful differences in damage and therefore no measurable signals. Approximately 70% of sector pairs with an SNR > 1.0 with either SAP or FDT2 had higher signal/noise ratios with FDT2 (Table 2). This is evidence of an overall gain and confirms that the benefits of the larger FDT2 signals are not offset by the larger variability of this technique (Table 1). For pairs with SNRs > 1.0, the SNRs of FDT2 were approximately 40% larger than those of SAP. This difference is similar in magnitude to the improvement in SNR that would be expected from repeating a test, since performing the same test twice can reduce the measurement variability, in theory, by a factor of √2 (1.41). A difference of this magnitude would mean a substantial net improvement in the detectability of early changes, for cross-sectional detection of visual field loss as well as for longitudinal measurement of visual field progression. For the former, empirical investigations on total- and pattern deviation probability maps31, 44, 46 with FTD2 and SAP, as well as global visual field indices such as pattern standard deviation,47 are in agreement with our results. How may signal/noise ratios, calculated from retest data, help to estimate performance in measuring progression? The rationale for using gradients in space as a surrogate for changes over time is illustrated in Fig. 9. Progression of visual field loss is a change in sensitivity over time, and the usefulness of a test for following patients over time depends on how well its data reflect these changes. In contrast, signal/noise ratios estimated from tests performed within a short period of time express the detectability of differences within a visual field at that particular time. Fig. 9) a) The SNR measured between locations At1 and Bt1 measures how reliably the technique represents the gradient of damage in space (vertical arrow). b) If B deteriorates over time, such that its deviation at Bt2 becomes equal to that of At1, the gradient in time between Bt1 and Bt2 is equal to that between At1 and Bt1. The ability to detect the change from Bt1 to Bt2 should be similar to that measured by the SNR between At1 and Bt1.

SNRs express how reliably a technique reflects gradients of damage within a visual field, and this is a function of the depth of loss, the variability of the measurements, and the dynamic range of the technique. A larger SNR therefore does not necessarily mean that the technique is more sensitive than another, nor does a more sensitive technique necessarily provide a larger SNR (Fig. 10).

14

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 10) Relationship between signals of 2 techniques A and B when A is more sensitive than B. a) A reveals superior loss (signal at nasal step, vertical arrow). There is no signal with B. b) A reveals more extensive loss than B but has reached the limit of its dynamic range; its signal is small compared to B. c) Owing to its larger dynamic range, B continues to provide signal even though both superior and inferior sectors are damaged.

The signal/noise methodology proposed in this paper has several limitations. To obtain robust estimates of signal and noise, multiple tests have to be performed. Nevertheless, a rigorous protocol with at least 5 examinations per patient has advantages also for the derivation of test-retest intervals, because a large number of combinations of test-retest examinations can be analysed.26, 48 SNRs depend on the sample of patients and cannot be compared across different studies. They also depend on the somewhat arbitrary choice of where in the visual field the signal and noise distributions are derived from. In this study, we used the superior-inferior sectors of the Glaucoma Hemifield Test, and therefore our finding of larger SNRs with FDT2 may strictly apply only to those analyses which make use of a similar clustering. In principle, however, other pairs of test locations or pairs of clusters could not be chosen. Finally, SNRs can be estimated only if focal losses are present in the visual field; diffuse reductions in sensitivity do not contribute a signal. As a consequence, the method is unsuitable for evaluating techniques that predominantly uncover diffuse loss. The assumption that gradients in space can be used as a first approximation for change over time appears reasonable but is, as yet, untested. Signal/noise estimates from test-retest studies will therefore not replace longitudinal studies for investigating new visual field tests’ ability to monitor patients with glaucoma, but they may provide early insight into properties that cannot be gained from analyses of testretest variability. They may help in hypothesis-building and in planning effective longitudinal studies of new visual field tests. 15

Artes & Chauhan, Signal/noise ratios in perimetry

Appendix Mean and SD are parametric estimates of central tendency and dispersion and can be highly affected by outliers, particularly so in small samples. Compared to mean and SD, the non-parametric median and Median Absolute Deviation (MAD) are more robust to outliers, but they are also less efficient (more variable) when there are no outliers. To investigate whether there were meaningful differences between parametric and non-parametric estimates of signal, noise, and SNRs, we computed non-parametric estimates of signal and noise from the median and MAD. The MAD was scaled by a factor of 1.483 to make it similar to the SD in a Normal distribution.49 It should be noted that the enumeration of all possible 36 differences between the superior and inferior sMDs as explained in Fig. 2 of the Methods section needs to be performed only to compute the nonparametric estimates. For the parametric estimates, the mean of the 36 differences is identical to the difference between the means of the 6 sMDs in the superior and inferior sectors (1), and the SD of the differences is identical to that obtained from pooling the SDs of the superior and inferior sectors (2). (1)

xdiff = xsup ! xinf

(2)

! diff = 2 (! sup ) 2 + (! inf ) 2

The Bland-Altman plots in Fig. 11 show good overall agreement between the parametric and nonparametric estimates of signal and noise. With both FDT2 and SAP, noise estimates >2 dB were systematically smaller with the non-parametric method (Fig. 11 c, d; p<0.10). The discrepancies observed in the SNR ratios of FDT2 increased with the magnitude of the SNR (Fig. 11f). However, none of our findings changed substantially when the analyses were performed with the non-parametric SNRs instead of the parametric alternatives.

16

Artes & Chauhan, Signal/noise ratios in perimetry

Fig. 11) Bland-Altman plots of parametric and nonparametric estimates of signal, noise, and signal/noise ratio with SAP and FDT2. Points on the horizontal line show perfect agreement between parametric and nonparametric estimates, and the shaded area encloses 95% of the differences.

17

Artes & Chauhan, Signal/noise ratios in perimetry

References 1.

Anderson DR, Patella VM. Automated Static Perimetry: Mosby St. Louis; 1999.

2.

Spry PGD, Johnson CA. Identification of progressive glaucomatous visual field loss. Survey of Ophthalmology 2002;47:158-173.

3.

Keltner JL, Johnson CA, Levine RA, et al. Normal visual field test results following glaucomatous visual field end points in the Ocular Hypertension Treatment Study. Archives of Ophthalmology 2005;123:1201-1206.

4.

Keltner JL, Johnson CA, Quigg JM, Cello KE, Kass MA, Gordon MO. Confirmation of visual field abnormalities in the ocular hypertension treatment study. Archives of Ophthalmology 2000;118:1187-1194.

5.

Heijl A, Leske MC, Bengtsson B, et al. Measuring visual field progression in the early manifest glaucoma trial. Acta Ophthalmologica Scandinavica 2003;81:286-293.

6.

Gardiner SK, Crabb DP. Examination of different pointwise linear regression methods for determining visual field progression. Investigative Ophthalmology and Visual Science 2002;43:1400-1407.

7.

Vesti E, Johnson CA, Chauhan BC. Comparison of Different Methods for Detecting Glaucomatous Visual Field Progression. Investigative ophthalmology & visual science 2003;44:3873-3879.

8.

Gardiner SK, Crabb DP. Frequency of testing for detecting visual field progression. British Journal of Ophthalmology 2002;86:560-564.

9.

Chauhan BC, Garway-Heath DF, Goñi FJ, et al. Practical recommendations for measuring rates of visual field change in glaucoma. British Journal of Ophthalmology 2008;92:569-573.

10.

Bengtsson B, Olsson J, Heijl A, Rootzén H. A new generation of algorithms for computerized threshold perimetry, SITA. Acta Ophthalmologica Scandinavica 1997;75:368-375.

11.

Vingrys AJ, Pianta MJ. A new look at threshold estimation algorithms for automated static perimetry. Optometry and Vision Science 1999;76:588-595.

12.

Anderson AJ, Johnson CA. Comparison of the ASA, MOBS, and ZEST threshold methods. Vision Research 2006;46:2403-2411.

13.

Turpin A, McKendrick AM, Johnson CA, Vingrys AJ. Performance of efficient test procedures for frequency-doubling technology perimetry in normal and glaucomatous eyes. Investigative Ophthalmology and Visual Science 2002;43:709-715.

14.

Schiefer U, Pascual JP, Edmunds B, et al. Comparison of the new perimetric GATE strategy with conventional full-threshold and SITA standard strategies. Investigative ophthalmology & visual science 2009;50:488-494.

15.

Frisén L. Computerized perimetry: Possibilities for individual adaptation and feedback. Documenta Ophthalmologica 1988;69:3-9.

16.

Frisen L. Perimetric variability: importance of criterion level. Doc Ophthalmol 1988;70:323330.

17.

Kutzko KE, Brito CF, Wall M. Effect of Instructions on Conventional Automated Perimetry. Investigative ophthalmology & visual science 2000;41:2006-2013.

18.

Wall M, Woodward KR, Brito CF. The effect of attention on conventional automated perimetry and luminance size threshold perimetry. Investigative ophthalmology & visual science 2004;45:342-350. 18

Artes & Chauhan, Signal/noise ratios in perimetry

19.

Miranda MA, Henson DB. Perimetric sensitivity and response variability in glaucoma with single-stimulus automated perimetry and multiple-stimulus perimetry with verbal feedback. Acta Ophthalmol Scand 2007.

20.

Anderson RS. The psychophysics of glaucoma: Improving the structure/function relationship. Progress in Retinal and Eye Research 2006;25:79-97.

21.

McKendrick AM. Recent developments in perimetry: test stimuli and procedures. Clinical & experimental optometry : journal of the Australian Optometrical Association 2005;88:73-80.

22.

Monhart M. What Are the Options of Psychophysical Approaches in Glaucoma? Survey of Ophthalmology 2007;52.

23.

Wall M. What's New in Perimetry. Journal of Neuro-Ophthalmology 2004;24:46-55.

24.

Fogagnolo P, Rossetti L, Ranno S, Ferreras A, Orzalesi N. Short-wavelength automated perimetry and frequency-doubling technology perimetry in glaucoma. Progress in Brain Research; 2008:101-124.

25.

Hot A, Dul MW, Swanson WH. Development and evaluation of a contrast sensitivity perimetry test for patients with glaucoma. Investigative ophthalmology & visual science 2008;49:30493057.

26.

Artes PH, Hutchison DM, Nicolela MT, LeBlanc RP, Chauhan BC. Threshold and variability properties of matrix frequency-doubling technology and standard automated perimetry in glaucoma. Investigative ophthalmology & visual science 2005;46:2451-2457.

27.

Chauhan BC, Johnson CA. Test-retest variability of frequency-doubling perimetry and conventional perimetry in glaucoma patients and normal subjects. Investigative Ophthalmology and Visual Science 1999;40:648-656.

28.

Wall M, Woodward KR, Doyle CK, Artes PH. Repeatability of Automated Perimetry: A Comparison between Standard Automated Perimetry with Stimulus Size III and V, Matrix, and Motion Perimetry. Invest Ophthalmol Vis Sci 2009;50:974-979.

29.

Spry PGD, Johnson CA, McKendrick AM, Turpin A. Variability components of standard automated perimetry and frequency-doubling technology perimetry. Investigative Ophthalmology and Visual Science 2001;42:1404-1410.

30.

Anderson AJ, Johnson CA. Anatomy of a supergroup: does a criterion of normal perimetric performance generate a supernormal population? Investigative ophthalmology & visual science 2003;44:5043-5048.

31.

Sakata LM, DeLeon-Ortega J, Arthur SN, Monheit BE, Girkin CA. Detecting visual function abnormalities using the Swedish interactive threshold algorithm and matrix perimetry in eyes with glaucomatous appearance of the optic disc. Archives of Ophthalmology 2007;125:340-345.

32.

Chauhan BC, House PH, McCormick TA, LeBlanc RP. Comparison of Conventional and HighPass Resolution Perimetry in a Prospective Study of Patients With Glaucoma and Healthy Controls. Am Med Assoc; 1999:24-33.

33.

Johnson CA, Adams AJ, Casson EJ, Brandt JD. Progression of early glaucomatous visual field loss as detected by blue- on-yellow and standard white-on-white automated perimetry. Archives of Ophthalmology 1993;111:651-656.

34.

Bayer AU, Erb C. Short wavelength automated perimetry, frequency doubling technology perimetry, and pattern electroretinography for prediction of progressive glaucomatous standard visual field defects. Ophthalmology 2002;109:1009-1017.

35.

Haymes SA, Hutchison DM, McCormick TA, et al. Glaucomatous Visual Field Progression with Frequency-Doubling Technology and Standard Automated Perimetry in a Longitudinal Prospective Study. Investigative ophthalmology & visual science 2005;46:547-554. 19

Artes & Chauhan, Signal/noise ratios in perimetry

36.

Asman P, Heijl A. Glaucoma Hemifield Test. Automated visual field evaluation. Archives of Ophthalmology 1992;110:812-819.

37.

Anderson AJ, Johnson CA, Fingeret M, et al. Characteristics of the Normative Database for the Humphrey Matrix Perimeter. Invest Ophthalmol Vis Sci 2005;46:1540-1548.

38.

Warton DI, Wright IJ, Falster DS, Westoby M. Bivariate line-fitting methods for allometry. Biological Reviews 2006;81:259-291.

39.

Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307-310.

40.

R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing 2004.

41.

Ihaka R, Gentleman R. R: A Language for Data Analysis and Graphics. Journal of Computational and Graphical Statistics 1996;5:299-314.

42.

Pinheiro JC, Bates DM. Mixed-Effects Models in S and S-Plus: Springer; 2000.

43.

Weisberg S. Applied Linear Regression: John Wiley & Sons New York; 1985.

44.

Racette L, Medeiros FA, Zangwill LM, Ng D, Weinreb RN, Sample PA. Diagnostic Accuracy of the Matrix 24-2 and Original N-30 Frequency-Doubling Technology Tests Compared with Standard Automated Perimetry. Invest Ophthalmol Vis Sci 2008;49:954-960.

45.

Sun HAO, Dul MW, Swanson WH. Linearity Can Account for the Similarity Among Conventional, Frequency-Doubling, and Gabor-Based Perimetric Tests in the Glaucomatous Macula. Optometry and Vision Science 2006;83:E455.

46.

Leeprechanon N, Giangiacomo A, Fontana H, Hoffman D, Caprioli J. Frequency-Doubling Perimetry: Comparison With Standard Automated Perimetry to Detect Glaucoma. American journal of ophthalmology 2007;143:263-271.e261.

47.

Medeiros FA, Sample PA, Zangwill LM, Liebmann JM, Girkin CA, Weinreb RN. A Statistical Approach to the Evaluation of Covariate Effects on the Receiver Operating Characteristic Curves of Diagnostic Tests in Glaucoma. Invest Ophthalmol Vis Sci 2006;47:2520-2527.

48.

Artes PH, Iwase A, Ohno Y, Kitazawa Y, Chauhan BC. Properties of perimetric threshold estimates from full threshold, SITA standard, and SITA fast strategies. Investigative Ophthalmology and Visual Science 2002;43:2654-2659.

49.

Huber PJ. Robust Statistics: Wiley-Interscience; 2004.

20