Skip to content

Advertisement

The Ultrasound Journal
The Ultrasound Journal Cover Image
  • Original Article
  • Open Access

A unique method for estimating the reliability learning curve of optic nerve sheath diameter ultrasound measurement

  • 1,
  • 1,
  • 2,
  • 3,
  • 1,
  • 4,
  • 5,
  • 1 and
  • 1Email authorView ORCID ID profile
Critical Ultrasound Journal20168:9

https://doi.org/10.1186/s13089-016-0044-x

©  The Author(s) 2016

  • Received: 15 April 2016
  • Accepted: 27 July 2016
  • Published:

Abstract

Background

Optic nerve sheath diameter (ONSD) measurement using ultrasound has been proposed as a rapid, non-invasive, point of care technique to estimate intra-cranial pressure (ICP). Ultrasonic measurement of the optic nerve sheath can be quite challenging and there is limited literature surrounding learning curves for this technique. We attempted to develop a method to estimate the reliability learning curve for ONSD measurement utilizing a unique definition of reliability: a plateau in within-subject variability with unchanged between-subject variability.

Methods

As part of a previously published study, a single operator measured the ONSD in 120 healthy volunteers over a 6-month period. Utilizing the assumption that the four measurements made on each subject during this study should be equal, the relationship of within-subject variance was described using a quadratic-plateau model as assessed by segmental polynomial (knot) regression.

Results

Segmental polynomial (knot) regression revealed a plateau in within-subject variance after the 21st subject. However, there was no difference in overall mean values [3.69 vs 3.68 mm (p = 0.884)] or between-subject variance [14.49 vs 11.92 (p = 0.54)] above or below this cutoff.

Conclusions

This study suggests a significant finite learning curve associated with ONSD measurements. It also offers a unique method of calculating the learning curve associated with ONSD measurement.

Keywords

  • Point of Care ultrasound
  • Optic nerve sheath diameter
  • Ultrasound
  • Intracranial pressure
  • Education
  • Learning curve

Background

In head injured patients, elevated intra-cranial pressure (ICP) is associated with worse outcomes [1]. One of the fundamental tenants of management involves identifying those with elevated ICP and intervening to lessen and control this pressure [2]. Optic nerve sheath diameter (ONSD) measurement using ultrasound has been proposed as a rapid, non-invasive, point of care technique to estimate ICP [3]. This has been suggested as a screening tool for, or even an alternative to, invasive ICP measurement. Traditionally ICP is measured via a surgically inserted, intracranial (subdural, intra-parenchymal or intra-ventricular) pressure monitor. While this is a fairly common procedure it carries a finite but not insignificant rate of complications (most notably intracranial hemorrhage and infection) [4] and is for the most part limited to centres with neurosurgical capabilities. Given the growing use of ultrasound in the emergency department, critical care units and even in the field, point-of-care estimation of ICP using ultrasound is appealing as it can be performed in a wide variety of locations, is easily repeatable and does not require transport of the patient or large diagnostic equipment (such as CT scanners). Its utility has been substantiated by a number of small studies in diverse populations including traumatic brain injury, mass lesion, infection and post transplant reperfusion injury [5–9].

The technique, however, can be quite challenging. It requires the measurement of an often moving, sub-centimeter structure, estimated to the nearest 0.1 of a millimeter, stretching both the limits of the operator and the resolution of the ultrasound unit. Significant inter-rater variance early in the learning curve is a potential concern that could affect the utility of this approach in more novice users. The literature surrounding learning curves for point of care ultrasound in general is limited [10] and even more so for the specific technique of ONSD measurement [11]. Here we attempt to employ a unique statistical method for estimating the potential learning curve for ONSD measurement.

Methods

Ultrasound technique

As part of a previously published study [12] a single trained operator (LG) measured the ONSD in 120 healthy volunteers over a 6-month period. This operator was trained in this technique during a World Interactive Network Focused On Critical Ultrasound (WINFOCUS) ultrasound course and has a number of previous publications in this area [12–14]. All subjects signed a consent form approved by the University of Manitoba Health Research Ethics Board (H2013:199).

Subjects were placed in the supine position and images were obtained through a closed eyelid. A 13–6 MHz linear array ultrasound transducer (L25x transducer with a Sonosite M-Turbo Ultrasound Machine, SonoSite Inc, Bothell, WA) was used with the ophthalmic preset thus minimizing the power and acoustic output of the ultrasound system. The probe was placed gently on the upper eyelid. The ONSD was measured in a direction perpendicular to the axis of the nerve, in each eye, in the horizontal and sagittal plane, 3 mm behind the optic nerve head (Fig. 1), for a total of four measurements per subject.
Fig. 1
Fig. 1

Sample image of ultrasonic ONSD measurement (measurement “B”), measured 3 mm behind the optic nerve head (measurement “A”)

Method for assessing the learning curve

For the sake of the learning curve analysis it was assumed that the four measurements (left, right, horizontal, sagittal) in each patient should be identical and any differences in these values are the result of measurement error.

The relationship of within-subject variance was described using a quadratic–plateau model as assessed by segmental polynomial regression [15] to determine how many subject measurements were needed before the operator achieved a stable within-subject variance. Segmental polynomial regression, or knot regression, allows the opportunity to fit multiple regression lines to different portions of a dataset, which then join at a “knot” value which represents the intersection of these two regression lines (in this case the plateau of the learning curve). In this specific variant of segmental polynomial regression, the quadratic-plateau model, a quadratic curve is fit to the early data and a flat line (plateau) to the later data. This model is able to predict the exact point of plateau in the learning curve as it describes how many subjects were needed before the ultrasonographer achieved a stable within-subject variance.

Assessing data before and after the “knot”

To ensure there was no inherent differences in the data other than the plateau of within-subject variance before and after the “knot” variable as described above, the means of the early (subjects before the “knot” or plateau) versus late (subjects after the “knot”) measurements were compared using Welch Two Sample t tests. As well, the homogeneity of the between-subject variance was examined for early versus late measurements utilizing Levene’s test for homogeneity of variance.

Analysis was completed using both the Statistical Analysis Software version 9.3 (SAS Institute, Cary, NC) and R Statistical Library (R Core Team, 2013).

Results

Subjects included 65 males and 55 females. Mean age was 29.3 (range 18–65). Overall mean ONSD measurement was 3.68 (95 % CI 2.85–4.40) [12]. Segmental polynomial (knot) regression revealed a plateau in within-subject variance, var(I), at subject index (I) 21 (Fig. 1). Below the knot (Subject 21), the best-fit equation is:
$$ {\text{var}}(I) = 19.73 - 1.614I + 0.03843I^{2} $$
Above the knot, the best-fit equation is simply the plateau value:
$$ {\text{var}}(I) = 2.78 $$
Based on the segmental polynomial regression and the plateau of within-subject variance at subject 21 we utilized this as the cut-off for defining early (subjects before the “knot” or plateau) vs late (subjects after the “knot”) measurements. The mean measurements for each subject were compared between the early and late groups using Welch Two Sample t tests (Table 1) with no difference seen. In addition, utilizing Levene’s test for homogeneity, the between-subject variance was compared in these two groups revealing no difference between early and late groups [14.49 (early) vs 11.92 (late) (p = 0.54)]. For males and females, respectively, it was 12.64 (early) vs 9.48 (late) (p = 0.33) and 12.50 (early) vs 11.71 (late) (p = 0.95).
Table 1

Welch two sample t test comparing mean of early versus late mean ONSD measurements using subject 21 as a cutoff

 

Early ONSD mean (mm)

Late ONSD mean (mm)

p value

Overall

3.69

3.68

0.884

Males

3.88

3.78

0.454

Females

3.52

3.55

0.869

Discussion

Determining the learning curve associated with point of care ultrasound has been investigated in multiple studies across a number of techniques. The most commonly studied technique is the focused assessment with sonography in trauma (FAST) exam [10]. Estimates for the learning curve associated with this technique have been extremely variable, however, and range from 10 [16, 17] to 200 studies [18] and even up to 500 scans [19]. The heterogeneity in these results likely results from the fact that there is no clear, accepted definition of what is an acceptable performance. The majority of these studies utilized clinical endpoints (correct identification of free fluid) to define the learning curve with varying thresholds of acceptable results [10, 16–18, 20–23], while only a single study utilized an assessment of image quality and technique [24].

To our knowledge, only a single study to date has commented on the learning curve associated with ONSD measurement. In their 2007 study, Tayal et al. [11] concluded that this technique can be learned relatively quickly with a learning curve of between 10 and 25 scans depending on previous ultrasound experience. However, the origin of these estimates is not clearly discussed within the manuscript and in discussion with the senior author of this paper, these figures were estimated based on qualitative observation and not based on objective measures (personal communication).

As in other forms of point of care sonography, one of the challenges associated with determining a learning curve involves the definition of acceptable performance. This is even more so an issue for ONSD measurement as it requires a quantitative assessment of a specific structure that is inherently difficult to measure making comparison to a control or reference value challenging. For example, in comparison with the work done in FAST one could use clinical outcomes and prediction of elevated ICP as a surrogate of competency but this does not determine if the numeric measurement of ONSD is accurate. Alternatively, one could use an assessment tool of ONSD image quality and generation but currently as apposed to FAST [25], no such validated tool exists. One could use comparison of results to another imaging technique (MRI or CT scan). While this has been used to assess the utility of ultrasound ONSD measurement [6] to date this has not been used to assess learning curves. One could also compare to the results of an expert but again this requires the definition of an expert. Finally, one could use a model with known ONSD measurements as was recently developed by our group [13, 14]. While measurements in the model tend to be easier than real life and may underestimate the learning curve-crisper borders to the simulated optic nerve sheath, and no eye movement to contend with-of the options available, it certainly seems the most feasible and reproducible and an area requiring further study.

One important dimension in proficiency with this skill is the ability to making reliable measurements. In this study, we utilized a unique definition of reliability. First, by assuming the presence of an internal control between each measurement in each eye and assuming any error/variance between these measurements is due measurement error. There are a number of recent ultrasound studies that give credence to this assumption by suggesting there may be no difference between right and left ONSD measurements [26–28]. In addition, since the variance did indeed plateau over time it does lend some credibility to the technique (Fig. 2). In addition, the use of the quadratic plateau polynomial regression model is unique to this study but presents a seemingly reliable way to determine the learning curve. This model allows a quadratic regression curve to be fit to the early data and a flat line (plateau) to the later data thus allowing the model to predict the exact point of plateau in the learning curve as it describes how many subjects were needed before the sonographer achieved a stable within-subject variance. Given four measurements per subject and a plateau at subject 21, this suggests that the error may plateau after approximately 80 individual measurements of ONSD.
Fig. 2
Fig. 2

Segmental polynomial (knot) regression of within-subject variance showing a plateau (knot) at subject 21

Due to the difference in variance in early versus late subjects we questioned the validity of the measurements in the initial 21 subjects in our previous publication. However, despite the increased variability, since our original data was reported using the mean of four measurements any random measurement error should cancel out. By defining the early group as all subjects measured up to subject 21 and the late group as subjects 22–120 we compared both the mean measurements and the between-subject variance between these two groups. Neither of these revealed a difference confirming that despite the increased within-subject variance in the first 21 subjects, the overall mean and between subject variance was equal between these groups meaning the first 21 measurements were still valid.

Study limitations

There are a number of significant limitations to this study performed in healthy volunteers. First, as mentioned earlier, when calculating the learning curve it is assumed that the actual ONSD measurement should be identical in each subject in both the right and left eye. However, even if these measurements are not identical [and our results would substantiate this since there still is some within-subject variability remaining even after the plateau (Fig. 2)] the plateau in this variability may still represent a useful definition of reliability. This is an area that requires further confirmation. Second, we have used a single operator to minimize inter-observer bias and hence to test the new statistical method for the estimation of the learning curve in a rather small group of healthy volunteers. This operator was not blinded to the measurements but the calculation of learning curve was a post hoc analysis and not the initial intention of the original study. Further studies are clearly required including in larger number of subjects and with more observers to confirm the current results. Finally, the operator in this study was in fact not a novice but his previous experience with this technique, although considerable, was on a more sporadic basis prior to this study. This does not lessen the finding of the learning curve but does suggest that the learning curve may be significantly longer than we have estimated in the case of true novices. In addition, it may also suggest a significant decay of skills if the technique is not practiced regularly and could suggest the need for retraining after a prolonged period of inactivity or regular maintenance of skills through simulation. The aforementioned concepts will be a focus of a future study.

Conclusions

Despite the obvious limitations, this study offers a unique method of calculating the learning curve associated with ONSD utilizing a new definition of reliability by looking at within-subject variability and a unique statistical model to calculate the plateau in this variability.

Abbreviations

ONSD: 

optic nerve sheath diameter

FAST: 

focused assessment with sonography in trauma

ICP: 

intra-cranial pressure

Declarations

Authors’ contributions

All authors were involved in study design, data analysis and manuscript composition. LG and PG were involved in data collection. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Ethics approval and consent to participate

All subjects signed a consent form approved by the University of Manitoba Health Research Ethics Board (H2013:199).

Funding

This study was funded by a general operating grant from the Health Science Centre Foundation in Winnipeg, Manitoba, Canada.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
Departments of Surgery, University of Manitoba, 409 Tache Avenue, Winnipeg, MB, R2H 2A6, Canada
(2)
Undergraduate Medical Education, University of Manitoba, Winnipeg, MB, Canada
(3)
Medical Education, University of Manitoba, Winnipeg, MB, Canada
(4)
Departments of Internal Medicine, University of South Carolina, School of Medicine, Columbia, SC, USA
(5)
Departments of Internal Medicine and Emergency Medicine, University of South Carolina, School of Medicine, Columbia, SC, USA

References

  1. Jones PA, Andrews PJ, Midgley S et al (1994) Measuring the burden of secondary insults in head-injured patients during intensive care. J Neurosurg Anesthesiol 6:4–14View ArticlePubMedGoogle Scholar
  2. Brain Trauma Foundation, American Association of Neurological Surgeons, Congress of Neurological Surgeons (2007) Guidelines for the management of severe traumatic brain injury. J Neurotrauma 24(Suppl 1):S1–106. doi:10.1089/neu.2007.9999 Google Scholar
  3. Gangemi M, Cennamo G, Maiuri F, D’Andrea F (1987) Echographic measurement of the optic nerve in patients with intracranial hypertension. Neurochirurgia 30:53–55. doi:10.1055/s-2008-1053656 PubMedGoogle Scholar
  4. Guyot LL, Dowling C, Diaz FG, Michael DB (1998) Cerebral monitoring devices: analysis of complications. Acta Neurochir Suppl 71:47–49PubMedGoogle Scholar
  5. Geeraerts T, Launey Y, Martin L et al (2007) Ultrasonography of the optic nerve sheath may be useful for detecting raised intracranial pressure after severe brain injury. Intensive Care Med 33:1704–1711. doi:10.1007/s00134-007-0797-6 View ArticlePubMedGoogle Scholar
  6. Geeraerts T, Newcombe VFJ, Coles JP et al (2008) Use of T2-weighted magnetic resonance imaging of the optic nerve sheath to detect raised intracranial pressure. Crit Care 12:R114. doi:10.1186/cc7006 View ArticlePubMedPubMed CentralGoogle Scholar
  7. Moretti R, Pizzi B (2011) Ultrasonography of the optic nerve in neurocritically ill patients. Acta Anaesthesiol Scand 55:644–652. doi:10.1111/j.1399-6576.2011.02432.x View ArticlePubMedGoogle Scholar
  8. Dubourg J, Javouhey E, Geeraerts T et al (2011) Ultrasonography of optic nerve sheath diameter for detection of raised intracranial pressure: a systematic review and meta-analysis. Intensive Care Med 37:1059–1068. doi:10.1007/s00134-011-2224-2 View ArticlePubMedGoogle Scholar
  9. Seo H, Kim Y-K, Shin WJ, Hwang GS (2013) Ultrasonographic optic nerve sheath diameter is correlated with arterial carbon dioxide concentration during reperfusion in liver transplant recipients. Transplant Proc 45:2272–2276. doi:10.1016/j.transproceed.2012.12.032 View ArticlePubMedGoogle Scholar
  10. Thomas B, Falcone RE, Vasquez D et al (1997) Ultrasound evaluation of blunt abdominal trauma: program implementation, initial experience, and learning curve. J Trauma Inj Infect Crit Care 42:384–388 (discussion 388–90) View ArticleGoogle Scholar
  11. Tayal VS, Neulander M, Norton HJ et al (2007) Emergency department sonographic measurement of optic nerve sheath diameter to detect findings of increased intracranial pressure in adult head injury patients. Ann Emerg Med 49:508–514. doi:10.1016/j.annemergmed.2006.06.040 View ArticlePubMedGoogle Scholar
  12. Goeres P, Zeiler FA, Unger B et al (2015) Ultrasound assessment of optic nerve sheath diameter in healthy volunteers. J Crit Care 31:168–171. doi:10.1016/j.jcrc.2015.10.009 View ArticlePubMedGoogle Scholar
  13. Zeiler FA, Unger B, Kramer AH et al (2013) A unique model for ultrasound assessment of optic nerve sheath diameter. Can J Neurol Sci 40:225–229View ArticlePubMedGoogle Scholar
  14. Zeiler FA, Unger B, Zhu Q et al (2014) A unique model for ONSD Part II: inter/intra-operator variability. Can J Neurol Sci 41:430–435View ArticlePubMedGoogle Scholar
  15. SAS Institute Inc. (2008) NLIN Procedure. SAS/STAT 9.2 user’s guide. SAS Institute Inc., CaryGoogle Scholar
  16. Smith RS, Kern SJ, Fry WR, Helmer SD (1998) Institutional learning curve of surgeon-performed trauma ultrasound. Arch Surg 133:530–535 (discussion 535–6) View ArticlePubMedGoogle Scholar
  17. Shackford SR, Rogers FB, Osler TM et al (1999) Focused abdominal sonogram for trauma: the learning curve of nonradiologist clinicians in detecting hemoperitoneum. J Trauma Inj Infect Crit Care 46:553–562 (discussion 562–4) View ArticleGoogle Scholar
  18. Scalea TM, Rodriguez A, Chiu WC et al (1999) Focused assessment with sonography for trauma (FAST): results from an international consensus conference. J Trauma 46:466–472View ArticlePubMedGoogle Scholar
  19. American Institute of Ultrasound in Medicine (2014) Training guidelines for physicians who evaluate and interpret diagnostic abdominal/general ultrasound examinations. http://www.aium.org/resources/viewStatement.aspx?id=47. Accessed 3 Nov 2015
  20. Rozycki GS (1998) Surgeon-performed ultrasound: its use in clinical practice. Ann Surg 228:16–28View ArticlePubMedPubMed CentralGoogle Scholar
  21. Frezza EEE, Solis RLR, Silich RJR et al (1999) Competency-based instruction to improve the surgical resident technique and accuracy of the trauma ultrasound. Am Surg 65:884–888PubMedGoogle Scholar
  22. McCarter FD, Luchette FA, Molloy M et al (2000) Institutional and individual learning curves for focused abdominal ultrasound for trauma: cumulative sum analysis. Ann Surg 231:689–700View ArticlePubMedPubMed CentralGoogle Scholar
  23. Gracias VH, Frankel HL, Gupta R et al (2001) Defining the learning curve for the focused abdominal sonogram for trauma (FAST) examination: implications for credentialing. Am Surg 67:364–368PubMedGoogle Scholar
  24. Jang T, Kryder G, Sineff S et al (2012) The technical errors of physicians learning to perform focused assessment with sonography in trauma. Acad Emerg Med 19:98–101. doi:10.1111/j.1553-2712.2011.01242.x View ArticlePubMedGoogle Scholar
  25. Ziesmann MT, Park J, Unger BJ et al (2015) Validation of the quality of ultrasound imaging and competence (QUICk) score as an objective assessment tool for the FAST examination. J Trauma Acute Care Surg 78:1008–1013. doi:10.1097/TA.0000000000000639 View ArticlePubMedGoogle Scholar
  26. Asghar A, Hashmi M, Hussain A (2015) Optic nerve sheath diameter evaluated by transorbital sonography in healthy volunteers from Pakistan. Anaesth Pain Intensive Care 19:282–286Google Scholar
  27. Lovrencic-Huzjan A, Simicevic DS, Popovic IM, Puretic MB, Cvetkovic VV, Gopcevic A et al (2012) Ultrasonography of the optic nerve sheath in brain death. Perspect Med 1:414–416View ArticleGoogle Scholar
  28. Chen H, Ding G-S, Zhao Y-C, Yu R-G, Zhou J-X (2015) Ultrasound measurement of optic nerve diameter and optic nerve sheath diameter in healthy Chinese adults. BMC Neurol 15:106View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s) 2016

Advertisement