Determining the learning curve associated with point of care ultrasound has been investigated in multiple studies across a number of techniques. The most commonly studied technique is the focused assessment with sonography in trauma (FAST) exam [10]. Estimates for the learning curve associated with this technique have been extremely variable, however, and range from 10 [16, 17] to 200 studies [18] and even up to 500 scans [19]. The heterogeneity in these results likely results from the fact that there is no clear, accepted definition of what is an acceptable performance. The majority of these studies utilized clinical endpoints (correct identification of free fluid) to define the learning curve with varying thresholds of acceptable results [10, 16–18, 20–23], while only a single study utilized an assessment of image quality and technique [24].
To our knowledge, only a single study to date has commented on the learning curve associated with ONSD measurement. In their 2007 study, Tayal et al. [11] concluded that this technique can be learned relatively quickly with a learning curve of between 10 and 25 scans depending on previous ultrasound experience. However, the origin of these estimates is not clearly discussed within the manuscript and in discussion with the senior author of this paper, these figures were estimated based on qualitative observation and not based on objective measures (personal communication).
As in other forms of point of care sonography, one of the challenges associated with determining a learning curve involves the definition of acceptable performance. This is even more so an issue for ONSD measurement as it requires a quantitative assessment of a specific structure that is inherently difficult to measure making comparison to a control or reference value challenging. For example, in comparison with the work done in FAST one could use clinical outcomes and prediction of elevated ICP as a surrogate of competency but this does not determine if the numeric measurement of ONSD is accurate. Alternatively, one could use an assessment tool of ONSD image quality and generation but currently as apposed to FAST [25], no such validated tool exists. One could use comparison of results to another imaging technique (MRI or CT scan). While this has been used to assess the utility of ultrasound ONSD measurement [6] to date this has not been used to assess learning curves. One could also compare to the results of an expert but again this requires the definition of an expert. Finally, one could use a model with known ONSD measurements as was recently developed by our group [13, 14]. While measurements in the model tend to be easier than real life and may underestimate the learning curve-crisper borders to the simulated optic nerve sheath, and no eye movement to contend with-of the options available, it certainly seems the most feasible and reproducible and an area requiring further study.
One important dimension in proficiency with this skill is the ability to making reliable measurements. In this study, we utilized a unique definition of reliability. First, by assuming the presence of an internal control between each measurement in each eye and assuming any error/variance between these measurements is due measurement error. There are a number of recent ultrasound studies that give credence to this assumption by suggesting there may be no difference between right and left ONSD measurements [26–28]. In addition, since the variance did indeed plateau over time it does lend some credibility to the technique (Fig. 2). In addition, the use of the quadratic plateau polynomial regression model is unique to this study but presents a seemingly reliable way to determine the learning curve. This model allows a quadratic regression curve to be fit to the early data and a flat line (plateau) to the later data thus allowing the model to predict the exact point of plateau in the learning curve as it describes how many subjects were needed before the sonographer achieved a stable within-subject variance. Given four measurements per subject and a plateau at subject 21, this suggests that the error may plateau after approximately 80 individual measurements of ONSD.
Due to the difference in variance in early versus late subjects we questioned the validity of the measurements in the initial 21 subjects in our previous publication. However, despite the increased variability, since our original data was reported using the mean of four measurements any random measurement error should cancel out. By defining the early group as all subjects measured up to subject 21 and the late group as subjects 22–120 we compared both the mean measurements and the between-subject variance between these two groups. Neither of these revealed a difference confirming that despite the increased within-subject variance in the first 21 subjects, the overall mean and between subject variance was equal between these groups meaning the first 21 measurements were still valid.
Study limitations
There are a number of significant limitations to this study performed in healthy volunteers. First, as mentioned earlier, when calculating the learning curve it is assumed that the actual ONSD measurement should be identical in each subject in both the right and left eye. However, even if these measurements are not identical [and our results would substantiate this since there still is some within-subject variability remaining even after the plateau (Fig. 2)] the plateau in this variability may still represent a useful definition of reliability. This is an area that requires further confirmation. Second, we have used a single operator to minimize inter-observer bias and hence to test the new statistical method for the estimation of the learning curve in a rather small group of healthy volunteers. This operator was not blinded to the measurements but the calculation of learning curve was a post hoc analysis and not the initial intention of the original study. Further studies are clearly required including in larger number of subjects and with more observers to confirm the current results. Finally, the operator in this study was in fact not a novice but his previous experience with this technique, although considerable, was on a more sporadic basis prior to this study. This does not lessen the finding of the learning curve but does suggest that the learning curve may be significantly longer than we have estimated in the case of true novices. In addition, it may also suggest a significant decay of skills if the technique is not practiced regularly and could suggest the need for retraining after a prolonged period of inactivity or regular maintenance of skills through simulation. The aforementioned concepts will be a focus of a future study.
