Estimating the accuracy of optic nerve sheath diameter measurement using a pocket-sized, handheld ultrasound on a simulation model

It is currently unknown if pocket ultrasound devices have the resolution to make ONSD measurements as accurately as their larger counterparts. We attempted to assess the accuracy of ONSD measurement in an ultra-portable ultrasound (Vscan) and compare it to a conventional sized portable unit (SonoSite). This was achieved by utilizing our previously validated model, which evaluated intra- and inter-observer variability [16, 18]. We report that the ICC between raters was 0.826 for the Vscan and 0.878 for the SonoSite and the between-machine agreement ICC was 0.752.

Our high inter-rater agreement for both ultrasound devices is equivalent and larger than we have previously reported (ICC = 0.643) using our ONSD model with the same SonoSite device [16]. These improvements are likely due to our exclusive reliance on expert operators for our data this time, as our results are comparable to those found in the previous studies examining inter-rater reliability of experts in small populations in vivo [7, 15, 20, 21]. Although achieving competency in ONSD ultrasonography is not very arduous [22], errors in probe positioning and caliper placement can lead to incorrect measurement with novice operators.

The inter-machine agreement ICC between the SonoSite and the Vscan was 0.752. This is only slightly lower than the ICC between different observers for either machine implying an “excellent” agreement, and that the variation in measurements between the machines was only slightly greater than the variation in measurements between two observers using an identical machine. This suggests that a skilled operator may be able to use either machine to make ONSD measurements with a similar degree of accuracy as two operators using the same machine.

In the Bland–Altman analysis comparing level of agreement between Vscan and SonoSite measurements, there was an even spread across the range of disc sizes suggesting no systematic bias. It did find that the Vscan tended to over-estimate ONSD by a small amount (0.33 ± 0.65 mm). We hypothesize that this may be due to the screen size and resolution of the Vscan as compared to the SonoSite. Since the screen on the Vscan device was smaller than that of the SonoSite, attempts to position, the calipers on the edge of the optic nerve sheath may have lead to an overestimation. Importantly, this strong agreement between ONSD measurements by both devices allows us to create a quadratic conversion formula between the two measures. After adjusting the Vscan results using this formula, the mean difference shrinks to 0 ± 0.53 mm. A conversion formula could potentially be used to translate previously established normal reference ranges from the SonoSite into those more useful for the Vscan. This conversion formula requires further validation in a prospective study.

The advantage of our ONSD model is: it creates a controlled environment where the known disc size was consistent across different measurements and readers. This allows for an isolated comparison of measurement accuracy inherent to each technique and observer. On the surface, comparing the resolution of two ultrasound units to measure a fixed structure may seem like a simple task. The Vscan linear probe has a frequency range of 8–3.4 MHz which is significantly lower than the SonoSite’s 13–6 MHz probe. However, one must consider more than just sheer resolution of the unit, as ultrasound imaging is an operator dependent task that involves optic nerve sheath interpretation, caliper measurement, and ease of software and probe usage. Hence, the primary outcome of agreement between the two measurements is not only important, but also the secondary outcomes of inter-rater and within-subject reliability.

Pocket ultrasound units are not currently approved in Canada or the United States for the measurement of ONSD in human subjects as the existing units do not have the required presets to make these measurements safely. For this reason, we elected to perform this initial pilot study utilizing our ONSD model. The recommended settings for ONSD ultrasound in human subjects requires a thermal index (TI) ≤1 and mechanical index (MI) ≤0.23 [23] which are not achievable with the current Vscan firmware. We, therefore, do not encourage the “off-label” usage of these machines in live human subjects until the appropriate presets are made available.

Despite the importance of our findings, there are limitations to our study. First, all of the data comparing the relative accuracies of the ultrasound devices were obtained in a controlled setting on a simulation model. This was done to limit external variables and limit random error to more precisely study machine and observer effects between devices. However, it is difficult to extrapolate our findings to in vivo as the model does not necessarily account for effects, such as patient variation in anatomy or movement, fluctuations in ICP, and time constraints. Our model may itself introduce some random error of its own, such as imaging the 3D printed disc on an angle causing the disc to produce a larger shadow than usual, or that the shadow generated by the disc is not the same size as the disc itself. We also used a relatively small group of “experts”, as ultrasound evaluation of ONSD is still largely considered experimental in our centre, and there are relatively few experts to recruit. In addition, by having the sonographers both measure using the SonoSite first followed by the Vscan, we may have unintentionally introduced some bias into the measurements, however, looking at the Bland–Altman plot (Fig. 2), the similar variability around the mean for both machines would suggest if present this was likely inconsequential. Finally, our model simulates what is known as the “black stripe method” of ONSD measurement—meaning that the operator measures the edges of the internal diameter of the ONS. However, new quality criteria recently introduced [24] suggest that instead the external hypo-echogenic layers surrounding the optic nerve should be measured. To date, we have been unable to simulate this appearance in our model. This does not invalidate the comparison between ultrasound units are we are still measuring the same structure, but the authors wish to caution the reader that in vivo images may appear different and the ability of the units to clearly differentiate the borders of the ONS may be different in vivo as well.