The artificial intelligence screening tool retina.help showed variations in detecting referable diabetic retinopathy across populations and imaging equipment, with sensitivity on handheld images ranging from 62% in a Tanzanian dataset to 90% in a Finnish dataset, according to a retrospective study published in Physiological Measurement. 

“These findings underscore the need for benchmarking AI-based diabetic retinopathy screening tools using standardized datasets that encompass diverse populations and imaging conditions,” the authors wrote. “Such evaluations can guide the development of equitable, reliable, and robust screening solutions.”  

The authors noted that several commercial artificial intelligence (AI) screening tools have been developed using data from specific populations collected with tabletop cameras, raising concerns about performance degradation and potential biases when applied to other populations or imaging devices. They also described portable retinal cameras as a more affordable option for expanding screening in low- and middle-income regions, where access to specialized clinicians and diagnostic facilities is limited.  

Performance across countries and cameras

The authors evaluated retina.help version 0.6 across eight datasets from six countries containing tabletop and handheld fundus images. Four datasets were publicly available, and four were private datasets from community screening programs. Dataset sizes ranged from 264 images in Tanzania to 12,701 in the Philippines. 

Three datasets contained tabletop-camera images from Brazil and China; five contained handheld-camera images from Brazil, the Philippines, Finland, Egypt, and Tanzania. 

In the primary image-level analysis, sensitivity and specificity for handheld images were: 

• Finland: 90% and 97%, respectively. 

• Egypt: 89% and 76%. 

• Philippines: 78% and 84%. 

• Brazil: 77% and 97%. 

• Tanzania: 62% and 98%. 

The authors reported lower sensitivity in datasets representing populations with greater retinal pigmentation, although differences in imaging conditions and other population characteristics may also have contributed to the findings.

The area under the receiver operating characteristic curve (AUROC) ranged from 0.88 in the Filipino dataset to 0.99 in the Finnish dataset. Values were 0.95 for Brazil, 0.93 for Egypt, and 0.92 for Tanzania. 

For tabletop images, sensitivity was 95% and specificity was 85% in both the Brazilian BRSET and Chinese DeepDRiD datasets, with an AUROC of 0.97. In the other Chinese dataset, DDR, sensitivity was 77%, specificity was 97%, and AUROC was 0.95. 

"We were positively surprised that retina.help works so well even on fundus images taken using handheld cameras, since it was developed using images from tabletop devices, which have a much higher contrast," lead author Philipp Schapotschnikow, PhD, said in an interveiw with AACE Endocrine AI.  

The authors suggested that the comparatively low specificity in the Egyptian handheld dataset could reflect the presence of microaneurysms in many nonreferable images. They noted that the software tends to classify images with numerous microaneurysms as moderate diabetic retinopathy, leading to false positives.  

Independent datasets and patient-level analyses 

Retina.help uses an ensemble of six convolutional neural networks, each with disease-classification and lesion-segmentation components. A random forest classifier combines the six models’ classification outputs. 

The researchers evaluated the model on datasets not used for training, without further tuning. Referable diabetic retinopathy was defined as moderate disease or worse when severity grades were available. Macular edema was not considered because its label was unavailable across datasets. 

Sensitivity and specificity were calculated using a 50% threshold for the summed probability of moderate or worse diabetic retinopathy. The authors did not apply the software’s image-quality rejection process, although they excluded images explicitly labeled ungradable in the Chinese DDR dataset. 

In three datasets with patient identifiers, assigning each patient the highest disease probability among their images increased sensitivity and decreased specificity compared with image-level analysis.  

Sensitivity increased from 95% per image to 96% per patient in the Brazilian BRSET tabletop dataset, while specificity decreased from 85% to 84%. In the Brazilian mBRSET handheld dataset, sensitivity increased from 77% to 84%, while specificity decreased from 97% to 94%. In the Filipino handheld dataset, sensitivity increased from 78% to 88%, while specificity decreased from 84% to 76%. AUROC remained practically unchanged across the three datasets. 

Limitations and validation needs 

The study had several limitations. Patient identifiers were unavailable for some datasets, preventing patient-level evaluation across all datasets. Because multiple images could come from the same patient, image-level observations were not statistically independent. 

The authors identified training-data composition as the most likely explanation for population-related performance differences, proposing underrepresentation of some ethnic groups as one hypothesis. They noted that retinal images from Southeast Asian patients were scarcely represented in training data.  

The authors emphasized the need for context-specific validation and optimization of AI-based diabetic retinopathy screening tools. 

“This study should raise awareness of differences in accuracy of automated screening techniques when applied to different ethnic groups. In particular, the accuracy may be significantly worse for minorities,” Dr. Schapotschnikow said. “We would advise treating the findings—or lack thereof—of AI-based techniques with caution in this case.” 

Dr. Schapotschnikow disclosed ownership of Brains for hire UG in Munich, Germany, which developed and owns retina.help, but the authors noted that the study's primary focus was not to validate or promote the clinical efficacy of the software. The remaining authors reported no conflicts of interest.