Could a smartphone photo help assess metabolic health? To explore that question, Ahmed Metwally, PhD, of Google’s Metabolic Health AI research program, and his colleagues developed PhotoScan, a deep learning method that estimates body composition from smartphone images. In their preprint study, adding PhotoScan estimates to age, sex, and body mass index (BMI) improved identification of insulin resistance, with performance approaching that achieved by adding dual-energy x-ray absorptiometry (DXA) measurements. 

The model estimated total body fat percentage, android-to-gynoid fat ratio, and visceral-to-subcutaneous fat area ratio. It was pretrained using magnetic resonance imaging (MRI)–derived body projections and DXA measurements from 35,323 UK Biobank participants, then fine-tuned using smartphone imagery and corresponding DXA measurements from a separate cohort of 677 participants. 

The researchers independently validated the body composition model in the MetabolicMosaic cohort, a 30-week prospective study with complete data from 132 participants across 195 visits. Compared with DXA, PhotoScan had a mean absolute error of 2.13 percentage points for total body fat percentage and 0.09 for each fat-distribution ratio.  

They also developed insulin resistance classifiers in MetabolicMosaic using repeated nested cross-validation, keeping each participant’s data within either the training or validation set within each fold. Insulin resistance was defined by a homeostatic model assessment of insulin resistance value greater than 2.9. Adding PhotoScan estimates to age, sex, and BMI increased the area under the receiver operating characteristic curve (AUROC) from 0.692 to 0.76. Adding DXA measurements yielded an AUROC of 0.773. 

These findings suggest that smartphone-derived body composition estimates capture information relevant to insulin resistance beyond age, sex, and BMI. However, the insulin resistance evaluation was limited to a small cohort, and PhotoScan remains an investigational research tool that has not been validated for clinical use. 

In part three of his interview with AACE Endocrine AI editor-in-chief Johnson Thomas, MD, FSCE, FEAA, Dr. Metwally discusses how the idea for this study came about and what the findings could mean for the future of metabolic health.  

(The following interview transcript has been edited for length and clarity.)   

Dr. Thomas: I want to focus on something we all carry in our pockets: our smartphones. You recently conducted a study using smartphone photos to estimate body composition. Can you talk a little bit about that? 

Dr. Metwally: This is still early research. We had this idea that photos of ourselves would carry information about metabolic health. Multiple studies have explored this. 

From a photo of someone wearing a specific type of clothing, can we predict body fat percentage and fat distribution? Can we use smartphone cameras to predict body fat percentage and body composition in general, or not? One question that came to mind was whether those predictions could tell us something about metabolic or cardiometabolic health. Does using photos actually improve predictive performance beyond what you get from demographics and measures like BMI and age? What is the added value of taking a picture?  

To conduct the study, we used data from the UK Biobank, which includes thousands of people who have had MRI and DXA scans. MRI is really the gold standard for body composition, and DXA provides a good reference measure. We used MRI and DXA data to pretrain a foundation model to predict body fat percentage and fat distribution.

But we knew we wanted to use smartphone photos. So, we recruited around 700 people and obtained usable PhotoScan images from 677. PhotoScan involves taking a photo of yourself wearing a specific type of clothing, such as fitted clothes. We then fine-tuned the pretrained model to analyze photos taken with a smartphone camera. From there, we predicted body fat percentage and fat distribution.

We found that the correlation between body fat percentage predicted by PhotoScan and body fat percentage measured by DXA was around 95%. 

When we conducted this study, I thought a correlation of around 80% would be great. If you base the prediction on BMI and demographics, the correlation would be about 85%. If you use bioelectrical impedance analysis (BIA), such as measurements from a smartwatch, it would be around 90%. PhotoScan used ordinary photos taken with a widely available smartphone camera.

But you have to take the images under specific conditions, and that's why the tool isn't ready for widespread use. We still need to conduct more studies to remove some of these requirements. But we found that we can predict body fat percentage and distribution very well.

If you take the predicted body fat percentage and distribution and add them to demographics to predict metabolic markers, such as insulin resistance, they provide additional predictive value compared with demographics alone. That added value is very similar to what you get with DXA.

If you use DXA, which not everyone has access to, you get better performance in predicting insulin resistance. But you get a similar gain with PhotoScan. That’s the key point. Stepping back, taking a picture in a certain way could eventually provide a screening tool for insulin resistance, diabetes, and hypertension. We just need more studies to test these hypotheses.

We also need to see whether we can relax some of the photo-taking requirements and make the process easier. For example, could we avoid having to put the phone on a table and rotate? If we could eventually use a selfie, that would be great.

Dr. Thomas: How do you address privacy concerns and the generalizability of these tools?

Dr. Metwally: I think you're hitting the right things: privacy, generalizability, but also trust and the personal aspect. All these models should run on the device itself. The images shouldn't leave your phone. If these tools are eventually deployed, they should protect users' privacy. You should be able to trust the company providing them, and the technology has to be very secure.

In my opinion, what would be really useful about these tools is being able to say, “This is where I am now. These are the changes I’m making to reach a specific goal, and I’m seeing the results.” Sometimes you do strength training and your BMI isn’t changing, but you’re actually getting healthier because you’re losing fat and gaining muscle.

BMI is not the metric to optimize for. As you said, is it visceral fat? Is it fat around your organs, such as your liver, or somewhere else? I think it’s useful to look more closely at where the fat is, which fat is good or bad, and how to optimize for that.

That’s why I think body composition is important. Scaling this approach is definitely important, as is making it accurate for a specific population. But accuracy is relative because it depends on how the tool is used. Are you trying to estimate body fat percentage to within 1%, or do you want to track whether it’s increasing or decreasing? How sensitive is the algorithm to these different changes?

This study takes us from PhotoScan to body composition to cardiometabolic prediction. I think it shows how this approach could work, but a lot more work is needed to generalize it and make it practical for broader use.