Food/Non-Food Classification of Real-Life Egocentric Images in Low- and Middle-Income Countries Based on Image Tagging Features

Authors

Guangzong Chen
Wenyan Jia
Yifan Zhao
Zhi-Hong Mao
Benny Lo
Alex K Anderson
Gary Frost
Modou L Jobarteh
Megan A McCrory
Edward Sazonov
Matilda Steiner-Asiedu
Richard S Ansong
Thomas Baranowski
Lora Burke
Mingui Sun

Publication Date

1-1-2021

Journal

Frontiers in Artificial Intelligence

DOI

10.3389/frai.2021.644712

PMID

33870184

PMCID

PMC8047062

PubMedCentral® Posted Date

4-1-2021

PubMedCentral® Full Text Version

Post-print

Published Open-Access

yes

Keywords

egocentric image, wearable device, technology-based dietary assessment, low- and middle-income country, artificial intelligence

Abstract

Malnutrition, including both undernutrition and obesity, is a significant problem in low- and middle-income countries (LMICs). In order to study malnutrition and develop effective intervention strategies, it is crucial to evaluate nutritional status in LMICs at the individual, household, and community levels. In a multinational research project supported by the Bill & Melinda Gates Foundation, we have been using a wearable technology to conduct objective dietary assessment in sub-Saharan Africa. Our assessment includes multiple diet-related activities in urban and rural families, including food sources (e.g., shopping, harvesting, and gathering), preservation/storage, preparation, cooking, and consumption (e.g., portion size and nutrition analysis). Our wearable device ("eButton" worn on the chest) acquires real-life images automatically during wake hours at preset time intervals. The recorded images, in amounts of tens of thousands per day, are post-processed to obtain the information of interest. Although we expect future Artificial Intelligence (AI) technology to extract the information automatically, at present we utilize AI to separate the acquired images into two binary classes: images with (Class 1) and without (Class 0) edible items. As a result, researchers need only to study Class-1 images, reducing their workload significantly. In this paper, we present a composite machine learning method to perform this classification, meeting the specific challenges of high complexity and diversity in the real-world LMIC data. Our method consists of a deep neural network (DNN) and a shallow learning network (SLN) connected by a novel probabilistic network interface layer. After presenting the details of our method, an image dataset acquired from Ghana is utilized to train and evaluate the machine learning system. Our comparative experiment indicates that the new composite method performs better than the conventional deep learning method assessed by integrated measures of sensitivity, specificity, and burden index, as indicated by the Receiver Operating Characteristic (ROC) curve.