Nowadays, people are increasingly conscious about their health and wellbeing due to the alarming rise in the death rates due to heart attacks, cancer, etc. Physicians’ foremost advice for an obese person is to keep track and control the intake of calories to reduce weight. Though there are quite a number of algorithms to identify images and provide a calorie based calculation for the requested image, there are limited data sets specifically on food items. This is a big challenge for deep learning systems in which learning rate and accuracy depends on the number of images in the training set. The main objective is to develop a novel deep learning-based system for classification of food images of South India.