JewelNet: A Custom CNN and Transfer Learning–Based Approach for Fine-Grained Jewelry Classification

Abstract

Fine-tuned jewelry image classification poses a considerable challenge to the computer vision application due to intraclass similarity, reflective metal material, complex background, and the absence of the corresponding training data. Traditional classification approaches using manually extracted features such as histograms of colors, texture description, and structural attributes have proven ineffective in solving the particular issue due to their limited ability to discriminate between structurally similar classes of items such as bangles and bracelets. Thus, the current research proposes a novel model for solving the given problem named JewelNet, which is an all-inclusive deep learning classifier that utilizes three different architectural approaches to classification including a custom CNN architecture, transfer learning using VGG16 and ResNet50, and the state-of-the-art EfficientNetB2 architecture. Two different sets of experiments were designed based on the same framework to compare various model architectures. A large dataset of 1,217 images of eight different types of jewelry items (bangle, bracelet, chain, earring, necklace, pendant, ring, and nose pin) was assembled. To diversify the collected data, various image augmentation techniques such as rotations, flips, zooming, changes to the brightness, and channels were employed. As a result, the size of the dataset increased to 8,519 images. Experimental evaluation revealed that the EfficientNetB2 architecture achieves the best classification performance with accuracy of 95.21%, with precision and recall being equal to 95.33% and 95.06%, respectively, leading to an F1-score of 94.97%. At the same time, the best classification accuracy for VGG16 is 94.19%, and Custom CNN yields 93.78%. The per- class classification demonstrates that recall for EfficientNetB2 equals 100% for earrings, necklaces, pendants, and rings. Besides, the deep features obtained from this network can be used for recommendation purposes with cosine similarity greater than 0.92.