Table of contents
-
Introduction
-
Tutorials
-
Survey papers
-
Benchmark datasets
-
Fine-grained image recognition
-
Fine-grained recognition by localization-classification subnetworks
-
Fine-grained recognition by end-to-end feature encoding
-
Fine-grained recognition with external information
-
Fine-grained recognition with web data / auxiliary data
-
Fine-grained recognition with multi-modality data
-
Fine-grained recognition with humans in the loop
-
-
-
Fine-grained image retrieval
-
Unsupervised with pre-trained models
-
Supervised with metric learning
-
-
Fine-grained image generation
-
Generating from fine-grained image distributions
-
Generating from text descriptions
-
-
Future directions of FGIA
-
Automatic fine-grained models
-
Fine-grained few shot learning
-
Fine-grained hashing
-
FGIA within more realistic settings
-
-
Leaderboard
Introduction
This homepage lists some representative papers/codes/datasets all about deep learning based fine-grained image, including fine-grained image recognition, fine-grained image retrieval, fine-grained image generation, etc. If you have any questions, please feel free to leave message.
Survey papers
-
A Survey on Deep Learning-based Fine-Grained Object Classification and Semantic Segmentation.
Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. International Journal of Automation and Computing, 2017.
Benchmark datasets
Summary of popular fine-grained image datasets. Note that ‘‘BBox’’ indicates whether this dataset provides object bounding box supervisions. ‘‘Part anno.’’ means providing the key part localizations. ‘‘HRCHY’’ corresponds to hierarchical labels. ‘‘ATR’’ represents the attribute labels (e.g., wing color, male, female, etc). ‘‘Texts’’ indicates whether fine-grained text descriptions of images are supplied.
Dataset name | Year | Meta-class | images | categories | BBox | Part anno. | HRCHY | ATR | Texts |
Oxford flower | 2008 | Flowers | 8,189 | 102 | |||||
CUB200 | 2011 | Birds | 11,788 | 200 | |||||
Stanford Dog | 2011 | Dogs | 20,580 | 120 | |||||
Stanford Car | 2013 | Cars | 16,185 | 196 | |||||
FGVC Aircraft | 2013 | Aircrafts | 10,000 | 100 | |||||
Birdsnap | 2014 | Birds | 49,829 | 500 | |||||
NABirds | 2015 | Birds | 48,562 | 555 | |||||
DeepFashion | 2016 | Clothes | 800,000 | 1,050 | |||||
Fru92 | 2017 | Fruits | 69,614 | 92 | |||||
Veg200 | 2017 | Vegetable | 91,117 | 200 | |||||
iNat2017 | 2017 | Plants & Animals | 859,000 | 5,089 | |||||
RPC | 2019 | Retail products | 83,739 | 200 |
Fine-grained image recognition
Fine-grained recognition by localization-classification subnetworks
-
Part-based R-CNNs for Fine-Grained Category Detection.
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. ECCV, 2014. [code]
-
Interaction Part Mining: A Mid-Level Approach for Fine-Grained Action Recognition.
Yang Zhou, Bingbing Ni, Richang Hong, Meng Wang, and Qi Tian. CVPR, 2015.
-
Fine-Grained Recognition without Part Annotations.
Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. CVPR, 2015. [code]
-
The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification.
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. CVPR, 2015.
-
Deep LAC: Deep Localization, Alignment and Classification for Fine-grained Recognition.
Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. CVPR, 2015.
-
Spatial Transformer Networks.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. NIPS, 2015. [code]
-
Part-Stacked CNN for Fine-Grained Visual Categorization.
Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. CVPR, 2016.
-
Mining Discriminative Triplets of Patches for Fine-Grained Classification.
Yaming Wang, Jonghyun Choi, Vlad I. Morariu, and Larry S. Davis. CVPR, 2016.
-
SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-grained Recognition.
Han Zhang, Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. CVPR, 2016.
-
Picking Deep Filter Responses for Fine-grained Image Recognition.
Xiaopeng Zhang, Hongkai, Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. CVPR, 2016.
-
Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition.
Jianlong Fu, Heliang Zheng, and Tao Mei. CVPR, 2017.
-
Fine-Grained Recognition as HSnet Search for Informative Image Parts.
Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. CVPR, 2017.
-
Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition.
Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. ICCV, 2017. [code]
-
Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-Grained Image Classification.
Xiangteng He, and Yuxin Peng. AAAI, 2017.
-
Localizing by Describing: Attribute-Guided Attention Localization for Fine-Grained Recognition.
Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin. AAAI, 2017.
-
Learning to Navigate for Fine-grained Classification.
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. ECCV, 2018. [code]
-
Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition.
Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. ECCV, 2018. [code]
-
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up.
Weifeng Ge, Xiangru Lin, and Yizhou Yu. CVPR, 2019.
Fine-grained recognition by end-to-end feature encoding
-
Hyper-Class Augmented and Regularized Deep Learning for Fine-Grained Image Classification.
Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. CVPR, 2015.
-
Subset Feature Learning for Fine-Grained Category Classification.
ZongYuan Ge, Christopher McCool, Conrad Sanderson, and Peter Corke. CVPR, 2015.
-
Bilinear CNN Models for Fine-grained Visual Recognition.
Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. ICCV, 2015. [code]
-
Multiple Granularity Descriptors for Fine-Grained Categorization.
Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. ICCV, 2015.
-
Compact Bilinear Pooling.
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. CVPR, 2016. [code]
-
Fine-Grained Image Classification by Exploring Bipartite-Graph Labels.
Feng Zhou, and Yuanqing Lin. CVPR, 2016. [project page]
-
Kernel Pooling for Convolutional Neural Networks.
Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. CVPR, 2017.
-
Low-rank Bilinear Pooling for Fine-Grained Classification.
Shu Kong, and Charless Fowlkes. CVPR, 2017. [code]
-
Higher-order Integration of Hierarchical Convolutional Activations for Fine-Grained Visual Categorization.
Sijia Cai, Wangmeng Zuo, and Lei Zhang. ICCV, 2017. [code]
-
Learning a Discriminative Filter Bank within a CNN for Fine-grained Recognition.
Yaming Wang, Vlad I. Morariu, and Larry S. Davis. CVPR, 2018. [code]
-
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization.
Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. CVPR, 2018. [code]
-
Maximum-Entropy Fine Grained Classification.
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. NIPS, 2018.
-
Pairwise Confusion for Fine-Grained Visual Classification.
Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik. ECCV, 2018. [code]
-
DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-Grained Image Recognition.
Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu. ECCV, 2018.
-
Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition.
Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. ECCV, 2018. [code]
-
Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification.
Xing Wei, Yue Zhang, Yihong Gong, Jiawei Zhang, and Nanning Zheng. ECCV, 2018.
-
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition.
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. CVPR, 2019. [code]
-
Destruction and Construction Learning for Fine-grained Image Recognition.
Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. CVPR, 2019. [code]
Fine-grained recognition with external information
Fine-grained recognition with web data / auxiliary data
-
Augmenting Strong Supervision Using Web Data for Fine-Grained Categorization.
Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao. ICCV, 2015.
-
Webly Supervised Learning Meets Zero-shot Learning: A Hybrid Approach for Fine-grained Classification.
Li Niu, Ashok Veeraraghavan, and Vshu Sabbarwal. CVPR, 2018.
-
Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data.
Yabin Zhang, Hui Tang, and Kai Jia. ECCV, 2018. [code]
-
Learning from Web Data using Adversarial Discriminative Neural Networks for Fine-Grained Classification.
Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. AAAI, 2019.
Fine-grained recognition with multi-modality data
-
Fine-Grained Image Classification via Combining Vision and Language.
Xiangteng He, and Yuxin Peng. CVPR, 2017.
-
Audio Visual Attribute Discovery for Fine-Grained Object Recognition.
Hua Zhang, Xiaochun Cao, and Rui Wang. AAAI, 2018.
-
Fine-grained Image Classification by Visual-Semantic Embedding.
Huapeng Xu, Guilin Qi, Jingjing Li, Meng Wang, Kang Xu, and Huan Gao. IJCAI, 2018.
-
Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition.
Tianshui Chen, Liang Lin, Riquan Chen, Yang Wu, and Xiannan Luo. IJCAI, 2018.
Fine-grained recognition with humans in the loop
-
Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop.
Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie. CVPR, 2016.
Fine-grained image retrieval
Unsupervised with pre-trained models
-
Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval.
Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. TIP, 2017. [project page]
Supervised with metric learning
-
Centralized Ranking Loss with Weakly Supervised Localization for Fine-Grained Object Retrieval.
Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. IJCAI, 2018.
-
Towards Optimal Fine Grained Retrieval via Decorrelated Centralized Loss with Normalize-Scale layer.
Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. AAAI, 2019.
Fine-grained image generation
Generating from fine-grained image distributions
-
CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training.
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. ICCV, 2017. [code]
-
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. CVPR, 2019. [code]
Generating from text descriptions
-
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. CVPR, 2018. [code]
Future directions of FGIA
Fine-grained few shot learning
-
Piecewise classifier mappings: Learning fine-grained learners for novel categories with few examples.
Xiu-Shen Wei, Peng Wang, Lingqiao Liu, Chunhua Shen, and Jianxin Wu. TIP, 2019.
FGIA within more realistic settings
-
Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach.
Timnit Geru, Judy Hoffman, and Li Fei-Fei. ICCV, 2017.
-
The iNaturalist Species Classification and Detection Dataset.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. CVPR 2018.
-
RPC: A Large-Scale Retail Product Checkout Dataset.
Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. arXiv: 1901.07249, 2019. [project page]
Leaderboard
The section is being continually updated. Since CUB200-2011 is the most popularly used fine-grained dataset, we list the fine-grained recognition leaderboard by treating it as the test bed.
Method | Publication | BBox? | Part? | External information? | Base model | Image resolution | Accuracy |
PB R-CNN | ECCV 2014 | Alex-Net | 224x224 | 73.9% | |||
MaxEnt | NIPS 2018 | GoogLeNet | TBD | 74.4% | |||
PB R-CNN | ECCV 2014 | Alex-Net | 224x224 | 76.4% | |||
PS-CNN | CVPR 2016 | CaffeNet | 454x454 | 76.6% | |||
MaxEnt | NIPS 2018 | VGG-16 | TBD | 77.0% | |||
Mask-CNN | PR 2018 | Alex-Net | 448x448 | 78.6% | |||
PC | ECCV 2018 | ResNet-50 | TBD | 80.2% | |||
DeepLAC | CVPR 2015 | Alex-Net | 227x227 | 80.3% | |||
MaxEnt | NIPS 2018 | ResNet-50 | TBD | 80.4% | |||
Triplet-A | CVPR 2016 | Manual labour | GoogLeNet | TBD | 80.7% | ||
Multi-grained | ICCV 2015 | WordNet etc. | VGG-19 | 224x224 | 81.7% | ||
Krause et al. | CVPR 2015 | CaffeNet | TBD | 82.0% | |||
Multi-grained | ICCV 2015 | WordNet etc. | VGG-19 | 224x224 | 83.0% | ||
TS | CVPR 2016 | VGGD+VGGM | 448x448 | 84.0% | |||
Bilinear CNN | ICCV 2015 | VGGD+VGGM | 448x448 | 84.1% | |||
STN | NIPS 2015 | GoogLeNet+BN | 448x448 | 84.1% | |||
LRBP | CVPR 2017 | VGG-16 | 224x224 | 84.2% | |||
PDFS | CVPR 2016 | VGG-16 | TBD | 84.5% | |||
Xu et al. | ICCV 2015 | Web data | CaffeNet | 224x224 | 84.6% | ||
Cai et al. | ICCV 2017 | VGG-16 | 448x448 | 85.3% | |||
RA-CNN | CVPR 2017 | VGG-19 | 448x448 | 85.3% | |||
MaxEnt | NIPS 2018 | Bilinear CNN | TBD | 85.3% | |||
PC | ECCV 2018 | Bilinear CNN | TBD | 85.6% | |||
CVL | CVPR 2017 | Texts | VGG | TBD | 85.6% | ||
Mask-CNN | PR 2018 | VGG-16 | 448x448 | 85.7% | |||
GP-256 | ECCV 2018 | VGG-16 | 448x448 | 85.8% | |||
KP | CVPR 2017 | VGG-16 | 224x224 | 86.2% | |||
T-CNN | IJCAI 2018 | ResNet | 224x224 | 86.2% | |||
MA-CNN | ICCV 2017 | VGG-19 | 448x448 | 86.5% | |||
MaxEnt | NIPS 2018 | DenseNet-161 | TBD | 86.5% | |||
DeepKSPD | ECCV 2018 | VGG-19 | 448x448 | 86.5% | |||
OSME+MAMC | ECCV 2018 | ResNet-101 | 448x448 | 86.5% | |||
StackDRL | IJCAI 2018 | VGG-19 | 224x224 | 86.6% | |||
DFL-CNN | CVPR 2018 | VGG-16 | 448x448 | 86.7% | |||
PC | ECCV 2018 | DenseNet-161 | TBD | 86.9% | |||
KERL | IJCAI 2018 | Attributes | VGG-16 | 224x224 | 87.0% | ||
HBP | ECCV 2018 | VGG-16 | 448x448 | 87.1% | |||
Mask-CNN | PR 2018 | ResNet-50 | 448x448 | 87.3% | |||
DFL-CNN | CVPR 2018 | ResNet-50 | 448x448 | 87.4% | |||
NTS-Net | ECCV 2018 | ResNet-50 | 448x448 | 87.5% | |||
HSnet | CVPR 2017 | GoogLeNet+BN | TBD | 87.5% | |||
MetaFGNet | ECCV 2018 | Auxiliary data | ResNet-34 | TBD | 87.6% | ||
DCL | CVPR 2019 | ResNet-50 | 448x448 | 87.8% | |||
TASN | CVPR 2019 | ResNet-50 | 448x448 | 87.9% | |||
Ge et al. | CVPR 2019 | GoogLeNet+BN | Shorter side is 800 px | 90.4% |