Hierarchical Joint Max-Margin Learning of Mid and Top Level Representations for Visual Recognition

Abstract

Currently, Bag-of-Visual-Words (BoVW) and part-based methods are the most popular approaches for visual recognition. In both cases, a mid-level representation is build on top of low level image descriptors while top levels classifiers use this mid-level representation to achieve visual recognition. While in current part-based approaches, mid and top level representations are usually jointly trained, this is not the usual case for BoVW schemes. A main reason is the complex data association problem associated to the larger size of the visual dictionary usually needed by BoVW approaches at the mid-level layer. As a further observation, typical solutions based on BoVW and part-based representations are usually limited to binary classification problems, a strategy that ignores relevant correlations among classes. In this work we propose a novel hierarchical approach for visual recognition that, in the context of a BoVW scheme, jointly learns suitable mid and top level representations. Furthermore, using a max-margin learning framework, the proposed approach directly handles the multiclass case at both levels of abstraction. We test our proposed method using several popular benchmarks datasets. As our main result, we demonstrate that by coupling learning of mid and top level representations, the proposed approach fosters sharing of discriminativity words among target classes, being able to achieve state-of-the-art recognition performance using far less visual words than previous approaches.