Home
Research
Publications
People

A Hybrid Deep Boltzmann Machine for Contextual Scene Modeling


(a) An overview of the proposed hybrid tri-way Boltzmann Machine, where the tri-way edges are shown in red, and (b) some examples for what it can provide to a robot

Abstract

Scene models allow robots to reason about what is in the scene, what else should be in it, and what should not be in it. In this paper, we propose a hybrid Boltzmann Machine (BM) for scene modeling where relations between objects are integrated. To be able to do that, we extend BM to include tri- way edges between visible (object) nodes and make the network to share the relations across different objects. We evaluate our method against several baseline models (Deep Boltzmann Machines, and Restricted Boltzmann Machines) on a scene classification dataset, and show that it performs better in several scene reasoning tasks.

Paper

    İlker Bozcan, Yağmur Oymak, İdil Zeynep Alemdar, Sinan Kalkan
    What is (missing or wrong) in the scene? A Hybrid Deep Boltzmann Machine For Contextualized Scene Modeling ( accepted for ICRA 2018 )
    [paper]

Method & Results

We propose a triway deep BM for scene modeling. Visible nodes are separated into two groups: relations(r) and objects(v). Relation nodes(visible) are shared between two object nodes. Visible nodes(both of relation and visible nodes) are connected to units at the first hidden layer. Hidden layers are stacked to make model deep.

Gibbs sampling is used for inference instead of variational inference since our dataset is relatively small (total 3485 samples) and input vectors are too sparse (i.e. slight number of relation nodes are active). It is a type of Monte Carlo Markov Chain Method that can guarantee precise inferences. Precise inference is important for our problem since only slight number of relation nodes are active for each sample.

We conduct several experiments that includes scenarios that can be performed by robots in real world problems. We compared our model’s results(Triway BM) with Restricted Boltzmann Machine(RBM) and General Boltzmann Machine(GBM) that is structured as bidirectional links exists within input layer and hidden layers are stacked.

The first experiment is estimation of relations between objects that are on the scene for current context. In this experiment, firstly, model determines context by using objects on the scene only, then using by context and activation of objects, it estimates relations among objects on the scene. For this task, we define accuracy as the percentage of relations correctly estimated with respect to the labeled relations in the test dataset. We see that our model provides highest accuracy(Table 1).

Table 1: Performance (accuracy) of the methods on estimating relations between a given set of objects. Higher is better.
RBM GBM Triway BM Chance level
12.06% 14.18% 23.35% 1.4310^-4%

The second experiment is finding missing object(s) on the scene according to current context. In this task, we randomly de-activate an object from the scene and expect the network to find the missing object. For this task, we define accuracy as the percentage of the missing objects found correctly in the test dataset. Our model is the best(Table 2).

Table 2: Performance (accuracy) of the methods on finding missing object in the scene. Higher is better.
RBM GBM Triway BM Chance level
35.12% 40.94% 43.28% 5.75*10^*6%

The third experiment is finding the objects that are out of current context on the scene. For this task, we select scenes, remove randomly an object and add randomly another object not in the scene. We define error measure to indicate how much reconstructed data is different form original data that is without extra objects. Our model is the best(Table 3).

Table 3: Performance (error) of the methods on finding what is out of context in the scene. Lower is better.
RBM GBM Triway BM Chance level
0.6446 0.1404 0.0789 0.5

The last experiment is random scene generation. In this task, we randomly activate hidden units and visible units are sampled according to context that is determined by hidden units.

Video


© KOVAN Research Labs ‒ Department of Computer Engineering @ Middle East Technical University ‒ Üniversiteler Mahallesi, Dumlupınar Bulvarı No:1 06800 Çankaya Ankara/TÜRKİYE.