Deep multi-object symbol learning with self-attention based predictors

Abstract

This paper proposes an architecture that can learn symbolic representations from the continuous sensorimotor experience of a robot interacting with a varying number of objects. Unlike previous works, this work aims to remove constraints on the learned symbols such as a fixed number of interacted objects or pre-defined symbolic structures. The proposed architecture can learn symbols for single objects and relations between them in a unified manner. The architecture is an encoder-decoder network with a binary activation layer followed by self-attention layers. Experiments are conducted in a robotic manipulation setup with a varying number of objects. Results showed that the robot successfully encodes the interaction dynamics between a varying number of objects using the discovered symbols. We also showed that the discovered symbols can be used for planning to reach symbolic goal states by training a higher-level neural network.