Representation Learning for Grounded Spatial Reasoning

July 13, 2017 · Entered Twilight · 🏛 Transactions of the Association for Computational Linguistics

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .gitattributes, LICENSE, README.md, data, download_data.sh, environment, generate_worlds.py, logs, models, pipeline, psiturk-vi, reinforcement.py, representation.py, requirements.txt, slurm, utils, visualization

Authors Michael Janner, Karthik Narasimhan, Regina Barzilay arXiv ID 1707.03938 Category cs.CL: Computation & Language Cross-listed cs.AI, cs.LG Citations 73 Venue Transactions of the Association for Computational Linguistics Repository https://github.com/jannerm/spatial-reasoning ⭐ 52 Last Checked 1 month ago

Abstract

The interpretation of spatial references is highly contextual, requiring joint inference over both language and the environment. We consider the task of spatial reasoning in a simulated environment, where an agent can act and receive rewards. The proposed model learns a representation of the world steered by instruction text. This design allows for precise alignment of local neighborhoods with corresponding verbalizations, while also handling global references in the instructions. We train our model with reinforcement learning using a variant of generalized value iteration. The model outperforms state-of-the-art approaches on several metrics, yielding a 45% reduction in goal localization error.