Continuous Scene Representations for Embodied AI

March 31, 2022 ยท Entered Twilight ยท ๐Ÿ› Computer Vision and Pattern Recognition

๐Ÿ’ค TWILIGHT: Eternal Rest
Repo abandoned since publication

"No code URL or promise found in abstract"
"Code repo scraped from project page (backfill)"

Evidence collected by the PWNC Scanner

Repo contents: README.md, aggrigate_metrics.py, configs_contrastive, configs_exploration, configs_rearrangement, environment.yml, explore_cache_test, explore_cache_val, figs, runner_cache_trajectories.py, runner_eval_rearrangement.py, scripts, src, train_csr.py

Authors Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, Roozbeh Mottaghi arXiv ID 2203.17251 Category cs.CV: Computer Vision Cross-listed cs.AI, cs.LG, cs.RO Citations 53 Venue Computer Vision and Pattern Recognition Repository https://github.com/allenai/CSR โญ 60 Last Checked 13 days ago
Abstract
We propose Continuous Scene Representations (CSR), a scene representation constructed by an embodied agent navigating within a space, where objects and their relationships are modeled by continuous valued embeddings. Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation. Our key insight is to embed pair-wise relationships between objects in a latent space. This allows for a richer representation compared to discrete relations (e.g., [support], [next-to]) commonly used for building scene representations. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations. Using CSR, we outperform state-of-the-art approaches for the challenging downstream task of visual room rearrangement, without any task specific training. Moreover, we show the learned embeddings capture salient spatial details of the scene and show applicability to real world data. A summery video and code is available at https://prior.allenai.org/projects/csr.
Community shame:
Not yet rated
Community Contributions

Found the code? Know the venue? Think something is wrong? Let us know!

๐Ÿ“œ Similar Papers

In the same crypt โ€” Computer Vision