Joint Unsupervised Learning of Optical Flow and Depth by Watching Stereo Videos

October 08, 2018 · Entered Twilight · 🏛 arXiv.org

"Last commit was 6.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .gitignore, .pre-commit-config.yaml, LICENSE, LICENSE_monodepth, README.md, __init__.py, eval, filenames, flowlib.py, loss_utils.py, main.py, models.py, monodepth_dataloader.py, monodepth_model.py, nets, optical_flow_warp_fwd.py, optical_flow_warp_old.py, pose_gt_data, test.py, utils.py

Authors Yang Wang, Zhenheng Yang, Peng Wang, Yi Yang, Chenxu Luo, Wei Xu arXiv ID 1810.03654 Category cs.CV: Computer Vision Citations 14 Venue arXiv.org Repository https://github.com/baidu-research/UnDepthflow ⭐ 131 Last Checked 2 months ago

Abstract

Learning depth and optical flow via deep neural networks by watching videos has made significant progress recently. In this paper, we jointly solve the two tasks by exploiting the underlying geometric rules within stereo videos. Specifically, given two consecutive stereo image pairs from a video, we first estimate depth, camera ego-motion and optical flow from three neural networks. Then the whole scene is decomposed into moving foreground and static background by compar- ing the estimated optical flow and rigid flow derived from the depth and ego-motion. We propose a novel consistency loss to let the optical flow learn from the more accurate rigid flow in static regions. We also design a rigid alignment module which helps refine ego-motion estimation by using the estimated depth and optical flow. Experiments on the KITTI dataset show that our results significantly outperform other state-of- the-art algorithms. Source codes can be found at https: //github.com/baidu-research/UnDepthflow