TwoStreamVAN: Improving Motion Modeling in Video Generation

December 03, 2018 · Entered Twilight · 🏛 IEEE Workshop/Winter Conference on Applications of Computer Vision

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .gitignore, README.md, __init__.py, bashes, classifier, data, figures, outer_prod_motion_mask, preprocess, separable_convolution, utils, videolist

Authors Ximeng Sun, Huijuan Xu, Kate Saenko arXiv ID 1812.01037 Category cs.CV: Computer Vision Citations 19 Venue IEEE Workshop/Winter Conference on Applications of Computer Vision Repository https://github.com/sunxm2357/TwoStreamVAN/ ⭐ 9 Last Checked 1 month ago

Abstract

Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content. Existing methods entangle the two intrinsically different tasks of motion and content creation in a single generator network, but this approach struggles to simultaneously generate plausible motion and content. To im-prove motion modeling in video generation tasks, we propose a two-stream model that disentangles motion generation from content generation, called a Two-Stream Variational Adversarial Network (TwoStreamVAN). Given an action label and a noise vector, our model is able to create clear and consistent motion, and thus yields photorealistic videos. The key idea is to progressively generate and fuse multi-scale motion with its corresponding spatial content. Our model significantly outperforms existing methods on the standard Weizmann Human Action, MUG Facial Expression, and VoxCeleb datasets, as well as our new dataset of diverse human actions with challenging and complex motion. Our code is available at https://github.com/sunxm2357/TwoStreamVAN/.