). Today, I would like to share a similar one: the CoDeF model developed by Ant Group and the Hong Kong University of Science and Technology. The paper link is arxiv.org/abs/2308.07926. Those who are interested can take a deeper look.
The core idea of the CoDeF model
CoDeF, which stands for Content Deformation Field, proposes a completely new video representation method. It mainly consists of two parts: one is the canonical content field, responsible for collecting static content from the video; the other is the temporal deformation field, recording the deformation process from the canonical image to each frame of the video. This approach feels very novel to me because it extends image processing techniques into the video domain.
Experience and feelings
The CoDeF model performs exceptionally well in converting images to video, even achieving keypoint tracking without training. It handles consistency between video frames very well, including non-rigid objects like water and smoke.
Colab
For friends who want to try the CoDeF model, you can do so via the official Colab: https://colab.research.google.com/github/camenduru/CoDeF-colab/blob/main/CoDeF_colab.ipynb
Check out the results