MotiF: Making Text Count in Image Animation with Motion Focal Loss

Abstract

Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%.

Motivation

A common limitation of existing TI2V methods is their tendency to generate videos with limited and identical motions when given an image and multiple prompts. We hypothesize that this limitation arises from insufficient emphasis on motion patterns. As illustrated in Figure (a), in a video with a static background, 97% of pixels remain unchanged, with only 3% showing meaningful motion. Such subtle motion is often overlooked in the standard TI2V training pipeline, where all regions are optimized equally in the L2 loss. This can result in "Condition Leakage," where the loss becomes low simply by copying the condition frames. To address this, we propose Motion Focal loss (MotiF) to guide TI2V training to focus on regions with more motion via motion heatmap re-weighting.

Previous TI2V generation methods mainly focused on deriving additional motion signals (motion score and/or motion mask) as inputs for the model. On the contrary, we focus on the learning objective and propose to weight the diffusion loss based on the motion intensity. During training, we first use optical flow to create a motion heatmap that represents the motion intensities of the video, then use the motion heatmap to assign loss weight for the video to focus on regions with more motion.

TI2V Bench

TI2V Bench comprises text-image pairs from 22 diverse scenarios featuring a variety of objects and scenes. Each scenario includes 3 to 5 images with similar content presented in different styles, alongside 3 to 5 distinct prompts designed to animate these images and generate varied outputs. The benchmark includes a total of 320 image-text pairs, consisting of 88 unique images and 133 unique prompts. We also include challenging scenarios when there are multiple objects in the initial image for fine-grained control or the text prompt describes novel objects.

Human evaluation on TI2V Bench

We conduct human evaluation to compare MotiF to nine open-sourced models on TI2V Bench. We achieved considerable improvements across the board with an average preference of 72%. Analysis of justification choices reveals that MotiF excels particularly in enhancing text alignment and generating accurate object motion.

BibTeX

@misc{wang2024motifmakingtextcount, title={MotiF: Making Text Count in Image Animation with Motion Focal Loss}, author={Shijie Wang and Samaneh Azadi and Rohit Girdhar and Saketh Rambhatla and Chen Sun and Xi Yin}, year={2024}, eprint={2412.16153}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2412.16153}, }

MotiF: Making Text Count in Image Animation with Motion Focal Loss

Abstract

Motivation