publications

2023

1. ICCV23

   Multiple Planar Object Tracking

   Zhicheng Zhang, Shengzhe Liu, and Jufeng Yang

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2023

   Abs Bib HTML PDF Code

   Tracking both location and pose of multiple planar objects (MPOT) is of great significance to numerous real-world applications. The greater degree-of-freedom of planar objects compared with common objects makes MPOT far more challenging than well-studied object tracking, especially when occlusion occurs. To address this challenging task, we are inspired by amodal perception that humans jointly track visible and invisible parts of the target, and propose a tracking framework that unifies appearance perception and occlusion reasoning. Specifically, we present a dual branch network to track the visible part of planar objects, including vertexes and mask. Then, we develop an occlusion area localization strategy to infer the invisible part, i.e., the occluded region, followed by a two-stream attention network finally refining the prediction. To alleviate the lack of data in this field, we build the first large-scale benchmark dataset, namely MPOT-3K. It consists of 3,717 planar objects from 356 videos, and contains 148,896 frames together with 687,417 annotations. The collected planar objects have 9 motion patterns and the videos are shot in 6 types of indoor and outdoor scenes. Extensive experiments demonstrate the superiority of our proposed method on the newly developed MPOT-3K as well as other two popular single planar object tracking datasets.

   @inproceedings{Zhang_2023_ICCV,
     author = {Zhang, Zhicheng and Liu, Shengzhe and Yang, Jufeng},
     title = {Multiple Planar Object Tracking},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
     year = {2023},
   }

2. CVPR23

   Weakly Supervised Video Emotion Detection and Prediction via Cross-Modal Temporal Erasing Network

   Zhicheng Zhang, Lijuan Wang, and Jufeng Yang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

   Abs Bib HTML PDF Code

   Automatically predicting the emotions of user-generated videos (UGVs) receives increasing interest recently. However, existing methods mainly focus on a few key visual frames, which may limit their capacity to encode the context that depicts the intended emotions. To tackle that, in this paper, we propose a cross-modal temporal erasing network that locates not only keyframes but also context and audio-related information in a weakly-supervised manner. In specific, we first leverage the intra- and inter-modal relationship among different segments to accurately select keyframes. Then, we iteratively erase keyframes to encourage the model to concentrate on the contexts that include complementary information. Extensive experiments on three challenging benchmark datasets demonstrate that the proposed method performs favorably against the state-of-the-art approaches.

   @inproceedings{Zhang_2023_CVPR,
     author = {Zhang, Zhicheng and Wang, Lijuan and Yang, Jufeng},
     title = {Weakly Supervised Video Emotion Detection and Prediction via Cross-Modal Temporal Erasing Network},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     year = {2023},
   }

3. TNNLS

   PlaneSeg: Building a Plug-In for Boosting Planar Region Segmentation

   Zhicheng Zhang, Song Chen, Zichuan Wang, and 1 more author

   IEEE Transactions on Neural Networks and Learning Systems 2023

   Abs Bib HTML PDF Code DOI: 10.1109/TNNLS.2023.3262544

   Existing methods in planar region segmentation suffer the problems of vague boundaries and failure to detect small-sized regions. To address these, this study presents an end-to-end framework, named PlaneSeg, which can be easily integrated into various plane segmentation models. Specifically, PlaneSeg contains three modules, namely the edge feature extraction module, the multi-scale module, and the resolution-adaptation module. First, the edge feature extraction module produces edge-aware feature maps for finer segmentation boundaries. The learned edge information acts as a constraint to mitigate inaccurate boundaries. Second, the multi-scale module combines feature maps of different layers to harvest spatial and semantic information from planar objects. The multiformity of object information can help recognize small-sized objects to produce more accurate segmentation results. Third, the resolution-adaptation module fuses the feature maps produced by the two aforementioned modules. For this module, a pair-wise feature fusion is adopted to resample the dropped pixels and extract more detailed features. Extensive experiments demonstrate that PlaneSeg outperforms other state-of-the-art approaches on three downstream tasks, including plane segmentation, 3D plane reconstruction, and depth prediction.

   @article{10097456,
     author = {Zhang, Zhicheng and Chen, Song and Wang, Zichuan and Yang, Jufeng},
     journal = {IEEE Transactions on Neural Networks and Learning Systems},
     title = {PlaneSeg: Building a Plug-In for Boosting Planar Region Segmentation},
     year = {2023},
     volume = {1},
     number = {1},
     pages = {1-15},
     doi = {10.1109/TNNLS.2023.3262544},
   }

2022

1. ACM MM22

   Temporal Sentiment Localization: Listen and Look in Untrimmed Videos

   Zhicheng Zhang, and Jufeng Yang

   In Proceedings of the 30th ACM International Conference on Multimedia 2022

   Abs Bib HTML PDF Code DOI: 10.1145/3503161.3548007

   Video sentiment analysis aims to uncover the underlying attitudes of viewers, which has a wide range of applications in real world. Existing works simply classify a video into a single sentimental category, ignoring the fact that sentiment in untrimmed videos may appear in multiple segments with varying lengths and unknown locations. To address this, we propose a challenging task, i.e., Temporal Sentiment Localization (TSL), to find which parts of the video convey sentiment. To systematically investigate fully- and weakly-supervised settings for TSL, we first build a benchmark dataset named TSL-300, which is consisting of 300 videos with a total length of 1,291 minutes. Each video is labeled in two ways, one of which is frame-by-frame annotation for the fully-supervised setting, and the other is single-frame annotation, i.e., only a single frame with strong sentiment is labeled per segment for the weakly-supervised setting. Due to the high cost of labeling a densely annotated dataset, we propose TSL-Net in this work, employing single-frame supervision to localize sentiment in videos. In detail, we generate the pseudo labels for unlabeled frames using a greedy search strategy, and fuse the affective features of both visual and audio modalities to predict the temporal sentiment distribution. Here, a reverse mapping strategy is designed for feature fusion, and a contrastive loss is utilized to maintain the consistency between the original feature and the reverse prediction. Extensive experiments show the superiority of our method against the state-of-the-art approaches.

   @inproceedings{10.1145/3503161.3548007,
     author = {Zhang, Zhicheng and Yang, Jufeng},
     title = {Temporal Sentiment Localization: Listen and Look in Untrimmed Videos},
     year = {2022},
     url = {https://doi.org/10.1145/3503161.3548007},
     doi = {10.1145/3503161.3548007},
     booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
   }