The side information quality has an immense effect on the compression efficiency of the distributed video coding (DVC) sys- tem. This article, based on the hierarchical motion estimation (HME), proposes a new side inf...The side information quality has an immense effect on the compression efficiency of the distributed video coding (DVC) sys- tem. This article, based on the hierarchical motion estimation (HME), proposes a new side information generation algorithm which is integrated into DVC system. First, forward motion estimation (FME) and bidirectional motion estimation (BME) on the basis of variable block size HME algorithm are used to acquire relatively accurate motion vectors. Second, a motion vector filter (MVF) is i...展开更多
Audio-visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms.Multimodal fusion methods can improve both the accuracy and rob...Audio-visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms.Multimodal fusion methods can improve both the accuracy and robustness of speaker tracking.However,in complex multispeaker tracking scenarios,critical challenges such as cross-modal feature discrepancy,weak sound source localisation ambiguity and frequent identity switch errors remain unresolved,which severely hinder the modelling of speaker identity consistency and consequently lead to degraded tracking accuracy and unstable tracking trajectories.To this end,this paper proposes a multimodal multispeaker tracking network using audio-visual contrastive learning(AVCLNet).By integrating heterogeneous modal representations into a unified space through audio-visual contrastive learning,which facilitates cross-modal feature alignment,mitigates cross-modal feature bias and enhances identity-consistent representations.In the audio-visual measurement stage,we design a vision-guided weak sound source weighted enhancement method,which leverages visual cues to establish cross-modal mappings and employs a spatiotemporal dynamic weighted mechanism to improve the detectability of weak sound sources.Furthermore,in the data association phase,a dual geometric constraint strategy is introduced by combining the 2D and 3D spatial geometric information,reducing frequent identity switch errors.Experiments on the AV16.3 and CAV3D datasets show that AVCLNet outperforms state-of-the-art methods,demonstrating superior robustness in multispeaker scenarios.展开更多
A novel Snake model with region information is proposed to detect and track moving objects. Generally, the region-information-based approach is sensitive to illumination changes and small movement in the background, w...A novel Snake model with region information is proposed to detect and track moving objects. Generally, the region-information-based approach is sensitive to illumination changes and small movement in the background, while the edge-information-based approach often obtains incorrect results for ambiguous images. The two types of information are introduced in computing the image force. Edge-information-based features make the algorithm fast and robust, and region information makes the active confour energy function obtains correct results for ambiguous images. Furthermore, an automatic contour initialization method using double difference images is given to meet the requirement of video sequence tracking. Meanwhile, a simple forecast section is added to estimate the position of the contour in the algorithm so that it can improve the convergence speed of the active contour. Experimental results show that the computation time of the algorithm is less than 0.1 s/frame. And it can be applied to a real-time system.展开更多
This paper proposes a new algorithm based on low-rank matrix recovery to remove salt &pepper noise from surveillance video. Unlike single image denoising techniques, noise removal from video sequences aims to util...This paper proposes a new algorithm based on low-rank matrix recovery to remove salt &pepper noise from surveillance video. Unlike single image denoising techniques, noise removal from video sequences aims to utilize both temporal and spatial information. By grouping neighboring frames based on similarities of the whole images in the temporal domain, we formulate the problem of removing salt &pepper noise from a video tracking sequence as a lowrank matrix recovery problem. The resulting nuclear norm and L1-norm related minimization problems can be efficiently solved by many recently developed methods. To determine the low-rank matrix, we use an averaging method based on other similar images. Our method can not only remove noise but also preserve edges and details. The performance of our proposed approach compares favorably to that of existing algorithms and gives better PSNR and SSIM results.展开更多
Based on the property analysis of interferential multispectral images, a novel compression algorithm of partial set partitioning in hierarchical trees （SPIHT） with classified weighted rate-distortion optimization is...Based on the property analysis of interferential multispectral images, a novel compression algorithm of partial set partitioning in hierarchical trees （SPIHT） with classified weighted rate-distortion optimization is presented. After wavelet decomposition, partial SPIHT is applied to each zero tree independently by adaptively selecting one of three coding modes according to the probability of the significant coefficients in each bitplane. Meanwhile the interferential multispectral image is partitioned into two kinds of regions in terms of luminous intensity, and the rate-distortion slopes of zero trees are then lifted with classified weights according to their distortion contribution to the constructed spectrum. Finally a global rate- distortion optimization truncation is performed. Compared with the conventional methods, the proposed algorithm not only improves the performance in spatial domain but also reduces the distortion in spectral domain.展开更多
基金National Natural Science Foundation of China (60702012)
文摘The side information quality has an immense effect on the compression efficiency of the distributed video coding (DVC) sys- tem. This article, based on the hierarchical motion estimation (HME), proposes a new side information generation algorithm which is integrated into DVC system. First, forward motion estimation (FME) and bidirectional motion estimation (BME) on the basis of variable block size HME algorithm are used to acquire relatively accurate motion vectors. Second, a motion vector filter (MVF) is i...
基金supported by the National Natural Science Foundation of China(62403345)the Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology(2024B1212010006)the Shanxi Provincial Department of Science and Technology Basic Research Project(202403021212174,202403021221074).
文摘Audio-visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms.Multimodal fusion methods can improve both the accuracy and robustness of speaker tracking.However,in complex multispeaker tracking scenarios,critical challenges such as cross-modal feature discrepancy,weak sound source localisation ambiguity and frequent identity switch errors remain unresolved,which severely hinder the modelling of speaker identity consistency and consequently lead to degraded tracking accuracy and unstable tracking trajectories.To this end,this paper proposes a multimodal multispeaker tracking network using audio-visual contrastive learning(AVCLNet).By integrating heterogeneous modal representations into a unified space through audio-visual contrastive learning,which facilitates cross-modal feature alignment,mitigates cross-modal feature bias and enhances identity-consistent representations.In the audio-visual measurement stage,we design a vision-guided weak sound source weighted enhancement method,which leverages visual cues to establish cross-modal mappings and employs a spatiotemporal dynamic weighted mechanism to improve the detectability of weak sound sources.Furthermore,in the data association phase,a dual geometric constraint strategy is introduced by combining the 2D and 3D spatial geometric information,reducing frequent identity switch errors.Experiments on the AV16.3 and CAV3D datasets show that AVCLNet outperforms state-of-the-art methods,demonstrating superior robustness in multispeaker scenarios.
文摘A novel Snake model with region information is proposed to detect and track moving objects. Generally, the region-information-based approach is sensitive to illumination changes and small movement in the background, while the edge-information-based approach often obtains incorrect results for ambiguous images. The two types of information are introduced in computing the image force. Edge-information-based features make the algorithm fast and robust, and region information makes the active confour energy function obtains correct results for ambiguous images. Furthermore, an automatic contour initialization method using double difference images is given to meet the requirement of video sequence tracking. Meanwhile, a simple forecast section is added to estimate the position of the contour in the algorithm so that it can improve the convergence speed of the active contour. Experimental results show that the computation time of the algorithm is less than 0.1 s/frame. And it can be applied to a real-time system.
基金supported by the National Nature Science Foundation of China (Nos. 61332015, 61373078, 61272245, and 61272430)NSFC Joint Fund with Guangdong (No. U1201258)
文摘This paper proposes a new algorithm based on low-rank matrix recovery to remove salt &pepper noise from surveillance video. Unlike single image denoising techniques, noise removal from video sequences aims to utilize both temporal and spatial information. By grouping neighboring frames based on similarities of the whole images in the temporal domain, we formulate the problem of removing salt &pepper noise from a video tracking sequence as a lowrank matrix recovery problem. The resulting nuclear norm and L1-norm related minimization problems can be efficiently solved by many recently developed methods. To determine the low-rank matrix, we use an averaging method based on other similar images. Our method can not only remove noise but also preserve edges and details. The performance of our proposed approach compares favorably to that of existing algorithms and gives better PSNR and SSIM results.
基金the National Natural Sci-ence Foundation of China (No.60532060 and 60507012)the Innovation Foundation of Xidian University (Chuang 05025).
文摘Based on the property analysis of interferential multispectral images, a novel compression algorithm of partial set partitioning in hierarchical trees （SPIHT） with classified weighted rate-distortion optimization is presented. After wavelet decomposition, partial SPIHT is applied to each zero tree independently by adaptively selecting one of three coding modes according to the probability of the significant coefficients in each bitplane. Meanwhile the interferential multispectral image is partitioned into two kinds of regions in terms of luminous intensity, and the rate-distortion slopes of zero trees are then lifted with classified weights according to their distortion contribution to the constructed spectrum. Finally a global rate- distortion optimization truncation is performed. Compared with the conventional methods, the proposed algorithm not only improves the performance in spatial domain but also reduces the distortion in spectral domain.
