Image captioning has seen significant research efforts over the last decade.The goal is to generate meaningful semantic sentences that describe visual content depicted in photographs and are syntactically accurate.Man...Image captioning has seen significant research efforts over the last decade.The goal is to generate meaningful semantic sentences that describe visual content depicted in photographs and are syntactically accurate.Many real-world applications rely on image captioning,such as helping people with visual impairments to see their surroundings.To formulate a coherent and relevant textual description,computer vision techniques are utilized to comprehend the visual content within an image,followed by natural language processing methods.Numerous approaches and models have been developed to deal with this multifaceted problem.Several models prove to be stateof-the-art solutions in this field.This work offers an exclusive perspective emphasizing the most critical strategies and techniques for enhancing image caption generation.Rather than reviewing all previous image captioning work,we analyze various techniques that significantly improve image caption generation and achieve significant performance improvements,including encompassing image captioning with visual attention methods,exploring semantic information types in captions,and employing multi-caption generation techniques.Further,advancements such as neural architecture search,few-shot learning,multi-phase learning,and cross-modal embedding within image caption networks are examined for their transformative effects.The comprehensive quantitative analysis conducted in this study identifies cutting-edgemethodologies and sheds light on their profound impact,driving forward the forefront of image captioning technology.展开更多
已有图像描述生成模型虽可以检测与表示图像目标实体及其视觉关系,但没有从文本句法关系角度关注模型的可解释性.因而,提出基于依存句法三元组的可解释图像描述生成模型(interpretable image caption generation based on dependency sy...已有图像描述生成模型虽可以检测与表示图像目标实体及其视觉关系,但没有从文本句法关系角度关注模型的可解释性.因而,提出基于依存句法三元组的可解释图像描述生成模型(interpretable image caption generation based on dependency syntax triplets modeling,IDSTM),以多任务学习的方式生成依存句法三元组序列和图像描述.IDSTM模型首先通过依存句法编码器从输入图像获得潜在的依存句法特征,并与依存句法三元组及文本词嵌入向量合并输入单层长短期记忆网络(long short-term memory,LSTM),生成依存句法三元组序列作为先验知识;接着,将依存句法特征输入到图像描述编码器中,提取视觉实体词特征;最后,采用硬限制和软限制2种机制,将依存句法和关系特征融合到双层LSTM,从而生成图像描述.通过依存句法三元组序列生成任务,IDSTM在未显著降低生成的图像描述精确度的前提下,提高了其可解释性.还提出了评测依存句法三元组序列生成质量的评价指标B1-DS(BLEU-1-DS),B4-DS(BLEU-4-DS),M-DS(METEOR-DS),并在MSCOCO数据集上的实验验证了IDSTM的有效性和可解释性.展开更多
基金supported by the National Natural Science Foundation of China(Nos.U22A2034,62177047)High Caliber Foreign Experts Introduction Plan funded by MOST,and Central South University Research Programme of Advanced Interdisciplinary Studies(No.2023QYJC020).
文摘Image captioning has seen significant research efforts over the last decade.The goal is to generate meaningful semantic sentences that describe visual content depicted in photographs and are syntactically accurate.Many real-world applications rely on image captioning,such as helping people with visual impairments to see their surroundings.To formulate a coherent and relevant textual description,computer vision techniques are utilized to comprehend the visual content within an image,followed by natural language processing methods.Numerous approaches and models have been developed to deal with this multifaceted problem.Several models prove to be stateof-the-art solutions in this field.This work offers an exclusive perspective emphasizing the most critical strategies and techniques for enhancing image caption generation.Rather than reviewing all previous image captioning work,we analyze various techniques that significantly improve image caption generation and achieve significant performance improvements,including encompassing image captioning with visual attention methods,exploring semantic information types in captions,and employing multi-caption generation techniques.Further,advancements such as neural architecture search,few-shot learning,multi-phase learning,and cross-modal embedding within image caption networks are examined for their transformative effects.The comprehensive quantitative analysis conducted in this study identifies cutting-edgemethodologies and sheds light on their profound impact,driving forward the forefront of image captioning technology.
