Aiming at the problems of incomplete characterization of text relations,poor guidance of potential representations,and low quality of model generation in the field of controllable long text generation,this paper propo...Aiming at the problems of incomplete characterization of text relations,poor guidance of potential representations,and low quality of model generation in the field of controllable long text generation,this paper proposes a new GSPT-CVAE model(Graph Structured Processing,Single Vector,and Potential Attention Com-puting Transformer-Based Conditioned Variational Autoencoder model).The model obtains a more comprehensive representation of textual relations by graph-structured processing of the input text,and at the same time obtains a single vector representation by weighted merging of the vector sequences after graph-structured processing to get an effective potential representation.In the process of potential representation guiding text generation,the model adopts a combination of traditional embedding and potential attention calculation to give full play to the guiding role of potential representation for generating text,to improve the controllability and effectiveness of text generation.The experimental results show that the model has excellent representation learning ability and can learn rich and useful textual relationship representations.The model also achieves satisfactory results in the effectiveness and controllability of text generation and can generate long texts that match the given constraints.The ROUGE-1 F1 score of this model is 0.243,the ROUGE-2 F1 score is 0.041,the ROUGE-L F1 score is 0.22,and the PPL-Word score is 34.303,which gives the GSPT-CVAE model a certain advantage over the baseline model.Meanwhile,this paper compares this model with the state-of-the-art generative models T5,GPT-4,Llama2,and so on,and the experimental results show that the GSPT-CVAE model has a certain competitiveness.展开更多
Purpose:A text generation based multidisciplinary problem identification method is proposed,which does not rely on a large amount of data annotation.Design/methodology/approach:The proposed method first identifies the...Purpose:A text generation based multidisciplinary problem identification method is proposed,which does not rely on a large amount of data annotation.Design/methodology/approach:The proposed method first identifies the research objective types and disciplinary labels of papers using a text classification technique;second,it generates abstractive titles for each paper based on abstract and research objective types using a generative pre-trained language model;third,it extracts problem phrases from generated titles according to regular expression rules;fourth,it creates problem relation networks and identifies the same problems by exploiting a weighted community detection algorithm;finally,it identifies multidisciplinary problems based on the disciplinary labels of papers.Findings:Experiments in the“Carbon Peaking and Carbon Neutrality”field show that the proposed method can effectively identify multidisciplinary research problems.The disciplinary distribution of the identified problems is consistent with our understanding of multidisciplinary collaboration in the field.Research limitations:It is necessary to use the proposed method in other multidisciplinary fields to validate its effectiveness.Practical implications:Multidisciplinary problem identification helps to gather multidisciplinary forces to solve complex real-world problems for the governments,fund valuable multidisciplinary problems for research management authorities,and borrow ideas from other disciplines for researchers.Originality/value:This approach proposes a novel multidisciplinary problem identification method based on text generation,which identifies multidisciplinary problems based on generative abstractive titles of papers without data annotation required by standard sequence labeling techniques.展开更多
Generation-based linguistic steganography is a popular research area of information hiding.The text generative steganographic method based on conditional probability coding is the direction that researchers have recen...Generation-based linguistic steganography is a popular research area of information hiding.The text generative steganographic method based on conditional probability coding is the direction that researchers have recently paid attention to.However,in the course of our experiment,we found that the secret information hiding in the text tends to destroy the statistical distribution characteristics of the original text,which indicates that this method has the problem of the obvious reduction of text quality when the embedding rate increases,and that the topic of generated texts is uncontrollable,so there is still room for improvement in concealment.In this paper,we propose a topic-controlled steganography method which is guided by graph-to-text generation.The proposed model can automatically generate steganographic texts carrying secret messages from knowledge graphs,and the topic of the generated texts is controllable.We also provide a graph path coding method with corresponding detailed algorithms for graph-to-text generation.Different from traditional linguistic steganography methods,we encode the secret information during graph path coding rather than using conditional probability.We test our method in different aspects and compare it with other text generative steganographic methods.The experimental results show that the model proposed in this paper can effectively improve the quality of the generated text and significantly improve the concealment of steganographic text.展开更多
Digitization,informatization,and Internet penetration have led to a significant rise in cross-border e-commerce(CBEC),attracting considerable interest from academia,government,and industry.This study employed a novel ...Digitization,informatization,and Internet penetration have led to a significant rise in cross-border e-commerce(CBEC),attracting considerable interest from academia,government,and industry.This study employed a novel method combining automatic text generation technology and traditional bibliometric analysis to summarize and categorize the research on CBEC evolution from 2000 to 2021.Articles were selected and examined with a focus on four dimensions:customer,risk,supply chain,and platform.Contradictions in these dimensions were found to result in two major obstacles to CBEC development,namely,dataset sharing and platform scalability.These obstacles prevent research on cross-border platforms from moving beyond theory-based studies.Further research needs to examine how soft computing can be used to accelerate and remodel the global trade ecosystem.展开更多
As an important subject of natural language generation,Controllable Text Generation(CTG)focuses on integrating additional constraints and controls while generating texts and has attracted a lot of attention.Existing c...As an important subject of natural language generation,Controllable Text Generation(CTG)focuses on integrating additional constraints and controls while generating texts and has attracted a lot of attention.Existing controllable text generation approaches mainly capture the statistical association implied within training texts,but generated texts lack causality consideration.This paper intends to review recent CTG approaches from a causal perspective.Firstly,according to previous research on basic types of CTG models,it is discovered that their essence is to obtain the association,and then four kinds of challenges caused by absence of causality are introduced.Next,this paper reviews the improvements to address these challenges from four aspects,namely representation disentanglement,causal inference,knowledge enhancement and multi-aspect CTG respectively.Additionally,this paper inspects existing evaluations of CTG,especially evaluations for causality of CTG.Finally,this review discusses some future research directions for the causality improvement of CTG and makes a conclusion.展开更多
Text generation is an essential research area in artificial intelligence(AI)technology and natural language processing and provides key technical support for the rapid development of AI-generated content(AIGC).It is b...Text generation is an essential research area in artificial intelligence(AI)technology and natural language processing and provides key technical support for the rapid development of AI-generated content(AIGC).It is based on technologies such as natural language processing,machine learning,and deep learning,which enable learning language rules through training models to automatically generate text that meets grammatical and semantic requirements.In this paper,we sort and systematically summarize the main research progress in text generation and review recent text generation papers,focusing on presenting a detailed understanding of the technical models.In addition,several typical text generation application systems are presented.Finally,we address some challenges and future directions in AI text generation.We conclude that improving the quality,quantity,interactivity,and adaptability of generated text can help fundamentally advance AI text generation development.展开更多
Recently,generative artificial intelligence(GenAI)has developed into a new form of technology that can create copy,image,audio,and video content and adapt it to individual preferences on every channel and moment autom...Recently,generative artificial intelligence(GenAI)has developed into a new form of technology that can create copy,image,audio,and video content and adapt it to individual preferences on every channel and moment automatically.But most fail at proof-of-concept,as the pipelines needed to govern data,generate it controllably,deliver it,and do causal evaluation are absent or poorly aligned.This paper puts forward a practical end-to-end framework concerning personalized advertising driven by GenAI,which combines representation learning,constrained generation,and experimentation into a single operating cycle.First,we pick a modular architecture:profiles and contexts go into controllable large language and diffusion models that yield brand-safe assets under deterministic conditioning,which are chosen via a contextual bandit and vetted by policy and equality guardrails.Second,we give a measurement stack going from straightforward A/B/n tests to doubly-robust uplift modeling,making it possible to find out diverse treatment effects that are good to use in business metrics(incremental conversions and profit).Third,we operationalize latency budgets,humans in the loop,red teams,safety filters,and post-deployment monitoring with clear escalation paths.We focus throughout the paper on reproducibility,privacy(consent,privacy,differential privacy,on-device inference),and on GDPR/CCPA-like governance specifications.We end on our actionable blueprint,algorithmic choices,sample prompts,KPIs,and step-wise rollout to achieve trustworthy performance upgrades without putting creative quality,fairness,or compliance to the test.展开更多
In recent years,deep neural network has achieved great success in solving many natural language processing tasks.Particularly,substantial progress has been made on neural text generation,which takes the linguistic and...In recent years,deep neural network has achieved great success in solving many natural language processing tasks.Particularly,substantial progress has been made on neural text generation,which takes the linguistic and non-linguistic input,and generates natural language text.This survey aims to provide an up-to-date synthesis of core tasks in neural text generation and the architectures adopted to handle these tasks,and draw attention to the challenges in neural text generation.We first outline the mainstream neural text generation frameworks,and then introduce datasets,advanced models and challenges of four core text generation tasks in detail,including AMR-to-text generation,data-to-text generation,and two text-to-text generation tasks(i.e.,text summarization and paraphrase generation).Finally,we present future research directions for neural text generation.This survey can be used as a guide and reference for researchers and practitioners in this area.展开更多
Generating diverse and factual text is challenging and is receiving increasing attention.By sampling from the latent space,variational autoencoder-based models have recently enhanced the diversity of generated text.Ho...Generating diverse and factual text is challenging and is receiving increasing attention.By sampling from the latent space,variational autoencoder-based models have recently enhanced the diversity of generated text.However,existing research predominantly depends on summarizationmodels to offer paragraph-level semantic information for enhancing factual correctness.The challenge lies in effectively generating factual text using sentence-level variational autoencoder-based models.In this paper,a novel model called fact-aware conditional variational autoencoder is proposed to balance the factual correctness and diversity of generated text.Specifically,our model encodes the input sentences and uses them as facts to build a conditional variational autoencoder network.By training a conditional variational autoencoder network,the model is enabled to generate text based on input facts.Building upon this foundation,the input text is passed to the discriminator along with the generated text.By employing adversarial training,the model is encouraged to generate text that is indistinguishable to the discriminator,thereby enhancing the quality of the generated text.To further improve the factual correctness,inspired by the natural language inference system,the entailment recognition task is introduced to be trained together with the discriminator via multi-task learning.Moreover,based on the entailment recognition results,a penalty term is further proposed to reconstruct the loss of our model,forcing the generator to generate text consistent with the facts.Experimental results demonstrate that compared with competitivemodels,ourmodel has achieved substantial improvements in both the quality and factual correctness of the text,despite only sacrificing a small amount of diversity.Furthermore,when considering a comprehensive evaluation of diversity and quality metrics,our model has also demonstrated the best performance.展开更多
To address the difficulty of training high-quality models in some specific domains due to the lack of fine-grained annotation resources, we propose in this paper a knowledge-integrated cross-domain data generation met...To address the difficulty of training high-quality models in some specific domains due to the lack of fine-grained annotation resources, we propose in this paper a knowledge-integrated cross-domain data generation method for unsupervised domain adaptation tasks. Specifically, we extract domain features, lexical and syntactic knowledge from source-domain and target-domain data, and use a masking model with an extended masking strategy and a re-masking strategy to obtain domain-specific data that remove domain-specific features. Finally, we improve the sequence generation model BART and use it to generate high-quality target domain data for the task of aspect and opinion co-extraction from the target domain. Experiments were performed on three conventional English datasets from different domains, and our method generates more accurate and diverse target domain data with the best results compared to previous methods.展开更多
Cyber security addresses the protection of information systems in cyberspace. These systems face multiple attacks on a daily basis, with the level of complication getting increasingly challenging. Despite the existenc...Cyber security addresses the protection of information systems in cyberspace. These systems face multiple attacks on a daily basis, with the level of complication getting increasingly challenging. Despite the existence of multiple solutions, attackers are still quite successful at identifying vulnerabilities to exploit. This is why cyber deception is increasingly being used to divert attackers’ attention and, therefore, enhance the security of information systems. To be effective, deception environments need fake data. This is where Natural Language (NLP) Processing comes in. Many cyber security models have used NLP for vulnerability detection in information systems, email classification, fake citation detection, and many others. Although it is used for text generation, existing models seem to be unsuitable for data generation in a deception environment. Our goal is to use text generation in NLP to generate data in the deception context that will be used to build multi-level deception in information systems. Our model consists of three (3) components, including the connection component, the deception component, composed of several states in which an attacker may be, depending on whether he is malicious or not, and the text generation component. The text generation component considers as input the real data of the information system and allows the production of several texts as output, which are usable at different deception levels.展开更多
Recent years have witnessed the widespread adoption of mobile applications(apps for short).For quality-of-service and commercial competitiveness,sufficient Graphical User Interface(GUI)testing is required to verify th...Recent years have witnessed the widespread adoption of mobile applications(apps for short).For quality-of-service and commercial competitiveness,sufficient Graphical User Interface(GUI)testing is required to verify the robustness of the apps.Given that testing with manual efforts is time-consuming and error-prone,automated GUI testing has been widely studied.However,existing approaches mostly focus on GUI exploration while lacking attention to complex interactions with apps,especially generating appropriate text inputs like real users.In this paper,we introduce CamDroid,a lightweight context-aware automated GUI testing tool,which can efficiently explore app activities through(1)a model-based UI-guided testing strategy informed by the context of previous event-activity transitions and(2)a data-driven text input generation approach regarding the GUI context.We evaluate CamDroid on 20 widely-used apps.The results show that CamDroid outperforms non-trivial baselines in activity coverage,crash detection,and test efficiency.展开更多
Steganography based on generative adversarial networks(GANs)has become a hot topic among researchers.Due to GANs being unsuitable for text fields with discrete characteristics,researchers have proposed GANbased stegan...Steganography based on generative adversarial networks(GANs)has become a hot topic among researchers.Due to GANs being unsuitable for text fields with discrete characteristics,researchers have proposed GANbased steganography methods that are less dependent on text.In this paper,we propose a new method of generative lyrics steganography based on GANs,called GAN-GLS.The proposed method uses the GAN model and the largescale lyrics corpus to construct and train a lyrics generator.In this method,the GAN uses a previously generated line of a lyric as the input sentence in order to generate the next line of the lyric.Using a strategy based on the penalty mechanism in training,the GAN model generates non-repetitive and diverse lyrics.The secret information is then processed according to the data characteristics of the generated lyrics in order to hide information.Unlike other text generation-based linguistic steganographic methods,our method changes the way that multiple generated candidate items are selected as the candidate groups in order to encode the conditional probability distribution.The experimental results demonstrate that our method can generate highquality lyrics as stego-texts.Moreover,compared with other similar methods,the proposed method achieves good performance in terms of imperceptibility,embedding rate,effectiveness,extraction success rate and security.展开更多
Chinese medicine(CM)diagnosis intellectualization is one of the hotspots in the research of CM modernization.The traditional CM intelligent diagnosis models transform the CM diagnosis issues into classification issues...Chinese medicine(CM)diagnosis intellectualization is one of the hotspots in the research of CM modernization.The traditional CM intelligent diagnosis models transform the CM diagnosis issues into classification issues,however,it is difficult to solve the problems such as excessive or similar categories.With the development of natural language processing techniques,text generation technique has become increasingly mature.In this study,we aimed to establish the CM diagnosis generation model by transforming the CM diagnosis issues into text generation issues.The semantic context characteristic learning capacity was enhanced referring to Bidirectional Long Short-Term Memory(BILSTM)with Transformer as the backbone network.Meanwhile,the CM diagnosis generation model Knowledge Graph Enhanced Transformer(KGET)was established by introducing the knowledge in medical field to enhance the inferential capability.The KGET model was established based on 566 CM case texts,and was compared with the classic text generation models including Long Short-Term Memory sequence-to-sequence(LSTM-seq2seq),Bidirectional and Auto-Regression Transformer(BART),and Chinese Pre-trained Unbalanced Transformer(CPT),so as to analyze the model manifestations.Finally,the ablation experiments were performed to explore the influence of the optimized part on the KGET model.The results of Bilingual Evaluation Understudy(BLEU),Recall-Oriented Understudy for Gisting Evaluation 1(ROUGE1),ROUGE2 and Edit distance of KGET model were 45.85,73.93,54.59 and 7.12,respectively in this study.Compared with LSTM-seq2seq,BART and CPT models,the KGET model was higher in BLEU,ROUGE1 and ROUGE2 by 6.00–17.09,1.65–9.39 and 0.51–17.62,respectively,and lower in Edit distance by 0.47–3.21.The ablation experiment results revealed that introduction of BILSTM model and prior knowledge could significantly increase the model performance.Additionally,the manual assessment indicated that the CM diagnosis results of the KGET model used in this study were highly consistent with the practical diagnosis results.In conclusion,text generation technology can be effectively applied to CM diagnostic modeling.It can effectively avoid the problem of poor diagnostic performance caused by excessive and similar categories in traditional CM diagnostic classification models.CM diagnostic text generation technology has broad application prospects in the future.展开更多
Software is a crucial component in the communication systems,and its security is of paramount importance.However,it is susceptible to different types of attacks due to potential vulnerabilities.Meanwhile,significant t...Software is a crucial component in the communication systems,and its security is of paramount importance.However,it is susceptible to different types of attacks due to potential vulnerabilities.Meanwhile,significant time and effort is required to fix such vulnerabilities.We propose an automated program repair method based on controlled text generation techniques.Specifically,we utilize a fine-tuned language model for patch generation and introduce a discriminator to evaluate the generation process,selecting results that contribute most to vulnerability fixes.Additionally,we perform static syntax analysis to expedite the patch verification process.The effectiveness of the proposed approach is validated using QuixBugs and Defects4J datasets,demonstrating significant improvements in generating correct patches compared to other existing methods.展开更多
Aiming at complex and changeable factors such as speech theme and environment,which make it difficult for a speaker to prepare the speech text in a short time,this paper proposes a speech generation and demonstration s...Aiming at complex and changeable factors such as speech theme and environment,which make it difficult for a speaker to prepare the speech text in a short time,this paper proposes a speech generation and demonstration system based on deep learning.This system is based on the Deep Learning Development Framework(PyTorch),trained through the theory of GPT-2 and the open source pretrained model,to generate multiple speeches according to the topics given by users,and the system generates thefinal speech and corresponding voice demon-stration audio through text modification,speech synthesis and other technologies to help users quickly obtain the target document and audio.Experiments show that the text generated by this model is smooth and easy to use,which helps shorten the preparation time of speakers and improves the confidence of the impromptu speaker.In addition,the paper explores the application prospects of text generation and has certain reference value.展开更多
1 Introduction With rapid development in computing power and breakthroughs in deep learning,the concept of“foundation models”has been introduced into the AI community.Generally,foundation models are large models tra...1 Introduction With rapid development in computing power and breakthroughs in deep learning,the concept of“foundation models”has been introduced into the AI community.Generally,foundation models are large models trained on massive data and can be easily adapted to different domains for various tasks.With specific prompts,foundation models can generate texts and images,or even animate scenarios based on the given descriptions.Due to powerful capabilities,there is a growing trend to build agents based on foundation models.In this paper,we conduct an investigation into agents empowered by the foundation models.展开更多
In this study,we explore the potential of Multiway Transformers for text-to-image generation to achieve performance improvements through a concise and efficient decoupled model design and the inference efficiency prov...In this study,we explore the potential of Multiway Transformers for text-to-image generation to achieve performance improvements through a concise and efficient decoupled model design and the inference efficiency provided by bidirectional encoding.We propose a method for improving the image tokenizer using pretrained Vision Transformers.Next,we employ bidirectional Multiway Transformers to restore the masked visual tokens combined with the unmasked text tokens.On the MS-COCO benchmark,our Multiway Transformers outperform vanilla Transformers,achieving superior FID scores and confirming the efficacy of the modality-specific parameter computation design.Ablation studies reveal that the fusion of visual and text tokens in bidirectional encoding contributes to improved model performance.Additionally,our proposed tokenizer outperforms VQGAN in image reconstruction quality and enhances the text-to-image generation results.By incorporating the additional CC-3M dataset for intermediate finetuning on our model with 688M parameters,we achieve competitive results with a finetuned FID score of 4.98 on MS-COCO.展开更多
This paper focuses on term-status pair extraction from medical dialogues(MD-TSPE),which is essential in diagnosis dia-logue systems and the automatic scribe of electronic medical records(EMRs).In the past few years,wo...This paper focuses on term-status pair extraction from medical dialogues(MD-TSPE),which is essential in diagnosis dia-logue systems and the automatic scribe of electronic medical records(EMRs).In the past few years,works on MD-TSPE have attracted increasing research attention,especially after the remarkable progress made by generative methods.However,these generative methods output a whole sequence consisting of term-status pairs in one stage and ignore integrating prior knowledge,which demands a deeper un-derstanding to model the relationship between terms and infer the status of each term.This paper presents a knowledge-enhanced two-stage generative framework(KTGF)to address the above challenges.Using task-specific prompts,we employ a single model to com-plete the MD-TSPE through two phases in a unified generative form:We generate all terms the first and then generate the status of each generated term.In this way,the relationship between terms can be learned more effectively from the sequence containing only terms in the first phase,and our designed knowledge-enhanced prompt in the second phase can leverage the category and status candidates of the generated term for status generation.Furthermore,our proposed special status"not mentioned"makes more terms available and en-riches the training data in the second phase,which is critical in the low-resource setting.The experiments on the Chunyu and CMDD datasets show that the proposed method achieves superior results compared to the state-of-the-art models in the full training and low-re-sourcesettings.展开更多
In the paper an intelligent speech production system is established by using language information processing technology. The concept of bi-directional grammar is proposed in Chinese language information processing and...In the paper an intelligent speech production system is established by using language information processing technology. The concept of bi-directional grammar is proposed in Chinese language information processing and a corresponding Chinese characteristic network is completed. Correct text can be generated through grammar parsing and some additional rules. According to the generated text the system generates speech which has good quality in naturalness and intelligibility using Chinese Text-to-Speech Conversion System.展开更多
文摘Aiming at the problems of incomplete characterization of text relations,poor guidance of potential representations,and low quality of model generation in the field of controllable long text generation,this paper proposes a new GSPT-CVAE model(Graph Structured Processing,Single Vector,and Potential Attention Com-puting Transformer-Based Conditioned Variational Autoencoder model).The model obtains a more comprehensive representation of textual relations by graph-structured processing of the input text,and at the same time obtains a single vector representation by weighted merging of the vector sequences after graph-structured processing to get an effective potential representation.In the process of potential representation guiding text generation,the model adopts a combination of traditional embedding and potential attention calculation to give full play to the guiding role of potential representation for generating text,to improve the controllability and effectiveness of text generation.The experimental results show that the model has excellent representation learning ability and can learn rich and useful textual relationship representations.The model also achieves satisfactory results in the effectiveness and controllability of text generation and can generate long texts that match the given constraints.The ROUGE-1 F1 score of this model is 0.243,the ROUGE-2 F1 score is 0.041,the ROUGE-L F1 score is 0.22,and the PPL-Word score is 34.303,which gives the GSPT-CVAE model a certain advantage over the baseline model.Meanwhile,this paper compares this model with the state-of-the-art generative models T5,GPT-4,Llama2,and so on,and the experimental results show that the GSPT-CVAE model has a certain competitiveness.
基金supported by the General Projects of ISTIC Innovation Foundation“Problem innovation solution mining based on text generation model”(MS2024-03).
文摘Purpose:A text generation based multidisciplinary problem identification method is proposed,which does not rely on a large amount of data annotation.Design/methodology/approach:The proposed method first identifies the research objective types and disciplinary labels of papers using a text classification technique;second,it generates abstractive titles for each paper based on abstract and research objective types using a generative pre-trained language model;third,it extracts problem phrases from generated titles according to regular expression rules;fourth,it creates problem relation networks and identifies the same problems by exploiting a weighted community detection algorithm;finally,it identifies multidisciplinary problems based on the disciplinary labels of papers.Findings:Experiments in the“Carbon Peaking and Carbon Neutrality”field show that the proposed method can effectively identify multidisciplinary research problems.The disciplinary distribution of the identified problems is consistent with our understanding of multidisciplinary collaboration in the field.Research limitations:It is necessary to use the proposed method in other multidisciplinary fields to validate its effectiveness.Practical implications:Multidisciplinary problem identification helps to gather multidisciplinary forces to solve complex real-world problems for the governments,fund valuable multidisciplinary problems for research management authorities,and borrow ideas from other disciplines for researchers.Originality/value:This approach proposes a novel multidisciplinary problem identification method based on text generation,which identifies multidisciplinary problems based on generative abstractive titles of papers without data annotation required by standard sequence labeling techniques.
基金supported in part by the National Natural Science Foundation of China [62102136]the 2020 Opening Fund for Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering [2020SDSJ06]the Construction Fund for Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering [2019ZYYD007].
文摘Generation-based linguistic steganography is a popular research area of information hiding.The text generative steganographic method based on conditional probability coding is the direction that researchers have recently paid attention to.However,in the course of our experiment,we found that the secret information hiding in the text tends to destroy the statistical distribution characteristics of the original text,which indicates that this method has the problem of the obvious reduction of text quality when the embedding rate increases,and that the topic of generated texts is uncontrollable,so there is still room for improvement in concealment.In this paper,we propose a topic-controlled steganography method which is guided by graph-to-text generation.The proposed model can automatically generate steganographic texts carrying secret messages from knowledge graphs,and the topic of the generated texts is controllable.We also provide a graph path coding method with corresponding detailed algorithms for graph-to-text generation.Different from traditional linguistic steganography methods,we encode the secret information during graph path coding rather than using conditional probability.We test our method in different aspects and compare it with other text generative steganographic methods.The experimental results show that the model proposed in this paper can effectively improve the quality of the generated text and significantly improve the concealment of steganographic text.
基金supported by the National Natural Science Foundation(NSFC)Programs of China(Grant Nos.:72011540408 and 72032006)the National Research Foundation of Korea(Grant No.:NRF-2020K2A9A2A06069972)the support of the Youth Innovation Team of Shaanxi Universities“Big data and Business Intelligent Innovation Team”and Shaanxi Superiority Funding Project for Scientific and Technological Activities of Overseas Scholars(Grant No.:2018017).
文摘Digitization,informatization,and Internet penetration have led to a significant rise in cross-border e-commerce(CBEC),attracting considerable interest from academia,government,and industry.This study employed a novel method combining automatic text generation technology and traditional bibliometric analysis to summarize and categorize the research on CBEC evolution from 2000 to 2021.Articles were selected and examined with a focus on four dimensions:customer,risk,supply chain,and platform.Contradictions in these dimensions were found to result in two major obstacles to CBEC development,namely,dataset sharing and platform scalability.These obstacles prevent research on cross-border platforms from moving beyond theory-based studies.Further research needs to examine how soft computing can be used to accelerate and remodel the global trade ecosystem.
基金supported in part by the National Key Research and Development Program of China(2022YFB4501704)in part by the Shanghai Science and Technology Innovation Action Plan Project(22511100700).
文摘As an important subject of natural language generation,Controllable Text Generation(CTG)focuses on integrating additional constraints and controls while generating texts and has attracted a lot of attention.Existing controllable text generation approaches mainly capture the statistical association implied within training texts,but generated texts lack causality consideration.This paper intends to review recent CTG approaches from a causal perspective.Firstly,according to previous research on basic types of CTG models,it is discovered that their essence is to obtain the association,and then four kinds of challenges caused by absence of causality are introduced.Next,this paper reviews the improvements to address these challenges from four aspects,namely representation disentanglement,causal inference,knowledge enhancement and multi-aspect CTG respectively.Additionally,this paper inspects existing evaluations of CTG,especially evaluations for causality of CTG.Finally,this review discusses some future research directions for the causality improvement of CTG and makes a conclusion.
基金Project supported by the National Natural Science Foundation of China(No.62272100)the Consulting Project of Chinese Academy of Engineering(No.2023-XY-09)+1 种基金the Major Project of the National Social Science Fund of China(No.21ZD11)the Fundamental Research Funds for the Central Universities,China。
文摘Text generation is an essential research area in artificial intelligence(AI)technology and natural language processing and provides key technical support for the rapid development of AI-generated content(AIGC).It is based on technologies such as natural language processing,machine learning,and deep learning,which enable learning language rules through training models to automatically generate text that meets grammatical and semantic requirements.In this paper,we sort and systematically summarize the main research progress in text generation and review recent text generation papers,focusing on presenting a detailed understanding of the technical models.In addition,several typical text generation application systems are presented.Finally,we address some challenges and future directions in AI text generation.We conclude that improving the quality,quantity,interactivity,and adaptability of generated text can help fundamentally advance AI text generation development.
文摘Recently,generative artificial intelligence(GenAI)has developed into a new form of technology that can create copy,image,audio,and video content and adapt it to individual preferences on every channel and moment automatically.But most fail at proof-of-concept,as the pipelines needed to govern data,generate it controllably,deliver it,and do causal evaluation are absent or poorly aligned.This paper puts forward a practical end-to-end framework concerning personalized advertising driven by GenAI,which combines representation learning,constrained generation,and experimentation into a single operating cycle.First,we pick a modular architecture:profiles and contexts go into controllable large language and diffusion models that yield brand-safe assets under deterministic conditioning,which are chosen via a contextual bandit and vetted by policy and equality guardrails.Second,we give a measurement stack going from straightforward A/B/n tests to doubly-robust uplift modeling,making it possible to find out diverse treatment effects that are good to use in business metrics(incremental conversions and profit).Third,we operationalize latency budgets,humans in the loop,red teams,safety filters,and post-deployment monitoring with clear escalation paths.We focus throughout the paper on reproducibility,privacy(consent,privacy,differential privacy,on-device inference),and on GDPR/CCPA-like governance specifications.We end on our actionable blueprint,algorithmic choices,sample prompts,KPIs,and step-wise rollout to achieve trustworthy performance upgrades without putting creative quality,fairness,or compliance to the test.
基金the National Natural Science Foundation of China(Grant No.61772036)the Key Laboratory of Science,Technology and Standard in Press Industry(Key Laboratory of Intelligent Press Media Technology)。
文摘In recent years,deep neural network has achieved great success in solving many natural language processing tasks.Particularly,substantial progress has been made on neural text generation,which takes the linguistic and non-linguistic input,and generates natural language text.This survey aims to provide an up-to-date synthesis of core tasks in neural text generation and the architectures adopted to handle these tasks,and draw attention to the challenges in neural text generation.We first outline the mainstream neural text generation frameworks,and then introduce datasets,advanced models and challenges of four core text generation tasks in detail,including AMR-to-text generation,data-to-text generation,and two text-to-text generation tasks(i.e.,text summarization and paraphrase generation).Finally,we present future research directions for neural text generation.This survey can be used as a guide and reference for researchers and practitioners in this area.
基金supported by the Science and Technology Department of Sichuan Province(No.2021YFG0156).
文摘Generating diverse and factual text is challenging and is receiving increasing attention.By sampling from the latent space,variational autoencoder-based models have recently enhanced the diversity of generated text.However,existing research predominantly depends on summarizationmodels to offer paragraph-level semantic information for enhancing factual correctness.The challenge lies in effectively generating factual text using sentence-level variational autoencoder-based models.In this paper,a novel model called fact-aware conditional variational autoencoder is proposed to balance the factual correctness and diversity of generated text.Specifically,our model encodes the input sentences and uses them as facts to build a conditional variational autoencoder network.By training a conditional variational autoencoder network,the model is enabled to generate text based on input facts.Building upon this foundation,the input text is passed to the discriminator along with the generated text.By employing adversarial training,the model is encouraged to generate text that is indistinguishable to the discriminator,thereby enhancing the quality of the generated text.To further improve the factual correctness,inspired by the natural language inference system,the entailment recognition task is introduced to be trained together with the discriminator via multi-task learning.Moreover,based on the entailment recognition results,a penalty term is further proposed to reconstruct the loss of our model,forcing the generator to generate text consistent with the facts.Experimental results demonstrate that compared with competitivemodels,ourmodel has achieved substantial improvements in both the quality and factual correctness of the text,despite only sacrificing a small amount of diversity.Furthermore,when considering a comprehensive evaluation of diversity and quality metrics,our model has also demonstrated the best performance.
文摘To address the difficulty of training high-quality models in some specific domains due to the lack of fine-grained annotation resources, we propose in this paper a knowledge-integrated cross-domain data generation method for unsupervised domain adaptation tasks. Specifically, we extract domain features, lexical and syntactic knowledge from source-domain and target-domain data, and use a masking model with an extended masking strategy and a re-masking strategy to obtain domain-specific data that remove domain-specific features. Finally, we improve the sequence generation model BART and use it to generate high-quality target domain data for the task of aspect and opinion co-extraction from the target domain. Experiments were performed on three conventional English datasets from different domains, and our method generates more accurate and diverse target domain data with the best results compared to previous methods.
文摘Cyber security addresses the protection of information systems in cyberspace. These systems face multiple attacks on a daily basis, with the level of complication getting increasingly challenging. Despite the existence of multiple solutions, attackers are still quite successful at identifying vulnerabilities to exploit. This is why cyber deception is increasingly being used to divert attackers’ attention and, therefore, enhance the security of information systems. To be effective, deception environments need fake data. This is where Natural Language (NLP) Processing comes in. Many cyber security models have used NLP for vulnerability detection in information systems, email classification, fake citation detection, and many others. Although it is used for text generation, existing models seem to be unsuitable for data generation in a deception environment. Our goal is to use text generation in NLP to generate data in the deception context that will be used to build multi-level deception in information systems. Our model consists of three (3) components, including the connection component, the deception component, composed of several states in which an attacker may be, depending on whether he is malicious or not, and the text generation component. The text generation component considers as input the real data of the information system and allows the production of several texts as output, which are usable at different deception levels.
基金supported by the National Key R&D Program of China(No.2022YFB4500703)the National Natural Science Foundation of China(Nos.61902211 and 62202266)+1 种基金the China Postdoctoral Science Foundation(No.2022M721831)Microsoft Research Asia(No.100336949).
文摘Recent years have witnessed the widespread adoption of mobile applications(apps for short).For quality-of-service and commercial competitiveness,sufficient Graphical User Interface(GUI)testing is required to verify the robustness of the apps.Given that testing with manual efforts is time-consuming and error-prone,automated GUI testing has been widely studied.However,existing approaches mostly focus on GUI exploration while lacking attention to complex interactions with apps,especially generating appropriate text inputs like real users.In this paper,we introduce CamDroid,a lightweight context-aware automated GUI testing tool,which can efficiently explore app activities through(1)a model-based UI-guided testing strategy informed by the context of previous event-activity transitions and(2)a data-driven text input generation approach regarding the GUI context.We evaluate CamDroid on 20 widely-used apps.The results show that CamDroid outperforms non-trivial baselines in activity coverage,crash detection,and test efficiency.
基金This work was supported in part by the National Natural Science Foundation of China under Grant 61872134,61672222,author Y.L.Liu,http://www.nsfc.gov.cn/in part by Science and Technology Development Center of the Ministry of Education under Grant 2019J01020,author Y.L.Liu,http://www.moe.gov.cn/+1 种基金in part by Science and Technology Project of Transport Department of Hunan Province under Grant 201935,author Y.L.Liu,http://jtt.hunan.gov.cn/Science and Technology Program of Changsha City under Grant kh200519,kq2004021,author Y.L.Liu,http://kjj.changsha.gov.cn/.
文摘Steganography based on generative adversarial networks(GANs)has become a hot topic among researchers.Due to GANs being unsuitable for text fields with discrete characteristics,researchers have proposed GANbased steganography methods that are less dependent on text.In this paper,we propose a new method of generative lyrics steganography based on GANs,called GAN-GLS.The proposed method uses the GAN model and the largescale lyrics corpus to construct and train a lyrics generator.In this method,the GAN uses a previously generated line of a lyric as the input sentence in order to generate the next line of the lyric.Using a strategy based on the penalty mechanism in training,the GAN model generates non-repetitive and diverse lyrics.The secret information is then processed according to the data characteristics of the generated lyrics in order to hide information.Unlike other text generation-based linguistic steganographic methods,our method changes the way that multiple generated candidate items are selected as the candidate groups in order to encode the conditional probability distribution.The experimental results demonstrate that our method can generate highquality lyrics as stego-texts.Moreover,compared with other similar methods,the proposed method achieves good performance in terms of imperceptibility,embedding rate,effectiveness,extraction success rate and security.
基金Supported by the National Natural Science Foundation of China(No.82174276 and 82074580)the Key Research and Development Program of Jiangsu Province(No.BE2022712)+2 种基金China Postdoctoral Foundation(No.2021M701674)Postdoctoral Research Program of Jiangsu Province(No.2021K457C)Qinglan Project of Jiangsu Universities 2021。
文摘Chinese medicine(CM)diagnosis intellectualization is one of the hotspots in the research of CM modernization.The traditional CM intelligent diagnosis models transform the CM diagnosis issues into classification issues,however,it is difficult to solve the problems such as excessive or similar categories.With the development of natural language processing techniques,text generation technique has become increasingly mature.In this study,we aimed to establish the CM diagnosis generation model by transforming the CM diagnosis issues into text generation issues.The semantic context characteristic learning capacity was enhanced referring to Bidirectional Long Short-Term Memory(BILSTM)with Transformer as the backbone network.Meanwhile,the CM diagnosis generation model Knowledge Graph Enhanced Transformer(KGET)was established by introducing the knowledge in medical field to enhance the inferential capability.The KGET model was established based on 566 CM case texts,and was compared with the classic text generation models including Long Short-Term Memory sequence-to-sequence(LSTM-seq2seq),Bidirectional and Auto-Regression Transformer(BART),and Chinese Pre-trained Unbalanced Transformer(CPT),so as to analyze the model manifestations.Finally,the ablation experiments were performed to explore the influence of the optimized part on the KGET model.The results of Bilingual Evaluation Understudy(BLEU),Recall-Oriented Understudy for Gisting Evaluation 1(ROUGE1),ROUGE2 and Edit distance of KGET model were 45.85,73.93,54.59 and 7.12,respectively in this study.Compared with LSTM-seq2seq,BART and CPT models,the KGET model was higher in BLEU,ROUGE1 and ROUGE2 by 6.00–17.09,1.65–9.39 and 0.51–17.62,respectively,and lower in Edit distance by 0.47–3.21.The ablation experiment results revealed that introduction of BILSTM model and prior knowledge could significantly increase the model performance.Additionally,the manual assessment indicated that the CM diagnosis results of the KGET model used in this study were highly consistent with the practical diagnosis results.In conclusion,text generation technology can be effectively applied to CM diagnostic modeling.It can effectively avoid the problem of poor diagnostic performance caused by excessive and similar categories in traditional CM diagnostic classification models.CM diagnostic text generation technology has broad application prospects in the future.
基金This work was supported by the National Natural Science Foundation of China(No.62372173).
文摘Software is a crucial component in the communication systems,and its security is of paramount importance.However,it is susceptible to different types of attacks due to potential vulnerabilities.Meanwhile,significant time and effort is required to fix such vulnerabilities.We propose an automated program repair method based on controlled text generation techniques.Specifically,we utilize a fine-tuned language model for patch generation and introduce a discriminator to evaluate the generation process,selecting results that contribute most to vulnerability fixes.Additionally,we perform static syntax analysis to expedite the patch verification process.The effectiveness of the proposed approach is validated using QuixBugs and Defects4J datasets,demonstrating significant improvements in generating correct patches compared to other existing methods.
文摘Aiming at complex and changeable factors such as speech theme and environment,which make it difficult for a speaker to prepare the speech text in a short time,this paper proposes a speech generation and demonstration system based on deep learning.This system is based on the Deep Learning Development Framework(PyTorch),trained through the theory of GPT-2 and the open source pretrained model,to generate multiple speeches according to the topics given by users,and the system generates thefinal speech and corresponding voice demon-stration audio through text modification,speech synthesis and other technologies to help users quickly obtain the target document and audio.Experiments show that the text generated by this model is smooth and easy to use,which helps shorten the preparation time of speakers and improves the confidence of the impromptu speaker.In addition,the paper explores the application prospects of text generation and has certain reference value.
文摘1 Introduction With rapid development in computing power and breakthroughs in deep learning,the concept of“foundation models”has been introduced into the AI community.Generally,foundation models are large models trained on massive data and can be easily adapted to different domains for various tasks.With specific prompts,foundation models can generate texts and images,or even animate scenarios based on the given descriptions.Due to powerful capabilities,there is a growing trend to build agents based on foundation models.In this paper,we conduct an investigation into agents empowered by the foundation models.
文摘In this study,we explore the potential of Multiway Transformers for text-to-image generation to achieve performance improvements through a concise and efficient decoupled model design and the inference efficiency provided by bidirectional encoding.We propose a method for improving the image tokenizer using pretrained Vision Transformers.Next,we employ bidirectional Multiway Transformers to restore the masked visual tokens combined with the unmasked text tokens.On the MS-COCO benchmark,our Multiway Transformers outperform vanilla Transformers,achieving superior FID scores and confirming the efficacy of the modality-specific parameter computation design.Ablation studies reveal that the fusion of visual and text tokens in bidirectional encoding contributes to improved model performance.Additionally,our proposed tokenizer outperforms VQGAN in image reconstruction quality and enhances the text-to-image generation results.By incorporating the additional CC-3M dataset for intermediate finetuning on our model with 688M parameters,we achieve competitive results with a finetuned FID score of 4.98 on MS-COCO.
基金This work was supported by the Key Research Program of the Chinese Academy of Sciences(No.ZDBSSSW-JSC006)the National Natural Science Foundation of China(No.62206294).
文摘This paper focuses on term-status pair extraction from medical dialogues(MD-TSPE),which is essential in diagnosis dia-logue systems and the automatic scribe of electronic medical records(EMRs).In the past few years,works on MD-TSPE have attracted increasing research attention,especially after the remarkable progress made by generative methods.However,these generative methods output a whole sequence consisting of term-status pairs in one stage and ignore integrating prior knowledge,which demands a deeper un-derstanding to model the relationship between terms and infer the status of each term.This paper presents a knowledge-enhanced two-stage generative framework(KTGF)to address the above challenges.Using task-specific prompts,we employ a single model to com-plete the MD-TSPE through two phases in a unified generative form:We generate all terms the first and then generate the status of each generated term.In this way,the relationship between terms can be learned more effectively from the sequence containing only terms in the first phase,and our designed knowledge-enhanced prompt in the second phase can leverage the category and status candidates of the generated term for status generation.Furthermore,our proposed special status"not mentioned"makes more terms available and en-riches the training data in the second phase,which is critical in the low-resource setting.The experiments on the Chunyu and CMDD datasets show that the proposed method achieves superior results compared to the state-of-the-art models in the full training and low-re-sourcesettings.
文摘In the paper an intelligent speech production system is established by using language information processing technology. The concept of bi-directional grammar is proposed in Chinese language information processing and a corresponding Chinese characteristic network is completed. Correct text can be generated through grammar parsing and some additional rules. According to the generated text the system generates speech which has good quality in naturalness and intelligibility using Chinese Text-to-Speech Conversion System.
