大型语言模型(LLM)已成为推进Text-to-SQL任务的强大工具。研究发现,基于LLM的模型在不同评估指标下,其性能表现与经过微调的模型存在显著差异。因此,文章分析了测试套件执行准确度(EXE)和精确集匹配准确度(ESM)在评估基于LLM的Text-to-...大型语言模型(LLM)已成为推进Text-to-SQL任务的强大工具。研究发现,基于LLM的模型在不同评估指标下,其性能表现与经过微调的模型存在显著差异。因此,文章分析了测试套件执行准确度(EXE)和精确集匹配准确度(ESM)在评估基于LLM的Text-to-SQL模型时的不足,并提出了改进指标EESM(Enhanced Exact Set Matching)。实验结果表明,EXE和ESM分别存在高达13.2%和10.8%的假阳性和假阴性率,而EESM的假阳性率和假阴性率分别仅为0.2%和1.8%,表明EESM能够提供更准确的评估。展开更多
Text-to-SQL is the task of translating a natural language query into a structured query language. Existing text-to-SQL approaches focus on improving the model’s architecture while ignoring the relationship between qu...Text-to-SQL is the task of translating a natural language query into a structured query language. Existing text-to-SQL approaches focus on improving the model’s architecture while ignoring the relationship between queries and table schemas and the differences in difficulty between examples in the dataset. To tackle these challenges, a two-stage curriculum learning framework for text-to-SQL(TSCL-SQL) is proposed in this paper. To exploit the relationship between the queries and the table schemas, a schema identification pre-training task is proposed to make the model choose the correct table schema from a set of candidates for a specific query. To leverage the differences in difficulty between examples, curriculum learning is applied to the text-to-SQL task, accompanied by an automatic curriculum learning solution, including a difficulty scorer and a training scheduler. Experiments show that the framework proposed in this paper is effective.展开更多
Recent advancements in large language models(LLMs)have significantly contributed to the progress of the Text-to-SQL task.A common requirement in many of these works is the post-correction of SQL queries.However,the ma...Recent advancements in large language models(LLMs)have significantly contributed to the progress of the Text-to-SQL task.A common requirement in many of these works is the post-correction of SQL queries.However,the majority of this process entails analyzing error cases to develop prompts with rules that eliminate model bias.And there is a weakness of execution verification for SQL queries.In addition,the prevalent techniques primarily depend on GPT-4 and few-shot prompts,resulting in expensive costs.To investigate the effective methods for SQL refinement in a cost-efficient manner,we introduce Semantic-Enhanced Text-to-SQL with Adaptive Refinement(SEA-SQL),which includes Adaptive Bias Elimination and Dynamic Execution Adjustment,aims to improve performance while minimizing resource expenditure with zero-shot prompts.Specifically,SEA-SQL employs a semantic-enhanced schema to augment database information and optimize SQL queries.During the SQL query generation,a fine-tuned adaptive bias eliminator is applied to mitigate inherent biases caused by the LLM.The dynamic execution adjustment is utilized to guarantee the executability of the bias eliminated SQL query.We conduct experiments on the Spider and BIRD datasets to demonstrate the effectiveness of this framework.The results demonstrate that SEA-SQL achieves state-of-the-art performance in the GPT-3.5 scenario with 9%-58% of the generation cost.Furthermore,SEA-SQL is comparable to GPT-4 with only 0.9%-5.3% of the generation cost.Our code is available at the website of github.com/545999961/SEA-SQL.展开更多
The increasing complexity of modern power systems,driven by factors such as the large-scale integration of renewable energy and the proliferation of distributed generation,has placed unprecedented demands on power dis...The increasing complexity of modern power systems,driven by factors such as the large-scale integration of renewable energy and the proliferation of distributed generation,has placed unprecedented demands on power dispatching operations.Ensuring grid stability and safety in this new environment requires real-time monitoring and swift,data-driven decision-making.Consequently,efficient and accurate data querying capabilities have become paramount.This study introduces Intelli-Dispatch-SQL,a novel agent-based Text-to-SQL framework that leverages the Large Language Model(LLM)to enhance the accuracy and reliability of generated SQL queries in the context of power dispatching.By integrating intent recognition and SQL validation modules,Intelli-Dispatch-SQL ensures that generated queries are not only syntactically correct but also semantically aligned with user intent and executable within the operational context.Through comprehensive experiments,including ablation studies and cross-model evaluations,we demonstrate that Intelli-Dispatch-SQL significantly outperforms existing Text-to-SQL models,achieving substantial improvements in both Exact Match(EM)and Execution Accuracy(EX).Notably,the incorporation of intent recognition and SQL validation modules is shown to be critical for performance enhancement.The framework’s effectiveness was further validated across various LLMs,confirming its robustness and applicability across diverse scenarios.Intelli-Dispatch-SQL offers a performance high-and generalizable solution for Text-to-SQL in power dispatching,paving the way for more efficient and intelligent power system management.展开更多
大语言模型(large language model,LLM)为数据库性能带来了极大的提升,将LLM与数据库相结合成为当前的研究热点。然而,目前大部分高校数据库实验课程仍停留于数据库基础操作层面,学生普遍缺乏数据库前沿技术相关知识。文本转结构化查询...大语言模型(large language model,LLM)为数据库性能带来了极大的提升,将LLM与数据库相结合成为当前的研究热点。然而,目前大部分高校数据库实验课程仍停留于数据库基础操作层面,学生普遍缺乏数据库前沿技术相关知识。文本转结构化查询语言(text to structured query language,Text-to-SQL)技术为基于LLM的数据库设计的重要研究方向之一。首先,设计实验课程系统地介绍基于LLM的Text-to-SQL方法的教学意义,以盘古大模型为例对实验课程相关技术进行介绍;其次,分析并划定基于LLM的数据库设计实验课程的教学目标,阐述具体的教学设计和实施;最后,对教学方法设计的各方面进行系统总结。展开更多
Text-to-SQL aims at translating textual questions into the corresponding SQL queries.Aggregate tables are widely created for high-frequent queries.Although text-to-SQL has emerged as an important task,recent studies p...Text-to-SQL aims at translating textual questions into the corresponding SQL queries.Aggregate tables are widely created for high-frequent queries.Although text-to-SQL has emerged as an important task,recent studies paid little attention to the task over aggregate tables.The increased aggregate tables bring two challenges:(1)mapping of natural language questions and relational databases will suffer from more ambiguity,(2)modern models usually adopt self-attention mechanism to encode database schema and question.The mechanism is of quadratic time complexity,which will make inferring more time-consuming as input sequence length grows.In this paper,we introduce a novel approach named WAGG for text-to-SQL over aggregate tables.To effectively select among ambiguous items,we propose a relation selection mechanism for relation computing.To deal with high computation costs,we introduce a dynamical pruning strategy to discard unrelated items that are common for aggregate tables.We also construct a new large-scale dataset SpiderwAGG extended from Spider dataset for validation,where extensive experiments show the effectiveness and efficiency of our proposed method with 4%increase of accuracy and 15%decrease of inference time w.r.t a strong baseline RAT-SQL.展开更多
文摘大型语言模型(LLM)已成为推进Text-to-SQL任务的强大工具。研究发现,基于LLM的模型在不同评估指标下,其性能表现与经过微调的模型存在显著差异。因此,文章分析了测试套件执行准确度(EXE)和精确集匹配准确度(ESM)在评估基于LLM的Text-to-SQL模型时的不足,并提出了改进指标EESM(Enhanced Exact Set Matching)。实验结果表明,EXE和ESM分别存在高达13.2%和10.8%的假阳性和假阴性率,而EESM的假阳性率和假阴性率分别仅为0.2%和1.8%,表明EESM能够提供更准确的评估。
基金Fundamental Research Funds for the Central Universities,China (No. 2232023D-19)。
文摘Text-to-SQL is the task of translating a natural language query into a structured query language. Existing text-to-SQL approaches focus on improving the model’s architecture while ignoring the relationship between queries and table schemas and the differences in difficulty between examples in the dataset. To tackle these challenges, a two-stage curriculum learning framework for text-to-SQL(TSCL-SQL) is proposed in this paper. To exploit the relationship between the queries and the table schemas, a schema identification pre-training task is proposed to make the model choose the correct table schema from a set of candidates for a specific query. To leverage the differences in difficulty between examples, curriculum learning is applied to the text-to-SQL task, accompanied by an automatic curriculum learning solution, including a difficulty scorer and a training scheduler. Experiments show that the framework proposed in this paper is effective.
基金supported by the National Natural Science Foundation of China(Grant Nos.62272054,62192784,62172056)the Beijing Nova Program(No.20230484319)the Xiaomi Young Talents Program.
文摘Recent advancements in large language models(LLMs)have significantly contributed to the progress of the Text-to-SQL task.A common requirement in many of these works is the post-correction of SQL queries.However,the majority of this process entails analyzing error cases to develop prompts with rules that eliminate model bias.And there is a weakness of execution verification for SQL queries.In addition,the prevalent techniques primarily depend on GPT-4 and few-shot prompts,resulting in expensive costs.To investigate the effective methods for SQL refinement in a cost-efficient manner,we introduce Semantic-Enhanced Text-to-SQL with Adaptive Refinement(SEA-SQL),which includes Adaptive Bias Elimination and Dynamic Execution Adjustment,aims to improve performance while minimizing resource expenditure with zero-shot prompts.Specifically,SEA-SQL employs a semantic-enhanced schema to augment database information and optimize SQL queries.During the SQL query generation,a fine-tuned adaptive bias eliminator is applied to mitigate inherent biases caused by the LLM.The dynamic execution adjustment is utilized to guarantee the executability of the bias eliminated SQL query.We conduct experiments on the Spider and BIRD datasets to demonstrate the effectiveness of this framework.The results demonstrate that SEA-SQL achieves state-of-the-art performance in the GPT-3.5 scenario with 9%-58% of the generation cost.Furthermore,SEA-SQL is comparable to GPT-4 with only 0.9%-5.3% of the generation cost.Our code is available at the website of github.com/545999961/SEA-SQL.
基金supported by the Guangdong Power Grid Com-pany(Grant Number:GDKJXM20231024)the National Natural Sci-ence Foundation of China(Grant Number:72331009,72171206 and 92270105)the Shenzhen Key Laboratory of Crowd Intelligence Em-powered Low-Carbon Energy Network(Grant number:ZDSYS20220606100601002).
文摘The increasing complexity of modern power systems,driven by factors such as the large-scale integration of renewable energy and the proliferation of distributed generation,has placed unprecedented demands on power dispatching operations.Ensuring grid stability and safety in this new environment requires real-time monitoring and swift,data-driven decision-making.Consequently,efficient and accurate data querying capabilities have become paramount.This study introduces Intelli-Dispatch-SQL,a novel agent-based Text-to-SQL framework that leverages the Large Language Model(LLM)to enhance the accuracy and reliability of generated SQL queries in the context of power dispatching.By integrating intent recognition and SQL validation modules,Intelli-Dispatch-SQL ensures that generated queries are not only syntactically correct but also semantically aligned with user intent and executable within the operational context.Through comprehensive experiments,including ablation studies and cross-model evaluations,we demonstrate that Intelli-Dispatch-SQL significantly outperforms existing Text-to-SQL models,achieving substantial improvements in both Exact Match(EM)and Execution Accuracy(EX).Notably,the incorporation of intent recognition and SQL validation modules is shown to be critical for performance enhancement.The framework’s effectiveness was further validated across various LLMs,confirming its robustness and applicability across diverse scenarios.Intelli-Dispatch-SQL offers a performance high-and generalizable solution for Text-to-SQL in power dispatching,paving the way for more efficient and intelligent power system management.
文摘大语言模型(large language model,LLM)为数据库性能带来了极大的提升,将LLM与数据库相结合成为当前的研究热点。然而,目前大部分高校数据库实验课程仍停留于数据库基础操作层面,学生普遍缺乏数据库前沿技术相关知识。文本转结构化查询语言(text to structured query language,Text-to-SQL)技术为基于LLM的数据库设计的重要研究方向之一。首先,设计实验课程系统地介绍基于LLM的Text-to-SQL方法的教学意义,以盘古大模型为例对实验课程相关技术进行介绍;其次,分析并划定基于LLM的数据库设计实验课程的教学目标,阐述具体的教学设计和实施;最后,对教学方法设计的各方面进行系统总结。
文摘Text-to-SQL aims at translating textual questions into the corresponding SQL queries.Aggregate tables are widely created for high-frequent queries.Although text-to-SQL has emerged as an important task,recent studies paid little attention to the task over aggregate tables.The increased aggregate tables bring two challenges:(1)mapping of natural language questions and relational databases will suffer from more ambiguity,(2)modern models usually adopt self-attention mechanism to encode database schema and question.The mechanism is of quadratic time complexity,which will make inferring more time-consuming as input sequence length grows.In this paper,we introduce a novel approach named WAGG for text-to-SQL over aggregate tables.To effectively select among ambiguous items,we propose a relation selection mechanism for relation computing.To deal with high computation costs,we introduce a dynamical pruning strategy to discard unrelated items that are common for aggregate tables.We also construct a new large-scale dataset SpiderwAGG extended from Spider dataset for validation,where extensive experiments show the effectiveness and efficiency of our proposed method with 4%increase of accuracy and 15%decrease of inference time w.r.t a strong baseline RAT-SQL.
