Fuzz testing is crucial for identifying software vulnerabilities,with coverage-guided grey-box fuzzers like AFL and Angora excelling in broad detection.However,as the need for targeted detection grows,directed grey-bo...Fuzz testing is crucial for identifying software vulnerabilities,with coverage-guided grey-box fuzzers like AFL and Angora excelling in broad detection.However,as the need for targeted detection grows,directed grey-box fuzzing(DGF)has become essential,focusing on specific vulnerabilities.The initial seed corpus,which consists of carefully selected input samples that the fuzzer uses as a starting point,is fundamental in determining the paths that the fuzzer explores.A well-designed seed corpus can guide the fuzzer more effectively towards critical areas of the code,improving the efficiency and success of the fuzzing process.Even with its importance,much work concentrates on refining guidance mechanisms while paying less attention to optimizing the initial seed corpus.In this paper,we introduce ISC4DGF,a novel approach to generating optimized initial seed corpus for DGF using large language models(LLMs).By leveraging LLMs’deep understanding of software and refined user inputs,ISC4DGF creates a precise seed corpus that efficiently triggers specific vulnerabilities through a multi-round validation process.Implemented on AFL and tested against state-of-the-art fuzzers such as Titan,BEACON,AFLGo,FairFuzz,and Entropic using the Magma benchmark,ISC4DGF achieves a 25.03x speedup with fewer target reaches.Moreover,ISC4DGF improves target vulnerabilities detection accuracy while narrowing the detection scope and reducing code coverage.展开更多
Large language models(LLMs)are increasingly applied across diverse software engineering tasks.Consequently,their ability to effectively rank code quality is crucial for applications like selecting optimal solutions an...Large language models(LLMs)are increasingly applied across diverse software engineering tasks.Consequently,their ability to effectively rank code quality is crucial for applications like selecting optimal solutions and aiding code review.However,evaluating this essential code ranking capability is hampered by a lack of benchmarks covering diverse paradigms and robustness testing.To address this,we introduce CodeRankEval,a benchmark suite for multiparadigm evaluation,and CodeRankEval-Perturbed for robustness testing against common code flaws.Our empirical study reveals key insights:pairwise ranking yields the highest accuracy but is costly;listwise is the cheapest and shows comparable performance with pairwise;pointwise generally exhibits lower performance with intermediate cost.Besides,ranking ability correlates positively with generation ability,models show reasonable robustness to perturbations but may exhibit positional bias.Overall,this work provides valuable resources and insights for understanding and improving LLM-based code ranking evaluation.展开更多
基金supported by the National Key Research and Development Program of China under Grant No.2021YFB3101802.
文摘Fuzz testing is crucial for identifying software vulnerabilities,with coverage-guided grey-box fuzzers like AFL and Angora excelling in broad detection.However,as the need for targeted detection grows,directed grey-box fuzzing(DGF)has become essential,focusing on specific vulnerabilities.The initial seed corpus,which consists of carefully selected input samples that the fuzzer uses as a starting point,is fundamental in determining the paths that the fuzzer explores.A well-designed seed corpus can guide the fuzzer more effectively towards critical areas of the code,improving the efficiency and success of the fuzzing process.Even with its importance,much work concentrates on refining guidance mechanisms while paying less attention to optimizing the initial seed corpus.In this paper,we introduce ISC4DGF,a novel approach to generating optimized initial seed corpus for DGF using large language models(LLMs).By leveraging LLMs’deep understanding of software and refined user inputs,ISC4DGF creates a precise seed corpus that efficiently triggers specific vulnerabilities through a multi-round validation process.Implemented on AFL and tested against state-of-the-art fuzzers such as Titan,BEACON,AFLGo,FairFuzz,and Entropic using the Magma benchmark,ISC4DGF achieves a 25.03x speedup with fewer target reaches.Moreover,ISC4DGF improves target vulnerabilities detection accuracy while narrowing the detection scope and reducing code coverage.
基金supported by the National Key Research and Development Program of China under Grant No.2021YFB3101802.
文摘Large language models(LLMs)are increasingly applied across diverse software engineering tasks.Consequently,their ability to effectively rank code quality is crucial for applications like selecting optimal solutions and aiding code review.However,evaluating this essential code ranking capability is hampered by a lack of benchmarks covering diverse paradigms and robustness testing.To address this,we introduce CodeRankEval,a benchmark suite for multiparadigm evaluation,and CodeRankEval-Perturbed for robustness testing against common code flaws.Our empirical study reveals key insights:pairwise ranking yields the highest accuracy but is costly;listwise is the cheapest and shows comparable performance with pairwise;pointwise generally exhibits lower performance with intermediate cost.Besides,ranking ability correlates positively with generation ability,models show reasonable robustness to perturbations but may exhibit positional bias.Overall,this work provides valuable resources and insights for understanding and improving LLM-based code ranking evaluation.
