Towards Explainability for Language Models in Security Testing

Abstract

Modern generative Language Models (LMs) present as black boxes, requiring significant trust in their capabilities and making it difficult to understand the reasoning behind their decisions. As these LMs are increasingly used for code and test-case generation, testers must trust them without knowing what drives the model's outputs. To improve accuracy, modern LMs rely on supplementary documentation, such as Retrieval-Augmented Generation (RAG), or other content directly provided in their prompts to enhance background knowledge. When testers use LM-generated test cases for other purposes, such as fuzz testing, they must place greater trust in their quality, as seed cases can significantly affect fuzzer coverage performance.

We adapt existing methods to build an analysis pipeline that explains document retrieval when the LM relies on documentation to generate test cases. We achieve this with only black-box access to the LMs under test. We use RFC-959 (the File Transfer Protocol (FTP) protocol) and two synthetic protocols to isolate the LM's reliance on data in its RAG system. Statistical analysis shows that the explanations from our pipeline capture real phenomena rather than random data.

To aid integration with automated security testing, we present a formal definition of protocol communication. This formalism helps map our pipeline's features to the protocol domain and lays a foundation for future work with fuzzers.

The explanations our pipeline generates yield plausible results, with some unexpected outputs, suggesting the need for tuning to improve explanations.