Insecure Output Handling: Toy Demo

This demo illustrates how improper LLM output handling can lead to security vulnerabilities, such as Cross-Site Scripting (XSS). Enter some text below and see how it is rendered.

<label>Simulated LLM output (edit this text):</label>
<textarea id="llmOutput">This is a normal response.</textarea>
<br>
<button onclick="renderVulnerable()">Insecure Rendering (innerHTML)</button>
<button onclick="renderSafe()">Secure Rendering (textContent)</button>

<div class="chat-box vulnerable">
    <strong>Vulnerable Result:</strong>
    <div id="vulnerableResult"></div>
</div>

<div class="chat-box safe">
    <strong>Safe Result:</strong>
    <div id="safeResult"></div>
</div>

<script>
    function renderVulnerable() {
        const output = document.getElementById('llmOutput').value;
        // dangerous: directly inserting LLM output into innerHTML
        // can lead to XSS if the output contains malicious scripts
        document.getElementById('vulnerableResult').innerHTML = output;
    }

    function renderSafe() {
        const output = document.getElementById('llmOutput').value;
        // safe: textContent treats any HTML tags in the output as
        // plain text, preventing XSS
        document.getElementById('safeResult').textContent = output;
    }
</script>

Real exploitation combines this with indirect injection — it’s rarely a standalone bug:

1. Attacker probes the chat window directly with a payload to confirm unsafe rendering.
2. Attacker plants the same payload somewhere the LLM will later read it from (a product review, a document) — this is indirect injection.
3. A different user asks the chatbot about that content; the LLM repeats the payload in its response; the victim’s browser executes it.

This exact flow is demonstrated in PortSwigger’s Web Security Academy lab on insecure output handling in LLMs.

Real CVEs:

• CVE-2023-29374 (LangChain, severity 9.8) — LLM output passed directly into Python’s exec(), leading to RCE.
• CVE-2023-36258 (Auto-GPT) — unsanitized filename parameter, high severity.

SQL injection through the LLM translation layer

Natural-language-to-SQL systems introduce a new twist on classic SQL injection: the attacker’s input isn’t inserted into SQL directly — it’s first “translated” by the LLM, then the generated SQL string is executed.

If the application doesn’t use parameterized queries for the LLM’s output, this is just as dangerous as classic SQL injection — arguably more dangerous, since the malicious SQL syntax is generated by the LLM, not typed directly by the user, so keyword-based WAF filtering is less effective.

Defense: never execute LLM-generated SQL directly. The LLM should only extract parameters; the SQL structure itself should be a fixed, parameterized template.

Relevance to NVR research direction

This maps directly onto how a local NVR’s natural-language camera query feature might be architected:

• Function calling mode: the LLM extracts structured parameters (camera ID, time range) and the backend applies a fixed, parameterized query template. Lower SQL injection risk, but parameter validation and authorization checks become the new attack surface.
• Direct SQL generation mode: the LLM outputs a raw SQL string that gets executed as-is. High SQL injection risk.

Determining which architecture a real device uses requires passive analysis — inspecting network requests via browser dev tools, and watching for database-level error messages leaking through natural language responses.