PREDICTIVE TREND INSIGHT
How to write system prompts that completely block automated red-teaming Illustration

How to write system prompts that completely block automated red-teaming

Direct Summary:

No system prompt "completely blocks" red-teaming, and treating that as the goal is the wrong frame — the honest goal is a prompt robust enough that your own red-team (Lesson 3) can't find a working jailbreak in a given attack category, using defense-in-depth rather than a single clever instruction. This lesson covers the advanced layers beyond Lesson 1: output-side validation, least-privilege system prompts, and honestly tracking what your defenses don't cover yet.

"Trust, but verify."

— Russian proverb

Key Insights

  • "Completely block" is the wrong target: no published defense claims to stop all prompt injection — OWASP frames it as a risk to manage and monitor, not one you fully close off. Prompts claiming otherwise should be treated skeptically, including your own.
  • Least privilege beats cleverer wording: a support bot that literally cannot call a refund API for more than $50 is safer than one told "never approve large refunds" — put hard limits in code and permissions, not just in the system prompt text.
  • Track what you haven't tested: a red-team pass that only covers known jailbreak phrasings says nothing about a novel attack technique published next month. Log what attack categories you've actually run, not just your pass rate on them.

This lesson's title promises something no one can honestly deliver. What's real and worth building is the next tier of hardening after Lesson 1's basics: moving critical decisions out of the prompt and into code, validating what the model actually outputs (not just what came in), and keeping an honest record of which attack categories you've verified against — rather than a system prompt that quietly claims to have solved prompt injection.

What advanced hardening actually looks like

1. Move hard limits into code, not prompt wording. "Never issue a refund over $50" in a system prompt is a suggestion the model might not follow under adversarial pressure. A tool call that's programmatically capped at $50 is a limit the model literally cannot exceed, regardless of what it's convinced to say.

2. Validate the output against your policy, not just the input against a blacklist. Check whether the response actually contains the system prompt, a policy-violating claim, or PII — that catches attacks whose input didn't match any known "jailbreak phrase" pattern.

3. Keep a defense coverage log. After each red-team pass (Lesson 3), record which specific attack categories were tested and which weren't — "we've verified against known jailbreak phrasings but haven't tested multi-turn escalation" is honest and useful; a blanket "our system is jailbreak-proof" is neither.

least_privilege_tool.py
# The limit lives in code, not in prompt wording the model could be talked past
def issue_refund(order_id: str, amount: float) -> dict:
    MAX_AUTO_REFUND = 50.00

    if amount > MAX_AUTO_REFUND:
        # No prompt-level instruction can bypass this - it's a hard branch
        return {"status": "requires_human_approval", "amount": amount}

    process_refund(order_id, amount)
    return {"status": "refunded", "amount": amount}
Defense layer Can the model talk its way past it?
"Never approve refunds over $50" (prompt instruction) Potentially, under the right adversarial pressure
Hard-coded amount check before the API call executes No — it's a code branch, not a suggestion

The honest framing matters for a business reason too: a written claim that your system "completely blocks" jailbreaks is a liability if it's ever tested and fails publicly. "We red-team against these specific categories on this schedule, and enforce hard limits in code for anything high-stakes" is both more accurate and more defensible.

Practical Challenge

Take one action your AI system can currently take that would be costly if misused (a refund, a data deletion, an email send) and rewrite it so the limit is enforced in code rather than only described in the system prompt.

Concept Check

Why is a hard-coded limit in a tool function more reliable than a prompt instruction like "never approve large refunds"?
Correct! Moving a hard limit into code means the model's output can request an action, but the system itself refuses to execute anything outside the limit — no phrasing changes that.
Incorrect. Try again! Hint: the model always reads its system prompt — the issue is that a natural-language rule can potentially be argued around, while a code-level check cannot.

Sources & Further Reading

Previous Guide Dashboard Next Guide