CINXE.COM
Embedding Security in LLM Workflows: Elastic's Proactive Approach — Elastic Security Labs
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width"/><title>Embedding Security in LLM Workflows: Elastic's Proactive Approach — Elastic Security Labs</title><meta name="description" content="Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications."/><meta property="og:title" content="Embedding Security in LLM Workflows: Elastic's Proactive Approach — Elastic Security Labs"/><meta property="og:description" content="Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications."/><meta property="og:image" content="https://www.elastic.co/security-labs/assets/images/embedding-security-in-llm-workflows/Security Labs Images 5.jpg?0b102f2087c210722a6f4e257de92c57"/><meta property="og:image:alt" content="Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications."/><meta property="og:site_name"/><meta property="og:url" content="https://www.elastic.co/security-labs/embedding-security-in-llm-workflows"/><meta property="og:type" content="website"/><meta name="twitter:card" content="summary_large_image"/><meta name="twitter:title" content="Embedding Security in LLM Workflows: Elastic's Proactive Approach — Elastic Security Labs"/><meta name="twitter:description" content="Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications."/><meta name="twitter:image" content="https://www.elastic.co/security-labs/assets/images/embedding-security-in-llm-workflows/Security Labs Images 5.jpg?0b102f2087c210722a6f4e257de92c57"/><meta name="twitter:image:alt" content="Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications."/><link rel="canonical" href="https://www.elastic.co/security-labs/embedding-security-in-llm-workflows"/><link rel="preload" href="/security-labs/logo.svg" as="image" fetchpriority="high"/><link rel="preload" as="image" imageSrcSet="/security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=640&q=75 640w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=750&q=75 750w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=828&q=75 828w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1080&q=75 1080w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1200&q=75 1200w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1920&q=75 1920w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=2048&q=75 2048w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=3840&q=75 3840w" imageSizes="100vw" fetchpriority="high"/><meta name="next-head-count" content="19"/><script src="https://play.vidyard.com/embed/v4.js" type="text/javascript" async=""></script><link rel="icon" href="/security-labs/favicon.svg"/><link rel="mask-icon" href="/security-labs/favicon.svg" color="#1C1E23"/><link rel="apple-touch-icon" href="/security-labs/favicon.svg"/><meta name="theme-color" content="#1C1E23"/><link rel="preload" href="/security-labs/_next/static/media/6d93bde91c0c2823-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/a34f9d1faa5f3315-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/369c6e283c5acc6e-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/92f44bb82993d879-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/ee71530a747ff30b-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/9fac010bc1f02be0-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><link rel="preload" href="/security-labs/_next/static/media/cbf5fbad4d73afac-s.p.woff2" as="font" type="font/woff2" crossorigin="anonymous" data-next-font="size-adjust"/><script id="google-tag-manager" data-nscript="beforeInteractive"> (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-KNJMG2M'); </script><link rel="preload" href="/security-labs/_next/static/css/265ed7605fd03477.css" as="style"/><link rel="stylesheet" href="/security-labs/_next/static/css/265ed7605fd03477.css" data-n-g=""/><link rel="preload" href="/security-labs/_next/static/css/1007ff9e696f6f88.css" as="style"/><link rel="stylesheet" href="/security-labs/_next/static/css/1007ff9e696f6f88.css" data-n-p=""/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/security-labs/_next/static/chunks/polyfills-78c92fac7aa8fdd8.js"></script><script src="/security-labs/_next/static/chunks/webpack-7987c6fda769d510.js" defer=""></script><script src="/security-labs/_next/static/chunks/framework-7a7e500878b44665.js" defer=""></script><script src="/security-labs/_next/static/chunks/main-ebd33a9f1cae5951.js" defer=""></script><script src="/security-labs/_next/static/chunks/pages/_app-cb8664d1d3df2511.js" defer=""></script><script src="/security-labs/_next/static/chunks/fec483df-43ee602fabdfe3a4.js" defer=""></script><script src="/security-labs/_next/static/chunks/877-34f408271ef44c22.js" defer=""></script><script src="/security-labs/_next/static/chunks/511-d08fe0fdd6f8a984.js" defer=""></script><script src="/security-labs/_next/static/chunks/683-a5053c37fe5bd0c9.js" defer=""></script><script src="/security-labs/_next/static/chunks/402-400478b95f2ce86a.js" defer=""></script><script src="/security-labs/_next/static/chunks/616-0b017b9cfa597392.js" defer=""></script><script src="/security-labs/_next/static/chunks/pages/%5Bslug%5D-b0c191de1a3710e4.js" defer=""></script><script src="/security-labs/_next/static/Zi9yjsbrUOekNuqEl_gO0/_buildManifest.js" defer=""></script><script src="/security-labs/_next/static/Zi9yjsbrUOekNuqEl_gO0/_ssgManifest.js" defer=""></script></head><body><noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-KNJMG2M" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript><div id="__next"><main class="__variable_0351a5 __variable_1f211e __variable_a5b5f5 flex flex-col min-h-screen"><div class="scroll-percentage-container"><div class="scroll-percentage-bar" style="width:0%"></div></div><nav class="fixed w-full z-40" data-headlessui-state=""><div class="bg-gradient-to-b from-zinc-900 from-20% h-[200%] to-transparent absolute inset-0 z-0 pointer-events-none"></div><div class="container relative z-10"><div class="flex h-16 items-center justify-between"><div class="flex items-center justify-start w-full"><div><a class="hover:opacity-50 transition" href="/security-labs"><img alt="elastic security labs logo" fetchpriority="high" width="200" height="30" decoding="async" data-nimg="1" style="color:transparent" src="/security-labs/logo.svg"/></a></div><div class="hidden lg:ml-6 lg:block"><div class="flex space-x-4"><a class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" href="/security-labs/about"><span>About</span></a><div class="relative" data-headlessui-state=""><div><button class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" id="headlessui-menu-button-:R2kpm:" type="button" aria-haspopup="menu" aria-expanded="false" data-headlessui-state="">Topics<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" class="ml-1 -mr-1 h-4 w-4 text-zinc-400 relative top-[1px]"><path fill-rule="evenodd" d="M5.23 7.21a.75.75 0 011.06.02L10 11.168l3.71-3.938a.75.75 0 111.08 1.04l-4.25 4.5a.75.75 0 01-1.08 0l-4.25-4.5a.75.75 0 01.02-1.06z" clip-rule="evenodd"></path></svg></button></div></div><a class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" href="/security-labs/category/vulnerability-updates"><span>Vulnerability updates</span></a><a class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" href="/security-labs/category/reports"><span>Reports</span></a><a class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" href="/security-labs/category/tools"><span>Tools</span></a></div></div><div class="hidden lg:ml-auto lg:block"><div class="flex items-center space-x-4"><a class="rounded flex items-center p-4 text-white focus:outline-none focus:ring-0 focus:ring-offset-1 focus:ring-offset-zinc-600 group" href="https://search.elastic.co/?location%5B0%5D=Security%20Labs&referrer=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows"><div class="flex items-center relative font-display"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="h-6 w-6"><path stroke-linecap="round" stroke-linejoin="round" d="M21 21l-5.197-5.197m0 0A7.5 7.5 0 105.196 5.196a7.5 7.5 0 0010.607 10.607z"></path></svg></div></a><a class="flex lg:inline-flex font-light my-1 py-1 px-2 font-display font-semibold lg:text-sm xl:text-base items-center transition hover:hover-link hover:text-white focus:accessible-link-focus" href="https://www.elastic.co/security-labs/rss/feed.xml"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" class="h-4 w-4 mr-1"><path d="M3.75 3a.75.75 0 00-.75.75v.5c0 .414.336.75.75.75H4c6.075 0 11 4.925 11 11v.25c0 .414.336.75.75.75h.5a.75.75 0 00.75-.75V16C17 8.82 11.18 3 4 3h-.25z"></path><path d="M3 8.75A.75.75 0 013.75 8H4a8 8 0 018 8v.25a.75.75 0 01-.75.75h-.5a.75.75 0 01-.75-.75V16a6 6 0 00-6-6h-.25A.75.75 0 013 9.25v-.5zM7 15a2 2 0 11-4 0 2 2 0 014 0z"></path></svg><span class="hidden xl:block">Subscribe</span></a><a class="font-display inline-flex items-center justify-center rounded font-semibold disabled:!select-none disabled:!bg-gray-400 bg-blue-600 text-white hover:bg-blue-500 enabled:hover:text-white/80 transition-colors px-4 py-2 text-sm flex-1 lg:flex-auto" href="https://cloud.elastic.co/registration?cta=cloud-registration&tech=trial&plcmt=navigation&pg=security-labs">Start free trial</a><a class="font-display inline-flex items-center justify-center rounded font-semibold text-white disabled:!select-none disabled:!bg-gray-400 button px-4 py-2 text-sm flex-1 lg:flex-auto" href="https://www.elastic.co/contact">Contact sales</a></div></div></div><div class="-mr-2 flex lg:hidden"><a class="rounded flex items-center p-4 text-white focus:outline-none focus:ring-0 focus:ring-offset-1 focus:ring-offset-zinc-600 group" href="https://search.elastic.co/?location%5B0%5D=Security%20Labs&referrer=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows"><div class="flex items-center relative font-display"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="h-6 w-6"><path stroke-linecap="round" stroke-linejoin="round" d="M21 21l-5.197-5.197m0 0A7.5 7.5 0 105.196 5.196a7.5 7.5 0 0010.607 10.607z"></path></svg></div></a><button class="inline-flex items-center justify-center rounded-md p-2 text-gray-400 hover:bg-gray-700 hover:text-white focus:outline-none focus:ring-2 focus:ring-inset focus:ring-white" id="headlessui-disclosure-button-:R59m:" type="button" aria-expanded="false" data-headlessui-state=""><span class="sr-only">Open navigation menu</span><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="block h-6 w-6"><path stroke-linecap="round" stroke-linejoin="round" d="M3.75 6.75h16.5M3.75 12h16.5m-16.5 5.25h16.5"></path></svg></button></div></div></div></nav><main class="mb-20 flex-1 flex flex-col"><div class="h-48 md:h-64"><div class="after:absolute after:block after:bg-blue-400 after:blur-3xl after:content-[' '] after:h-96 after:opacity-5 after:right-0 after:rounded-full after:top-20 after:w-1/2 after:z-0 before:absolute before:block before:blur-3xl before:bg-orange-400 before:content-[' '] before:h-96 before:left-0 before:opacity-5 before:rounded-full before:w-1/2 before:z-0 w-full h-full relative"><div class="relative z-10 w-full h-[125%] -top-[25%] bg-no-repeat bg-cover bg-bottom flex items-center justify-center" style="background-image:url(/security-labs/grid.svg)"></div></div></div><article class="px-4"><div class="max-w-7xl mx-auto relative z-10 flex flex-col space-y-4"><div class="eyebrow break-words"><time class="block mb-2 md:mb-0 md:inline-block article-published-date" dateTime="2024-04-25T00:00:00.000Z">25 April 2024</time><span class="hidden md:inline-block md:mx-2">•</span><a class="hover:text-blue-400 text-xs md:text-sm whitespace-nowrap author-name" href="/security-labs/author/mika-ayenson">Mika Ayenson, PhD</a></div><h1 class="font-bold leading-tighter text-3xl md:text-5xl"><span>Embedding Security in LLM Workflows: Elastic's Proactive Approach</span></h1><p class="text-zinc-200 text-base md:text-xl">Exploring Elastic's innovative approach for integrating security into the lifecycle of LLMs to safeguard against vulnerabilities featuring Elastic’s AI Assistant.</p><div class="flex items-center mt-4 text-zinc-200 text-sm space-x-4 border-t border-white/25 pt-4"><span class="flex items-center space-x-1"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="h-4 w-4 text-zinc-400"><path stroke-linecap="round" stroke-linejoin="round" d="M12 6v6h4.5m4.5 0a9 9 0 11-18 0 9 9 0 0118 0z"></path></svg><span>28 min read</span></span><span class="flex items-center space-x-1"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="h-4 w-4 text-zinc-400"><path stroke-linecap="round" stroke-linejoin="round" d="M9.568 3H5.25A2.25 2.25 0 003 5.25v4.318c0 .597.237 1.17.659 1.591l9.581 9.581c.699.699 1.78.872 2.607.33a18.095 18.095 0 005.223-5.223c.542-.827.369-1.908-.33-2.607L11.16 3.66A2.25 2.25 0 009.568 3z"></path><path stroke-linecap="round" stroke-linejoin="round" d="M6 6h.008v.008H6V6z"></path></svg><span><a class="hover:text-blue-400 whitespace-nowrap" href="/security-labs/category/detection-science">Detection science</a>, </span><span><a class="hover:text-blue-400 whitespace-nowrap" href="/security-labs/category/machine-learning">Machine learning</a>, </span><span><a class="hover:text-blue-400 whitespace-nowrap" href="/security-labs/category/generative-ai">Generative AI</a></span></span></div></div><div class="max-w-7xl mx-auto"><div class="bg-zinc-900 border border-zinc-800 drop-shadow-lg p-5 sm:p-8 md:p-10 rounded-3xl mt-5 md:mt-10"><div class="relative w-full rounded-lg overflow-hidden aspect-video"><img alt="Embedding Security in LLM Workflows: Elastic's Proactive Approach" fetchpriority="high" decoding="async" data-nimg="fill" class="object-cover absolute h-full w-full" style="position:absolute;height:100%;width:100%;left:0;top:0;right:0;bottom:0;color:transparent" sizes="100vw" srcSet="/security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=640&q=75 640w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=750&q=75 750w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=828&q=75 828w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1080&q=75 1080w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1200&q=75 1200w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=1920&q=75 1920w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=2048&q=75 2048w, /security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=3840&q=75 3840w" src="/security-labs/_next/image?url=%2Fsecurity-labs%2Fassets%2Fimages%2Fembedding-security-in-llm-workflows%2FSecurity%20Labs%20Images%205.jpg&w=3840&q=75"/><div class="absolute border border-white/50 inset-0 mix-blend-overlay rounded-lg z-10"></div></div></div></div><div class="lg:max-w-7xl mx-auto relative mt-12 lg:grid lg:grid-cols-4 lg:gap-8 items-start"><div class="flex justify-center lg:col-span-3"><div class="prose lg:prose-lg prose-invert w-full article-content"><div><p>We recently concluded one of our quarterly Elastic OnWeek events, which provides a unique week to explore opportunities outside of our regular day-to-day. In line with recent publications from <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">OWASP</a> and the <a href="https://media.defense.gov/2024/Apr/15/2003439257/-1/-1/0/CSI-DEPLOYING-AI-SYSTEMS-SECURELY.PDF">NSA AISC</a>, we decided to spend some time with the OWASP Top Ten vulnerabilities for LLMs natively in Elastic. In this article, we touch on a few opportunities to detect malicious LLM activity with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html">ES|QL</a>, namely:</p> <ul> <li>LLM01: Prompt Injection</li> <li>LLM02: Insecure Output Handling</li> <li>LLM04: Model Denial of Service</li> <li>LLM06: Sensitive Information Disclosure</li> </ul> <p>Elastic provides the ability to audit LLM applications for malicious behaviors; we’ll show you one approach with just four steps:</p> <ol> <li>Intercepting and analyzing the LLM requests and responses</li> <li>Enriching data with LLM-specific analysis results</li> <li>Sending data to Elastic Security</li> <li>Writing ES|QL detection rules that can later be used to respond</li> </ol> <p>This approach reflects our ongoing efforts to explore and implement advanced detection strategies, including developing detection rules tailored specifically for LLMs, while keeping pace with emerging generative AI technologies and security challenges. Building on this foundation, last year marked a significant enhancement to our toolkit and overall capability to continue this proactive path forward.</p> <p>Elastic <a href="https://www.elastic.co/blog/introducing-elastic-ai-assistant">released</a> the AI Assistant for Security, introducing how the open generative AI sidekick is powered by the <a href="https://www.elastic.co/platform">Search AI Platform</a> — a collection of relevant tools for developing advanced search applications. Backed by machine learning (ML) and artificial intelligence (AI), this AI Assistant provides powerful pre-built workflows like alert summarization, workflow suggestions, query conversions, and agent integration advice. I highly recommend you read more on Elastic’s <a href="https://www.elastic.co/elasticsearch/ai-assistant">AI Assistant</a> about how the capabilities seamlessly span across Observability and Security.</p> <p>We can use the AI Assistant’s capabilities as a third-party LLM application to capture, audit, and analyze requests and responses for convenience and to run experiments. Once data is in an index, writing behavioral detections on it becomes business as usual — we can also leverage the entire security detection engine. Even though we’re proxying the Elastic AI Assistant LLM activity in this experiment, it’s merely used as a vehicle to demonstrate auditing LLM-based applications. Furthermore, this proxy approach is intended for third-party applications to ship data to <a href="https://www.elastic.co/guide/en/security/current/es-overview.html">Elastic Security</a>.</p> <p>We can introduce security mechanisms into the application's lifecycle by intercepting LLM activity or leveraging observable LLM metrics. It’s common practice to address prompt-based threats by <a href="https://platform.openai.com/docs/guides/safety-best-practices">implementing various safety tactics</a>:</p> <ol> <li><strong>Clean Inputs</strong>: Sanitize and validate user inputs before feeding them to the model</li> <li><strong>Content Moderation</strong>: Use OpenAI tools to filter harmful prompts and outputs</li> <li><strong>Rate Limits and Monitoring</strong>: Track usage patterns to detect suspicious activity</li> <li><strong>Allow/Blocklists</strong>: Define acceptable or forbidden inputs for specific applications</li> <li><strong>Safe Prompt Engineering</strong>: Design prebuilt prompts that guide the model towards intended outcomes</li> <li><strong>User Role Management</strong>: Control user access to prevent unauthorized actions</li> <li><strong>Educate End-Users</strong>: Promote responsible use of the model to mitigate risks</li> <li><strong>Red Teaming & Monitoring</strong>: Test for vulnerabilities and continuously monitor for unexpected outputs</li> <li><strong>HITL Feedback for Model Training</strong>: Learn from human-in-the-loop, flagged issues to refine the model over time</li> <li><strong>Restrict API Access</strong>: Limit model access based on specific needs and user verification</li> </ol> <p>Two powerful features provided by OpenAI, and many other LLM implementers, is the ability to <a href="https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids">submit end-user IDs</a> and check content against a <a href="https://platform.openai.com/docs/guides/moderation">moderation API</a>, features that set the bar for LLM safety. Sending hashed IDs along with the original request aids in abuse detection and provides targeted feedback, allowing unique user identification without sending personal information. Alternatively, OpenAI's moderation endpoint helps developers identify potentially harmful content like hate speech, self-harm encouragement, or violence, allowing them to filter such content. It even goes a step further by detecting threats and intent to self-harm.</p> <p>Despite all of the recommendations and best practices to protect against malicious prompts, we recognize that there is no single perfect solution. When using capabilities like OpenAI’s API, some of these threats may be detected by the content filter, which will respond with a usage policy violation notification:</p> <p></p> <p>This content filtering is beneficial to address many issues; however, it cannot identify further threats in the broader context of the environment, application ecosystem, or other alerts that may appear. The more we can integrate generative AI use cases into our existing protection capabilities, the more control and possibilities we have to address potential threats. Furthermore, even if LLM safeguards are in place to stop rudimentary attacks, we can still use the detection engine to alert and take future remediation actions instead of silently blocking or permitting abuse.</p> <h2 class="font-bold text-2xl md:text-4xl relative"><span id="proxying-llm-requests-and-setup" class="absolute -top-32"></span>Proxying LLM Requests and Setup</h2> <p>The optimal security solution integrates additional safeguards directly within the LLM application's ecosystem. This allows enriching alerts with the complete context surrounding requests and responses. As requests are sent to the LLM, we can intercept and analyze them for potential malicious activity. If necessary, a response action can be triggered to defer subsequent HTTP calls. Similarly, inspecting the LLM's response can uncover further signs of malicious behavior.</p> <p>Using a proxy to handle these interactions offers several advantages:</p> <ul> <li><strong>Ease of Integration and Management</strong>: By managing the new security code within a dedicated proxy application, you avoid embedding complex security logic directly into the main application. This approach minimizes changes needed in the existing application structure, allowing for easier maintenance and clearer separation of security from business logic. The main application must only be reconfigured to route its LLM requests through the proxy.</li> <li><strong>Performance and Scalability</strong>: Placing the proxy on a separate server isolates the security mechanisms and helps distribute the computational load. This can be crucial when scaling up operations or managing performance-intensive tasks, ensuring that the main application's performance remains unaffected by the additional security processing.</li> </ul> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="quick-start-option-proxy-with-flask" class="absolute -top-32"></span>Quick Start Option: Proxy with Flask</h3> <p>You can proxy incoming and outgoing LLM connections for a faster initial setup. This approach can be generalized for other LLM applications by creating a simple Python-based <a href="https://flask.palletsprojects.com/en/3.0.x/">Flask</a> application. This application would intercept the communication, analyze it for security risks, and log relevant information before forwarding the response.</p> <p></p> <p>Multiple SDKs exist to connect to Elasticsearch and handle OpenAI LLM requests. The provided <a href="https://github.com/elastic/llm-detection-proxy">llm-detection-proxy</a> repo demonstrates the available Elastic and OpenAI clients. This snippet highlights the bulk of the experimental proxy in a single Flask route.</p> <pre><code>@app.route("/proxy/openai", methods=["POST"]) def azure_openai_proxy(): """Proxy endpoint for Azure OpenAI requests.""" data = request.get_json() messages = data.get("messages", []) response_content = "" error_response = None try: # Forward the request to Azure OpenAI response = client.chat.completions.create(model=deployment_name, messages=messages) response_content = response.choices[0].message.content # Assuming one choice for simplicity choices = response.choices[0].model_dump() except openai.BadRequestError as e: # If BadRequestError is raised, capture the error details error_response = e.response.json().get("error", {}).get("innererror", {}) response_content = e.response.json().get("error", {}).get("message") # Structure the response with the error details choices = {**error_response.get("content_filter_result", {}), "error": response_content, "message": {"content": response_content}} # Perform additional analysis and create the Elastic document additional_analysis = analyze_and_enrich_request(prompt=messages[-1], response_text=response_content, error_response=error_response) log_data = {"request": {"messages": messages[-1]}, "response": {"choices": response_content}, **additional_analysis} # Log the last message and response log_to_elasticsearch(log_data) # Calculate token usage prompt_tokens = sum(len(message["content"]) for message in messages) completion_tokens = len(response_content) total_tokens = prompt_tokens + completion_tokens # Structure and return the response return jsonify({ "choices": [choices], "usage": { "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens, "total_tokens": total_tokens, } })</code></pre> <p>With the Flask server, you can configure the <a href="https://www.elastic.co/guide/en/kibana/current/openai-action-type.html">OpenAI Kibana Connector</a> to use your proxy.</p> <p></p> <p>Since this proxy to your LLM is running locally, credentials and connection information are managed outside of Elastic, and an empty string can be provided in the API key section. Before moving forward, testing your connection is generally a good idea. It is important to consider other security implications if you are considering implementing a proxy solution in a real environment - not something this prototype considered for brevity.</p> <p></p> <p>We can now index our LLM requests and responses and begin to write detections on the available data in the <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">azure-openai-logs</code> index created in this experiment. Optionally, we could preprocess the data using an Elastic <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html">ingestion pipeline</a>, but in this contrived example, we can effectively write detections with the power of ES|QL.</p> <p> Sample AzureOpenAI LLM Request/Response Data</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="langsmith-proxy" class="absolute -top-32"></span>Langsmith Proxy</h3> <p><em>Note: The <a href="https://docs.smith.langchain.com/proxy/quickstart">Langsmith Proxy</a> project provides a dockerized proxy for your LLM APIs. While it offers a minimized solution, as of this writing, it lacks native capabilities for incorporating custom security analysis tools or integrating directly with Elastic Security.</em></p> <p>The LangSmith Proxy is designed to simplify LLM API interaction. It's a sidecar application requiring minimal configuration (e.g., LLM API URL). It enhances performance (caching, streaming) for high-traffic scenarios. It uses NGINX for efficiency and supports optional tracing for detailed LLM interaction tracking. Currently, it works with OpenAI and AzureOpenAI, with future support planned for other LLMs.</p> <h2 class="font-bold text-2xl md:text-4xl relative"><span id="llm-potential-attacks-and-detection-rule-opportunities" class="absolute -top-32"></span>LLM Potential Attacks and Detection Rule Opportunities</h2> <p><strong>It’s important to understand that even though documented lists of protections do not accompany some LLMs, simply trying some of these prompts may be immediately denied or result in banning on whatever platform used to submit the prompt. We recommend experimenting with caution and understand the SLA prior to sending any malicious prompts. Since this exploration leverages OpenAI’s resources, we recommend following the bugcrowd <a href="https://bugcrowd.com/openai">guidance</a> and sign up for an additional testing account using your @bugcrowdninja.com email address.</strong></p> <p>Here is a list of several plausible examples to illustrate detection opportunities. Each LLM topic includes the OWASP description, an example prompt, a sample document, the detection opportunity, and potential actions users could take if integrating additional security mechanisms in their workflow.</p> <p>While this list is currently not extensive, Elastic Security Labs is currently undertaking a number of initiatives to ensure future development, and formalization of rules will continue.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="llm01---prompt-injection" class="absolute -top-32"></span>LLM01 - prompt injection</h3> <p><strong>OWASP Description</strong>: Manipulating LLMs via crafted inputs can lead to unauthorized access, data breaches, and compromised decision-making. Reference <a href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM01_PromptInjection.md">here</a>.</p> <p><strong>Example</strong>: An adversary might try to craft prompts that trick the LLM into executing unintended actions or revealing sensitive information. <em>Note: Tools like <a href="https://github.com/utkusen/promptmap">promptmap</a> are available to generate creative prompt injection ideas and automate the testing process.</em></p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p><strong>Detection Rule Opportunity</strong>: In this example, the LLM responded by refusing to handle database connection strings due to security risks. It emphasizes keeping credentials private and suggests using secure methods like environment variables or vaults to protect them.</p> <p>A very brittle but basic indicator-matching query may look like this:</p> <pre><code>FROM azure-openai-logs | WHERE request.messages.content LIKE "*generate*connection*string*" OR request.messages.content LIKE "*credentials*password*username*" OR response.choices LIKE "*I'm sorry, but I can't assist*"</code></pre> <p>A slightly more advanced query detects more than two similar attempts within the last day.</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY | WHERE request.messages.content LIKE "*credentials*password*username*" OR response.choices LIKE "*I'm*sorry,*but*I*can't*assist*" OR response.choices LIKE "*I*can’t*process*actual*sensitive*" | stats total_attempts = count(*) by connectorId | WHERE total_attempts >= 2</code></pre> <p><em>Note that there are many approaches to detect malicious prompts and protect LLM responses. Relying on these indicators alone is not the best approach; however, we can gradually improve the detection with additional enrichment or numerous response attempts. Furthermore, if we introduce an ID into our documents, we can further enhance our query by aggregating attempts based on the field that correlates to a specific user.</em></p> <p><strong>Example 2</strong>: The <a href="https://arxiv.org/abs/2404.01833v1">Crescendo</a> effect is a realistic jailbreak attack where an adversary gradually manipulates a language model through a series of seemingly innocent inquiries that shift towards asking the model to describe hypothetical scenarios involving the unauthorized access and manipulation of secure systems. By doing so, they aim to extract methods that could potentially bypass the LLM’s security constraints.</p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p>With the additional analysis from OpenAI’s filtering, we can immediately detect the first occurrence of abuse.</p> <p><strong>Detection Rule Opportunity</strong>:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY AND ( request.messages.content LIKE "*credentials*password*username*" OR response.choices LIKE "*I'm sorry, but I can't assist*" OR analysis.openai.code == "ResponsibleAIPolicyViolation" OR malicious ) | STATS total_attempts = COUNT(*) BY connectorId | WHERE total_attempts > 1 | SORT total_attempts DESC</code></pre> <p>However, as you continue to use the Crescendo Effect, we notice that the conversation pivot goes unblocked after the initial content filter by OpenAI. It’s important to understand that even if tactics like this are difficult to prevent, we still have opportunities to detect.</p> <p></p> <p>Additional analysis tools, like LLM-Guard, detect if the conversation is sensitive, which, in this case, is inaccurate. However, it hints at potential opportunities to track malicious behavior over multiple prompts. Note: We could also take advantage of EQL sequences as an alternative to this ES|QL query to help track behaviors over multiple events.</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY AND ( request.messages.content LIKE "*Molotov*" OR analysis.openai.code == "ResponsibleAIPolicyViolation" OR malicious ) | STATS attempts = count(*), max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive) BY connectorId | WHERE attempts >= 1 AND max_sensitivity > 0.5 | SORT attempts DESC</code></pre> <p>This query detects suspicious behavior related to Molotov Cocktails across multiple events by analyzing sequences of log entries associated with a single user/session (identified by connectorId). The query core filters events based on:</p> <ul> <li><strong>Content Matching</strong>: It searches for mentions of "Molotov" in conversation content (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">request.messages.content LIKE "*Molotov*"</code>)</li> <li>**Policy Violations: It identifies attempts blocked by OpenAI's safety filters (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">analysis.openai.code == "ResponsibleAIPolicyViolation"</code>), indicating the start of potentially suspicious behavior</li> <li><strong>Malicious Flag Consideration</strong>: It includes logs where the system flagged the content as malicious (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">malicious == true</code>), capturing potentially subtle or varied mentions</li> <li><strong>Session-Level Analysis</strong>: By grouping events by connectorId, it analyzes the complete sequence of attempts within a session. It then calculates the total number of attempts (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">attempts = count(*)</code>) and the highest sensitivity score (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive)</code>) across all attempts in that session</li> <li><strong>Flagging High-Risk Sessions</strong>: It filters sessions with at least one attempt (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">attempts >= 1</code>) and a maximum sensitivity score exceeding 0.5 (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">max_sensitivity > 0.5</code>). This threshold helps focus on sessions where users persistently discussed or revealed potentially risky content.</li> </ul> <p>By analyzing these factors across multiple events within a session, we can start building an approach to detect a pattern of escalating discussions, even if individual events might not be flagged alone.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="llm02---insecure-output-handling" class="absolute -top-32"></span>LLM02 - insecure output handling</h3> <p><strong>OWASP Description</strong>: Neglecting to validate LLM outputs may lead to downstream security exploits, including code execution that compromises systems and exposes data. Reference <a href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM02_InsecureOutputHandling.md">here</a>.</p> <p><strong>Example</strong>: An adversary may attempt to exploit the LLM to generate outputs that can be used for cross-site scripting (XSS) or other injection attacks.</p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p><strong>Detection Rule Opportunity</strong>:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY | WHERE ( response.choices LIKE "*<script>*" OR response.choices LIKE "*document.cookie*" OR response.choices LIKE "*<img src=x onerror=*" OR response.choices LIKE "*<svg/onload=*" OR response.choices LIKE "*javascript:alert*" OR response.choices LIKE "*<iframe src=# onmouseover=*" OR response.choices LIKE "*<img ''><script>*" OR response.choices LIKE "*<IMG SRC=javascript:alert(String.fromCharCode(88,83,83))>*" OR response.choices LIKE "*<IMG SRC=# onmouseover=alert('xxs')>*" OR response.choices LIKE "*<IMG onmouseover=alert('xxs')>*" OR response.choices LIKE "*<IMG SRC=/ onerror=alert(String.fromCharCode(88,83,83))>*" OR response.choices LIKE "*&#0000106&#0000097&#0000118&#0000097&#0000115&#0000099&#0000114&#0000105&#0000112&#0000116&#0000058&#0000097&#0000108&#0000101&#0000114&#0000116&#0000040&#0000039&#0000088&#0000083&#0000083&#0000039&#0000041>*" OR response.choices LIKE "*<IMG SRC=&#106;&#97;&#118;&#97;&#115;&#99;&#114;&#105;&#112;&#116;&#58;&#97;&#108;&#101;&#114;&#116;&#40;&#39;&#88;&#83;&#83;&#39;&#41;>*" OR response.choices LIKE "*<IMG SRC=\"jav&#x0A;ascript:alert('XSS');\">*" ) | stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId) | WHERE total_attempts >= 2</code></pre> <p>This pseudo query detects potential insecure output handling by identifying LLM responses containing scripting elements or cookie access attempts, which are common in Cross-Site Scripting (XSS) attacks. It is a shell that could be extended by allow or block lists for well-known keywords.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="llm04---model-dos" class="absolute -top-32"></span>LLM04 - model DoS</h3> <p><strong>OWASP Description</strong>: Overloading LLMs with resource-heavy operations can cause service disruptions and increased costs. Reference <a href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM04_ModelDoS.md">here</a>.</p> <p><strong>Example</strong>: An adversary may send complex prompts that consume excessive computational resources.</p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p>Detection Rule Opportunity:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY | WHERE response.choices LIKE "*requires*significant*computational*resources*" | stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId) | WHERE total_attempts >= 2</code></pre> <p>This detection illustrates another simple example of how the LLM response is used to identify potentially abusive behavior. Although this example may not represent a traditional security threat, it could emulate how adversaries can impose costs on victims, either consuming resources or tokens.</p> <p><strong>Example 2</strong>: An adversary may send complex prompts that consume excessive computational resources.</p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p>At a glance, this prompt appears to be benign. However, excessive requests and verbose responses in a short time can significantly increase costs.</p> <p><strong>Detection Rule Opportunity</strong>:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 HOUR | STATS request_count = COUNT(*), distinct_prompts = COUNT_DISTINCT(request.messages.content) BY connectorId | WHERE request_count > 50 AND distinct_prompts > 10 | SORT request_count DESC</code></pre> <p>In the context of example 2, this working query efficiently tracks and analyzes usage patterns by counting all requests and distinct prompt contents for each <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">connectorId</code> from the <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">azure-openai-logs</code> over the past hour. If any <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">connectorId</code> submits over 50 requests with more than ten unique prompts within this timeframe, it indicates a potential misuse pattern like the one described, where an adversary might be modifying queries to probe encryption algorithm details, potentially causing undue load or evading detection systems. The results are then ordered to prioritize the <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">connectorIds</code> with the highest request counts, which helps quickly identify the most active or suspicious sources.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="llm06---sensitive-information-disclosure" class="absolute -top-32"></span>LLM06 - sensitive information disclosure</h3> <p><strong>OWASP Description</strong>: Failure to protect against disclosure of sensitive information in LLM outputs can result in legal consequences or a loss of competitive advantage. Reference <a href="https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM06_SensitiveInformationDisclosure.md">here</a>.</p> <p><strong>Example</strong>: An adversary may craft prompts to extract sensitive information embedded in the training data.</p> <p><strong>Prompt</strong>: </p> <p><strong>Sample Response</strong>: </p> <p>Detection Rule Opportunity:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY | WHERE request.messages.content LIKE "*common*passwords*used*in*the*dataset*" AND malicious | stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId) BY identified_threats.keyword, analysis.langkit_score | WHERE total_attempts >= 2</code></pre> <p>This query searches for logs from the last day that contain requests specifically asking about "common passwords used in the dataset" to train the model and where such requests are flagged as malicious. It aggregates these logs to count the number of such attempts and identifies distinct users involved, grouped by any identified threats and the language kit score. With some enrichment, the inclusion of <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">AND malicious = true</code> ensures focus on requests already flagged as potentially harmful, helping to prioritize investigation and response actions.</p> <h2 class="font-bold text-2xl md:text-4xl relative"><span id="enriching-detection-rules-with-security-insights" class="absolute -top-32"></span>Enriching Detection Rules with Security Insights</h2> <p>By routing LLM requests through a proxy, we can capitalize on specialized security tools to analyze each request for signs of malicious intent. Upon detection, the original request can be enriched with additional metadata indicating the likelihood of malicious content and the specific type of threat it represents. This enriched data is then indexed in Elasticsearch, creating a robust monitoring, alerting, and retrospective analysis dataset. With this enrichment, the LLM detection opportunities from the last section are possible.</p> <p>We don’t deep-dive on every tool available, but several open-source tools have emerged to offer varying approaches to analyzing and securing LLM interactions. Some of these tools are backed by machine learning models trained to detect malicious prompts:</p> <ul> <li><strong>Rebuff</strong> (<a href="https://github.com/protectai/rebuff">GitHub</a>): Utilizes machine learning to identify and mitigate attempts at social engineering, phishing, and other malicious activities through LLM interactions. Example usage involves passing request content through Rebuff's analysis engine and tagging requests with a "malicious" boolean field based on the findings.</li> <li><strong>LLM-Guard</strong> (<a href="https://github.com/protectai/llm-guard">GitHub</a>): Provides a rule-based engine for detecting harmful patterns in LLM requests. LLM-Guard can categorize detected threats based on predefined categories, enriching requests with detailed threat classifications.</li> <li><strong>LangKit</strong> (<a href="https://github.com/whylabs/langkit/tree/main">GitHub</a>): A toolkit designed for monitoring and securing LLMs, LangKit can analyze request content for signs of adversarial inputs or unintended model behaviors. It offers hooks for integrating custom analysis functions.</li> <li><strong>Vigil-LLM</strong> (<a href="https://github.com/deadbits/vigil-llm">GitHub</a>): Focuses on real-time monitoring and alerting for suspicious LLM requests. Integration into the proxy layer allows for immediate flagging potential security issues, enriching the request data with vigilance scores.</li> <li><strong>Open-Prompt Injection</strong> (<a href="https://github.com/liu00222/Open-Prompt-Injection">GitHub</a>): Offers methodologies and tools for detecting prompt injection attacks, allowing for the enrichment of request data with specific indicators of compromise related to prompt injection techniques.</li> </ul> <p><em>Note: Most of these tools require additional calls/costs to an external LLM, and would require further infrastructure to threat hunt effectively.</em></p> <p>One simple example implementation that uses LLM-guard and LangKit might look like this:</p> <pre><code>def analyze_and_enrich_request( prompt: str, response_text: str, error_response: Optional[dict] = None ) -> dict: """Analyze the prompt and response text for malicious content and enrich the document.""" # LLM Guard analysis sanitized_prompt, results_valid_prompt, results_score_prompt = scan_prompt( input_scanners, prompt["content"] ) ( sanitized_response_text, results_valid_response, results_score_response, ) = scan_output(output_scanners, sanitized_prompt, response_text) # LangKit for additional analysis schema = injections.init() langkit_result = extract({"prompt": prompt["content"]}, schema=schema) # Initialize identified threats and malicious flag identified_threats = [] # Check LLM Guard results for prompt if not any(results_valid_prompt.values()): identified_threats.append("LLM Guard Prompt Invalid") # Check LLM Guard results for response if not any(results_valid_response.values()): identified_threats.append("LLM Guard Response Invalid") # Check LangKit result for prompt injection prompt_injection_score = langkit_result.get("prompt.injection", 0) if prompt_injection_score > 0.4: # Adjust threshold as needed identified_threats.append("LangKit Injection") # Identify threats based on LLM Guard scores for category, score in results_score_response.items(): if score > 0.5: identified_threats.append(category) # Combine results and enrich document # llm_guard scores map scanner names to float values of risk scores, # where 0 is no risk, and 1 is high risk. # langkit_score is a float value of the risk score for prompt injection # based on known threats. enriched_document = { "analysis": { "llm_guard_prompt_scores": results_score_prompt, "llm_guard_response_scores": results_score_response, "langkit_score": prompt_injection_score, }, "malicious": any(identified_threats), "identified_threats": identified_threats, } # Check if there was an error from OpenAI and enrich the analysis if error_response: code = error_response.get("code") filtered_categories = { category: info["filtered"] for category, info in error_response.get( "content_filter_result", {} ).items() } enriched_document["analysis"]["openai"] = { "code": code, "filtered_categories": filtered_categories, } if code == "ResponsibleAIPolicyViolation": enriched_document["malicious"] = True return enriched_document</code></pre> <p>This function could be called for each request passing through the proxy, with the returned data being appended to the request document before it's sent to Elasticsearch. The result is a detailed and actionable dataset that captures the raw interactions with the LLM and provides immediate security insights to embed in our detection rules based on the request and response. Going full circle with the prompt injection LLM01 example, the query could be updated to something like this:</p> <pre><code>FROM azure-openai-logs | WHERE @timestamp > NOW() - 1 DAY | WHERE identified_threats.keyword == "LangKit Injection" OR analysis.langkit_score > 0.4 | stats total_attempts = count(*), users = count_distinct(connectorId) by identified_threats.keyword, analysis.langkit_score | WHERE users == 1 and total_attempts >= 2</code></pre> <p>As you can see, both scoring mechanisms are subjective based on the results returned from the open source prompt analysis tools. This query filters logs from the past day where the identified threat is "LangKit Injection" or the LangKit score is above <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">0.4</code>. It then calculates the total attempts and counts the number of unique users (agents) associated with each identified threat category and LangKit score, filtering to include only cases where there's a single user involved (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">users == 1</code>) and the total attempts are two or more (<code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">total_attempts >= 2</code>).</p> <p>With these additional tools, we have a variety of analysis result fields available to improve our detection rules. In these examples, we shipped most of the data as-is for simplicity. However, in a production environment, it's crucial to normalize these fields across all tools and LLM responses to a schema like <a href="https://www.elastic.co/guide/en/ecs/current/ecs-reference.html">Elastic Common Schema</a> (ECS). Normalizing data to ECS enhances interoperability between different data sources, simplifies analysis, and streamlines the creation of more effective and cohesive security rules.</p> <p>In Part two of this series, we will discuss how we’ve taken a more formal approach to ECS field mapping, and integrations.</p> <h2 class="font-bold text-2xl md:text-4xl relative"><span id="alternative-options-for-llm-application-auditing" class="absolute -top-32"></span>Alternative Options for LLM Application Auditing</h2> <p>While using a proxy may be straightforward, other approaches may better suit a production setup; for example:</p> <ul> <li>Utilizing <a href="https://www.elastic.co/observability/application-performance-monitoring">application performance monitoring</a> (APM)</li> <li>Using the OpenTelemetry integration</li> <li>Modifying changes in Kibana directly to audit and trace LLM activity</li> </ul> <p>Unsurprisingly, these approaches have potential limitations like not natively ingesting all the LLM security analysis tool data generated without developing custom logic to support third-party tools.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="leveraging-elastic-apm-for-in-depth-application-insights" class="absolute -top-32"></span>Leveraging Elastic APM for In-Depth Application Insights</h3> <p>Elastic <a href="https://www.elastic.co/guide/en/observability/current/apm.html">APM</a> provides an alternative solution for monitoring applications in real-time, essential for detecting performance bottlenecks and identifying problematic requests or queries. By integrating Elastic APM, users gain detailed insights into transaction times, database query performance, external API call efficiency, and more. This comprehensive visibility makes it easier to address and resolve performance issues or errors quickly. Unlike the proxy approach, APM automatically ingests logs into Elastic about your application, providing an opportunity to create security detection rules based on the behaviors seen within your data.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="utilizing-opentelemetry-for-enhanced-observability" class="absolute -top-32"></span>Utilizing OpenTelemetry for Enhanced Observability</h3> <p>For applications already employing OpenTelemetry, leveraging its <a href="https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html">integration</a> with Elastic APM can enhance observability without requiring extensive instrumentation changes. This integration supports capturing a wide array of telemetry data, including traces and metrics, which can be seamlessly sent to the Elastic Stack. This approach allows developers to continue using familiar libraries while benefiting from the robust monitoring capabilities of Elastic. OpenTelemetry’s compatibility across multiple programming languages and its <a href="https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html">support through Elastic’s native protocol</a> (OTLP) facilitate straightforward data transmission, providing a robust foundation for monitoring distributed systems. Compared to the proxy example, this approach more natively ingests data than maintaining an independent index and logging mechanism to Elastic.</p> <h3 class="font-bold leading-tight text-xl md:text-3xl relative"><span id="llm-auditing-with-kibana" class="absolute -top-32"></span>LLM Auditing with Kibana</h3> <p>Like writing custom logic for your LLM application to audit and ship data, you can test the approach with Elastic’s AI Assistant. If you're comfortable with TypeScript, consider deploying a local Elastic instance using the Kibana <a href="https://www.elastic.co/guide/en/kibana/current/development-getting-started.html">Getting Started Guide</a>. Once set up, navigate to the <a href="https://github.com/elastic/kibana/tree/main/x-pack/plugins/elastic_assistant">Elastic AI Assistant</a> and configure it to intercept LLM requests and responses for auditing and analysis. Note: This approach primarily tracks Elastic-specific LLM integration compared to using APM and other integrations or a proxy to track third-party applications. It should only be considered for experimentation and exploratory testing purposes.</p> <p>Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting <code class="px-1.5 py-1 rounded not-prose bg-[var(--tw-prose-invert-pre-bg)] whitespace-break-spaces text-[85%] text-emerald-600">elastic.apm.active: true</code>). See the <a href="https://github.com/elastic/kibana/blob/main/x-pack/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx">README</a> for more details.</p> <h2 class="font-bold text-2xl md:text-4xl relative"><span id="closing-thoughts" class="absolute -top-32"></span>Closing Thoughts</h2> <p>As we continue with this exploration into integrating security practices within the lifecycle of large language models at Elastic, it's clear that embedding security into LLM workflows can provide a path forward for creating safer and more reliable applications. These contrived examples, drawn from our work during OnWeek, illustrate how someone can proactively detect, alert, and triage malicious activity, leveraging the security solutions that analysts find most intuitive and effective.</p> <p>It’s also worth noting that with the example proxy approach, we can incorporate a model to actively detect and prevent requests. Additionally, we can triage the LLM response before sending it back to the user if we’ve identified malicious threats. At this point, we have the flexibility to extend our security protections to cover a variety of defensive approaches. In this case, there is a fine line between security and performance, as each additional check will consume time and impede the natural conversational flow that users would expect.</p> <p>Feel free to check out the proof-of-concept proxy at <a href="https://github.com/elastic/llm-detection-proxy">llm-detection-proxy</a> and adapt it to fit your needs!</p> <p>We’re always interested in hearing use cases and workflows like these, so as always, reach out to us via <a href="https://github.com/elastic/detection-rules/issues">GitHub issues</a>, chat with us in our <a href="http://ela.st/slack">community Slack</a>, and ask questions in our <a href="https://discuss.elastic.co/c/security/endpoint-security/80">Discuss forums</a>.</p> <p><em>The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.</em></p></div></div></div><div class="hidden lg:flex lg:col-span-1 text-sm lg:flex-col lg:space-y-6"><div class="toc"><h4 class="font-bold leading-tight text-lg md:text-2xl mb-3">Jump to section</h4><ul class="flex flex-col space-y-2"><li><a class="flex items-center space-x-1 hover:text-white" href="/security-labs/embedding-security-in-llm-workflows#proxying-llm-requests-and-setup"><span>Proxying LLM Requests and Setup</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#quick-start-option-proxy-with-flask"><span>Quick Start Option: Proxy with Flask</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#langsmith-proxy"><span>Langsmith Proxy</span></a></li><li><a class="flex items-center space-x-1 hover:text-white" href="/security-labs/embedding-security-in-llm-workflows#llm-potential-attacks-and-detection-rule-opportunities"><span>LLM Potential Attacks and Detection Rule Opportunities</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#llm01---prompt-injection"><span>LLM01 - prompt injection</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#llm02---insecure-output-handling"><span>LLM02 - insecure output handling</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#llm04---model-dos"><span>LLM04 - model DoS</span></a></li><li><a class="flex items-center space-x-1 hover:text-white ml-4" href="/security-labs/embedding-security-in-llm-workflows#llm06---sensitive-information-disclosure"><span>LLM06 - sensitive information disclosure</span></a></li><li><a class="flex items-center space-x-1 hover:text-white" href="/security-labs/embedding-security-in-llm-workflows#enriching-detection-rules-with-security-insights"><span>Enriching Detection Rules with Security Insights</span></a></li><li><a class="flex items-center space-x-1 hover:text-white" href="/security-labs/embedding-security-in-llm-workflows#alternative-options-for-llm-application-auditing"><span>Alternative Options for LLM Application Auditing</span></a></li></ul><button class="border-t border-white/20 w-full mt-3 py-2 flex items-center space-x-1 text-xs font-medium uppercase tracking-wide hover:text-white"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" class="w-3 h-3"><path d="M10.75 4.75a.75.75 0 00-1.5 0v4.5h-4.5a.75.75 0 000 1.5h4.5v4.5a.75.75 0 001.5 0v-4.5h4.5a.75.75 0 000-1.5h-4.5v-4.5z"></path></svg><span>Show more</span></button></div><div class="bg-zinc-900 border border-zinc-800 drop-shadow-lg p-5 md:p-2 sm:p-4 md:px-6 md:py-4 rounded-xl"><h4 class="font-bold leading-tight text-lg md:text-2xl mb-3">Elastic Security Labs Newsletter</h4><div><a target="_blank" class="button inline-flex" href="https://www.elastic.co/elastic-security-labs/newsletter?utm_source=security-labs">Sign Up</a></div></div></div></div><div class="bg-zinc-900 border border-zinc-800 drop-shadow-lg p-5 md:p-2 sm:p-4 md:px-6 md:py-4 rounded-xl my-5 md:my-10 max-w-3xl mx-auto flex flex-col items-center shadow-2xl"><h4 class="font-bold leading-tight text-lg md:text-2xl">Share this article</h4><div class="flex flex-wrap items-center justify-center mt-4 space-x-4"><a class="flex items-center space-x-2 button" href="https://twitter.com/intent/tweet?text=Embedding Security in LLM Workflows: Elastic's Proactive Approach&url=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows" target="_blank" rel="noopener noreferrer" aria-label="Share this article on Twitter" title="Share this article on Twitter"><svg class="w-4 h-4" viewBox="0 0 24 24"><path fill="currentColor" d="M23.954 4.569c-.885.389-1.83.653-2.825.772a4.98 4.98 0 002.187-2.746 9.955 9.955 0 01-3.157 1.204 4.98 4.98 0 00-8.49 4.54A14.128 14.128 0 011.69 3.05a4.98 4.98 0 001.54 6.638A4.94 4.94 0 011.2 8.62v.06a4.98 4.98 0 004 4.87 4.94 4.94 0 01-2.24.086 4.98 4.98 0 004.64 3.45A9.97 9.97 0 010 20.35a14.075 14.075 0 007.59 2.22c9.16 0 14.17-7.583 14.17-14.17 0-.217-.005-.434-.015-.65a10.128 10.128 0 002.485-2.58l-.001-.001z"></path></svg><span>Twitter</span></a><a class="flex items-center space-x-2 button" href="https://www.facebook.com/sharer/sharer.php?u=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows" target="_blank" rel="noopener noreferrer" aria-label="Share this article on Facebook" title="Share this article on Facebook"><svg class="w-4 h-4" viewBox="0 0 24 24"><path fill="currentColor" d="M22.5 12c0-5.799-4.701-10.5-10.5-10.5S1.5 6.201 1.5 12c0 5.301 3.901 9.699 9 10.401V14.4h-2.7v-2.7h2.7v-2.1c0-2.7 1.8-4.2 4.2-4.2 1.2 0 2.1.1 2.4.2v2.4h-1.5c-1.2 0-1.5.6-1.5 1.5v1.8h3l-.3 2.7h-2.7V22C18.599 21.3 22.5 17.301 22.5 12z"></path></svg><span>Facebook</span></a><a class="flex items-center space-x-2 button" href="https://www.linkedin.com/shareArticle?mini=true&url=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows&title=Embedding Security in LLM Workflows: Elastic's Proactive Approach" target="_blank" rel="noopener noreferrer" aria-label="Share this article on LinkedIn" title="Share this article on LinkedIn"><svg class="w-4 h-4" viewBox="0 0 24 24"><path fill="currentColor" d="M19 0h-14c-2.761 0-5 2.239-5 5v14c0 2.761 2.239 5 5 5h14c2.762 0 5-2.239 5-5v-14c0-2.761-2.238-5-5-5zm-11 19h-3v-11h3v11zm-1.5-12.268c-.966 0-1.75-.79-1.75-1.764s.784-1.764 1.75-1.764 1.75.79 1.75 1.764-.783 1.764-1.75 1.764zm13.5 12.268h-3v-5.604c0-3.368-4-3.113-4 0v5.604h-3v-11h3v1.765c1.396-2.586 7-2.777 7 2.476v6.759z"></path></svg><span>LinkedIn</span></a><a class="flex items-center space-x-2 button" href="https://reddit.com/submit?url=https://www.elastic.co/security-labs/embedding-security-in-llm-workflows&title=Embedding Security in LLM Workflows: Elastic's Proactive Approach" target="_blank" rel="noopener noreferrer" aria-label="Share this article on Reddit" title="Share this article on Reddit"><svg class="w-4 h-4" viewBox="0 0 24 24"><path fill-rule="evenodd" clip-rule="evenodd" d="M24 12C24 18.6274 18.6274 24 12 24C5.37258 24 0 18.6274 0 12C0 5.37258 5.37258 0 12 0C18.6274 0 24 5.37258 24 12ZM19.6879 11.0584C19.8819 11.3352 19.9916 11.6622 20.004 12C20.0091 12.3306 19.9205 12.656 19.7485 12.9384C19.5765 13.2208 19.3281 13.4488 19.032 13.596C19.0455 13.7717 19.0455 13.9483 19.032 14.124C19.032 16.812 15.9 18.996 12.036 18.996C8.172 18.996 5.04 16.812 5.04 14.124C5.02649 13.9483 5.02649 13.7717 5.04 13.596C4.80919 13.49 4.6042 13.335 4.43923 13.1419C4.27427 12.9487 4.15327 12.722 4.08462 12.4775C4.01598 12.2329 4.00133 11.9764 4.04169 11.7256C4.08205 11.4748 4.17646 11.2358 4.31837 11.0251C4.46028 10.8145 4.6463 10.6372 4.86354 10.5056C5.08078 10.3739 5.32404 10.2911 5.57646 10.2629C5.82889 10.2346 6.08444 10.2616 6.32541 10.3419C6.56638 10.4222 6.78701 10.5539 6.972 10.728C8.35473 9.79023 9.98146 9.27718 11.652 9.252L12.54 5.088C12.55 5.03979 12.5694 4.99405 12.5972 4.95341C12.625 4.91277 12.6605 4.87805 12.7018 4.85127C12.7431 4.82448 12.7894 4.80615 12.8378 4.79735C12.8862 4.78855 12.9359 4.78945 12.984 4.8L15.924 5.388C16.0676 5.14132 16.2944 4.9539 16.5637 4.85937C16.833 4.76484 17.1272 4.7694 17.3934 4.87222C17.6597 4.97505 17.8806 5.1694 18.0164 5.42041C18.1523 5.67141 18.1942 5.96262 18.1348 6.24177C18.0753 6.52092 17.9182 6.76972 17.6918 6.94352C17.4654 7.11732 17.1845 7.20473 16.8995 7.19006C16.6144 7.1754 16.3439 7.05962 16.1366 6.8635C15.9292 6.66738 15.7985 6.40378 15.768 6.12L13.2 5.58L12.42 9.324C14.0702 9.3594 15.6749 9.87206 17.04 10.8C17.2839 10.566 17.5902 10.4074 17.9221 10.3436C18.254 10.2797 18.5973 10.3132 18.9106 10.4401C19.2239 10.5669 19.4939 10.7817 19.6879 11.0584ZM8.20624 12.5333C8.07438 12.7307 8.004 12.9627 8.004 13.2C8.004 13.5183 8.13043 13.8235 8.35547 14.0485C8.58051 14.2736 8.88574 14.4 9.204 14.4C9.44134 14.4 9.67335 14.3296 9.87068 14.1978C10.068 14.0659 10.2218 13.8785 10.3127 13.6592C10.4035 13.4399 10.4272 13.1987 10.3809 12.9659C10.3346 12.7331 10.2204 12.5193 10.0525 12.3515C9.8847 12.1836 9.67089 12.0694 9.43811 12.0231C9.20533 11.9768 8.96405 12.0005 8.74478 12.0913C8.52551 12.1822 8.33809 12.336 8.20624 12.5333ZM12.012 17.424C13.0771 17.4681 14.1246 17.1416 14.976 16.5V16.548C15.0075 16.5173 15.0327 16.4806 15.05 16.4402C15.0674 16.3997 15.0766 16.3563 15.0772 16.3122C15.0777 16.2682 15.0696 16.2245 15.0533 16.1837C15.0369 16.1428 15.0127 16.1055 14.982 16.074C14.9513 16.0425 14.9146 16.0173 14.8742 16C14.8337 15.9826 14.7903 15.9734 14.7462 15.9728C14.7022 15.9723 14.6585 15.9804 14.6177 15.9967C14.5768 16.0131 14.5395 16.0373 14.508 16.068C13.7797 16.5904 12.895 16.8487 12 16.8C11.1061 16.8399 10.2255 16.5732 9.504 16.044C9.44182 15.993 9.36289 15.9669 9.28256 15.9708C9.20222 15.9748 9.12622 16.0085 9.06935 16.0653C9.01247 16.1222 8.97879 16.1982 8.97484 16.2786C8.97089 16.3589 8.99697 16.4378 9.048 16.5C9.89937 17.1416 10.9469 17.4681 12.012 17.424ZM14.0933 14.2458C14.2907 14.3776 14.5227 14.448 14.76 14.448L14.748 14.496C14.9107 14.4978 15.0721 14.4664 15.2223 14.4038C15.3725 14.3413 15.5084 14.2488 15.6218 14.1321C15.7352 14.0154 15.8236 13.8768 15.8818 13.7248C15.9399 13.5728 15.9665 13.4106 15.96 13.248C15.96 13.0107 15.8896 12.7787 15.7578 12.5813C15.6259 12.384 15.4385 12.2302 15.2192 12.1393C14.9999 12.0485 14.7587 12.0248 14.5259 12.0711C14.2931 12.1174 14.0793 12.2316 13.9115 12.3995C13.7436 12.5673 13.6294 12.7811 13.5831 13.0139C13.5368 13.2467 13.5605 13.4879 13.6513 13.7072C13.7422 13.9265 13.896 14.1139 14.0933 14.2458Z" fill="currentColor"></path></svg><span>Reddit</span></a></div></div></article></main><footer class="mt-auto text-xs md:text-sm"><div class="container py-6 flex flex-col md:flex-row gap-2 md:gap-0 justify-between items-center"><div class="text-zinc-300"><nav><ul class="flex space-x-4"><li><a class="hover:text-white font-medium" href="/security-labs/sitemap.xml">Sitemap</a></li><li><a class="hover:text-white font-medium flex items-center space-x-1" href="https://elastic.co?utm_source=elastic-search-labs&utm_medium=referral&utm_campaign=search-labs&utm_content=footer"><svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" aria-hidden="true" class="inline-block w-3 h-3"><path stroke-linecap="round" stroke-linejoin="round" d="M13.5 6H5.25A2.25 2.25 0 003 8.25v10.5A2.25 2.25 0 005.25 21h10.5A2.25 2.25 0 0018 18.75V10.5m-10.5 6L21 3m0 0h-5.25M21 3v5.25"></path></svg><span>Elastic.co</span></a></li><li><a class="hover:text-white font-medium flex items-center space-x-1" href="https://twitter.com/elasticseclabs"><svg class="w-4 h-4 inline-block w-3 h-3" viewBox="0 0 24 24"><path fill="currentColor" d="M23.954 4.569c-.885.389-1.83.653-2.825.772a4.98 4.98 0 002.187-2.746 9.955 9.955 0 01-3.157 1.204 4.98 4.98 0 00-8.49 4.54A14.128 14.128 0 011.69 3.05a4.98 4.98 0 001.54 6.638A4.94 4.94 0 011.2 8.62v.06a4.98 4.98 0 004 4.87 4.94 4.94 0 01-2.24.086 4.98 4.98 0 004.64 3.45A9.97 9.97 0 010 20.35a14.075 14.075 0 007.59 2.22c9.16 0 14.17-7.583 14.17-14.17 0-.217-.005-.434-.015-.65a10.128 10.128 0 002.485-2.58l-.001-.001z"></path></svg><span>@elasticseclabs</span></a></li></ul></nav></div><div class="flex flex-col space-y-1 text-zinc-300"><p>© <!-- -->2025<!-- -->. Elasticsearch B.V. All Rights Reserved.</p></div></div></footer></main></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"article":{"title":"Embedding Security in LLM Workflows: Elastic's Proactive Approach","slug":"embedding-security-in-llm-workflows","date":"2024-04-25","description":"Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications.","image":"Security Labs Images 5.jpg","subtitle":"Exploring Elastic's innovative approach for integrating security into the lifecycle of LLMs to safeguard against vulnerabilities featuring Elastic’s AI Assistant.","body":{"raw":"\nWe recently concluded one of our quarterly Elastic OnWeek events, which provides a unique week to explore opportunities outside of our regular day-to-day. In line with recent publications from [OWASP](https://owasp.org/www-project-top-10-for-large-language-model-applications/) and the [NSA AISC](https://media.defense.gov/2024/Apr/15/2003439257/-1/-1/0/CSI-DEPLOYING-AI-SYSTEMS-SECURELY.PDF), we decided to spend some time with the OWASP Top Ten vulnerabilities for LLMs natively in Elastic. In this article, we touch on a few opportunities to detect malicious LLM activity with [ES|QL](https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html), namely:\n\n - LLM01: Prompt Injection\n - LLM02: Insecure Output Handling\n - LLM04: Model Denial of Service\n - LLM06: Sensitive Information Disclosure\n\nElastic provides the ability to audit LLM applications for malicious behaviors; we’ll show you one approach with just four steps:\n\n 1. Intercepting and analyzing the LLM requests and responses \n 2. Enriching data with LLM-specific analysis results\n 3. Sending data to Elastic Security\n 4. Writing ES|QL detection rules that can later be used to respond \n\nThis approach reflects our ongoing efforts to explore and implement advanced detection strategies, including developing detection rules tailored specifically for LLMs, while keeping pace with emerging generative AI technologies and security challenges. Building on this foundation, last year marked a significant enhancement to our toolkit and overall capability to continue this proactive path forward. \n\nElastic [released](https://www.elastic.co/blog/introducing-elastic-ai-assistant) the AI Assistant for Security, introducing how the open generative AI sidekick is powered by the [Search AI Platform](https://www.elastic.co/platform) — a collection of relevant tools for developing advanced search applications. Backed by machine learning (ML) and artificial intelligence (AI), this AI Assistant provides powerful pre-built workflows like alert summarization, workflow suggestions, query conversions, and agent integration advice. I highly recommend you read more on Elastic’s [AI Assistant](https://www.elastic.co/elasticsearch/ai-assistant) about how the capabilities seamlessly span across Observability and Security.\n\nWe can use the AI Assistant’s capabilities as a third-party LLM application to capture, audit, and analyze requests and responses for convenience and to run experiments. Once data is in an index, writing behavioral detections on it becomes business as usual — we can also leverage the entire security detection engine. Even though we’re proxying the Elastic AI Assistant LLM activity in this experiment, it’s merely used as a vehicle to demonstrate auditing LLM-based applications. Furthermore, this proxy approach is intended for third-party applications to ship data to [Elastic Security](https://www.elastic.co/guide/en/security/current/es-overview.html). \n\nWe can introduce security mechanisms into the application's lifecycle by intercepting LLM activity or leveraging observable LLM metrics. It’s common practice to address prompt-based threats by [implementing various safety tactics](https://platform.openai.com/docs/guides/safety-best-practices):\n\n 1. **Clean Inputs**: Sanitize and validate user inputs before feeding them to the model\n 2. **Content Moderation**: Use OpenAI tools to filter harmful prompts and outputs\n 3. **Rate Limits and Monitoring**: Track usage patterns to detect suspicious activity\n 4. **Allow/Blocklists**: Define acceptable or forbidden inputs for specific applications\n 5. **Safe Prompt Engineering**: Design prebuilt prompts that guide the model towards intended outcomes\n 6. **User Role Management**: Control user access to prevent unauthorized actions\n 7. **Educate End-Users**: Promote responsible use of the model to mitigate risks\n 8. **Red Teaming \u0026 Monitoring**: Test for vulnerabilities and continuously monitor for unexpected outputs\n 9. **HITL Feedback for Model Training**: Learn from human-in-the-loop, flagged issues to refine the model over time\n 10. **Restrict API Access**: Limit model access based on specific needs and user verification\n\nTwo powerful features provided by OpenAI, and many other LLM implementers, is the ability to [submit end-user IDs](https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids) and check content against a [moderation API](https://platform.openai.com/docs/guides/moderation), features that set the bar for LLM safety. Sending hashed IDs along with the original request aids in abuse detection and provides targeted feedback, allowing unique user identification without sending personal information. Alternatively, OpenAI's moderation endpoint helps developers identify potentially harmful content like hate speech, self-harm encouragement, or violence, allowing them to filter such content. It even goes a step further by detecting threats and intent to self-harm. \n\nDespite all of the recommendations and best practices to protect against malicious prompts, we recognize that there is no single perfect solution. When using capabilities like OpenAI’s API, some of these threats may be detected by the content filter, which will respond with a usage policy violation notification: \n\n\n\n\nThis content filtering is beneficial to address many issues; however, it cannot identify further threats in the broader context of the environment, application ecosystem, or other alerts that may appear. The more we can integrate generative AI use cases into our existing protection capabilities, the more control and possibilities we have to address potential threats. Furthermore, even if LLM safeguards are in place to stop rudimentary attacks, we can still use the detection engine to alert and take future remediation actions instead of silently blocking or permitting abuse. \n\n## Proxying LLM Requests and Setup\n\nThe optimal security solution integrates additional safeguards directly within the LLM application's ecosystem. This allows enriching alerts with the complete context surrounding requests and responses. As requests are sent to the LLM, we can intercept and analyze them for potential malicious activity. If necessary, a response action can be triggered to defer subsequent HTTP calls. Similarly, inspecting the LLM's response can uncover further signs of malicious behavior. \n\nUsing a proxy to handle these interactions offers several advantages:\n\n - **Ease of Integration and Management**: By managing the new security code within a dedicated proxy application, you avoid embedding complex security logic directly into the main application. This approach minimizes changes needed in the existing application structure, allowing for easier maintenance and clearer separation of security from business logic. The main application must only be reconfigured to route its LLM requests through the proxy.\n - **Performance and Scalability**: Placing the proxy on a separate server isolates the security mechanisms and helps distribute the computational load. This can be crucial when scaling up operations or managing performance-intensive tasks, ensuring that the main application's performance remains unaffected by the additional security processing.\n\n### Quick Start Option: Proxy with Flask\n\nYou can proxy incoming and outgoing LLM connections for a faster initial setup. This approach can be generalized for other LLM applications by creating a simple Python-based [Flask](https://flask.palletsprojects.com/en/3.0.x/) application. This application would intercept the communication, analyze it for security risks, and log relevant information before forwarding the response.\n\n\n\n\nMultiple SDKs exist to connect to Elasticsearch and handle OpenAI LLM requests. The provided [llm-detection-proxy](https://github.com/elastic/llm-detection-proxy) repo demonstrates the available Elastic and OpenAI clients. This snippet highlights the bulk of the experimental proxy in a single Flask route.\n\n``` python\n@app.route(\"/proxy/openai\", methods=[\"POST\"])\ndef azure_openai_proxy():\n \"\"\"Proxy endpoint for Azure OpenAI requests.\"\"\"\n data = request.get_json()\n messages = data.get(\"messages\", [])\n response_content = \"\"\n error_response = None\n\n try:\n # Forward the request to Azure OpenAI\n response = client.chat.completions.create(model=deployment_name, messages=messages)\n response_content = response.choices[0].message.content # Assuming one choice for simplicity\n choices = response.choices[0].model_dump()\n except openai.BadRequestError as e:\n # If BadRequestError is raised, capture the error details\n error_response = e.response.json().get(\"error\", {}).get(\"innererror\", {})\n response_content = e.response.json().get(\"error\", {}).get(\"message\")\n\n # Structure the response with the error details\n choices = {**error_response.get(\"content_filter_result\", {}),\n \"error\": response_content, \"message\": {\"content\": response_content}}\n\n # Perform additional analysis and create the Elastic document\n additional_analysis = analyze_and_enrich_request(prompt=messages[-1],\n response_text=response_content,\n error_response=error_response)\n log_data = {\"request\": {\"messages\": messages[-1]},\n \"response\": {\"choices\": response_content},\n **additional_analysis}\n\n # Log the last message and response\n log_to_elasticsearch(log_data)\n\n # Calculate token usage\n prompt_tokens = sum(len(message[\"content\"]) for message in messages)\n completion_tokens = len(response_content)\n total_tokens = prompt_tokens + completion_tokens\n\n # Structure and return the response\n return jsonify({\n \"choices\": [choices],\n \"usage\": {\n \"prompt_tokens\": prompt_tokens,\n \"completion_tokens\": completion_tokens,\n \"total_tokens\": total_tokens,\n }\n })\n```\n\nWith the Flask server, you can configure the [OpenAI Kibana Connector](https://www.elastic.co/guide/en/kibana/current/openai-action-type.html) to use your proxy. \n\n\n\nSince this proxy to your LLM is running locally, credentials and connection information are managed outside of Elastic, and an empty string can be provided in the API key section. Before moving forward, testing your connection is generally a good idea. It is important to consider other security implications if you are considering implementing a proxy solution in a real environment - not something this prototype considered for brevity.\n\n\n\n\nWe can now index our LLM requests and responses and begin to write detections on the available data in the ```azure-openai-logs``` index created in this experiment. Optionally, we could preprocess the data using an Elastic [ingestion pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html), but in this contrived example, we can effectively write detections with the power of ES|QL.\n\n\nSample AzureOpenAI LLM Request/Response Data\n\n### Langsmith Proxy\n\n*Note: The [Langsmith Proxy](https://docs.smith.langchain.com/proxy/quickstart) project provides a dockerized proxy for your LLM APIs. While it offers a minimized solution, as of this writing, it lacks native capabilities for incorporating custom security analysis tools or integrating directly with Elastic Security.*\n\nThe LangSmith Proxy is designed to simplify LLM API interaction. It's a sidecar application requiring minimal configuration (e.g., LLM API URL). It enhances performance (caching, streaming) for high-traffic scenarios. It uses NGINX for efficiency and supports optional tracing for detailed LLM interaction tracking. Currently, it works with OpenAI and AzureOpenAI, with future support planned for other LLMs.\n\n## LLM Potential Attacks and Detection Rule Opportunities\n\n**It’s important to understand that even though documented lists of protections do not accompany some LLMs, simply trying some of these prompts may be immediately denied or result in banning on whatever platform used to submit the prompt. We recommend experimenting with caution and understand the SLA prior to sending any malicious prompts. Since this exploration leverages OpenAI’s resources, we recommend following the bugcrowd [guidance](https://bugcrowd.com/openai) and sign up for an additional testing account using your @bugcrowdninja.com email address.**\n\nHere is a list of several plausible examples to illustrate detection opportunities. Each LLM topic includes the OWASP description, an example prompt, a sample document, the detection opportunity, and potential actions users could take if integrating additional security mechanisms in their workflow. \n\nWhile this list is currently not extensive, Elastic Security Labs is currently undertaking a number of initiatives to ensure future development, and formalization of rules will continue.\n\n### LLM01 - prompt injection\n\n**OWASP Description**: Manipulating LLMs via crafted inputs can lead to unauthorized access, data breaches, and compromised decision-making. Reference [here](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM01_PromptInjection.md).\n\n**Example**: An adversary might try to craft prompts that trick the LLM into executing unintended actions or revealing sensitive information. *Note: Tools like [promptmap](https://github.com/utkusen/promptmap) are available to generate creative prompt injection ideas and automate the testing process.*\n\n**Prompt**:\n\n\n**Sample Response**:\n\n\n**Detection Rule Opportunity**: In this example, the LLM responded by refusing to handle database connection strings due to security risks. It emphasizes keeping credentials private and suggests using secure methods like environment variables or vaults to protect them.\n\nA very brittle but basic indicator-matching query may look like this:\n\n``` sql\nFROM azure-openai-logs |\n WHERE request.messages.content LIKE \"*generate*connection*string*\"\n OR request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm sorry, but I can't assist*\"\n```\n\nA slightly more advanced query detects more than two similar attempts within the last day. \n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm*sorry,*but*I*can't*assist*\"\n OR response.choices LIKE \"*I*can’t*process*actual*sensitive*\"\n| stats total_attempts = count(*) by connectorId\n| WHERE total_attempts \u003e= 2\n```\n\n*Note that there are many approaches to detect malicious prompts and protect LLM responses. Relying on these indicators alone is not the best approach; however, we can gradually improve the detection with additional enrichment or numerous response attempts. Furthermore, if we introduce an ID into our documents, we can further enhance our query by aggregating attempts based on the field that correlates to a specific user.*\n\n**Example 2**: The [Crescendo](https://arxiv.org/abs/2404.01833v1) effect is a realistic jailbreak attack where an adversary gradually manipulates a language model through a series of seemingly innocent inquiries that shift towards asking the model to describe hypothetical scenarios involving the unauthorized access and manipulation of secure systems. By doing so, they aim to extract methods that could potentially bypass the LLM’s security constraints.\n\n**Prompt**:\n\n\n**Sample Response**:\n\n\nWith the additional analysis from OpenAI’s filtering, we can immediately detect the first occurrence of abuse. \n\n**Detection Rule Opportunity**:\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n AND (\n request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm sorry, but I can't assist*\"\n OR analysis.openai.code == \"ResponsibleAIPolicyViolation\"\n OR malicious\n )\n| STATS total_attempts = COUNT(*) BY connectorId\n| WHERE total_attempts \u003e 1\n| SORT total_attempts DESC\n```\n\nHowever, as you continue to use the Crescendo Effect, we notice that the conversation pivot goes unblocked after the initial content filter by OpenAI. It’s important to understand that even if tactics like this are difficult to prevent, we still have opportunities to detect.\n\n\n\nAdditional analysis tools, like LLM-Guard, detect if the conversation is sensitive, which, in this case, is inaccurate. However, it hints at potential opportunities to track malicious behavior over multiple prompts. Note: We could also take advantage of EQL sequences as an alternative to this ES|QL query to help track behaviors over multiple events.\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n AND (\n request.messages.content LIKE \"*Molotov*\"\n OR analysis.openai.code == \"ResponsibleAIPolicyViolation\"\n OR malicious\n )\n| STATS attempts = count(*), max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive) BY connectorId\n| WHERE attempts \u003e= 1 AND max_sensitivity \u003e 0.5\n| SORT attempts DESC\n```\n\nThis query detects suspicious behavior related to Molotov Cocktails across multiple events by analyzing sequences of log entries associated with a single user/session (identified by connectorId). The query core filters events based on:\n\n - **Content Matching**: It searches for mentions of \"Molotov\" in conversation content (```request.messages.content LIKE \"*Molotov*\"```)\n - **Policy Violations: It identifies attempts blocked by OpenAI's safety filters (```analysis.openai.code == \"ResponsibleAIPolicyViolation\"```), indicating the start of potentially suspicious behavior\n - **Malicious Flag Consideration**: It includes logs where the system flagged the content as malicious (```malicious == true```), capturing potentially subtle or varied mentions\n - **Session-Level Analysis**: By grouping events by connectorId, it analyzes the complete sequence of attempts within a session. It then calculates the total number of attempts (```attempts = count(*)```) and the highest sensitivity score (```max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive)```) across all attempts in that session\n - **Flagging High-Risk Sessions**: It filters sessions with at least one attempt (```attempts \u003e= 1```) and a maximum sensitivity score exceeding 0.5 (```max_sensitivity \u003e 0.5```). This threshold helps focus on sessions where users persistently discussed or revealed potentially risky content.\n\nBy analyzing these factors across multiple events within a session, we can start building an approach to detect a pattern of escalating discussions, even if individual events might not be flagged alone. \n\n### LLM02 - insecure output handling\n\n**OWASP Description**: Neglecting to validate LLM outputs may lead to downstream security exploits, including code execution that compromises systems and exposes data. Reference [here](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM02_InsecureOutputHandling.md).\n\n**Example**: An adversary may attempt to exploit the LLM to generate outputs that can be used for cross-site scripting (XSS) or other injection attacks.\n\n**Prompt**:\n\n\n**Sample Response**:\n\n\n**Detection Rule Opportunity**:\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE (\n response.choices LIKE \"*\u003cscript\u003e*\"\n OR response.choices LIKE \"*document.cookie*\"\n OR response.choices LIKE \"*\u003cimg src=x onerror=*\"\n OR response.choices LIKE \"*\u003csvg/onload=*\"\n OR response.choices LIKE \"*javascript:alert*\"\n OR response.choices LIKE \"*\u003ciframe src=# onmouseover=*\"\n OR response.choices LIKE \"*\u003cimg ''\u003e\u003cscript\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=javascript:alert(String.fromCharCode(88,83,83))\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=# onmouseover=alert('xxs')\u003e*\"\n OR response.choices LIKE \"*\u003cIMG onmouseover=alert('xxs')\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=/ onerror=alert(String.fromCharCode(88,83,83))\u003e*\"\n OR response.choices LIKE \"*\u0026#0000106\u0026#0000097\u0026#0000118\u0026#0000097\u0026#0000115\u0026#0000099\u0026#0000114\u0026#0000105\u0026#0000112\u0026#0000116\u0026#0000058\u0026#0000097\u0026#0000108\u0026#0000101\u0026#0000114\u0026#0000116\u0026#0000040\u0026#0000039\u0026#0000088\u0026#0000083\u0026#0000083\u0026#0000039\u0026#0000041\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=\u0026#106;\u0026#97;\u0026#118;\u0026#97;\u0026#115;\u0026#99;\u0026#114;\u0026#105;\u0026#112;\u0026#116;\u0026#58;\u0026#97;\u0026#108;\u0026#101;\u0026#114;\u0026#116;\u0026#40;\u0026#39;\u0026#88;\u0026#83;\u0026#83;\u0026#39;\u0026#41;\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=\\\"jav\u0026#x0A;ascript:alert('XSS');\\\"\u003e*\"\n)\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n| WHERE total_attempts \u003e= 2\n```\n\nThis pseudo query detects potential insecure output handling by identifying LLM responses containing scripting elements or cookie access attempts, which are common in Cross-Site Scripting (XSS) attacks. It is a shell that could be extended by allow or block lists for well-known keywords. \n\n### LLM04 - model DoS\n\n**OWASP Description**: Overloading LLMs with resource-heavy operations can cause service disruptions and increased costs. Reference [here](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM04_ModelDoS.md).\n\n**Example**: An adversary may send complex prompts that consume excessive computational resources. \n\n**Prompt**:\n\n\n**Sample Response**:\n\n\nDetection Rule Opportunity:\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE response.choices LIKE \"*requires*significant*computational*resources*\"\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n| WHERE total_attempts \u003e= 2\n```\n\nThis detection illustrates another simple example of how the LLM response is used to identify potentially abusive behavior. Although this example may not represent a traditional security threat, it could emulate how adversaries can impose costs on victims, either consuming resources or tokens. \n\n**Example 2**: An adversary may send complex prompts that consume excessive computational resources. \n\n**Prompt**:\n\n\n**Sample Response**:\n\n\nAt a glance, this prompt appears to be benign. However, excessive requests and verbose responses in a short time can significantly increase costs. \n\n**Detection Rule Opportunity**:\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 HOUR\n| STATS request_count = COUNT(*), distinct_prompts = COUNT_DISTINCT(request.messages.content) BY connectorId\n| WHERE request_count \u003e 50 AND distinct_prompts \u003e 10\n| SORT request_count DESC\n```\n\nIn the context of example 2, this working query efficiently tracks and analyzes usage patterns by counting all requests and distinct prompt contents for each ```connectorId``` from the ```azure-openai-logs``` over the past hour. If any ```connectorId``` submits over 50 requests with more than ten unique prompts within this timeframe, it indicates a potential misuse pattern like the one described, where an adversary might be modifying queries to probe encryption algorithm details, potentially causing undue load or evading detection systems. The results are then ordered to prioritize the ```connectorIds``` with the highest request counts, which helps quickly identify the most active or suspicious sources.\n\n### LLM06 - sensitive information disclosure\n\n**OWASP Description**: Failure to protect against disclosure of sensitive information in LLM outputs can result in legal consequences or a loss of competitive advantage. Reference [here](https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM06_SensitiveInformationDisclosure.md).\n\n**Example**: An adversary may craft prompts to extract sensitive information embedded in the training data. \n\n**Prompt**:\n\n\n**Sample Response**:\n\n\nDetection Rule Opportunity:\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE request.messages.content LIKE \"*common*passwords*used*in*the*dataset*\" AND malicious\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n BY identified_threats.keyword, analysis.langkit_score\n| WHERE total_attempts \u003e= 2\n```\n\nThis query searches for logs from the last day that contain requests specifically asking about \"common passwords used in the dataset\" to train the model and where such requests are flagged as malicious. It aggregates these logs to count the number of such attempts and identifies distinct users involved, grouped by any identified threats and the language kit score. With some enrichment, the inclusion of ```AND malicious = true``` ensures focus on requests already flagged as potentially harmful, helping to prioritize investigation and response actions.\n\n## Enriching Detection Rules with Security Insights\n\nBy routing LLM requests through a proxy, we can capitalize on specialized security tools to analyze each request for signs of malicious intent. Upon detection, the original request can be enriched with additional metadata indicating the likelihood of malicious content and the specific type of threat it represents. This enriched data is then indexed in Elasticsearch, creating a robust monitoring, alerting, and retrospective analysis dataset. With this enrichment, the LLM detection opportunities from the last section are possible.\n\nWe don’t deep-dive on every tool available, but several open-source tools have emerged to offer varying approaches to analyzing and securing LLM interactions. Some of these tools are backed by machine learning models trained to detect malicious prompts:\n\n - **Rebuff** ([GitHub](https://github.com/protectai/rebuff)): Utilizes machine learning to identify and mitigate attempts at social engineering, phishing, and other malicious activities through LLM interactions. Example usage involves passing request content through Rebuff's analysis engine and tagging requests with a \"malicious\" boolean field based on the findings. \n - **LLM-Guard** ([GitHub](https://github.com/protectai/llm-guard)): Provides a rule-based engine for detecting harmful patterns in LLM requests. LLM-Guard can categorize detected threats based on predefined categories, enriching requests with detailed threat classifications. \n - **LangKit** ([GitHub](https://github.com/whylabs/langkit/tree/main)): A toolkit designed for monitoring and securing LLMs, LangKit can analyze request content for signs of adversarial inputs or unintended model behaviors. It offers hooks for integrating custom analysis functions.\n - **Vigil-LLM** ([GitHub](https://github.com/deadbits/vigil-llm)): Focuses on real-time monitoring and alerting for suspicious LLM requests. Integration into the proxy layer allows for immediate flagging potential security issues, enriching the request data with vigilance scores.\n - **Open-Prompt Injection** ([GitHub](https://github.com/liu00222/Open-Prompt-Injection)): Offers methodologies and tools for detecting prompt injection attacks, allowing for the enrichment of request data with specific indicators of compromise related to prompt injection techniques.\n\n*Note: Most of these tools require additional calls/costs to an external LLM, and would require further infrastructure to threat hunt effectively.*\n\nOne simple example implementation that uses LLM-guard and LangKit might look like this:\n\n``` python\ndef analyze_and_enrich_request(\n prompt: str, response_text: str, error_response: Optional[dict] = None\n) -\u003e dict:\n \"\"\"Analyze the prompt and response text for malicious content and enrich the document.\"\"\"\n\n # LLM Guard analysis\n sanitized_prompt, results_valid_prompt, results_score_prompt = scan_prompt(\n input_scanners, prompt[\"content\"]\n )\n (\n sanitized_response_text,\n results_valid_response,\n results_score_response,\n ) = scan_output(output_scanners, sanitized_prompt, response_text)\n\n # LangKit for additional analysis\n schema = injections.init()\n langkit_result = extract({\"prompt\": prompt[\"content\"]}, schema=schema)\n\n # Initialize identified threats and malicious flag\n identified_threats = []\n\n # Check LLM Guard results for prompt\n if not any(results_valid_prompt.values()):\n identified_threats.append(\"LLM Guard Prompt Invalid\")\n\n # Check LLM Guard results for response\n if not any(results_valid_response.values()):\n identified_threats.append(\"LLM Guard Response Invalid\")\n\n # Check LangKit result for prompt injection\n prompt_injection_score = langkit_result.get(\"prompt.injection\", 0)\n if prompt_injection_score \u003e 0.4: # Adjust threshold as needed\n identified_threats.append(\"LangKit Injection\")\n\n # Identify threats based on LLM Guard scores\n for category, score in results_score_response.items():\n if score \u003e 0.5:\n identified_threats.append(category)\n\n # Combine results and enrich document\n # llm_guard scores map scanner names to float values of risk scores,\n # where 0 is no risk, and 1 is high risk.\n # langkit_score is a float value of the risk score for prompt injection\n # based on known threats.\n enriched_document = {\n \"analysis\": {\n \"llm_guard_prompt_scores\": results_score_prompt,\n \"llm_guard_response_scores\": results_score_response,\n \"langkit_score\": prompt_injection_score,\n },\n \"malicious\": any(identified_threats),\n \"identified_threats\": identified_threats,\n }\n\n # Check if there was an error from OpenAI and enrich the analysis\n if error_response:\n code = error_response.get(\"code\")\n filtered_categories = {\n category: info[\"filtered\"]\n for category, info in error_response.get(\n \"content_filter_result\", {}\n ).items()\n }\n\n enriched_document[\"analysis\"][\"openai\"] = {\n \"code\": code,\n \"filtered_categories\": filtered_categories,\n }\n if code == \"ResponsibleAIPolicyViolation\":\n enriched_document[\"malicious\"] = True\n\n return enriched_document\n```\n\nThis function could be called for each request passing through the proxy, with the returned data being appended to the request document before it's sent to Elasticsearch. The result is a detailed and actionable dataset that captures the raw interactions with the LLM and provides immediate security insights to embed in our detection rules based on the request and response. Going full circle with the prompt injection LLM01 example, the query could be updated to something like this:\n\n``` sql\nFROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE identified_threats.keyword == \"LangKit Injection\" OR analysis.langkit_score \u003e 0.4\n| stats total_attempts = count(*), users = count_distinct(connectorId) by identified_threats.keyword, analysis.langkit_score\n| WHERE users == 1 and total_attempts \u003e= 2\n```\n\nAs you can see, both scoring mechanisms are subjective based on the results returned from the open source prompt analysis tools. This query filters logs from the past day where the identified threat is \"LangKit Injection\" or the LangKit score is above ```0.4```. It then calculates the total attempts and counts the number of unique users (agents) associated with each identified threat category and LangKit score, filtering to include only cases where there's a single user involved (```users == 1```) and the total attempts are two or more (```total_attempts \u003e= 2```).\n\nWith these additional tools, we have a variety of analysis result fields available to improve our detection rules. In these examples, we shipped most of the data as-is for simplicity. However, in a production environment, it's crucial to normalize these fields across all tools and LLM responses to a schema like [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) (ECS). Normalizing data to ECS enhances interoperability between different data sources, simplifies analysis, and streamlines the creation of more effective and cohesive security rules.\n\nIn Part two of this series, we will discuss how we’ve taken a more formal approach to ECS field mapping, and integrations.\n\n## Alternative Options for LLM Application Auditing\n\nWhile using a proxy may be straightforward, other approaches may better suit a production setup; for example: \n\n - Utilizing [application performance monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) \n - Using the OpenTelemetry integration\n - Modifying changes in Kibana directly to audit and trace LLM activity\n\nUnsurprisingly, these approaches have potential limitations like not natively ingesting all the LLM security analysis tool data generated without developing custom logic to support third-party tools.\n\n### Leveraging Elastic APM for In-Depth Application Insights\n\nElastic [APM](https://www.elastic.co/guide/en/observability/current/apm.html) provides an alternative solution for monitoring applications in real-time, essential for detecting performance bottlenecks and identifying problematic requests or queries. By integrating Elastic APM, users gain detailed insights into transaction times, database query performance, external API call efficiency, and more. This comprehensive visibility makes it easier to address and resolve performance issues or errors quickly. Unlike the proxy approach, APM automatically ingests logs into Elastic about your application, providing an opportunity to create security detection rules based on the behaviors seen within your data.\n\n### Utilizing OpenTelemetry for Enhanced Observability\n\nFor applications already employing OpenTelemetry, leveraging its [integration](https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html) with Elastic APM can enhance observability without requiring extensive instrumentation changes. This integration supports capturing a wide array of telemetry data, including traces and metrics, which can be seamlessly sent to the Elastic Stack. This approach allows developers to continue using familiar libraries while benefiting from the robust monitoring capabilities of Elastic. OpenTelemetry’s compatibility across multiple programming languages and its [support through Elastic’s native protocol](https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html) (OTLP) facilitate straightforward data transmission, providing a robust foundation for monitoring distributed systems. Compared to the proxy example, this approach more natively ingests data than maintaining an independent index and logging mechanism to Elastic. \n\n### LLM Auditing with Kibana\n\nLike writing custom logic for your LLM application to audit and ship data, you can test the approach with Elastic’s AI Assistant. If you're comfortable with TypeScript, consider deploying a local Elastic instance using the Kibana [Getting Started Guide](https://www.elastic.co/guide/en/kibana/current/development-getting-started.html). Once set up, navigate to the [Elastic AI Assistant](https://github.com/elastic/kibana/tree/main/x-pack/plugins/elastic_assistant) and configure it to intercept LLM requests and responses for auditing and analysis. Note: This approach primarily tracks Elastic-specific LLM integration compared to using APM and other integrations or a proxy to track third-party applications. It should only be considered for experimentation and exploratory testing purposes. \n\nFortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting ```elastic.apm.active: true```). See the [README](https://github.com/elastic/kibana/blob/main/x-pack/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx) for more details.\n\n## Closing Thoughts\n\nAs we continue with this exploration into integrating security practices within the lifecycle of large language models at Elastic, it's clear that embedding security into LLM workflows can provide a path forward for creating safer and more reliable applications. These contrived examples, drawn from our work during OnWeek, illustrate how someone can proactively detect, alert, and triage malicious activity, leveraging the security solutions that analysts find most intuitive and effective. \n\nIt’s also worth noting that with the example proxy approach, we can incorporate a model to actively detect and prevent requests. Additionally, we can triage the LLM response before sending it back to the user if we’ve identified malicious threats. At this point, we have the flexibility to extend our security protections to cover a variety of defensive approaches. In this case, there is a fine line between security and performance, as each additional check will consume time and impede the natural conversational flow that users would expect.\n\nFeel free to check out the proof-of-concept proxy at [llm-detection-proxy](https://github.com/elastic/llm-detection-proxy) and adapt it to fit your needs!\n\nWe’re always interested in hearing use cases and workflows like these, so as always, reach out to us via [GitHub issues](https://github.com/elastic/detection-rules/issues), chat with us in our [community Slack](http://ela.st/slack), and ask questions in our [Discuss forums](https://discuss.elastic.co/c/security/endpoint-security/80).\n\n*The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.*\n","code":"var Component=(()=\u003e{var h=Object.create;var o=Object.defineProperty;var p=Object.getOwnPropertyDescriptor;var u=Object.getOwnPropertyNames;var m=Object.getPrototypeOf,g=Object.prototype.hasOwnProperty;var f=(n,e)=\u003e()=\u003e(e||n((e={exports:{}}).exports,e),e.exports),y=(n,e)=\u003e{for(var i in e)o(n,i,{get:e[i],enumerable:!0})},a=(n,e,i,r)=\u003e{if(e\u0026\u0026typeof e==\"object\"||typeof e==\"function\")for(let s of u(e))!g.call(n,s)\u0026\u0026s!==i\u0026\u0026o(n,s,{get:()=\u003ee[s],enumerable:!(r=p(e,s))||r.enumerable});return n};var w=(n,e,i)=\u003e(i=n!=null?h(m(n)):{},a(e||!n||!n.__esModule?o(i,\"default\",{value:n,enumerable:!0}):i,n)),b=n=\u003ea(o({},\"__esModule\",{value:!0}),n);var c=f((E,l)=\u003e{l.exports=_jsx_runtime});var _={};y(_,{default:()=\u003eI,frontmatter:()=\u003ev});var t=w(c()),v={title:\"Embedding Security in LLM Workflows: Elastic's Proactive Approach\",slug:\"embedding-security-in-llm-workflows\",date:\"2024-04-25\",subtitle:\"Exploring Elastic's innovative approach for integrating security into the lifecycle of LLMs to safeguard against vulnerabilities featuring Elastic\\u2019s AI Assistant.\",description:\"Dive into Elastic's exploration of embedding security directly within Large Language Models (LLMs). Discover our strategies for detecting and mitigating several of the top OWASP vulnerabilities in LLM applications, ensuring safer and more secure AI-driven applications.\",author:[{slug:\"mika-ayenson\"}],image:\"Security Labs Images 5.jpg\",category:[{slug:\"detection-science\"},{slug:\"machine-learning\"},{slug:\"generative-ai\"}]};function d(n){let e=Object.assign({p:\"p\",a:\"a\",ul:\"ul\",li:\"li\",ol:\"ol\",strong:\"strong\",img:\"img\",h2:\"h2\",h3:\"h3\",pre:\"pre\",code:\"code\",em:\"em\"},n.components);return(0,t.jsxs)(t.Fragment,{children:[(0,t.jsxs)(e.p,{children:[\"We recently concluded one of our quarterly Elastic OnWeek events, which provides a unique week to explore opportunities outside of our regular day-to-day. In line with recent publications from \",(0,t.jsx)(e.a,{href:\"https://owasp.org/www-project-top-10-for-large-language-model-applications/\",rel:\"nofollow\",children:\"OWASP\"}),\" and the \",(0,t.jsx)(e.a,{href:\"https://media.defense.gov/2024/Apr/15/2003439257/-1/-1/0/CSI-DEPLOYING-AI-SYSTEMS-SECURELY.PDF\",rel:\"nofollow\",children:\"NSA AISC\"}),\", we decided to spend some time with the OWASP Top Ten vulnerabilities for LLMs natively in Elastic. In this article, we touch on a few opportunities to detect malicious LLM activity with \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html\",rel:\"nofollow\",children:\"ES|QL\"}),\", namely:\"]}),`\n`,(0,t.jsxs)(e.ul,{children:[`\n`,(0,t.jsx)(e.li,{children:\"LLM01: Prompt Injection\"}),`\n`,(0,t.jsx)(e.li,{children:\"LLM02: Insecure Output Handling\"}),`\n`,(0,t.jsx)(e.li,{children:\"LLM04: Model Denial of Service\"}),`\n`,(0,t.jsx)(e.li,{children:\"LLM06: Sensitive Information Disclosure\"}),`\n`]}),`\n`,(0,t.jsx)(e.p,{children:\"Elastic provides the ability to audit LLM applications for malicious behaviors; we\\u2019ll show you one approach with just four steps:\"}),`\n`,(0,t.jsxs)(e.ol,{children:[`\n`,(0,t.jsx)(e.li,{children:\"Intercepting and analyzing the LLM requests and responses\"}),`\n`,(0,t.jsx)(e.li,{children:\"Enriching data with LLM-specific analysis results\"}),`\n`,(0,t.jsx)(e.li,{children:\"Sending data to Elastic Security\"}),`\n`,(0,t.jsx)(e.li,{children:\"Writing ES|QL detection rules that can later be used to respond\"}),`\n`]}),`\n`,(0,t.jsx)(e.p,{children:\"This approach reflects our ongoing efforts to explore and implement advanced detection strategies, including developing detection rules tailored specifically for LLMs, while keeping pace with emerging generative AI technologies and security challenges. Building on this foundation, last year marked a significant enhancement to our toolkit and overall capability to continue this proactive path forward.\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"Elastic \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/blog/introducing-elastic-ai-assistant\",rel:\"nofollow\",children:\"released\"}),\" the AI Assistant for Security, introducing how the open generative AI sidekick is powered by the \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/platform\",rel:\"nofollow\",children:\"Search AI Platform\"}),\" \\u2014 a collection of relevant tools for developing advanced search applications. Backed by machine learning (ML) and artificial intelligence (AI), this AI Assistant provides powerful pre-built workflows like alert summarization, workflow suggestions, query conversions, and agent integration advice. I highly recommend you read more on Elastic\\u2019s \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/elasticsearch/ai-assistant\",rel:\"nofollow\",children:\"AI Assistant\"}),\" about how the capabilities seamlessly span across Observability and Security.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[\"We can use the AI Assistant\\u2019s capabilities as a third-party LLM application to capture, audit, and analyze requests and responses for convenience and to run experiments. Once data is in an index, writing behavioral detections on it becomes business as usual \\u2014 we can also leverage the entire security detection engine. Even though we\\u2019re proxying the Elastic AI Assistant LLM activity in this experiment, it\\u2019s merely used as a vehicle to demonstrate auditing LLM-based applications. Furthermore, this proxy approach is intended for third-party applications to ship data to \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/security/current/es-overview.html\",rel:\"nofollow\",children:\"Elastic Security\"}),\".\"]}),`\n`,(0,t.jsxs)(e.p,{children:[\"We can introduce security mechanisms into the application's lifecycle by intercepting LLM activity or leveraging observable LLM metrics. It\\u2019s common practice to address prompt-based threats by \",(0,t.jsx)(e.a,{href:\"https://platform.openai.com/docs/guides/safety-best-practices\",rel:\"nofollow\",children:\"implementing various safety tactics\"}),\":\"]}),`\n`,(0,t.jsxs)(e.ol,{children:[`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Clean Inputs\"}),\": Sanitize and validate user inputs before feeding them to the model\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Content Moderation\"}),\": Use OpenAI tools to filter harmful prompts and outputs\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Rate Limits and Monitoring\"}),\": Track usage patterns to detect suspicious activity\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Allow/Blocklists\"}),\": Define acceptable or forbidden inputs for specific applications\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Safe Prompt Engineering\"}),\": Design prebuilt prompts that guide the model towards intended outcomes\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"User Role Management\"}),\": Control user access to prevent unauthorized actions\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Educate End-Users\"}),\": Promote responsible use of the model to mitigate risks\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Red Teaming \u0026 Monitoring\"}),\": Test for vulnerabilities and continuously monitor for unexpected outputs\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"HITL Feedback for Model Training\"}),\": Learn from human-in-the-loop, flagged issues to refine the model over time\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Restrict API Access\"}),\": Limit model access based on specific needs and user verification\"]}),`\n`]}),`\n`,(0,t.jsxs)(e.p,{children:[\"Two powerful features provided by OpenAI, and many other LLM implementers, is the ability to \",(0,t.jsx)(e.a,{href:\"https://platform.openai.com/docs/guides/safety-best-practices/end-user-ids\",rel:\"nofollow\",children:\"submit end-user IDs\"}),\" and check content against a \",(0,t.jsx)(e.a,{href:\"https://platform.openai.com/docs/guides/moderation\",rel:\"nofollow\",children:\"moderation API\"}),\", features that set the bar for LLM safety. Sending hashed IDs along with the original request aids in abuse detection and provides targeted feedback, allowing unique user identification without sending personal information. Alternatively, OpenAI's moderation endpoint helps developers identify potentially harmful content like hate speech, self-harm encouragement, or violence, allowing them to filter such content. It even goes a step further by detecting threats and intent to self-harm.\"]}),`\n`,(0,t.jsx)(e.p,{children:\"Despite all of the recommendations and best practices to protect against malicious prompts, we recognize that there is no single perfect solution. When using capabilities like OpenAI\\u2019s API, some of these threats may be detected by the content filter, which will respond with a usage policy violation notification:\"}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image5.png\",alt:\"Violation notification from OpenAI\",width:\"618\",height:\"106\"})}),`\n`,(0,t.jsx)(e.p,{children:\"This content filtering is beneficial to address many issues; however, it cannot identify further threats in the broader context of the environment, application ecosystem, or other alerts that may appear. The more we can integrate generative AI use cases into our existing protection capabilities, the more control and possibilities we have to address potential threats. Furthermore, even if LLM safeguards are in place to stop rudimentary attacks, we can still use the detection engine to alert and take future remediation actions instead of silently blocking or permitting abuse.\"}),`\n`,(0,t.jsx)(e.h2,{id:\"proxying-llm-requests-and-setup\",children:\"Proxying LLM Requests and Setup\"}),`\n`,(0,t.jsx)(e.p,{children:\"The optimal security solution integrates additional safeguards directly within the LLM application's ecosystem. This allows enriching alerts with the complete context surrounding requests and responses. As requests are sent to the LLM, we can intercept and analyze them for potential malicious activity. If necessary, a response action can be triggered to defer subsequent HTTP calls. Similarly, inspecting the LLM's response can uncover further signs of malicious behavior.\"}),`\n`,(0,t.jsx)(e.p,{children:\"Using a proxy to handle these interactions offers several advantages:\"}),`\n`,(0,t.jsxs)(e.ul,{children:[`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Ease of Integration and Management\"}),\": By managing the new security code within a dedicated proxy application, you avoid embedding complex security logic directly into the main application. This approach minimizes changes needed in the existing application structure, allowing for easier maintenance and clearer separation of security from business logic. The main application must only be reconfigured to route its LLM requests through the proxy.\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Performance and Scalability\"}),\": Placing the proxy on a separate server isolates the security mechanisms and helps distribute the computational load. This can be crucial when scaling up operations or managing performance-intensive tasks, ensuring that the main application's performance remains unaffected by the additional security processing.\"]}),`\n`]}),`\n`,(0,t.jsx)(e.h3,{id:\"quick-start-option-proxy-with-flask\",children:\"Quick Start Option: Proxy with Flask\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"You can proxy incoming and outgoing LLM connections for a faster initial setup. This approach can be generalized for other LLM applications by creating a simple Python-based \",(0,t.jsx)(e.a,{href:\"https://flask.palletsprojects.com/en/3.0.x/\",rel:\"nofollow\",children:\"Flask\"}),\" application. This application would intercept the communication, analyze it for security risks, and log relevant information before forwarding the response.\"]}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image3.png\",alt:\"Approach to Intercept Elastic Request/Responses\",width:\"1440\",height:\"835\"})}),`\n`,(0,t.jsxs)(e.p,{children:[\"Multiple SDKs exist to connect to Elasticsearch and handle OpenAI LLM requests. The provided \",(0,t.jsx)(e.a,{href:\"https://github.com/elastic/llm-detection-proxy\",rel:\"nofollow\",children:\"llm-detection-proxy\"}),\" repo demonstrates the available Elastic and OpenAI clients. This snippet highlights the bulk of the experimental proxy in a single Flask route.\"]}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-python\",children:`@app.route(\"/proxy/openai\", methods=[\"POST\"])\ndef azure_openai_proxy():\n \"\"\"Proxy endpoint for Azure OpenAI requests.\"\"\"\n data = request.get_json()\n messages = data.get(\"messages\", [])\n response_content = \"\"\n error_response = None\n\n try:\n # Forward the request to Azure OpenAI\n response = client.chat.completions.create(model=deployment_name, messages=messages)\n response_content = response.choices[0].message.content # Assuming one choice for simplicity\n choices = response.choices[0].model_dump()\n except openai.BadRequestError as e:\n # If BadRequestError is raised, capture the error details\n error_response = e.response.json().get(\"error\", {}).get(\"innererror\", {})\n response_content = e.response.json().get(\"error\", {}).get(\"message\")\n\n # Structure the response with the error details\n choices = {**error_response.get(\"content_filter_result\", {}),\n \"error\": response_content, \"message\": {\"content\": response_content}}\n\n # Perform additional analysis and create the Elastic document\n additional_analysis = analyze_and_enrich_request(prompt=messages[-1],\n response_text=response_content,\n error_response=error_response)\n log_data = {\"request\": {\"messages\": messages[-1]},\n \"response\": {\"choices\": response_content},\n **additional_analysis}\n\n # Log the last message and response\n log_to_elasticsearch(log_data)\n\n # Calculate token usage\n prompt_tokens = sum(len(message[\"content\"]) for message in messages)\n completion_tokens = len(response_content)\n total_tokens = prompt_tokens + completion_tokens\n\n # Structure and return the response\n return jsonify({\n \"choices\": [choices],\n \"usage\": {\n \"prompt_tokens\": prompt_tokens,\n \"completion_tokens\": completion_tokens,\n \"total_tokens\": total_tokens,\n }\n })\n`})}),`\n`,(0,t.jsxs)(e.p,{children:[\"With the Flask server, you can configure the \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/kibana/current/openai-action-type.html\",rel:\"nofollow\",children:\"OpenAI Kibana Connector\"}),\" to use your proxy.\"]}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image10.png\",alt:\"\",width:\"1360\",height:\"1008\"})}),`\n`,(0,t.jsx)(e.p,{children:\"Since this proxy to your LLM is running locally, credentials and connection information are managed outside of Elastic, and an empty string can be provided in the API key section. Before moving forward, testing your connection is generally a good idea. It is important to consider other security implications if you are considering implementing a proxy solution in a real environment - not something this prototype considered for brevity.\"}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image4.png\",alt:\"Sample screenshot of the AI Assistant operating through the prototype proxy\",width:\"1440\",height:\"662\"})}),`\n`,(0,t.jsxs)(e.p,{children:[\"We can now index our LLM requests and responses and begin to write detections on the available data in the \",(0,t.jsx)(e.code,{children:\"azure-openai-logs\"}),\" index created in this experiment. Optionally, we could preprocess the data using an Elastic \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html\",rel:\"nofollow\",children:\"ingestion pipeline\"}),\", but in this contrived example, we can effectively write detections with the power of ES|QL.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image13.png\",alt:`Sample AzureOpenAI LLM Request/Response Data\nLangsmith Proxy`,width:\"1440\",height:\"581\"}),`\nSample AzureOpenAI LLM Request/Response Data`]}),`\n`,(0,t.jsx)(e.h3,{id:\"langsmith-proxy\",children:\"Langsmith Proxy\"}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsxs)(e.em,{children:[\"Note: The \",(0,t.jsx)(e.a,{href:\"https://docs.smith.langchain.com/proxy/quickstart\",rel:\"nofollow\",children:\"Langsmith Proxy\"}),\" project provides a dockerized proxy for your LLM APIs. While it offers a minimized solution, as of this writing, it lacks native capabilities for incorporating custom security analysis tools or integrating directly with Elastic Security.\"]})}),`\n`,(0,t.jsx)(e.p,{children:\"The LangSmith Proxy is designed to simplify LLM API interaction. It's a sidecar application requiring minimal configuration (e.g., LLM API URL). It enhances performance (caching, streaming) for high-traffic scenarios. It uses NGINX for efficiency and supports optional tracing for detailed LLM interaction tracking. Currently, it works with OpenAI and AzureOpenAI, with future support planned for other LLMs.\"}),`\n`,(0,t.jsx)(e.h2,{id:\"llm-potential-attacks-and-detection-rule-opportunities\",children:\"LLM Potential Attacks and Detection Rule Opportunities\"}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsxs)(e.strong,{children:[\"It\\u2019s important to understand that even though documented lists of protections do not accompany some LLMs, simply trying some of these prompts may be immediately denied or result in banning on whatever platform used to submit the prompt. We recommend experimenting with caution and understand the SLA prior to sending any malicious prompts. Since this exploration leverages OpenAI\\u2019s resources, we recommend following the bugcrowd \",(0,t.jsx)(e.a,{href:\"https://bugcrowd.com/openai\",rel:\"nofollow\",children:\"guidance\"}),\" and sign up for an additional testing account using your @bugcrowdninja.com email address.\"]})}),`\n`,(0,t.jsx)(e.p,{children:\"Here is a list of several plausible examples to illustrate detection opportunities. Each LLM topic includes the OWASP description, an example prompt, a sample document, the detection opportunity, and potential actions users could take if integrating additional security mechanisms in their workflow.\"}),`\n`,(0,t.jsx)(e.p,{children:\"While this list is currently not extensive, Elastic Security Labs is currently undertaking a number of initiatives to ensure future development, and formalization of rules will continue.\"}),`\n`,(0,t.jsx)(e.h3,{id:\"llm01---prompt-injection\",children:\"LLM01 - prompt injection\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"OWASP Description\"}),\": Manipulating LLMs via crafted inputs can lead to unauthorized access, data breaches, and compromised decision-making. Reference \",(0,t.jsx)(e.a,{href:\"https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM01_PromptInjection.md\",rel:\"nofollow\",children:\"here\"}),\".\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example\"}),\": An adversary might try to craft prompts that trick the LLM into executing unintended actions or revealing sensitive information. \",(0,t.jsxs)(e.em,{children:[\"Note: Tools like \",(0,t.jsx)(e.a,{href:\"https://github.com/utkusen/promptmap\",rel:\"nofollow\",children:\"promptmap\"}),\" are available to generate creative prompt injection ideas and automate the testing process.\"]})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image7.png\",alt:\"\",width:\"1432\",height:\"130\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image8.png\",alt:\"\",width:\"1440\",height:\"244\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Detection Rule Opportunity\"}),\": In this example, the LLM responded by refusing to handle database connection strings due to security risks. It emphasizes keeping credentials private and suggests using secure methods like environment variables or vaults to protect them.\"]}),`\n`,(0,t.jsx)(e.p,{children:\"A very brittle but basic indicator-matching query may look like this:\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs |\n WHERE request.messages.content LIKE \"*generate*connection*string*\"\n OR request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm sorry, but I can't assist*\"\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"A slightly more advanced query detects more than two similar attempts within the last day.\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm*sorry,*but*I*can't*assist*\"\n OR response.choices LIKE \"*I*can\\u2019t*process*actual*sensitive*\"\n| stats total_attempts = count(*) by connectorId\n| WHERE total_attempts \u003e= 2\n`})}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.em,{children:\"Note that there are many approaches to detect malicious prompts and protect LLM responses. Relying on these indicators alone is not the best approach; however, we can gradually improve the detection with additional enrichment or numerous response attempts. Furthermore, if we introduce an ID into our documents, we can further enhance our query by aggregating attempts based on the field that correlates to a specific user.\"})}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example 2\"}),\": The \",(0,t.jsx)(e.a,{href:\"https://arxiv.org/abs/2404.01833v1\",rel:\"nofollow\",children:\"Crescendo\"}),\" effect is a realistic jailbreak attack where an adversary gradually manipulates a language model through a series of seemingly innocent inquiries that shift towards asking the model to describe hypothetical scenarios involving the unauthorized access and manipulation of secure systems. By doing so, they aim to extract methods that could potentially bypass the LLM\\u2019s security constraints.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image15.png\",alt:\"\",width:\"874\",height:\"104\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image17.png\",alt:\"\",width:\"1440\",height:\"1398\"})]}),`\n`,(0,t.jsx)(e.p,{children:\"With the additional analysis from OpenAI\\u2019s filtering, we can immediately detect the first occurrence of abuse.\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Detection Rule Opportunity\"}),\":\"]}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n AND (\n request.messages.content LIKE \"*credentials*password*username*\"\n OR response.choices LIKE \"*I'm sorry, but I can't assist*\"\n OR analysis.openai.code == \"ResponsibleAIPolicyViolation\"\n OR malicious\n )\n| STATS total_attempts = COUNT(*) BY connectorId\n| WHERE total_attempts \u003e 1\n| SORT total_attempts DESC\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"However, as you continue to use the Crescendo Effect, we notice that the conversation pivot goes unblocked after the initial content filter by OpenAI. It\\u2019s important to understand that even if tactics like this are difficult to prevent, we still have opportunities to detect.\"}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image6.png\",alt:\"\",width:\"1440\",height:\"280\"})}),`\n`,(0,t.jsx)(e.p,{children:\"Additional analysis tools, like LLM-Guard, detect if the conversation is sensitive, which, in this case, is inaccurate. However, it hints at potential opportunities to track malicious behavior over multiple prompts. Note: We could also take advantage of EQL sequences as an alternative to this ES|QL query to help track behaviors over multiple events.\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n AND (\n request.messages.content LIKE \"*Molotov*\"\n OR analysis.openai.code == \"ResponsibleAIPolicyViolation\"\n OR malicious\n )\n| STATS attempts = count(*), max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive) BY connectorId\n| WHERE attempts \u003e= 1 AND max_sensitivity \u003e 0.5\n| SORT attempts DESC\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"This query detects suspicious behavior related to Molotov Cocktails across multiple events by analyzing sequences of log entries associated with a single user/session (identified by connectorId). The query core filters events based on:\"}),`\n`,(0,t.jsxs)(e.ul,{children:[`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Content Matching\"}),': It searches for mentions of \"Molotov\" in conversation content (',(0,t.jsx)(e.code,{children:'request.messages.content LIKE \"*Molotov*\"'}),\")\"]}),`\n`,(0,t.jsxs)(e.li,{children:[\"**Policy Violations: It identifies attempts blocked by OpenAI's safety filters (\",(0,t.jsx)(e.code,{children:'analysis.openai.code == \"ResponsibleAIPolicyViolation\"'}),\"), indicating the start of potentially suspicious behavior\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Malicious Flag Consideration\"}),\": It includes logs where the system flagged the content as malicious (\",(0,t.jsx)(e.code,{children:\"malicious == true\"}),\"), capturing potentially subtle or varied mentions\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Session-Level Analysis\"}),\": By grouping events by connectorId, it analyzes the complete sequence of attempts within a session. It then calculates the total number of attempts (\",(0,t.jsx)(e.code,{children:\"attempts = count(*)\"}),\") and the highest sensitivity score (\",(0,t.jsx)(e.code,{children:\"max_sensitivity = max(analysis.llm_guard_response_scores.Sensitive)\"}),\") across all attempts in that session\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Flagging High-Risk Sessions\"}),\": It filters sessions with at least one attempt (\",(0,t.jsx)(e.code,{children:\"attempts \u003e= 1\"}),\") and a maximum sensitivity score exceeding 0.5 (\",(0,t.jsx)(e.code,{children:\"max_sensitivity \u003e 0.5\"}),\"). This threshold helps focus on sessions where users persistently discussed or revealed potentially risky content.\"]}),`\n`]}),`\n`,(0,t.jsx)(e.p,{children:\"By analyzing these factors across multiple events within a session, we can start building an approach to detect a pattern of escalating discussions, even if individual events might not be flagged alone.\"}),`\n`,(0,t.jsx)(e.h3,{id:\"llm02---insecure-output-handling\",children:\"LLM02 - insecure output handling\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"OWASP Description\"}),\": Neglecting to validate LLM outputs may lead to downstream security exploits, including code execution that compromises systems and exposes data. Reference \",(0,t.jsx)(e.a,{href:\"https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM02_InsecureOutputHandling.md\",rel:\"nofollow\",children:\"here\"}),\".\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example\"}),\": An adversary may attempt to exploit the LLM to generate outputs that can be used for cross-site scripting (XSS) or other injection attacks.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image9.png\",alt:\"\",width:\"1274\",height:\"108\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image12.png\",alt:\"\",width:\"1428\",height:\"1114\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Detection Rule Opportunity\"}),\":\"]}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE (\n response.choices LIKE \"*\u003cscript\u003e*\"\n OR response.choices LIKE \"*document.cookie*\"\n OR response.choices LIKE \"*\u003cimg src=x onerror=*\"\n OR response.choices LIKE \"*\u003csvg/onload=*\"\n OR response.choices LIKE \"*javascript:alert*\"\n OR response.choices LIKE \"*\u003ciframe src=# onmouseover=*\"\n OR response.choices LIKE \"*\u003cimg ''\u003e\u003cscript\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=javascript:alert(String.fromCharCode(88,83,83))\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=# onmouseover=alert('xxs')\u003e*\"\n OR response.choices LIKE \"*\u003cIMG onmouseover=alert('xxs')\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=/ onerror=alert(String.fromCharCode(88,83,83))\u003e*\"\n OR response.choices LIKE \"*\u0026#0000106\u0026#0000097\u0026#0000118\u0026#0000097\u0026#0000115\u0026#0000099\u0026#0000114\u0026#0000105\u0026#0000112\u0026#0000116\u0026#0000058\u0026#0000097\u0026#0000108\u0026#0000101\u0026#0000114\u0026#0000116\u0026#0000040\u0026#0000039\u0026#0000088\u0026#0000083\u0026#0000083\u0026#0000039\u0026#0000041\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=\u0026#106;\u0026#97;\u0026#118;\u0026#97;\u0026#115;\u0026#99;\u0026#114;\u0026#105;\u0026#112;\u0026#116;\u0026#58;\u0026#97;\u0026#108;\u0026#101;\u0026#114;\u0026#116;\u0026#40;\u0026#39;\u0026#88;\u0026#83;\u0026#83;\u0026#39;\u0026#41;\u003e*\"\n OR response.choices LIKE \"*\u003cIMG SRC=\\\\\"jav\u0026#x0A;ascript:alert('XSS');\\\\\"\u003e*\"\n)\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n| WHERE total_attempts \u003e= 2\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"This pseudo query detects potential insecure output handling by identifying LLM responses containing scripting elements or cookie access attempts, which are common in Cross-Site Scripting (XSS) attacks. It is a shell that could be extended by allow or block lists for well-known keywords.\"}),`\n`,(0,t.jsx)(e.h3,{id:\"llm04---model-dos\",children:\"LLM04 - model DoS\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"OWASP Description\"}),\": Overloading LLMs with resource-heavy operations can cause service disruptions and increased costs. Reference \",(0,t.jsx)(e.a,{href:\"https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM04_ModelDoS.md\",rel:\"nofollow\",children:\"here\"}),\".\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example\"}),\": An adversary may send complex prompts that consume excessive computational resources.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image2.png\",alt:\"\",width:\"1402\",height:\"110\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image18.png\",alt:\"\",width:\"1414\",height:\"654\"})]}),`\n`,(0,t.jsx)(e.p,{children:\"Detection Rule Opportunity:\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE response.choices LIKE \"*requires*significant*computational*resources*\"\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n| WHERE total_attempts \u003e= 2\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"This detection illustrates another simple example of how the LLM response is used to identify potentially abusive behavior. Although this example may not represent a traditional security threat, it could emulate how adversaries can impose costs on victims, either consuming resources or tokens.\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example 2\"}),\": An adversary may send complex prompts that consume excessive computational resources.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image16.png\",alt:\"\",width:\"1094\",height:\"104\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image14.png\",alt:\"\",width:\"1406\",height:\"860\"})]}),`\n`,(0,t.jsx)(e.p,{children:\"At a glance, this prompt appears to be benign. However, excessive requests and verbose responses in a short time can significantly increase costs.\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Detection Rule Opportunity\"}),\":\"]}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 HOUR\n| STATS request_count = COUNT(*), distinct_prompts = COUNT_DISTINCT(request.messages.content) BY connectorId\n| WHERE request_count \u003e 50 AND distinct_prompts \u003e 10\n| SORT request_count DESC\n`})}),`\n`,(0,t.jsxs)(e.p,{children:[\"In the context of example 2, this working query efficiently tracks and analyzes usage patterns by counting all requests and distinct prompt contents for each \",(0,t.jsx)(e.code,{children:\"connectorId\"}),\" from the \",(0,t.jsx)(e.code,{children:\"azure-openai-logs\"}),\" over the past hour. If any \",(0,t.jsx)(e.code,{children:\"connectorId\"}),\" submits over 50 requests with more than ten unique prompts within this timeframe, it indicates a potential misuse pattern like the one described, where an adversary might be modifying queries to probe encryption algorithm details, potentially causing undue load or evading detection systems. The results are then ordered to prioritize the \",(0,t.jsx)(e.code,{children:\"connectorIds\"}),\" with the highest request counts, which helps quickly identify the most active or suspicious sources.\"]}),`\n`,(0,t.jsx)(e.h3,{id:\"llm06---sensitive-information-disclosure\",children:\"LLM06 - sensitive information disclosure\"}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"OWASP Description\"}),\": Failure to protect against disclosure of sensitive information in LLM outputs can result in legal consequences or a loss of competitive advantage. Reference \",(0,t.jsx)(e.a,{href:\"https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM06_SensitiveInformationDisclosure.md\",rel:\"nofollow\",children:\"here\"}),\".\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Example\"}),\": An adversary may craft prompts to extract sensitive information embedded in the training data.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Prompt\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image1.png\",alt:\"\",width:\"1344\",height:\"110\"})]}),`\n`,(0,t.jsxs)(e.p,{children:[(0,t.jsx)(e.strong,{children:\"Sample Response\"}),`:\n`,(0,t.jsx)(e.img,{src:\"/assets/images/embedding-security-in-llm-workflows/image19.png\",alt:\"\",width:\"1410\",height:\"594\"})]}),`\n`,(0,t.jsx)(e.p,{children:\"Detection Rule Opportunity:\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE request.messages.content LIKE \"*common*passwords*used*in*the*dataset*\" AND malicious\n| stats total_attempts = COUNT(*), users = COUNT_DISTINCT(connectorId)\n BY identified_threats.keyword, analysis.langkit_score\n| WHERE total_attempts \u003e= 2\n`})}),`\n`,(0,t.jsxs)(e.p,{children:['This query searches for logs from the last day that contain requests specifically asking about \"common passwords used in the dataset\" to train the model and where such requests are flagged as malicious. It aggregates these logs to count the number of such attempts and identifies distinct users involved, grouped by any identified threats and the language kit score. With some enrichment, the inclusion of ',(0,t.jsx)(e.code,{children:\"AND malicious = true\"}),\" ensures focus on requests already flagged as potentially harmful, helping to prioritize investigation and response actions.\"]}),`\n`,(0,t.jsx)(e.h2,{id:\"enriching-detection-rules-with-security-insights\",children:\"Enriching Detection Rules with Security Insights\"}),`\n`,(0,t.jsx)(e.p,{children:\"By routing LLM requests through a proxy, we can capitalize on specialized security tools to analyze each request for signs of malicious intent. Upon detection, the original request can be enriched with additional metadata indicating the likelihood of malicious content and the specific type of threat it represents. This enriched data is then indexed in Elasticsearch, creating a robust monitoring, alerting, and retrospective analysis dataset. With this enrichment, the LLM detection opportunities from the last section are possible.\"}),`\n`,(0,t.jsx)(e.p,{children:\"We don\\u2019t deep-dive on every tool available, but several open-source tools have emerged to offer varying approaches to analyzing and securing LLM interactions. Some of these tools are backed by machine learning models trained to detect malicious prompts:\"}),`\n`,(0,t.jsxs)(e.ul,{children:[`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Rebuff\"}),\" (\",(0,t.jsx)(e.a,{href:\"https://github.com/protectai/rebuff\",rel:\"nofollow\",children:\"GitHub\"}),`): Utilizes machine learning to identify and mitigate attempts at social engineering, phishing, and other malicious activities through LLM interactions. Example usage involves passing request content through Rebuff's analysis engine and tagging requests with a \"malicious\" boolean field based on the findings.`]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"LLM-Guard\"}),\" (\",(0,t.jsx)(e.a,{href:\"https://github.com/protectai/llm-guard\",rel:\"nofollow\",children:\"GitHub\"}),\"): Provides a rule-based engine for detecting harmful patterns in LLM requests. LLM-Guard can categorize detected threats based on predefined categories, enriching requests with detailed threat classifications.\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"LangKit\"}),\" (\",(0,t.jsx)(e.a,{href:\"https://github.com/whylabs/langkit/tree/main\",rel:\"nofollow\",children:\"GitHub\"}),\"): A toolkit designed for monitoring and securing LLMs, LangKit can analyze request content for signs of adversarial inputs or unintended model behaviors. It offers hooks for integrating custom analysis functions.\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Vigil-LLM\"}),\" (\",(0,t.jsx)(e.a,{href:\"https://github.com/deadbits/vigil-llm\",rel:\"nofollow\",children:\"GitHub\"}),\"): Focuses on real-time monitoring and alerting for suspicious LLM requests. Integration into the proxy layer allows for immediate flagging potential security issues, enriching the request data with vigilance scores.\"]}),`\n`,(0,t.jsxs)(e.li,{children:[(0,t.jsx)(e.strong,{children:\"Open-Prompt Injection\"}),\" (\",(0,t.jsx)(e.a,{href:\"https://github.com/liu00222/Open-Prompt-Injection\",rel:\"nofollow\",children:\"GitHub\"}),\"): Offers methodologies and tools for detecting prompt injection attacks, allowing for the enrichment of request data with specific indicators of compromise related to prompt injection techniques.\"]}),`\n`]}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.em,{children:\"Note: Most of these tools require additional calls/costs to an external LLM, and would require further infrastructure to threat hunt effectively.\"})}),`\n`,(0,t.jsx)(e.p,{children:\"One simple example implementation that uses LLM-guard and LangKit might look like this:\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-python\",children:`def analyze_and_enrich_request(\n prompt: str, response_text: str, error_response: Optional[dict] = None\n) -\u003e dict:\n \"\"\"Analyze the prompt and response text for malicious content and enrich the document.\"\"\"\n\n # LLM Guard analysis\n sanitized_prompt, results_valid_prompt, results_score_prompt = scan_prompt(\n input_scanners, prompt[\"content\"]\n )\n (\n sanitized_response_text,\n results_valid_response,\n results_score_response,\n ) = scan_output(output_scanners, sanitized_prompt, response_text)\n\n # LangKit for additional analysis\n schema = injections.init()\n langkit_result = extract({\"prompt\": prompt[\"content\"]}, schema=schema)\n\n # Initialize identified threats and malicious flag\n identified_threats = []\n\n # Check LLM Guard results for prompt\n if not any(results_valid_prompt.values()):\n identified_threats.append(\"LLM Guard Prompt Invalid\")\n\n # Check LLM Guard results for response\n if not any(results_valid_response.values()):\n identified_threats.append(\"LLM Guard Response Invalid\")\n\n # Check LangKit result for prompt injection\n prompt_injection_score = langkit_result.get(\"prompt.injection\", 0)\n if prompt_injection_score \u003e 0.4: # Adjust threshold as needed\n identified_threats.append(\"LangKit Injection\")\n\n # Identify threats based on LLM Guard scores\n for category, score in results_score_response.items():\n if score \u003e 0.5:\n identified_threats.append(category)\n\n # Combine results and enrich document\n # llm_guard scores map scanner names to float values of risk scores,\n # where 0 is no risk, and 1 is high risk.\n # langkit_score is a float value of the risk score for prompt injection\n # based on known threats.\n enriched_document = {\n \"analysis\": {\n \"llm_guard_prompt_scores\": results_score_prompt,\n \"llm_guard_response_scores\": results_score_response,\n \"langkit_score\": prompt_injection_score,\n },\n \"malicious\": any(identified_threats),\n \"identified_threats\": identified_threats,\n }\n\n # Check if there was an error from OpenAI and enrich the analysis\n if error_response:\n code = error_response.get(\"code\")\n filtered_categories = {\n category: info[\"filtered\"]\n for category, info in error_response.get(\n \"content_filter_result\", {}\n ).items()\n }\n\n enriched_document[\"analysis\"][\"openai\"] = {\n \"code\": code,\n \"filtered_categories\": filtered_categories,\n }\n if code == \"ResponsibleAIPolicyViolation\":\n enriched_document[\"malicious\"] = True\n\n return enriched_document\n`})}),`\n`,(0,t.jsx)(e.p,{children:\"This function could be called for each request passing through the proxy, with the returned data being appended to the request document before it's sent to Elasticsearch. The result is a detailed and actionable dataset that captures the raw interactions with the LLM and provides immediate security insights to embed in our detection rules based on the request and response. Going full circle with the prompt injection LLM01 example, the query could be updated to something like this:\"}),`\n`,(0,t.jsx)(e.pre,{children:(0,t.jsx)(e.code,{className:\"language-sql\",children:`FROM azure-openai-logs\n| WHERE @timestamp \u003e NOW() - 1 DAY\n| WHERE identified_threats.keyword == \"LangKit Injection\" OR analysis.langkit_score \u003e 0.4\n| stats total_attempts = count(*), users = count_distinct(connectorId) by identified_threats.keyword, analysis.langkit_score\n| WHERE users == 1 and total_attempts \u003e= 2\n`})}),`\n`,(0,t.jsxs)(e.p,{children:['As you can see, both scoring mechanisms are subjective based on the results returned from the open source prompt analysis tools. This query filters logs from the past day where the identified threat is \"LangKit Injection\" or the LangKit score is above ',(0,t.jsx)(e.code,{children:\"0.4\"}),\". It then calculates the total attempts and counts the number of unique users (agents) associated with each identified threat category and LangKit score, filtering to include only cases where there's a single user involved (\",(0,t.jsx)(e.code,{children:\"users == 1\"}),\") and the total attempts are two or more (\",(0,t.jsx)(e.code,{children:\"total_attempts \u003e= 2\"}),\").\"]}),`\n`,(0,t.jsxs)(e.p,{children:[\"With these additional tools, we have a variety of analysis result fields available to improve our detection rules. In these examples, we shipped most of the data as-is for simplicity. However, in a production environment, it's crucial to normalize these fields across all tools and LLM responses to a schema like \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/ecs/current/ecs-reference.html\",rel:\"nofollow\",children:\"Elastic Common Schema\"}),\" (ECS). Normalizing data to ECS enhances interoperability between different data sources, simplifies analysis, and streamlines the creation of more effective and cohesive security rules.\"]}),`\n`,(0,t.jsx)(e.p,{children:\"In Part two of this series, we will discuss how we\\u2019ve taken a more formal approach to ECS field mapping, and integrations.\"}),`\n`,(0,t.jsx)(e.h2,{id:\"alternative-options-for-llm-application-auditing\",children:\"Alternative Options for LLM Application Auditing\"}),`\n`,(0,t.jsx)(e.p,{children:\"While using a proxy may be straightforward, other approaches may better suit a production setup; for example:\"}),`\n`,(0,t.jsxs)(e.ul,{children:[`\n`,(0,t.jsxs)(e.li,{children:[\"Utilizing \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/observability/application-performance-monitoring\",rel:\"nofollow\",children:\"application performance monitoring\"}),\" (APM)\"]}),`\n`,(0,t.jsx)(e.li,{children:\"Using the OpenTelemetry integration\"}),`\n`,(0,t.jsx)(e.li,{children:\"Modifying changes in Kibana directly to audit and trace LLM activity\"}),`\n`]}),`\n`,(0,t.jsx)(e.p,{children:\"Unsurprisingly, these approaches have potential limitations like not natively ingesting all the LLM security analysis tool data generated without developing custom logic to support third-party tools.\"}),`\n`,(0,t.jsx)(e.h3,{id:\"leveraging-elastic-apm-for-in-depth-application-insights\",children:\"Leveraging Elastic APM for In-Depth Application Insights\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"Elastic \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/observability/current/apm.html\",rel:\"nofollow\",children:\"APM\"}),\" provides an alternative solution for monitoring applications in real-time, essential for detecting performance bottlenecks and identifying problematic requests or queries. By integrating Elastic APM, users gain detailed insights into transaction times, database query performance, external API call efficiency, and more. This comprehensive visibility makes it easier to address and resolve performance issues or errors quickly. Unlike the proxy approach, APM automatically ingests logs into Elastic about your application, providing an opportunity to create security detection rules based on the behaviors seen within your data.\"]}),`\n`,(0,t.jsx)(e.h3,{id:\"utilizing-opentelemetry-for-enhanced-observability\",children:\"Utilizing OpenTelemetry for Enhanced Observability\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"For applications already employing OpenTelemetry, leveraging its \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html\",rel:\"nofollow\",children:\"integration\"}),\" with Elastic APM can enhance observability without requiring extensive instrumentation changes. This integration supports capturing a wide array of telemetry data, including traces and metrics, which can be seamlessly sent to the Elastic Stack. This approach allows developers to continue using familiar libraries while benefiting from the robust monitoring capabilities of Elastic. OpenTelemetry\\u2019s compatibility across multiple programming languages and its \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html\",rel:\"nofollow\",children:\"support through Elastic\\u2019s native protocol\"}),\" (OTLP) facilitate straightforward data transmission, providing a robust foundation for monitoring distributed systems. Compared to the proxy example, this approach more natively ingests data than maintaining an independent index and logging mechanism to Elastic.\"]}),`\n`,(0,t.jsx)(e.h3,{id:\"llm-auditing-with-kibana\",children:\"LLM Auditing with Kibana\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"Like writing custom logic for your LLM application to audit and ship data, you can test the approach with Elastic\\u2019s AI Assistant. If you're comfortable with TypeScript, consider deploying a local Elastic instance using the Kibana \",(0,t.jsx)(e.a,{href:\"https://www.elastic.co/guide/en/kibana/current/development-getting-started.html\",rel:\"nofollow\",children:\"Getting Started Guide\"}),\". Once set up, navigate to the \",(0,t.jsx)(e.a,{href:\"https://github.com/elastic/kibana/tree/main/x-pack/plugins/elastic_assistant\",rel:\"nofollow\",children:\"Elastic AI Assistant\"}),\" and configure it to intercept LLM requests and responses for auditing and analysis. Note: This approach primarily tracks Elastic-specific LLM integration compared to using APM and other integrations or a proxy to track third-party applications. It should only be considered for experimentation and exploratory testing purposes.\"]}),`\n`,(0,t.jsxs)(e.p,{children:[\"Fortunately, Kibana is already instrumented with APM, so if you configure an APM server, you will automatically start ingesting logs from this source (by setting \",(0,t.jsx)(e.code,{children:\"elastic.apm.active: true\"}),\"). See the \",(0,t.jsx)(e.a,{href:\"https://github.com/elastic/kibana/blob/main/x-pack/plugins/elastic_assistant/server/lib/langchain/tracers/README.mdx\",rel:\"nofollow\",children:\"README\"}),\" for more details.\"]}),`\n`,(0,t.jsx)(e.h2,{id:\"closing-thoughts\",children:\"Closing Thoughts\"}),`\n`,(0,t.jsx)(e.p,{children:\"As we continue with this exploration into integrating security practices within the lifecycle of large language models at Elastic, it's clear that embedding security into LLM workflows can provide a path forward for creating safer and more reliable applications. These contrived examples, drawn from our work during OnWeek, illustrate how someone can proactively detect, alert, and triage malicious activity, leveraging the security solutions that analysts find most intuitive and effective.\"}),`\n`,(0,t.jsx)(e.p,{children:\"It\\u2019s also worth noting that with the example proxy approach, we can incorporate a model to actively detect and prevent requests. Additionally, we can triage the LLM response before sending it back to the user if we\\u2019ve identified malicious threats. At this point, we have the flexibility to extend our security protections to cover a variety of defensive approaches. In this case, there is a fine line between security and performance, as each additional check will consume time and impede the natural conversational flow that users would expect.\"}),`\n`,(0,t.jsxs)(e.p,{children:[\"Feel free to check out the proof-of-concept proxy at \",(0,t.jsx)(e.a,{href:\"https://github.com/elastic/llm-detection-proxy\",rel:\"nofollow\",children:\"llm-detection-proxy\"}),\" and adapt it to fit your needs!\"]}),`\n`,(0,t.jsxs)(e.p,{children:[\"We\\u2019re always interested in hearing use cases and workflows like these, so as always, reach out to us via \",(0,t.jsx)(e.a,{href:\"https://github.com/elastic/detection-rules/issues\",rel:\"nofollow\",children:\"GitHub issues\"}),\", chat with us in our \",(0,t.jsx)(e.a,{href:\"http://ela.st/slack\",rel:\"nofollow\",children:\"community Slack\"}),\", and ask questions in our \",(0,t.jsx)(e.a,{href:\"https://discuss.elastic.co/c/security/endpoint-security/80\",rel:\"nofollow\",children:\"Discuss forums\"}),\".\"]}),`\n`,(0,t.jsx)(e.p,{children:(0,t.jsx)(e.em,{children:\"The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.\"})})]})}function L(n={}){let{wrapper:e}=n.components||{};return e?(0,t.jsx)(e,Object.assign({},n,{children:(0,t.jsx)(d,n)})):d(n)}var I=L;return b(_);})();\n;return Component;"},"_id":"articles/embedding-security-in-llm-workflows.mdx","_raw":{"sourceFilePath":"articles/embedding-security-in-llm-workflows.mdx","sourceFileName":"embedding-security-in-llm-workflows.mdx","sourceFileDir":"articles","contentType":"mdx","flattenedPath":"articles/embedding-security-in-llm-workflows"},"type":"Article","imageUrl":"/assets/images/embedding-security-in-llm-workflows/Security Labs Images 5.jpg","readingTime":"28 min read","series":"","url":"/embedding-security-in-llm-workflows","headings":[{"level":2,"title":"Proxying LLM Requests and Setup","href":"#proxying-llm-requests-and-setup"},{"level":3,"title":"Quick Start Option: Proxy with Flask","href":"#quick-start-option-proxy-with-flask"},{"level":3,"title":"Langsmith Proxy","href":"#langsmith-proxy"},{"level":2,"title":"LLM Potential Attacks and Detection Rule Opportunities","href":"#llm-potential-attacks-and-detection-rule-opportunities"},{"level":3,"title":"LLM01 - prompt injection","href":"#llm01---prompt-injection"},{"level":3,"title":"LLM02 - insecure output handling","href":"#llm02---insecure-output-handling"},{"level":3,"title":"LLM04 - model DoS","href":"#llm04---model-dos"},{"level":3,"title":"LLM06 - sensitive information disclosure","href":"#llm06---sensitive-information-disclosure"},{"level":2,"title":"Enriching Detection Rules with Security Insights","href":"#enriching-detection-rules-with-security-insights"},{"level":2,"title":"Alternative Options for LLM Application Auditing","href":"#alternative-options-for-llm-application-auditing"},{"level":3,"title":"Leveraging Elastic APM for In-Depth Application Insights","href":"#leveraging-elastic-apm-for-in-depth-application-insights"},{"level":3,"title":"Utilizing OpenTelemetry for Enhanced Observability","href":"#utilizing-opentelemetry-for-enhanced-observability"},{"level":3,"title":"LLM Auditing with Kibana","href":"#llm-auditing-with-kibana"},{"level":2,"title":"Closing Thoughts","href":"#closing-thoughts"}],"author":[{"title":"Mika Ayenson, PhD","slug":"mika-ayenson","body":{"raw":"","code":"var Component=(()=\u003e{var x=Object.create;var a=Object.defineProperty;var f=Object.getOwnPropertyDescriptor;var _=Object.getOwnPropertyNames;var g=Object.getPrototypeOf,j=Object.prototype.hasOwnProperty;var l=(t,n)=\u003e()=\u003e(n||t((n={exports:{}}).exports,n),n.exports),M=(t,n)=\u003e{for(var e in n)a(t,e,{get:n[e],enumerable:!0})},i=(t,n,e,s)=\u003e{if(n\u0026\u0026typeof n==\"object\"||typeof n==\"function\")for(let o of _(n))!j.call(t,o)\u0026\u0026o!==e\u0026\u0026a(t,o,{get:()=\u003en[o],enumerable:!(s=f(n,o))||s.enumerable});return t};var d=(t,n,e)=\u003e(e=t!=null?x(g(t)):{},i(n||!t||!t.__esModule?a(e,\"default\",{value:t,enumerable:!0}):e,t)),p=t=\u003ei(a({},\"__esModule\",{value:!0}),t);var m=l((F,c)=\u003e{c.exports=_jsx_runtime});var k={};M(k,{default:()=\u003eh,frontmatter:()=\u003ey});var r=d(m()),y={title:\"Mika Ayenson, PhD\",slug:\"mika-ayenson\"};function u(t){return(0,r.jsx)(r.Fragment,{})}function D(t={}){let{wrapper:n}=t.components||{};return n?(0,r.jsx)(n,Object.assign({},t,{children:(0,r.jsx)(u,t)})):u(t)}var h=D;return p(k);})();\n;return Component;"},"_id":"authors/mika-ayenson.mdx","_raw":{"sourceFilePath":"authors/mika-ayenson.mdx","sourceFileName":"mika-ayenson.mdx","sourceFileDir":"authors","contentType":"mdx","flattenedPath":"authors/mika-ayenson"},"type":"Author","imageUrl":"","url":"/authors/mika-ayenson"}],"category":[{"title":"Detection science","slug":"detection-science","body":{"raw":"","code":"var Component=(()=\u003e{var x=Object.create;var c=Object.defineProperty;var f=Object.getOwnPropertyDescriptor;var _=Object.getOwnPropertyNames;var d=Object.getPrototypeOf,g=Object.prototype.hasOwnProperty;var j=(t,e)=\u003e()=\u003e(e||t((e={exports:{}}).exports,e),e.exports),l=(t,e)=\u003e{for(var n in e)c(t,n,{get:e[n],enumerable:!0})},i=(t,e,n,s)=\u003e{if(e\u0026\u0026typeof e==\"object\"||typeof e==\"function\")for(let o of _(e))!g.call(t,o)\u0026\u0026o!==n\u0026\u0026c(t,o,{get:()=\u003ee[o],enumerable:!(s=f(e,o))||s.enumerable});return t};var p=(t,e,n)=\u003e(n=t!=null?x(d(t)):{},i(e||!t||!t.__esModule?c(n,\"default\",{value:t,enumerable:!0}):n,t)),D=t=\u003ei(c({},\"__esModule\",{value:!0}),t);var u=j((h,a)=\u003e{a.exports=_jsx_runtime});var X={};l(X,{default:()=\u003eF,frontmatter:()=\u003eM});var r=p(u()),M={title:\"Detection science\",slug:\"detection-science\"};function m(t){return(0,r.jsx)(r.Fragment,{})}function C(t={}){let{wrapper:e}=t.components||{};return e?(0,r.jsx)(e,Object.assign({},t,{children:(0,r.jsx)(m,t)})):m(t)}var F=C;return D(X);})();\n;return Component;"},"_id":"categories/detection-science.mdx","_raw":{"sourceFilePath":"categories/detection-science.mdx","sourceFileName":"detection-science.mdx","sourceFileDir":"categories","contentType":"mdx","flattenedPath":"categories/detection-science"},"type":"Category","url":"/categories/detection-science"},{"title":"Machine learning","slug":"machine-learning","body":{"raw":"","code":"var Component=(()=\u003e{var x=Object.create;var o=Object.defineProperty;var g=Object.getOwnPropertyDescriptor;var l=Object.getOwnPropertyNames;var f=Object.getPrototypeOf,_=Object.prototype.hasOwnProperty;var j=(t,n)=\u003e()=\u003e(n||t((n={exports:{}}).exports,n),n.exports),M=(t,n)=\u003e{for(var e in n)o(t,e,{get:n[e],enumerable:!0})},c=(t,n,e,i)=\u003e{if(n\u0026\u0026typeof n==\"object\"||typeof n==\"function\")for(let a of l(n))!_.call(t,a)\u0026\u0026a!==e\u0026\u0026o(t,a,{get:()=\u003en[a],enumerable:!(i=g(n,a))||i.enumerable});return t};var d=(t,n,e)=\u003e(e=t!=null?x(f(t)):{},c(n||!t||!t.__esModule?o(e,\"default\",{value:t,enumerable:!0}):e,t)),h=t=\u003ec(o({},\"__esModule\",{value:!0}),t);var m=j((b,s)=\u003e{s.exports=_jsx_runtime});var F={};M(F,{default:()=\u003eD,frontmatter:()=\u003ep});var r=d(m()),p={title:\"Machine learning\",slug:\"machine-learning\"};function u(t){return(0,r.jsx)(r.Fragment,{})}function C(t={}){let{wrapper:n}=t.components||{};return n?(0,r.jsx)(n,Object.assign({},t,{children:(0,r.jsx)(u,t)})):u(t)}var D=C;return h(F);})();\n;return Component;"},"_id":"categories/machine-learning.mdx","_raw":{"sourceFilePath":"categories/machine-learning.mdx","sourceFileName":"machine-learning.mdx","sourceFileDir":"categories","contentType":"mdx","flattenedPath":"categories/machine-learning"},"type":"Category","url":"/categories/machine-learning"},{"title":"Generative AI","slug":"generative-ai","body":{"raw":"","code":"var Component=(()=\u003e{var x=Object.create;var o=Object.defineProperty;var f=Object.getOwnPropertyDescriptor;var g=Object.getOwnPropertyNames;var _=Object.getPrototypeOf,j=Object.prototype.hasOwnProperty;var l=(t,e)=\u003e()=\u003e(e||t((e={exports:{}}).exports,e),e.exports),d=(t,e)=\u003e{for(var n in e)o(t,n,{get:e[n],enumerable:!0})},s=(t,e,n,i)=\u003e{if(e\u0026\u0026typeof e==\"object\"||typeof e==\"function\")for(let a of g(e))!j.call(t,a)\u0026\u0026a!==n\u0026\u0026o(t,a,{get:()=\u003ee[a],enumerable:!(i=f(e,a))||i.enumerable});return t};var p=(t,e,n)=\u003e(n=t!=null?x(_(t)):{},s(e||!t||!t.__esModule?o(n,\"default\",{value:t,enumerable:!0}):n,t)),M=t=\u003es(o({},\"__esModule\",{value:!0}),t);var u=l((b,c)=\u003e{c.exports=_jsx_runtime});var F={};d(F,{default:()=\u003eD,frontmatter:()=\u003ev});var r=p(u()),v={title:\"Generative AI\",slug:\"generative-ai\"};function m(t){return(0,r.jsx)(r.Fragment,{})}function C(t={}){let{wrapper:e}=t.components||{};return e?(0,r.jsx)(e,Object.assign({},t,{children:(0,r.jsx)(m,t)})):m(t)}var D=C;return M(F);})();\n;return Component;"},"_id":"categories/generative-ai.mdx","_raw":{"sourceFilePath":"categories/generative-ai.mdx","sourceFileName":"generative-ai.mdx","sourceFileDir":"categories","contentType":"mdx","flattenedPath":"categories/generative-ai"},"type":"Category","url":"/categories/generative-ai"}]},"seriesArticles":null},"__N_SSG":true},"page":"/[slug]","query":{"slug":"embedding-security-in-llm-workflows"},"buildId":"Zi9yjsbrUOekNuqEl_gO0","assetPrefix":"/security-labs","isFallback":false,"gsp":true,"scriptLoader":[]}</script></body></html>