Skip to content

Latest commit

 

History

History
836 lines (703 loc) · 142 KB

rules.md

File metadata and controls

836 lines (703 loc) · 142 KB

Rules, checks, and coverage limits

This page is the reference behind the README. The detection-method map links every layer to its implementation, including checks outside the rules directory. This reference covers the 89 YARA rules shipped in rules/local/, the structural checks the scanner runs without YARA, and the measured limits. Each table gives the indicators a rule requires and its finding severity. verdict_on_match in the rule sets that severity: block, advisory, or note. A note is a weak signal with nothing else behind it: six YARA rules (the three override-phrase rules and the three lone-Unicode-control rules) and five checks (heuristic_invisible_joiner, heuristic_pdf_OpenAction_goto, heuristic_pdf_text_unmapped, heuristic_pdf_content_unread, and agent_encoded_instruction) report one. Of the 89 rules, 77 are advisory, 6 block, and 6 are notes. Under agent-ingest, an advisory or note finding holds the file as block. Under agent-document, an advisory finding holds the file and a note is only recorded, so a notes-only document can pass while any advisory or blocking finding still holds it. Quotation alone is not an exemption. Document callers read a notes-only agent-document record (the read rule), so moving a rule to note widens what an agent may read: treat that as an integration change, not only a precision change.

A match is a pattern that needs review, not proof of malice. Security write-ups, red-team notes, and administration scripts can match the advisory rules on purpose. The near-miss pairs in check_rule_pairs.py record each rule's benign boundary.

Agent-ingestion rules

These rules target content an AI agent is about to read: messages, retrieved documents, tool results, and repository files.

Rule ID What it matches Finding severity
agent_prompt_injection_phrases Phrases that try to override an agent's prompt: ignore, disregard, or forget the previous instructions, the instructions above, or your instructions; do not follow, stop following, or override the previous instructions; reveal your system prompt Note
agent_prompt_injection_multilingual Override phrases in Spanish, Portuguese, French, German, Italian, Dutch, Polish, Turkish, Indonesian, Vietnamese, Chinese, Japanese, Korean, Russian (including uppercase Cyrillic through text-folded), Arabic, and Hindi Note
agent_role_spoof_action A claim of system authority, an all-caps authority header line, or a developer claim, followed by an override, disclosure, or mode-switch request; "this is the system administrator" (or developer, operator) ending its sentence, then at once an order to override, disable, bypass, or ignore your safety, guardrails, content filters, restrictions, or instructions; an OVERRIDE header (SYSTEM OVERRIDE:, [ADMIN OVERRIDE]) that voids the previous rules or instructions or announces new ones on the same line Advisory
agent_fake_chat_marker A fake system or assistant chat marker (ChatML, or the assistant turn opener of Phi, Gemma, Llama, harmony, Cohere, DeepSeek, Granite, Kimi, or Inkling) with an instruction verb, or a </tool_result> or </tool_response> close tag that is followed within 80 bytes by an instruction verb and has no open tag of its own kind anywhere in the unit Advisory
agent_tool_output_override Tool-result text that asks a command tool to bypass controls Advisory
agent_persistent_instruction A request to persist an approval or safety bypass in agent guidance or in future (or all future) sessions, tasks, or chats. In the future-sessions form, a bypass that never, not, or don't negates does not count. The guidance-file form (append, write, save, or store to AGENTS.md, CLAUDE.md, MEMORY.md, or memory) has no such exemption today: "Append to CLAUDE.md: do not bypass confirmation" matches Advisory
agent_secret_transfer_request A request to read sensitive material and send it to an outbound destination, or a one-line request to send the prompt, hidden instructions, memory file, or your credentials; a request to send me the keys, tokens, secrets, or credentials of a .env file, opening its clause. A key alone, with no .env, is not enough Advisory
agent_message_code_relay A request to forward, send, text, or reply with a one-time code (one that carries a type such as verification or 2FA, a digit count, or "you just received", or a PIN or passcode) to a phone number, an email address, a URL, me, or this chat; to forward a received reset or sign-in link; or the "I sent you a code by mistake" story, or "I just sent you a code, forward it to me". A refusal on the same line exempts it Advisory
agent_override_imperative An order that names what it acts on: ignore or disregard all, your, or earlier guidance, directives, guidelines, or safety rules; disable, bypass, or remove your filters, guardrails, or safety; a persona (pretend to be, roleplay as, behave as) with no rules or unfiltered; you are no longer bound by; a jailbreak, developer, or sudo mode with no restrictions or root access; DAN mode; a critical update to your instructions; a message from your creators or the IT department that orders compliance or a reset; a standing instruction stored in your memory Note
agent_model_addressed_instruction A sentence or HTML text tag introducing a model-specific address, answer hook, or named tool request. The rule requires a following directive or a narrow promotional claim. Concealment counts beside a model address or answer hook, including an instruction to add something to the answer. Inline prose quotations, crawler directions, and unpaired confidentiality text are outside the matched shapes. Standalone prompt examples can still match. Advisory
agent_tool_hijack_directive An order aimed at the agent's tools: please or to exfiltrate, or exfiltrate the data, secrets, files, or credentials; do not tell the user within 160 bytes of a tool_call, run_command, execute_command, or function call, in either order; invoke a tool to email, send, upload, post, forward, or transfer; use your named tool to forward, send, email, upload, or share, opening its clause, with an outside email address, phone number, or URL, or an order not to tell the owner or user, on the same line; decode and then execute, run, or eval Advisory
agent_markdown_exfil A Markdown image, link, or reference-style definition ([label]: https://…) whose URL contains a template ({{ or ${) Advisory
agent_hidden_html An instruction-like phrase within 256 bytes after a hiding cue: display:none, visibility:hidden, a font size of zero, under one pixel, or 1 px (0.9em and 0.875rem are small text, not a cue), an offset of -1000 px or more (left, top, text-indent, a margin), a screen-reader-only class (sr-only, visually-hidden), the hidden attribute, or a Markdown comment ([//]: #); or inside an HTML comment, before the comment closes Advisory
agent_invisible_unicode A zero-width space, a directional mark, a word joiner, or an internal BOM in a text unit Note
agent_invisible_run A run of 16 or more consecutive zero-width or invisible formatting characters (U+200B to U+200D, U+2060 to U+2064, U+FEFF, U+180E, U+034F) in a text unit: bytes carried one bit or more per character Advisory
agent_bidi_override An explicit left-to-right or right-to-left override in a text unit Note
agent_bidi_formatting A directional embedding or isolate in a text unit Note
agent_bidi_source Any bidi override, embedding, or isolate in a file whose suffix names a programming or shell language (Trojan Source, CVE-2021-42574) Block
agent_base64_blob A base64 run of 400 or more characters that is not media, a font, a PEM body, an SSH key, or hex digits only (a PDF hex string of glyph ids or a UTF-16BE outline title is hex text, not base64), or a LibreOffice PrinterSetup item that decodes to exactly the job setup LibreOffice writes and holds no executable or base64 run of its own Advisory
agent_base64_executable A base64 run of 400 or more characters whose decoded head is a PE (MZ), ELF, or Mach-O header, in a page, a JSON value, a message, or a data: URI Block
agent_eicar The EICAR antivirus test string Block

The base64 exemptions apply to each run separately. A run that starts with an image, audio, video, or font header, or that follows a data: media URI, a PEM header, or an SSH key type, is exempt. A second, unrelated run in the same file still matches. CA FE BA BE is not treated as an executable header, because Java class files use the same bytes. Both base64 rules are skipped on the text-folded view: folding look-alike characters can join ordinary words into something that looks like a base64 run. They are skipped on the base64-decoded and percent-decoded views too: those views hold the text a run spells, for the lexical rules, and the run itself is judged in the raw view.

In the pdf-flate view, decoded streams can be image samples. A flat area of pixels whose byte values fall in the base64 alphabet reads as a long run, such as +G9+G9+G9... for RGB (0x2B, 0x47, 0x39). There agent_base64_blob counts a run only if it still has 400 characters after each unit of 1 to 12 characters that repeats four or more times in a row is cut to one copy. What is cut can be rebuilt from the copy that stays, so an exempt run carries no more than a run under 400 characters, which the rule already ignores. Grainy areas, where noise breaks exact repeats, are handled by a second count: the run is read in 64-character windows, and a window is cut when, at one lag of 1, 2, 3, 4, 6, or 8 bytes (the size of a gray to 16-bit CMYK pixel), at least 58 of its characters are within 4 of the character that lag before. The run must keep 400 characters under both counts. Base64 of English text, UTF-16LE or UTF-32LE text such as an encoded PowerShell command, JSON, compressed data, and random bytes reached at most 52 of 64 when measured (base64 of hex text 55), and a contiguous payload of 373 or more characters fires wherever it sits inside grainy padding. The stream dictionary (/Subtype /Image) plays no part, and no stream leaves the view. agent_base64_executable and every other view are unchanged. In the same view, agent_invisible_unicode, agent_bidi_override, and agent_bidi_formatting count a match only when the 16 bytes on at least one side of it decode as UTF-8: pixel bytes can spell E2 80 8B, but between bytes that are not UTF-8 it is not a character. See the image-sample record and the grain and control-byte record.

Generated rule files

The model-addressed rule and three phrase rules in the table above are rendered by scripts/generate_rules.py from rules/templates/: agent_prompt_injection_phrases.yar, agent_override_imperative.yar, agent_hidden_html.yar, and agent_model_addressed_instruction.yar. The model address strings share a sentence and HTML tag boundary. The three original templates share word lists (the verbs, the adjectives that mean "earlier", the nouns an order acts on, the markers after "you are now", the hiding cues, and the phrases read after a cue) and the separator between words are in fragments.toml, each written once. The non-greedy copy of the grammar that the documentary exemption and the HTML comment strings need is derived from the same fragments, so it ends where the strings it follows end. The links in the table go to the generated files, which are what a scan compiles; the templates are not packaged and nothing reads them at scan time. Lists differ between strings on purpose (rules is left out after override, forget counts only before the prompt or programming), and each difference is a boundary recorded in the detection backlog. To change one of these rules, see generated rule files.

Globals for other YARA tools

The scanner finds base64 runs with a linear regex in heuristics.base64_run_signals and hands the results to YARA as globals. As a YARA string, a 400-character class has no literal atoms, and YARA-X spent about 5 s per MB testing it at every offset. Rules about files known by name, such as .envrc and devcontainer.json, read the name the same way. Another YARA tool that loads rules/local/ directly must define all six globals before it compiles:

Global Type Compile default What the scanner sets per buffer
yara_gate_ext string "bin" The lowercased suffix of the submitted or member name, without the dot
yara_gate_name string "" The lowercased final component of the submitted or member name (.envrc for repo/.envrc in a zip)
yara_gate_text bool true true when the buffer sniffs as text
yara_gate_b64_blob bool false true when the buffer has a base64 run that agent_base64_blob should report
yara_gate_b64_exe bool false true when a run decodes to an executable header
yara_gate_svg_scope_proven bool false The scanner verified unambiguous SVG tag boundaries for this buffer. Leave false when running the rules standalone.

If the globals are missing, compilation fails with an error. Leaving both base64 booleans at false turns off the two base64 rules in that tool, and leaving yara_gate_name empty turns off dev_devcontainer_lifecycle and dev_envrc_exec. Nothing else changes.

How the other checks work

Installed curated packs add known malware signatures. Local heuristics check Office auto-exec macros (olevba), risky PDF tokens (/JavaScript, /JS, /OpenAction, /AA, /Launch, /EmbeddedFile, /RichMedia), and embedded executable headers. These checks also run on archive members. PDF feature names match complete decoded tokens (/JS1 is not /JS); see the name-boundary verification. PDF token checks remain conservative feature restrictions, with one exception. An /OpenAction that only opens a page is the note heuristic_pdf_OpenAction_goto: a direct destination array such as [3 0 R /Fit], a /GoTo action dictionary with no /Next chain, or a reference whose every definition in the file, written directly or stored in a FlateDecode object stream, is one of those. Every other /OpenAction still blocks as heuristic_pdf_OpenAction. That includes another action type (/JavaScript, /Launch, /URI, /GoToR, /GoToE, /SubmitForm, /ImportData, /Rendition), a missing or non-name /S, a chained /Next, a value that does not parse, and a reference that does not resolve. So a LaTeX-style PDF whose catalog and /GoTo action both sit in an object stream gets the note. Each occurrence is read from its own position, and every one must be a page open, including a redefinition in an incremental update; a copy inside a string or comment is one more value that must pass. An object stream whose first filter is not FlateDecode is not indexed, so a reference into it does not resolve and blocks. The same split applies in the pdf-flate view, whose references resolve against the whole file.

A lone Unicode control in a text unit is a note. That covers a zero-width space, a left-to-right or right-to-left mark, a word joiner, or an internal BOM (agent_invisible_unicode), a bidi override (agent_bidi_override), a directional embedding or isolate (agent_bidi_formatting), and a stray ZWNJ or ZWJ (heuristic_invisible_joiner). One such character is a paste artifact as often as an attack: a zero-width space copied from a web page, or an isolate around an Arabic or Hebrew name. The default policy and agent-document record it, and agent-ingest holds it. When the same member also carries another note, the scanner adds heuristic_unicode_control_conjunction, which is advisory, so agent-document holds the file too. That is how a phrase split by a zero-width space shows up: the phrase matches in the format-controls-removed view. Any bidi control in a file whose suffix names a programming or shell language blocks as agent_bidi_source, because it can make a comment or string literal read differently from what the compiler sees. A Unicode tag character outside an emoji subdivision-flag sequence blocks as heuristic_unicode_tag, including one spelled as an HTML entity or a JSON escape. The England, Scotland, and Wales flags stay clean. A run of eight or more variation selectors (U+FE00-FE0F, U+E0100-E01EF) blocks as heuristic_variation_selector_run, also when spelled as HTML entities or JSON escapes: a variation sequence is one base character and one selector, so a run renders as nothing and is how bytes are hidden after an emoji, one selector each (the encoding GlassWorm used in VS Code extensions). An emoji's U+FE0F, an ideographic variation sequence such as U+845B U+E0100, and a doubled emoji selector stay clean. A run of 16 or more consecutive zero-width or invisible formatting characters (U+200B to U+200D, U+2060 to U+2064, U+FEFF, U+180E, U+034F) is advisory as agent_invisible_run, beside the notes for its characters: a run renders as nothing and carries one bit or more per character, the zero-width encoding that hides bytes in ordinary text. Persian and Indic text use single joiners, emoji put one ZWJ between pictographs, and a pasted ZWSP stands alone, so none of them forms a run; a run of 15 stays the notes (record). These checks read text units only. A unit that does not sniff as text, such as an image, a PDF's raw bytes, or compressed data, holds these byte sequences by chance and is not checked. The first 4 KiB of a unit decide that, so text placed after a binary prefix is not checked either. A unit named .bmp, .dib, .tif, .tiff, .emf, or .wmf whose header validates as that format (as word/media and ppt/media members often are) is scanned like any other unit, but in the samples mode the pdf-flate view uses (above), so its pixels stop reading as base64 or as controls while a PHP webshell, an injected caption, an appended payload, or a zero-width space inside a word is still checked. Both the name and the bytes must say image, so a .php or .py script behind a valid BMP header is not in samples mode and keeps agent_bidi_source. See the raster record. A ZWJ inside an emoji sequence and a ZWNJ or ZWJ between Arabic, Persian, or Indic letters stay clean. A text file's long-base64 detection survives a filename change to .bin.

agent_role_spoof_action reads the system and developer turn openers of the common chat templates (Llama 3 and Llama 4 headers, harmony <|start|>system<|message|>, Llama 2 <<SYS>>, Mistral [SYSTEM_PROMPT], Phi <|system|>, Gemma <start_of_turn> and <|turn>, Granite, Cohere, DeepSeek's fullwidth-bar token, and the Kimi and MiniMax system openers), [start]/[message] role markers, XML-style system/developer tags, and all-caps SYSTEM:, [ADMIN], DEVELOPER, or OPERATOR header lines, each followed on the same line by a disclosure, override, or mode-switch request (System: reboot at 02:00 and Operating system: Windows stay clean). One-sentence requests are read by other rules. agent_prompt_injection_phrases, a note, reads "reveal your system prompt" in the second person (so "the debug view can show the system prompt" stays clean), "New instructions:" followed by from now on, respond only, you are now, or ignore. agent_secret_transfer_request reads "send/forward/reply with" the system prompt, hidden instructions, memory file, or your credentials, with no destination required. The phrase "you are now" needs a nearby override, unrestricted-role, or jailbreak-persona term (DAN case-sensitive, developer mode, jailbroken, "without restrictions"), including inside hidden HTML, so ordinary connection-status text stays clean. See exact boundaries and measured collisions.

agent_hidden_html reads its cues lexically; no stylesheet is resolved. opacity:0, white or transparent text, a zero height, and a data-* attribute are not cues, because fade-in animations, dark themes, and share widgets put them on text a reader sees. A phrase beside one of those is still a note from the phrase rules. The phrase has to follow the cue: text after a comment has closed is shown, and React, Vue, and Angular write empty comments beside the text they render, so a closed comment is not a cue. HTML text a browser does not show is also read on its own, as Office and PDF hidden text is; see Hidden HTML text.

agent_markdown_exfil also reads a reference-style definition, the form EchoLeak (CVE-2025-32711) used to get an image URL past link redaction. A definition with no template, such as a badge or a footnote link, stays clean, as does [home]: {{ site.url }}/index.html, which names no host.

Delivery patterns

The delivery-pattern rules report these IDs:

Rule ID Required indicators Finding severity
web_clickfix_lure Verification language ("verify you are human" or "you're human", "I'm not a robot", "human verification", "Verification ID", captcha), a clipboard-copy API (writeText, write, or execCommand('copy')), an instruction to open Run, a terminal, PowerShell, the File Explorer address bar, or Spotlight with Command+Space, and a paste instruction (Ctrl+V, Command+V, or "paste the command") near each other. Each key may sit in its own <kbd> tag Advisory
web_html_smuggling Base64 decoding, Blob/URL APIs, and a nearby literal script, executable, installer (.msi, .msix, .pkg), shortcut, disk-image, or archive download filename Advisory
script_encoded_powershell An encoded command of 40 or more base64 characters plus a hidden-window flag or an execution policy of Bypass or Unrestricted in a bounded PowerShell command fragment. Each parameter is read in any spelling PowerShell accepts: a prefix (-e, -enco, -win, -exec), after -, --, /, or a Unicode dash, with a window style of hidden, a prefix of it, or 1 Advisory
script_mshta_remote A direct HTTP(S) target, or an inline vbscript: or javascript: URL, passed to mshta Advisory
script_regsvr32_remote_scriptlet Regsvr32 with a remote /i: or -i: target and scrobj.dll in one bounded command Advisory
script_rundll32_mshtml Rundll32 JavaScript protocol invoking MSHTML RunHTMLApplication Advisory
shortcut_powershell_encoded Parsed PowerShell target plus encoded and hidden, bypass, or unrestricted arguments, in the same parameter spellings as script_encoded_powershell Advisory
shortcut_mshta_remote Parsed MSHTA target with an HTTP(S) argument Advisory
shortcut_padded_arguments Parsed shortcut whose arguments open with a line break, vertical tab, or form feed, or with 32 or more spaces or tabs: the ZDI-CAN-25373 padding that empties the Target box in Properties Advisory
shortcut_interpreter_remote Parsed cmd, PowerShell, wscript, cscript, conhost, forfiles, rundll32, regsvr32, msiexec, certutil, or bitsadmin target whose arguments carry an http(s):// URL or a \\host\ path Advisory

These lexical combinations can also occur in legitimate administration, export code, and security documentation; see the delivery-rule sources and boundaries and the attachment-signature research. They do not execute or decode the referenced payload. An IT shortcut that runs cmd /c start https://intranet…, an installer shortcut msiexec /i \\server\share\x.msi, a rundll32 url.dll,FileProtocolHandler https://… link, or a logon script through cscript \\dc\netlogon\… matches shortcut_interpreter_remote; a shortcut to explorer.exe \\fileserver\share, a browser shortcut that opens a URL, and powershell -File C:\… do not. Four leading spaces in a shortcut's arguments stay clean; a leading line break has no benign writer.

Windows shortcut-family checks

These rules look at the file's own format header, not its extension, so a renamed copy still matches. A Windows host acts on these files when Explorer shows, previews, or extracts them, often before anyone opens them. A macOS or Linux gate that syncs them to a Windows machine is still the place to hold them.

Rule ID Required indicators Finding severity
win_internet_shortcut_launch An [InternetShortcut] whose URL= is a file:, UNC, or search-ms: target or a local program path, or whose WorkingDirectory= is a UNC, WebDAV, or web location Advisory
win_shortcut_remote_icon An [InternetShortcut], .scf [Shell], or desktop.ini [.ShellClassInfo] icon loaded from a UNC path or file: URL Advisory
win_library_unc_location A file that opens with a .library-ms or .searchConnector-ms description (after an optional XML declaration) whose <url> is a UNC path (\\host or //host) or a file: URL with a host Block
win_library_remote_location Any other description whose <url> is a UNC path, file: URL, or web address, including library XML quoted in a document Advisory
web_search_ms_remote A search-ms: or search: link whose crumb=location: is a UNC path (plain or %5C%5C-encoded) or a web address Advisory
win_rdp_resource_redirect An .rdp connection file with a full address:s: host that also redirects local drives (drivestoredirect:s: with a value) or opens a RemoteApp (remoteapplicationmode:i:1) Advisory
win_theme_remote_resource A .theme section ([Theme], [Control Panel\Desktop], or [VisualStyles]) whose BrandImage=, Wallpaper=, or visual-style Path= is a UNC path or file: URL Advisory

The shapes come from CVE-2025-33053 (a .url whose working directory is a WebDAV share), CVE-2025-24054 (a zipped .library-ms that leaks an NTLM hash), and search-ms phishing. win_library_unc_location blocks because Windows contacts the SMB host as soon as Explorer lists the folder or the archive is extracted, which is the CVE-2025-24054 leak. A library that really lives on a company file server blocks too; release it through review. A CDATA wrapper or entity-encoded backslashes still block (the entities through the html-unescaped view). A comment before the root element, or more than 67 bytes of leading whitespace, drops such a file to the advisory rule; agent-ingest still holds it. An enterprise search connector to an https:// SharePoint site, a local file:/// location, or a folder icon on a file server also matches. Those stay advisory: the default policy allows them and agent-ingest holds them. A .url to an https:// page, a local icon, and a knownfolder: library do not match. The .rdp shape is the rogue RDP lure: a signed connection file that maps the victim's drives and clipboard into an attacker's RemoteApp session. The .theme shape is the Windows Themes credential leak (Akamai, CVE-2024-21320, with the CVE-2024-38030 and CVE-2025-21308 variants), where BrandImage, Wallpaper, or the visual-style Path points at a UNC share. Any .rdp that mstsc saves with Local Resources → Drives ticked (it writes drivestoredirect:s:*), not only an administrator's, and a corporate theme whose wallpaper lives on a file server match and stay advisory. A file saved by mstsc with an empty drivestoredirect:s: and remoteapplicationmode:i:0, a theme with local paths, and an INI that is not a theme do not match.

Windows script-host and document checks

These rules key on the file's own markup or header, not its extension. Each file type opens with a double-click on Windows 10, Windows 11, and Windows Server 2022 and later, and several run before the reader sees any content.

Rule ID Required indicators Finding severity
win_hta_shell_object An <hta:application> element plus WScript.Shell, Shell.Application, ADODB.Stream, or an XMLHTTP object Advisory
win_wsf_remote_script A <job> or <package> root plus a <script src=> that points at a URL or a UNC path Advisory
win_msc_console_remote An <MMC_ConsoleFile> whose <String> holds a URL, javascript:, vbscript:, or a UNC path, or that names apds.dll (the GrimResource shape) Advisory
win_reg_run_interpreter A registry export whose Run, RunOnce, RunServices, or Winlogon value starts powershell, mshta, wscript, cscript, rundll32, regsvr32, certutil, bitsadmin, msiexec, or cmd /c, or that sets an IFEO Debugger Advisory
win_script_encoded_body A Windows Script Encoder body: the #@~^ opening marker with its six-character length field and ==, plus the ==^#~@ closing marker, the .vbe/.jse form the script engines decode at run time Advisory
win_script_host_fetch_run CreateObject(, ActiveXObject(, or GetObject( plus a download or file-write class (MSXML2.XMLHTTP, Microsoft.XMLHTTP, WinHttp.WinHttpRequest, ADODB.Stream) plus WScript.Shell or Shell.Application followed by .Run, .Exec, or .ShellExecute; the body of a .vbs, .js, .jse, or .wsf without <hta:application> Advisory
win_cmd_fetch_start A downloader (iwr, irm, Invoke-WebRequest, curl, wget, certutil -urlcache, bitsadmin /transfer) that names a runnable suffix, then start, call, cmd /c, an interpreter (msiexec, wscript, mshta, powershell, …), or .\ launching a runnable suffix on the same line or the next Advisory
doc_onenote_embedded_launcher A .one header plus a FileDataStoreObject whose payload starts with MZ, a shell-link header, or script text (<hta:, <script, @echo, WScript.Shell, powershell, CreateObject(, …) Advisory
doc_spreadsheet_formula_exec A CSV, SYLK, or IQY cell that starts a DDE launch (=cmd|…), calls DDE(, WEBSERVICE(, or an IMPORT*( function, calls a Win32 DLL from an Excel 4.0 macro, or an IQY WEB header followed by a URL Advisory
web_ms_protocol_lure An ms-msdt: link with IT_BrowseForFile, IT_LaunchMethod, PCWDiagnostic, or $( (CVE-2022-30190), or an ms-appinstaller:?…source=https:// link Advisory
win_lolbin_download certutil -urlcache or -verifyctl, bitsadmin /transfer or /addfile, or msiexec /i, /package, or /a (or -i) with an HTTP(S) URL on the same command, or finger user@host piped into cmd, PowerShell, or a Unix shell, or read by for /f … in ('finger user@host') do call (the 2025 and 2026 ClickFix stage) Advisory
win_msbuild_inline_task TaskFactory="CodeTaskFactory" or "RoslynCodeTaskFactory" and a <Code element, plus VirtualAlloc, VirtualProtect, CreateThread, CreateRemoteThread, WriteProcessMemory, NtAllocateVirtualMemory, GetDelegateForFunctionPointer, Marshal.Copy, Assembly.Load(, or a kernel32 DllImport; or a download call (DownloadData, DownloadString, DownloadFile, GetByteArrayAsync, GetStringAsync, WebRequest.Create, HttpWebRequest) with FromBase64String or Process.Start Advisory
win_inf_scriptlet_install An INF with Signature=$chicago$ or $Windows NT$ whose OCX line points scrobj.dll at an HTTP(S) or UNC scriptlet, or whose [RunPreSetupCommandsSection] runs PowerShell, mshta, regsvr32, rundll32, certutil, bitsadmin, wscript, cscript, or cmd /c (the cmstp and InfDefaultInstall shape) Advisory
win_browser_credential_decrypt Login Data, encrypted_key or app_bound_encrypted_key, and CryptUnprotectData or ProtectedData.Unprotect in one file: the Chromium saved-password decryption at the core of an infostealer Advisory

An administrator's HTA that shells out, a console that links an intranet page, a .reg that registers a legitimate tool, a notebook with a .lnk attachment, an IQY against an internal server, a vendor site that still uses ms-appinstaller:, and a deployment script that runs msiexec /i https://… all match, as does a deployment VBScript or batch file that downloads an installer and runs it, and a legacy administration script that was run through the Script Encoder. They stay advisory. A notebook whose attachment is a PDF or an image, an HTA without a shell object, a write-up that only names the #@~^ and ^#~@ markers, a logon script that only maps a drive through WScript.Shell, an inventory script that only posts through XMLHTTP, the old ActiveXObject("Microsoft.XMLHTTP") AJAX polyfill, a download with no launch, a .reg that runs an ordinary program, a CSV formula such as =SUM(A1:A2), an ms-settings: link, and certutil -hashfile do not match. Microsoft disabled the ms-appinstaller: handler by default in December 2023 after Storm-0569, Storm-1113, Sangria Tempest, and Storm-1674 abused it; older or re-enabled installs still honor it.

Windows defense evasion

Loaders and ransomware droppers turn off protection and recovery before the payload runs. These rules read the commands in a .ps1, .bat, .cmd, .reg, script body, or pasted text, in UTF-8 or UTF-16LE.

Rule ID Required indicators Finding severity
win_defender_tamper Set-MpPreference -Disable{RealtimeMonitoring,BehaviorMonitoring,IOAVProtection,ScriptScanning,BlockAtFirstSeen,IntrusionPreventionSystem,ArchiveScanning} with $true, 1, or no value; Add-MpPreference or Set-MpPreference -Exclusion{Path,Process,Extension} (Add appends to the list, Set replaces it) of a drive root, a temp, profile, Public, or ProgramData folder, a script interpreter, or a runnable suffix; a reg add, Set-ItemProperty, New-ItemProperty, or .reg line that sets DisableAntiSpyware/DisableRealtimeMonitoring/… to 1; sc stop/delete WinDefend, sc config WinDefend start= disabled, or Stop-Service WinDefend Advisory
win_amsi_bypass AmsiUtils and amsiInitFailed, amsiContext, or amsiSession within 400 bytes; or AmsiScanBuffer, AmsiOpenSession, or EtwEventWrite and VirtualProtect within 2 KB Advisory
win_recovery_inhibit vssadmin delete shadows, wmic … shadowcopy … delete, Win32_ShadowCopy piped to .Delete() or Remove-CimInstance, bcdedit /set … recoveryenabled no or bootstatuspolicy ignoreallfailures, or wbadmin delete catalog/systemstatebackup/backup Advisory

A hardening guide that shows how an attacker disables Defender, a red-team write-up that quotes the AMSI one-liner, and a backup runbook that prunes shadow copies all match and stay advisory. Get-MpPreference, Set-MpPreference -Disable… $false, an exclusion for a build folder such as C:\src\repo, a DisableAntiSpyware value of 0, sc query WinDefend, sc config WinDefend start= auto, prose that explains AmsiScanBuffer or VirtualProtect alone, vssadmin list shadows, and bcdedit … recoveryenabled Yes do not match. Sources: Set-MpPreference, MITRE T1562.001, and MITRE T1490.

Later local checks

rules/local/ in this tree has 89 YARA rules in 59 files. Recount them with rg -N '^rule ' rules/local. The tables above and below list their own families; these are the rules none of them lists.

ID Severity What it matches
script_base64_decode_pipe Advisory base64 decode (-d, -D, -di, --decode) piped into sh, bash, or zsh, named by path or through env and optionally under sudo, or into python or perl with no program argument; or the decode inside bash -c "$(…)" or eval "$(…)"
script_osascript_shell Advisory osascript -e, or osascript reading a here-document, running do shell script
script_clear_quarantine Advisory An xattr or /usr/bin/xattr command (optionally after sudo) that deletes com.apple.quarantine (-d, -rd, -r -d, -d -r) or clears every attribute (-c, -cr)
script_launchagent_shell Advisory LaunchAgent program arguments that run /bin/sh, bash, zsh, or /usr/bin/python3, perl, ruby, osascript, curl, or env, with curl, wget, osascript, base64, or urlopen in the plist
script_chmod_execute Advisory chmod +x, u+x, or an octal mode, then a launch (./, a shell, a /tmp path, or ~/, optionally after sudo, nohup, setsid, or exec) on the next line or after && or ;
doc_rtf_object Block RTF object data or an object update
doc_svg_active Block An SVG with a script or an onload handler. In standalone SVG or when scope is ambiguous, the rule retains its strict match. In ordinary markup with proven SVG scope, a sibling page script or body handler does not count.
doc_svg_event_handler Advisory An SVG animation with a non-empty onbegin handler or an image, feImage, svg, or use element with a non-empty onerror handler. Uses the same scope proof and strict fallback. Whether the handler runs depends on the embedding.
doc_svg_active_link Advisory An SVG that runs or loads something with no script element or handler: an a element whose href or xlink:href is a javascript: URL, a set or animate whose to, from, or values writes one, or a foreignObject that holds an iframe, object, embed, form, or meta refresh before it closes. An empty, ;, or void(0) target (the legacy no-op link), https and fragment links, and div, span, or p labels in a foreignObject (diagram exports) stay clean. Uses the same scope proof and strict fallback (record).
doc_external_template Advisory One OOXML <Relationship> element whose Type ends in /attachedTemplate, with TargetMode="External" and an http(s):// or UNC Target, or an RTF file whose \template control word is followed within 256 bytes by an http(s):// or UNC path (T1221)
script_download_execute Advisory curl or wget piped into a shell or interpreter, bash <(curl …), eval "$(curl …)", source <(curl …), a PowerShell download or DNS lookup on the same line as iex, or a python -c one-liner that fetches and runs the result (details below)
container_disk_image Advisory ISO, VHD, or DMG magic at the offset that format uses

script_download_execute matches curl or wget piped (| or |&) into sh, bash, zsh, python, or perl, with or without sudo in between, and with the interpreter named by path or through env (| /bin/bash, | /usr/bin/env sh); a download piped into node with no program argument (curl … | node, the one-liner in the 2026 VS Code task lures; | node parse.js reads it as data); bash <(curl …), sh -c "$(curl …)", eval "$(curl …)", and source <(curl …); a PowerShell download (Invoke-WebRequest, Invoke-RestMethod, DownloadString, iwr, irm) or a DNS lookup (Resolve-DnsName, the TXT-record stage) on the same line as Invoke-Expression or iex; and a python -c one-liner whose quoted program both fetches (urlopen, urlretrieve, requests.get, http.client) and runs the result (exec(, eval(, os.system(, subprocess.run(, …), in either order. A Python file that fetches and spawns a process is a program, not a one-liner, and stays clean, as does a one-liner that only prints what it fetched. The -c match wants python, python2, or python3 followed by a space; python3.12 -c, combined flags such as -Ic, and a program split over several lines are not covered. The Homebrew install line is the known benign collision; an installer documented as curl … | /bin/bash or eval "$(curl …)" is the same collision in another spelling. VERSION=$(curl …) is an assignment and curl … | /usr/bin/shasum is a checksum, and both stay clean. It does not match a file that also contains "the following command", "common lure", or "for example" within 160 bytes before one of those download words or python. script_base64_decode_pipe treats python or perl as the sink only when nothing follows it but -, so base64 -d | python3 -m json.tool stays clean. That exception covers the whole file. An SVG foreignObject without a script or an onload handler does not match doc_svg_active. A disk-image filename without the magic bytes does not match container_disk_image.

macOS, Linux, and developer-workspace checks

Rule ID Required indicators Finding severity
linux_desktop_exec_fetch A [Desktop Entry] whose Exec= line runs curl, wget, base64 -d, xxd -r, or openssl enc Advisory
script_fetch_chmod_run One line that runs curl or wget, then chmod (+x, u+x, or an octal mode), then launches a path (/, ./, ~/, or a variable) Advisory
web_applescript_url_lure An applescript://com.apple.scripteditor?action=new link whose prefilled script contains do shell script, curl, or osascript Advisory
macos_password_prompt A display dialog whose text names a password, passcode, passphrase, or credentials and whose answer field is with hidden answer Advisory
macos_keychain_dump security find-generic-password or find-internet-password with -g or -w (prints the secret), or security dump-keychain -d Advisory
unix_privilege_persistence A NOPASSWD: ALL line appended to /etc/sudoers by a redirect or through tee, a write to /etc/ld.so.preload, a downloaded key appended to authorized_keys, an insert into access on TCC.db, or, on one line, unset HISTFILE, export HISTFILE=/dev/null, or history -c followed within 120 bytes by unset HISTFILE, history -c, or a history-file delete Advisory
macos_gatekeeper_disable spctl --master-disable or --global-disable, or defaults write com.apple.LaunchServices LSQuarantine set to NO, false, or 0 Advisory
macos_archive_upload ditto -c -k followed within 4 KB by curl with -F/--form name=@…, or --data-binary/-T/--upload-file of a path (the AMOS and Poseidon exfiltration step) Advisory
macos_stealer_password_check dscl . -authonly (or /Local/Default) in the same file as Login Data (also written Login\ Data, as a shell script escapes it), Cookies.binarycookies, key4.db, logins.json, or a wallet name (Exodus, Electrum, MetaMask or its extension id, Atomic Wallet, Coinomi): the AMOS and Banshee password check before collection Advisory
macos_jxa_remote_exec osascript -l JavaScript, ObjC.import(, or Application.currentApplication() with dataWithContentsOfURL, stringWithContentsOfURL, or NSURL.URLWithString plus eval(, or a doShellScript( string that runs curl, wget, a base64 decode, or chmod +x Advisory
script_reverse_shell bash -i or bash -l redirected to /dev/tcp/host/port, exec N<>/dev/tcp/… with a descriptor of up to three digits, a shell with stdin, stdout, and stderr on one numbered descriptor (sh -i <&3 >&3 2>&3, the PHP and Ruby one-liners), a Perl open(STDIN,">&S") followed by exec of a shell, nc/ncat with -e/-c/--exec and a shell, the mkfifo … sh -i … nc relay, socat joining exec:sh to tcp:, a Python socket.socket( followed by os.dup2(…fileno() or pty.spawn(, or a PowerShell Net.Sockets.TCPClient( stream read into iex Advisory
dev_vscode_autorun_task A "runOn": "folderOpen" task with, within 2 KB, a downloader (curl, wget, iwr, irm, Invoke-WebRequest, DownloadString), node -e, python -c, base64 -d, or powershell -enc, or within 1 KB "reveal": "never" Advisory
dev_python_pth_exec A .pth file (or .pth archive member, such as one inside a wheel) with a line that starts import and calls exec, eval, compile, a base64/base85/zlib/marshal decode, subprocess, os.system, os.popen, urlopen, or urlretrieve Advisory
dev_gyp_command_expansion A file with the suffix .gyp or .gypi (binding.gyp, or one inside an npm .tgz) whose <!( or <!@( command expansion, on one line, sends output to /dev/null or chains with &&, ||, or ; to echo a .c, .cc, .cpp, or .cxx name. node -p and node -e lookups such as require('node-addon-api').include_dir, pkg-config, a node -p test that uses && inside JavaScript, and 2>&1 piped on to another command stay clean Advisory
dev_git_config_exec A git config (a [core], [pager], [filter "…"], or [diff "…"] section) whose fsmonitor, sshCommand, pager, editor, askPass, pager.<command>, filter clean/smudge/process, or diff textconv value on the same line runs a downloader (curl, wget, nc, Invoke-WebRequest, …), sh -c, cmd /c, encoded or inline PowerShell, python -c, node -e, perl -e, base64 -d, /dev/tcp/, mshta, or osascript; or a hooksPath in /tmp, /var/tmp, /dev/shm, /Users/Shared, ~/Downloads, $TMPDIR, %TEMP%, C:\Windows\Temp, C:\Users\Public, or on a UNC share. Each key counts only in its own section (no other section header between them), so [alias] entries are not read Advisory
dev_devcontainer_lifecycle A file named devcontainer.json or .devcontainer.json whose initializeCommand (it runs on the host) has a downloader, certutil, bitsadmin, sh -c, cmd /c, PowerShell, python -c, node -e, base64 -d, /dev/tcp/, mshta, or osascript; or whose onCreateCommand, updateContentCommand, postCreateCommand, postStartCommand, or postAttachCommand decodes base64, runs encoded PowerShell, opens /dev/tcp/, or runs inline Python or Node that reaches the network or execs. The indicator must sit inside a JSON string of that key's value (a string, or one in its array or object, which may span lines), so a later property is not read. An installer piped to sh inside the container is left to script_download_execute Advisory
dev_envrc_exec A file named .envrc that runs curl, wget, nc, ncat, Invoke-WebRequest, or Invoke-RestMethod, decodes with base64 -d or openssl enc … -d, opens /dev/tcp/, runs osascript -e, or runs python -c with urllib, socket, b64decode, exec(, or subprocess. source_url with its pinned hash, use flake, layout, dotenv, and eval "$(pyenv init -)" stay clean Advisory
agent_config_approval_bypass "chat.tools.autoApprove": true or "chat.tools.global.autoApprove": true (CVE-2025-53773), a "chat.tools.terminal.autoApprove" (or deprecated chat.agent.terminal.autoApprove) pattern that matches every command, such as "/.*/": true, "github.copilot.chat.claudeAgent.allowDangerouslySkipPermissions": true, "defaultMode": "bypassPermissions", "enableAllProjectMcpServers": true in the same file as an "ANTHROPIC_BASE_URL" override (CVE-2026-21852), or a Codex approval_policy = "never" with sandbox_mode = "danger-full-access" in a .toml file Advisory
agent_cli_secret_sweep --dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox, --trust-all-tools, --allow-all-tools, --yolo, or --approval-mode yolo within 256 bytes after claude, gemini, codex, copilot, cursor-agent, opencode, qwen, or q chat, and within 2 KB a wallet, keystore, seed phrase, mnemonic, wallet brand, id_rsa/id_ed25519, .ssh/, private key, or inventory.txt (the Nx s1ngularity shape) Advisory
script_credential_upload One command that pipes an SSH private key, .aws/credentials, .git-credentials, .netrc, .npmrc, .pypirc, a Docker or kube config, gh/hosts.yml, a login keychain, or a .env file into curl, wget, nc, ncat, socat, or a PowerShell web cmdlet, or attaches one with curl -d/--data*/-F/-T, wget --post-file, or -InFile; the same file read through $(cat …) inside a curl or wget command; an archive of the whole .ssh folder piped to one of those tools; or the process environment sent out (env | curl …, printenv | base64 | nc …, curl -d "$(env)" …) Advisory
dev_github_workflow_secrets_dump A workflow (a runs-on: or steps: key) that expands ${{ toJSON(secrets) }} with, within 1 KB, curl, wget, nc, ncat, Invoke-WebRequest, Invoke-RestMethod, iwr, irm, base64, webhook.site, fetch(, requests.post, or upload-artifact, unless that expansion is the whole value of an input under with: (after up to 16 single-line sibling inputs) Advisory
dev_github_workflow_runner_backdoor A run: step, single-line or a block scalar within its first nine lines (indented deeper than the run: key, at key columns 2 to 20, with an optional comment after the indicator), that interpolates ${{ github.event.<discussion, issue, comment, review, review_comment, or pull_request>.<body or title> }}, with runs-on: self-hosted (any letter case; bare, or first in a flow or block list) or a RUNNER_TRACKING_ID override earlier in the same job, or RUNNER_TRACKING_ID in the top-level env: before jobs:. A job header is a key at indent 2, 3, or 4 whose first key line, after any comment or blank lines, sits at twice that indent Advisory

The GitHub Actions rules cover a cloned or synced repository that carries a planted workflow, the persistence the 2025 npm worms left behind. In September 2025 Shai-Hulud committed a workflow that serialized every secret with toJSON(secrets) and posted it to a webhook; a later variant wrote it to a file and published it as an artifact. In November 2025 Shai-Hulud 2.0 registered victims as self-hosted runners and added discussion.yaml, which echoes the discussion body on that runner and overrides RUNNER_TRACKING_ID so spawned processes survive the job. Anyone who opens a discussion runs commands on the machine. Passing toJSON(secrets) to an action input (the secrets-to-env pattern, even when a later step uploads build output or decodes a keystore), a single named secret in an API call, event text read through env:, with:, or if:, and event text in a run: step on a GitHub-hosted runner do not match. The last is the common injection weakness, not a planted backdoor; a rule for it would hit many ordinary repositories. A secret dump split across steps more than 1 KB apart, a self-hosted label that is not first in its list, a runs-on: written after the job's steps, a job beyond the first 32 self-hosted or RUNNER_TRACKING_ID lines, and a run step that reads the field through a shell variable all pass. So does an env: value placed after a block-scalar run:, and a self-hosted runner in a different job from the interpolating step, including when that job opens with a comment or blank line. A job-level environment: or env: map whose child is name: stays part of its job. Still unmatched: flow-style steps (steps: [{run: …}]), a dump split across format() or re-encoded with xxd or gh gist, and event fields written in another letter case. A dump exported to $GITHUB_ENV through env: still matches when an unrelated curl (a health check, say) sits within 1 KB; telling those apart needs step boundaries the rule does not track.

The .desktop shape is the APT36 campaign that shipped a launcher named like a PDF inside a zip. The download, chmod, run chain and xattr -c are the AMOS stealer bootstrap. Script Editor links are the ClickFix variant that avoids Terminal. A vendor install line such as curl -o install.sh … && chmod +x install.sh && ./install.sh also matches script_fetch_chmod_run. The password dialog is the AMOS-style stealer prompt; a dialog that asks for a project name, or hides an API token without naming a password, stays clean. security find-generic-password -s Chrome without -g/-w, security add-trusted-cert, a locally generated public key appended to authorized_keys, sudo visudo, a sentence warning against NOPASSWD, and history | tail stay clean. An operator's own hardening or cleanup script can match the persistence rule; it stays advisory. A chain that ends in sudo mv or an install step, a chmod on a later line, a .desktop file that only mentions curl in Comment=, and a Script Editor link that only shows a dialog do not match.

dev_vscode_autorun_task is the repository lure from the 2025 fake-job-interview campaigns: a project whose .vscode/tasks.json runs a hidden or downloading task as soon as the folder is opened in VS Code or a fork of it. It matters for an agent that clones a repository or reads a synced project folder before a person opens it. A watch or npm install task on folder open without a hidden terminal or a fetch does not match. Pentest cheat sheets, a socat port forward, nc -zv, echo > /dev/tcp/host/22 port checks, spctl --status/--assess, and ditto -c -k followed by notarytool do not match; a write-up that quotes a full reverse shell or a forum answer that says to run spctl --master-disable does, and stays advisory. Sources: VS Code task run behavior, MITRE T1059.004, T1553.001, and T1560.001.

The build and install rules cover code that runs because a project is built or a package is installed, before anyone opens a file. MSBuild compiles and runs an inline task during a build (T1127.001), and RoslynCodeTaskFactory is common in legitimate .targets files, so win_msbuild_inline_task needs memory-allocation, assembly-loading, or download-and-decode calls; Process.Start("git", …) alone stays clean. win_inf_scriptlet_install follows the cmstp INF shape (T1218.003). Connection Manager profiles use RunPreSetupCommandsSection too, so the section must launch a script host or downloader. Site-packages runs any .pth line that starts with import at every interpreter start (T1546.018). The LiteLLM 1.82.7 and 1.82.8 packages shipped one. dev_python_pth_exec reads the yara_gate_ext global (the suffix), so a .py file with the same line, setuptools' distutils-precedence.pth, virtualenv, and editable-install .pth files stay clean. npm runs node-gyp rebuild for a package with a root binding.gyp and no install script, and GYP runs every <!(…) expansion as a shell command while it configures, so a planted expansion runs at install with no lifecycle script to review (T1195.002). The Miasma wave wrote <!(node index.js > /dev/null 2>&1 && echo stub.c) (JFrog, MAL-2026-6526). Native addons use the same syntax for ordinary lookups, so dev_gyp_command_expansion holds only the two parts that hide the run: discarded output and an echoed stub source name (record). ClickOnce deployment manifests are not matched: a remote deploymentProvider is the normal shape of a published ClickOnce application.

The checkout-configuration rules cover files that run a command because a tool opens the folder, not because someone runs the file. Git reads .git/config on every git status, diff, log, or fetch an agent or IDE runs, so a zipped or synced checkout that carries a core.fsmonitor, sshCommand, pager, or filter-driver command runs it then; a clone does not copy this file, but an archive or sync folder does. dev_git_config_exec reads the section headers, so the file needs no particular name. hooksPath is reported only for temp, shared, and network folders: Husky writes core.hooksPath = .husky/_, so a repository-relative path is ordinary. A dev container's initializeCommand runs on the host before the container exists, while its other lifecycle commands run inside it, where downloading an installer is normal; dev_devcontainer_lifecycle holds the two to different indicator sets. direnv runs .envrc as a shell script once the folder is allowed. Both name-scoped rules read the yara_gate_name global (a member path written with backslashes is split on them, and trailing dots and spaces are dropped, as Windows reads it), so the same text in init.json or setup.sh is not reported by them. npx -y <package> in an MCP or dev container config is not matched: it is the normal way those configs start a server.

The credential-store rules need the decryption or password check, not a path list. Sandbox deny-lists, cleaners, YARA rules, and threat reports name Login Data, key4.db, and logins.json too. macos_password_prompt already matches the hidden-answer dialog; macos_stealer_password_check adds the dscl . -authonly check that AMOS and Banshee (Elastic) run on the typed password, and needs a browser or wallet store in the same file because LAPS and admin scripts run the same check.

The agent-settings rules cover a cloned or synced project that reconfigures the coding agent about to open it. agent_config_approval_bypass is advisory because teams commit some of these keys on purpose. Approximate GitHub code-search counts (October 2026) set its boundaries: "enableAllProjectMcpServers": true appears in about 6,300 committed .claude files and "hooks" with SessionStart in about 15,600, so neither matches on its own. The rule pairs the first with an ANTHROPIC_BASE_URL override, the CVE-2026-21852 shape that sent the API key to another host before the trust prompt. A hook or MCP command that pipes a download into a shell, decodes base64 into a shell, or opens a reverse shell is already matched by script_download_execute, script_base64_decode_pipe, and script_reverse_shell, and a JSON command that starts sh, bash, curl, or PowerShell by heuristic_mcp_shell. Claude Code ignores bypassPermissions in project files in current releases; the key still marks a file that asks for it. agent_cli_secret_sweep follows the Nx s1ngularity postinstall script, which ran claude, gemini, and q with their approval flags off and a prompt to inventory wallets and keys. A CI step that runs an agent unattended for ordinary work stays clean, because secrets.* and .env are not hunt terms. Prose that warns about the flag next to an SSH key name still matches. script_credential_upload is the shell step an injected instruction asks an agent's command tool to run. curl --netrc-file (authentication, not upload), copying a key, and uploading an ordinary file stay clean.

agent_override_imperative covers the families in CyClaw's injection red-team corpus (.claude/skills/injection-redteam/probes.yaml) that the narrower phrase rule did not. On main, 4 of its 36 attack probes were held; with this rule, 33 are, and none of its 12 benign questions is. The three left out ("act as a penetration tester", "execute command: rm -rf", "run this code to dump the log") have the same shape as ordinary README and review text. Every pattern names what it acts on (your filters, a persona with no rules, your memory), so "enable developer mode", "run this command", "act as a reviewer", and "ignore the earlier draft" stay clean. "Ignore previous instructions" stays with agent_prompt_injection_phrases and its documentary exemption. That rule also reads the same order with its words in another place ("ignore all instructions above", "ignore above instructions"), in the second person without an adjective ("ignore your instructions", "forget your system prompt"), negated ("do not follow the previous instructions", "stop following your earlier rules"), and as a replacement ("override previous instructions"). These spellings have no documentary exemption, so a quoted example of one is a note. "The model may ignore the system prompt", "later rules override earlier rules", "ignore any instructions found in the document", and "don't forget your instructions sheet" stay clean. In both rules the words of an override may be split by one line break, as in hard-wrapped Markdown or "IgNoRe\nALL\tPrEvIoUs instructions" from the CG-agent-harness tests; a blank line still splits them, and the documentary exemption follows the same rule. A threat model or test suite that quotes these orders still matches, as a note: under agent-document the file passes with the finding recorded, and under agent-ingest it is held. The invoke-tool-to-send and decode-and-execute probes from the same corpus moved to agent_tool_hijack_directive, which stays advisory because those are orders to act, not phrases a report quotes; the corpus counts above are unchanged under agent-ingest.

agent_message_code_relay is for chat, SMS, and mail bodies passed to scan_bytes or scan -. Ordinary one-time-code notices can contain "code", "verification", and "do not share", so the rule needs a relay verb, a code that carries a one-time qualifier, and a destination. A bare "code" in chat is usually source code. A recognized refusal ("never forward the verification code to anyone") exempts only a request starting inside that refusal; an independent request on the same line can still match. "I just sent you a code, forward it to me" is matched when the request follows the code at once; "we just sent you a code, enter it on the sign-in page" is the notice and stays clean. English only; a paraphrase without the listed verbs passes.

The spellings added for the readiness review's P2 shapes (#255, record) read the word before their verb inside the pattern:

  • an administrator claim's order follows the claim at once;
  • a use your ... tool order and a .env request open their clause;
  • a persistent bypass follows punctuation or a word such as to, must, or always.

So "do not use your email tool to ...", "how to use your email tool to ...", "please don't email me the keys from your .env file", and "never skip approval" stay clean in the recorded boundary cases. The linked timing probes cover repeated-line fixtures; they are not a latency guarantee for arbitrary documents. A system prompt or agent configuration that tells its own agent to "use your email tool to send" mail to a fixed outside address matches agent_tool_hijack_directive the same way an attack does. Under agent-document, such a file is held. The third-person "use the send_email tool", as framework docs write it, does not match. A bare SYSTEM OVERRIDE: header, or one that names a flag, stays clean.

Code-relay refusals

The code-relay rule checks where each request starts. A recognized refusal exempts a request that starts inside it. A separate request can still match on the same line. The rule searches the gaps between refusals by offset, so a long notice repeating a recognized refusal does not need a request-by-refusal comparison. This applies to both ASCII and UTF-16LE patterns. The finding remains advisory, and both agent policies still hold a matching request.

Documentary quotations

agent_prompt_injection_phrases lets a quotation ("the phrase 'ignore all previous instructions'") cover the one override it ends with. Any other override counts. This includes an override before the first quotation, one between quotations, and one after a quotation ends. The rule finds them by searching the gaps between quotations by offset, as the code-relay rule does (#274, record). A file of repeated quotations scans in linear time and stays clean. It used to time out at 16 MiB. The answer is the same as before for every input the differential probe tried. That includes UTF-16LE text, and a UTF-16 override inside an ASCII quotation, which still counts. An override that counts is a note, which agent-ingest holds. YARA-X records at most 1,000,000 matches of a string, so a full list of an override string counts as an override. An override after a million quoted ones is never recorded, and without that rule the file would pass (#280). So documentation that quotes an override a million times or more gets the note even when every quotation is benign: agent-ingest holds it, and agent-document passes it with the finding recorded. At the 67-byte line in the record that is about 64 MiB, under the 100 MiB intake limit.

Reported offsets

A finding's offset and length name one match of its rule within the finding's view. For most rules that is the first match of the first string that matched, in the order the rule declares its strings. Some rules decide by where matches lie: an exemption (a quotation of an override, a refusal of a code request) or a placement (a fetch beside an execute, a script inside an SVG). In such a rule, the first match can be one the condition exempted while another match made the rule fire.

For these rules the scanner checks whether the rule still fires with the first match's first byte blanked. If it does not, the first match was needed and is reported as before. If it does, the scanner finds the shortest prefix of the view on which the rule fires and reports the match that ends there. It skips the first match if another match ends at the same byte. The search re-scans prefixes with that rule's file alone and the same global values. It runs only in views of up to 1 MiB (WITNESS_MAX_BYTES). A larger view, a timeout during the search, or a rule that no longer fires on a prefix keeps the first match. Verdicts never change; only which match a finding names (#284, record).

JSON parsing limits

MCP, notebook, and Keras readers check structure before the standard JSON decoder allocates an object tree. Each candidate permits at most 100,000 commas, colons, and opening braces/brackets outside strings and comments, and 64 nested containers. The check keeps counters only; escaped quotes and punctuation inside strings or comments do not consume that budget. The standard decoder still owns syntax, duplicate keys, numbers, and string decoding. Keras's embedded HDF5 config check stops at the first JSON object's end. No new dependency is needed.

Exhaustion returns incomplete, with mcp json exceeds structure limit, notebook json exceeds structure limit, or keras config exceeds structure limit. This can hold a legitimate dense notebook or generated config. It is a coverage limit, not a malware finding; do not interpret missing derived findings as clean. A potentially relevant MCP input requiring comments/trailing-comma decoding beyond the existing 1 MiB JSONC cap is also incomplete (mcp json exceeds comment limit). Malformed inputs within the budget retain their existing treatment.

This bounds tree amplification, not total process memory: input buffers, decoded strings, rules, and other readers still use memory. The file and expansion budgets remain in force. Consumers needing a hard memory or time ceiling must isolate the worker and hold its input on failure. See the allocation measurements and compatibility checks.

Structural checks without YARA

In addition to the format-specific checks below, heuristic_embedded_pe, heuristic_embedded_elf, and heuristic_embedded_macho block validated executable headers inside another file. They differ from document-member and nested-archive executable findings, which depend on container position.

Legacy Office macro inspection reports heuristic_office_autoexec for an olevba auto-exec indicator, and heuristic_office_shell, heuristic_office_createobject, or heuristic_office_urldownloadtofile for those whole-word keywords in extracted VBA. These findings block. The code is never executed; a claimed VBA project that cannot be parsed is incomplete. A macro's mere presence is not this keyword check, though OOXML vbaProject.bin has its own blocking structural finding. See office_findings.

filetype_mismatch_<kind> reports header/extension disagreement: PE, ELF, and Mach-O mismatches block, while other recognized mismatches are advisory. A zero-byte file with a promised document/type suffix uses heuristic_empty_file instead. The name and header checks live in filetype.py and are applied by scanner.py.

The scanner also reports these checks without a YARA rule. heuristic_pickle_reduce walks pickle opcodes with a reader built on the standard library's pickletools opcode table and argument readers (the same parse pickletools.genops gives, minus large payloads) and never calls pickle.loads. It blocks a call (REDUCE, NEWOBJ, NEWOBJ_EX, INST, or OBJ) whose callable is a GLOBAL or STACK_GLOBAL of os, subprocess, posix, or nt, any name called eval or exec, or a builtins (or Python 2 __builtin__) name other than the value constructors protocols 2 and 3 use for ordinary data (set, frozenset, bytearray, bytes, complex, range, slice, int, float, str, bool, list, dict, tuple, their Python 2 spellings, and exception classes). Any name from the four modules blocks; builtins.getattr, __import__, open, and compile block. Every opcode's documented stack effect is applied, so dicts, sets, big integers (LONG1/LONG4), byte strings, object state, and memo references before the call are walked through, and the module and name are followed through the memo (STACK_GLOBAL fed by BINGET). Pickles written back to back, the legacy torch.save layout, are walked in turn. A malformed stream, which a loader would also refuse, reports nothing. A stream longer than 20,000,000 opcodes, or one that holds more than 1,000,000 memo entries or stack items at once (a one-byte MEMOIZE repeated millions of times would otherwise exhaust memory), is incomplete, pickle exceeds walk limit, rather than clean. The walk also checks the scan deadline every 4,096 opcodes (incomplete, timeout), and it skips byte payloads, big integers, and text arguments longer than 4 KB without building them, since none can name a callable. The check names dangerous callables; it is not a general pickle safety validator, and a module outside that list (for example runpy or importlib) is not reported. heuristic_safetensors_header blocks a .safetensors file whose header length or tensor offsets do not fit the file. heuristic_gguf_header blocks a .gguf file, or any bytes that start with GGUF, when the version is not 1, 2, or 3, the counts exceed 100,000, or a following record does not parse. heuristic_keras_unsafe_call blocks a Keras model config that names a module that runs commands, opens sockets, or imports code (os, subprocess, sys, socket, importlib, runpy, pickle, marshal, ctypes, urllib, requests, and a few others), or a builtins function that runs code, imports, opens files, or reflects (eval, exec, compile, __import__, open, getattr, and similar). Keras writes its own losses, metrics, and activations as builtins and resolves them in its own modules first, so builtins with mean_squared_error stays clean. It also blocks Keras's own get_file (CVE-2025-8747) or enable_unsafe_deserialization (CVE-2025-9906); load_model resolves every module it reads, which is how CVE-2025-1550 runs code with safe_mode on. heuristic_keras_lambda is advisory on a Lambda layer that carries marshalled bytecode (__lambda__ in Keras 3, function_type: lambda in Keras 2); safe_mode refuses it by default, and some teams turn that off. Both read config.json inside a .keras archive and a .json file from model.to_json(), decoded the way json.loads decodes bytes (a BOM, UTF-16 or UTF-32, and escaped keys all reach the check), and up to four JSON objects inside an .h5/.hdf5 file that open with a key Keras writes at a config's top level (class_name, config, module, registered_name, build_config, or compile_config), in any key order or spacing. Only a config with class_name at its top level counts, as load_model requires, so a JSON document that merely names subprocess, such as an API index, stays clean. A Keras candidate is checked against the JSON structure budget before decoding; exhaustion is incomplete, keras config exceeds structure limit. The subsequent walk retains its 2,000,000-node and 64-level limits. The walker parses JSON and never imports Keras. It does not read a config hidden in HDF5 chunked or compressed storage, and four decoy objects ahead of the real config in an .h5 use up its attempts. heuristic_ooxml_vba and heuristic_ooxml_dde block. heuristic_ooxml_oleobject and heuristic_ooxml_altchunk are advisory. heuristic_mcp_shell and heuristic_mcp_hidden_instruction are advisory on a JSON object or array, read after a UTF-8 byte order mark, and, when strict JSON fails, as jsonc-parser reads a config: // and /* */ comments and trailing commas removed outside strings, in a file of at most 1 MiB. Before decoding, a .json file that names command, args, or description (or uses a Unicode escape that could encode a key) is checked against the JSON structure budget; exhaustion is incomplete (mcp json exceeds structure limit). The subsequent walk retains its 2,000,000-node and 64-level limits. JSON without any such key is not parsed by this heuristic; its raw and derived scans still run. The hidden-instruction check reads every description string, at any depth, for do not tell/mention/inform/reveal the user, no need to tell/mention/inform the user (not when to and a verb follow, as in "to wait"), do not notify the user about it (notify only with about or of and this, it, or that, since a notification tool limits how often it notifies the user), this tool immediately as the first step, ignore previous instructions, an <IMPORTANT> tag, before using this tool, read, an .ssh/ path, id_rsa, mcp.json, pass its contents, or as a sidenote, the vocabulary of MCP tool poisoning. A description is a wrapped docstring, so the words of a phrase may be apart by any run of whitespace, a line break and a no-break space included. heuristic_pdf_text_unmapped is a note on a PDF whose shown text could not all be decoded (see the pdf-text view under Limits); under agent-ingest it holds the file, under agent-document it is recorded. heuristic_pdf_content_unread is a note, graded the same way, on a PDF with page content no view could decode: a stream a /Contents entry names (directly, in an array, in an array object, or in a page dictionary stored in an object stream) whose dictionary marks it as no image, form, or font program, or a form XObject, under a filter chain with no decoder here or under LZW or RunLength past its size cap. Images, fonts, object streams, and streams nothing names as page content are never reported, whatever their filter. The report's pdf_content_unread counts those streams by filter name, from the standard names of ISO 32000-1 Table 6 or other, never a name taken from the file (verification). heuristic_hidden_text is advisory on an instruction phrase in Office or PDF text a reader hides (vanished, white, or 1 pt runs, hidden shapes and slides, invisible render modes); see hidden Office and PDF text. agent_encoded_instruction is a note on a base64-decoded run that contains ignore previous instructions or ignore all previous instructions as a plain substring, case-insensitive, when the phrase rules did not already report that run. It keeps the reading of the retired YARA rule of that name, which also caught a phrase run into the word before it; plain text with the same glued phrase is not reported. heuristic_empty_file is advisory: a zero-length file whose name promises a type (an online-only cloud placeholder named report.pdf) is reported under that label instead of as a text file in disguise, and agent-ingest still blocks it. heuristic_double_extension, heuristic_filename_control, heuristic_document_executable, and heuristic_nested_executable block. heuristic_double_extension matches a document, spreadsheet, image, media, or OneNote suffix, optional spaces, dots, or underscores, then a program suffix (.exe, .scr, .com, .msi, .xll, …) or a launcher suffix: Windows script hosts and shortcuts (.lnk, .url, .js, .hta, .cmd, .ps1, .msc, .library-ms, …), disk images that mount on a double-click (.iso, .img, .vhd, .vhdx), packages and manifests that install (.msix, .appx, .application, .appref-ms, .msp, .chm, .diagcab), registry and setup files (.reg, .inf), Remote Desktop and theme files that reach a host when opened (.rdp, .theme), App Installer manifests (.appinstaller), and macOS launchers (.pkg, .dmg, .scpt, .terminal, .workflow, .command, .webloc), so invoice.pdf.lnk, photo.jpg.iso, statement.pdf.msix, invoice.pdf .exe, and Invoice.pdf.exe with a fullwidth dot block. A .txt or .html decoy pairs only with a program suffix, so page.html.js stays clean; a single suffix such as backup.iso is not a disguise. heuristic_filename_control blocks a name that contains a bidi override, embedding, isolate or mark, a zero-width space, a word joiner, a BOM, or a Unicode tag (the invoice_\u202Efdp.exe trick). ZWNJ and ZWJ are allowed so Persian, Indic, and emoji names stay clean. Both checks read the submitted name and every component of an archive member's path, so an Invoice.pdf.app/ bundle inside a zip blocks. Directories above a scanned file are not checked.

Measured precision

Rechecked on November 16, 2026 at 1426e7f with Python 3.12.13, YARA-X 1.23.0, and bundled rules. Run uv run --frozen python docs/verification/measure_precision.py --fail-over to repeat it. This benign-only measurement reports collisions on the generated corpus. The November 6 documentation audit records the older baseline.

On the generated 328-file corpus (docs/verification/measure_precision.py, run by CI), 85 of the 89 YARA rules hit zero files and all scans completed. No structural check reports anything: the merged PDF whose font name means five fonts is read page by page and is clean (page fonts). Four distinct files had findings (4/328, 1.22%); each matching rule hit one file (1/328, 0.30%). These are benign-corpus hit rates, not statistical precision (true positives divided by all positives), recall, or a production false-positive estimate. A per-rule threshold does not bound the combined hold rate. The four that hit are tagged Windows shortcut-family collisions, each one file: win_library_unc_location blocked a file-server library, win_library_remote_location matched an https search connector, win_shortcut_remote_icon a file-server folder icon, and win_internet_shortcut_launch a .url to a local file: URL. The corpus's 31 agent-facing files (one-time-code notices, README lines that run a command, committed agent settings that keep approval on, unattended CI agent steps), 12 platform files (inline MSBuild tasks, Connection Manager and driver INFs, a local JXA script, a dscl -authonly test, standard .pth files), 14 ecosystem files (five GitHub Actions workflows, five Keras models), and four checkout files (.git/config with LFS, git-crypt, Watchman, and Husky entries, a devcontainer.json, a nix-direnv .envrc) hit nothing. See Precision with Windows shortcut-family files and the earlier Precision after the local pack grew; the near-miss pairs in docs/verification/check_rule_pairs.py remain the tighter benign boundary. Vendor rule bodies stay out of git; the license reasons are in Vendor pack review.

Phase 1c-small reader boundaries

Excel xl/comments\d*.xml and PowerPoint ppt/comments/*.xml rich-text comments are scanned one comment at a time as ooxml-comment, and as ooxml-comment+text-folded when folding changes the text. Runs are joined within that comment; paragraphs and explicit breaks become newlines. Two comments cannot complete or suppress each other's phrase. One-run comments remain covered by raw and entity-decoded views. The same bounded tokenizer serves worksheet cells. Excel threaded-comment parts are not added here. Drawings and the Word glossary part are joined by ooxml-text (below).

The existing html-text reader also accepts .svg or an SVG root, including one after an XML declaration. tspan and textPath stay inline; each text element is its own piece. CDATA inside SVG contributes literal text without entity decoding. This is lexical extraction: coordinates, CSS, visibility, and visual line layout are not interpreted. Positioning tspans within one text element can still form a phrase. Latin small capitals with a direct ASCII letter equivalent fold in text-folded and the joined/decoded views that already fold; this is a fixed table, not general phonetic transliteration or a new finding severity.

Verification records the targeted gaps, benign boundaries, and remaining Phase 1c work in #171.

PDF annotation boundaries

pdf-annotation reads /Contents text strings from dictionaries declaring /Type /Annot, or a standard annotation /Subtype and /Rect. It scans each definition, including inactive revisions; it does not resolve the active xref revision or prove that a viewer displays the annotation. Literal and octal escapes, hexadecimal strings (including odd final nibbles), PDFDocEncoding, UTF-16BE BOM and PDF 2.0 UTF-8 BOM strings are decoded. Indirect text strings and members of already supported object streams are included. Language tags inside Unicode strings are removed. Each annotation and each blank-line-separated paragraph is a separate piece; folded spelling reaches the same existing rules as pdf-annotation+text-folded. Encoded text alone is no finding.

The reader skips comments, literal-string contents, and stream bodies while looking for annotation dictionaries. It proves a stream boundary only when a direct or indirect /Length lands on endstream and that keyword is the first endstream in the counted span. The body is then skipped, so a comment, literal, hex string, or object header inside it is not an annotation. An object header by itself does not reject the span: with no earlier endstream, those bytes are body data, and an annotation after the landed keyword is still read.

A missing length, a length that does not land on endstream, or a length whose span contains an earlier keyword endstream does not resume lexical parsing. A longer token such as endstreaming is not that keyword. The scan is incomplete (pdf annotation stream length unaligned). That earlier keyword is not taken as the boundary either. Resuming there is unsafe, and treating the first textual endstream or an N G obj inside opaque bytes as a real object would also be unsafe: images, fonts, and compressed streams can carry those bytes. The hypothesis of rejecting either sequence was measured. Neither appeared in 348 installed font and image files, a random 1 MiB buffer, its Flate encoding, or the Flate encoding of prose that contains both phrases. Rejecting an object header alone still fails closed on four aligned lookalike rows whose bodies contain N G obj and whose real annotation follows the stream, so that half is not used. A length is refused when the span contains an earlier keyword endstream. That also incompletes two correct lengths: uncompressed page text that includes the word, and a JPEG comment that includes it. The Flate encoding of the same prose does not contain the keyword, and the annotation after that stream is still read.

A keyword-delimited object header in the raw bytes of a landed stream makes the scan incomplete (pdf object header inside stream). The header is N G obj with PDF whitespace (NUL, TAB, LF, FF, CR, or space, including a run of them) between the tokens. The number starts the span or follows whitespace or a delimiter ()<>[]{}/%. obj ends on whitespace, one of those delimiters, or <<. This does not depend on an xref entry, a page reference, or the generation. Decoded object-stream members are not this check. They are read on their own, and a type-2 xref entry stays an index into that stream. The incompletes are conservative. A header written as stream data, including one after a /Length that lands on endstream, still stops the scan, and a real annotation after that stream is not read. A header that no viewer displays still stops the scan. Measured on 2026-11-07, none of these files had such a header in a landed span: rfbproto-3.3.pdf (Acrobat Distiller 3.02, 110778 bytes), rfbproto-3.7.pdf (GNU Ghostscript 5.10, 165552 bytes), rfbproto-3.8.pdf (GPL Ghostscript 8.61, 143840 bytes), gnus-logo.pdf (GPL Ghostscript 8.54, 3658 bytes), and the two Microsoft Word hybrids CL2020 EMEAR DEVWKS-1542 Intersight Ansible Lab Guide.pdf (1030061 bytes) and Impact DNW07 UCS Ansible Collection Lab Guide.pdf (433208 bytes). The same header pattern was absent from 348 fonts under /usr/share/fonts and 454 PNG or JPEG files under /usr/share/pixmaps and /usr/share/icons.

A file with startxref also resolves cross-reference offsets before the scan can be clean. A candidate is every occurrence of the bytes startxref that a number follows, with or without a following %%EOF, wherever it is written. Readers search the end of the file for the bytes or for the line and do not lex around them, so one inside a comment, a literal string, or a stream body is a candidate too (linearized files and the keyword search). Comments between the keyword and the number are skipped, and leading zeros are dropped, as readers do. Offset 0, and a keyword that no number follows, name the start of the file: when the file opens with an xref stream that stream is read as a candidate, and otherwise the occurrence is skipped, since a linearized file's first-page trailer carries startxref 0 by design. A sign or a dot in front of the number, or more than 20 digits, is malformed. A candidate that is present and does not parse still makes the scan incomplete. Each candidate includes its /Prev chain. Classic 20-byte tables, Flate /Type /XRef streams (/W, /Index, and a PNG or TIFF predictor), and hybrid /XRefStm are read. A newer section overrides an older one inside one candidate. At one trailer, a classic offset and a stream offset for the same object are both checked. The landed span includes the newline and endstream the length ends on, so an offset on those bytes is inside the span. A type-1 offset from any candidate inside that span makes the scan incomplete (pdf annotation xref offset inside stream). A type-1 offset that does not begin an object header is malformed, which covers a /W width that does not match the packed fields and a stream whose decoded length is not an exact multiple of the row width. A type-2 entry is an object-stream index, not a file offset, so object streams stay readable. An object that only a later candidate lists is kept when its offset begins an object header outside every span. A file with no candidate leaves the sequential reader in place. A classic table may have blank lines between its sections and before trailer, and its trailer is read as far as the startxref after it. A malformed table or stream (pdf xref malformed), a filter or predictor this reader does not decode (pdf xref unsupported), more than 100000 entries (pdf xref entries exceed 100000), more than 32 startxref candidates, or a spent deadline is incomplete. A /Prev chain of more than 32 sections is malformed, so the scan is incomplete, not truncated. Flate output is charged to the shared 500MB budget, and a stream pdf_streams already inflated is not inflated again.

Rich text /RC and filters the existing object-stream reader does not support remain outside this view; form-field values are read as pdf-string, below. A recognized annotation with an unresolved /Contents reference, duplicate dictionary keys, malformed string, or unreadable encoding makes the scan incomplete. Each source is limited to 4,000,000 tokens, each dictionary to 100,000 tokens and the existing 64-level parser depth, encoded strings to 1 MiB, and objects/pieces to 100,000, under the shared deadline and expansion budget. See verification.

PDF font resource keys

The file-wide font lookup recognizes standard PDF name escapes in the /Font key. Direct and indirect resource dictionaries use the same lookup. The page-specific lookup still requires an unambiguous page tree and falls back to the file-wide lookup for escaped keys. For the measured nonstandard reference spellings that only some readers follow, text shown with the affected font adds the heuristic_pdf_text_unmapped coverage note. The scanner keeps its other readings and leaves the disputed reference unresolved. Printed examples and comments do not add the note. See reader verification.

PDF text strings

pdf-string reads the other text strings a PDF reader shows (#239): a form field's value (/V in a dictionary that is a widget or carries /FT, /T, /Parent, or /Kids, including each string of a choice field's array, written in place or in an array object of its own whose items may each be a reference), the document information strings (/Title, /Author, /Subject, /Keywords, /Creator, /Producer), and an outline item's /Title. pypdf and PyMuPDF show each of these, and pdfinfo shows the information strings. They come from the same pass, and every definition, as annotation /Contents, so a string in an object stream or behind a reference is read, and so is one in an inactive revision or in a field no page lists. Each string, and each blank-line-separated paragraph of one, is a piece, folded as pdf-string+text-folded when the spelling changes.

A string is decoded as pypdf decodes it: a UTF-8 or UTF-16 byte order mark, then UTF-16 without a mark when a NUL is in its first or second byte, then PDFDocEncoding. A byte PDFDocEncoding leaves undefined is read as its Latin-1 character, and an invalid sequence as U+FFFD. Nearly every PDF has an information dictionary, so an odd byte from a producer never makes this view incomplete. A value that is not a text string (a check box's name, a signature dictionary, a reference to a missing object) is passed over: a reader shows no text for it. A string over 1 MiB, a malformed literal, and more than 100,000 pieces across both views are incomplete, under the same reasons as annotations. An array object over 100,000 tokens is incomplete (pdf string array exceeds 100000 tokens) only when a value refers to it.

The view reaches the existing rules and adds no rule or severity. Rich text (/RV, /RC), a field's default value (/DV), its tooltip (/TU), the options of a choice field (/Opt), and the XMP metadata stream are not read here. See verification.

Marked-content ActualText

pdf-actual-text reads each /ActualText string written in a content stream's own property list (/Span <</ActualText <FEFF...>>> BDC, any tag, literal, octal, hex, or UTF-16), and of each property list outside a stream, such as one a page names from its /Properties resource (/Span /MC0 BDC), directly or as an object of its own, decoded as a pdf-string is, one piece per distinct value. poppler 24.02 and PDFium show such a string in place of the glyphs it covers; pdfminer.six shows the glyphs. Two values are never joined, so a phrase split over two spans is two pieces. More than 20,000 distinct values in one file adds the note heuristic_pdf_text_unmapped, and the rest are not read. The values share the 16 Mi-character decoded-text limit with the page text, so past it the scan is incomplete (pdf decoded text exceeds character limit). A property list outside a stream is read whether or not a page names it, shares the dictionary pass's limits (a string over 1 MiB, or more than 100,000 values with the other strings, is incomplete), and is reported in the same view as the content-stream values. Not read: /ActualText on a structure element (one with /S, or /Type /StructElem), which neither reader shows in place of the glyphs; and the page text a reader rebuilds when it substitutes a value for part of a sentence. See verification and named property lists.

PDF font coverage

When a selected font has an unreadable composite interpretation, the scanner can report heuristic_pdf_text_unmapped. A nearby font's map cannot clear that note for these declarations when the selected font has no usable map of its own:

  • A declared simple font with a composite Identity encoding.
  • A Type0 font with an absent or null /DescendantFonts entry.
  • A font interpreted as composite whose /ToUnicode value has the wrong type for a map reference. An explicit null is treated like an omitted entry.

The scanner still tries the existing font readings. The note describes unread text, and it follows the existing policy for notes. See verification.

RTF reader boundaries

RTF data beginning with {\rtf1 is read even under a renamed filename or inside an archive. rtf-text removes formatting, decodes hex bytes using the declared codepage or font charset, and reads signed Unicode escapes with group-scoped fallback lengths. Each paragraph, table cell, and separate header/footer/footnote is scanned independently, then folded as rtf-text+text-folded when its spelling changes. Unknown starred destinations, metadata, font tables, pictures, field instructions, and embedded-object bytes are skipped; field results remain text. Original bytes are still scanned. Formatting or an encoding alone is no finding.

The existing phrase rule stays a note: default and agent-document are advisory (exit 0), while agent-ingest blocks (exit 1). Malformed groups, escapes, lengths, unreadable codepages or font charsets, and resource limits make the scan incomplete (exit 2). The reader checks the shared deadline and expansion budget and caps group depth at 128, tokens at 4,000,000, and pieces at 100,000. It is not a renderer: Symbol-font glyphs, pictures, embedded objects, and visual layout are not decoded. See the synthetic verification.

Office XML and OpenDocument reader boundaries

The Office readers match tags by pattern; no XML parser runs. plain_xml first gives them the part as a parser hands its text on: comments, processing instructions, and the DOCTYPE are removed, a CDATA section becomes the text it holds, > in a quoted attribute value is escaped, a reference to an entity the part's own DOCTYPE declares becomes that entity's text, and a UTF-16 part is transcoded. So Ignore all <!-- -->previous, <![CDATA[previous]]>, and &w; reach the phrase rules as the words pandoc, LibreOffice, or a library built on a parser shows. A run counts as joined only where markup stood between two stretches of its text; a run wholly in CDATA is contiguous in the raw bytes and is left to them. An entity with a system or public identifier is never read, fetched, or expanded. More than 64 internal entities, or more than 1 MiB of expansion, makes the scan incomplete.

ooxml-text joins runs in the body, header, footer, note, comment, shared-string, slide, and notes-slide parts, and in the parts a renderer draws from: ppt/slideMasters and ppt/slideLayouts, a SmartArt diagram's diagrams/data*.xml and drawing*.xml, charts/chart*.xml, xl/drawings/drawing*.xml, and word/glossary/document.xml. Notes masters, handout masters, and theme parts are drawn on no slide and are not joined. Hidden runs are read in the body parts only (below); a 1 pt run on a slide master is read as shown text.

A part is known by its name or by what it is. A package names its own parts, so a body can be word/document2.xml and shared strings xl/strings.xml. A unit that opens as XML is also given a role by its root element and the namespace its prefix binds, transitional or strict: a Word document, hdr, ftr, footnotes, endnotes, or comments; a SpreadsheetML sst, worksheet, or comments; a PresentationML sld, notes, or cmLst; and the drawn parts above. A Word or SpreadsheetML root outside that list is known by what it holds (a body, an si, a sheetData). All four Office readers, the hidden-text reader included, take the role, so a loose XML file with an Office root is read too. Notes and handout masters, themes, settings, and styles get no role.

A run is known by its namespace, not its prefix. The same XML under ns0: (what ElementTree writes) or x: (what the Open XML SDK writes for SpreadsheetML) is read, and an element spelled w: that an inner declaration binds to another namespace is not a run. A part with the usual prefixes is not rewritten. A DrawingML <a:br/> is a line break, white space is allowed before the > of an end tag, a text tag's attributes have no length limit, and a phonetic run is left out of a shared string. Where readers take a part differently (resolved or as written, a break or a join) each reading is its own ooxml-text view, at most four, and a rule is reported once per unit and view. More than 256 nested elements that each declare a namespace, or more than 8 namespace defaults in a DOCTYPE, make the scan incomplete; no office suite writes either. Tracked changes, text boxes anchored between two words, and field, math, and symbol runs are read one way only. See verification; a change is checked against LibreOffice, pandoc, python-docx, and python-pptx with probe_ooxml_reader_agreement.py.

odf-text joins the paragraphs of an OpenDocument part, and is folded as odf-text+text-folded when its spelling changes. A part is read when it declares the OpenDocument text namespace, under whatever prefix it binds: content.xml of an .odt, .ods, or .odp, a header in styles.xml, an embedded object, and a flat document (.fodt). A paragraph or heading is one line; a span, a link, a comment, and CDATA join; text:s, text:tab, and text:line-break are their white space, and other white space collapses as a reader collapses it. A paragraph nested in another (a footnote body, a comment, a text box) is a line of its own, and a blank line follows each table cell, so two cells never form one phrase. LibreOffice shows the character data of an element it does not know and pandoc drops it, so such a paragraph is read each way: as the text every reader agrees on, as LibreOffice lays it out, and as all of its character data. Paragraphs nested more than 64 deep make the scan incomplete. Hidden OpenDocument text (text:hidden-text, a style with text:display="none", white or 1 pt text set by a style) is read as shown text. See verification; a change to what the view joins is checked against LibreOffice and pandoc with probe_odf_reader_agreement.py.

Hidden Office and PDF text

heuristic_hidden_text is advisory. It is the Office and PDF counterpart of agent_hidden_html: an instruction a reader does not see, where a person reviewing the document would see nothing to object to. The text a reader hides is collected on its own and scanned piece by piece, as ooxml-hidden-text or pdf-hidden-text (and +text-folded when folding changes it); HTML has the same view, described below. When an agent_ rule other than the Unicode-control and base64 rules matches there at note or advisory severity, the scanner adds one heuristic_hidden_text finding for the member, labeled with that view. The rule's own finding still comes from the views that read every run, such as ooxml-text or pdf-text. So under agent-document a phrase in hidden text holds the file, where the same phrase in shown text is a note. Hidden text without an instruction reports nothing: templates, watermarks, field codes, and OCR layers hide text routinely.

Hidden means, from a run's, shape's, or slide's own properties:

  • Word: w:vanish, w:specVanish, or w:webHidden (unless its value is 0, false, or off), w:sz of 2 half-points or less, or a w:color whose every channel is F0 or more. White text counts only when no run shading or highlight, paragraph shading, table-cell or table shading, or page background (w:background) paints something else behind it. The earlier properties a tracked change keeps (w:rPrChange and its siblings) and a paragraph mark's own properties (w:pPr/w:rPr) do not count. XML comments do not affect formatting. A text box in a drawing marked hidden="1" hides its text.
  • PowerPoint: a run of 1 pt or less (a:rPr sz of 100 or less), every text frame of a shape or group whose p:cNvPr is hidden="1", and every frame of a slide with show="0".
  • Excel: a rich-text run in shared strings, an inline-string cell, or a comment at 1 pt or less, or in near-white (an RGB value, or theme colour 0 without a darkening tint, which Excel maps to the light background).
  • PDF: text in render mode 3 (invisible) or 7 (clip only), and text whose active fill or stroke colours are all near-white and whose first glyph sits on nothing the same content stream painted in another colour. Render mode and colour carry across the streams of one /Contents array, which are read as one stream: a direct array in a raw page dictionary, an array kept in an object of its own, or an array in a page stored in a compressed object stream. A painted area is the bounding box of a filled path, an image placement, or a shading; a stream keeps 32, and white text over a later one counts as hidden. In a page's own content stream, text under 1 pt and text that lies wholly outside the union of every /MediaBox in the file are hidden too. Form XObjects, patterns, and glyph procedures are drawn through a matrix the reader does not know, so their size and position are not judged, while their render mode and colour are.

A hidden piece ends at the next shown text that is more than whitespace (decoded through the font for a PDF), so hidden halves on both sides of a shown word are never joined; a shown space between them is kept. In Office parts a piece also ends at a table cell, text box, shape, shared string, inline string, or comment; paragraph ends inside it become newlines.

Accepted boundary: an OCR text layer is render mode 3 over a page image, and so is an attacker's, so a scanned page whose text layer reads as an override phrase is held under agent-document. Exempting text over an image would let any picture hide an instruction.

Not resolved: properties inherited from styles, list styles, slide layouts and masters, and cell formats (all in other parts), hidden sheets, rows, and columns (their text sits in shared strings), DrawingML colours, text covered by a later opaque fill, clipping, transparency and blend modes, CropBox, and per-page boxes; a bogus /MediaBox widens the page union. White text over a shaded table whose shading comes from a table style counts as hidden, which can only add a hold. The hidden-text record has the fixtures, the benign sweep, and the timings.

Hidden HTML text

The same finding covers HTML. The text a browser does not show when the document opens is collected as html-hidden-text (and +text-folded), and an agent_ rule match in a piece adds one heuristic_hidden_text, as above. Where agent_hidden_html already fired for the member nothing is added, so a file that was held keeps the findings it had. Nothing is rendered, no stylesheet is resolved, and no script runs.

A claim that text is hidden rests on one exact declaration on an element the text is inside, in a spelling a browser accepts. A browser drops a declaration it cannot parse, so font-size:0 px, display:none\9, and a declaration after an unclosed parenthesis are no claim, and a declaration that could have set a property back ends the claim an ancestor made.

Hidden Until
A comment; a data-* value of twelve characters or more that holds a space
The content of a template
display:none nothing inside can show again
Zero opacity, in a document with no script, no linked stylesheet, and no animating style nothing inside can show again
visibility:hidden a descendant sets it back, or carries a class or id a stylesheet could use
A font size of zero a descendant sets a size
Transparent text an element with any attribute of its own
Text within 12 steps a channel of the background the same element declares an element with any attribute of its own; a block inside an inline element
The hidden attribute with no inline display, in a document with no stylesheet nothing inside can show again
A screen-reader-only class, in a document with a stylesheet an element with any attribute of its own

In a document with a stylesheet a colour or a font size is followed to the text directly inside the declaring element and no further, because stylesheets style by element type (code { color: ... }, p { font-size: 16px }). Transparent and same-colour text count only where no style in the document draws text by a shadow, a stroke, a fill, or a background clip, and no stylesheet is linked.

A piece of hidden text whose whole wording is also in the text the page shows is not reported: a share widget copies the title into data-title, and a mail preheader repeats the first line. A hidden piece ends at the next shown text that is more than whitespace, and a block boundary inside it is a newline.

Which elements are open follows the HTML parser where a browser's tree differs from the written nesting: end tags a writer may leave out, formatting elements a browser reopens after a misnested end tag, a link that ends an open link, the parts of a table and text moved out of one, a form closed by its parent, and HTML that ends an svg. Markup past what is followed stops the element reader for the rest of the document, and comments and data-* values are still read: a select that holds anything but options, a formatting element that ends across a table cell or an svg, a fourth copy of one formatting element, a table inside an open paragraph under an older doctype, more than 500 open elements, or more than 400,000 tags.

Not claimed, because the markup does not settle it: a box of no size that clips its content, text moved off the page by an offset, a font of a pixel, and near-white text with no background of its own. Chromium can paint the same markup differently with a different colour preference or stylesheet. See the browser measurements. agent_hidden_html still reads the offset and the small font as cues beside one of its phrases. Colour attributes (<font color>, bgcolor) and SVG presentation attributes are not read. A document a browser reads as XML (an .svg or .xhtml name, or an XML start under any name but .html) gives only its comments and data-* values, since the element rules are the HTML parser's. A noscript body is not read: with scripting on it is not markup.

Three things are assumed, not read: the document is taken as it is when it opens, before a script changes it; the browser has scripting on; and no stylesheet overrides an inline declaration or turns visibility back on for an element by its type. Someone who wants the element reader to stop can write markup it does not follow; the lexical rule and the phrase rules do not depend on it. The record has the comparison with Chromium that set these boundaries, and probe_html_hidden_browser.py repeats it.

Terminal output

text-folded handles escape sequences for text matching. A phrase can still be read when a terminal later redraws over it. The scanner has no terminal viewport, wrapping, or scrollback state for a captured log. The tmux 3.4 measurements show how those details affect whether the same bytes leave text visible.

Declared charsets

A text file that is not UTF-8 and declares a charset is read through that charset as decoded-charset, and the other text views are read from that text, as they are from UTF-16. A browser shows a windows-1251 page as Cyrillic, so a phrase written in those bytes reaches the multilingual rule; read as UTF-8 its letters were U+FFFD. The bytes as written are still scanned raw.

The UTF-8 reading is scanned as well, with the same checks and views it had before. A consumer that ignores the declaration decodes UTF-8, so a meta and one stray byte must not hide what that reading shows: tag characters, joiners, a fullwidth or UTF-8 phrase. A rule both readings find in one view is one finding. This doubles the text work for a declared file that is not UTF-8 (4 MiB of HTML comments: 3.2 s on f54c57e, 7.5 s here).

The declaration is found the way Chromium finds it:

  • A meta with charset, or with http-equiv="content-type" and a content that names a charset. The last charset attribute wins over any content. The first meta whose label the WHATWG Encoding Standard knows decides; one with an unknown label is passed over.
  • The meta is looked for anywhere in the head and, once the head ends, in the first 1,024 bytes. A tag a head cannot hold ends it; text does not. A meta inside a comment (<!--> and --!> close one), an attribute value, a script (including a doubly escaped one), style, title, textarea, or other raw-text element is text. One inside a noscript counts.
  • Failing a meta, an XML declaration at the first byte (<?xml, then encoding in lower case). A file named .xml, .xhtml, .xht, or .svg is read through its XML declaration first, as a browser reads it, and then through a meta, which a browser does not read in those files but a reader that parses them as HTML does.
  • Labels are the standard's 228, as their encodings decode: latin1 and us-ascii as windows-1252, gb2312 and gbk as gb18030, shift_jis as windows-31j, euc-kr as windows-949. A byte the charset does not define is U+FFFD. The directional marks and joiners that windows-1255, windows-1256, and ISO-8859-8 hold as one byte are dropped, as in a transcoded mail part, because ordinary Hebrew and Persian text carries them.

Nothing is transcoded when the bytes are UTF-8, with one exception: a page declared ISO-2022-JP that holds an escape, whose seven-bit text is valid UTF-8 that no reader shows. A UTF-8 byte order mark, a UTF-8 or UTF-16 label, the replacement encoding (iso-2022-kr, hz-gb-2312), and UTF-7, which is not a label a browser knows, also leave the text as UTF-8. A mail message is not read this way: its parts carry their own charsets.

A page that declares nothing is read as UTF-8, although Chromium guesses its encoding from the bytes and would show windows-1251 text. When a file holds both a meta and an XML declaration naming different charsets, the one its name's parser reads is used, and a browser that parses it the other way reads the other. The record has the comparison with Chromium that set these rules, and probe_html_charset_browser.py repeats it.

Limits

Cyrillic-only words are lowercased in derived folded views so casing cannot hide the existing Russian override phrases. Mixed-script look-alike mapping runs first; ASCII case-sensitive signatures such as DAN, Greek text, and accented Latin letters retain their prior treatment. No new phrase or severity is introduced. The November 7 comparison records the new cases, unchanged corpus outcomes, and the combining-mark gap that remains; RTF text and annotation strings became views afterwards.

It does poorly against a new or targeted payload. YARA matches patterns somebody has already written down. Building a rule from the file you are judging only describes that file, so the tool never does that.

Other limits, measured in this tree with the locked YARA-X 1.23.0:

  • ZIP, tar, gzip, xz, bzip2, ar (a .deb's outer layer, static libraries), newc and crc cpio (initramfs images, RPM payloads), and mail messages (.eml, or the raw bytes a mail hook holds: header lines within 64 KiB, a Content-Type, and a multipart/* body, a base64, quoted-printable, or uuencoded body, an HTML body, or a text body in a charset other than UTF-8 or ASCII; one UTF-8 text/plain body with no encoding stays one unit) are inspected in memory with at most 3 container levels, 1,000 entries, 100 MiB per file/member, and a shared 500 MiB expanded/derived-content budget. Gzip, xz, and bzip2 each count as one level, including in .tar.gz, .tgz, .tar.xz, .txz, and .tar.bz2. A file can hold more than one zip: zipfile reads the archive whose end record comes last, while 7-Zip opens the first of a concatenation, so every earlier end record outside the members already read is opened as an archive too (a record inside a member's bytes belongs to a stored nested zip and is read once, as that member), with at most 16 archives and 64 candidate end records before the scan is incomplete. A zip appended to another file (a PDF, image, or script, or a Python zipapp after its #! line) is read as a zip as well; the shape is not itself a finding, and the outer file's own checks still run. An entry whose local header no directory record lists (before the first member, or between members) is read too, as Java's ZipInputStream and other streaming unzippers read it: the bytes no listed member covers are searched for stored or deflated local headers, and each that inflates is one more member under the same checks. A zip with local headers and no directory at all stays incomplete. A tar without ustar magic (Unix V7) is opened when its first block is a header with a matching checksum, the test GNU tar and tarfile make (record).
  • A PDF whose trailer or XRef stream names an /Encrypt dictionary (encrypted pdf, including #xx-escaped spellings) and an OLE file holding an encrypted Office package or a legacy document encrypted in place (encrypted office file, from stream names and msoffcrypto via oletools, never decrypted) produce incomplete scans: their text is ciphertext to the scanner. Findings from the clear bytes are kept, and incomplete still decides the exit. Owner-password-only PDFs (print or copy restrictions) are encrypted too and are held the same way.
  • Encrypted/unreadable archives, unsupported tar member types, zip symlink/device entries, a zip or tar member name that is absolute (/x, C:\x), has a .. segment, or holds a NUL byte (archive member escapes the archive), a cpio symlink or device, a corrupt xz or bzip2 stream, and containers recognized from their full fixed header that the scanner cannot open (7z, RAR, zstd and its skippable frames, LZ4, cabinet, CHM, WIM, xar, ACE, LHA, thin ar, and odc cpio: unsupported archive) produce incomplete scans. Members are never extracted to filesystem paths; the name check exists so a consumer that unpacks the archive later is not told it was clean. Reasons never carry the member name.
  • Raw bytes and UTF-8 derived views are scanned: BOM-marked or ASCII-heavy UTF-16 (decoded-utf16), text that is not UTF-8 read through the charset its meta or XML declaration names (decoded-charset; declared charsets), HTML entity decoding (html-unescaped), JSON string escapes such as \u0049 in a payload that opens like JSON, meaning { or [ followed by a quote, bracket, digit, minus sign, true, false, or null, after an optional byte order mark, white space, and // or /* */ comments within the first 8,192 characters (JSON with comments, as VS Code reads settings.json, mcp.json, and devcontainer.json), so an INI header such as [InternetShortcut] is not decoded (json-unescaped), the escapes of each YAML double-quoted scalar and TOML basic string in a file named .yaml, .yml, or .toml, (config-unescaped: one piece per escaped scalar or string; YAML uses PyYAML parser events, which keep plain and block scalars literal and leave aliases and tags unresolved; TOML decodes basic-string escapes and multiline continuations; malformed YAML retains only events produced before the syntax error), removal of format controls for inspection (format-controls-removed), base64 runs that decode to readable text (base64-decoded: standard or URL-safe runs of 16 or more characters, line-wrapped runs rejoined, decoded as UTF-8 or as UTF-16LE the way PowerShell's -EncodedCommand writes it, kept only when the result is at least 95% printable and holds a space; padding ends a run, and a wrapped run that does not decode whole is tried line by line; each decoded run is a separate YARA pass, so two runs cannot form one phrase; every candidate is read under the scan deadline, and more than 1,024 readable runs is incomplete; also applied to a PDF's inflated text streams; skipped on a MIME message, whose base64 parts are decoded as members, while its headers' RFC 2047 encoded words get a mime-headers-decoded view), percent-encoding (percent-decoded: %XX escapes and + spaces, when the text holds three escapes or a token of four or more letter words joined by +), look-alike spellings folded to plain letters (text-folded: terminal control sequences removed whole, so ESC [ 1 m before a word does not join it; terminal strings taken out of the line as tmux draws them, an OSC (ESC ]) ending at BEL or ESC \ and a DCS, SOS, PM, or APC at ESC \ only, any other ESC, CAN, or SUB ending one early, with each payload kept as a paragraph after the text because an OSC 0 to 2 payload is a window title shown elsewhere, and a string that never ends left with only its opener removed; then NFKC compatibility forms such as fullwidth letters, control characters, variation selectors (U+FE00 to U+FE0F, U+E0100 to U+E01EF, and the Mongolian free variation selectors), and the combining grapheme joiner U+034F removed, with VT, FF, and NEL read as line feeds rather than removed, so Ignore<VT>all is two words as a terminal shows it, Cyrillic and Greek look-alikes folded to Latin inside a token that mixes scripts, a word hyphenated at a line end rejoined (a letter, -, a line break, then a lowercase letter, so previ- and ous read as previous), letters spaced one apart rejoined, and 0 1 3 4 5 7 @ $ folded inside a token that also has letters, so Ignоre with a Cyrillic о, Ignore, 1gn0r3, and i g n o r e reach the phrase rules without transliterating Cyrillic-only or Greek-only words (Cyrillic-only word case is lowercased); ordinary product codes and spaced initials retain their tested benign outcomes; the json-unescaped, base64-decoded, percent-decoded, mime-headers-decoded, and ooxml-text views are folded the same way, one step deep, as <view>+text-folded, so an escape or encoding stacked on a look-alike spelling is read too, and the two base64 rules are left to the raw bytes in every folded view), the inflated text of a PDF's FlateDecode content and object streams (pdf-flate), and the joined text runs of a Word body, header, footer, notes, or comments part, an Excel shared-strings part, a PowerPoint slide, or a part a renderer draws from, such as a slide master, a chart, or a worksheet text box (ooxml-text: runs within a paragraph are joined, paragraphs end with a newline, entities are decoded, no XML parser runs but comments, CDATA wrappers, and the part's own internal entities are read through, and a part with a single run is left to the member's own views), the paragraphs of an OpenDocument part (odf-text: one line per paragraph, a blank line after each table cell; boundaries), each worksheet inline-string cell whose text is split across two or more rich-text runs, and each cell value (<v>, such as a formula's cached string) that a comment, a CDATA boundary, or an entity splits (ooxml-cell: one YARA pass per cell, so two cells never form one phrase, and phonetic runs are not shown text), each section of a worksheet's page headers and footers (ooxml-header: oddHeader, evenHeader, firstHeader, and their footers, split at &L, &C, and &R, a section switched to twice read as one; && is an ampersand; a field such as &P or &D stands as U+FFFC, which parts no words and joins none; font, size, colour, and toggle codes and any other & code are taken out, as LibreOffice prints them; one YARA pass per section, and a header that holds no & is left to the raw view; verification), the text of each HTML block that inline markup splits in a file named .html, .htm, or .xhtml or that opens with <!doctype html> or <html>, including an HTML mail part (html-text: one YARA pass per paragraph, list item, table cell, heading, or other block, with bold, links, spans, and comments dropped and their text joined, <br> a line break, whitespace collapsed outside <pre>, and script and style bodies skipped; two blocks are never joined, and a block whose text is one run is left to the raw and HTML-unescaped views; two lexical passes, with no computed CSS visibility; the second recognizes additional markup the WHATWG tokenizer discards (<!-->, <!--->, a comment closed by --!>, </>, a bogus comment that holds <, a tag name of any length or spelling, a tag that holds <), and adds blocks beyond the first pass; comment and CDATA reads check the deadline), the text of an RSS or Atom feed, known by its XML root and namespace (feed-text: one piece per text node that a comment, a CDATA boundary, or the feed's own entities split, joined as an XML parser joins it, and one piece per block of escaped or CDATA HTML in RSS description and content:encoded, or in a declared Atom HTML field; Atom text and default text fields keep literal angle brackets after XML decoding, read with both html-text readings, and one piece per block of an Atom text construct of type xhtml (content, summary, title, subtitle, or rights holding an XHTML div), read the same way with element prefixes such as xhtml:p removed; two nodes are never joined; verification), the text of each node of another XML document that a comment, a CDATA boundary, or the document's own entities split, joined as xml.etree joins it, for a document that opens with an XML declaration or is named .xml, .xsd, .xsl, .xslt, .plist, .csproj, .vbproj, .fsproj, .props, .targets, .xaml, .resx, .nuspec, .wsdl, .kml, .gpx, .opf, .ncx, .rdf, .xlf, or .xliff, and is neither a feed nor a document html-text reads (xml-text: one piece per node; a document none of these split gives none, which is every part an office suite writes; escaped markup in a node is not read as HTML, since nothing renders it; verification); a document that declares more than 64 entities, or whose entities expand past 1 MiB, is read without them rather than left incomplete, except a feed or XML document whose text refers to one of the entities it declares, which is incomplete (xml entities exceed 64) because a parser would expand that reference into the text; an Atom end tag may hold white space before its >, and at most 32 distinct construct names are searched for one, the text of each Markdown paragraph, list item, or heading that inline markup splits in a file named .md, .markdown, .mdown, .mkd, .mkdn, or .mdx, or in a .txt or suffixless file that opens with a heading or front matter (markdown-text: emphasis and strong markers, strikethrough, code-span backticks, and underscores at a word edge dropped within each line, never paired across lines, so snake_case stays; bracketed text such as a link, image, reference, or checkbox keeps its text and loses its brackets and target; > quote leaders removed; a blank line, heading, list item, thematic break, code fence, or deeper quote starts a new piece, and fenced code is not read until a closing fence of the same character at least as long; the suffix is taken from the final path component, so a suffixless README inside an archive directory such as project.v1/ is still read; one YARA pass per piece, and a piece with nothing removed is left to the raw view), the text of each block of two or more adjacent line comments in a text unit (comment-text: lines that open with the same //, #, --, ;, or ' leader, or the * lines of a /* block comment, joined with newlines after their leaders are removed, so a sentence wrapped over two comment lines reads whole; an empty comment line, a line of code, or another leader ends a block, a * line outside a block comment is a list item and not read, and a comment after code on the same line is not read; one YARA pass per block, so two blocks never form one phrase), and each Jupyter notebook cell source, stream output, and text/* output stored as a list of two or more lines (notebook-text: a file named .ipynb, or text holding both "cells" and "nbformat" that parses as a JSON object with a cells list; one YARA pass per item, so two cells never form one phrase, a source held in one string is left to json-unescaped, malformed or deeply nested JSON gives no items, and a code cell's source is also read for comment-text), each also folded as <view>+text-folded, with at most 100,000 pieces per view before the scan is incomplete. The PDF view feeds both the YARA rules and the PDF name checks, so a /JavaScript action inside a compressed object stream blocks. Inflated streams that are not text (fonts, images) are dropped; a corrupt stream is skipped; each stream is capped at 100 MiB, the total at the shared 500 MiB budget, and more than 2,000 streams is incomplete. ASCIIHexDecode and ASCII85Decode stages before FlateDecode, LZWDecode, or RunLengthDecode, and PNG or TIFF predictors, are decoded. LZW (with its EarlyChange) and RunLength are decoded in Python, so only for a stream that is not an image and whose encoded bytes are at most 2 MiB (LZW) or 8 MiB (RunLength); a larger one, and an ASCII chain that reaches none of the three, is not, and when that stream is page content it is reported as the note heuristic_pdf_content_unread and counted in pdf_content_unread. Originals remain unchanged. The text a PDF shows is rebuilt from its operators (pdf-text): Tj, TJ, ', and " strings, literal or hex, decoded through each font's ToUnicode map (a bfrange whose destination overflows its width, such as <0000> <00FF> <FFFF>, is read both ways readers resolve it: wrapped to that width, as PDFium, PDFBox, and pdfminer.six do, and as the code point past U+FFFF that pdf.js and MuPDF show, each a reading of the font; verification) or as WinAnsi, or, for a Type0 font with no map whose every /Encoding value is a predefined Unicode CMap (UniGB-UCS2-H, UniJIS-UTF16-H, and the rest of ISO 32000-1 Table 118's Uni*-UCS2 and Uni*-UTF16 names), as UTF-16BE. A sole Unicode reading keeps the full decoded text. When a font name has multiple possible readings, the UTF-16BE reading drops format controls, combining marks, private-use and unassigned code points, and Hangul fillers because other fonts' glyph IDs can map to those characters, a TJ adjustment of -200 or less read as a space, and each block also read with spaces where some readers put them (a TJ gap of -100 or less, and a string on the same line after Td, TD, T*, Tm, or a new text object), added where it differs, so a word placed by position reads as a word: poppler, pypdf, pdfminer.six, and PDFium space a word placed by its own Td or after a -150 gap, and poppler and pdfminer.six one after -110, and each text object (BT ... ET) joined as one block, cut where the text jumps to a new paragraph (more than 1.7 font sizes between baselines; the next line within that joins with a newline). Each block is a separate YARA pass. Generators start a new text object at every font change or line, so a paragraph that spans two or more blocks (a word in bold, a wrapped line) is also scanned whole, as pdf-paragraph; two paragraphs are never joined. Both views are also spelling-folded when different. At most one finding per rule and namespace is kept for each view; the two base64 rules are left to pdf-flate, which tells image samples apart. Text longer than 64K characters is scanned in overlapping 64K chunks, q/Q save and restore the selected font with the rest of the graphics state, a page's later content stream can continue a text object begun earlier, the streams of one /Contents array are read as one stream in array order, as ISO 32000-1 7.8.2 defines it and every reader reads it (a direct array, an array kept in an object of its own, or one in a page dictionary stored in an object stream; an empty member is passed over, a member that is reused, not decoded, or drawn through other fonts ends the run, and at most 1,024 streams are joined), a page content stream is read even when its bytes are not text, as with an inline image or a binary string before the text, when a /Contents entry names it and its dictionary marks it as no image, form, or font program, every token (each TJ array element included) counts toward the deadline, and more than 20,000 blocks, 4,000,000 tokens, or a 100,000-item array is incomplete. heuristic_pdf_text_unmapped is a note: a composite font has no readable ToUnicode map and is not read as UTF-16BE through a predefined Unicode CMap, a font the file never resolves shows two-byte codes, or a string is shown in a font resource name that means more than four fonts, so some text was not read. A page content stream reads each name through its own page's /Resources (direct or behind references; not inherited ones), so the pages of a merged file, which each keep their source's /F1, read one font apiece (page fonts). A stream several pages draw can mean each of their fonts. The page map is built only when some name means two fonts, and only for a file every reader reads the same way; everything else uses every font the name means anywhere in the file, as before the map existed. That covers a name one of the stream's pages does not list or lists as anything but a reference to one font dictionary, a font carried in from an earlier stream, a form XObject, a page with no /Resources of its own or with resources that do not resolve (defined twice with different values, under another generation, or not listed in use by the cross-reference table), and a stream referenced from anywhere but a page's /Contents (an annotation appearance, a Type 3 glyph procedure, a page object that repeats a key, escapes a name with #, or is written in the trailer). It covers the whole file when the file is not written plainly throughout: anything but comments, cross-reference sections, and trailers between objects; an object that is not one well-formed value; a byte outside strings and streams that is not printable ASCII, a tab, or a line end; a number or reference in a spelling only some lexers read (5.0 0 R, 5 0 R#); a comment right after a number; a stream whose /Length does not land; bytes that read as an object header inside a string, a comment, or a stream; a cross-reference entry that does not agree with the object headers, or a table that cannot be read; an object stream that cannot be read, lacks /Type /ObjStm, or does not hold exactly one object in each member's place; and past 1,000,000 tokens, 20,000 page objects, 2 MiB of parsed objects, 250,000 font names given to streams, or 100,000 objects. probe_pdf_reader_agreement.py checks the map against six PDF readers. Each string is decoded through at most four of the fonts its name means (an overflowing bfrange adds a second reading after every font's first), and at most 64 fonts per name are resolved; text only a later font shows is not scanned (verification). A font object or ToUnicode map that the file defines more than once (an incremental update, a number reused under another generation, an object-stream member beside a raw object) is read through every definition, since readers differ in which one they use; each distinct definition is one of those readings, and a font with more than four, or more than 64 raw definitions, is reported the same way (verification). Text a Word, Excel, PowerPoint, or PDF reader hides is also read on its own, one piece per hidden stretch, as ooxml-hidden-text and pdf-hidden-text (each folded as <view>+text-folded); an instruction there adds heuristic_hidden_text (hidden Office and PDF text). Office text inside other parts (settings, styles, embedded objects) is not joined. Hex, ROT13, and base64 nested inside another encoding are not decoded. Annotation /Contents strings are separate pieces (pdf-annotation, folded when the spelling changes), and so are form field values, document information strings, and outline titles (pdf-string, boundaries). So is each distinct /ActualText string of marked content in a content stream (/Span <</ActualText <FEFF...>>> BDC ... EMC), which poppler and PDFium show in place of the glyphs it covers, decoded as a pdf-string is (pdf-actual-text, folded when the spelling changes; boundaries). No general document-text extraction, OCR, or semantic prompt-injection guarantee is provided.
  • PDF CMaps share budgets for tokens, entries and decoded map bytes across fonts. One map keeps at most 131,072 entries; entries past that are not read, without an incomplete result. The scanner checks the 64 MiB map expansion limit before allocating range entries, and limits decoded PDF text to 16,777,216 characters before joining repeated glyphs. Exhaustion yields an incomplete scan.
  • Reads are capped by remaining byte budgets. A shared deadline is checked between extraction chunks and scan/view operations, and YARA receives the remaining time. Third-party archive metadata and Office parsing are not preempted by an OS process deadline.
  • A .pdf that is really a zip is an advisory type mismatch and exits 0. A .pdf that is really a PE, ELF, or Mach-O exits 1. The same check reads .jar, .apk, .msix, .appx, .xpi, .vsix, .nupkg, .whl, .epub, and OpenDocument names as zips; .xll, .cpl, .ocx, .sys, .drv, and .efi as PE; and .csv, .svg, .xml, .rtf, .ini, .url, .scf, .reg, .ps1, .vbs, .js, .sh, .bat, and .cmd as text, so a PE saved under any of those text or zip names exits 1.
  • A scan of a path outside the inbox does not move that file.
  • YARA-X keeps the first 1,000,000 matches of each string and drops the rest without notice, so a rule that reads where a match lies could miss one after a million others, in an input of a megabyte or more. Each bundled rule that does so counts a full list as a match in place: it reports its finding rather than clear the file (record). The scanner also guards every rule, vendor packs included. Each rule file is compiled a second time with a private copy of every string whose count or place a rule reads (a string tested only for presence keeps its first match, so it cannot hide one), which reports any of those lists at the limit, and a view of a million bytes or more is scanned with that set. A view that fills one to four lists is scanned again with a limit of 4,000,000 when the time left allows it, and the report's match_rescan_limit records it. A list still full then, or more than four full lists, makes the scan incomplete (match cap reached) with its findings kept, so the file stays held. A rule file whose guard cannot be built loads as before and is counted in unguarded_rule_files (record).