This page is the reference behind the README. The detection-method map links every layer to its implementation, including checks outside the rules directory. This reference covers the 89 YARA rules shipped in rules/local/, the structural checks the scanner runs without YARA, and the measured limits. Each table gives the indicators a rule requires and its finding severity. verdict_on_match in the rule sets that severity: block, advisory, or note. A note is a weak signal with nothing else behind it: six YARA rules (the three override-phrase rules and the three lone-Unicode-control rules) and five checks (heuristic_invisible_joiner, heuristic_pdf_OpenAction_goto, heuristic_pdf_text_unmapped, heuristic_pdf_content_unread, and agent_encoded_instruction) report one. Of the 89 rules, 77 are advisory, 6 block, and 6 are notes. Under agent-ingest, an advisory or note finding holds the file as block. Under agent-document, an advisory finding holds the file and a note is only recorded, so a notes-only document can pass while any advisory or blocking finding still holds it. Quotation alone is not an exemption. Document callers read a notes-only agent-document record (the read rule), so moving a rule to note widens what an agent may read: treat that as an integration change, not only a precision change.
A match is a pattern that needs review, not proof of malice. Security write-ups, red-team notes, and administration scripts can match the advisory rules on purpose. The near-miss pairs in check_rule_pairs.py record each rule's benign boundary.
These rules target content an AI agent is about to read: messages, retrieved documents, tool results, and repository files.
| Rule ID | What it matches | Finding severity |
|---|---|---|
agent_prompt_injection_phrases |
Phrases that try to override an agent's prompt: ignore, disregard, or forget the previous instructions, the instructions above, or your instructions; do not follow, stop following, or override the previous instructions; reveal your system prompt | Note |
agent_prompt_injection_multilingual |
Override phrases in Spanish, Portuguese, French, German, Italian, Dutch, Polish, Turkish, Indonesian, Vietnamese, Chinese, Japanese, Korean, Russian (including uppercase Cyrillic through text-folded), Arabic, and Hindi |
Note |
agent_role_spoof_action |
A claim of system authority, an all-caps authority header line, or a developer claim, followed by an override, disclosure, or mode-switch request; "this is the system administrator" (or developer, operator) ending its sentence, then at once an order to override, disable, bypass, or ignore your safety, guardrails, content filters, restrictions, or instructions; an OVERRIDE header (SYSTEM OVERRIDE:, [ADMIN OVERRIDE]) that voids the previous rules or instructions or announces new ones on the same line |
Advisory |
agent_fake_chat_marker |
A fake system or assistant chat marker (ChatML, or the assistant turn opener of Phi, Gemma, Llama, harmony, Cohere, DeepSeek, Granite, Kimi, or Inkling) with an instruction verb, or a </tool_result> or </tool_response> close tag that is followed within 80 bytes by an instruction verb and has no open tag of its own kind anywhere in the unit |
Advisory |
agent_tool_output_override |
Tool-result text that asks a command tool to bypass controls | Advisory |
agent_persistent_instruction |
A request to persist an approval or safety bypass in agent guidance or in future (or all future) sessions, tasks, or chats. In the future-sessions form, a bypass that never, not, or don't negates does not count. The guidance-file form (append, write, save, or store to AGENTS.md, CLAUDE.md, MEMORY.md, or memory) has no such exemption today: "Append to CLAUDE.md: do not bypass confirmation" matches |
Advisory |
agent_secret_transfer_request |
A request to read sensitive material and send it to an outbound destination, or a one-line request to send the prompt, hidden instructions, memory file, or your credentials; a request to send me the keys, tokens, secrets, or credentials of a .env file, opening its clause. A key alone, with no .env, is not enough |
Advisory |
agent_message_code_relay |
A request to forward, send, text, or reply with a one-time code (one that carries a type such as verification or 2FA, a digit count, or "you just received", or a PIN or passcode) to a phone number, an email address, a URL, me, or this chat; to forward a received reset or sign-in link; or the "I sent you a code by mistake" story, or "I just sent you a code, forward it to me". A refusal on the same line exempts it |
Advisory |
agent_override_imperative |
An order that names what it acts on: ignore or disregard all, your, or earlier guidance, directives, guidelines, or safety rules; disable, bypass, or remove your filters, guardrails, or safety; a persona (pretend to be, roleplay as, behave as) with no rules or unfiltered; you are no longer bound by; a jailbreak, developer, or sudo mode with no restrictions or root access; DAN mode; a critical update to your instructions; a message from your creators or the IT department that orders compliance or a reset; a standing instruction stored in your memory |
Note |
agent_model_addressed_instruction |
A sentence or HTML text tag introducing a model-specific address, answer hook, or named tool request. The rule requires a following directive or a narrow promotional claim. Concealment counts beside a model address or answer hook, including an instruction to add something to the answer. Inline prose quotations, crawler directions, and unpaired confidentiality text are outside the matched shapes. Standalone prompt examples can still match. | Advisory |
agent_tool_hijack_directive |
An order aimed at the agent's tools: please or to exfiltrate, or exfiltrate the data, secrets, files, or credentials; do not tell the user within 160 bytes of a tool_call, run_command, execute_command, or function call, in either order; invoke a tool to email, send, upload, post, forward, or transfer; use your named tool to forward, send, email, upload, or share, opening its clause, with an outside email address, phone number, or URL, or an order not to tell the owner or user, on the same line; decode and then execute, run, or eval |
Advisory |
agent_markdown_exfil |
A Markdown image, link, or reference-style definition ([label]: https://…) whose URL contains a template ({{ or ${) |
Advisory |
agent_hidden_html |
An instruction-like phrase within 256 bytes after a hiding cue: display:none, visibility:hidden, a font size of zero, under one pixel, or 1 px (0.9em and 0.875rem are small text, not a cue), an offset of -1000 px or more (left, top, text-indent, a margin), a screen-reader-only class (sr-only, visually-hidden), the hidden attribute, or a Markdown comment ([//]: #); or inside an HTML comment, before the comment closes |
Advisory |
agent_invisible_unicode |
A zero-width space, a directional mark, a word joiner, or an internal BOM in a text unit | Note |
agent_invisible_run |
A run of 16 or more consecutive zero-width or invisible formatting characters (U+200B to U+200D, U+2060 to U+2064, U+FEFF, U+180E, U+034F) in a text unit: bytes carried one bit or more per character | Advisory |
agent_bidi_override |
An explicit left-to-right or right-to-left override in a text unit | Note |
agent_bidi_formatting |
A directional embedding or isolate in a text unit | Note |
agent_bidi_source |
Any bidi override, embedding, or isolate in a file whose suffix names a programming or shell language (Trojan Source, CVE-2021-42574) | Block |
agent_base64_blob |
A base64 run of 400 or more characters that is not media, a font, a PEM body, an SSH key, or hex digits only (a PDF hex string of glyph ids or a UTF-16BE outline title is hex text, not base64), or a LibreOffice PrinterSetup item that decodes to exactly the job setup LibreOffice writes and holds no executable or base64 run of its own |
Advisory |
agent_base64_executable |
A base64 run of 400 or more characters whose decoded head is a PE (MZ), ELF, or Mach-O header, in a page, a JSON value, a message, or a data: URI |
Block |
agent_eicar |
The EICAR antivirus test string | Block |
The base64 exemptions apply to each run separately. A run that starts with an image, audio, video, or font header, or that follows a data: media URI, a PEM header, or an SSH key type, is exempt. A second, unrelated run in the same file still matches. CA FE BA BE is not treated as an executable header, because Java class files use the same bytes. Both base64 rules are skipped on the text-folded view: folding look-alike characters can join ordinary words into something that looks like a base64 run. They are skipped on the base64-decoded and percent-decoded views too: those views hold the text a run spells, for the lexical rules, and the run itself is judged in the raw view.
In the pdf-flate view, decoded streams can be image samples. A flat area of pixels whose byte values fall in the base64 alphabet reads as a long run, such as +G9+G9+G9... for RGB (0x2B, 0x47, 0x39). There agent_base64_blob counts a run only if it still has 400 characters after each unit of 1 to 12 characters that repeats four or more times in a row is cut to one copy. What is cut can be rebuilt from the copy that stays, so an exempt run carries no more than a run under 400 characters, which the rule already ignores. Grainy areas, where noise breaks exact repeats, are handled by a second count: the run is read in 64-character windows, and a window is cut when, at one lag of 1, 2, 3, 4, 6, or 8 bytes (the size of a gray to 16-bit CMYK pixel), at least 58 of its characters are within 4 of the character that lag before. The run must keep 400 characters under both counts. Base64 of English text, UTF-16LE or UTF-32LE text such as an encoded PowerShell command, JSON, compressed data, and random bytes reached at most 52 of 64 when measured (base64 of hex text 55), and a contiguous payload of 373 or more characters fires wherever it sits inside grainy padding. The stream dictionary (/Subtype /Image) plays no part, and no stream leaves the view. agent_base64_executable and every other view are unchanged. In the same view, agent_invisible_unicode, agent_bidi_override, and agent_bidi_formatting count a match only when the 16 bytes on at least one side of it decode as UTF-8: pixel bytes can spell E2 80 8B, but between bytes that are not UTF-8 it is not a character. See the image-sample record and the grain and control-byte record.
The model-addressed rule and three phrase rules in the table above are rendered by scripts/generate_rules.py from rules/templates/: agent_prompt_injection_phrases.yar, agent_override_imperative.yar, agent_hidden_html.yar, and agent_model_addressed_instruction.yar. The model address strings share a sentence and HTML tag boundary. The three original templates share word lists (the verbs, the adjectives that mean "earlier", the nouns an order acts on, the markers after "you are now", the hiding cues, and the phrases read after a cue) and the separator between words are in fragments.toml, each written once. The non-greedy copy of the grammar that the documentary exemption and the HTML comment strings need is derived from the same fragments, so it ends where the strings it follows end. The links in the table go to the generated files, which are what a scan compiles; the templates are not packaged and nothing reads them at scan time. Lists differ between strings on purpose (rules is left out after override, forget counts only before the prompt or programming), and each difference is a boundary recorded in the detection backlog. To change one of these rules, see generated rule files.
The scanner finds base64 runs with a linear regex in heuristics.base64_run_signals and hands the results to YARA as globals. As a YARA string, a 400-character class has no literal atoms, and YARA-X spent about 5 s per MB testing it at every offset. Rules about files known by name, such as .envrc and devcontainer.json, read the name the same way. Another YARA tool that loads rules/local/ directly must define all six globals before it compiles:
| Global | Type | Compile default | What the scanner sets per buffer |
|---|---|---|---|
yara_gate_ext |
string | "bin" |
The lowercased suffix of the submitted or member name, without the dot |
yara_gate_name |
string | "" |
The lowercased final component of the submitted or member name (.envrc for repo/.envrc in a zip) |
yara_gate_text |
bool | true |
true when the buffer sniffs as text |
yara_gate_b64_blob |
bool | false |
true when the buffer has a base64 run that agent_base64_blob should report |
yara_gate_b64_exe |
bool | false |
true when a run decodes to an executable header |
yara_gate_svg_scope_proven |
bool | false |
The scanner verified unambiguous SVG tag boundaries for this buffer. Leave false when running the rules standalone. |
If the globals are missing, compilation fails with an error. Leaving both base64 booleans at false turns off the two base64 rules in that tool, and leaving yara_gate_name empty turns off dev_devcontainer_lifecycle and dev_envrc_exec. Nothing else changes.
Installed curated packs add known malware signatures. Local heuristics check Office auto-exec macros (olevba), risky PDF tokens (/JavaScript, /JS, /OpenAction, /AA, /Launch, /EmbeddedFile, /RichMedia), and embedded executable headers. These checks also run on archive members. PDF feature names match complete decoded tokens (/JS1 is not /JS); see the name-boundary verification. PDF token checks remain conservative feature restrictions, with one exception. An /OpenAction that only opens a page is the note heuristic_pdf_OpenAction_goto: a direct destination array such as [3 0 R /Fit], a /GoTo action dictionary with no /Next chain, or a reference whose every definition in the file, written directly or stored in a FlateDecode object stream, is one of those. Every other /OpenAction still blocks as heuristic_pdf_OpenAction. That includes another action type (/JavaScript, /Launch, /URI, /GoToR, /GoToE, /SubmitForm, /ImportData, /Rendition), a missing or non-name /S, a chained /Next, a value that does not parse, and a reference that does not resolve. So a LaTeX-style PDF whose catalog and /GoTo action both sit in an object stream gets the note. Each occurrence is read from its own position, and every one must be a page open, including a redefinition in an incremental update; a copy inside a string or comment is one more value that must pass. An object stream whose first filter is not FlateDecode is not indexed, so a reference into it does not resolve and blocks. The same split applies in the pdf-flate view, whose references resolve against the whole file.
A lone Unicode control in a text unit is a note. That covers a zero-width space, a left-to-right or right-to-left mark, a word joiner, or an internal BOM (agent_invisible_unicode), a bidi override (agent_bidi_override), a directional embedding or isolate (agent_bidi_formatting), and a stray ZWNJ or ZWJ (heuristic_invisible_joiner). One such character is a paste artifact as often as an attack: a zero-width space copied from a web page, or an isolate around an Arabic or Hebrew name. The default policy and agent-document record it, and agent-ingest holds it. When the same member also carries another note, the scanner adds heuristic_unicode_control_conjunction, which is advisory, so agent-document holds the file too. That is how a phrase split by a zero-width space shows up: the phrase matches in the format-controls-removed view. Any bidi control in a file whose suffix names a programming or shell language blocks as agent_bidi_source, because it can make a comment or string literal read differently from what the compiler sees. A Unicode tag character outside an emoji subdivision-flag sequence blocks as heuristic_unicode_tag, including one spelled as an HTML entity or a JSON escape. The England, Scotland, and Wales flags stay clean. A run of eight or more variation selectors (U+FE00-FE0F, U+E0100-E01EF) blocks as heuristic_variation_selector_run, also when spelled as HTML entities or JSON escapes: a variation sequence is one base character and one selector, so a run renders as nothing and is how bytes are hidden after an emoji, one selector each (the encoding GlassWorm used in VS Code extensions). An emoji's U+FE0F, an ideographic variation sequence such as U+845B U+E0100, and a doubled emoji selector stay clean. A run of 16 or more consecutive zero-width or invisible formatting characters (U+200B to U+200D, U+2060 to U+2064, U+FEFF, U+180E, U+034F) is advisory as agent_invisible_run, beside the notes for its characters: a run renders as nothing and carries one bit or more per character, the zero-width encoding that hides bytes in ordinary text. Persian and Indic text use single joiners, emoji put one ZWJ between pictographs, and a pasted ZWSP stands alone, so none of them forms a run; a run of 15 stays the notes (record). These checks read text units only. A unit that does not sniff as text, such as an image, a PDF's raw bytes, or compressed data, holds these byte sequences by chance and is not checked. The first 4 KiB of a unit decide that, so text placed after a binary prefix is not checked either. A unit named .bmp, .dib, .tif, .tiff, .emf, or .wmf whose header validates as that format (as word/media and ppt/media members often are) is scanned like any other unit, but in the samples mode the pdf-flate view uses (above), so its pixels stop reading as base64 or as controls while a PHP webshell, an injected caption, an appended payload, or a zero-width space inside a word is still checked. Both the name and the bytes must say image, so a .php or .py script behind a valid BMP header is not in samples mode and keeps agent_bidi_source. See the raster record. A ZWJ inside an emoji sequence and a ZWNJ or ZWJ between Arabic, Persian, or Indic letters stay clean. A text file's long-base64 detection survives a filename change to .bin.
agent_role_spoof_action reads the system and developer turn openers of the common
chat templates (Llama 3 and Llama 4 headers, harmony <|start|>system<|message|>, Llama 2
<<SYS>>, Mistral [SYSTEM_PROMPT], Phi <|system|>, Gemma <start_of_turn> and
<|turn>, Granite, Cohere, DeepSeek's fullwidth-bar token, and the Kimi and MiniMax
system openers), [start]/[message] role
markers, XML-style system/developer tags, and all-caps SYSTEM:, [ADMIN],
DEVELOPER, or OPERATOR header lines, each followed on the same line by a
disclosure, override, or mode-switch request (System: reboot at 02:00 and
Operating system: Windows stay clean). One-sentence requests are read by other rules.
agent_prompt_injection_phrases, a note, reads "reveal
your system prompt" in the second person (so "the debug view can show the system
prompt" stays clean), "New instructions:" followed by from now on, respond only,
you are now, or ignore. agent_secret_transfer_request reads "send/forward/reply with" the system prompt, hidden
instructions, memory file, or your credentials, with no destination required. The
phrase "you are now" needs a nearby override, unrestricted-role, or jailbreak-persona
term (DAN case-sensitive, developer mode, jailbroken, "without restrictions"),
including inside hidden HTML, so ordinary connection-status text stays clean. See
exact boundaries and measured collisions.
agent_hidden_html reads its cues lexically; no stylesheet is resolved. opacity:0, white
or transparent text, a zero height, and a data-* attribute are not cues, because fade-in
animations, dark themes, and share widgets put them on text a reader sees. A phrase beside
one of those is still a note from the phrase rules. The phrase has to follow the cue: text
after a comment has closed is shown, and React, Vue, and Angular write empty comments
beside the text they render, so a closed comment is not a cue. HTML text a browser does not
show is also read on its own, as Office and PDF hidden text is; see
Hidden HTML text.
agent_markdown_exfil also reads a reference-style definition, the form EchoLeak
(CVE-2025-32711) used to get an image
URL past link redaction. A definition with no template, such as a badge or a footnote
link, stays clean, as does [home]: {{ site.url }}/index.html, which names no host.
The delivery-pattern rules report these IDs:
| Rule ID | Required indicators | Finding severity |
|---|---|---|
web_clickfix_lure |
Verification language ("verify you are human" or "you're human", "I'm not a robot", "human verification", "Verification ID", captcha), a clipboard-copy API (writeText, write, or execCommand('copy')), an instruction to open Run, a terminal, PowerShell, the File Explorer address bar, or Spotlight with Command+Space, and a paste instruction (Ctrl+V, Command+V, or "paste the command") near each other. Each key may sit in its own <kbd> tag |
Advisory |
web_html_smuggling |
Base64 decoding, Blob/URL APIs, and a nearby literal script, executable, installer (.msi, .msix, .pkg), shortcut, disk-image, or archive download filename |
Advisory |
script_encoded_powershell |
An encoded command of 40 or more base64 characters plus a hidden-window flag or an execution policy of Bypass or Unrestricted in a bounded PowerShell command fragment. Each parameter is read in any spelling PowerShell accepts: a prefix (-e, -enco, -win, -exec), after -, --, /, or a Unicode dash, with a window style of hidden, a prefix of it, or 1 |
Advisory |
script_mshta_remote |
A direct HTTP(S) target, or an inline vbscript: or javascript: URL, passed to mshta |
Advisory |
script_regsvr32_remote_scriptlet |
Regsvr32 with a remote /i: or -i: target and scrobj.dll in one bounded command |
Advisory |
script_rundll32_mshtml |
Rundll32 JavaScript protocol invoking MSHTML RunHTMLApplication |
Advisory |
shortcut_powershell_encoded |
Parsed PowerShell target plus encoded and hidden, bypass, or unrestricted arguments, in the same parameter spellings as script_encoded_powershell |
Advisory |
shortcut_mshta_remote |
Parsed MSHTA target with an HTTP(S) argument | Advisory |
shortcut_padded_arguments |
Parsed shortcut whose arguments open with a line break, vertical tab, or form feed, or with 32 or more spaces or tabs: the ZDI-CAN-25373 padding that empties the Target box in Properties | Advisory |
shortcut_interpreter_remote |
Parsed cmd, PowerShell, wscript, cscript, conhost, forfiles, rundll32, regsvr32, msiexec, certutil, or bitsadmin target whose arguments carry an http(s):// URL or a \\host\ path |
Advisory |
These lexical combinations can also occur in legitimate administration, export code, and security documentation; see the delivery-rule sources and boundaries and the attachment-signature research. They do not execute or decode the referenced payload. An IT shortcut that runs cmd /c start https://intranet…, an installer shortcut msiexec /i \\server\share\x.msi, a rundll32 url.dll,FileProtocolHandler https://… link, or a logon script through cscript \\dc\netlogon\… matches shortcut_interpreter_remote; a shortcut to explorer.exe \\fileserver\share, a browser shortcut that opens a URL, and powershell -File C:\… do not. Four leading spaces in a shortcut's arguments stay clean; a leading line break has no benign writer.
These rules look at the file's own format header, not its extension, so a renamed copy still matches. A Windows host acts on these files when Explorer shows, previews, or extracts them, often before anyone opens them. A macOS or Linux gate that syncs them to a Windows machine is still the place to hold them.
| Rule ID | Required indicators | Finding severity |
|---|---|---|
win_internet_shortcut_launch |
An [InternetShortcut] whose URL= is a file:, UNC, or search-ms: target or a local program path, or whose WorkingDirectory= is a UNC, WebDAV, or web location |
Advisory |
win_shortcut_remote_icon |
An [InternetShortcut], .scf [Shell], or desktop.ini [.ShellClassInfo] icon loaded from a UNC path or file: URL |
Advisory |
win_library_unc_location |
A file that opens with a .library-ms or .searchConnector-ms description (after an optional XML declaration) whose <url> is a UNC path (\\host or //host) or a file: URL with a host |
Block |
win_library_remote_location |
Any other description whose <url> is a UNC path, file: URL, or web address, including library XML quoted in a document |
Advisory |
web_search_ms_remote |
A search-ms: or search: link whose crumb=location: is a UNC path (plain or %5C%5C-encoded) or a web address |
Advisory |
win_rdp_resource_redirect |
An .rdp connection file with a full address:s: host that also redirects local drives (drivestoredirect:s: with a value) or opens a RemoteApp (remoteapplicationmode:i:1) |
Advisory |
win_theme_remote_resource |
A .theme section ([Theme], [Control Panel\Desktop], or [VisualStyles]) whose BrandImage=, Wallpaper=, or visual-style Path= is a UNC path or file: URL |
Advisory |
The shapes come from CVE-2025-33053 (a .url whose working directory is a WebDAV share), CVE-2025-24054 (a zipped .library-ms that leaks an NTLM hash), and search-ms phishing. win_library_unc_location blocks because Windows contacts the SMB host as soon as Explorer lists the folder or the archive is extracted, which is the CVE-2025-24054 leak. A library that really lives on a company file server blocks too; release it through review. A CDATA wrapper or entity-encoded backslashes still block (the entities through the html-unescaped view). A comment before the root element, or more than 67 bytes of leading whitespace, drops such a file to the advisory rule; agent-ingest still holds it. An enterprise search connector to an https:// SharePoint site, a local file:/// location, or a folder icon on a file server also matches. Those stay advisory: the default policy allows them and agent-ingest holds them. A .url to an https:// page, a local icon, and a knownfolder: library do not match. The .rdp shape is the rogue RDP lure: a signed connection file that maps the victim's drives and clipboard into an attacker's RemoteApp session. The .theme shape is the Windows Themes credential leak (Akamai, CVE-2024-21320, with the CVE-2024-38030 and CVE-2025-21308 variants), where BrandImage, Wallpaper, or the visual-style Path points at a UNC share. Any .rdp that mstsc saves with Local Resources → Drives ticked (it writes drivestoredirect:s:*), not only an administrator's, and a corporate theme whose wallpaper lives on a file server match and stay advisory. A file saved by mstsc with an empty drivestoredirect:s: and remoteapplicationmode:i:0, a theme with local paths, and an INI that is not a theme do not match.
These rules key on the file's own markup or header, not its extension. Each file type opens with a double-click on Windows 10, Windows 11, and Windows Server 2022 and later, and several run before the reader sees any content.
| Rule ID | Required indicators | Finding severity |
|---|---|---|
win_hta_shell_object |
An <hta:application> element plus WScript.Shell, Shell.Application, ADODB.Stream, or an XMLHTTP object |
Advisory |
win_wsf_remote_script |
A <job> or <package> root plus a <script src=> that points at a URL or a UNC path |
Advisory |
win_msc_console_remote |
An <MMC_ConsoleFile> whose <String> holds a URL, javascript:, vbscript:, or a UNC path, or that names apds.dll (the GrimResource shape) |
Advisory |
win_reg_run_interpreter |
A registry export whose Run, RunOnce, RunServices, or Winlogon value starts powershell, mshta, wscript, cscript, rundll32, regsvr32, certutil, bitsadmin, msiexec, or cmd /c, or that sets an IFEO Debugger |
Advisory |
win_script_encoded_body |
A Windows Script Encoder body: the #@~^ opening marker with its six-character length field and ==, plus the ==^#~@ closing marker, the .vbe/.jse form the script engines decode at run time |
Advisory |
win_script_host_fetch_run |
CreateObject(, ActiveXObject(, or GetObject( plus a download or file-write class (MSXML2.XMLHTTP, Microsoft.XMLHTTP, WinHttp.WinHttpRequest, ADODB.Stream) plus WScript.Shell or Shell.Application followed by .Run, .Exec, or .ShellExecute; the body of a .vbs, .js, .jse, or .wsf without <hta:application> |
Advisory |
win_cmd_fetch_start |
A downloader (iwr, irm, Invoke-WebRequest, curl, wget, certutil -urlcache, bitsadmin /transfer) that names a runnable suffix, then start, call, cmd /c, an interpreter (msiexec, wscript, mshta, powershell, …), or .\ launching a runnable suffix on the same line or the next |
Advisory |
doc_onenote_embedded_launcher |
A .one header plus a FileDataStoreObject whose payload starts with MZ, a shell-link header, or script text (<hta:, <script, @echo, WScript.Shell, powershell, CreateObject(, …) |
Advisory |
doc_spreadsheet_formula_exec |
A CSV, SYLK, or IQY cell that starts a DDE launch (=cmd|…), calls DDE(, WEBSERVICE(, or an IMPORT*( function, calls a Win32 DLL from an Excel 4.0 macro, or an IQY WEB header followed by a URL |
Advisory |
web_ms_protocol_lure |
An ms-msdt: link with IT_BrowseForFile, IT_LaunchMethod, PCWDiagnostic, or $( (CVE-2022-30190), or an ms-appinstaller:?…source=https:// link |
Advisory |
win_lolbin_download |
certutil -urlcache or -verifyctl, bitsadmin /transfer or /addfile, or msiexec /i, /package, or /a (or -i) with an HTTP(S) URL on the same command, or finger user@host piped into cmd, PowerShell, or a Unix shell, or read by for /f … in ('finger user@host') do call (the 2025 and 2026 ClickFix stage) |
Advisory |
win_msbuild_inline_task |
TaskFactory="CodeTaskFactory" or "RoslynCodeTaskFactory" and a <Code element, plus VirtualAlloc, VirtualProtect, CreateThread, CreateRemoteThread, WriteProcessMemory, NtAllocateVirtualMemory, GetDelegateForFunctionPointer, Marshal.Copy, Assembly.Load(, or a kernel32 DllImport; or a download call (DownloadData, DownloadString, DownloadFile, GetByteArrayAsync, GetStringAsync, WebRequest.Create, HttpWebRequest) with FromBase64String or Process.Start |
Advisory |
win_inf_scriptlet_install |
An INF with Signature=$chicago$ or $Windows NT$ whose OCX line points scrobj.dll at an HTTP(S) or UNC scriptlet, or whose [RunPreSetupCommandsSection] runs PowerShell, mshta, regsvr32, rundll32, certutil, bitsadmin, wscript, cscript, or cmd /c (the cmstp and InfDefaultInstall shape) |
Advisory |
win_browser_credential_decrypt |
Login Data, encrypted_key or app_bound_encrypted_key, and CryptUnprotectData or ProtectedData.Unprotect in one file: the Chromium saved-password decryption at the core of an infostealer |
Advisory |
An administrator's HTA that shells out, a console that links an intranet page, a .reg that registers a legitimate tool, a notebook with a .lnk attachment, an IQY against an internal server, a vendor site that still uses ms-appinstaller:, and a deployment script that runs msiexec /i https://… all match, as does a deployment VBScript or batch file that downloads an installer and runs it, and a legacy administration script that was run through the Script Encoder. They stay advisory. A notebook whose attachment is a PDF or an image, an HTA without a shell object, a write-up that only names the #@~^ and ^#~@ markers, a logon script that only maps a drive through WScript.Shell, an inventory script that only posts through XMLHTTP, the old ActiveXObject("Microsoft.XMLHTTP") AJAX polyfill, a download with no launch, a .reg that runs an ordinary program, a CSV formula such as =SUM(A1:A2), an ms-settings: link, and certutil -hashfile do not match. Microsoft disabled the ms-appinstaller: handler by default in December 2023 after Storm-0569, Storm-1113, Sangria Tempest, and Storm-1674 abused it; older or re-enabled installs still honor it.
Loaders and ransomware droppers turn off protection and recovery before the payload runs. These rules read the commands in a .ps1, .bat, .cmd, .reg, script body, or pasted text, in UTF-8 or UTF-16LE.
| Rule ID | Required indicators | Finding severity |
|---|---|---|
win_defender_tamper |
Set-MpPreference -Disable{RealtimeMonitoring,BehaviorMonitoring,IOAVProtection,ScriptScanning,BlockAtFirstSeen,IntrusionPreventionSystem,ArchiveScanning} with $true, 1, or no value; Add-MpPreference or Set-MpPreference -Exclusion{Path,Process,Extension} (Add appends to the list, Set replaces it) of a drive root, a temp, profile, Public, or ProgramData folder, a script interpreter, or a runnable suffix; a reg add, Set-ItemProperty, New-ItemProperty, or .reg line that sets DisableAntiSpyware/DisableRealtimeMonitoring/… to 1; sc stop/delete WinDefend, sc config WinDefend start= disabled, or Stop-Service WinDefend |
Advisory |
win_amsi_bypass |
AmsiUtils and amsiInitFailed, amsiContext, or amsiSession within 400 bytes; or AmsiScanBuffer, AmsiOpenSession, or EtwEventWrite and VirtualProtect within 2 KB |
Advisory |
win_recovery_inhibit |
vssadmin delete shadows, wmic … shadowcopy … delete, Win32_ShadowCopy piped to .Delete() or Remove-CimInstance, bcdedit /set … recoveryenabled no or bootstatuspolicy ignoreallfailures, or wbadmin delete catalog/systemstatebackup/backup |
Advisory |
A hardening guide that shows how an attacker disables Defender, a red-team write-up that quotes the AMSI one-liner, and a backup runbook that prunes shadow copies all match and stay advisory. Get-MpPreference, Set-MpPreference -Disable… $false, an exclusion for a build folder such as C:\src\repo, a DisableAntiSpyware value of 0, sc query WinDefend, sc config WinDefend start= auto, prose that explains AmsiScanBuffer or VirtualProtect alone, vssadmin list shadows, and bcdedit … recoveryenabled Yes do not match. Sources: Set-MpPreference, MITRE T1562.001, and MITRE T1490.
rules/local/ in this tree has 89 YARA rules in 59 files. Recount them with rg -N '^rule ' rules/local. The tables above and below list their own families; these are the rules none of them lists.
| ID | Severity | What it matches |
|---|---|---|
script_base64_decode_pipe |
Advisory | base64 decode (-d, -D, -di, --decode) piped into sh, bash, or zsh, named by path or through env and optionally under sudo, or into python or perl with no program argument; or the decode inside bash -c "$(…)" or eval "$(…)" |
script_osascript_shell |
Advisory | osascript -e, or osascript reading a here-document, running do shell script |
script_clear_quarantine |
Advisory | An xattr or /usr/bin/xattr command (optionally after sudo) that deletes com.apple.quarantine (-d, -rd, -r -d, -d -r) or clears every attribute (-c, -cr) |
script_launchagent_shell |
Advisory | LaunchAgent program arguments that run /bin/sh, bash, zsh, or /usr/bin/python3, perl, ruby, osascript, curl, or env, with curl, wget, osascript, base64, or urlopen in the plist |
script_chmod_execute |
Advisory | chmod +x, u+x, or an octal mode, then a launch (./, a shell, a /tmp path, or ~/, optionally after sudo, nohup, setsid, or exec) on the next line or after && or ; |
doc_rtf_object |
Block | RTF object data or an object update |
doc_svg_active |
Block | An SVG with a script or an onload handler. In standalone SVG or when scope is ambiguous, the rule retains its strict match. In ordinary markup with proven SVG scope, a sibling page script or body handler does not count. |
doc_svg_event_handler |
Advisory | An SVG animation with a non-empty onbegin handler or an image, feImage, svg, or use element with a non-empty onerror handler. Uses the same scope proof and strict fallback. Whether the handler runs depends on the embedding. |
doc_svg_active_link |
Advisory | An SVG that runs or loads something with no script element or handler: an a element whose href or xlink:href is a javascript: URL, a set or animate whose to, from, or values writes one, or a foreignObject that holds an iframe, object, embed, form, or meta refresh before it closes. An empty, ;, or void(0) target (the legacy no-op link), https and fragment links, and div, span, or p labels in a foreignObject (diagram exports) stay clean. Uses the same scope proof and strict fallback (record). |
doc_external_template |
Advisory | One OOXML <Relationship> element whose Type ends in /attachedTemplate, with TargetMode="External" and an http(s):// or UNC Target, or an RTF file whose \template control word is followed within 256 bytes by an http(s):// or UNC path (T1221) |
script_download_execute |
Advisory | curl or wget piped into a shell or interpreter, bash <(curl …), eval "$(curl …)", source <(curl …), a PowerShell download or DNS lookup on the same line as iex, or a python -c one-liner that fetches and runs the result (details below) |
container_disk_image |
Advisory | ISO, VHD, or DMG magic at the offset that format uses |
script_download_execute matches curl or wget piped (| or |&) into sh, bash, zsh, python, or perl, with or without sudo in between, and with the interpreter named by path or through env (| /bin/bash, | /usr/bin/env sh); a download piped into node with no program argument (curl … | node, the one-liner in the 2026 VS Code task lures; | node parse.js reads it as data); bash <(curl …), sh -c "$(curl …)", eval "$(curl …)", and source <(curl …); a PowerShell download (Invoke-WebRequest, Invoke-RestMethod, DownloadString, iwr, irm) or a DNS lookup (Resolve-DnsName, the TXT-record stage) on the same line as Invoke-Expression or iex; and a python -c one-liner whose quoted program both fetches (urlopen, urlretrieve, requests.get, http.client) and runs the result (exec(, eval(, os.system(, subprocess.run(, …), in either order. A Python file that fetches and spawns a process is a program, not a one-liner, and stays clean, as does a one-liner that only prints what it fetched. The -c match wants python, python2, or python3 followed by a space; python3.12 -c, combined flags such as -Ic, and a program split over several lines are not covered. The Homebrew install line is the known benign collision; an installer documented as curl … | /bin/bash or eval "$(curl …)" is the same collision in another spelling. VERSION=$(curl …) is an assignment and curl … | /usr/bin/shasum is a checksum, and both stay clean. It does not match a file that also contains "the following command", "common lure", or "for example" within 160 bytes before one of those download words or python. script_base64_decode_pipe treats python or perl as the sink only when nothing follows it but -, so base64 -d | python3 -m json.tool stays clean. That exception covers the whole file. An SVG foreignObject without a script or an onload handler does not match doc_svg_active. A disk-image filename without the magic bytes does not match container_disk_image.
| Rule ID | Required indicators | Finding severity |
|---|---|---|
linux_desktop_exec_fetch |
A [Desktop Entry] whose Exec= line runs curl, wget, base64 -d, xxd -r, or openssl enc |
Advisory |
script_fetch_chmod_run |
One line that runs curl or wget, then chmod (+x, u+x, or an octal mode), then launches a path (/, ./, ~/, or a variable) |
Advisory |
web_applescript_url_lure |
An applescript://com.apple.scripteditor?action=new link whose prefilled script contains do shell script, curl, or osascript |
Advisory |
macos_password_prompt |
A display dialog whose text names a password, passcode, passphrase, or credentials and whose answer field is with hidden answer |
Advisory |
macos_keychain_dump |
security find-generic-password or find-internet-password with -g or -w (prints the secret), or security dump-keychain -d |
Advisory |
unix_privilege_persistence |
A NOPASSWD: ALL line appended to /etc/sudoers by a redirect or through tee, a write to /etc/ld.so.preload, a downloaded key appended to authorized_keys, an insert into access on TCC.db, or, on one line, unset HISTFILE, export HISTFILE=/dev/null, or history -c followed within 120 bytes by unset HISTFILE, history -c, or a history-file delete |
Advisory |
macos_gatekeeper_disable |
spctl --master-disable or --global-disable, or defaults write com.apple.LaunchServices LSQuarantine set to NO, false, or 0 |
Advisory |
macos_archive_upload |
ditto -c -k followed within 4 KB by curl with -F/--form name=@…, or --data-binary/-T/--upload-file of a path (the AMOS and Poseidon exfiltration step) |
Advisory |
macos_stealer_password_check |
dscl . -authonly (or /Local/Default) in the same file as Login Data (also written Login\ Data, as a shell script escapes it), Cookies.binarycookies, key4.db, logins.json, or a wallet name (Exodus, Electrum, MetaMask or its extension id, Atomic Wallet, Coinomi): the AMOS and Banshee password check before collection |
Advisory |
macos_jxa_remote_exec |
osascript -l JavaScript, ObjC.import(, or Application.currentApplication() with dataWithContentsOfURL, stringWithContentsOfURL, or NSURL.URLWithString plus eval(, or a doShellScript( string that runs curl, wget, a base64 decode, or chmod +x |
Advisory |
script_reverse_shell |
bash -i or bash -l redirected to /dev/tcp/host/port, exec N<>/dev/tcp/… with a descriptor of up to three digits, a shell with stdin, stdout, and stderr on one numbered descriptor (sh -i <&3 >&3 2>&3, the PHP and Ruby one-liners), a Perl open(STDIN,">&S") followed by exec of a shell, nc/ncat with -e/-c/--exec and a shell, the mkfifo … sh -i … nc relay, socat joining exec:sh to tcp:, a Python socket.socket( followed by os.dup2(…fileno() or pty.spawn(, or a PowerShell Net.Sockets.TCPClient( stream read into iex |
Advisory |
dev_vscode_autorun_task |
A "runOn": "folderOpen" task with, within 2 KB, a downloader (curl, wget, iwr, irm, Invoke-WebRequest, DownloadString), node -e, python -c, base64 -d, or powershell -enc, or within 1 KB "reveal": "never" |
Advisory |
dev_python_pth_exec |
A .pth file (or .pth archive member, such as one inside a wheel) with a line that starts import and calls exec, eval, compile, a base64/base85/zlib/marshal decode, subprocess, os.system, os.popen, urlopen, or urlretrieve |
Advisory |
dev_gyp_command_expansion |
A file with the suffix .gyp or .gypi (binding.gyp, or one inside an npm .tgz) whose <!( or <!@( command expansion, on one line, sends output to /dev/null or chains with &&, ||, or ; to echo a .c, .cc, .cpp, or .cxx name. node -p and node -e lookups such as require('node-addon-api').include_dir, pkg-config, a node -p test that uses && inside JavaScript, and 2>&1 piped on to another command stay clean |
Advisory |
dev_git_config_exec |
A git config (a [core], [pager], [filter "…"], or [diff "…"] section) whose fsmonitor, sshCommand, pager, editor, askPass, pager.<command>, filter clean/smudge/process, or diff textconv value on the same line runs a downloader (curl, wget, nc, Invoke-WebRequest, …), sh -c, cmd /c, encoded or inline PowerShell, python -c, node -e, perl -e, base64 -d, /dev/tcp/, mshta, or osascript; or a hooksPath in /tmp, /var/tmp, /dev/shm, /Users/Shared, ~/Downloads, $TMPDIR, %TEMP%, C:\Windows\Temp, C:\Users\Public, or on a UNC share. Each key counts only in its own section (no other section header between them), so [alias] entries are not read |
Advisory |
dev_devcontainer_lifecycle |
A file named devcontainer.json or .devcontainer.json whose initializeCommand (it runs on the host) has a downloader, certutil, bitsadmin, sh -c, cmd /c, PowerShell, python -c, node -e, base64 -d, /dev/tcp/, mshta, or osascript; or whose onCreateCommand, updateContentCommand, postCreateCommand, postStartCommand, or postAttachCommand decodes base64, runs encoded PowerShell, opens /dev/tcp/, or runs inline Python or Node that reaches the network or execs. The indicator must sit inside a JSON string of that key's value (a string, or one in its array or object, which may span lines), so a later property is not read. An installer piped to sh inside the container is left to script_download_execute |
Advisory |
dev_envrc_exec |
A file named .envrc that runs curl, wget, nc, ncat, Invoke-WebRequest, or Invoke-RestMethod, decodes with base64 -d or openssl enc … -d, opens /dev/tcp/, runs osascript -e, or runs python -c with urllib, socket, b64decode, exec(, or subprocess. source_url with its pinned hash, use flake, layout, dotenv, and eval "$(pyenv init -)" stay clean |
Advisory |
agent_config_approval_bypass |
"chat.tools.autoApprove": true or "chat.tools.global.autoApprove": true (CVE-2025-53773), a "chat.tools.terminal.autoApprove" (or deprecated chat.agent.terminal.autoApprove) pattern that matches every command, such as "/.*/": true, "github.copilot.chat.claudeAgent.allowDangerouslySkipPermissions": true, "defaultMode": "bypassPermissions", "enableAllProjectMcpServers": true in the same file as an "ANTHROPIC_BASE_URL" override (CVE-2026-21852), or a Codex approval_policy = "never" with sandbox_mode = "danger-full-access" in a .toml file |
Advisory |
agent_cli_secret_sweep |
--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox, --trust-all-tools, --allow-all-tools, --yolo, or --approval-mode yolo within 256 bytes after claude, gemini, codex, copilot, cursor-agent, opencode, qwen, or q chat, and within 2 KB a wallet, keystore, seed phrase, mnemonic, wallet brand, id_rsa/id_ed25519, .ssh/, private key, or inventory.txt (the Nx s1ngularity shape) |
Advisory |
script_credential_upload |
One command that pipes an SSH private key, .aws/credentials, .git-credentials, .netrc, .npmrc, .pypirc, a Docker or kube config, gh/hosts.yml, a login keychain, or a .env file into curl, wget, nc, ncat, socat, or a PowerShell web cmdlet, or attaches one with curl -d/--data*/-F/-T, wget --post-file, or -InFile; the same file read through $(cat …) inside a curl or wget command; an archive of the whole .ssh folder piped to one of those tools; or the process environment sent out (env | curl …, printenv | base64 | nc …, curl -d "$(env)" …) |
Advisory |
dev_github_workflow_secrets_dump |
A workflow (a runs-on: or steps: key) that expands ${{ toJSON(secrets) }} with, within 1 KB, curl, wget, nc, ncat, Invoke-WebRequest, Invoke-RestMethod, iwr, irm, base64, webhook.site, fetch(, requests.post, or upload-artifact, unless that expansion is the whole value of an input under with: (after up to 16 single-line sibling inputs) |
Advisory |
dev_github_workflow_runner_backdoor |
A run: step, single-line or a block scalar within its first nine lines (indented deeper than the run: key, at key columns 2 to 20, with an optional comment after the indicator), that interpolates ${{ github.event.<discussion, issue, comment, review, review_comment, or pull_request>.<body or title> }}, with runs-on: self-hosted (any letter case; bare, or first in a flow or block list) or a RUNNER_TRACKING_ID override earlier in the same job, or RUNNER_TRACKING_ID in the top-level env: before jobs:. A job header is a key at indent 2, 3, or 4 whose first key line, after any comment or blank lines, sits at twice that indent |
Advisory |
The GitHub Actions rules cover a cloned or synced repository that carries a planted workflow, the persistence the 2025 npm worms left behind. In September 2025 Shai-Hulud committed a workflow that serialized every secret with toJSON(secrets) and posted it to a webhook; a later variant wrote it to a file and published it as an artifact. In November 2025 Shai-Hulud 2.0 registered victims as self-hosted runners and added discussion.yaml, which echoes the discussion body on that runner and overrides RUNNER_TRACKING_ID so spawned processes survive the job. Anyone who opens a discussion runs commands on the machine. Passing toJSON(secrets) to an action input (the secrets-to-env pattern, even when a later step uploads build output or decodes a keystore), a single named secret in an API call, event text read through env:, with:, or if:, and event text in a run: step on a GitHub-hosted runner do not match. The last is the common injection weakness, not a planted backdoor; a rule for it would hit many ordinary repositories. A secret dump split across steps more than 1 KB apart, a self-hosted label that is not first in its list, a runs-on: written after the job's steps, a job beyond the first 32 self-hosted or RUNNER_TRACKING_ID lines, and a run step that reads the field through a shell variable all pass. So does an env: value placed after a block-scalar run:, and a self-hosted runner in a different job from the interpolating step, including when that job opens with a comment or blank line. A job-level environment: or env: map whose child is name: stays part of its job. Still unmatched: flow-style steps (steps: [{run: …}]), a dump split across format() or re-encoded with xxd or gh gist, and event fields written in another letter case. A dump exported to $GITHUB_ENV through env: still matches when an unrelated curl (a health check, say) sits within 1 KB; telling those apart needs step boundaries the rule does not track.
The .desktop shape is the APT36 campaign that shipped a launcher named like a PDF inside a zip. The download, chmod, run chain and xattr -c are the AMOS stealer bootstrap. Script Editor links are the ClickFix variant that avoids Terminal. A vendor install line such as curl -o install.sh … && chmod +x install.sh && ./install.sh also matches script_fetch_chmod_run. The password dialog is the AMOS-style stealer prompt; a dialog that asks for a project name, or hides an API token without naming a password, stays clean. security find-generic-password -s Chrome without -g/-w, security add-trusted-cert, a locally generated public key appended to authorized_keys, sudo visudo, a sentence warning against NOPASSWD, and history | tail stay clean. An operator's own hardening or cleanup script can match the persistence rule; it stays advisory. A chain that ends in sudo mv or an install step, a chmod on a later line, a .desktop file that only mentions curl in Comment=, and a Script Editor link that only shows a dialog do not match.
dev_vscode_autorun_task is the repository lure from the 2025 fake-job-interview campaigns: a project whose .vscode/tasks.json runs a hidden or downloading task as soon as the folder is opened in VS Code or a fork of it. It matters for an agent that clones a repository or reads a synced project folder before a person opens it. A watch or npm install task on folder open without a hidden terminal or a fetch does not match. Pentest cheat sheets, a socat port forward, nc -zv, echo > /dev/tcp/host/22 port checks, spctl --status/--assess, and ditto -c -k followed by notarytool do not match; a write-up that quotes a full reverse shell or a forum answer that says to run spctl --master-disable does, and stays advisory. Sources: VS Code task run behavior, MITRE T1059.004, T1553.001, and T1560.001.
The build and install rules cover code that runs because a project is built or a package is installed, before anyone opens a file. MSBuild compiles and runs an inline task during a build (T1127.001), and RoslynCodeTaskFactory is common in legitimate .targets files, so win_msbuild_inline_task needs memory-allocation, assembly-loading, or download-and-decode calls; Process.Start("git", …) alone stays clean. win_inf_scriptlet_install follows the cmstp INF shape (T1218.003). Connection Manager profiles use RunPreSetupCommandsSection too, so the section must launch a script host or downloader. Site-packages runs any .pth line that starts with import at every interpreter start (T1546.018). The LiteLLM 1.82.7 and 1.82.8 packages shipped one. dev_python_pth_exec reads the yara_gate_ext global (the suffix), so a .py file with the same line, setuptools' distutils-precedence.pth, virtualenv, and editable-install .pth files stay clean. npm runs node-gyp rebuild for a package with a root binding.gyp and no install script, and GYP runs every <!(…) expansion as a shell command while it configures, so a planted expansion runs at install with no lifecycle script to review (T1195.002). The Miasma wave wrote <!(node index.js > /dev/null 2>&1 && echo stub.c) (JFrog, MAL-2026-6526). Native addons use the same syntax for ordinary lookups, so dev_gyp_command_expansion holds only the two parts that hide the run: discarded output and an echoed stub source name (record). ClickOnce deployment manifests are not matched: a remote deploymentProvider is the normal shape of a published ClickOnce application.
The checkout-configuration rules cover files that run a command because a tool opens the folder, not because someone runs the file. Git reads .git/config on every git status, diff, log, or fetch an agent or IDE runs, so a zipped or synced checkout that carries a core.fsmonitor, sshCommand, pager, or filter-driver command runs it then; a clone does not copy this file, but an archive or sync folder does. dev_git_config_exec reads the section headers, so the file needs no particular name. hooksPath is reported only for temp, shared, and network folders: Husky writes core.hooksPath = .husky/_, so a repository-relative path is ordinary. A dev container's initializeCommand runs on the host before the container exists, while its other lifecycle commands run inside it, where downloading an installer is normal; dev_devcontainer_lifecycle holds the two to different indicator sets. direnv runs .envrc as a shell script once the folder is allowed. Both name-scoped rules read the yara_gate_name global (a member path written with backslashes is split on them, and trailing dots and spaces are dropped, as Windows reads it), so the same text in init.json or setup.sh is not reported by them. npx -y <package> in an MCP or dev container config is not matched: it is the normal way those configs start a server.
The credential-store rules need the decryption or password check, not a path list. Sandbox deny-lists, cleaners, YARA rules, and threat reports name Login Data, key4.db, and logins.json too. macos_password_prompt already matches the hidden-answer dialog; macos_stealer_password_check adds the dscl . -authonly check that AMOS and Banshee (Elastic) run on the typed password, and needs a browser or wallet store in the same file because LAPS and admin scripts run the same check.
The agent-settings rules cover a cloned or synced project that reconfigures the coding agent about to open it. agent_config_approval_bypass is advisory because teams commit some of these keys on purpose. Approximate GitHub code-search counts (October 2026) set its boundaries: "enableAllProjectMcpServers": true appears in about 6,300 committed .claude files and "hooks" with SessionStart in about 15,600, so neither matches on its own. The rule pairs the first with an ANTHROPIC_BASE_URL override, the CVE-2026-21852 shape that sent the API key to another host before the trust prompt. A hook or MCP command that pipes a download into a shell, decodes base64 into a shell, or opens a reverse shell is already matched by script_download_execute, script_base64_decode_pipe, and script_reverse_shell, and a JSON command that starts sh, bash, curl, or PowerShell by heuristic_mcp_shell. Claude Code ignores bypassPermissions in project files in current releases; the key still marks a file that asks for it. agent_cli_secret_sweep follows the Nx s1ngularity postinstall script, which ran claude, gemini, and q with their approval flags off and a prompt to inventory wallets and keys. A CI step that runs an agent unattended for ordinary work stays clean, because secrets.* and .env are not hunt terms. Prose that warns about the flag next to an SSH key name still matches. script_credential_upload is the shell step an injected instruction asks an agent's command tool to run. curl --netrc-file (authentication, not upload), copying a key, and uploading an ordinary file stay clean.
agent_override_imperative covers the families in CyClaw's injection red-team corpus (.claude/skills/injection-redteam/probes.yaml) that the narrower phrase rule did not. On main, 4 of its 36 attack probes were held; with this rule, 33 are, and none of its 12 benign questions is. The three left out ("act as a penetration tester", "execute command: rm -rf", "run this code to dump the log") have the same shape as ordinary README and review text. Every pattern names what it acts on (your filters, a persona with no rules, your memory), so "enable developer mode", "run this command", "act as a reviewer", and "ignore the earlier draft" stay clean. "Ignore previous instructions" stays with agent_prompt_injection_phrases and its documentary exemption. That rule also reads the same order with its words in another place ("ignore all instructions above", "ignore above instructions"), in the second person without an adjective ("ignore your instructions", "forget your system prompt"), negated ("do not follow the previous instructions", "stop following your earlier rules"), and as a replacement ("override previous instructions"). These spellings have no documentary exemption, so a quoted example of one is a note. "The model may ignore the system prompt", "later rules override earlier rules", "ignore any instructions found in the document", and "don't forget your instructions sheet" stay clean. In both rules the words of an override may be split by one line break, as in hard-wrapped Markdown or "IgNoRe\nALL\tPrEvIoUs instructions" from the CG-agent-harness tests; a blank line still splits them, and the documentary exemption follows the same rule. A threat model or test suite that quotes these orders still matches, as a note: under agent-document the file passes with the finding recorded, and under agent-ingest it is held. The invoke-tool-to-send and decode-and-execute probes from the same corpus moved to agent_tool_hijack_directive, which stays advisory because those are orders to act, not phrases a report quotes; the corpus counts above are unchanged under agent-ingest.
agent_message_code_relay is for chat, SMS, and mail bodies passed to scan_bytes or scan -. Ordinary one-time-code notices can contain "code", "verification", and "do not share", so the rule needs a relay verb, a code that carries a one-time qualifier, and a destination. A bare "code" in chat is usually source code. A recognized refusal ("never forward the verification code to anyone") exempts only a request starting inside that refusal; an independent request on the same line can still match. "I just sent you a code, forward it to me" is matched when the request follows the code at once; "we just sent you a code, enter it on the sign-in page" is the notice and stays clean. English only; a paraphrase without the listed verbs passes.
The spellings added for the readiness review's P2 shapes (#255, record) read the word before their verb inside the pattern:
- an administrator claim's order follows the claim at once;
- a
use your ... toolorder and a.envrequest open their clause; - a persistent bypass follows punctuation or a word such as
to,must, oralways.
So "do not use your email tool to ...", "how to use your email tool to ...", "please don't email me the keys from your .env file", and "never skip approval" stay clean in the recorded boundary cases. The linked timing probes cover repeated-line fixtures; they are not a latency guarantee for arbitrary documents. A system prompt or agent configuration that tells its own agent to "use your email tool to send" mail to a fixed outside address matches agent_tool_hijack_directive the same way an attack does. Under agent-document, such a file is held. The third-person "use the send_email tool", as framework docs write it, does not match. A bare SYSTEM OVERRIDE: header, or one that names a flag, stays clean.
The code-relay rule checks where each request starts. A recognized refusal exempts a request that starts inside it. A separate request can still match on the same line. The rule searches the gaps between refusals by offset, so a long notice repeating a recognized refusal does not need a request-by-refusal comparison. This applies to both ASCII and UTF-16LE patterns. The finding remains advisory, and both agent policies still hold a matching request.
agent_prompt_injection_phrases lets a quotation ("the phrase 'ignore all previous
instructions'") cover the one override it ends with. Any other override counts. This
includes an override before the first quotation, one between quotations, and one after
a quotation ends. The rule finds them by searching the gaps between quotations by
offset, as the code-relay rule does
(#274,
record). A file of repeated
quotations scans in linear time and stays clean. It used to time out at 16 MiB. The
answer is the same as before for every input the differential probe tried. That
includes UTF-16LE text, and a UTF-16 override inside an ASCII quotation, which still
counts. An override that counts is a note, which agent-ingest holds. YARA-X records at most
1,000,000 matches of a string, so a full list of an override string counts as an
override. An override after a million quoted ones is never recorded, and without that
rule the file would pass (#280).
So documentation that quotes an override a million times or more gets the note even
when every quotation is benign: agent-ingest holds it, and agent-document passes
it with the finding recorded. At the 67-byte line in the record that is about 64 MiB,
under the 100 MiB intake limit.
A finding's offset and length name one match of its rule within the finding's view.
For most rules that is the first match of the first string that matched, in the order
the rule declares its strings. Some rules decide by where matches lie: an exemption
(a quotation of an override, a refusal of a code request) or a placement (a fetch
beside an execute, a script inside an SVG). In such a rule, the first match can be
one the condition exempted while another match made the rule fire.
For these rules the scanner checks whether the rule still fires with the first match's
first byte blanked. If it does not, the first match was needed and is reported as before.
If it does, the scanner finds the shortest prefix of the view on which the rule fires and
reports the match that ends there. It skips the first match if another match ends at the
same byte. The search re-scans prefixes with that rule's file alone and the same global
values. It runs only in views of up to 1 MiB (WITNESS_MAX_BYTES). A larger view, a
timeout during the search, or a rule that no longer fires on a prefix keeps the first
match. Verdicts never change; only which match a finding names
(#284,
record).
MCP, notebook, and Keras readers check structure before the standard JSON decoder allocates an object tree. Each candidate permits at most 100,000 commas, colons, and opening braces/brackets outside strings and comments, and 64 nested containers. The check keeps counters only; escaped quotes and punctuation inside strings or comments do not consume that budget. The standard decoder still owns syntax, duplicate keys, numbers, and string decoding. Keras's embedded HDF5 config check stops at the first JSON object's end. No new dependency is needed.
Exhaustion returns incomplete, with mcp json exceeds structure limit,
notebook json exceeds structure limit, or keras config exceeds structure limit.
This can hold a legitimate dense notebook or generated config. It is a coverage
limit, not a malware finding; do not interpret missing derived findings as clean.
A potentially relevant MCP input requiring comments/trailing-comma decoding beyond
the existing 1 MiB JSONC cap is also incomplete (mcp json exceeds comment limit).
Malformed inputs within the budget retain their existing treatment.
This bounds tree amplification, not total process memory: input buffers, decoded strings, rules, and other readers still use memory. The file and expansion budgets remain in force. Consumers needing a hard memory or time ceiling must isolate the worker and hold its input on failure. See the allocation measurements and compatibility checks.
In addition to the format-specific checks below, heuristic_embedded_pe,
heuristic_embedded_elf, and heuristic_embedded_macho block validated executable
headers inside another file. They differ from document-member and nested-archive
executable findings, which depend on container position.
Legacy Office macro inspection reports heuristic_office_autoexec for an
olevba auto-exec indicator, and heuristic_office_shell,
heuristic_office_createobject, or heuristic_office_urldownloadtofile for those
whole-word keywords in extracted VBA. These findings block. The code is never
executed; a claimed VBA project that cannot be parsed is incomplete. A macro's
mere presence is not this keyword check, though OOXML vbaProject.bin has its own
blocking structural finding. See office_findings.
filetype_mismatch_<kind> reports header/extension disagreement: PE, ELF, and
Mach-O mismatches block, while other recognized mismatches are advisory. A
zero-byte file with a promised document/type suffix uses heuristic_empty_file
instead. The name and header checks live in filetype.py
and are applied by scanner.py.
The scanner also reports these checks without a YARA rule. heuristic_pickle_reduce walks pickle opcodes with a reader built on the standard library's pickletools opcode table and argument readers (the same parse pickletools.genops gives, minus large payloads) and never calls pickle.loads. It blocks a call (REDUCE, NEWOBJ, NEWOBJ_EX, INST, or OBJ) whose callable is a GLOBAL or STACK_GLOBAL of os, subprocess, posix, or nt, any name called eval or exec, or a builtins (or Python 2 __builtin__) name other than the value constructors protocols 2 and 3 use for ordinary data (set, frozenset, bytearray, bytes, complex, range, slice, int, float, str, bool, list, dict, tuple, their Python 2 spellings, and exception classes). Any name from the four modules blocks; builtins.getattr, __import__, open, and compile block. Every opcode's documented stack effect is applied, so dicts, sets, big integers (LONG1/LONG4), byte strings, object state, and memo references before the call are walked through, and the module and name are followed through the memo (STACK_GLOBAL fed by BINGET). Pickles written back to back, the legacy torch.save layout, are walked in turn. A malformed stream, which a loader would also refuse, reports nothing. A stream longer than 20,000,000 opcodes, or one that holds more than 1,000,000 memo entries or stack items at once (a one-byte MEMOIZE repeated millions of times would otherwise exhaust memory), is incomplete, pickle exceeds walk limit, rather than clean. The walk also checks the scan deadline every 4,096 opcodes (incomplete, timeout), and it skips byte payloads, big integers, and text arguments longer than 4 KB without building them, since none can name a callable. The check names dangerous callables; it is not a general pickle safety validator, and a module outside that list (for example runpy or importlib) is not reported. heuristic_safetensors_header blocks a .safetensors file whose header length or tensor offsets do not fit the file. heuristic_gguf_header blocks a .gguf file, or any bytes that start with GGUF, when the version is not 1, 2, or 3, the counts exceed 100,000, or a following record does not parse. heuristic_keras_unsafe_call blocks a Keras model config that names a module that runs commands, opens sockets, or imports code (os, subprocess, sys, socket, importlib, runpy, pickle, marshal, ctypes, urllib, requests, and a few others), or a builtins function that runs code, imports, opens files, or reflects (eval, exec, compile, __import__, open, getattr, and similar). Keras writes its own losses, metrics, and activations as builtins and resolves them in its own modules first, so builtins with mean_squared_error stays clean. It also blocks Keras's own get_file (CVE-2025-8747) or enable_unsafe_deserialization (CVE-2025-9906); load_model resolves every module it reads, which is how CVE-2025-1550 runs code with safe_mode on. heuristic_keras_lambda is advisory on a Lambda layer that carries marshalled bytecode (__lambda__ in Keras 3, function_type: lambda in Keras 2); safe_mode refuses it by default, and some teams turn that off. Both read config.json inside a .keras archive and a .json file from model.to_json(), decoded the way json.loads decodes bytes (a BOM, UTF-16 or UTF-32, and escaped keys all reach the check), and up to four JSON objects inside an .h5/.hdf5 file that open with a key Keras writes at a config's top level (class_name, config, module, registered_name, build_config, or compile_config), in any key order or spacing. Only a config with class_name at its top level counts, as load_model requires, so a JSON document that merely names subprocess, such as an API index, stays clean. A Keras candidate is checked against the JSON structure budget before decoding; exhaustion is incomplete, keras config exceeds structure limit. The subsequent walk retains its 2,000,000-node and 64-level limits. The walker parses JSON and never imports Keras. It does not read a config hidden in HDF5 chunked or compressed storage, and four decoy objects ahead of the real config in an .h5 use up its attempts. heuristic_ooxml_vba and heuristic_ooxml_dde block. heuristic_ooxml_oleobject and heuristic_ooxml_altchunk are advisory. heuristic_mcp_shell and heuristic_mcp_hidden_instruction are advisory on a JSON object or array, read after a UTF-8 byte order mark, and, when strict JSON fails, as jsonc-parser reads a config: // and /* */ comments and trailing commas removed outside strings, in a file of at most 1 MiB. Before decoding, a .json file that names command, args, or description (or uses a Unicode escape that could encode a key) is checked against the JSON structure budget; exhaustion is incomplete (mcp json exceeds structure limit). The subsequent walk retains its 2,000,000-node and 64-level limits. JSON without any such key is not parsed by this heuristic; its raw and derived scans still run. The hidden-instruction check reads every description string, at any depth, for do not tell/mention/inform/reveal the user, no need to tell/mention/inform the user (not when to and a verb follow, as in "to wait"), do not notify the user about it (notify only with about or of and this, it, or that, since a notification tool limits how often it notifies the user), this tool immediately as the first step, ignore previous instructions, an <IMPORTANT> tag, before using this tool, read, an .ssh/ path, id_rsa, mcp.json, pass its contents, or as a sidenote, the vocabulary of MCP tool poisoning. A description is a wrapped docstring, so the words of a phrase may be apart by any run of whitespace, a line break and a no-break space included. heuristic_pdf_text_unmapped is a note on a PDF whose shown text could not all be decoded (see the pdf-text view under Limits); under agent-ingest it holds the file, under agent-document it is recorded. heuristic_pdf_content_unread is a note, graded the same way, on a PDF with page content no view could decode: a stream a /Contents entry names (directly, in an array, in an array object, or in a page dictionary stored in an object stream) whose dictionary marks it as no image, form, or font program, or a form XObject, under a filter chain with no decoder here or under LZW or RunLength past its size cap. Images, fonts, object streams, and streams nothing names as page content are never reported, whatever their filter. The report's pdf_content_unread counts those streams by filter name, from the standard names of ISO 32000-1 Table 6 or other, never a name taken from the file (verification). heuristic_hidden_text is advisory on an instruction phrase in Office or PDF text a reader hides (vanished, white, or 1 pt runs, hidden shapes and slides, invisible render modes); see hidden Office and PDF text. agent_encoded_instruction is a note on a base64-decoded run that contains ignore previous instructions or ignore all previous instructions as a plain substring, case-insensitive, when the phrase rules did not already report that run. It keeps the reading of the retired YARA rule of that name, which also caught a phrase run into the word before it; plain text with the same glued phrase is not reported. heuristic_empty_file is advisory: a zero-length file whose name promises a type (an online-only cloud placeholder named report.pdf) is reported under that label instead of as a text file in disguise, and agent-ingest still blocks it. heuristic_double_extension, heuristic_filename_control, heuristic_document_executable, and heuristic_nested_executable block. heuristic_double_extension matches a document, spreadsheet, image, media, or OneNote suffix, optional spaces, dots, or underscores, then a program suffix (.exe, .scr, .com, .msi, .xll, …) or a launcher suffix: Windows script hosts and shortcuts (.lnk, .url, .js, .hta, .cmd, .ps1, .msc, .library-ms, …), disk images that mount on a double-click (.iso, .img, .vhd, .vhdx), packages and manifests that install (.msix, .appx, .application, .appref-ms, .msp, .chm, .diagcab), registry and setup files (.reg, .inf), Remote Desktop and theme files that reach a host when opened (.rdp, .theme), App Installer manifests (.appinstaller), and macOS launchers (.pkg, .dmg, .scpt, .terminal, .workflow, .command, .webloc), so invoice.pdf.lnk, photo.jpg.iso, statement.pdf.msix, invoice.pdf .exe, and Invoice.pdf.exe with a fullwidth dot block. A .txt or .html decoy pairs only with a program suffix, so page.html.js stays clean; a single suffix such as backup.iso is not a disguise. heuristic_filename_control blocks a name that contains a bidi override, embedding, isolate or mark, a zero-width space, a word joiner, a BOM, or a Unicode tag (the invoice_\u202Efdp.exe trick). ZWNJ and ZWJ are allowed so Persian, Indic, and emoji names stay clean. Both checks read the submitted name and every component of an archive member's path, so an Invoice.pdf.app/ bundle inside a zip blocks. Directories above a scanned file are not checked.
Rechecked on November 16, 2026 at 1426e7f with Python 3.12.13, YARA-X 1.23.0, and bundled rules. Run uv run --frozen python docs/verification/measure_precision.py --fail-over to repeat it. This benign-only measurement reports collisions on the generated corpus. The November 6 documentation audit records the older baseline.
On the generated 328-file corpus (docs/verification/measure_precision.py, run by CI), 85 of the 89 YARA rules hit zero files and all scans completed. No structural check reports anything: the merged PDF whose font name means five fonts is read page by page and is clean (page fonts). Four distinct files had findings (4/328, 1.22%); each matching rule hit one file (1/328, 0.30%). These are benign-corpus hit rates, not statistical precision (true positives divided by all positives), recall, or a production false-positive estimate. A per-rule threshold does not bound the combined hold rate. The four that hit are tagged Windows shortcut-family collisions, each one file: win_library_unc_location blocked a file-server library, win_library_remote_location matched an https search connector, win_shortcut_remote_icon a file-server folder icon, and win_internet_shortcut_launch a .url to a local file: URL. The corpus's 31 agent-facing files (one-time-code notices, README lines that run a command, committed agent settings that keep approval on, unattended CI agent steps), 12 platform files (inline MSBuild tasks, Connection Manager and driver INFs, a local JXA script, a dscl -authonly test, standard .pth files), 14 ecosystem files (five GitHub Actions workflows, five Keras models), and four checkout files (.git/config with LFS, git-crypt, Watchman, and Husky entries, a devcontainer.json, a nix-direnv .envrc) hit nothing. See Precision with Windows shortcut-family files and the earlier Precision after the local pack grew; the near-miss pairs in docs/verification/check_rule_pairs.py remain the tighter benign boundary. Vendor rule bodies stay out of git; the license reasons are in Vendor pack review.
Excel xl/comments\d*.xml and PowerPoint ppt/comments/*.xml rich-text comments
are scanned one comment at a time as ooxml-comment, and as
ooxml-comment+text-folded when folding changes the text. Runs are joined within
that comment; paragraphs and explicit breaks become newlines. Two comments cannot
complete or suppress each other's phrase. One-run comments remain covered by raw
and entity-decoded views. The same bounded tokenizer serves worksheet cells.
Excel threaded-comment parts are not added here. Drawings and the Word glossary part are
joined by ooxml-text (below).
The existing html-text reader also accepts .svg or an SVG root, including one
after an XML declaration. tspan and textPath stay inline; each text element is
its own piece. CDATA inside SVG contributes literal text without entity decoding.
This is lexical extraction: coordinates, CSS, visibility, and visual
line layout are not interpreted. Positioning tspans within one text element can
still form a phrase. Latin small capitals with a direct ASCII letter equivalent
fold in text-folded and the joined/decoded views that already fold; this is a
fixed table, not general phonetic transliteration or a new finding severity.
Verification records the targeted gaps, benign boundaries, and remaining Phase 1c work in #171.
pdf-annotation reads /Contents text strings from dictionaries declaring
/Type /Annot, or a standard annotation /Subtype and /Rect. It scans each
definition, including inactive revisions; it does not resolve the active xref
revision or prove that a viewer displays the annotation. Literal and octal
escapes, hexadecimal strings (including odd final nibbles), PDFDocEncoding,
UTF-16BE BOM and PDF 2.0 UTF-8 BOM strings are decoded. Indirect text strings and
members of already supported object streams are included. Language tags inside
Unicode strings are removed. Each annotation and each blank-line-separated
paragraph is a separate piece; folded spelling reaches the same existing rules
as pdf-annotation+text-folded. Encoded text alone is no finding.
The reader skips comments, literal-string contents, and stream bodies while
looking for annotation dictionaries. It proves a stream boundary only when a
direct or indirect /Length lands on endstream and that keyword is the first
endstream in the counted span. The body is then skipped, so a comment,
literal, hex string, or object header inside it is not an annotation. An object
header by itself does not reject the span: with no earlier endstream, those
bytes are body data, and an annotation after the landed keyword is still read.
A missing length, a length that does not land on endstream, or a length whose
span contains an earlier keyword endstream does not resume lexical parsing.
A longer token such as endstreaming is not that keyword. The scan
is incomplete (pdf annotation stream length unaligned). That earlier keyword
is not taken as the boundary either. Resuming there is unsafe, and treating the
first textual endstream or an N G obj inside opaque bytes as a real object
would also be unsafe: images, fonts, and compressed streams can carry those
bytes. The hypothesis of rejecting either sequence was measured. Neither
appeared in 348 installed font and image files, a random 1 MiB buffer, its
Flate encoding, or the Flate encoding of prose that contains both phrases.
Rejecting an object header alone still fails closed on four aligned lookalike
rows whose bodies contain N G obj and whose real annotation follows the
stream, so that half is not used. A length is refused when the span contains
an earlier keyword endstream. That also incompletes two correct lengths:
uncompressed page text that includes the word, and a JPEG comment that includes
it. The Flate encoding of the same prose does not contain the keyword, and the
annotation after that stream is still read.
A keyword-delimited object header in the raw bytes of a landed stream makes
the scan incomplete (pdf object header inside stream). The header is
N G obj with PDF whitespace (NUL, TAB, LF, FF, CR, or space, including a
run of them) between the tokens. The number starts the span or follows
whitespace or a delimiter ()<>[]{}/%. obj ends on whitespace, one of those
delimiters, or <<. This does not depend on an xref entry, a page reference,
or the generation. Decoded object-stream members are not this check. They are
read on their own, and a type-2 xref entry stays an index into that stream.
The incompletes are conservative. A header written as stream data, including
one after a /Length that lands on endstream, still stops the scan, and a
real annotation after that stream is not read. A header that no viewer displays
still stops the scan. Measured on 2026-11-07, none of these files had such a header
in a landed span: rfbproto-3.3.pdf (Acrobat Distiller 3.02, 110778 bytes),
rfbproto-3.7.pdf (GNU Ghostscript 5.10, 165552 bytes), rfbproto-3.8.pdf
(GPL Ghostscript 8.61, 143840 bytes), gnus-logo.pdf (GPL Ghostscript 8.54,
3658 bytes), and the two Microsoft Word hybrids
CL2020 EMEAR DEVWKS-1542 Intersight Ansible Lab Guide.pdf (1030061 bytes)
and Impact DNW07 UCS Ansible Collection Lab Guide.pdf (433208 bytes). The
same header pattern was absent from 348 fonts under /usr/share/fonts and
454 PNG or JPEG files under /usr/share/pixmaps and /usr/share/icons.
A file with startxref also resolves cross-reference offsets before the scan
can be clean. A candidate is every occurrence of the bytes startxref that a
number follows, with or without a following %%EOF, wherever it is written.
Readers search the end of the file for the bytes or for the line and do not
lex around them, so one inside a comment, a literal string, or a stream body
is a candidate too
(linearized files and the keyword search).
Comments between the keyword and the number are skipped, and leading zeros
are dropped, as readers do. Offset 0, and a keyword that no number follows,
name the start of the file: when the file opens with an xref stream that
stream is read as a candidate, and otherwise the occurrence is skipped, since
a linearized file's first-page trailer carries startxref 0 by design.
A sign or a dot in front of the number, or more than 20 digits, is malformed. A candidate that is
present and does not parse still makes the scan incomplete.
Each candidate includes its /Prev chain. Classic 20-byte tables,
Flate /Type /XRef streams (/W, /Index, and a PNG or TIFF predictor), and
hybrid /XRefStm are read. A newer section overrides an older one inside one
candidate. At one trailer, a classic offset and a stream offset for the same
object are both checked. The landed span includes the newline and endstream
the length ends on, so an offset on those bytes is inside the span. A type-1
offset from any candidate inside that span makes the scan incomplete
(pdf annotation xref offset inside stream). A type-1 offset that does not
begin an object header is malformed, which covers a /W width that does not
match the packed fields and a stream whose decoded length is not an exact
multiple of the row width. A type-2 entry is an object-stream index, not a
file offset, so object streams stay readable. An object that only a later
candidate lists is kept when its offset begins an object header outside every
span. A file with no candidate leaves the sequential reader in place. A
classic table may have blank lines between its sections and before trailer,
and its trailer is read as far as the startxref after it. A malformed table
or stream (pdf xref malformed), a filter or predictor this reader does not
decode (pdf xref unsupported), more than 100000 entries
(pdf xref entries exceed 100000), more than 32 startxref candidates, or a
spent deadline is incomplete. A /Prev chain of more than 32 sections is
malformed, so the scan is incomplete, not truncated. Flate output is
charged to the shared 500MB budget, and a stream pdf_streams already inflated
is not inflated again.
Rich text /RC and filters the existing object-stream reader does not support
remain outside this view; form-field values are read as pdf-string, below. A recognized annotation with an
unresolved /Contents reference, duplicate dictionary keys, malformed string,
or unreadable encoding makes the scan incomplete. Each source is limited to
4,000,000 tokens, each dictionary to 100,000 tokens and the existing 64-level
parser depth, encoded strings to 1 MiB, and objects/pieces to 100,000, under the
shared deadline and expansion budget. See verification.
The file-wide font lookup recognizes standard PDF name escapes in the /Font key.
Direct and indirect resource dictionaries use the same lookup. The page-specific
lookup still requires an unambiguous page tree and falls back to the file-wide
lookup for escaped keys. For the measured nonstandard reference spellings that
only some readers follow, text shown with the affected font adds the
heuristic_pdf_text_unmapped coverage note. The scanner keeps its other readings
and leaves the disputed reference unresolved. Printed examples and comments do
not add the note.
See reader verification.
pdf-string reads the other text strings a PDF reader shows (#239):
a form field's value (/V in a dictionary that is a widget or carries /FT, /T,
/Parent, or /Kids, including each string of a choice field's array, written in place
or in an array object of its own whose items may each be a reference), the document
information strings (/Title, /Author, /Subject, /Keywords, /Creator,
/Producer), and an outline item's /Title. pypdf and PyMuPDF show each of these, and
pdfinfo shows the information strings. They come from the same pass, and every
definition, as annotation /Contents, so a string in an object stream or behind a
reference is read, and so is one in an inactive revision or in a field no page lists.
Each string, and each blank-line-separated paragraph of one, is a piece, folded as
pdf-string+text-folded when the spelling changes.
A string is decoded as pypdf decodes it: a UTF-8 or UTF-16 byte order mark, then UTF-16
without a mark when a NUL is in its first or second byte, then PDFDocEncoding. A byte
PDFDocEncoding leaves undefined is read as its Latin-1 character, and an invalid sequence
as U+FFFD. Nearly every PDF has an information dictionary, so an odd byte from a producer
never makes this view incomplete. A value that is not a text string (a check box's
name, a signature dictionary, a reference to a missing object) is passed over: a reader
shows no text for it. A string over 1 MiB, a malformed literal, and more than 100,000
pieces across both views are incomplete, under the same reasons as annotations. An array
object over 100,000 tokens is incomplete (pdf string array exceeds 100000 tokens) only
when a value refers to it.
The view reaches the existing rules and adds no rule or severity. Rich text (/RV,
/RC), a field's default value (/DV), its tooltip (/TU), the options of a choice
field (/Opt), and the XMP metadata stream are not read here. See
verification.
pdf-actual-text reads each /ActualText string written in a content stream's own
property list (/Span <</ActualText <FEFF...>>> BDC, any tag, literal, octal, hex, or
UTF-16), and of each property list outside a stream, such as one a page names from its
/Properties resource (/Span /MC0 BDC), directly or as an object of its own, decoded as a
pdf-string is, one piece per distinct value. poppler 24.02 and
PDFium show such a string in place of the glyphs it covers; pdfminer.six shows the glyphs.
Two values are never joined, so a phrase split over two spans is two pieces. More than
20,000 distinct values in one file adds the note heuristic_pdf_text_unmapped, and the
rest are not read. The values share the 16 Mi-character decoded-text limit with the page
text, so past it the scan is incomplete (pdf decoded text exceeds character limit). A
property list outside a stream is read whether or not a page names it, shares the
dictionary pass's limits (a string over 1 MiB, or more than 100,000 values with the other
strings, is incomplete), and is reported in the same view as the content-stream values.
Not read: /ActualText on a structure element (one with /S, or /Type /StructElem),
which neither reader shows in place of the glyphs; and the page text a reader rebuilds when
it substitutes a value for part of a sentence. See
verification and
named property lists.
When a selected font has an unreadable composite interpretation, the scanner can report
heuristic_pdf_text_unmapped. A nearby font's map cannot clear that note for these declarations
when the selected font has no usable map of its own:
- A declared simple font with a composite Identity encoding.
- A Type0 font with an absent or null
/DescendantFontsentry. - A font interpreted as composite whose
/ToUnicodevalue has the wrong type for a map reference. An explicit null is treated like an omitted entry.
The scanner still tries the existing font readings. The note describes unread text, and it follows the existing policy for notes. See verification.
RTF data beginning with {\rtf1 is read even under a renamed filename or inside
an archive. rtf-text removes formatting, decodes hex bytes using the declared
codepage or font charset, and reads signed Unicode escapes with group-scoped
fallback lengths. Each paragraph, table cell, and separate header/footer/footnote
is scanned independently, then folded as rtf-text+text-folded when its spelling
changes. Unknown starred destinations, metadata, font tables, pictures, field
instructions, and embedded-object bytes are skipped; field results remain text.
Original bytes are still scanned. Formatting or an encoding alone is no finding.
The existing phrase rule stays a note: default and agent-document are advisory
(exit 0), while agent-ingest blocks (exit 1). Malformed groups, escapes, lengths,
unreadable codepages or font charsets, and resource limits make the scan incomplete
(exit 2). The reader checks the shared deadline and expansion budget and caps group
depth at 128, tokens at 4,000,000, and pieces at 100,000. It is not a renderer:
Symbol-font glyphs, pictures, embedded objects, and visual layout are not decoded.
See the synthetic verification.
The Office readers match tags by pattern; no XML parser runs. plain_xml first gives
them the part as a parser hands its text on: comments, processing instructions, and the
DOCTYPE are removed, a CDATA section becomes the text it holds, > in a quoted attribute
value is escaped, a reference to an entity the part's own DOCTYPE declares becomes that
entity's text, and a UTF-16 part is transcoded. So Ignore all <!-- -->previous,
<![CDATA[previous]]>, and &w; reach the phrase rules as the words pandoc, LibreOffice,
or a library built on a parser shows. A run counts as joined only where markup stood
between two stretches of its text; a run wholly in CDATA is contiguous in the raw bytes and
is left to them. An entity with a system or public identifier is never read, fetched, or
expanded. More than 64 internal entities, or more than 1 MiB of expansion, makes the scan
incomplete.
ooxml-text joins runs in the body, header, footer, note, comment, shared-string, slide,
and notes-slide parts, and in the parts a renderer draws from: ppt/slideMasters and
ppt/slideLayouts, a SmartArt diagram's diagrams/data*.xml and drawing*.xml,
charts/chart*.xml, xl/drawings/drawing*.xml, and word/glossary/document.xml. Notes
masters, handout masters, and theme parts are drawn on no slide and are not joined. Hidden
runs are read in the body parts only (below); a 1 pt run on a
slide master is read as shown text.
A part is known by its name or by what it is. A package names its own parts, so a body
can be word/document2.xml and shared strings xl/strings.xml. A unit that opens as XML
is also given a role by its root element and the namespace its prefix binds, transitional
or strict: a Word document, hdr, ftr, footnotes, endnotes, or comments; a
SpreadsheetML sst, worksheet, or comments; a PresentationML sld, notes, or
cmLst; and the drawn parts above. A Word or SpreadsheetML root outside that list is known
by what it holds (a body, an si, a sheetData). All four Office readers, the
hidden-text reader included, take the role, so a loose XML file with an Office root is read
too. Notes and handout masters, themes, settings, and styles get no role.
A run is known by its namespace, not its prefix. The same XML under ns0: (what
ElementTree writes) or x: (what the Open XML SDK writes for SpreadsheetML) is read, and an
element spelled w: that an inner declaration binds to another namespace is not a run. A
part with the usual prefixes is not rewritten. A DrawingML <a:br/> is a line break, white
space is allowed before the > of an end tag, a text tag's attributes have no length
limit, and a phonetic run is left out of a shared string. Where readers take a part
differently (resolved or as written, a break or a join) each reading is its own ooxml-text
view, at most four, and a rule is reported once per unit and view. More than 256 nested
elements that each declare a namespace, or more than 8 namespace defaults in a DOCTYPE,
make the scan incomplete; no office suite writes either. Tracked changes, text boxes
anchored between two words, and field, math, and symbol runs are read one way only. See
verification; a change is checked against LibreOffice, pandoc, python-docx,
and python-pptx with probe_ooxml_reader_agreement.py.
odf-text joins the paragraphs of an OpenDocument part, and is folded as
odf-text+text-folded when its spelling changes. A part is read when it declares the
OpenDocument text namespace, under whatever prefix it binds: content.xml of an .odt,
.ods, or .odp, a header in styles.xml, an embedded object, and a flat document
(.fodt). A paragraph or heading is one line; a span, a link, a comment, and CDATA join;
text:s, text:tab, and text:line-break are their white space, and other white space
collapses as a reader collapses it. A paragraph nested in another (a footnote body, a
comment, a text box) is a line of its own, and a blank line follows each table cell, so two
cells never form one phrase. LibreOffice shows the character data of an element it does not
know and pandoc drops it, so such a paragraph is read each way: as the text every reader
agrees on, as LibreOffice lays it out, and as all of its character data. Paragraphs nested
more than 64 deep make the scan incomplete. Hidden OpenDocument text (text:hidden-text, a
style with text:display="none", white or 1 pt text set by a style) is read as shown text.
See verification; a change to what the view
joins is checked against LibreOffice and pandoc with
probe_odf_reader_agreement.py.
Hidden Office and PDF text
heuristic_hidden_text is advisory. It is the Office and PDF counterpart of
agent_hidden_html: an instruction a reader does not see, where a person reviewing
the document would see nothing to object to. The text a reader hides is collected on
its own and scanned piece by piece, as ooxml-hidden-text or pdf-hidden-text
(and +text-folded when folding changes it); HTML has the same view, described
below. When an agent_ rule other than the
Unicode-control and base64 rules matches there at note or advisory severity, the
scanner adds one heuristic_hidden_text finding for the member, labeled with that
view. The rule's own finding still comes from the views that read every run, such as
ooxml-text or pdf-text. So under agent-document a phrase in hidden text holds
the file, where the same phrase in shown text is a note. Hidden text without an
instruction reports nothing: templates, watermarks, field codes, and OCR layers hide
text routinely.
Hidden means, from a run's, shape's, or slide's own properties:
- Word:
w:vanish,w:specVanish, orw:webHidden(unless its value is0,false, oroff),w:szof 2 half-points or less, or aw:colorwhose every channel isF0or more. White text counts only when no run shading or highlight, paragraph shading, table-cell or table shading, or page background (w:background) paints something else behind it. The earlier properties a tracked change keeps (w:rPrChangeand its siblings) and a paragraph mark's own properties (w:pPr/w:rPr) do not count. XML comments do not affect formatting. A text box in a drawing markedhidden="1"hides its text. - PowerPoint: a run of 1 pt or less (
a:rPr szof 100 or less), every text frame of a shape or group whosep:cNvPrishidden="1", and every frame of a slide withshow="0". - Excel: a rich-text run in shared strings, an inline-string cell, or a comment at 1 pt or less, or in near-white (an RGB value, or theme colour 0 without a darkening tint, which Excel maps to the light background).
- PDF: text in render mode 3 (invisible) or 7 (clip only), and text whose active
fill or stroke colours are all near-white and whose first glyph sits on nothing
the same content stream painted in another colour. Render mode and colour carry
across the streams of one
/Contentsarray, which are read as one stream: a direct array in a raw page dictionary, an array kept in an object of its own, or an array in a page stored in a compressed object stream. A painted area is the bounding box of a filled path, an image placement, or a shading; a stream keeps 32, and white text over a later one counts as hidden. In a page's own content stream, text under 1 pt and text that lies wholly outside the union of every/MediaBoxin the file are hidden too. Form XObjects, patterns, and glyph procedures are drawn through a matrix the reader does not know, so their size and position are not judged, while their render mode and colour are.
A hidden piece ends at the next shown text that is more than whitespace (decoded through the font for a PDF), so hidden halves on both sides of a shown word are never joined; a shown space between them is kept. In Office parts a piece also ends at a table cell, text box, shape, shared string, inline string, or comment; paragraph ends inside it become newlines.
Accepted boundary: an OCR text layer is render mode 3 over a page image, and so is an
attacker's, so a scanned page whose text layer reads as an override phrase is held
under agent-document. Exempting text over an image would let any picture hide an
instruction.
Not resolved: properties inherited from styles, list styles, slide layouts and
masters, and cell formats (all in other parts), hidden sheets, rows, and columns
(their text sits in shared strings), DrawingML colours, text covered by a later
opaque fill, clipping, transparency and blend modes, CropBox, and per-page boxes; a
bogus /MediaBox widens the page union. White text over a shaded table whose shading
comes from a table style counts as hidden, which can only add a hold. The
hidden-text record has the fixtures, the
benign sweep, and the timings.
Hidden HTML text
The same finding covers HTML. The text a browser does not show when the document opens is
collected as html-hidden-text (and +text-folded), and an agent_ rule match in a piece
adds one heuristic_hidden_text, as above. Where agent_hidden_html already fired for the
member nothing is added, so a file that was held keeps the findings it had. Nothing is
rendered, no stylesheet is resolved, and no script runs.
A claim that text is hidden rests on one exact declaration on an element the text is
inside, in a spelling a browser accepts. A browser drops a declaration it cannot parse, so
font-size:0 px, display:none\9, and a declaration after an unclosed parenthesis are no
claim, and a declaration that could have set a property back ends the claim an ancestor
made.
| Hidden | Until |
|---|---|
A comment; a data-* value of twelve characters or more that holds a space |
|
The content of a template |
|
display:none |
nothing inside can show again |
| Zero opacity, in a document with no script, no linked stylesheet, and no animating style | nothing inside can show again |
visibility:hidden |
a descendant sets it back, or carries a class or id a stylesheet could use |
| A font size of zero | a descendant sets a size |
| Transparent text | an element with any attribute of its own |
| Text within 12 steps a channel of the background the same element declares | an element with any attribute of its own; a block inside an inline element |
The hidden attribute with no inline display, in a document with no stylesheet |
nothing inside can show again |
| A screen-reader-only class, in a document with a stylesheet | an element with any attribute of its own |
In a document with a stylesheet a colour or a font size is followed to the text directly
inside the declaring element and no further, because stylesheets style by element type
(code { color: ... }, p { font-size: 16px }). Transparent and same-colour text count
only where no style in the document draws text by a shadow, a stroke, a fill, or a
background clip, and no stylesheet is linked.
A piece of hidden text whose whole wording is also in the text the page shows is not
reported: a share widget copies the title into data-title, and a mail preheader repeats
the first line. A hidden piece ends at the next shown text that is more than whitespace,
and a block boundary inside it is a newline.
Which elements are open follows the HTML parser where a browser's tree differs from the
written nesting: end tags a writer may leave out, formatting elements a browser reopens
after a misnested end tag, a link that ends an open link, the parts of a table and text
moved out of one, a form closed by its parent, and HTML that ends an svg. Markup past
what is followed stops the element reader for the rest of the document, and comments and
data-* values are still read: a select that holds anything but options, a formatting
element that ends across a table cell or an svg, a fourth copy of one formatting element,
a table inside an open paragraph under an older doctype, more than 500 open elements, or
more than 400,000 tags.
Not claimed, because the markup does not settle it: a box of no size that clips its
content, text moved off the page by an offset, a font of a pixel, and near-white text with
no background of its own. Chromium can paint the same markup differently with a different
colour preference or stylesheet. See the
browser measurements.
agent_hidden_html still reads the offset and the small font as cues beside one of its phrases. Colour attributes
(<font color>, bgcolor) and SVG presentation attributes are not read. A document a
browser reads as XML (an .svg or .xhtml name, or an XML start under any name but
.html) gives only its comments and data-* values, since the element rules are the HTML
parser's. A noscript body is not read: with scripting on it is not markup.
Three things are assumed, not read: the document is taken as it is when it opens, before a
script changes it; the browser has scripting on; and no stylesheet overrides an inline
declaration or turns visibility back on for an element by its type. Someone who wants
the element reader to stop can write markup it does not follow; the lexical rule and the
phrase rules do not depend on it. The
record has the comparison with Chromium
that set these boundaries, and probe_html_hidden_browser.py
repeats it.
text-folded handles escape sequences for text matching. A phrase can still be
read when a terminal later redraws over it. The scanner has no terminal viewport,
wrapping, or scrollback state for a captured log. The
tmux 3.4 measurements
show how those details affect whether the same bytes leave text visible.
A text file that is not UTF-8 and declares a charset is read through that charset as
decoded-charset, and the other text views are read from that text, as they are from
UTF-16. A browser shows a windows-1251 page as Cyrillic, so a phrase written in those bytes
reaches the multilingual rule; read as UTF-8 its letters were U+FFFD. The bytes as written
are still scanned raw.
The UTF-8 reading is scanned as well, with the same checks and views it had before. A
consumer that ignores the declaration decodes UTF-8, so a meta and one stray byte must not
hide what that reading shows: tag characters, joiners, a fullwidth or UTF-8 phrase. A rule
both readings find in one view is one finding. This doubles the text work for a declared
file that is not UTF-8 (4 MiB of HTML comments: 3.2 s on f54c57e, 7.5 s here).
The declaration is found the way Chromium finds it:
- A
metawithcharset, or withhttp-equiv="content-type"and acontentthat names a charset. The lastcharsetattribute wins over anycontent. The first meta whose label the WHATWG Encoding Standard knows decides; one with an unknown label is passed over. - The meta is looked for anywhere in the head and, once the head ends, in the first 1,024
bytes. A tag a head cannot hold ends it; text does not. A meta inside a comment
(
<!-->and--!>close one), an attribute value, ascript(including a doubly escaped one),style,title,textarea, or other raw-text element is text. One inside anoscriptcounts. - Failing a meta, an XML declaration at the first byte (
<?xml, thenencodingin lower case). A file named.xml,.xhtml,.xht, or.svgis read through its XML declaration first, as a browser reads it, and then through a meta, which a browser does not read in those files but a reader that parses them as HTML does. - Labels are the standard's 228, as their encodings decode:
latin1andus-asciias windows-1252,gb2312andgbkas gb18030,shift_jisas windows-31j,euc-kras windows-949. A byte the charset does not define is U+FFFD. The directional marks and joiners that windows-1255, windows-1256, and ISO-8859-8 hold as one byte are dropped, as in a transcoded mail part, because ordinary Hebrew and Persian text carries them.
Nothing is transcoded when the bytes are UTF-8, with one exception: a page declared
ISO-2022-JP that holds an escape, whose seven-bit text is valid UTF-8 that no reader shows.
A UTF-8 byte order mark, a UTF-8 or UTF-16 label, the replacement encoding (iso-2022-kr,
hz-gb-2312), and UTF-7, which is not a label a browser knows, also leave the text as
UTF-8. A mail message is not read this way: its parts carry their own charsets.
A page that declares nothing is read as UTF-8, although Chromium guesses its encoding from
the bytes and would show windows-1251 text. When a file holds both a meta and an XML
declaration naming different charsets, the one its name's parser reads is used, and a
browser that parses it the other way reads the other. The
record has the comparison with Chromium
that set these rules, and
probe_html_charset_browser.py repeats it.
Cyrillic-only words are lowercased in derived folded views so casing cannot hide the existing Russian override phrases. Mixed-script look-alike mapping runs first; ASCII case-sensitive signatures such as DAN, Greek text, and accented Latin letters retain their prior treatment. No new phrase or severity is introduced. The November 7 comparison records the new cases, unchanged corpus outcomes, and the combining-mark gap that remains; RTF text and annotation strings became views afterwards.
It does poorly against a new or targeted payload. YARA matches patterns somebody has already written down. Building a rule from the file you are judging only describes that file, so the tool never does that.
Other limits, measured in this tree with the locked YARA-X 1.23.0:
- ZIP, tar, gzip, xz, bzip2, ar (a
.deb's outer layer, static libraries), newc and crc cpio (initramfs images, RPM payloads), and mail messages (.eml, or the raw bytes a mail hook holds: header lines within 64 KiB, aContent-Type, and amultipart/*body, a base64, quoted-printable, or uuencoded body, an HTML body, or a text body in a charset other than UTF-8 or ASCII; one UTF-8text/plainbody with no encoding stays one unit) are inspected in memory with at most 3 container levels, 1,000 entries, 100 MiB per file/member, and a shared 500 MiB expanded/derived-content budget. Gzip, xz, and bzip2 each count as one level, including in.tar.gz,.tgz,.tar.xz,.txz, and.tar.bz2. A file can hold more than one zip:zipfilereads the archive whose end record comes last, while 7-Zip opens the first of a concatenation, so every earlier end record outside the members already read is opened as an archive too (a record inside a member's bytes belongs to a stored nested zip and is read once, as that member), with at most 16 archives and 64 candidate end records before the scan is incomplete. A zip appended to another file (a PDF, image, or script, or a Python zipapp after its#!line) is read as a zip as well; the shape is not itself a finding, and the outer file's own checks still run. An entry whose local header no directory record lists (before the first member, or between members) is read too, as Java'sZipInputStreamand other streaming unzippers read it: the bytes no listed member covers are searched for stored or deflated local headers, and each that inflates is one more member under the same checks. A zip with local headers and no directory at all stays incomplete. A tar withoutustarmagic (Unix V7) is opened when its first block is a header with a matching checksum, the test GNU tar andtarfilemake (record). - A PDF whose trailer or XRef stream names an
/Encryptdictionary (encrypted pdf, including#xx-escaped spellings) and an OLE file holding an encrypted Office package or a legacy document encrypted in place (encrypted office file, from stream names andmsoffcryptovia oletools, never decrypted) produce incomplete scans: their text is ciphertext to the scanner. Findings from the clear bytes are kept, and incomplete still decides the exit. Owner-password-only PDFs (print or copy restrictions) are encrypted too and are held the same way. - Encrypted/unreadable archives, unsupported tar member types, zip symlink/device entries, a zip or tar member name that is absolute (
/x,C:\x), has a..segment, or holds a NUL byte (archive member escapes the archive), a cpio symlink or device, a corrupt xz or bzip2 stream, and containers recognized from their full fixed header that the scanner cannot open (7z, RAR, zstd and its skippable frames, LZ4, cabinet, CHM, WIM, xar, ACE, LHA, thin ar, and odc cpio:unsupported archive) produce incomplete scans. Members are never extracted to filesystem paths; the name check exists so a consumer that unpacks the archive later is not told it was clean. Reasons never carry the member name. - Raw bytes and UTF-8 derived views are scanned: BOM-marked or ASCII-heavy UTF-16 (
decoded-utf16), text that is not UTF-8 read through the charset its meta or XML declaration names (decoded-charset; declared charsets), HTML entity decoding (html-unescaped), JSON string escapes such as\u0049in a payload that opens like JSON, meaning{or[followed by a quote, bracket, digit, minus sign,true,false, ornull, after an optional byte order mark, white space, and//or/* */comments within the first 8,192 characters (JSON with comments, as VS Code readssettings.json,mcp.json, anddevcontainer.json), so an INI header such as[InternetShortcut]is not decoded (json-unescaped), the escapes of each YAML double-quoted scalar and TOML basic string in a file named.yaml,.yml, or.toml, (config-unescaped: one piece per escaped scalar or string; YAML uses PyYAML parser events, which keep plain and block scalars literal and leave aliases and tags unresolved; TOML decodes basic-string escapes and multiline continuations; malformed YAML retains only events produced before the syntax error), removal of format controls for inspection (format-controls-removed), base64 runs that decode to readable text (base64-decoded: standard or URL-safe runs of 16 or more characters, line-wrapped runs rejoined, decoded as UTF-8 or as UTF-16LE the way PowerShell's-EncodedCommandwrites it, kept only when the result is at least 95% printable and holds a space; padding ends a run, and a wrapped run that does not decode whole is tried line by line; each decoded run is a separate YARA pass, so two runs cannot form one phrase; every candidate is read under the scan deadline, and more than 1,024 readable runs is incomplete; also applied to a PDF's inflated text streams; skipped on a MIME message, whose base64 parts are decoded as members, while its headers' RFC 2047 encoded words get amime-headers-decodedview), percent-encoding (percent-decoded:%XXescapes and+spaces, when the text holds three escapes or a token of four or more letter words joined by+), look-alike spellings folded to plain letters (text-folded: terminal control sequences removed whole, soESC [ 1 mbefore a word does not join it; terminal strings taken out of the line as tmux draws them, an OSC (ESC ]) ending at BEL orESC \and a DCS, SOS, PM, or APC atESC \only, any other ESC, CAN, or SUB ending one early, with each payload kept as a paragraph after the text because an OSC 0 to 2 payload is a window title shown elsewhere, and a string that never ends left with only its opener removed; then NFKC compatibility forms such as fullwidth letters, control characters, variation selectors (U+FE00 to U+FE0F, U+E0100 to U+E01EF, and the Mongolian free variation selectors), and the combining grapheme joiner U+034F removed, with VT, FF, and NEL read as line feeds rather than removed, soIgnore<VT>allis two words as a terminal shows it, Cyrillic and Greek look-alikes folded to Latin inside a token that mixes scripts, a word hyphenated at a line end rejoined (a letter,-, a line break, then a lowercase letter, soprevi-andousread asprevious), letters spaced one apart rejoined, and0 1 3 4 5 7 @ $folded inside a token that also has letters, soIgnоrewith a Cyrillic о,Ignore,1gn0r3, andi g n o r ereach the phrase rules without transliterating Cyrillic-only or Greek-only words (Cyrillic-only word case is lowercased); ordinary product codes and spaced initials retain their tested benign outcomes; thejson-unescaped,base64-decoded,percent-decoded,mime-headers-decoded, andooxml-textviews are folded the same way, one step deep, as<view>+text-folded, so an escape or encoding stacked on a look-alike spelling is read too, and the two base64 rules are left to the raw bytes in every folded view), the inflated text of a PDF'sFlateDecodecontent and object streams (pdf-flate), and the joined text runs of a Word body, header, footer, notes, or comments part, an Excel shared-strings part, a PowerPoint slide, or a part a renderer draws from, such as a slide master, a chart, or a worksheet text box (ooxml-text: runs within a paragraph are joined, paragraphs end with a newline, entities are decoded, no XML parser runs but comments, CDATA wrappers, and the part's own internal entities are read through, and a part with a single run is left to the member's own views), the paragraphs of an OpenDocument part (odf-text: one line per paragraph, a blank line after each table cell; boundaries), each worksheet inline-string cell whose text is split across two or more rich-text runs, and each cell value (<v>, such as a formula's cached string) that a comment, a CDATA boundary, or an entity splits (ooxml-cell: one YARA pass per cell, so two cells never form one phrase, and phonetic runs are not shown text), each section of a worksheet's page headers and footers (ooxml-header:oddHeader,evenHeader,firstHeader, and their footers, split at&L,&C, and&R, a section switched to twice read as one;&&is an ampersand; a field such as&Por&Dstands as U+FFFC, which parts no words and joins none; font, size, colour, and toggle codes and any other&code are taken out, as LibreOffice prints them; one YARA pass per section, and a header that holds no&is left to the raw view; verification), the text of each HTML block that inline markup splits in a file named.html,.htm, or.xhtmlor that opens with<!doctype html>or<html>, including an HTML mail part (html-text: one YARA pass per paragraph, list item, table cell, heading, or other block, with bold, links, spans, and comments dropped and their text joined,<br>a line break, whitespace collapsed outside<pre>, and script and style bodies skipped; two blocks are never joined, and a block whose text is one run is left to the raw and HTML-unescaped views; two lexical passes, with no computed CSS visibility; the second recognizes additional markup the WHATWG tokenizer discards (<!-->,<!--->, a comment closed by--!>,</>, a bogus comment that holds<, a tag name of any length or spelling, a tag that holds<), and adds blocks beyond the first pass; comment and CDATA reads check the deadline), the text of an RSS or Atom feed, known by its XML root and namespace (feed-text: one piece per text node that a comment, a CDATA boundary, or the feed's own entities split, joined as an XML parser joins it, and one piece per block of escaped or CDATA HTML in RSSdescriptionandcontent:encoded, or in a declared Atom HTML field; Atom text and default text fields keep literal angle brackets after XML decoding, read with bothhtml-textreadings, and one piece per block of an Atom text construct of typexhtml(content,summary,title,subtitle, orrightsholding an XHTMLdiv), read the same way with element prefixes such asxhtml:premoved; two nodes are never joined; verification), the text of each node of another XML document that a comment, a CDATA boundary, or the document's own entities split, joined asxml.etreejoins it, for a document that opens with an XML declaration or is named.xml,.xsd,.xsl,.xslt,.plist,.csproj,.vbproj,.fsproj,.props,.targets,.xaml,.resx,.nuspec,.wsdl,.kml,.gpx,.opf,.ncx,.rdf,.xlf, or.xliff, and is neither a feed nor a documenthtml-textreads (xml-text: one piece per node; a document none of these split gives none, which is every part an office suite writes; escaped markup in a node is not read as HTML, since nothing renders it; verification); a document that declares more than 64 entities, or whose entities expand past 1 MiB, is read without them rather than left incomplete, except a feed or XML document whose text refers to one of the entities it declares, which is incomplete (xml entities exceed 64) because a parser would expand that reference into the text; an Atom end tag may hold white space before its>, and at most 32 distinct construct names are searched for one, the text of each Markdown paragraph, list item, or heading that inline markup splits in a file named.md,.markdown,.mdown,.mkd,.mkdn, or.mdx, or in a.txtor suffixless file that opens with a heading or front matter (markdown-text: emphasis and strong markers, strikethrough, code-span backticks, and underscores at a word edge dropped within each line, never paired across lines, sosnake_casestays; bracketed text such as a link, image, reference, or checkbox keeps its text and loses its brackets and target;>quote leaders removed; a blank line, heading, list item, thematic break, code fence, or deeper quote starts a new piece, and fenced code is not read until a closing fence of the same character at least as long; the suffix is taken from the final path component, so a suffixlessREADMEinside an archive directory such asproject.v1/is still read; one YARA pass per piece, and a piece with nothing removed is left to the raw view), the text of each block of two or more adjacent line comments in a text unit (comment-text: lines that open with the same//,#,--,;, or'leader, or the*lines of a/*block comment, joined with newlines after their leaders are removed, so a sentence wrapped over two comment lines reads whole; an empty comment line, a line of code, or another leader ends a block, a*line outside a block comment is a list item and not read, and a comment after code on the same line is not read; one YARA pass per block, so two blocks never form one phrase), and each Jupyter notebook cell source, stream output, andtext/*output stored as a list of two or more lines (notebook-text: a file named.ipynb, or text holding both"cells"and"nbformat"that parses as a JSON object with acellslist; one YARA pass per item, so two cells never form one phrase, a source held in one string is left tojson-unescaped, malformed or deeply nested JSON gives no items, and a code cell's source is also read forcomment-text), each also folded as<view>+text-folded, with at most 100,000 pieces per view before the scan is incomplete. The PDF view feeds both the YARA rules and the PDF name checks, so a/JavaScriptaction inside a compressed object stream blocks. Inflated streams that are not text (fonts, images) are dropped; a corrupt stream is skipped; each stream is capped at 100 MiB, the total at the shared 500 MiB budget, and more than 2,000 streams is incomplete.ASCIIHexDecodeandASCII85Decodestages beforeFlateDecode,LZWDecode, orRunLengthDecode, and PNG or TIFF predictors, are decoded. LZW (with itsEarlyChange) and RunLength are decoded in Python, so only for a stream that is not an image and whose encoded bytes are at most 2 MiB (LZW) or 8 MiB (RunLength); a larger one, and an ASCII chain that reaches none of the three, is not, and when that stream is page content it is reported as the noteheuristic_pdf_content_unreadand counted inpdf_content_unread. Originals remain unchanged. The text a PDF shows is rebuilt from its operators (pdf-text):Tj,TJ,', and"strings, literal or hex, decoded through each font'sToUnicodemap (abfrangewhose destination overflows its width, such as<0000> <00FF> <FFFF>, is read both ways readers resolve it: wrapped to that width, as PDFium, PDFBox, and pdfminer.six do, and as the code point past U+FFFF that pdf.js and MuPDF show, each a reading of the font; verification) or as WinAnsi, or, for a Type0 font with no map whose every/Encodingvalue is a predefined Unicode CMap (UniGB-UCS2-H,UniJIS-UTF16-H, and the rest of ISO 32000-1 Table 118'sUni*-UCS2andUni*-UTF16names), as UTF-16BE. A sole Unicode reading keeps the full decoded text. When a font name has multiple possible readings, the UTF-16BE reading drops format controls, combining marks, private-use and unassigned code points, and Hangul fillers because other fonts' glyph IDs can map to those characters, aTJadjustment of -200 or less read as a space, and each block also read with spaces where some readers put them (aTJgap of -100 or less, and a string on the same line afterTd,TD,T*,Tm, or a new text object), added where it differs, so a word placed by position reads as a word: poppler, pypdf, pdfminer.six, and PDFium space a word placed by its ownTdor after a -150 gap, and poppler and pdfminer.six one after -110, and each text object (BT...ET) joined as one block, cut where the text jumps to a new paragraph (more than 1.7 font sizes between baselines; the next line within that joins with a newline). Each block is a separate YARA pass. Generators start a new text object at every font change or line, so a paragraph that spans two or more blocks (a word in bold, a wrapped line) is also scanned whole, aspdf-paragraph; two paragraphs are never joined. Both views are also spelling-folded when different. At most one finding per rule and namespace is kept for each view; the two base64 rules are left topdf-flate, which tells image samples apart. Text longer than 64K characters is scanned in overlapping 64K chunks,q/Qsave and restore the selected font with the rest of the graphics state, a page's later content stream can continue a text object begun earlier, the streams of one/Contentsarray are read as one stream in array order, as ISO 32000-1 7.8.2 defines it and every reader reads it (a direct array, an array kept in an object of its own, or one in a page dictionary stored in an object stream; an empty member is passed over, a member that is reused, not decoded, or drawn through other fonts ends the run, and at most 1,024 streams are joined), a page content stream is read even when its bytes are not text, as with an inline image or a binary string before the text, when a/Contentsentry names it and its dictionary marks it as no image, form, or font program, every token (eachTJarray element included) counts toward the deadline, and more than 20,000 blocks, 4,000,000 tokens, or a 100,000-item array is incomplete.heuristic_pdf_text_unmappedis a note: a composite font has no readableToUnicodemap and is not read as UTF-16BE through a predefined Unicode CMap, a font the file never resolves shows two-byte codes, or a string is shown in a font resource name that means more than four fonts, so some text was not read. A page content stream reads each name through its own page's/Resources(direct or behind references; not inherited ones), so the pages of a merged file, which each keep their source's/F1, read one font apiece (page fonts). A stream several pages draw can mean each of their fonts. The page map is built only when some name means two fonts, and only for a file every reader reads the same way; everything else uses every font the name means anywhere in the file, as before the map existed. That covers a name one of the stream's pages does not list or lists as anything but a reference to one font dictionary, a font carried in from an earlier stream, a form XObject, a page with no/Resourcesof its own or with resources that do not resolve (defined twice with different values, under another generation, or not listed in use by the cross-reference table), and a stream referenced from anywhere but a page's/Contents(an annotation appearance, a Type 3 glyph procedure, a page object that repeats a key, escapes a name with#, or is written in the trailer). It covers the whole file when the file is not written plainly throughout: anything but comments, cross-reference sections, and trailers between objects; an object that is not one well-formed value; a byte outside strings and streams that is not printable ASCII, a tab, or a line end; a number or reference in a spelling only some lexers read (5.0 0 R,5 0 R#); a comment right after a number; a stream whose/Lengthdoes not land; bytes that read as an object header inside a string, a comment, or a stream; a cross-reference entry that does not agree with the object headers, or a table that cannot be read; an object stream that cannot be read, lacks/Type /ObjStm, or does not hold exactly one object in each member's place; and past 1,000,000 tokens, 20,000 page objects, 2 MiB of parsed objects, 250,000 font names given to streams, or 100,000 objects. probe_pdf_reader_agreement.py checks the map against six PDF readers. Each string is decoded through at most four of the fonts its name means (an overflowingbfrangeadds a second reading after every font's first), and at most 64 fonts per name are resolved; text only a later font shows is not scanned (verification). A font object orToUnicodemap that the file defines more than once (an incremental update, a number reused under another generation, an object-stream member beside a raw object) is read through every definition, since readers differ in which one they use; each distinct definition is one of those readings, and a font with more than four, or more than 64 raw definitions, is reported the same way (verification). Text a Word, Excel, PowerPoint, or PDF reader hides is also read on its own, one piece per hidden stretch, asooxml-hidden-textandpdf-hidden-text(each folded as<view>+text-folded); an instruction there addsheuristic_hidden_text(hidden Office and PDF text). Office text inside other parts (settings, styles, embedded objects) is not joined. Hex, ROT13, and base64 nested inside another encoding are not decoded. Annotation/Contentsstrings are separate pieces (pdf-annotation, folded when the spelling changes), and so are form field values, document information strings, and outline titles (pdf-string, boundaries). So is each distinct/ActualTextstring of marked content in a content stream (/Span <</ActualText <FEFF...>>> BDC ... EMC), which poppler and PDFium show in place of the glyphs it covers, decoded as apdf-stringis (pdf-actual-text, folded when the spelling changes; boundaries). No general document-text extraction, OCR, or semantic prompt-injection guarantee is provided. - PDF CMaps share budgets for tokens, entries and decoded map bytes across fonts. One map keeps at most 131,072 entries; entries past that are not read, without an incomplete result. The scanner checks the 64 MiB map expansion limit before allocating range entries, and limits decoded PDF text to 16,777,216 characters before joining repeated glyphs. Exhaustion yields an incomplete scan.
- Reads are capped by remaining byte budgets. A shared deadline is checked between extraction chunks and scan/view operations, and YARA receives the remaining time. Third-party archive metadata and Office parsing are not preempted by an OS process deadline.
- A
.pdfthat is really a zip is an advisory type mismatch and exits 0. A.pdfthat is really a PE, ELF, or Mach-O exits 1. The same check reads.jar,.apk,.msix,.appx,.xpi,.vsix,.nupkg,.whl,.epub, and OpenDocument names as zips;.xll,.cpl,.ocx,.sys,.drv, and.efias PE; and.csv,.svg,.xml,.rtf,.ini,.url,.scf,.reg,.ps1,.vbs,.js,.sh,.bat, and.cmdas text, so a PE saved under any of those text or zip names exits 1. - A scan of a path outside the inbox does not move that file.
- YARA-X keeps the first 1,000,000 matches of each string and drops the rest without notice, so a rule that reads where a match lies could miss one after a million others, in an input of a megabyte or more. Each bundled rule that does so counts a full list as a match in place: it reports its finding rather than clear the file (record). The scanner also guards every rule, vendor packs included. Each rule file is compiled a second time with a private copy of every string whose count or place a rule reads (a string tested only for presence keeps its first match, so it cannot hide one), which reports any of those lists at the limit, and a view of a million bytes or more is scanned with that set. A view that fills one to four lists is scanned again with a limit of 4,000,000 when the time left allows it, and the report's
match_rescan_limitrecords it. A list still full then, or more than four full lists, makes the scan incomplete (match cap reached) with its findings kept, so the file stays held. A rule file whose guard cannot be built loads as before and is counted inunguarded_rule_files(record).