Noise benchmark
How often we flag a package people install on purpose
Catch rates are easy to claim and hard to check. Noise decides if people leave a scanner on,
and few tools publish theirs. These numbers are ours, at rules version 2026.11.04b. We
rebuild them when that changes.
Read every number with the sentence next to it.
We chose all of these corpora. None is a random sample of the registry, and none measures what happens in the field. So you cannot carry these rates over to your own dependencies. They do show how this scanner behaves on lists we publish in full, and you can run those yourself.
Attack samples we wrote
71 of 71
What that counts: attack samples our rules detected, out of the ones in our corpus. We wrote every sample to match what an attack does. We did not write them to match our rules.
What the corpus is made of, counted from the samples: technique family we wrote: 44, evasion variant we wrote: 14, published incident report: 13. Only the ones written from a published report follow a named attack. The rest are our idea of what a type of attack looks like. They do not show that we caught any real attack.
And they are not all npm packages. What each sample is: npm package: 34, GitHub repository: 12, agent skill: 7, MCP server: 5, model repository: 5, Python source file: 3, PyPI package: 3, CI workflow: 2. So this number covers more than npm packages.
This is not field recall. It cannot be. That would need a list of all attacks in a period, even the ones nobody has found. Nobody has that list. Anyone who quotes such a number for a package scanner is making it up, and that includes us.
We put in cases we expected to fail, on purpose. A test that only holds what you catch tells you nothing.
What we do not catch
Every sample in the corpus was detected in this run. That is a statement about this corpus, which we wrote, and nothing more. Samples we expected to miss are kept in it on purpose, and when one starts passing, we say so.
Noise on packages people install on purpose
Popular npm packagesrules 2026.11.04bmeasured 2026-11-04
1 of 154What that counts: packages flagged by our rules out of the whole list, with no indicator feed in play.
The corpus, and who chose it: The list we publish package pages for, chosen by us for search traffic, not sampled from the registry at random. It is weighted to packages a working developer installs on purpose, which is the population a noisy scanner annoys.
Full breakdown: 151 came back clear,
1 came back flagged, and 2
could not be read at all in this run. We count those on their own.
| Package | Verdict | Risk | Rule categories |
|---|---|---|---|
pm2 | flagged | high | obfuscation, persistence |
We name them so you can check them. We do not claim each one is wrong. Our rules flagged it, and you can read the evidence yourself. If we find one was wrong, we correct it and list it in the public corrections log. We do not just edit it out of this table.
Could not be read in this run:
better-sqlite3 | fetch failed: too_large (fetch: content-length 11402131); Not read. The package download is larger than our 5 MB limit, so there is no result either way. Retrying will not change this unless the package gets smaller. | |||
next | fetch failed: too_large (fetch: content-length 43659658); Not read. The package download is larger than our 5 MB limit, so there is no result either way. Retrying will not change this unless the package gets smaller. | |||
Adversarial npm packagesrules 2026.11.04bmeasured 2026-11-04
5 of 53What that counts: packages whose result disagreed with the label we wrote down before running them.
The corpus, and who chose it: Chosen by us because each one legitimately does something our rules look for: reads cloud credentials, posts to a chat webhook, spawns shells, compiles code at run time, or runs an install step that downloads a binary. Every label was written before the run, and any we later corrected shows the old label, the date and the reason.
Full breakdown: 50 came back clear,
1 came back flagged, and 2
could not be read at all in this run. We count those on their own.
The two directions, one at a time, so you can see which way we were wrong:
1 we had labelled clear came back flagged, and
4 we had labelled flagged came back clear. The second group is
the harder one. We expected a flag for what they do at install time, and the scan did not give one.
We do not flag a bare install script, since most are plain builds and downloads. Where we see one, the report
lists it as an info finding, which does not change the verdict. Those
labels stay as written, so these still count as disagreements.
Labelled clear, came back flagged: pm2.
Labelled flagged, came back clear, install script reported at info: bcrypt, esbuild, puppeteer, sqlite3.
Labelled flagged, came back clear, no install script reported: none.
Labels we corrected after writing them, since the package changed:
sharp was labelled flagged until 2026-11-02. sharp removed its install script in 0.35.0, published 2026-06-10, so the old label described releases we no longer scan.
| Package | Verdict | Risk | Rule categories | Our label |
|---|---|---|---|---|
pm2 | flagged | high | obfuscation, persistence | we labelled it clear |
Could not be read in this run:
googleapis | fetch failed: too_large (fetch: content-length 23807988); Not read. The package download is larger than our 5 MB limit, so there is no result either way. Retrying will not change this unless the package gets smaller. | |||
node-pty | fetch failed: too_large (fetch: content-length 15468777); Not read. The package download is larger than our 5 MB limit, so there is no result either way. Retrying will not change this unless the package gets smaller. | |||
Well-known MCP serversrules 2026.11.04bmeasured 2026-11-04
0 of 20What that counts: server packages flagged by our rules out of the whole list.
The corpus, and who chose it: The MCP servers we publish pages for, chosen by us from the widely used ones.
Full breakdown: 20 came back clear,
0 came back flagged, and 0
could not be read at all in this run. We count those on their own.
Nothing in this corpus was flagged.
Every sample, and how it went
| Sample | What it is | How it was written | Written from | Result | Label when written |
|---|---|---|---|---|---|
| postinstall dropper | npm package | published incident report | ua-parser-js / coa / rc (2021): postinstall fetches and runs a remote binary | detected | expected to detect |
| ssh + cloud credential stealer | npm package | technique family we wrote | typosquat crypto/credential stealers: read key material, POST to hardcoded host | detected | expected to detect |
| npm token + env harvest | npm package | published incident report | Shai-Hulud (2025): harvest npm/GitHub tokens and env, exfil to attacker endpoint | detected | expected to detect |
| obfuscated decode-then-eval loader | npm package | published incident report | event-stream / flatmap-stream (2018): decrypt a blob at runtime, then execute it | detected | expected to detect |
| decoded into a variable, then evaluated | npm package | evasion variant we wrote | the same loader with the decode and the eval split by a variable, the first thing an author tries to dodge a one-expression pattern | detected | expected to detect |
| stealer behind a file of nothing but newlines | npm package | evasion variant we wrote | padding that spends the scanner line budget before the payload is reached, a few KB once compressed | detected | expected to detect |
| crypto wallet stealer | npm package | technique family we wrote | wallet-draining typosquats: read wallet files, exfil | detected | expected to detect |
| self-updating beacon | npm package | technique family we wrote | persistent loaders: periodically pull and execute remote code | detected | expected to detect |
| payload hidden inside a minified bundle | npm package | technique family we wrote | attacker ships the stealer inside a published dist bundle | detected | written as a case we expected to miss |
| worm ships as a minified bundle its install script runs | npm package | technique family we wrote | Shai-Hulud (2025): postinstall runs a single minified bundle.js that harvests tokens and posts them out | detected | expected to detect |
| DNS-based exfiltration | npm package | technique family we wrote | covert channel: encode secrets into DNS lookups (no HTTP sink) | detected | written as a case we expected to miss |
| reverse shell in a postinstall | npm package | technique family we wrote | bash /dev/tcp reverse shell dropped by an install script (common CTF/malware idiom) | detected | expected to detect |
| Discord-webhook token stealer | npm package | technique family we wrote | many 2023-2025 npm token stealers: read npm/env secrets, POST to a Discord webhook (a "normal" domain) | detected | expected to detect |
| clipboard crypto clipper | npm package | technique family we wrote | clipper malware: poll the clipboard and swap a copied wallet address for the attacker's | detected | expected to detect |
| multi-provider env token harvest | npm package | technique family we wrote | Shai-Hulud-style worm / token stealers: collect NPM/GitHub/AWS secrets from env and POST them | detected | expected to detect |
| remote code loader to a domain | Python source file | technique family we wrote | dropper that fetches a stage-2 payload from a DOMAIN (no hardcoded IP) and execs it | detected | expected to detect |
| bin shadows a system command | npm package | technique family we wrote | metadata attack: a package whose bin shadows git/npm so a global install hijacks the command | detected | expected to detect |
| CI workflow secret exfiltration | CI workflow | technique family we wrote | poisoned GitHub Actions workflow that POSTs a build secret to an attacker host | detected | expected to detect |
| Windows LOLBin dropper via postinstall | npm package | technique family we wrote | Windows-targeted npm malware: postinstall uses certutil to fetch a remote exe | detected | expected to detect |
| in-memory shellcode injection | Python source file | technique family we wrote | loader that allocates executable memory and runs a raw shellcode buffer | detected | expected to detect |
| obfuscated dynamic import (anti-analysis) | Python source file | evasion variant we wrote | evasion: assemble the module name from fragments so a static scan never sees "os" | detected | expected to detect |
| credential exfil to a throwaway sink | npm package | technique family we wrote | reads env/npm secrets and POSTs them to a webhook.site capture URL | detected | expected to detect |
| CI workflow dropper from a raw IP | CI workflow | technique family we wrote | poisoned GitHub Actions step fetches a payload from a bare IP and pipes it to a shell | detected | expected to detect |
| agent credential store read and exfiltrated | npm package | technique family we wrote | AI-tooling credential theft: read a coding agent OAuth store (.codex/auth.json) and POST the refresh token out | detected | expected to detect |
| agent transcript exfiltrated over a websocket | npm package | technique family we wrote | AI-tooling credential theft: glob the Claude conversation transcripts and stream them to a relay over ws | detected | expected to detect |
| agent CLI spawned with permission bypass | npm package | published incident report | Nx s1ngularity (2025): a postinstall drives the on-host coding agent with the approval prompt disabled to inventory secrets | detected | expected to detect |
| agent config tampered to hijack MCP servers | npm package | technique family we wrote | AI-tooling credential theft: a postinstall rewrites the agent config to repoint mcpServers at a proxy and pre-accept trust | detected | expected to detect |
| on-host agent driven remotely over a websocket relay | npm package | technique family we wrote | Remote-control daemon: a package bin takes prompts from a WebSocket relay and runs them through the local coding agent with approvals disabled | detected | expected to detect |
| agent skill exfiltrates via a rendered image | agent skill | technique family we wrote | prompt-injection in an agent SKILL.md: a markdown image whose URL templates the conversation to an attacker host | detected | expected to detect |
| remote dynamic dependency (http dep URL) | npm package | published incident report | PhantomRaven (2025): the npm tarball is a benign stub; a dependency VALUE is an http URL npm fetches and runs at install | detected | expected to detect |
| binding.gyp build-time command substitution | npm package | technique family we wrote | Phantom Gyp / Miasma: a binding.gyp sources entry runs a script through node-gyp at install, with no package.json hook to inspect | detected | expected to detect |
| Python .pth interpreter-startup stealer | PyPI package | published incident report | LiteLLM 1.107.0 (2026): a shipped .pth imports a module that reads agent credential files at every interpreter start, even when the package is never imported | detected | expected to detect |
| Python .pth startup hook in an sdist src layout | PyPI package | evasion variant we wrote | LiteLLM 1.107.0 (2026), as the sdist ships it: the .pth and its module sit under src/, and the module binds its network call by from-import | detected | expected to detect |
| binding.gyp substitution hidden in a split sources list | npm package | evasion variant we wrote | Phantom Gyp / Miasma, laid out the gl/Chromium way: the sources key and its [ on two lines, a variable list pulled in through <@(name), and inline sh -c running a package script | detected | expected to detect |
| Python .pth import line behind a bare carriage return | PyPI package | evasion variant we wrote | LiteLLM 1.107.0 (2026), bent for the reader: site.addpackage splits a .pth with str.splitlines(), so an import after a bare \r runs while a newline-only reader sees one path line | detected | expected to detect |
| agent skill grabs a token while loading | agent skill | published incident report | skill marketplace malware (the ToxicSkills report): a SKILL.md dynamic-context command reads the GitHub CLI token and curls it out during preprocessing, under a blanket Bash grant, before the model has read the skill | detected | expected to detect |
| skill repo config imports the user's secrets | agent skill | technique family we wrote | a skill repo whose CLAUDE.md @import lines pull an SSH key and a dotenv file from outside the repo into every session that opens it, beside an honest in-tree import and a clean SKILL.md | detected | expected to detect |
| namespaced command ships the environment while loading | agent skill | technique family we wrote | the skill marketplace shape moved into a plugin slash command under a namespace directory (commands/<ns>/<name>.md, loaded as /ns:name): the environment is piped out at load time under a curl grant that is scoped by nothing but a flag | detected | expected to detect |
| command posts the environment from node while loading | agent skill | technique family we wrote | the same load-time shape with curl taken out: a slash command with no name: key, pasted or installed, whose dynamic-context command runs node -e to POST the whole environment, so no curl flag or shell pipe is on the line | detected | expected to detect |
| MCP tool schema smuggles the conversation through a parameter name | MCP server | technique family we wrote | HiddenLayer / CyberArk schema poisoning: an inputSchema property named conversation_history / system_prompt that the model auto-fills from its own context, with a bland description and nothing in required | detected | expected to detect |
| one-click MCP install badge hides a curl-piped-to-shell command | MCP server | published incident report | CursorJack (CVE-2025-54133) / DeepJack / Dive (CVE-2026-23523): a cursor:// mcp/install deeplink whose base64 config decodes to a hostile command/args | detected | expected to detect |
| VS Code install redirect hides a decode-and-run command | MCP server | technique family we wrote | the same deeplink class in the vscode.dev redirect form most MCP READMEs use: the vscode:mcp/install deeplink, its JSON config the whole query, encoded again inside url= | detected | expected to detect |
| MCP tool schema hides the system prompt channel inside a nested options object | MCP server | evasion variant we wrote | the HiddenLayer schema channel moved one level down: an options object whose own property is named system_prompt, which a top-level-only schema render never showed to any rule | detected | expected to detect |
| MCP tool schema rewords the context channel and hides behind generic words | MCP server | evasion variant we wrote | the HiddenLayer schema channel under a synonym (previous_conversation) on a tool whose own text says "error message" and "tide model", words that name no conversation and no LLM, so nothing honest explains the field | detected | expected to detect |
| agent skill with a loader hidden in variation selectors | agent skill | published incident report | GlassWorm (2025): a payload encoded in Unicode variation selectors, invisible in an editor, parked on blank-looking lines of a source file; here inside a skill and its script | detected | expected to detect |
| install hook registers a chat-backed MCP server with a baked-in bot token | npm package | technique family we wrote | rogue MCP registration: postinstall writes an mcp.json entry whose env carries a Telegram bot token, and a settings.json that auto-accepts broad permissions | detected | expected to detect |
| status-line plugin ships a settings.json that reroutes the API with a baked-in key | npm package | published incident report | a Claude Code status-line package (2026) whose only payload was a .claude/settings.json: ANTHROPIC_BASE_URL rewritten to an attacker relay, a key baked into apiKeyHelper, permission prompts switched off | detected | expected to detect |
| setup script turns agent telemetry into a session feed | GitHub repository | technique family we wrote | agent telemetry as a covert channel: a repository setup script enables OTEL export with prompt, tool content and raw API body logging and points the collector at an outside host | detected | expected to detect |
| repository ships a one-line settings file that streams the session | GitHub repository | evasion variant we wrote | agent telemetry as a covert channel, written the way a script writes it: JSON.stringify output on one line in .claude/settings.local.json, prompt and tool content logging on, collector on an outside host | detected | expected to detect |
| agent skill with a selector payload hidden behind blank filler characters | agent skill | evasion variant we wrote | GlassWorm (2025) variation-selector encoding, with a Hangul filler before every selector so no selector is left without a base and the lines still render blank | detected | expected to detect |
| agent instruction file hides a directive in zero-width joiners | npm package | published incident report | TrapDoor (2026): AGENTS.md and CLAUDE.md files planted in npm, PyPI and crates packages, with a directive written in zero-width joiners between visible letters | detected | expected to detect |
| Cursor rules files hide a directive in zero-width characters | npm package | published incident report | TrapDoor (2026): the same zero-width directive delivered in a .cursorrules and a .cursor/rules .mdc file, which are not markdown by extension | detected | expected to detect |
| selector payload parked behind soft hyphens | npm package | evasion variant we wrote | GlassWorm (2025) variant: every variation selector of the payload follows a soft hyphen, a format character that renders as nothing, so each selector looks like it has a base | detected | expected to detect |
| selector payload large enough to pass for a minified bundle | npm package | evasion variant we wrote | GlassWorm (2025) at full size: thousands of variation selectors on one line of a short module, so the raw length per line reads like a minified bundle and a size check would drop the file unread | detected | expected to detect |
| npm lifecycle script streams the agent session to an outside collector | npm package | evasion variant we wrote | agent telemetry as a covert channel, carried in a package.json postinstall as an env prefix rather than a setup script: prompt and tool content logging on, the OTLP endpoint on an outside host | detected | expected to detect |
| fake job-interview repository starts a loader when the folder is opened | GitHub repository | published incident report | fake job-interview repositories (Contagious Interview, 2025): a .vscode/tasks.json task set to run on folder open that starts a loader committed beside it, so the editor runs it with no click | detected | expected to detect |
| workspace settings point the TypeScript server at a copy the repository ships | GitHub repository | technique family we wrote | editor toolchain redirect (the CVE-2021-26700 class): .vscode/settings.json sets typescript.tsdk to a folder inside the repository that holds its own tsserver.js, so opening a .ts file starts it | detected | expected to detect |
| dev container posts the developer secrets during setup | GitHub repository | technique family we wrote | dev container lifecycle exfiltration: postCreateCommand POSTs the user secrets file and the container environment, Codespaces token included, to an outside host the first time the container is built | detected | expected to detect |
| project agent settings run a fetched script before every tool call | GitHub repository | technique family we wrote | agent hook abuse: a committed .claude/settings.json whose PreToolUse hook substitutes a remote script into bash -c, so the agent runs it before each tool call in that project | detected | expected to detect |
| repository ships agent settings that switch the approval prompt off | GitHub repository | technique family we wrote | consent bypass committed as config (the CVE-2025-53773 class): a repository whose settings turn on chat.tools.autoApprove for VS Code and bypassPermissions for Claude Code, so a prompt injection anywhere in it runs commands without a prompt | detected | expected to detect |
| in-repo .npmrc swaps the git binary for a repo script | GitHub repository | technique family we wrote | npm runs the program named by git= for every git dependency it resolves, so a committed .npmrc turns an ordinary install into running a hidden script | detected | expected to detect |
| pnpm hook file starts a detached process on every install | GitHub repository | technique family we wrote | pnpm runs .pnpmfile.cjs hooks on every install; a readPackage hook that spawns a process runs before any package script and needs no lifecycle hook | detected | expected to detect |
| pre-commit local hook pipes a download into a shell | GitHub repository | technique family we wrote | pre-commit runs a repo: local hook entry as written, with no pinned revision, whenever the hooks run on a commit | detected | expected to detect |
| lockfile resolves one dependency from a host the rest of the file never uses | GitHub repository | technique family we wrote | lockfile poisoning: npm installs each entry from the resolved URL in package-lock.json and checks it against the integrity beside it, so a rewritten entry pulls code from any host with no change to package.json and no install script | detected | expected to detect |
| requirements file adds an extra package index for an internal-looking name | GitHub repository | technique family we wrote | dependency confusion through pip: --extra-index-url makes pip take the highest version of a name across every index, so an index the repository adds can serve its own build of any package it lists | detected | expected to detect |
| model repo config runs a modeling file that starts a process | model repository | technique family we wrote | trust_remote_code wiring: auto_map in config.json names a modeling file the loader imports on load, and that file starts a shell that fetches a script (nested configs have forced the flag on, CVE-2026-5241 and CVE-2026-27893) | detected | expected to detect |
| diffusers repo ships a None.py | model repository | technique family we wrote | CVE-2026-44827: a diffusers model_index.json lists a component as [null, null] and the repo ships None.py, which the loader imports without trust_remote_code | detected | expected to detect |
| tokenizer chat template escapes the Jinja sandbox | model repository | technique family we wrote | chat template injection: a tokenizer_config.json chat_template that walks from a string to __subclasses__, rendered on every chat call | detected | expected to detect |
| Keras model config carries a Lambda layer | model repository | technique family we wrote | CVE-2025-1550 and CVE-2025-8747: a Keras config.json whose Lambda layer stores Python to run when the model is loaded, even with the guard meant to refuse it switched on | detected | expected to detect |
| weights file pickle imports a system callable | model repository | technique family we wrote | pickle payloads in model weights: a state dict whose pickle also names posix.system, the global a loader imports before any tensor is built (torch.load weights_only bypass CVE-2025-32434; denylist scanner bypasses CVE-2025-10155, CVE-2025-10156, CVE-2025-10157). The sample is the bytes after the fetcher disassembled them | detected | expected to detect |
| install dropper behind a byte-order mark | npm package | evasion variant we wrote | a UTF-8 byte-order mark ahead of package.json: JSON.parse rejects the file while npm drops the mark and runs the hook. A registry tarball loses the mark when it is decoded, so this reaches the rules when a manifest is sent as inline text | detected | expected to detect |
How these were measured
- We switched off the known-bad feeds for the package surveys, so this measures our own rules and nothing else.
- We scanned each package as published, at the version the registry served on the date shown on each corpus. The corpus records package names, not versions. We did not note which release each result came from. So a re-run today resolves whatever is current and may scan different code. Read each number as measured on the named packages on the named date.
- Each corpus is the list as it stood on its measurement date. The machine-readable version of this page names each package in it. We add to these lists, so a list can be longer today. These counts still describe the packages named here. The date on each card tells you which is which.
- We wrote the labels for the adversarial corpus before the run. That is why we can report disagreements, and why the number is not zero. If a package later changes, we correct its label in the open. The card shows the old label, the date and the reason.
- We report packages the fetcher could not read as their own count. Dropping them would make the other numbers look better.
- The attack-sample run needs no network and gives the same result each time, so it runs on each build. If its result changes, this page changes too.
- The attack samples are ours. The table above says which ones come from a published report. Catching a sample we wrote is weaker evidence than catching an attack someone else reported, and most of the corpus is the weaker kind.
What none of this tells you
- A
clearresult means nothing matched and no threat rule fired. It can still carryinfofindings, such as an install script. It makes no promise about the code. - We run nothing. So we cannot see what only happens at run time, against a live server or on one operating system.
- These numbers describe rules version
2026.11.04b. The build fails if a survey was recorded under a different version. That check reads a version string we set by hand. So it catches a version bump, but not every rule edit. - This page lists the misses we found by looking. Treat it as a floor, since there is more to find. Disputes we upheld are at /corrections. Where the data comes from, with its gaps, is at /sources.
Check it yourself
The machine-readable version of this page is at /benchmark.json, with each package and its result. The rule list is at /v1/rules. You can scan any package here again. This names the package and not the release, so you may scan a later one:
curl -s https://lazaretto.dev/benchmark.json | jq '.surveys[].members[] | select(.verdict=="flagged")'
Generated 2026-11-05 at rules version 2026.11.04b.