Why a one-time review is not enough
A plugin review is a snapshot. The npm package behind it publishes new versions, the maintainer goes quiet, a dependency gets a CVE. None of that shows up in the review you did three months ago. If your only gate is a human reading a README once, you are trusting a decision that expires.
What to scan, in order of signal
- Maintenance recency — when was the last publish, and is the repo still alive
- Dependency surface — how many transitive packages you are inheriting
- Install scripts — anything running postinstall deserves a read before it runs on your machine
- Permission scope — what the plugin asks for versus what it actually needs
- Docs quality — a plugin that cannot explain itself is a plugin you will misconfigure
Where the gate belongs
Put the scan at the point where the plugin list changes, not at deploy time. A plugin addition is a dependency change, so it belongs in the same pull request as the manifest edit. That way the diff shows both the code change and the quality delta.
A minimal pipeline shape
- On pull request, resolve the plugin manifest and emit the current plugin set
- Fetch quality signals for each plugin (publish date, dependency count, install scripts)
- Compare against the previous set and fail only on regressions
- Post the delta as a PR comment so a human sees what changed
Keeping the gate from crying wolf
The fastest way to kill a security gate is false positives. If the check fails on every commit, people start ignoring it, and then it protects nothing. Fail on regressions rather than absolute thresholds, because a plugin that was always mediocre is not the problem to solve today.
FAQ
Should the scan block merges?
Only for regressions in the risky categories — install scripts and permission scope. Everything else should warn.
How often should signals refresh?
On every plugin-list change, and on a weekly schedule so slow rot still surfaces.
Does this replace manual review?
No. It replaces the part of review that a machine does better, and frees the human to read the code that matters.