HyperChecker
Send an Instagram Reel by DM, and this agent checks the developer tools it mentions against public sources and replies.
The problem
Reels are full of lines like "this one MCP is all you need" or "tens of thousands of stars." Checking them means finding the repo, ruling out clones with the same name, comparing the claims with the README, and reading the install script. HyperChecker turns that into a single share.
Principle: code decides the verdict
It never asks an LLM "is this tool good?" Code calculates the grade and score from measurements. The LLM handles jobs such as extracting tools, reading on-screen text, and finding evidence in the source, and its output is checked by code before it affects a verdict. When it cannot confirm something, it withholds a verdict instead of guessing.
The grade answers "how well was the post verified?" The score (0–10) answers "how strongly is the tool itself recommended?" The score uses only stars, recent activity, and warning signals, and the reply shows how it was calculated.
| Grade | Meaning |
|---|---|
| 👍 Verified against official docs | Location, measurements, and activity check out, and the post matches the official docs |
| 🤔 Use with care | Found the official location, but some claims are unverified or there are warnings |
| 👎 Not recommended | Claims contradict the docs, a security placeholder package, or an impersonating clone |
| ❔ Verdict withheld | Could not settle on a single official location |
Real replies
Output from running real posts through the current code. Replies are in Korean; a confirmed tool shows the score breakdown.
2. morphic — 추천 7/10 · 👍 🔗 github.com/miurla/morphic ⭐ 9,257 · 최근 30일 안에 업데이트 👍 좋은 점 · 게시물의 '나만의 AI 검색엔진' 설명이 공식 문서와 맞아요 · 공식 위치 하나를 찾았어요 📊 기본 5점 · 스타 규모 +1 · 업데이트 +1 → 7점
Recommended 7/10 · 👍. The post's "your own AI search engine" claim matches the official docs; one official location found. Base 5 + star count 1 + recent update 1 = 7.
When it cannot confirm, it shows candidates and says they are unconfirmed.
1. puck — 추천 판단 보류 · ❔ 🔎 이 도구일 수 있어요(확정 아님): github.com/puckeditor/puck ⭐ 13,606 ⚠️ 살펴볼 점 · 공식 위치를 하나로 정하지 못했어요 · 측정한 코드 정보가 없어요
Verdict withheld · ❔. "This might be the tool (unconfirmed)." Could not settle on one official location; no code measurements.
Architecture
Instagram DM → serve signed webhook, file queue → work one item at a time → read video transcript · on-screen text → extract tools Claude → identify caption links → registries → name search → read-only measurement GitHub · npm · PyPI · MCP registry → compare claims verify quoted source text → grade · score code rules → DM reply
| Area | Stack |
|---|---|
| Runtime | Python 3.14, mostly standard library; Docker Compose; Oracle Cloud |
| LLM | Claude Haiku 5.5 (extraction, on-screen text, claim comparison, same-tool checks) |
| Video | yt-dlp, ffmpeg, Groq Whisper |
| Measurement | GitHub API, npm, PyPI, MCP registry, deps.dev |
| Storage | File queue; SQLite cache (same post 7 days, same tool 3 days) |
| Entry | Meta Instagram API, Cloudflare Tunnel + WAF |
Guardrails
The starting assumption: neither LLM output nor post content is trusted.
- LLM claims are checked. To say a claim matches the docs, Claude must return a paragraph number and a quoted passage, and code confirms the passage is really in that paragraph.
- Posts are untrusted input. Captions, transcripts, on-screen text, and READMEs go into
<untrusted>blocks as data only, and output is limited to fixed values and source quotes, which reduces how much instructions inside a video can affect the verdict. - Tools under review are never executed. No installs, no
npx; only public metadata and docs are read. Subprocesses are allow-listed, and the container runs with a read-only root and all capabilities dropped. - Cost and abuse are capped. Two requests per account per day, 60 in total, and an estimated $2 per day. Each item has a 180-second deadline, and plain-text DMs never reach Claude.
- Retention is short. Videos and frames are deleted right after processing, and processing records after 7 days. Sender account IDs are not passed to external APIs; captions and public official docs are sent to Anthropic.
Evaluation
The answer set has 32 cases built from human-judged cards and adversarial examples. To pass: 👍 precision of at least 90% (automatic 👍 that a human rated 👍 or 🤔), zero critical errors (human 👎 → automatic 👍), at most one 👎 false positive, and at most 40% ❔ (tools whose correct answer is ❔ are left out of that ratio). Results vary between runs, so it was run several times and compared (2026-11-03).
| Before fixes | Current code | |
|---|---|---|
| Dev set (19 tools) passed | 2 of 2 runs | 2 of 2 runs |
| Holdout (9 tools) passed | 1 of 3 runs | 2 of 3 runs |
| Holdout runs with a critical error | 2 of 3 runs | 1 of 3 runs |
Across all five runs, the current code kept the grade cap on adversarial cases such as prompt injection.
The limits of these numbers: the sample is small, and the holdout was looked at repeatedly while the rules were being fixed, so it is not an independent test. In the earlier CLI stage, a validation set was committed and frozen before its first run; first-run identification was only 9/36, but all 9 confirmed identifications were correct. In 24 of the 27 misses, the right repo was in the top 3 but same-name clones and impersonators kept it from being settled.
In the earlier CLI stage, extraction and identification were scored as separate tasks, with a fixed denominator for identification. A separate session with no access to the answer set picked candidates from the frames, to check that exposure to the answers had not inflated the scores. For today's DM verdicts, the table above is the whole result. Details:eval-history (Korean) ↗
What production logs changed
Reading real replies after launch turned up problems the answer set had missed.
| Found | Cause | Fix |
|---|---|---|
| No 👍 ever; 3 of 42 claims matched | Code rejected 9 of the 20 claims Claude confirmed ([3] markers, Markdown differences) | Normalize formatting, but keep the rule that the quote must be in the paragraph |
| ❔ despite a GitHub link in the caption | Searched by name only | Use links in the post before searching |
| 4 found tools dropped from a "TOP 10" post | Unfound tools took the verdict slots | Judge found tools first and list the missing names |
| 0 stars and 7,000 stars both scored 5 | Activity bonus earned by the owner's own commits | No bonus with few users, and a stronger penalty |
Star-spike warning on NSA's ghidra | One viral moment flagged as manipulation | Keep a notice, drop the penalty and the 👍 block |
Design decision: take the criteria, not the fork
Before using Starguard (★500, Apache-2.0) for star-manipulation detection, I read its code without installing or running it. Its collection layer returned partial data as if it had succeeded when errors occurred, its two key signals (lock-step and clone impersonation) had bugs that broke their reasoning, and it had no automated tests.
So instead of forking, I reimplemented only its criteria, with attribution. Account-based signals cannot be computed for someone else's repo, so only the star-count time-series signals and the fork/star ratio came over. A fork would have given a few dozen lines of constants, while fixing and maintaining it alone would have cost more.Review notes (Korean) ↗
Known limits
- It cannot catch exaggeration in the README itself. 👍 means "the developer's description matches the post." If the docs are right but the behavior differs, it can miss that.
- It does not check that features actually work, because it never runs them.
- Up to six tools per post are judged, for cost reasons; the rest are listed by name.
- One item at a time. About a minute each, so replies slow down under load.
- Automatic results can be wrong. Every reply says so, and users can answer
정정("correction").