Effect Bloom Filter URL Frontier
Crawler-grade dedupe in a fixed bit array: k hash positions per URL (double hashing over FNV-1a with an avalanche finisher), textbook sizing from capacity and target false-positive rate, and a bounded frontier Queue so discovery backpressures fetchers instead of buffering the web in memory. The trade sits on the safe side by construction: false negatives are impossible, so a seen URL is never re-crawled and the crawler cannot loop, while false positives arrive at the configured rate and each costs one missed page, never correctness. The demo measures 100k URLs in ~117KB against a ~12MB exact Set and confirms the observed false-positive rate against the predicted one. Pinned to effect 4.0.0-beta.98.
npx shadcn@latest add @compronents/effect-bloom-url-frontierInstalls from ui.aryank.space. Needs a registry token and a one-time @compronents entry in components.json. The docs walk through both.
The crawler remembers every URL it has fetched in an exact in-memory set, and the set grows with the web: 100k URLs is 12MB, a billion is tens of gigabytes per worker. Watch the resident memory climb until the fleet is sized by its dedupe structure instead of its work.