Why bulk checking breaks so easily
Checking one domain's WHOIS record is trivial. Checking two thousand of them without getting throttled, temporarily banned, or fed stale cached data is a genuinely different problem. WHOIS servers and RDAP endpoints are built for occasional lookups, not bulk scanning — and they enforce that with rate limits that vary wildly by registry, often undocumented, and sometimes IP-based rather than per-request.
Where people get it wrong
Hammering a single server
The most common failure: firing hundreds of requests per second at one WHOIS server from one IP. Most registries respond with a temporary block ranging from minutes to 24 hours. Once blocked, every check from that source returns garbage or empty responses — silently corrupting your data instead of failing loudly.
Ignoring RDAP in favor of legacy WHOIS
RDAP (Registration Data Access Protocol) is the modern replacement for WHOIS — structured JSON instead of free-text parsing, and generally more generous rate limits since it's built for programmatic access. Falling back to legacy WHOIS-only checking means dealing with the older protocol's stricter limits and inconsistent formatting across registries, for no real benefit when RDAP is available.
No backoff on failure
A naive bulk checker retries a failed request immediately, which — against a server that just rate-limited you — guarantees the next request fails too. Exponential backoff with jitter (wait longer each retry, with some randomness so requests from different sources don't retry in lockstep) is the difference between recovering gracefully and getting your entire batch blocked.
What actually works at scale
- RDAP-first, WHOIS fallback: try RDAP where the registry supports it, fall back to legacy WHOIS only when necessary.
- Rotate query sources: spreading queries across multiple IPs or checking cadences avoids any single source tripping a registry's per-IP threshold.
- Respect per-domain check frequency: a domain that hasn't changed status doesn't need re-checking every few minutes. Widening the interval for stable domains and tightening it only for domains near a known milestone (like an expiration date) cuts total query volume dramatically without losing signal.
- Cache and diff, don't re-fetch blindly: compare the new response against the last known state before treating it as a "change" — this catches formatting noise instead of firing false alerts, and lets you skip unnecessary downstream processing.
The tradeoff to accept
You cannot check thousands of domains in real time with zero delay and zero risk of throttling — that constraint is enforced by the registries, not by your tooling. The goal isn't to eliminate rate limits, it's to check frequently enough that nothing meaningful slips through, while staying well under the threshold that gets you blocked. Domains close to a known event (expiration, pending delete) earn tighter check intervals; everything else can wait longer between passes.