Patrick WoodheadThe Rising Tide of Pay-Per-Request Protocols
The internet is shifting. Its traditional business models are creaking in the age of AI. For decades, web content was essentially free to scrape. Anyone with a bot could harvest massive amounts of data at negligible cost. But the rise of greedy LLMs has changed the stakes. As large models vacuum up the web to train themselves, platforms and protocols are pushing back with pay-per-request economics.
Cloudflare and Pay-Per-Crawl
Cloudflare recently launched a pay-per-crawl system, giving website owners the option to charge automated crawlers for access. This represents a major shift: instead of simply blocking bots, publishers can now monetize them. For AI companies, this means the free lunch of web-scale training data is starting to come with a price tag.
Coinbase and the x402 Spec
Coinbase’s recent x402 spec introduces a standardized way to handle payments for API calls. Think of it as Stripe, but native to crypto and machine-readable. APIs, crawlers, and data providers can now require automatic, per-request payments, creating a new “metered web” economy.
Micropayments and Wallet-Enabled Agents
At the same time, Crypto wallets can now unlock micropayments for content. Imagine reading an article and paying fractions of a cent to the publisher without friction. Extend that idea further: AI agents can carry wallets themselves, paying for data, compute, or content access in real time. This transforms AI from passive consumers of free data into active participants in economic exchange.
The Pushback Against AI Scraping
Underlying all of this is a cultural and economic pushback. The open web was never designed for large-scale model training, and many creators feel that AI companies are extracting value without compensation. Pay-per-request protocols may provide a middle ground - not blocking access outright, but forcing a shift toward permissioned, monetized data flows.
Investors are also paying attention. As Liz Harkavy notes in a16z’s AI x Crypto Crossovers, the rise of AI is destabilizing the open internet’s economic model. To rebalance incentives, she highlights the need for micropayment systems and attribution protocols that allow revenues to be automatically distributed across all the contributors in an information chain. Blockchains, she argues, are particularly well-suited to this, enabling nanopayments, enforceable smart contracts, and transparent, programmable revenue splits.
Carra Wu, also writing in the same a16z essay, observes that the AI agent with the strongest product-market fit today is the webcrawler. Nearly half of internet traffic now comes from non-human sources, many of which ignore robots.txt and leave websites footing the bill for serving unwanted traffic. In her view, the current patchwork of blocking services is unsustainable. Instead, web crawlers themselves could participate in onchain negotiations, carrying crypto and paying websites directly via standards like x402. Humans, meanwhile, could still access content for free, while bots would help fund the system. This would re-establish the economic covenant between content creators and platforms in a way that feels more aligned and sustainable.
FilCDN and the Original Vision of the Web
The rise of pay-per-request isn’t just a defensive move against AI scraping - it’s the rebirth of the web’s original economic design. Hidden in the very first HTTP spec was status code 402: Payment Required, a placeholder for a machine-native payments layer that never fully materialized.
FilCDN is bringing that vision to life. By introducing a pay-per-byte model, FilCDN enables clients to pay precisely for the egress they consume. No waste, no guesswork - just direct, transparent, verifiable metered value exchange between providers and consumers.
This isn’t an incremental tweak. It’s a fundamental shift:
- For creators, every request for your content is monetizable
- For providers, every byte of egress becomes revenue.
- For clients, costs align exactly with usage.
- For the ecosystem, it creates a sustainable and scalable foundation for the data economy.
Where others are experimenting with per-request tolls, FilCDN plans to go deeper - turning the raw material of the internet, data itself, into a native, liquid, and fairly priced resource.
FilCDN isn’t just aligned with the pay-per-request movement - it’s leading it, realizing the promise of 402 and unlocking the true economic engine of the web.