The web is complicated. Getting your data shouldn’t be.
One API finds a cost-effective route to clean, usable content.
Pay as you goNo subscriptionNo card to sign up
ScrapeGoat# Example Domain This domain is for use in illustrative examples.
Fits the stack
you already use.
Turn public pages into inputs for your application. Keep the same request shape as your data needs change.
curl "$SCRAPEGOAT_URL/v1/fetch" \
-H "Authorization: Bearer $SCRAPEGOAT_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"budget": {
"cost_ceiling_micro_usd": 50000,
"max_attempts": 3
}
}'JSON extraction, screenshots, and JavaScript rendering require a configured provider with that capability.
ScrapeGoat checks the page, controls the spend, and keeps the details. You keep building.
Set a cost ceiling before the first attempt. The router reserves cost before it calls a provider and stops within the allowed budget.
Set a request budget200presentclearmatchedReject known challenge pages and empty responses. Add checks for content type, size, and the text your application expects.
See validation optionsRequire JavaScript rendering or a supported location. Only routes that meet the request’s capabilities enter the selection.
Choose your requirementsBring current page content into your own workflows. Here are four good places to start.
Fetch documentation and reference pages as Markdown. Feed the result into your own chunking, embedding, and retrieval pipeline.
formats: ["markdown"]You provide the URLs and run the agent or retrieval workflow.
Know which route ran, what it returned, and what each attempt cost. The console connects your keys, requests, and balance in one place.
example.comKnow what you can ship today. ScrapeGoat uses its own API contract; it is not a drop-in replacement for another provider.
| Capability | ScrapeGoat API | Availability |
|---|---|---|
| Single-page scraping | POST /v1/fetchMarkdown and HTML | Available |
| JavaScript, JSON & screenshots | POST /v1/fetchRendering requirements, schema, or output format | Provider-dependent |
| Request history & cost trace | GET /v1/requests/{id}Owned requests and attempt metadata | Available |
| Crawl, map & web search | No hosted endpoint | Not available |
| Browser sessions & interaction | No hosted endpoint | Not available |
| Batch jobs, schedules & scraping webhooks | Orchestrate fetch requests in your application | No built-in workflow |
Current hosted limits: 60 admitted requests per minute and 5 pending requests per account.
Read limits & errors ↗We optimize for the lowest expected cost per valid result across eligible routes. You get the cost and the fee, in plain sight.
One transparent service fee. No monthly plan to outgrow.
Payments use Stripe Checkout when billing is enabled.
Estimate a workload using your own provider cost.
Illustrative estimate, not a quote. Uses the same per-request micro-USD rounding as billing. Include the cost of all attempts.
Lowest expected cost is an optimization target, not a market-wide price guarantee. Route availability, reliability, and latency affect selection. Failed attempts can be billable; estimated costs are labeled, and unknown costs remain pending. Charges are capped at your provider budget plus the service fee. Read the billing details.
The details that matter before your first request.
Read the developer guide ↗ScrapeGoat is a web scraping API and developer console. Send a public URL and your requirements. The router selects an eligible route, checks the content, and returns the result with a record of each attempt.
Routes are ranked by expected cost per valid result, using available price estimates and past reliability and latency for that page profile. The router can choose a simpler route when it meets your needs. A 20% service fee applies. We do not claim that every request is cheaper than every direct provider plan.
Set requirements.rendering to required for JavaScript pages. Rendering and other advanced features need a capable, configured provider. Known challenge responses can trigger fallback within your budget. No scraper can promise success on every page.
The hosted API supports single-page fetching, multiple output formats, and account-owned request history. It has its own request and response shape. Crawl, map, search, persistent browser interaction, and managed batch workflows are not available. Check the capability table before migrating.
They can. An upstream provider may charge for a failed attempt. ScrapeGoat records that cost and the service fee, up to your reserved ceiling. Unknown costs retain their reservation until they can be resolved; they are never shown as a zero-cost request.
Page bodies are returned to your client and are not stored in console history. The service retains account and request metadata, including route, timing, status, and cost. Target query values, cookies, and authorization header values are excluded from console history. Upstream providers process the target and supported request options.
Inspect the request history before sending a new request. Admitted request IDs survive restarts and cannot be replayed. Reusing an admitted or uncertain ID returns a conflict, which helps prevent a second possibly charged exchange.
Yes. Open the sample console to explore usage, activity, keys, and billing with clearly labeled sample data. Sign in to create your own API key. Trial balance and payment availability depend on the deployment.
You bring the idea. We’ll find a route to the data.