Environment variables
Set in your Laravel .env. These configure the default transport, the command prefix, and the addresses of the optional anti-bot services.
| Variable | Default | Description |
|---|---|---|
CRAWLER_TRANSPORT | guzzle | Global default transport: guzzle, browser, flaresolverr, scraping_api, auto. A robot's own --transport overrides this |
CRAWLER_COMMAND_PREFIX | datahelm | Artisan command prefix (datahelm:scrap:generate, …) |
BROWSERLESS_URL | http://browserless:3000 | Address of the local browserless (headless Chrome) container — used by the browser transport |
BROWSERLESS_TOKEN | (empty) | Auth token, only if you password-protect your own browserless instance |
FLARESOLVERR_URL | http://flaresolverr:8191 | Address of the local FlareSolverr container — used by the flaresolverr transport |
FLARESOLVERR_MAX_TIMEOUT | 60000 | Max milliseconds FlareSolverr may spend solving one Cloudflare challenge before giving up |
CRAWLER_PROXY_URL | (empty) | Upstream proxy for the browser / flaresolverr transports |
SCRAPING_API_URL | (empty) | Managed scraping API base URL |
SCRAPING_API_KEY | (empty) | API key for the scraping_api transport |
SCRAPING_API_KEY_PARAM | — | Query-param name the provider expects for the key (e.g. apikey) |
SCRAPING_API_PARAMS | — | Extra provider flags, e.g. js_render=true&antibot=true&premium_proxy=true |
Local vs. external services
browserless and flaresolverr are local Docker containers, not external paid services — everything runs on your own machine and nothing is billed. The only paid option is scraping_api, which stays off until you add a key.
Docker vs. host hostnames
When Laravel runs inside Docker on the same network as the services, use the service hostnames:
ini
BROWSERLESS_URL=http://browserless:3000
FLARESOLVERR_URL=http://flaresolverr:8191
When Laravel runs on the host machine, point at the published host ports instead:
ini
BROWSERLESS_URL=http://localhost:3010
FLARESOLVERR_URL=http://localhost:8191
Managed scraping API example
ini
CRAWLER_TRANSPORT=scraping_api
SCRAPING_API_URL=https://api.zenrows.com/v1/
SCRAPING_API_KEY=your_key
SCRAPING_API_KEY_PARAM=apikey
SCRAPING_API_PARAMS=js_render=true&antibot=true&premium_proxy=true&proxy_country=br
See config/crawler.php for ready-made examples (ZenRows, ScraperAPI, ScrapingBee) and the Transports guide for when each transport is needed.