How a browser tab that stays on the site took Lowe's from 2.6% to 98% delivered pages, and what it means for anyone collecting web data.
If you collect data from retail or review sites, you know the pattern. A page loads fine in your own browser. The same page, requested by a scraper, comes back empty, blocked, or as a CAPTCHA. On some sites it happens to almost every request.
For a long time we treated this as an anti-bot problem. Better fingerprints, more proxies, smarter retries. It helped, but on the hardest sites the numbers stayed ugly: Lowe's gave us 2.6% of the pages we asked for. Yelp, under 1%.
Then we looked at it from the site's side. A fresh browser that appears out of nowhere and jumps straight to a product page doesn't look like a bot. It looks like a stranger. Real visitors don't arrive that way.
What we changed
Our new live engine keeps a real browser tab open on the site. Each page a customer asks for is then served from that tab. We call them live tabs.
In two weeks we moved seven sites to live tabs. Here is what happened.
| Site | Delivered, before → after | Median time |
|---|
| Lowe's | 2.6% → 98% | 5.1 s → 1.1 s |
| Yelp | 0.7% → 90% | 9.2 s → 4.6 s |
| Target | 53% → 98% | 1.5 s → 0.7 s |
| Autoweek | 14% → 99% | 14.7 s → 1.3 s |
| Allegro | 49% → 87% | 2.8 s → 5.6 s |
| Leroy Merlin | 89% → 99.9% | 16.6 s → 2.4 s |
| Amazon.pl | 98.1% → 99.6% | 14.4 s → 1.4 s |
Before: the 7 days before each site moved. After: 7–8 November 2026.
That is about 145,000 pages a day that customers used to lose and now receive. Most of them arrive in one to two seconds. Allegro is slower because its category pages still run on Firefox tabs, which load the whole page. Before, it delivered only the pages a quick request could get.
The side effect we didn't plan for: live tabs need far fewer browsers. On these sites we went from roughly 750 browser hours a day to about 10, while traffic to Amazon.pl doubled. One tab serving thousands of pages costs a fraction of thousands of fresh browsers serving one page each. Fewer browsers per page also makes each page cheaper for us to deliver.
At scale
In the 7 days to 8 November, 2.2 million pages came through live tabs, 10% of everything the Scraper API delivered. 95% of live requests succeeded, at about two seconds each. In the last 24 hours it was 98%, about 360,000 pages.
How it compares
We took real customer URLs from these sites and sent the same URLs to six other scraping APIs, each in its strongest anti-bot mode, from the same country. Only real pages with the data counted. Test run: 6–8 November 2026. Each service got the same URLs as us, at the same time. Time is the median on pages both delivered.
| Service | URLs | Delivered, them / us | Time, them / us | Price per 1k pages |
|---|
| Oxylabs | 160 | 77% / 90% | 12.6 s / 5.2 s | ~$1.50 |
| Zyte | 80 | 58% / 85% | 8.3 s / 6.3 s | ~$3 |
| Scrapfly | 24 | 63% / 92% | 31.9 s / 7.8 s | $2.70–4.50 |
| ScrapingBee | 8 | 75% / 100% | 19.6 s / 7.1 s | $5.60–19 |
| Firecrawl | 160 | 24% / 82% | 7.9 s / 5.1 s | $0.60–3.80 |
| ScraperAPI* | 32 | 28% / 91% | 3.3 s / 5.7 s | $0.90–4.90 |
| Geonode | | | | $0.20–0.40 |
*ScraperAPI ran on its free plan, which has no premium proxies.
We delivered more pages than all six. We were faster than five of them. The sixth, ScraperAPI, was quicker on the pages it did return, but it delivered only 28% of them – the easiest ones. On Yelp, only we and one other service got any pages at all. On Lowe's list pages we got 20 of 20, where one competitor returned "Website Ban" and another nothing. And the cheapest service in the test costs more per page than our most expensive plan.
What this means if you buy web data
Success rate is not a vanity metric. Every page you don't get is a product missing from a price comparison, a review that never reaches your model, a listing your client asked for. At 2.6% you don't have a dataset, you have a sample. At 98% you have the site.
Speed matters for the same reason. Pages that arrive in a second can be refreshed every hour. Pages that take 15 seconds, or never come, cannot.
What's next
The system now looks for the next sites on its own. It tries live tabs on page types that keep failing, which is how it found Yelp. A dozen large sites still run on heavy browsers, and we are moving them one by one.
If there is a site that keeps blocking you, send us the URLs. We'll run them and show you the numbers.
