Best Web Scraping Tools for Python 11 Stacks vs. 6 Live Bot Checks

In July 2026 Michael Mintz, the author of SeleniumBase, published a run putting ten Python browser-automation stacks through the same six live checkpoints: two bot-detection audits, a Cloudflare interstitial, a Turnstile widget, a fingerprint-coherence check that cross-examines the browser against itself, and one real grocery storefront that verifies the device before painting a product tile.

Every pass and fail in rows 1–10 below is his observation, not ours. We regrouped his checkpoints into tiers, applied our own scoring vocabulary and drew our own operational conclusions, but we did not re-run his test, and we flag that wherever it matters. The one row we did measure ourselves is the last one. Gologin was never in his bracket, so on 24 August 2026 we ran the same six checkpoints against one profile ourselves, and we publish that run with its confounder declared. What we add beyond it is an analysis of why the failures cluster where they do, and an honest account of what six checkpoints can and cannot tell you.

Four of the ten stacks broke. Only one of those four was stopped by a static audit page, and it failed for a reason worth the whole article. If you came here to pick the best web scraping tools for Python, the results table below is the short answer; the findings after it explain why that answer decays every four weeks.

The crawl that worked on Tuesday and 403s on Thursday

Nothing in your repo changed. The crawler still exits 0, the scheduler still reports green, and extraction has drifted from "fine" to "most of these rows are empty." Price feeds go stale, and challenge HTML gets parsed as a valid document with zero fields.

The question is not which library is fastest. It is which stack still returns data on day 40 of a scheduled crawl.

Two philosophies: patch the browser, or be a browser profile

Patch the browser. Launch generic Chromium, then overwrite what gives automation away at runtime: navigator.webdriver, plugin arrays, canvas noise, CDP artifacts. Every open-source stack below does this.

Be a browser profile. Ship a browser whose identity was coherent before any script ran: a fingerprint sampled from a real device, bound to its own proxy, persisted between sessions. That is the cloud browser category: a different bet, absent from the original ten, and the one we put through the same six checkpoints ourselves.

The four layers a real site scores

1. JS-visible fingerprint. Canvas, WebGL, fonts, screen metrics, timezone, and whether those values are mutually consistent.
2. TLS and HTTP/2 handshake. JA3/JA4 signatures and header ordering, judged before a line of JavaScript runs.
3. IP reputation and ASN. Datacenter vs. residential, request density, and whether the IP's geolocation matches the declared timezone.
4. Session continuity. Cookies, storage, and whether this identity has history with the site or arrives a stranger every run.

All six checkpoints score layer 1, and it is the only layer this test can isolate. The Cloudflare and DataDome checkpoints certainly weigh layers 2 and 3 internally, but each returns a single verdict, so no pass or fail can be attributed to one layer. Layer 4 is untouched, because every run was a cold start.

Methodology: six checkpoints, three tiers

Scoring states. Pass: cleared unattended. Fail: challenged with no path through. Not shown: the run was never displayed, so we record a gap rather than infer a result. Fault: ran, but needed a human for reasons unrelated to detection.

What this test does not measure

1. One cold start per stack. Nothing returned twice as the same identity; layer 4 is untested.
2. One IP, held constant. Network reputation was controlled, so the table says nothing about layer 3, the layer that blocks most operations first.
3. No layer isolation on the gates. Cloudflare and DataDome return one verdict over several layers at once.
4. No concurrency. Success rate falling as concurrency rises, rather than as volume rises, is invisible here.
5. Results decay. Anti-bot vendors ship constantly. Any row is a timestamp, not a property of the software.

Results

Rows 1–10 are observations from Mintz's run. Row 11 is Gologin, which was not in that test. Those six cells are our own measurement, dated 24 August 2026, and footnote ⁴ carries the method and the one confounder.

Two stacks, Patchright and SeleniumBase, were observed clean across all six checkpoints. Four broke. Three are unresolved, because cells were never displayed and we are not filling them by inference. Gologin's row is ours, not Mintz's: six for six on 24 August 2026, measured behind the residential proxy its profile ships with. That last clause is both the result and the caveat (⁴).

Footnotes. 

• ¹ undetected-chromedriver's tier-3 run was not shown; Mintz's closing summary lists it among the stacks still running clean.
• ² NoDriver needed a hand-added encoding declaration to start under Python 3.14. That is a maintenance cost, not a detection failure, so it is recorded separately. Scrapling also produced unrequested console output, which Mintz explicitly declined to penalize; we follow him.
• ³ The source contradicts itself here: the spoken clean-finisher list names PyAutoGUI, while the closing recap names Pydoll and omits PyAutoGUI. We record both as unresolved rather than pick a side.
• ⁴ Gologin was not in Mintz's test; its row is our own run. Method: one profile (Chrome 149, macOS fingerprint), one pass per checkpoint on 24 August 2026, driven headful over CDP, every verdict read from a full-page screenshot, nothing auto-scored. What the screenshots show: BrowserScan's headline verdict "Normal" with even its CDP-detection panel green; every sannysoft row passed; GitLab rendered its sign-in form with no interstitial; the Turnstile widget resolved to "Success!" unaided; pixelscan reported "no masking detected" and "no automated behavior detected," with the WebRTC address matching the exit IP and the timezone matching both; Stop & Shop served its full homepage without offering the slider. The confounder: the profile ran behind its bundled US residential exit while Mintz's stacks ran bare, so read the row as the product as shipped, not a same-conditions comparison. One run on one date is an anecdote, not a rate. The harness, raw results.json and all six screenshots accompany this post.

The run, in screenshots

Notable absentee: Camoufox, a Firefox build injecting fingerprints below the JS layer. Missing from the source run.

Finding 1: passing the audit pages predicts almost nothing

Eight stacks cleared BrowserScan and bot.sannysoft.com. Those pages test for known automation artifacts; a live storefront tests whether the whole device story hangs together. A benchmark nearly everyone passes carries no selection signal. Your smoke test going green is not evidence your crawler works.

Finding 2: shared lineage means shared failure

ZenDriver is a fork of NoDriver, and Mintz called this one before running it: same lineage, same expected outcome. It landed that way: both cleared all five earlier checkpoints, and both failed the same device check.

The operational read is ours. A fork of your current tool is not a fallback, it is the same bet with a different package name. Pick a fallback from a different family, so a CDP-native primary keeps a WebDriver-patch option warm.

Finding 3: coherence beats concealment

Cloak Browser was the only stack stopped at tier 1, and it failed at pixelscan for inconsistent masking. That check does not look for hiding. It cross-examines declared values against each other:

• The user agent claims macOS while the font list is Windows-typical.
• The WebGL renderer reports an Apple GPU while other signals say Linux.
• The browser's timezone says Berlin while the exit IP geolocates to Virginia.

Overriding properties one at a time produces combinations no shipped device ever had. The verdict is not "we caught you concealing something," it is "this device does not exist." A fingerprint is safe when it was coherent to begin with, not when it was made coherent by patch. In your logs this shows up as failures correlating with region or proxy pool rather than with volume.

Finding 4: detection and CAPTCHAs are separate problems

Scrapling passed all five earlier checkpoints and was stopped by a DataDome slider. Not being flagged and resolving an active challenge are different capabilities, and most tools have only the first. A CAPTCHA is also not a retryable error: blind retries burn IP reputation and turn a soft challenge into a hard block.

Web crawling vs. web scraping, and why it changes tooling

Crawling is breadth and revisit cadence: discovering URLs, following links, returning on a schedule. It breaks on rate limits and IP reputation. Scraping is depth: rendering one page and extracting fields. It breaks on rendering and identity.

A crawler hammering 50,000 URLs from one datacenter subnet gets blocked however good its fingerprint is. A scraper hitting 200 product pages gets blocked despite clean IPs if the browser identity is incoherent. Different failure, different fix. And most pipelines do both.

The checkpoint nobody ran: session two

Every result above is a cold start. A 200-page extraction is a scripting problem; a continuous pipeline is a distributed-identity problem: the same identity returning hourly for months, often logged in, scored on continuity rather than on any single page load.

A freshly patched Chromium per run arrives a brand-new stranger every time, which is anomalous anywhere that keeps history. Persistent browser profiles stop being a convenience there and become the reliability mechanism.

Concurrency is the other untested axis. How much a stack varies its fingerprint across simultaneous workers differs by library and is worth checking before you scale out. A fleet presenting one identity across 200 workers is a cohort, not 200 visitors. Nothing in this run measured it.

The layer none of these tools touch: your IP

All ten stacks ran from one address, so network reputation was held constant. Real operations rarely get that. ASN classification is the first cut, since datacenter ranges score badly before anything renders. Then comes request density, then the check that ties back to Finding 3: an exit IP in Virginia under a browser declaring Europe/Berlin fails coherence whatever the fingerprint quality.

Rotate per request for broad crawls, where no session needs to survive. Bind one durable IP to one profile for anything logged in, where rotating mid-session is a login-from-three-countries event. Proxy rotation is a crawling tactic, not a universal setting.

Match the stack to the job

One rule: choose by how long a single identity has to survive.

Honest carve-out: for low-volume collection of public pages, a maintained patch-philosophy library is the right call. Two were observed clean here and they cost nothing. Paying for infrastructure you do not need is its own failure mode.

Where a cloud browser fits: architecture, not a score

Gologin was not in Mintz's test, so we ran the six checkpoints ourselves: six passes, with method and caveat in footnote ⁴. One run, though, is thin evidence, which is why the case here rests on design. These are properties you can verify, not a score to take on faith.

A cloud browser is a hosted browser whose identity (fingerprint, proxy, cookies, storage) lives in a server-side profile rather than in flags on your machine. Three properties map onto findings above:

1. Fingerprints are sampled from real devices, per profile, not overridden at runtime on a generic build. Runtime overriding is the failure class that stopped Cloak Browser at pixelscan (Finding 3).
2. Residential and mobile proxies bind per profile, so the IP and declared timezone move together. That handles the layer-3 coherence problem in config rather than in glue code.
3. Profiles persist across runs with cookies and storage intact, which is the session-two problem this test could not score. For multi-account work, isolation is the point: one flagged account should not teach a platform what the others look like.

One thing not to oversell: every stack in the table drives a real Chromium binary, so the TLS handshake is Chrome's in all of them. Layer 2 is a floor here, not a differentiator.

Trade-offs, plainly: it costs money, it is not open source, and you trust someone else's update cadence.

Switching cost depends on your current API. The control plane is CDP, so if you are already CDP-native (NoDriver, ZenDriver, Pydoll), migration is mostly a change of launch step. If you are on a WebDriver or Playwright API (undetected-chromedriver, SeleniumBase, Patchright), your locator and extraction code needs porting too.

Running the six checkpoints in Python

import base64, json, os, time
import requests, websocket
from gologin import GoLogin

PORT = 9222
CHECKPOINTS = [
    "https://www.browserscan.net/bot-detection",
    "https://bot.sannysoft.com/",
    "https://pixelscan.net/fingerprint-check",
    "https://gitlab.com/users/sign_in",
    "https://www.planetminecraft.com/account/sign_in/",
    "https://stopandshop.com/",
]

gl = GoLogin({
    "token": os.environ["GOLOGIN_TOKEN"],
    "profile_id": os.environ["GOLOGIN_PROFILE_ID"],  # proxy attached to profile
    "port": PORT,
    # No browser path needed: the SDK fetches the Orbita build matching
    # the profile's Chrome version into ~/.gologin/browser.
})
gl.start()

try:
    tabs = requests.get(f"http://127.0.0.1:{PORT}/json/list", timeout=10).json()
    # The proxy helper extension shows up here as a service_worker target,
    # and only a page target accepts Page.* commands.
    page = next(t for t in tabs if t["type"] == "page")
    ws = websocket.create_connection(page["webSocketDebuggerUrl"], timeout=30)

    # --window-size is unreliable headless; set page metrics explicitly.
    ws.send(json.dumps({"id": 1, "method": "Emulation.setDeviceMetricsOverride",
                        "params": {"width": 1920, "height": 1080,
                                   "deviceScaleFactor": 1, "mobile": False}}))
    ws.recv()

    for i, url in enumerate(CHECKPOINTS, start=2):
        ws.send(json.dumps({"id": i, "method": "Page.navigate",
                            "params": {"url": url}}))
        ws.recv()
        time.sleep(10)  # let challenges resolve before capture
        ws.send(json.dumps({"id": 900 + i, "method": "Page.captureScreenshot",
                            "params": {"format": "png"}}))
        shot = json.loads(ws.recv())["result"]["data"]
        with open(f"checkpoint_{i}.png", "wb") as fh:
            fh.write(base64.b64decode(shot))
    ws.close()
finally:
    gl.stop()

Three notes learned the hard way. --window-size is unreliable headless, so set Emulation.setDeviceMetricsOverride explicitly. The profile's proxy comes up a beat after the browser does, applied by a helper extension; until the tunnel is live every navigation stalls, so open a cheap page first and confirm it loads before spending checkpoint time. And do not layer spoofing on top of a profile: it already carries a fingerprint, and a second override is how you manufacture the incoherence in Finding 3.

Reproduce it, and keep it honest

Score every screenshot by hand: a Turnstile widget can render and never resolve, and pixelscan's verdict is a banner that is easy to misparse. Record the environment (fresh profile, one IP held constant, headed or headless) and stamp the run date; ours is stamped 24 August 2026 for exactly this reason. A single cold start is an anecdote, ours included, so the row ships with a spec for a better test than the one being reported:

• Success rate at 1, 50 and 500 concurrent profiles.
• Challenge rate on run 1 vs. run 30 from the same profile.
• Profile survival across a week of hourly revisits.
• Cost per successfully extracted page, using the formula above.
• Every stack behind the same proxy exit, so egress is equalized.

Corrections welcome. Source for the open-source results: Michael Mintz's run, 10 July 2026.

Questions this keeps getting

What is a cloud browser, and how does it differ from a headless browser on a VM? A cloud browser is a hosted browser whose identity lives in a server-side profile: a device-sampled fingerprint, a bound proxy, persistent cookies and storage. A headless browser on a VM is a generic build on a datacenter IP with no persistent identity, so you supply all three yourself.

Will a residential proxy alone fix my blocks? No. A clean residential IP paired with an incoherent fingerprint still fails a consistency check, and a mismatch between IP geolocation and declared timezone is itself a signal. IP and browser identity have to agree.

Do browser profiles mean I can stop rotating IPs? No, and the inverse is costlier. Rotate per request for broad crawling; bind one durable IP per profile for logged-in sessions, where rotating mid-session resembles account takeover.

Is a headless browser automatically a giveaway? Not by itself. Modern headless modes are close to headed, and several stacks here cleared audits headless. Inconsistency is what gives you away: headless-typical metrics contradicting the rest of the profile.

Can I just use the stacks observed clean and skip the rest? Yes, until the next Chromium release. That is Finding 5. Two stacks finishing clean on one date is a snapshot, and the maintenance clock resets roughly every four weeks.

Does this mean Playwright is bad? No. Playwright was the control. It failed everything because it was supposed to. An unpatched baseline is how you prove the checkpoints work. As a driver it is excellent.

What about robots.txt and terms of service? Respect them and rate-limit yourself. Collect public data rather than anything behind an account you are not entitled to use. Technical capability and legal permission are separate questions, and the second is what ends projects.