Ghosts in the beacon log
Not every visit is a person. Some are crawlers, some are headless browsers, and some are scripts firing too fast to be human. Here is how this site marks those ghosts, why it keeps the rows instead of deleting them, and how the bots lab shows the count.
Halloween is the one night a year when it is fine to have a few ghosts in the house. Your analytics is different. Ghost visits show up every night: a crawler reading every page, a headless browser running a test script, a loop that fires the same event forty times a minute. None of them is a person, and all of them can land in your reports looking like one.
The bots lab shows how this site handles them. It counts the latest stored beacons, splits them into four buckets, and lists the exact rules that put a beacon in each one. This post walks through those rules, why the marked rows are kept instead of deleted, and what that means if you are building your own collection.
How many ghosts are out there
The scale is not small. The 2025 Imperva Bad Bot Report found that automated traffic made up 51% of all web traffic in 2024, the first time in a decade it passed human activity. It put malicious bots at 37% of all internet traffic, up from 32% in 2023, and good bots at 14%. That is one vendor's view of its own network, not a measurement of your site, but it is a fair warning that "a visit" and "a person" are not the same thing.
Not every bot is bad, either. Search crawlers and AI crawlers read pages so they can be found. You want them on the site. You just do not want them counted as customers.
What Google Analytics does for you, and what it hides
If you use Google Analytics, some of this is already handled. Google's help page on known bot-traffic exclusion says traffic from known bots and spiders is excluded automatically, identified with a combination of Google research and the IAB/ABC International Spiders and Bots List. The same page says you cannot turn that exclusion off and cannot see how much known bot traffic was excluded.
That is a reasonable default. It is also a black box. You get a cleaner number, but you do not get to see what was removed or why. The point of this site is to open that box, so its own first-party beacon pipeline does the filtering out in the open.
The three rules this site actually runs
Every consented beacon goes through three checks when it is stored. The bots lab lists them, and they match the code.
- Crawler. The server reads the request's User-Agent with the Next.js
userAgenthelper, which returns anisBotflag for requests from known bots. If it is true, the row is markedcrawler. That list covers names such as Googlebot and GPTBot. - Headless. If the browser string contains
HeadlessChrome,PhantomJS,Puppeteer,Playwright, orSelenium, the row is markedheadless. This is a separate check becauseHeadlessChromeis not on the Next.js crawler list. A headless browser is a real browser engine driven by a script, often a test runner or a scraper. - Rate. If the same anonymous visitor key already stored 40 beacons in the previous 60 seconds, the next row is marked
rate. A person reading pages does not fire that fast. A loop does.
Anything that matches none of these is included. Those four counters are the boxes at the top of the lab.
What the rules do not do
The lab is just as plain about the limits. This site does not read navigator.webdriver, mouse paths, or screen size to decide. It does not block the request, and it does not delete the beacon.
That list matters, because it tells you what slips through. The User-Agent is a string the client sends, and a script can send any string it likes. A headless browser that reports itself as ordinary Chrome passes rule 2. A scraper that paces itself under 40 beacons a minute passes rule 3. The W3C WebDriver spec defines navigator.webdriver as a way for a browser under automation to say so, but a script that wants to hide can avoid it too. So these rules catch the honest ghosts and the clumsy ones. They do not catch a bot that is working hard to look human, and no simple rule set does.
That is fine for a demo site, and it is a useful baseline for a small one. Commercial bot management adds behavior signals, IP reputation, and challenges on top. You can add those later. Start by marking the obvious cases so they stop skewing your numbers.
Mark, don't delete
Here is the design choice worth copying. A marked row stays in the collection log. The reports that answer "what did people do?" skip it. On this site that means attribution, funnels, cohorts, anomalies, and the other Understand counts leave out crawler, headless, and rate rows.
Why keep them at all?
- You can audit the rule. If a real visitor gets marked by mistake, the row is still there to find. Deleting it would hide the error forever.
- You can change the rule later. Tighten the rate limit or add a pattern, and you can re-run it over history instead of only fixing the future.
- The ghosts are data too. A spike in
crawlerrows after a launch tells you search and AI crawlers found the new pages. A spike inraterows can point to a broken script or someone hammering a form.
The lab is also honest about where the mark does not reach. Pathing, recommendations, and Google Analytics do not read it. Google Analytics has its own exclusion, described above. The other two are places where a determined bot could still leave footprints, and the lab says so instead of pretending the filter is universal.
Ghosts and personalization
This matters beyond reports. If you personalize from behavior, a bot can teach your model the wrong lesson. A scraper that walks every product page in a category looks like a visitor with a strong interest in that category. A test script that loads the pricing page a hundred times looks like a buyer.
The segments lab and the home page's "Why am I seeing this?" panel work from first-party signals in the browser. Marking bot rows at storage time is what keeps the aggregate side of that picture, the counts and funnels that tell you which segments matter, from being tuned by things that are not people. You do not need an expensive vendor to get this first layer right. You need a few clear rules, applied when the beacon arrives, with the evidence kept.
Try it
- Open the bots lab and look at the four counters for the latest stored beacons. A zero means the rule matched nothing in that set, not that the rule is off.
- If fewer than eight beacons are stored, the lab says the mix is not a rate yet. Click Sample data to see example counters. The sample is labeled as not coming from the database.
- Read the line under the counters. It tells you how many of these beacons are left out of the Understand reports.
- Compare that with what you know about your own analytics. If you use Google Analytics, ask yourself where you would see the bots it removed. On this site, you can see them.
The takeaway for anyone building collection: you will always have a few ghosts. Name them, mark them when they arrive, keep the evidence, and make sure your reports know the difference between a visitor and a costume with nobody inside. Happy Halloween.
Sources
Labs that show this
- Bot traffic filtering explained
Bot traffic filtering explained: crawler, headless, and high-rate beacons stay stored, and the Understand reports skip them. Free to inspect the rules.
- LinkedIn attribution, compared
Attribution models compared on one journey, including a LinkedIn-style path. Last click, first click, linear, time decay, and position, side by side.
- See live audience segments
See live audience segments built from signals already in this browser, plus a count of consented visitors. The rules are on the page. No third-party profile.
Recommended next
Why these picksReading path transitions…