Bot traffic to websites has become so large that it has crowded out human visits. Visits from real people are currently only 48% of the total.

The new data about bots come from Imperva, a provider of data and application security solutions to protect business information “in the cloud and on-premises.” According to its “Bot Traffic Report 2016″:

In 2015 we documented a downward shift in bot activity on our network, resulting in a drop below the 50 percent line for the first time in years. In 2016 we witnessed a correction of that trend, with bot traffic scaling back to 51.8 percent—only slightly higher than what it was in 2012.

Some 28.9% of traffic to sites is from “bad bots.” Most of these are from “impersonators.” These are defined as follows:

Impersonators are attack bots masking themselves as legitimate visitors so as to circumvent security solutions. Clearly such bypass capabilities complement all malicious activities and that makes impersonators the ‘weapon of choice’ for the majority of automated attacks. More so, because a rudimentary level of obfuscation is relatively easy to achieve.

Thus, the more primitive impersonators are simply bots that hide behind a fake ‘user-agent’—a HTTP/S header that declares the visitors identity to the application. By modifying the content of that header these attackers proclaim themselves as either good bots or humans, hoping that this would be enough to gain access.

More advanced offenders take things a step further by thoroughly modifying their HTTP/S signatures to mimic browser-like behavior—including the ability to capture cookies and parse JavaScript. In many cases they’re doing both at the same time.

These bad bots are used by hackers and to take down sites.

Good bots, which made up 22.9% of internet activity last year, fall into several categories:

• Feed fetcher – Bots that ferry website content to mobile and web applications, which they then display to users.
• Search engine bots – Bots that collect information for search engine algorithms, which is then used to make ranking decisions.
• Commercial crawlers – Spiders used for authorized data extractions, usually on behalf of digital marketing tools.
• Monitoring bots – Bots that monitor website availability and the proper functioning of various online features.

One thing that the report makes clear, even if its authors do not say so, is that humans will never take back the web.

Methodology:

The data presented herein is based on a sample of over 16.7 billion bot and human visits collected from August 9, 2016 to November 6, 2016. The traffic data came from 100,000 randomly chosen domains on the Incapsula CDN. …

Geographically, the observed traffic includes all of the world’s 249 countries, territories, or areas of geographical interest (per codes provided by an ISO 3166-1 standard).

Get Our Free Investment Newsletter

I have read, and agree to the Terms of Use

Thank you for reading! Have some feedback for us?
Contact the 24/7 Wall St. editorial team.