Unmasking the Bot Threat: Exploring Bad Bot Analyzer Tool, Part 1

What is Bad Bot Analyzer

Radware’s Bad Bot Analyzer is an innovative tool that is available free of charge. It serves as an industry-first solution, offering organizations valuable insights into the extent of bot traffic on their website, app, and APIs. By using this tool, businesses gain crucial information to enhance the security of their digital assets and the impact on their revenue streams. Furthermore, it assesses the effectiveness of existing defenses against bots and helps to determine whether advanced bot mitigation solutions are necessary to enable or not.

What it takes to run Bad Bot Analyzer

In order to effectively analyze and identify patterns and use cases, we require customers to share access logs that contain specific parameters. These parameters include:

Time Stamp: The timestamp provides the exact date and time when a request was made, allowing for chronological analysis, and tracking of bot activities. Ex- 09/Jul/2021:00:51:48 +0600

IP Address: The IP address of the requester (Source IP/Client IP) helps identify the source of the request, enabling us to differentiate between legitimate users and potentially malicious bots. Ex: IP: 23.155.24.5

URL: The URL indicates the specific page or resource that was requested, giving insights into the behavior and intent behind each request. Ex: seller.xyz.com

Referrer: The referrer field indicates the URL of the previous webpage that referred the requester to the current page. This information helps in understanding the source of the traffic and identifying potential bots or malicious activity. Ex: https://seller.xyz.com/sell-online/pricing

User Agent: The user agent provides details about the software and device used to make the request. It includes information such as the browser, operating system, and device type. Analyzing the user agent helps in distinguishing between different types of traffic, such as bots, mobile users, or desktop users. Ex: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36

By having access to these parameters within the access logs, our Bad Bot Analyzer (BBA) engine can dive into the details of each request, effectively identify patterns, and determine the specific use cases behind the bot activity. This enables us to provide accurate analysis and insights to our customers and allows to understand why Bot Manger must be enabled.

Explore the Analyzer outcome in detail

We have conducted an analysis on a website hosted in Germany, utilizing access logs spanning a period of 7 days. The total number of records scanned for this analysis amounts to 110 million.

Analyzer Summary

The Bad Bot Analyzer Report provides insights into the overall threat posed by bots and identifies the specific use cases and pain points related to web and mobile applications.

Good Bots: Good Bots encompass both Legitimate Bots and Crawlers. They play a positive role in enhancing a website’s search engine performance by indexing its content.

• Legitimate Bots: Legitimate Bots include various types such as monitoring bots for websites, social networking site bots, backlink checking bots, partner website bots, and bots that gather information from other websites. These bots offer essential services and are considered legitimate.
• Crawlers: Web crawlers systematically browse webpages to understand the content of each page, allowing for indexing, updating, and retrieval of information when users make search queries. Common examples of web crawlers are Googlebot, Bingbot, Yahoobot, Yandexbot, and others.

Bad Bots: Bad Bots are primarily employed for malicious purposes. They are designed to engage in harmful activities like scraping website content, Account Takeover, Fake Form Submissions, and various other malicious actions.

Humans: Human Traffic represents the number of genuine users who have accessed the web or mobile application.

Analyzing Bad Bot Traffic Trend

The Bad Bot Analyzer Report includes a Bot trend graph that displays the recorded activity of Bad Bots over the specified timeframe. This graph offers a visual representation of the bot activity, allowing for a better understanding of the patterns and trends observed on a daily basis throughout the designated period.

The report lists the Top IPs (Internet Protocol addresses) that contribute to the Bad Bot attacks. By analyzing the data from the above-mentioned chart, it is evident that over the course of six days, a total of 18.8 million bad bot attempts were observed. Within this timeframe, a notable spike in bot activity occurred on a specific date, the 28th, with approximately 256,000 bad bot hits recorded. This spike indicates a significant surge in malicious bot traffic on that particular day. Furthermore, the analysis highlights the top three IPs responsible for most of the attacks. These IPs played a prominent role in generating and executing the bad bot activity observed during the timeframe.

Which Geo is impacting the most

Geo-based statistics provide valuable insights into Bad Bot attacks originating from different countries. In this case, it has been observed that a significant amount of bad traffic is coming from Germany, Austria, and Switzerland. The web application is accessible to users from these countries due to the nature of the business. Furthermore, it is possible to map the specific URLs that have been targeted by bots from each country.

Among these countries, Germany stands out as the major contributor, accounting for more than 80% of the bad traffic. This information highlights the need for focused attention on mitigating Bad Bot attacks originating from Germany, while also considering the bot activity from Austria and Switzerland.

Why a customer should opt for Bad Bot Analyzer

This tool is free and provides comprehensive visibility into bot traffic on websites, applications, and APIs, by looking at the log shared and allows organizations to gain deep insights into the extent of the bot problem and identify specific pain points. The tool’s capability to differentiate between good bots, legitimate crawlers, and human traffic provides valuable technical insights. Customers can optimize website performance based on the impact of each type of interaction, ensuring a seamless user experience for genuine visitors.

Furthermore, Bad Bot Analyzer provides industry-specific insights, catering to the unique challenges faced by different verticals.

In the upcoming section of the bad bot analyzer blog, we will delve into a comprehensive analysis of various use cases, exploring their business implications and practical applications.