Connection: Bot Protection

This article describes the recommended order of primary setup for the bot protection service.

  1. Configure the request processing rules

  2. Configure exceptions for trusted users

  3. Test the configuration

  4. Incrementally enable bot protection for all users

Bot protection setup check list

The complete configuration should meet the following conditions:

1. Configure the request processing rules

In Bot protection β†’ Locations section set up the request processing rules according to the structure of your resource. The rules define how the system should treat each incoming user request depending on its HTTP method, host and path values. A possible variant of the ruleset is listed in the sample config.

On your first use of Bot protection β†’ Locations you also need to activate Enabled switch.

2. Configure exceptions for trusted users

In Bot protection β†’ Visitors section you can specify pre-requisites for requests from the users who should be treated as trusted and allowed to proceed without checks and limitations.

Such exceptions can be defined for the user requests:

On your first use of Bot protection β†’ Visitors you also need to activate Enabled switch.

3. Test the configuration

Note

Before testing, make sure that Bot protection β†’ Locations and Bot protection β†’ Visitors sections have Enabled switch activated, and the IP addresses used to make test requests are added to Force checks For IPs list.

You can test the correctness of your ruleset without affecting all visitors by using a limited number of test devices. Add one or several IP addresses to Force checks For IPs list and send test requests from these addresses. Make sure that your ruleset is letting through legitimate users while preventing automated and scripted requests.

We recommend using web browsers such as Google Chrome and console utilities such as curl for configuration tests.

You can find several configuration testing scenarios below.

Example: test access to web pages

This scenario is used to test web pages to which the access is allowed for browsers but not for bots.

  • Use your web browser to open the page. It should load correctly and your browser should receive a cookie named qrator_jsid.

  • Make several requests to the page using curl in your console. Each time the command should return 401 Unauthorized response. The data about all your attempts (bar the first one) should appear on the Bot protection β†’ Dashboard.

  • Make a request to the page using curl in your console and adding the header names and values listed in Exclude by headers setting. The command should receive the contents of the page from your server without errors.

If your ruleset works incorrectly:

  • make sure that in the request processing rules you have set Accept & Inject JS Challenge actions for the corresponding conditions;

  • check whether the IP address used for testing is added to Force checks For IPs list and is NOT added to Exclude IPs & Networks list;

  • if you have recently changed and saved your configuration, wait several minutes for the changes to take effect.

Example: test access to protected API endpoints

This scenario tests the API locations available only to the users who previously completed the checks on one of your web pages and received a tracking cookie.

  • Use your web browser to visit any location which has a rule for it with Accept & Inject JS Challenge action. The browser should receive a cookie named qrator_jsid. After that make a request to the API endpoint which has a rule for it with Accept With Cookies Only action. The API request should be completed successfully.

  • Delete the qrator_jsid cookie from your browser using developer tools and repeat the request to the API endpoint. You should receive a 403 Forbidden error page. The data about your attempt should appear on the Bot protection β†’ Dashboard.

  • Make a request to the API location using curl in your console. The command should return 403 Forbidden response. The data about your attempt should appear on the Bot protection β†’ Dashboard.

  • Make a request to the API location curl in your console and adding the header names and values listed in Exclude by headers setting. The command should receive the contents of the page from your server without errors.

If your ruleset works incorrectly:

  • make sure that the web page you are accessing meets the conditions for any of the traffic processing rules that have Accept & Inject JS Challenge action, and the API endpoints you are testing meet the conditions for the rules with Accept With Cookies Only action;

  • check whether the IP address used for testing is added to Force checks For IPs list and is NOT added to Exclude IPs & Networks list;

  • if you have recently changed and saved your configuration, wait several minutes for the changes to take effect.

Example: test access to unprotected API endpoints

This scenario is for API locations available to any users including all bots.

  • Make a request to the API location using curl in your console. The command should receive the response from your server without errors.

If your ruleset works incorrectly:

make sure that in the request processing rules you have set Do Nothing actions for the corresponding conditions;

  • check whether the IP address used for testing is added to Force checks For IPs list and is NOT added to Exclude IPs & Networks list;

  • if you have recently changed and saved your configuration, wait several minutes for the changes to take effect.

4. Incrementally enable bot protection for all users

It is recommended to enable bot protection for your user base within several stages, incrementally increasing the share of users whose requests will be checked. This approach helps minimize the impact of possible configuration errors on your user base during deployment.

For step-by-step deployment use A/B checks distribution setting. You can start with 5% users checked and keep increasing this value by 5% more each step after a brief observation interval. Keep in mind that applying a new value to this setting may take 5–7 minutes.

Track the changes on Traffic and Bot protection graphs in Statistics section after each increment. Usually the increasing share of checked users leads to the corresponding increase of the number of detected bots and decrease of the user traffic passed to the server. Deviations from this pattern may also happen due to large bot networks being detected as well as possible false-positive blockings of legitimate users.

If you observe drastic changes on your graphs, study the detailed data in Bot protection β†’ Dashboard section to find out possible causes. In case of false-positive blockings it is recommended to cancel the deployment by setting A/B checks distribution back to 0%, fix the errors in Bot protection β†’ Locations and Bot protection β†’ Visitors sections and start the incremental deployment again.

In case you do not observe any false-positive blockings you can keep increasing A/B checks distribution up to 100%.

expand_less