Sending log files

Sending web server logs to Qrator Labs servers allows you to analyze and clean up HTTP traffic going to these web servers at the application layer (L7 under the OSI/ISO model) in situations where it is not technically possible to analyze HTTP traffic itself.

For instance, it is recommended to configure log sending if you're using reverse TCP proxy Qrator Labs. This proxy analyzes traffic at the transport layer (L4), but, with the help of log analysis, can also protect against some application layer (L7) attacks.

To transmit logs to the Qrator Labs network:

Choose the IP addresses and ports for log sending
Configure the log format.
Configure log collection and sending.

Warning

For analysis, you need the logs of the web server on your network that receives external requests first, regardless of where it routes or proxies them later. In this case, only access logs are needed. It is not necessary to send error logs.

1. Choose the IP addresses and ports for log sending

This step should be done by you and Qrator Labs technical support team. On your side, you need to set which IP address the logs will be sent from. On the Qrator Labs side, the IP address and port of the log listener will be assigned. The assigned IP address may be different from Qrator IP specified in your dashboard.

2. Configure the log format

Configure the server such that access logs are written in a certain format from which the Qrator Labs reverse TCP proxy can extract all the information needed for analysis. Log format requirements may change in the future, in which case Qrator Labs tech support specialists will inform you about the changes beforehand.

Below are the current log formats for popular web servers.

NGINX

Apache

F5 BIG-IP

HAProxy

Specify the following line in the log_format option.

'2xFFFFFF|:,_|$bytes_sent|:,_|$http_accept|:,_|$http_accept_encoding|:,_|$http_accept_language|:,_|$http_cookie|:,_|$http_host|:,_|$http_referer|:,_|$http_user_agent|:,_|$http_x_forwarded_for|:,_|$remote_addr|:,_|$request|:,_|$request_time|:,_|$status|:,_|$time_local|:,_|$upstream_response_time|:,_|$http_x_requested_with|:,_|$http_authorization|:,_|$remote_port|:,_|-|:,_|$msec|:,_|'

Specify the following line in CustomLog.

'5xFFFFFF|:,_|%O|:,_|%{Accept}i|:,_|%{Accept-Encoding}i|:,_|%{Accept-Language}i|:,_|%{Cookie}i|:,_|%{Host}i|:,_|%{Referer}i|:,_|%{User-Agent}i|:,_|%{X-Forwarded-For}i|:,_|%a|:,_|%r|:,_|%{ms}T|:,_|%s|:,_|%t|:,_|%D|:,_|%{X-Requested-With}i|:,_|%{Authorization}i|:,_|%{remote}p|:,_|%{sec}t.%{msec_frac}t|:,_|-|:,_|'

Specify the following line when configuring request logging according to the Configuring Request Logging guide.

'3xFFFFFF|:,_|$RESPONSE_SIZE|:,_|$Accept|:,_|${Accept-Encoding}|:,_|${Accept-Language}|:,_|$Cookie|:,_|$Host|:,_|$Referer|:,_|${User-Agent}|:,_|${X-Forwarded-For}|:,_|$CLIENT_IP|:,_|$CLIENT_PORT|:,_|$HTTP_REQUEST|:,_|$RESPONSE_MSECS|:,_|$HTTP_STATCODE|:,_|$DATE_YYYY/$DATE_MM/${DATE_DD}_$TIME_HH24:$TIME_MM:$TIME_SS|:,_|$HTTP_KEEPALIVE|:,_|$SERVER_IP|:,_|$SERVER_PORT|:,_|$BIGIP_CACHED|:,_|${X-Requested-With}|:,_|${Authorization}|:,_|$CLIENT_PORT|:,_|$TIME_MSECS|:,_|-|:,_|'

Insert the following snippet into configuration file HAProxy.

capture request header Accept len 4096
capture request header Accept-Encoding len 4096
capture request header Accept-Language len 4096
capture request header Cookie len 16384
capture request header Host len 4096
capture request header Referer len 4096
capture request header User-Agent len 4096
capture request header X-Forwarded-For len 4096
capture request header X-Requested-With len 4096
capture request header Authorization len 4096
log-format "4xFFFFFF|:,_|%B|:,_|%[capture.req.hdr(0)]|:,_|%[capture.req.hdr(1)]|:,_|%[capture.req.hdr(2)]|:,_|%[capture.req.hdr(3)]|:,_|%[capture.req.hdr(4)]|:,_|%[capture.req.hdr(5)]|:,_|%[capture.req.hdr(6)]|:,_|%[capture.req.hdr(7)]|:,_|%ci|:,_|%r|:,_|%Ta|:,_|%ST|:,_|%Tl|:,_|%Tr|:,_|%[capture.req.hdr(8)]|:,_|%[capture.req.hdr(9)]|:,_|%cp|:,_|%Ts.%ms|:,_|-|:,_|"

3. Configure log collection and sending

Depending on the web server you use, you may have different ways of sending logs to Qrator Labs. The easiest way is to send logs directly from the web server if it supports such. A more universal solution is to use a log management system for any web server.

Note that the log listener on the Qrator Labs side accepts logs via syslog over the TCP transport protocol. Some web servers only support syslog over UDP or over Unix sockets. In that case, it is recommended to collect them using a logging system that would redirect the logs to Qrator Labs.

sending logs

Send logs directly

If the web server handling user requests is capable of sending logs to Qrator Labs via the syslog protocol itself, then you only need to configure that web server accordingly. For example, instructions on how to send logs directly from F5 BIG-IP can be found here: Configuring BIG-IP to log to a remote syslog server.

To use the secure TLS to send logs, even if the web server itself does not support this protocol, you can use the optional Stunnel as described here: SSL Encrypting Syslog with Stunnel.

Using log management systems

A logging system is software that can act as an intermediate host between your web server and Qrator Labs log receiver. Examples of such systems for Unix-like operating systems are rsyslog and syslog-ng; for Windows – Snare. When selecting a logging system, note that Qrator Labs accepts syslog over TCP transport protocol, not UDP.

Note

Popular logging systems also support encryption of sent logs using TLS, for example:

Interaction between the web server and the logging system can be configured in a variety of ways, including reading logs from a text file, over a Unix socket, or over TCP or UDP. The method you choose depends on whether the web server and the logging system are on the same server or on different servers.

You also usually need to configure the web server itself in order to send logs to the right file, socket, or address. Read more in the documentation for the particular web server you are using, e.g:

Warning

When using logging system rsyslog, you must additionally configure the action queues and their discard rules. You can read more here: Action Queue cfg discarding local messages.

rsyslog configuration examples

Below are examples of rsyslog configuration files for HAProxy and NGINX. The required configuration file should be placed in the /etc/rsyslog.d/ directory and then rsyslog should be restarted. To use TLS, it is also required to substitute the path to Qrator Labs certificate in the configuration file.

HAProxy without TLS

HAProxy with TLS

NGINX without TLS

NGINX with TLS

$AddUnixListenSocket /var/lib/haproxy/dev/log
if ($programname startswith 'haproxy') then {
    if ($msg contains '4xFFFFFF') then {
         $template codename-fwd,"<%PRI%>1 %timegenerated:::date-rfc3339% codename %syslogtag:1:32% - - %msg:::drop-last-lf%\n"
         action(type="omfwd"
                Target="<receiver IP address>"
                Port="<receiver port>"
                protocol="tcp"
                queue.filename="codename-fwd-q1"
                queue.discardSeverity="4"
                queue.discardMark="25000"
                queue.size="30000"
                queue.highWatermark="10000"
                queue.lowWatermark="0"
                queue.timeoutEnqueue="500"
                queue.maxDiskSpace="20m"
                queue.timeoutWorkerthreadShutdown="-1"
                queue.saveOnShutdown="on"
                queue.type="LinkedList"
                Action.ResumeRetryCount="-1"
                template="codename-fwd"
          )
    }
}

$AddUnixListenSocket /var/lib/haproxy/dev/log
global(
    DefaultNetstreamDriverCAFile="<path to the receiver certificate file>"
)
if ($programname startswith 'haproxy') then {
    if ($msg contains '4xFFFFFF') then {
         $template codename-fwd,"<%PRI%>1 %timegenerated:::date-rfc3339% codename %syslogtag:1:32% - - %msg:::drop-last-lf%\n"
         action(type="omfwd"
                Target="<receiver IP address>"
                Port="<receiver port>"
                protocol="tcp"
                queue.filename="codename-fwd-q1"
                queue.discardSeverity="4"
                queue.discardMark="25000"
                queue.size="30000"
                queue.highWatermark="10000"
                queue.lowWatermark="0"
                queue.timeoutEnqueue="500"
                queue.maxDiskSpace="20m"
                queue.timeoutWorkerthreadShutdown="-1"
                queue.saveOnShutdown="on"
                queue.type="LinkedList"
                Action.ResumeRetryCount="-1"
                template="codename-fwd"
          )
    }
}

input(type="imuxsock"
      Socket="/var/log/nginx/nginx.sock")

template(name="codename-fwd" type="string"
         string="<%PRI%>1 %timegenerated:::date-rfc3339% codename %syslogtag:1:32% - - %msg:::drop-last-lf%\n"
)
if ($programname startswith 'nginx') then {
    if ($msg contains '2xFFFFFF') then {
        action(type="omfwd"
               Target="<receiver IP address>"
               Port="<receiver port>"
               protocol="tcp"
               StreamDriver="gtls"
               StreamDriverMode="1"
               StreamDriverAuthMode="anon"
               queue.filename="codename-fwd-q1"
               queue.discardSeverity="4"
               queue.discardMark="30000"
               queue.size="30000"
               queue.highWatermark="10000"
               queue.lowWatermark="0"
               queue.timeoutEnqueue="500"
               queue.maxDiskSpace="20m"
               queue.timeoutWorkerthreadShutdown="-1"
               queue.saveOnShutdown="on"
               queue.type="LinkedList"
               Action.ResumeRetryCount="-1"
               template="codename-fwd"
        )
        & stop
    }
}

input(type="imuxsock"
      Socket="/var/log/nginx/nginx.sock")

template(name="codename-fwd" type="string"
         string="<%PRI%>1 %timegenerated:::date-rfc3339% codename %syslogtag:1:32% - - %msg:::drop-last-lf%\n"
)
global(
    DefaultNetstreamDriverCAFile="<path to the receiver certificate file>"
)
if ($programname startswith 'nginx') then {
    if ($msg contains '2xFFFFFF') then {
        action(type="omfwd"
               Target="<receiver IP address>"
               Port="<receiver port>"
               protocol="tcp"
               StreamDriver="gtls"
               StreamDriverMode="1"
               StreamDriverAuthMode="anon"
               queue.filename="codename-fwd-q1"
               queue.discardSeverity="4"
               queue.discardMark="30000"
               queue.size="30000"
               queue.highWatermark="10000"
               queue.lowWatermark="0"
               queue.timeoutEnqueue="500"
               queue.maxDiskSpace="20m"
               queue.timeoutWorkerthreadShutdown="-1"
               queue.saveOnShutdown="on"
               queue.type="LinkedList"
               Action.ResumeRetryCount="-1"
               template="codename-fwd"
        )
        & stop
    }
}

Qrator Labs certificates

To send data over the TLS transport protocol, you must use a Qrator Labs log receiver certificate.

Different receivers on the Qrator Labs side use different certificates. Contact Qrator Labs tech support to find out which receiver you need to send the logs to. You can also check the authenticity of the CA (certificate authority) that signed these certificates.

Save the required certificate to a file, e.g. /etc/rsyslog.d/codename-syslog-ca.crt, and specify this in the logging system configuration path. Configuration files for rsyslog can be found at rsyslog configuration examples.

Certificate for 178.248.239.29

Certificate for 178.248.239.30

Certificate for 185.94.108.254

Certificate for 100.64.1.254

Certificate for CA