Send Squid Logs to Graylog

13 thoughts on “Send Squid Logs to Graylog”

1. Hello,
   How I get the size of download from user? How I get the total size of traffic in session by a user? The objective is build a report (in graylog) that shows the users that consumes more band.

   Reply

   • Hi, I think that you’re going to need to check Squid’s size counters for this. There is a weblink provided (check the blog post) on where to find various format arguments for logformat.

2. Anyone know of a tutorial that teaches how to configure squid with graylog?

   Reply

   • This very blog post shows you a way to configure Squid log shipping to Graylog. Check logformat for more arguments if you need to, and add them to your JSON message. I don’t think there is anything else left to configure.

3. Does anyone know how to generate reports with Graylog2? Squid access reports on screen and exporting to pdf?

   Reply

   • The way I’ve got Squid (and other services) reports configured was to create an empty dashboard and add widgets. I’m sure there are dozens of different ways to achieve this, Graylog documentation is the best place to start. Not sure about exporting to PDF though.

4. Hi,
   where do you save graylog_vhost format under graylog? It appears that we can use the same method to define new log format and send to graylog?

   Reply

   • It’s not under Graylog, but it goes into Squid’s configuration file.

   • I need some directions in doing the following. I have jira server access_date.log (few hundred log files) copied on a local drive.
     I have create nxlog config following https://stackoverflow.com/questions/32410888/using-nxlog-with-iis-and-have-json-output (replaced the fields with access log fields). The nxlog starts fine with no error but there is no data in the graylog.

     the graylog has few others inputs which are working fine and receiving the data.
     ****************************************************************************************************
     ## This is a sample configuration file. See the nxlog reference manual about the
     ## configuration options. It should be installed locally and is also available
     ## online at http://nxlog.org/docs/

     ## Please set the ROOT to the folder your nxlog was installed into,
     ## otherwise it will not start.

     #define ROOT C:\Program Files\nxlog
     define ROOT C:\Program Files (x86)\nxlog

     Moduledir %ROOT%\modules
     CacheDir %ROOT%\data
     Pidfile %ROOT%\data\nxlog.pid
     SpoolDir %ROOT%\data
     LogFile %ROOT%\data\nxlog.log

     Module xm_json

     Module xm_fileop

     Module xm_syslog

     Module xm_gelf

     Module xm_csv
     Fields $IPAddress,$UserName,$DateTime,$HTTPAction,$ResponseCode,$Column10,$Column11
     FieldTypes string,string,string,string,string,string,string
     Delimiter “,”

     Module im_file
     File “C:\\Users\\jira\\Documents\\TempOut\\JiraAccessLogs\\accessLog.csv”

     if $raw_event =~ /^#/ drop(); \
     else
     {
     jira->parse_csv(); \
     to_json(); \
     }

     Module om_udp
     Host 172.17.1.87
     Port 5046
     OutputType GELF
     Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
     Exec $collector_node_id = ‘JiraClient’;
     Exec $Hostname = hostname_fqdn();
     #Use the following line for debugging (uncomment the fileop extension above as well)
     Exec file_write(“C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log”, $raw_event);

     Path in => out

   • I’m sorry, I cannot help you with your work assignment.

5. hello
   I cannot make access_log udp://graylog.example.com:12201 graylog_vhost work
   Graylog never receiver the message, i tried in tcp as well…
   i succeed to send the message if i pass throught filebeat but the field message isnt parsed properly
   Any help with that ? thanks in advance

   Reply

   • Try generating some traffic to see if it’s a network issue. You can use Netcat for both UDP and TCP. If you can connect, then you will see a hit on iptables (under pkts).

     If it’s not a network issue, then use Netcat to send a test packet to see if Graylog receives it. If it does, then check message formatting and make sure it matches Graylog’s parsing rules.

6. I don’t get this working: The GELF messages arrive at the target host (checked via tcpdump), the format looks good (checked by writing the same format to a file on the squid host):

   {
   “version” : “1.1”,
   “_user_agent” : “curl/7.80.0”,
   “short_message” : “CONNECT http://www.cnn.com:443 HTTP/1.1″,
   “_squid_hierarchy_status” : “HIER_DIRECT”,
   “_request_size” : “1011”,
   “_http_status” : “200”,
   “_server_ip” : “151.101.113.67”,
   “_squid_request_status” : “TCP_TUNNEL”,
   “_http_method” : “CONNECT”,
   “_response_time” : “51”,
   “_client_ip” : “129.70.xx.yy”,
   “_squid_ip” : “129.70.xx.zz”,
   “level” : 6,
   “_http_referer” : “-“,
   “_http_url” : “www.cnn.com:443”,
   “timestamp” : 1641479750,
   “host” : “www.cnn.com:443”,
   “_reply_size” : “5572”,
   “_from_squid” : “true”
   }

   I did change

   “timestamp”:”%tl”

   to

   “timestamp”:%ts

   to comply with the specification (this should be seconds since epoch). But the importer shows no received messages. Any ideas? I am using Graylog 4.2.5.

   Reply