Standard log format for firewall and IDS?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Dalis

New Member
Apr 20, 2021
11
0
1
Arizona, USA
Hi all – Is there a standard log format that appliances and firewall software like pfSense, Suricata, Snort, Untangle, Sophos, et al use?

In my research I'm not seeing much mention of the actual log format these systems use. Are they still emitting syslog-style logs? What about rsyslog and syslog-ng? Are those different formats or just different clients?

Thanks.
 

RTM

Well-Known Member
Jan 26, 2014
956
359
63
No... well... yes, see: Standards

There are many standards for log formats, log delivery and log collection (may not be the right term, here I am using it to cover receiving and storing it on disk).

Syslog is both a protocol to deliver logs over a network and a piece of software on a system that services can log to.
rsyslog and syslog-ng are both the latter, they allow services to log to them, and to forward logs elsewhere.

However you don't have to use Syslog to transmit logs, as a couple of examples you could use Windows Event Forwarding (on Windows obviously) or Elastic's Beat clients. From my experience SIEM vendors usually have their own clients as well, that use proprietary protocols.

While syslog is a standard, it does not prevent the log format (the actual content) from different vendors from being completely different from one another.

When dealing with logging, you should expect each to look up the documentation for the devices, it is often better to go with the recommended method.
It may also be worth keeping in mind that Suricata on pfSense, may be different to plain Suricata on some Linux box.
Many SIEM's and similar platforms, come with parsers for many of the common log formats (think windows event logs, Apache access logs and so on), so if you need to search the logs (which is part of what a SIEM allows you to do) using key/value pairs you can do so, without having to write custom parsers for each log source yourself. The caveat here is that if you customize the logs your applications generate, you may well end up having to write a custom parser anyway.
 
  • Like
Reactions: Dalis

Dalis

New Member
Apr 20, 2021
11
0
1
Arizona, USA
Thanks. I wish there was a clean binary format. These text log formats are so wasteful and bloated, they'll use a ton of disk. For example, an IPv4 address could be encoded as four bytes – four 8-bit unsigned integers – instead of potentially 15 bytes of ASCII or UTF-8 (12 bytes for the numbers and three for periods).