The full article was first available on The New Stack - read it here.

Audit logs are immutable records that describe a system’s changes over time. Each event and attempt that occurs should be captured in a log stating what happened, the time it occurred and the users that were involved.

Effective audit logging creates an audit trail that sequentially documents your system’s activity. This information is invaluable for compliance, debugging and investigation of security breaches.

This article will teach you what audit logs are and explain why they’re essential for production environments.

What are audit logs?

Audit logs are centrally stored records of the events that affect a system’s state or behavior. Each record is a historical footprint of a single change. It logs the event and contextual information about how it was initiated. Logs can also incorporate information about why the event occurred.

The following data fields are usually tracked as part of an audit log event:

• The type of event that occurred.
• The time it was triggered.
• The identity of the client that initiated the event, such as a user ID or API token.
• Detailed information about the client device, such as its IP address, to track from where the event originated.
• Event-specific references to relevant data in your system, such as the IDs of any entities modified by the event.

An audit log entry for a user authorization check could look similar to the following:

An administrator who’s concerned about a spike in authorization failures could consult the audit log as part of their investigation. They’d see that Demo User tried to use a feature that required the CreateNewUser role, but the system denied their access. Without the audit log, there would be no record of this potentially problematic request.

Audit logs vs. application logs

Audit logs have a different purpose and intended audience when compared to the system logs written by your application’s code. Whereas those logs are usually designed to help developers debug unexpected technical errors, audit logs are primarily a compliance control for monitoring your system’s operation. They’re helpful to regulatory teams, system administrators and security practitioners who need to check that correct processes are being followed.

Audit logs also differ in the way they’re stored and retained. They’re usually stored for much longer periods than application logs, which are unlikely to be kept after an issue is solved. Because audit logs are a historical record, they could be retained indefinitely if you have the storage available. This enables you to investigate incidents that only come to light years after the events occurred.

The information within an audit log typically falls into one of three main categories:

• Informational events and notifications – These events include logging in to the system and accessing a specific piece of data.
• State change events – Your system’s state is modified by actions such as creating a new user, assigning a permission or raising an invoice. Tracking each change with an audit event lets you understand how your system has reached its current state.
• System events – These are events that relate directly to the system’s internal operations. Events could be logged when a process completes, such as flushing the email spool or deleting expired API tokens, so you can check that these activities have completed successfully.

These examples show how audit events directly relate to your system’s business functions. Anyone involved with your project should be able to understand the data in its audit trail. This accessibility contrasts with application logs that are often only useful to developers. As they focus on errors within the code, app logs tend to be technical, verbose and low-level.

Authorization vs. authentication logs

Audit logs are a key tool for monitoring and investigating authentication and authorization patterns. Distinguishing between these terms is important so you know what each log includes and when to access them:

• Authentication – Authentication verifies that a person is who they claim to be. Users are typically authenticated when they log in by entering their username and password.
• Authorization – Authorization determines whether a known individual can perform a specific action. After logging in, the user has been authenticated, but they still need to be authorized each time they try to create a new user, assign a permission or access a sensitive resource.

The information provided by an audit log entry will vary depending on whether the record relates to authentication or authorization. If it’s an authentication attempt, the request will have occurred in the context of a public session. Authorization logs, written by AuthZ services like Cerbos, will include the identity of the logged-in user.

Assessing both types of logs will expose the widest range of attacks. Brute force credential-guessing attempts will show up as failed authentications in the audit trail. Meanwhile, a spike in denied authorizations for a user could indicate that their API token has been intercepted by an attacker.

How to implement audit logging

Audit logs should be written by your application each time a significant event occurs. At its simplest, an audit log is simply a file or database table where a new record is added as each event occurs. To generate an audit event, capture the information that’s relevant to your system, then label it with the type of event that’s occurred.

Audit logs can be stored in several different formats:

• Plain text – Plain text audit logs in a format such as CSV are the simplest way to get started, but they can be challenging to use with log-indexing tools.
• JSON – Storing audit logs in JSON format makes them easier for other tools to parse. However, JSON files lack a schema and are relatively large, which can make them cumbersome to work with.
• Protobufs – Protocol buffers are an emerging solution for serializing structured data like audit log records into a compact-typed format. They’re usually smaller than equivalent JSON while allowing you to stipulate that EventName is always a string, and EventTimestamp is a valid time.
• Inside a database table – You can store audit logs inside a relational or document-based database. This method lets you use familiar query methods to search and filter your audit data. Consider separating your audit database from your primary database, however, to prevent attackers who compromise the primary from being able to modify old audit events.

You’ve also got options for where audit events are sent to be indexed and consumed. Manually interacting with any of the methods above is clunky and error-prone. Setting up a dedicated log viewer provides convenient access to your records:

• Streaming audit logs to stdout – Most common log aggregation tools such as Fluentd and Elastic can collect logs written directly to your application’s standard output stream, if you use formats such as JSON with a consistent structure. This isn’t always possible to accommodate, however, such as if you’re already emitting application logs to stdout.
• Collecting logs with Logstash – Logstash is part of the Elastic suite. It’s a data collector that can read logs from many different sources, including local files, events in a database and requests sent to an HTTP webhook or websocket. Once the data’s been ingested, it’s forwarded to your preferred log aggregator.
• Viewing logs – Datadog and Grafana are two popular tools for aggregating, browsing and searching log data. Once you’ve ingested the audit logs from your system, you can use these applications to query for events by properties such as type and timespan.

Choose a processing pipeline that’s resilient enough to handle audit logs. Occasional drops of an error log can be acceptable, but a failed audit log write could alter the outcome of a future compliance investigation. For example, delivering logs from your application to an HTTP webhook is risky if there’s no mechanism to retry the submission. Events sent while the network’s down would be irretrievably lost.

Why audit logs are important

Audit logs enable you to maintain compliance by proving whether certain actions have been performed. Having the ability to retrieve the sequence of changes that a particular user’s initiated, or which occurred on a specific day, can be critical when you need to demonstrate you’ve met the requirements of regulatory frameworks such as ICO 27001 or SOC2.

Selecting technologies that produce audit logs will ensure you’ve always got the information you need on hand. For example, a PCI DSS audit might result in questions about who has accessed payment information and the context of their request. If you use Cerbos as your authorization platform, you could retrieve its audit logs and list out the “accept” decisions that relate to payment access.

Another side to audit logs and compliance is the ability to provide legal evidence that breaches have occurred. Without audit logs, breaches could go undetected for months or years, and any signs of their existence might be non-conclusive. A detailed audit trail informs when customers and regulators need to be notified. It can prove to insurers that a breach did occur or defend a lawsuit by demonstrating your system’s integrity.

Audit logs also allow you to reconstruct the events that led up to unexpected behavior or a security breach. After observing an issue, you can access the audit log and repeat the actions it’s recorded in a fresh development environment. This can help reproduce the bug and identify how attackers gained access. Audit logs don’t usually contain precise technical data, however, as they’re focused on answering business queries about who did what in the system. Application-level logs are still required to reveal the specific code paths that handled a user action.

Finally, a robust audit log implementation challenges you to identify the turnkey points in your system that have a bearing on your organization’s operations, both past and future. Each audit event increases accountability, provides visibility into changes and helps you recognize where risks are lurking. They demonstrate to developers, legal teams, customers and regulators that you’re proactively managing risks.

Conclusion

Audit logs create a historical record that’s maintained independently of your system’s current state. Administrators and compliance teams can use the audit logs to investigate user actions, spot suspicious activity and adhere to regulatory frameworks.

Comprehensive audit logs are especially critical for your application’s authorization requests. Cerbos’ open source authorization platform provides full visibility into incoming requests and their corresponding responses, explaining why specific allow or deny decisions were made. This equips you to sustain compliance and identify suspect behavior. Audit logs can be retrieved from the Cerbos admin API or written to a JSON file or standard output stream, ready to be forwarded to a log aggregation platform.

The full article was first available on The New Stack - read it here.