Audit logs security: cryptographically signed tamper-proof logs

Logs, audit logs, and security events are must-have components of a secure system, which help to monitor ongoing behaviour and provide forensic evidence in case of an incident. Let’s cut through complexity.

In this article, we cover cryptographically signed audit logging, aka “secure logging”, when logs are generated in a certain way which prevents tampering messages, removing, adding or changing the order of log entries.

We explain why signed logs are essential for security software, how we’ve built-in secure audit logging in Acra, and how to use it together with other defense in-depth layers in your systems.

This article is a part of a series about defense-in-depth security features that Acra provides. Previously we’ve described how searching in encrypted data works and how using distributed tracing helps our customers to monitor the whole secure data flow within their systems.

1. What is cryptographically signed audit logging
   • When and why you might need crypto signed audit logs?
   • Audit logs security in Acra
   • Security requirements towards crypto signed audit logs
2. Cryptographic scheme for secure audit logging
3. Implementing secure audit logs
   • Producing secure logs
   • Verifying secure logs
4. Audit logs meet real world
5. Open sourcing audit logging module
6. Closing thoughts

What is cryptographically signed audit logging #

Audit log, aka “audit trail”, is a set of security-relevant log records in chronological order, which identifies a source and a reason of a log and provides relevant context for further investigation. 

Audit logs contain information about access management events (like getting or revoking access, access errors), sensitive actions made by users (deleting data), events from services, and critical errors. 

The purpose of the audit log is to provide documentary evidence of the sequence of activities and help to reconstruct what happened (as a response to an incident or regulator inquiry).

As audit logs become the security source of truth, they require protection. In many cases, audit logs are available to superusers only, stored in a special storage and backed up.

However, as logs are just text messages, they require a set of security procedures (see NIST SP 800-92 ) and often become an easy target to manipulate and tamper.

When and why you might need cryptographically signed audit logs? #

The ability to generate logs in a tamper-free way and then efficiently verify all (or some) log entries are vital to any application employing secure logging techniques. Cryptographically signed audit log is a set of log entries, protected from modification with a help of cryptographic binding. Logging process and log entries structure are designed to mitigate the possibility of making any unnoticed adversarial changes in the logs.

Having a cryptographic validation of each log entry makes it possible to verify the whole log chain and find the first log message that was tampered.

Precise identification of tampered log entries helps to understand when a system was compromised, define trust scope, and reconstruct the attack.

Audit logs security in Acra #

Acra is a database protection suite that provides tools for protecting data stored in databases, like “on the fly” encryption and decryption, data pseudonymisation and masking, search through encrypted data, etc.

Acra logs contain information about application and database connection attempts, API usage, key management actions, accepting / denying incoming queries (for SQL firewall), data processing events (encryption, decryption, pseudonymisation, etc.).

Since Acra suite is intended to provide data security across different parts of infrastructure, we use a complex secure approach and introduce cryptographically signed audit logging for each Acra tool.

All Acra tools that have access to users’ data generate crypto signed logs. Acra calculates integrity checks for each log entry. Each check depends on the current and previous log entry, thus building a log chain (think about blockchain). Acra provides a separate utility, AL Verifier, that takes a crypto signed log and verifies the authenticity of each log entry moving along the chain.

Security requirements towards cryptographically signed audit logs #

While studying various secure logging systems ( “A New Approach to Secure Logging” paper had a significant influence on us ), we gathered a set of security requirements for crypto signed audit logging:

1. Log messages in each log entry should be protected against tampering (symbols editing).
2. The structure of the log chain should be protected against modification: adding or removing separate log entries.
3. Log chains should be protected against truncation attacks when logs are truncated from the end of the chain.
4. Crypto signed audit logs should be initialised as early as possible at startup of logging service—to reduce the number of unprotected log entries in the log chain.
5. Service restart/shutdown should not lead to inconsistent audit logs. If a service was shut down due to an emergency, audit logs should be verifiable.
6. Key security: a cryptographic key (used for computing integrity checks) should be stored in a dedicated key storage and reside minimum time in service memory (take a look at aes-finder utility that scans memory for AES keys).
7. Acceptable performance: the ability to verify production volumes of protected logs in seconds. Performance directly affects incident detection and reaction time (MTTD, MTTR), if verification of audit logs is an automated monitored procedure.
8. Log rotation friendliness: audit logs should be compatible with log rotation policies typical for distributed systems.

   ---

Cryptographic scheme for secure audit logging #

After carefully analysing two cryptographic schemes described in the above mentioned paper, gathering security and usability requirements, we have chosen a scheme based on symmetric keys (“private verifiable” scheme).

This scheme has a simple and boring cryptographic design and higher performance than the second scheme (“public verifiable”). It has its own limitations: using the same key for log generation and verification leads to strict security requirements towards a verification service.

The private verifiable scheme is built using two cryptographic primitives. 

The first one is a keyed hash function (HMAC), and the second is a usual hash function (hash). HMAC is configured by a symmetric cryptographic key (k) which is the one and only security secret ( one key to rule them all! ).

Each audit log entry consists of a message, date, action, error code, etc. For each log entry we add integrity check computed iteratively by the following formal principle:

As a result, the audit log chain will consist of:

• LE[1] ║ IC[1]
• LE[2] ║ IC[2]
• …
• LE[n] ║ IC[n]

where LE[n] is the body of the n-th entry in the log stream, IC[n] is integrity check of the n-th log entry, and ║ means concatenation.

The value of IC[n] depends on LE[n] and internal state[n]. Internal state is not presented in a log chain but is kept in memory during log generation/verification. IC[n] value is computed only if the value of k and previous state state[n-1] are known, while IC[n] itself does not reveal information about previous log entries, internal state, etc.

Implementing secure audit log #

Audit logging usage consists of two stages: producing crypto signed logs and, later, verifying their integrity.

Producing secure logs #

AL Logger is an internal component of Acra tools. That’s the last step of processing logs before output. AL Logger takes each log entry, computes its integrity check, and includes it into the log entry structure (producing LE[n] ║ IC[n]).

AL Logger uses cryptographic key k to calculate integrity checks. This key is generated by AcraKeymaker during Acra setup and configuration. In the case of multi-instance installation, each Acra tool can use their own log key.

As the key k is required only to generate the first integrity check IC[1] (then the key is re-hashed for each next entry), it stays in process memory for a minimal time.

func (f *LogEntryIntegrityCalculator) calculateHmac(input []byte) []byte {
	h := hmac.New(sha256.New, f.cryptoKey)
	h.Write(input)
	h.Write(f.previousLogEntryIntegrityCheck)
	return h.Sum(nil)
}

Calculating internal state[i] as HMAC of k[i], LE[i] and state[i-1]. #

Acra supports various log rotation policies, so it can be configured to produce smaller log files, resetting the log chain upon some event (by signal or after N log entries). In this case, AL Logger finalizes the current log chain and starts a new one.

This approach makes Acra easier to use in high load distributed systems: verifying smaller chains takes less time and is more convenient for log rotation.

Have a look at an example of an audit log stream from one of the Acra tools:

Audit log’s integrity checks are underlined with the green lines. Note that the first five entries of the log stream do not have integrity checks, while further entries do have them (the 5th log entry starts a new log chain, LE[1] ║ IC[1]).

This limitation comes from real-world engineering: once started, an Acra tool performs initialization and self-checks procedures first, then accesses the key storage, and only then it’s ready to produce cryptographically signed logs.

Verifying secure logs #

AL Verifier is a separate tool that runs independently of other Acra services.

AL Verifier takes audit log file(s), processes each log entry, calculates expected IC[n], and compares it with the actual IC[n] of the log entry. Processing log entries one by one, AL Verifier validates the authenticity and integrity of each log entry, log chain, or the entire log stream (a chain of chains).

AL Verifier can indicate only the line of the first invalid log entry because each integrity check depends on the previous log entry.

AL Verifier requires the same cryptographic key k to validate log entries. As we mentioned before, using a symmetric scheme requires AL Verifier to run in a trusted environment. Crypto key k stays in memory for a short time as it’s necessary only to verify a first log entry.

If AL Verifier finds log entries within the log chain that don’t have integrity checks, it displays warnings or errors on such log entries. It’s possible to configure “treat all warnings as errors” for systems with strict security requirements.

When verifying a list of files, AL Verifier works similarly to a popular logrotate tool, making the adoption of continuous audit log verification as easy as log rotation.

Check out the example of successful verification of an audit log stream:

As we mentioned before, several log entries at the beginning don’t have integrity checks because they are a part of the initialization procedure. As this is expected behaviour and these log entries happened before the start of a new crypto signed log chain, AL Verifier reports them as warnings. All following log entries have valid integrity checks, and verification is successful.

Now, let’s take this audit log, modify a log message on the 9th line (we deliberately removed one symbol from the integrity string), and verify it.

As expected, verification fails now:

AL Verifier reported the error and the line number of the first invalid integrity check (caused by a mismatch between the expected value and the actual one).

Logs meet real world #

Cryptographically signed audit logs and their verification should work together with other security controls in the system, building layers of defense in depth.

Previously we explained how Acra can help you to build defense in depth around data, here we outline some practical considerations on how to protect your systems with secure audit logging.

Protect log contents #

Acra logs are signed not encrypted, because they don’t contain sensitive information (any potentially sensitive information, like query params, is removed by Acra’s log handlers).

If your system’s logs contain sensitive data, consider using data encryption for the log files at rest.

Protect against log deletion #

It’s important to remember that cryptography helps to verify the authenticity of crypto signed audit logs, but it can’t actually prevent attackers from deleting log entries or log files themselves.

Secure audit logs help to find invalid log entry but don’t help to identify what exactly had happened based on the log itself.

It makes sense to use a combination of security controls to ensure that logs are untouched: using special storage devices (like WORM drives), ensuring multiple backups in the physically independent locations, monitoring sudden changes in log files’ size, etc.

Protect against attacks on logging producers and verifiers #

If attackers gain control of logging or verifying service (i.e., one of the Acra tools), they might affect produced logs or verification results.

Acra tools should run inside a trusted perimeter on a host machine protected by several security measures: access control, monitoring, repeated authentication. 

It makes sense to configure an intrusion detection system on the host machine which checks that the audit log file has been successfully stored, copied or verified. (H)IDS should alert on errors with creating / copying files and log verification.

Automate log verification #

Use log verification as an ongoing automated procedure (verify logs every X hours, every Y days, or on log rotate event).

Ongoing verification of audit logs helps to detect potential issues earlier. Acra can be configured to reset the log chain and start a new one according to a specific schedule.

Log backups #

Configure audit log backups into different locations. Depending on the industry and regulations, backups can be required to be stored for a long time.

Log key rotation #

Log key should be periodically updated/rotated using a remote trusted server or local trusted hardware. This should be a part of regular key management procedures run in the system.

Note that rotated log keys might need to be stored to verify old audit log files.

We work with companies on demanding markets. Read how we use Acra to protect data in critical infrastructure.

Critical infrastructure

Industrial

Protecting telemetry data of power grids

Protecting data signals transmitted over the air between power distribution stations and central dispatch system.

Read story

Open sourcing audit logging module #

Currently, cryptographically signed audit logging is available in Acra Enterprise Edition only.

We have plans to open source the audit logging module as a separate library (AL Logger and AL Verifier) later. Audit logging library will fulfil our collection of free and open-source security tools that help developers to protect their users’ data.

Closing thoughts #

Using cryptographically signed audit logs is one of the ways to protect audit logs' integrity and authenticity.

We use a cryptographic scheme where the integrity check of each log entry depends on the previous log entry. Because of the chaining, verification of the audit log file shows if the whole log stream is valid and indicates the first invalid log entry for tampered logs.

Combining crypto signed audit logging with ongoing automated verification shortens MTTD and MTTR on potential incidents.

We believe that companies and users should own their data. Therefore, we provide fast, reliable, and secure tools for data protection that work inside your system.

Acra helps you to protect data, adding application level encryption, searchable encryption, data masking, etc. Use Acra whenever you want to control your sensitive data and keep it protected.