An audit log is a documented record of events and activity within an information technology system. Audit logging is critical for application owners and administrators to track activity within their systems.

In this post, we'll first dive into what auditing entails and what to consider when audit logging. We'll then explore some options for implementing audit logs, including PaperTrail, Audited, AuditLog, AppSignal, and a custom implementation.

Let's get started!

Auditing and Audit Logs: An Introduction

Auditing provides an understanding of system behavior using verifiable, well-defined events. It has a different goal than application logging, which records general-purpose messages often used to diagnose or troubleshoot issues within an application.

Auditing is typically a foundational requirement for many systems, yet it is often one of the last concerns of development teams. The use of simple, effective patterns upfront can change this trend.

One motivation for teams to tackle this capability early on is the feedback it provides in terms of metrics for testing and evaluating systems. The observability of audit events can highlight system issues or even missed requirements.

Audit logs typically have a defined structure that includes the following event information:

• Timestamp
• Event type
• Description
• User or system process that initiated or requested the event
• Impacted entities in the system
• Any additional relevant detail

Any system, service, or device can have its own audit log. A collection of audit logs spanning a particular dimension is considered an audit trail of related activity. An audit trail could be specific to a user, application, or system entity. For example, the AWS CloudTrail service records all activity within an AWS account.

Audit logs support, among others, the following key functions and use cases:

• Compliance and regulatory requirements
• Security reviews and investigations of potential security breaches
• Determining what users make particular changes in a system and when

Design Considerations for Audit Capabilities

Audit logging solutions typically involve a centralized log management system. Each application instance can directly send audit events to the centralized repository, or the solution may include agents running on compute nodes that facilitate this communication.

Distributed architectures can make auditing challenging because there are many sources of audit events throughout the infrastructure. Centralized solutions aim to funnel all of the disparate audit events through a central pipeline for processing.

Audit management systems provide storage and search capabilities for the collection of audit events. There are several persistence options, although many Ruby and Rails-based solutions use database persistence to store audit events as a natural extension of ActiveRecord capabilities.

As an alternative, cloud data streaming services can also be used. For example, AWS Kinesis is designed to handle extremely large volumes of streaming data and is well suited for the audit use case. Another option is to store audit data in AWS S3 buckets and then query it using AWS Athena.

What Constitutes an Audit Event in Ruby and Rails?

Countless events occur within a system, so one of the most important decisions is determining what exactly to audit. Audit events can be based on functional requirements (e.g., a user submits an order) or non-functional requirements (e.g., a security-relevant event occurs, such as a failed login attempt). Both types are important and serve different purposes. Certainly, security auditing is of foremost concern in today’s operating environment.

The level of detail is another key consideration. A single business transaction may involve updates to multiple database tables. While a change in an individual database entity could be the subject of an audit event, by itself this may not contain enough information about the overarching event taking place. It is often beneficial to audit the business function in addition to, or even in place of, individual database rows. Business events typically occur within the service or controller level code.

Some use cases require all of the detail you have. For example, requirements may dictate that all user HTTP requests must be audited. You may need to know which users queried for particular information in comprehensive security audits. As is often the case in software engineering, let your requirements be your guide in making these design decisions.

Ruby Audit Log Implementation Options

Most Ruby gems in this space are focused on the Rails use case, as it has ORM capabilities to identify changes to ActiveRecord instances and store them as audit events. This enables basic audit capabilities without much development effort. Keep in mind, however, the level of detail of your audit events. You may need to add code to specifically audit business events in your service or controller classes.

AppSignal also has a new logging feature as part of its integration for Ruby. Applications can log messages and structured event data from their applications which are stored on managed AppSignal servers. Once you've signed up, you can use AppSignal dashboards to search, filter, and view your logs.

PaperTrail

The PaperTrail gem is a popular choice for audit logging in Ruby. It is focused on tracking changes to ActiveRecord model classes using a versions table. It allows you to see how a model looks at any point in its lifecycle. Several configuration options determine what type of model changes result in creating an audit event, or version. For example, you can choose to audit all changes or only create audit events where designated attributes are modified.

If you have an application that requires maintaining a history of changes over time, you will find this gem particularly useful. There is a good deal of overlap between auditing and versioning capabilities, and PaperTrail provides a nice solution for both. A versions method is added to each model class.

You can easily see the changes between versions as well, as shown in this code snippet from the PaperTrail documentation:

widget = Widget.create name: 'Bob'
widget.versions.last.changeset # reads object_changes column
# {
#   "name"=>[nil, "Bob"],
#   "created_at"=>[nil, 2015-08-10 04:10:40 UTC],
#   "updated_at"=>[nil, 2015-08-10 04:10:40 UTC],
#   "id"=>[nil, 1]
# }
widget.update name: 'Robert'
widget.versions.last.changeset
# {
#   "name"=>["Bob", "Robert"],
#   "updated_at"=>[2015-08-10 04:13:19 UTC, 2015-08-10 04:13:19 UTC]
# }
widget.destroy
widget.versions.last.changeset
# {}

As long as your controller has a current_user method, it will track who is responsible for changes using the following callback:

class ApplicationController
  before_action :set_paper_trail_whodunnit
end

One drawback is the lack of a well-defined pattern for custom audit events. A workaround is to create a database entity that captures relevant audit information, although that somewhat defeats the purpose of this gem’s paradigm.

Audited

The Audited gem is another implementation similar in scope to PaperTrail. It now only focuses on ActiveRecord support. An opt-in approach is used to denote which model classes should be audited, as shown below.

class User < ActiveRecord::Base
  audited
end

Model classes get a revisions method similar to PaperTrail’s versions. Audited models also get an audits method that provides change-tracking information.

However, this choice suffers from the same drawback as PaperTrail: a lack of support for custom audit events.

AuditLog

AuditLog takes a different approach. Rather than centering the design around model classes, it uses explicit calls to create audit events, as shown in the code below.

AuditLog.audit!(:update_password, @user, payload: { ip: request.remote_ip })
AuditLog.audit!(:sign_in, @user, payload: { ip: request.remote_ip })
AuditLog.audit!(:create_address, nil, payload: params)

If you want to audit individual model entities, you can use code similar to the following to accomplish this.

class TicketsController < ApplicationController
  def index
    audit! :list_tickets, nil
  end
 
  def create
    if @ticket.save
      audit! :create_ticket, @ticket, payload: ticket_params
    else
      render :new
    end
  end
 
  def update
    if @ticket.save
      audit! :update_ticket, @ticket, payload: ticket_params
    else
      render :edit
    end
  end
  ...
end

PaperTrail and Audited can do this automatically for you. However, in this code sample, you can also see the use of action-based auditing in the index method. This equates to an audit event capturing a view tickets operation.

AuditLog also provides a simple web interface to view audit information. While your operational requirements may dictate a more comprehensive reporting capability, this is a nice feature to have right out of the box.

Audit Logging with AppSignal

AppSignal provides a managed logging mechanism that offloads the storage of log data and provides a comprehensive set of search and filter capabilities. Each application has a default log source, or you can create additional sources as shown below.

The following code shows an example of creating an audit log event from a sample blog application. The logger constructor takes a group name as a parameter that can be used to identify event sources. This code example comes from an articles controller class where new articles are created.

logger = Appsignal::Logger.new("articles")
logger.info("Created new article #{@article.id}", { article_id: @article.id, date: DateTime.now })

AppSignal dashboards can be used to view log data, both at the aggregate level as well as detailed audit log information.

You can drill down into specific log events and view the structured data included with the event, as shown below.

Custom Audit Logging Implementation

You can also roll your own implementation if you have specific requirements that don’t fit one of these solutions. In most cases, you will need to implement additional reporting capabilities using the stored audit data.

For this, you may choose to:

• Use another Ruby gem.
• Push the data to a data warehouse or cloud service for analysis and long-term storage.

Choosing an Audit Logging Implementation

Here are a few key considerations to help you choose between audit solutions:

• What types of events do you need to audit? Are they primarily database-driven events? If so, a solution like PaperTrail is a great choice. If you have higher-level business events, then you need support for custom audit events similar to the pattern implemented by AuditLog.
• Do gems with higher activity levels give you more confidence? If so, PaperTrail and Audited are two great choices. AuditLog provides more flexibility, but it has less of an installed user base.
• What volume of audit events do you need to capture? Will the storage of audit events in the same application database add significant overhead to your application? If so, consider using AppSignal for minimal impact to your application's performance.
• Remember that you also need query and reporting capabilities. The out-of-the-box capability here varies widely across the different options. AppSignal provides a comprehensive search and filter capability over log data.

Wrapping Up

Audit logging is a fundamental capability for almost all business applications. We've seen that it is beneficial to establish a design pattern early on for the observability of a system as development progresses.

There are some great implementations available for Ruby and Rails developers: PaperTrail, Audited, AuditLog, and AppSignal. Evaluate your requirements using the criteria discussed above to determine the best choice for your needs.

Happy audit logging!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!