Ruby Magic by AppSignal

Data Sovereignty: How to Keep All of Your Services in Europe (AppSignal + Hatchbox)

Roy Tomeij — 2026-05-07T00:00:00+00:00

Over the last decade, a great deal of data privacy regulations have been passed in the European Union. Like it or not, measures like GDPR, the Digital Services Act, and the upcoming Artificial Intelligence Act are exerting increasing influence across industries over how and especially where the data of European customers is stored.

In this article, we will explore the ways to keep the simplicity of a Platform as a Service (PaaS) while utilizing only European providers.

Why Data Sovereignty Matters More Than Ever

The General Data Protection Regulation (GDPR), which was ratified in 2016, granted fundamental rights to the "data subject". These include the right of access, the right of rectification, the right of erasure, and similar. One of the implications of this regulation was that personal data of EU citizens can only be stored inside the Union or countries with adequate protection levels.

In the 2020 Schrems II ruling, the EU Court of Justice invalidated the EU-US Privacy Shield over concerns of data privacy adequacy. The ripple effect caused by this decision (and others) has led to greater vigilance about data privacy among European citizens, and customers have started to wonder where their personal information is stored.

All this has put European companies under increased pressure to both comply with the GDPR requirements and make those efforts visible to target groups.

Traditional Trade-offs (and Why They Suck)

Before it has switched to “maintenance mode” with no clear successor emerging, Heroku was undoubtedly the go-to provider of the last decade. As a traditional PaaS, it defined deployment ergonomics: git push heroku master was the gold standard of developer experience. But even though it set the bar, it was never without drawbacks (costs, limited European options, and infrastructure lock-in, to name a few). Developers were always limited to databases, key-value stores, and logging services it provided. Then, there were limited scaling options and the inability to fine-tune resources; compromises worth making, but far from ideal.

Another option is to build your own operations pipeline: implement a DIY solution on bare metal, create the necessary tooling, and deploy every change manually. Even with modern container orchestration tools like Kamal, this approach introduces a massive DevOps overhead. Permissions have to be managed, passwords need to be stored securely and shared among the team, you know the drill. Basically, it boils down to hiring manpower to synchronize the inner and outer loops (development and operations).

The youngest "traditional" way to approach deployment and operations is managed Kubernetes. However, it’s important to point out that it has a steep learning curve and is probably overkill for most Rails apps. It does incorporate some of the developer experience improvements that Heroku pioneered, but you'll still want to have a person on the team to manage the configuration. And if you're just starting out, that's not an ideal situation.

The Third Way: Own Your Cloud Without the Complexity

But what if I told you you can combine developer ergonomics with the freedom to choose your cloud provider? To deploy with a single git push while keeping your data in Europe? To connect to an application and performance monitoring service (APM) that also serves as log storage and uptime watchdog? There is a way to achieve this, and while it's not a silver bullet, it comes close.

Hatchbox is a DevOps service provider designed with one goal in mind: to simplify the deployment pipeline for you. You pick a (European) cloud provider like Scaleway or Hetzner (or any other Ubuntu-based server with root access), create a server, and Hatchbox takes care of the rest. You can deploy your Rails application with one click, or even git push.

AppSignal is a EU-hosted monitoring service that boasts a full suite of observation tools: error and performance monitoring, uptime watchdog, log management, server metrics dashboards, etc.

Below, we will analyze this powerful combination using a real-world example.

Practical Setup: A European-First Rails Stack

We’ll begin by setting up our Hatchbox account, provisioning a server, and deploying a Rails application. First, sign up at Hatchbox and enter a payment method to start adding servers.

Next, we have to create a new cluster. In Hatchbox, a cluster is simply a group of servers where each one has a different role. That can be a web server, load balancer, database server, background processors, and so on.

For the sake of this example, I will use a European option, Hetzner. However, like I’ve said, you can connect any provider via the "custom" option, as long as it can provision a server running an Ubuntu image for you. My second go-to European cloud provider is Scaleway, which also allows you to prepare a managed database and Redis services.

After setting up the cluster, we’re asked to provide an API token for our Hetzner account. To create one, navigate to a Hetzner project and click on Security. From there, create an API token with a meaningful description. Remember to give it read and write access:

Now use it to connect your Hatchbox account to Hetzner:

After choosing a region, you're ready to add a first server:

For this demo, we are going with a small CX 23 compute instance, which you can choose right inside this form. In other words, you don't need to leave Hatchbox to manage resources in Hetzner.

As noted above, a server can have a couple of distinct roles. We're going to activate four of them here, for demonstration purposes:

Web server (this one is required)
Cron server (also required for migrations during deployments)
Background worker
PostgreSQL
Redis

After we click "Create server", Hatchbox will harden our server and install the necessary services and dependencies:

The next step is to actually deploy an application. To do this, we need to create it within our Hatchbox cluster and give it a name:

Then we're asked to provide the Git repository the app should deploy from. I've already connected my Github account, so I can now simply specify the path.

Once that is saved, we'll have to provision the required PostgreSQL and Redis databases. We can do this by adding new unmanaged instances from the "Databases" tab. (For production, it's advised to use a managed database service, but we'll get to that in a second).

Now I can conveniently click the "Deploy" button in the upper right corner, and Hatchbox will work its magic; that is, prepare the correct environment variables (RAILS_ENV, DATABASE_URL etc.), run the deploy script, and so on.

Aside: You can also mimic Heroku-like behavior by enabling automatic deploys on git push:

What I am leaving out of this tutorial is the fact that you may want to connect your application to a domain or set up SSL. The former requires you to purchase a domain, add it to your Hatchbox app via "Domains & SSL," and point the DNS records to the server. The latter, on the other hand, is more straightforward because Hatchbox uses Caddy, which manages Let’s Encrypt certificates automatically.

At the end of this process, you've got a Rails application running on European infrastructure. But your operation’s requirements likely don't end here. You'll want access to server and business metrics, error and log management, and the like. Luckily, AppSignal is a cost-effective Dutch provider that ticks all these boxes and is very easy to install.

In your application, you only have to add this to your Gemfile:

gem "appsignal"

bundle install it, and after creating an organization in AppSignal, add a Ruby on Rails application.

AppSignal will then provide you with the necessary installation steps:

The CLI will guide you through the installation, and AppSignal will verify the success:

Now that everything’s been set up, you can enjoy some peace of mind, courtesy of AppSignal. I recommend reading the docs for further instructions and configuration options.

To complete your migration to European data services, consider extending your toolbox. Cloud providers like Scaleway or IONOS offer a full suite of services, from managed databases and container registries to CDNs and object storage. For a more comprehensive list of alternatives, consult directories such as European Alternatives.

Benefits and Drawbacks

The approach we’ve outlined in this article comes with several benefits:

First of all, you’ll worry less about whether your application is GDPR-compliant. While there are other challenges to tackle in this domain, keeping your data in European territory makes them all easier to handle.
Hatchbox gives you the flexibility of choosing your own cloud provider and instance configurations, while remaining your single point of contact for management.
Deployment to Hatchbox is just as simple as pushing new releases to Heroku used to be.
Enlisting AppSignal as your APM keeps the bulk of operational concerns manageable in a cost-effective and comprehensive way. Gone are the days when you needed separate services for uptime monitoring, log management, performance tracking, and error monitoring. It’s effectively an operations management one-stop shop.

Keep in mind that there are some drawbacks, as well:

There is a bit of DevOps overhead attached to Hatchbox if you ever need to customize your deploys or builds. Tasks like custom TLS certificates, installing OS packages, and similar can all be pulled off, but they will require a bit of manual work.
You’ll be working with more than one company, so you’ll have to manage multiple billing/subscription plans and release cycles.

Clearly, the benefits outweigh the downsides in this approach. All things considered, what’s not to love?

Conclusion

You don't need a DevOps team or complex PaaS to keep your data in Europe. This article has shown a simple and cost-effective way of deploying your Ruby on Rails application to a European data center. By using AppSignal, you can go a step further and ensure your application metrics and other monitoring data remain within domestic borders. And with agentic AI tools improving over time, achieving data sovereignty is bound to become even easier.

Go ahead and try AppSignal and Hatchbox for a fully European stack.

Happy prompting!

Setting Up Server Monitoring for a Rails App on Hatchbox

Roy Tomeij — 2026-04-30T00:00:00+00:00

Owning your server stack shouldn't be a source of anxiety. Unfortunately, it often is, especially if you only pay attention to the problems you can feel in your gut: Is the app running? Is it throwing exceptions? Does it seem fast enough? These are great intuitive measurements, but just as a doctor uses diagnostics to catch high blood pressure before it becomes a crisis, you need deeper visibility to detect memory leaks, CPU spikes, and disk consumption before they bring your project to a halt.

Hatchbox and AppSignal give you that "deeper visibility." They simplify infrastructure management by replacing manual monitoring with automated, real-time feedback. Together, they transform complex data into actionable insights and make server management accessible to any developer. This gives you the diagnostic depth you’d expect from a much larger operation, without the overhead cost.

What AppSignal Captures at the Host Level

Hatchbox is built to coordinate and manage your Ruby on Rails servers, providing high-level observability for your cluster. To offer greater insights, it has teamed up with AppSignal through a seamless integration. This partnership takes you from manual dashboard monitoring to automated instrumentation, historical trend analysis, alerting, and in-depth metrics all within the context of your application.

When you add the AppSignal gem to your app, you get access to both APM (Application Performance Monitoring) measurements and server-level instrumentation. This includes load average, CPU/memory usage, network traffic, and disk I/O. The table below breaks down how these two layers of visibility work together:

	Server-level	App-level (APM)
Provider	Hatchbox/Host, AppSignal	AppSignal
Focus	Infrastructure & hardware	Code & business logic
Visibility	"Is the server up?"	"Is the `/checkout` route slow?"
Typical data	Load avg, memory %, disk I/O	Throughput, error rates, traces

Unlike other services, AppSignal doesn't require a separate tool or daemon; everything is built into the AppSignal gem. The Hatchbox integration enables you to connect your stack in just two clicks, after which you can view real-time insights within Hatchbox or jump directly into AppSignal to debug.

Reading the Host Dashboard

To see the host-level measurements captured by AppSignal, you'll need to navigate to the Host metrics page first. You can find that navigation item under Host monitoring. Once there, click the specific host you're interested in. Alternatively, you can click the specific host listed under the Worst 10 hosts by section within the Overview dashboard.

The Host metrics dashboard displays several charts. These can be broken down into groups: Memory, CPU and load, Disk usage, and Network traffic. Let's take a closer look at each.

Note: Host metrics are only available for apps deployed to a server.

Memory

Ruby is a memory intensive language, and Rails is not the leanest framework.

Even before it handles its first request, a base Rails application with standard boilerplate can easily consume 100 MB to 150 MB of RAM at boot. This makes memory utilization one of the primary measurements to keep an eye on.

In general, a healthy server runs between 40% and 70% memory utilization (used / total * 100.0). When you’re looking for a "normal" memory picture for Hatchbox managed servers, don’t expect to see a flat line. It’s more of a waveform. You'll see peaks during times of high traffic and valleys during low-usage periods. You'll also notice regular drops in memory as Ruby's garbage collector frees up unused memory.

As you continue developing your app (and as you attract more users), you'll notice "used memory" slowly start to increase. This is referred to as "drift," and you won't be able to see it unless you expand your time-frame to the 30-day view.

This drift shouldn't be confused with a memory leak. It’s caused by organic data growth or adding dependencies. Leaks, on the other hand, tend to look like a gradual increase in memory over a shorter time period. While memory usage may drop with traffic usage, the "floor" will continue to grow every day.

Another thing to note: If you see memory climb steadily and then suddenly "crash" to a low point before climbing again, this may signal a process (like a Sidekiq or Resque worker) crashing and getting restarted by the OS or Hatchbox after hitting a limit.

CPU and Load

It's not uncommon to view CPU and load averages as the same thing, but they are actually not. A great way to look at these differences is with a doctor's office analogy. CPU usage would be the speed at which a doctor is able to see patients. Similarly, load average could be imagined as the number of people sitting in the waiting room.

A load of 1.0 on a 1-core machine means the cashier is busy, but there is no line.
A load of 2.0 on a 1-core machine means the cashier is busy and there is another person waiting their turn.

Spikes in CPU and load don't always indicate a problem. Often, they show a process kicking off. For example, it's not uncommon for your CPU to max out during the assets:precompile step of a deploy. Likewise, worker restarts may cause a jump in CPU usage and load as it retrieves and starts working on a new batch of jobs.

Disk Usage

When memory, CPU, or load averages spike, your users will experience degraded performance and slower response times. However, if disk usage spikes, they experience a cessation of all activities. Disk usage of 100% on your server means databases can't write recovery logs, temp files can't be created for uploads, and the operating system itself often locks up.

When you're alerted in the middle of the night because of a server crash, these are the three things to look for:

Logs (the usual suspects): Even when you're rotating logs, a "noisy" app can generate gigabytes of text faster than the rotation script can clean it up.
Temp files (/tmp): Image processing and file generation usually happen in the background. Processes such as Resque and Sidekiq can quickly fill up the /tmp and leave no trace of a problem after the server restarts.
Database WAL segments: Both Postgres and MySQL write changes to a WAL file before updating data. There are some instances where a replica loses connection and these segments pile up until the disk is full.

As a rule of thumb, treat 80% disk usage as your warning signal to start investigating. Once you cross 95%, the operating system loses the overhead it needs for routine tasks, potentially leading to a silent yet total failure.

Connecting Host Metrics to App Problems

A measurement by itself is just a number. For it to provide value and help you make good decisions, it needs to be correlated and compared to other data. When taken alone, measurements such as CPU %, memory usage, and disk usage are known as “vanity metrics.”

Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action.

— Lean Analytics

Although you can use the Host metrics page to compare measurements, it's worth your while to set up a custom dashboard and display measurements either within the same chart or next to one another.

Below, you’ll find a metric correlation cheat sheet:

If you see...	Check this Host Metric...	Likely Culprit
High response times / low CPU	Disk I/O wait	Database bottleneck or slow logs
Unpredictable "jittery" latency	Swap usage	Memory exhaustion / leaks
High request queueing	Load average	Undersized server (need more cores)
High CPU / low throughput	CPU usage (per process)	Inefficient code or background job "storms"
High response times and/or DB errors	Disk usage	Insufficient space for swapping or Write-Ahead Logs (WAL)
CPU spikes	Deployment marker	Investigate if no deployment marker

Alerts Worth Setting Up

A dashboard is only useful while you're looking at it. To stay ahead of issues, you need automated notifications. AppSignal’s Anomaly Detection handles this by alerting you the moment metrics exceed your set thresholds. Setting up these alerts is covered in How to Monitor Your Host Metrics Automatically. For now, these are the triggers you'll want defined for any Hatchbox setup:

Disk usage: You should set up a warning for 80% and a critical alert for 95%.
Sustained memory usage: Set this for 80% usage sustained for 5-10 minutes. If your app stays above that threshold, it's likely "swapping."
Load average anomaly: Define this as a 15-minute average that exceeds your server's core count + 1 (for example, set it for 3.0 on a 2-core server)

Wrapping Up

Monitoring your application's performance without looking at the host is like checking a patient's temperature but ignoring their pulse and breathing. APM tells you what is slow, but host metrics tell you why. Is it the disk at capacity? Is there a memory leak? Is the CPU undersized? Observability is about more than just keeping the lights on. It's about understanding the environment your code lives in.

Hatchbox makes deployment easy, but the responsibility for server health still rests with you. Leveraging AppSignal’s host-level instrumentation lets you keep your finger on the pulse of both your application and the infrastructure hosting it. However, visibility is only the first half of the equation; you cannot spend your entire day watching graphs. By setting up alerts, you move from reactively “firefighting” to proactively managing your stack.

Monitoring Sidekiq Job Performance with AppSignal

Roy Tomeij — 2026-04-28T00:00:00+00:00

When my Sidekiq job starts failing or slowing down, I often feel frustrated, especially if I don’t know how to fix it.

If you’re using Sidekiq to run your background jobs, you know what I’m talking about. It’s a vital element of your stack, handling everything from data exports to password reset requests. It runs silently in the background, and most of the time, you’re not even giving it a second thought.

However, when things go wrong, like pages loading slowly, emails never being sent, or exports failing, the impact can be huge. Getting to the bottom of the issue can be like finding a needle in a haystack.

That’s where AppSignal comes in. It can monitor your Sidekiq jobs, alert you when things go wrong, and give you all the context you need to fix and rerun failed jobs without having to spend hours debugging or dealing with customer complaints.

In this tutorial, you will learn how to monitor your Sidekiq jobs’ performance with AppSignal. To follow along, you need to already have a Ruby application with some Sidekiq jobs.

Note: Even though we’ll be focusing on Sidekiq jobs here, it’s worth noting that AppSignal can monitor all aspects of your Rails application.

The Problem

Sidekiq jobs live in a different part of the web application, which can make issues hard to spot.

When a controller action fails, you get an alert within seconds. But what happens when a background job silently retries four times and gives up? You usually find out only when a user emails support asking where their CSV export went. These jobs slowly start piling up in the queue and degrading in performance without any obvious signals.

When Sidekiq jobs fail, you might see things like:

Queues backing up and throughput collapsing
Vital jobs silently exhausting their retries and dying
Exceptions getting swallowed with no trace in your logs
Users feeling the impact long before you realize something is happening

What You Get Out of the Box

A great thing about AppSignal is how easy it is to get started with. You don’t need a bunch of customization. Just add the appsignal gem, and you’re good to go.

From that point on, every Sidekiq job execution is tracked automatically. You get insights like job duration, queue wait time, and many other instrumented events that fire within the job itself. In practice, this means you get a structured performance data and error visibility of your jobs immediately after AppSignal has been activated.

If you’d like a full breakdown of what is captured and how the integration works under the hood, the AppSignal Sidekiq documentation is worth a look. However, here’s a categorized list of what you get out of the box:

Job duration and throughput per worker, or how long each job takes and how many are being completed;
Queue length and latency, or how many jobs are waiting and how long they've been sitting there;
Job status per queue, which is a breakdown of succeeded, failed, and retried jobs across every queue;
Job count, which is the number of workers and processes that are active at any given time;
Connection count, or the number of open Sidekiq connections;
Redis memory usage, which refers to both allocated and RSS memory consumed by your Redis instance.

Three Signals That Matter

Instead of just going through different random job performance metrics, we will focus on the three most important ones:

Job duration lets you spot your slowest jobs before they become outages. A job that averaged 2 seconds and now averages 12 is a problem you want to catch early.
Queue latency has to do with the time a job spends sitting in the queue waiting to be picked up by a worker.
Error rate and retry count provides a comparison between the error rate and the retry count.

You will learn how to find those metrics in the AppSignal dashboard and why they actually matter. But, before we go ahead with this section, it’s worth mentioning that Sidekiq provides its own dashboard, as well.

Still, as you can see from the image below, this dashboard doesn’t give you the full picture. It misses some important metrics and additional insights that are available on the AppSignal dashboard.

Adding AppSignal to the Ruby/Sidekiq Project

Assuming you already have a project set up (as stated in the introduction), these steps will guide you through how you can add AppSignal to your Ruby project. And since AppSignal automatically tracks the Sidekiq metrics, there isn’t much configuration necessary on your end.

Start by adding 'appsignal' to your Gemfile, then:

bundle install

Next, sign in to AppSignal and follow the steps below:

Create a new application using the “Add app” button.
Choose Ruby & Rails as the language.
Fill in the app name.
Install Rails for your framework.
Run the CLI installer provided in the root of your project and follow the instructions. The installer should look like this bundle exec appsignal install YOUR_PUSH_API_KEY .

This creates config/appsignal.yml automatically. Here’s what mine looks like:

# AppSignal Ruby gem configuration
# Visit our documentation for a list of all available configuration options.
# https://docs.appsignal.com/ruby/configuration/options.html
Appsignal.configure do |config|
  config.activate_if_environment("development", "production", "staging")
  config.name = "Sidekiq"
  # The application's Push API key
  # We recommend removing this line and setting this option with the
  # APPSIGNAL_PUSH_API_KEY environment variable instead.
  # https://docs.appsignal.com/ruby/configuration/options.html#option-push_api_key
  config.push_api_key = "APPSIGNAL_PUSH_API_KEY"

  # Configure actions that should not be monitored by AppSignal.
  # For more information see our docs:
  # https://docs.appsignal.com/ruby/configuration/ignore-actions.html
  # config.ignore_actions << "ApplicationController#isup"

  # Configure errors that should not be recorded by AppSignal.
  # For more information see our docs:
  # https://docs.appsignal.com/ruby/configuration/ignore-errors.html
  # config.ignore_errors << "MyCustomError"
end

Update the config/boot.rb (the file that contains the setup for processing jobs) to load AppSignal before Sidekiq:

require 'appsignal'
require 'appsignal/integrations/sidekiq'

Appsignal.start

require 'sidekiq'
require 'app/jobs/welcome_email_job'
require 'app/jobs/order_processing_job'
require 'app/jobs/log_cleanup_job'

Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0') }
end

Now you can run Sidekiq as normal:

bundle exec sidekiq -r ./config/boot.rb -C ./config/sidekiq.yml

AppSignal will automatically track overall job status, job status per queue, duration per worker, and much more, which is enough for what we want to accomplish in this article.

Job Duration

A job that used to take 2 seconds and now averages 12 is your next outage waiting to happen. Any time a job starts taking longer than expected to execute, it’s a sign that something’s off.

For example:

A database query inside the job is hitting an unindexed table as data grows;
An external API the job depends on has slowed down or started timing out;
A job that once processed 10 records is now processing 10,000 with no change to the underlying code.

You want to catch this early because that way, you’ll be fixing a performance issue instead of fighting a full-blown incident. You can easily spot this metric on Duration per worker graph on the dashboard, which shows the amount of time that it took jobs to execute, grouped by namespace and action.

Queue Latency

Latency is the time between when a job is enqueued and when a worker actually picks it up. It shows you how long a job has been sitting around before it starts.

This makes queue latency your earliest warning for backlog problems. It starts climbing before jobs begin failing or erroring, which gives you a chance to catch issues early. If you spot a spike on the AppSignal Sidekiq dashboard, it means your workers can't keep up with the current load, even if every job that runs is technically succeeding.

In the image below, you can see that there is no latency. However, if you have an application used by multiple users at a time, you’ll probably see some readings in the Queue latency chart.

Error Rate vs. Retry Count

Sidekiq retries can easily mask failures. A job that errors five times before actually succeeding looks fine from the outside. That’s where AppSignal provides a clearer picture by showing the actual error count and backtraces.

Without that visibility, you can end up missing real issues, such as:

a payment processing job that retries silently after hitting a rate limit and fails repeatedly, which can cause the charge to never go through
a CSV export job that fails three times due to a sudden memory spike before it finally succeeds, adding delays that the user has to deal with
a SendWebhookJob to a partner API that fails three times with a 503 error before it gets a 200 response. Sidekiq marks it as successful; meanwhile, your partner is experiencing delays in data, and their support team is filing tickets with yours. Luckily, AppSignal shows the 503 errors and identifies which partner endpoint is having issues.

Sidekiq's built-in retry mechanism is great for resilience, but it can be a hindrance when monitoring your application because it can lead to you missing a potential problem. The job we’ve mentioned above that errors 5 times before eventually succeeding still shows up as zero failures in the Sidekiq web UI, and it just disappears from the queue as "done." AppSignal captures every failed attempt, its backtrace, and the exact job arguments that triggered it. Without this, you can't really know for sure how reliable your jobs are.

You can find these errors in the Dashboard overview.

Get Alerted of Failures Before Users Do

You don’t want to be sitting there, watching the metrics and searching for issues manually. That simply wouldn’t be productive. This is exactly where the Anomaly detection feature comes in.

It lets you set triggers that send notifications when a metric crosses a defined threshold. In the context of Sidekiq jobs, it means getting an alert the moment queue latency spikes or a job class starts returning errors at an unusual rate, ideally before any user opens a support ticket.

The detection runs on a minute-by-minute basis, so the gap between a problem starting and you knowing about it is kept to a minimum.

You can configure Triggers with a variety of metrics, including error count, throughput, error rate increases, slow actions, and queue time.

For the sake of this tutorial, we will set up triggers on the last three.

Before you do that, though, you need to create triggers, the functionality that executes these alerts.

Go to your app navigation and click on the Anomaly detection tab. Choose Triggers, then click the button to add a new trigger. Use the image above as a guide.

Slow Actions

The slow action trigger fires when the mean duration of a specific job class exceeds a time limit you’ve defined. With it, you can catch performance regressions, like when a job that used to run in 2 seconds suddenly starts running in 12 for one reason or another.

To set this up, go to Slow actions tab under Actions. Choose the following options and save the trigger:

Namespace: background
Alert me when the: Mean
Is above (ms): 2000
Tick Default email notifier

When the mean duration of that job crosses 3 seconds, AppSignal sends an alert with the job class name, the current mean, and a direct link to its performance page.

Error Rate Increase

The error rate trigger is activated when the percentage of job executions that raise exceptions goes above your defined limit. Without this kind of metric, a job could start failing 20% of the time and still go unnoticed for hours because Sidekiq retries keep the queue visually clean.

To set this up, click Error rate under Actions. Then select the following options, and save the trigger:

Namespace: Background
Alert me when error rate is above: more than, 5%
Check Default email notifier

With the 5% error rate threshold set here, you get alerted after roughly 1 in 20 job executions fail.

Note: You will need to set a lower threshold for payment processing or sensitive jobs where even a 1% failure rate is unacceptable.

Queue Time

The queue time trigger fires when the latency between enqueuing a job and it being executed exceeds your threshold. This is the earliest signal that there is a backlog issue because latency rises long before jobs begin to fail or workers start to crash.

To set this up, click Queue time under Actions. Then select the following options and save the trigger:

Namespace: Background
Alert me when the: Mean
Is above: more than, 30000ms
Check Default email notifier

With this configuration, you will receive an email whenever the queue time on any monitored queue crosses its threshold. It will include all the information you need in order to identify the defective job.

Here’s an example of what this email looks like:

Wrap-Up

In this article, we started by explaining how AppSignal can help you monitor your Sidekiq job performance. Then, we demonstrated how you can set it up in a Ruby project and explored the three key signals that matter: job duration, queue latency, and error rate versus retry count. Finally, you learned how to set triggers and alerts for slow actions, error rate increases, and queue time thresholds.

This is just one of many ways AppSignal helps you build stable, scalable applications. If you want to create a more rounded observability setup for your background jobs, you can build on the knowledge gained here by exploring AppSignal's other monitoring features.

An Introduction to Ruby Parsing with Prism

Roy Tomeij — 2026-01-07T00:00:00+00:00

You might have heard about Prism, the new Ruby parser. Perhaps you've heard it's faster, more reliable, and more powerful than what we had before. Or maybe you never took a compilers class and aren't sure about what this actually means.

I'm here to tell you all about it, and how it's changing our lives as Ruby developers. Today, I want to take you from square one to writing your first transpiler.

Interpreters 101

Before we begin our journey, let's start with the basics of how an interpreter works, so we're all on the same page. Interpreting a programming language usually involves three main steps:

Tokenizing input (a.k.a. lexing): Breaking the input text into a list of meaningful tokens. That's like converting your code into something like this:

tokens = [
  { type: :integer,    literal: "0",      value: 0,   line: 1 },
  { type: :operator,   literal: "+",      value: nil, line: 1 },
  { type: :integer,    literal: "1",      value: 1,   line: 1 },
  { type: :keyword,    literal: "if",     value: 1,   line: 1 },
  { type: :identifier, literal: "admin?", value: nil, line: 1 }
  # ...
]

Parsing: Analyzing the tokens to understand the program structure (what to do and in which order) and building a data representation that holds that information (known as an Abstract Syntax Tree). For example:

 ast = {
   node_type: :binary,
   operation: "+",
   left: {
     node_type: :number,
     value: 0
   },
   right: {
     node_type: :number,
     value: 1
   },
 }

Evaluating: Executing the parsed input and producing an output. This is where your code actually runs.

For a deeper dive into this topic, I recommend the Crafting Interpreters book or my RailsConf talk on the subject.

Now let's dive into what Prism can do and how it helps with parsing.

Why Is Prism Useful for Ruby Parsing?

Ruby historically used a parser called parse.y, built with Yacc. The catch? It was made specifically for CRuby, forcing other Ruby implementations (like JRuby and TruffleRuby) to create their own parsers from scratch.

That's why tools like RuboCop, code editors, and even other Ruby implementations often lagged behind or had incompatibilities with newer Ruby syntax. Developers building Ruby analysis tools had to write their own parsers too, spawning projects like whitequark/parser and ruby_parser.

Prism solves this by becoming the de facto parser for all Ruby tools and implementations. And it's working: it is now used in CRuby, JRuby, TruffleRuby, Rails, RuboCop, and more.

Okay, enough talk. Prism can lex and parse Ruby, which allows us to build fun things. How about we build a ✨ transpiler ✨ with it? Wait! Don't go away. This will be simple. I promise.

Your First Transpiler

The full code for the examples in this post is available in this repository.

First, we'll build a tool that converts our Ruby code into Emoruby. If you have never seen Emoruby, this is what it looks like:

📋 ❤️
  🔜 👋
    👀 💬😃 🌏💬
  🔚
🔚

❤️▪️🐣▪️👋

Equivalent to this in Ruby:

class Heart
  def wave
    puts "smiley earth_asia"
  end
end

Heart.new.wave

Ready? Ok, here we go. We'll need a Gemfile to install emoruby and prism:

source 'https://rubygems.org'

gem "emoruby", git: "https://github.com/searls/emoruby", branch: "master"
gem "prism"

After bundle installing, let's create the entry point for our transpiler — the Rubyemo.ruby_to_emoji method:

require 'emoruby'
require 'prism'

module Rubyemo
  extend self

  def ruby_to_emoji(src)
    tokenize(src)
  end

  private

  def tokenize(src)
    result = Prism.lex(src)
    raise "Invalid Ruby code" if result.errors.any?

    result.value.map(&:first)
  end
end

For now, it only tokenizes the input with Prism. The lex method returns a result object that contains either the tokens or errors. If the source code contains invalid Ruby, we'll just raise an exception.

Then we get the value attribute, which contains a list of tokens and some other stuff. We only care about the tokens, so we grab them with map(&:first).

Emojify the Ruby Tokens

Now onto the fun part. How do we emojify our Ruby tokens? Emoruby has a very simple design, so we can basically replace tokens one by one with an emoji alternative:

module Rubyemo
  extend self

  def ruby_to_emoji(src)
    tokenize(src)
      .then { emojify it }
      .join
  end

  private

  def emojify(tokens)
    tokens.filter_map do |token|
      next if token.type == :EOF

      token_to_emoji(token)
    end
  end

  # ...
end

To do that mapping, we'll use Emoruby's translation file and EmojiData to translate token values to emojis by name.

module Rubyemo
  # ...

  TRANSLATIONS = Emoruby::ConvertsRubyToEmoji::TRANSLATIONS

  def token_to_emoji(token)
    case token
    in {type: :COMMENT, value:}
      "💭#{value[1..]}"
    in {type: :IGNORED_NEWLINE | :NEWLINE}
      "\n"
    else
      token.value.to_s.split.map do |part|
        TRANSLATIONS[part] || EmojiData.from_short_name(part)&.to_s || part
      end.join(" ")
    end
  end

If a token has no mapping, we leave it as-is so the code still runs. Pattern matching is pretty handy here to match a particular type and grab its value at the same time.

And... that's it! Let's see our little transpiler in action:

emo = Rubyemo.ruby_to_emoji('puts "Hello, world!"')
puts emo # 👀💬Hello, world!💬

Emoruby.eval(emo) # Hello, world!

Fix Indentation and Spacing

You might not have noticed, but our code doesn't handle indentation or spacing. See how the space between puts and the opening quote got lost? We can do better than this, so let's handle that. Luckily, the Prism::Token instances have information about their location, including the start/end line and column.

Let's change emojify to this:

def emojify(tokens)
  previous_line, previous_column = 1, 0

  tokens.filter_map do |token|
    next if token.type == :EOF

    emoji = token_to_emoji(token)
    indentation, previous_line, previous_column = indentation_for(
      token,
      previous_line,
      previous_column
    )

    indentation + emoji
  end

  # ...
end

And implement indentation_for:

def indentation_for(token, previous_line, previous_column)
  if token.location.start_line != previous_line
    previous_line = token.location.start_line
    previous_column = 0
  end
  indentation = " " * (token.location.start_column - previous_column)
  previous_column = token.location.end_column

  [indentation, previous_line, previous_column]
end

Try emojifying this now:

ruby = <<~RUBY
  class Heart
    public def jeans
      puts "purse"
    end

    protected def shirt
      puts "yellow_heart"
    end

    private def wave
      puts "smiley earth_asia"
    end
  end

  Heart.new.wave
RUBY

puts Rubyemo.ruby_to_emoji(ruby)
# 📋 ❤️
#   🔓 🔜 👖
#     👀 💬👛💬
#   🔚
#
#   🔒 🔜 👕
#     👀 💬💛💬
#   🔚
#
#   ⛔️ 🔜 👋
#     👀 💬😃 🌏💬
#   🔚
# 🔚
#
# ❤️▪️🐣▪️👋

Done — for real.

Onto Parsing with Prism for Ruby

Rubyemo helped us to learn Prism lexing, but we didn't need any parsing. Let's try another example. Let's say you learned about Ruby 3.2's Data class and you want to rewrite your old structs with it. Why spend 10 minutes doing it manually when you can write a script in one hour that does it?

Note: If you think this sounds like a RuboCop cop, you're 100% correct! You could turn this into a custom cop if you wanted.

While Prism has a method to parse code, for this, we'll use its Visitor class instead. The visitor design pattern allows us to add new operations to objects (in this case, the Prism AST nodes) without changing their classes. In other words, for each node type it finds, the Visitor class will call a method, and we decide what to do in that situation.

We need to act when we find Struct.new, so that's a method call, which in Prism is identified by the CallNode type. Let's filter those:

require "prism"

class StructToData < Prism::Visitor
  Fix = Data.define(:location, :replacement)

  attr_reader :fixes

  def initialize(src)
    super()
    @src = src
    @fixes = []
  end

  def visit_call_node(node)
    if struct_new?(node)
      # todo
    end

    super
  end

  private

  def struct_new?(node)
    node.name == :new &&
      node.receiver.is_a?(Prism::ConstantReadNode) &&
      node.receiver.name == :Struct
  end
end

Note that we need to call super on the visit methods, which makes Prism keep walking inner nodes (like the body of a class definition).

Now we need to collect the struct arguments and build our fix object with the replacement code. We'll also skip named structs, as those don't map 1:1 to Data classes:

# ...
def visit_call_node(node)
  if struct_new?(node) && !named_struct?(node)
    members = struct_members(node)
    replacement = build_replacement(members, node.block)
    @fixes << Fix.new(node.location, replacement)
  end

  super
end

private

# ...

# skips interpolated symbols for simplicity
def struct_members(node)
  (node.arguments&.arguments || [])
    .take_while { it.is_a?(Prism::SymbolNode) }
    .map(&:slice)
end

def named_struct?(node)
  (node.arguments&.arguments || [])
    .first
    .is_a?(Prism::StringNode)
end

def build_replacement(node_members, node_block)
  call = "Data.define(#{node_members.join(", ")})"
  if node_block
    call += " #{node_block.slice}"
  end
  call
end

Make Fixes with a Method

Now let's write a method to apply the fixes. We'll process them in ascending order of start position, so offsets remain correct as we build the new source.

class StructToData < Prism::Visitor
  #...
  def self.rewrite(source)
    ast = Prism.parse(source)
    return [source, []] unless ast.success?

    v = new(source)
    v.visit(ast.value)
    [v.apply_fixes, v.fixes]
  end

  # ...

  private

  def apply_fixes
    return @src if @fixes.empty?

    pos = 0
    out = +""
    @fixes.sort_by { it.location.start_offset }.each do |fix|
      out << @src.byteslice(pos...fix.location.start_offset)
      out << fix.replacement
      pos = fix.location.end_offset
    end
    out << @src.byteslice(pos..-1)

    out
  end

Note: We have to use byteslice because Prism offsets are in bytes, while replacing using something like String#[]= would fail on multibyte characters.

This is enough for us to test our code. Let's see it in action:

source = "Point = Struct.new(:x, :y, keyword_init: true)"

rewritten, _fixes = StructToData.rewrite(source)

rewritten =="Point = Data.define(:x, :y)" # => true

It works!

Mutation: The Source of All Evil

There's one big problem with our current approach: Data objects are immutable, while structs aren't, so we can't always convert them. We have to also skip structs that mutate internal state.

We'll use an inner visitor to check if a struct body contains mutations:

class MutationScanner < Prism::Visitor
  def initialize
    super
    @mutates = false
  end

  def mutates? = @mutates

  def visit_call_node(n)
    # self.x = ..., self[:k] = ...
    if n.receiver.is_a?(Prism::SelfNode) && n.name.to_s.end_with?("=")
      @mutates = true
    end

    # adding writers via macros
    if n.name == :attr_writer || n.name == :attr_accessor
      @mutates = true
    end

    # define_method(:x=) { ... }
    if n.name == :define_method
      arg = n.arguments&.arguments&.first
      if (arg.is_a?(Prism::SymbolNode) || arg.is_a?(Prism::StringNode)) && arg.unescaped.end_with?("=")
        @mutates = true
      end
    end

    super
  end

  # def x=(...) / def []=(...)
  def visit_def_node(n)
    if n.name.to_s.end_with?("=")
      @mutates = true
    end
    super
  end
end

This catches many cases, but there are many more ways to mutate a value in Ruby (i.e., by writing to an instance variable): writing to ivars directly, using attribute writers, or memoization, to name a few. It would be a chore to define methods for each one manually.

Luckily, Prism consistently names the nodes for these mutation methods, so we'll just flag any nodes that perform a write:

class MutationScanner < Prism::Visitor
  # ... after def visit_def_node

  Prism
    .constants
    .filter_map do |const_name|
      next if const_name !~ /Write/ || const_name =~ /GlobalVariable|LocalVariable|Constant/

      Prism.const_get(const_name)
    end
    .each do |node_class|
      define_method("visit_#{node_class.type}") do |n|
        @mutates = true
        super(n)
      end
    end

So we get all "write" constants and define methods, but ignore writing to constants, local variables, and global variables (as those don't change a struct's internal state).

Let's wire up this scanner now:

class StructToData < Prism::Visitor
  #...
  def visit_call_node(node)
    if struct_new?(node) && !named_struct?(node) && !mutates_instance_state?(node.block)
      # build fix
    end
  end

  # ...

  def mutates_instance_state?(block_node)
    return false if block_node.nil?

    scanner = MutationScanner.new
    scanner.visit(block_node)
    scanner.mutates?
  end
end

That's it! Now our rewriter is ready to dance. Try it out with some of your structs.

Wrapping Up

Prism has already reshaped the Ruby landscape by making our tools faster, more portable, and more consistent. But its real impact will come from what you build with it.

Think bigger than just parsing: a Ruby-to-JS transpiler, a test runner that knows exactly which test to run from a file and line number, or even something that turns your code into pixel art. The parser is no longer the bottleneck — your imagination is.

Go make something amazing!

AppSignal’s Top 5 Ruby Posts in 2025

Roy Tomeij — 2025-12-17T00:00:00+00:00

As the year comes to a close, we’re thrilled to highlight our top five most-read Ruby articles of 2025:

Season's Greetings ❆ ⛄

Have a joyful festive season, and we’ll see you in the new year!

If you haven’t already, don’t forget to subscribe to our Ruby Magic newsletter, so you never miss an upcoming post.

Create a Markdown Editor in Ruby on Rails

Roy Tomeij — 2025-12-10T00:00:00+00:00

In recent years, Markdown has become the lingua franca of plain-text files on the web. If you're a developer, you have read — and maybe even written — hundreds of Markdown documents over the course of your career.

GitHub repositories use README files written in Markdown. Stack Overflow and Reddit use it to format posts. Technical documentation, blog posts, and entire books are written in Markdown. And it's not just for humans either! AI tools such as Claude Code and Cursor use Markdown documents to improve the effectiveness of AI agents. This article you are reading is — you guessed it — written in Markdown!

Ruby on Rails, of course, has its own tooling around Markdown, and in this post, we'll build a Markdown editor using Rails.

Markdown in Ruby on Rails

Rails 8.1 brings Markdown as content type, as well as a new rich text editor with Markdown support. And if you can't or don't want to use Rails defaults, there are always gems, such as Marksmith.

But what does it actually take to build a GitHub-like Markdown editor? Something simple that supports editing Markdown text and previewing the result. Something like this:

Let's find out!

Markdown and Markdown Flavors

Before we start with the implementation, let's talk about the Markdown language itself. First, it's essential to understand that there is no single, definitive Markdown language. When Markdown was conceived in 2004, its original description left quite some room for interpretation. As a result, a plethora of Markdown dialects — or flavors — sprang into existence, each of them supporting different features and slightly different syntaxes.

The flavor you might be most familiar with is GitHub Flavored Markdown, or GFM in short. It is based on CommonMark, an attempt to create a formal, unified specification for Markdown, which was created in 2014. The CommonMark specification has been widely adopted, but even so, several different Markdown flavors remain common today.

Now, what makes Markdown so interesting, and why was it so widely adopted? The beautiful thing about Markdown is that it strikes a perfect balance between being structured and unstructured. It is not as verbose as other, more structured languages (such as XML, HTML, or JSON), and that makes it easy to read and write. But it provides just enough structure to be processed by computers. And that allows us to do interesting things with it.

One of those interesting things is converting Markdown to other formats, such as HTML. As a matter of fact, that was exactly what its designers had in mind!

Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid HTML. Source: Markdown, John Gruber's website

Using Ruby, we have several fine options for converting Markdown into HTML: redcarpet, commonmarker, and Kramdown, to name a few. What they all have in common is that they convert Markdown to HTML. For example:

Markdown: Markdown supports *italic* and **bold** text.

HTML: Markdown supports italic and bold text.

As I mentioned earlier, different flavors of Markdown work differently. We'll be using GitHub Flavored Markdown (GFM) for the rest of this article. If you're curious about how GFM differs from other Markdown flavors, take a look at the specification. Note the sections tagged as extension — these are features that are unique to GFM and not usually found in other Markdown dialects.

Now that we have covered the theory, let's get building.

Creating a Markdown Editor with Ruby on Rails

Our Markdown editor will require three things. First, a way for users to write — and save — plain text. This is easily accomplished using off-the-shelf Rails features. Second, we'll utilize a third-party gem to render Markdown to HTML. And finally, a tiny bit of JavaScript to tie these things together and create a nice user experience.

While the quickest way to add a Markdown editor to your application is simply by using a gem — such as the excellent Marksmith — it's always worthwhile to learn what goes on under the hood. You can follow along with the code used in this post by viewing or cloning this repository.

Let's start a new Rails app with a simple post model that will hold our Markdown text. We use Tailwind CSS for styling and also enable ActiveStorage. Why do we need ActiveStorage, you ask?

It's a surprise: you'll see later.

rails new github-markdown-editor --css=tailwind && cd github-markdown-editor
rails active_storage:install
# Add our Post model and migrate
rails g scaffold Post body:text
rails db:migrate

The post scaffold creates a form with a text area, which is all we need for users to write and save Markdown documents. Now we need to render those documents to HTML. We'll use Commonmarker. It supports GitHub Flavored Markdown and is easily customizable, which makes it the logical choice.

bundle add commonmarker

Before we tackle the editor itself, let's update our app so it displays rendered Markdown when showing a post. We only need to add a handful of lines to both our posts controller and the post partial.

Note our use of html_safe. Otherwise, the HTML created by Commonmarker will be sanitized by Rails. We don't want sanitized HTML — we want our HTML to be shown to the user as rendered.

# app/controllers/posts_controller.rb
class PostsController < ApplicationController

  def show
    require 'commonmarker'
    @markdown = Commonmarker.to_html(@post.body).html_safe
  end

  #...
end

<!-- app/views/posts/_post.html.erb -->
<div id="<%= dom_id post %>" class="w-full sm:w-auto my-5 space-y-5">
  <!-- ... -->
  <div>
    <strong class="block font-medium mb-1">Markdown:</strong>
    <div>
      <%= @markdown %>
    </div>
  </div>
</div>

To try it out, let's create a new post with the following Markdown content.

# Markdown Rendering

Let's add some Markdown rendering to a Rails application using the [Commonmarker](https://github.com/gjtorikian/commonmarker) gem.

Markdown supports **bold** and _italic_ text. GFM adds features like ~~strike through text~~.

Autolinking and emojis are supported by Commonmarker, check it out: https://www.appsignal.com/ :heart:

There's also syntax highlighting:

```ruby
def greet(name)
  puts "Hello, #{name}"
end
```

As well as tables and lots of other features!

| Gem          | Main Feature                                |
| ------------ | ------------------------------------------- |
| CommonMarker | GitHub-flavored Markdown                    |
| AppSignal    | Performance monitoring for your application |

If we view the post, things are clearly working. We can see bold and italic text, links, and even syntax highlighting. But we are still far from a GitHub-like editor.

We don't want to create a post to see what the resulting HTML looks like. What we need is a live preview.

Preview with Turbo Streams

Rather than showing rendered HTML only after a post is created, we want to display it while in the writing process. We'll easily be able to do so using Turbo Streams and StimulusJS. Let's add a Turbo stream action to our posts controller and the corresponding route to our routes file.

# app/controllers/posts_controller.rb
class PostController
  def preview
    require "commonmarker"
    @markdown = Commonmarker.to_html(params[:body])
    render turbo_stream: turbo_stream.update("preview", @markdown)
  end
  # ...
end

# config/routes.rb
Rails.application.routes.draw do
  resources :posts do
    post :preview, on: :collection
  end
  # ...
end

Our Turbo stream targets the preview element. Accordingly, we create a div with the preview ID in the post form. We also add a button that will trigger the Turbo stream action using a Stimulus controller — which we'll create next.

<!-- app/views/posts/_form.html.erb -->
<%= form_with(model: post, class: "contents",
              data: {
                controller: "preview",
                preview_url_value: preview_posts_path
              }) do |form| %>
  <!-- ... -->
  <div>
    <%= form.textarea :body, rows: 22,
                  data: {
                    preview_target: "editorContent",
                  },
                  class: "..."
                  %>
  </div>
  <!-- Add a button to trigger the preview and div to display it -->
  <div class="my-5">
    <button type="button" data-action="click->preview#show" class="...">
      Preview
    </button>
  </div>
  <div id="preview" class="my-5 p-4 border rounded-md bg-white"></div>
</form>

Our Stimulus controller is nothing special. It takes the content of our editor, which we refer to as the editorContentTarget. It then sends a request to the post preview action. The rest is done automagically by Turbo. On a side note, we are using rails/request.js here, which makes it a lot simpler to send Turbo Stream requests — or any other requests for that matter — to your Rails application from JavaScript.

// app/javascript/controllers/preview_controller.js
import { Controller } from "@hotwired/stimulus";
import { post } from "@rails/request.js";

export default class extends Controller {
  static targets = ["editorContent"];
  static values = { url: String };

  show() {
    post(this.urlValue, {
      body: {
        body: this.editorContentTarget.value,
      },
      responseKind: "turbo-stream",
    });
  }
}

In our post form, we can now preview the rendered Markdown at the click of a button. Great success!

Improving the User Experience

Now, there are two more things to address. For one, if you've been playing along, you've probably noticed our HTML doesn't really look right. We are rendering tags such as h1 or table correctly, but the styling is off. Since we use Tailwind, there is an easy way to fix this — the TailwindCSS typography plugin.

/* app/assets/tailwind/application.css */
@import "tailwindcss";
/* Add the typography plugin */
@plugin "@tailwindcss/typography";

What does it do, you ask?

The official Tailwind CSS Typography plugin provides a set of prose classes you can use to add beautiful typographic defaults to any vanilla HTML you don't control, like HTML rendered from Markdown, or pulled from a CMS. Source: Tailwind CSS Typography GitHub README

Sounds perfect for our purpose. To make our HTML look pretty, we just need to add the prose class to our preview pane.

<!-- app/views/posts/_form.html.erb -->
<div id="preview" class="my-5 p-4 ... prose"></div>

Much better. If you are not using Tailwind or the typography plugin, you can still make your HTML look nice, but you'll need to create the necessary CSS by hand.

Now, finally, let's touch up our editor. We want to switch between our editor (the text area where we write our Markdown) and our preview at the click of a button. For that, we'll update our Stimulus controller so that it toggles between these two panes by hiding one and displaying the other. We'll introduce the necessary targets, along with new targets for the buttons (so we can change the style of the active one).

// app/javascript/controllers/preview_controller.js
export default class extends Controller {
  static targets = [
    "editor",
    "editorContent",
    "preview",
    "editorButton",
    "previewButton",
  ];
  static values = { url: String };

  editor(event) {
    event.preventDefault();
    this.#toggleButton(this.editorButtonTarget, this.previewButtonTarget);
    this.#togglePane(this.editorTarget, this.previewTarget);
  }

  preview(event) {
    event.preventDefault();
    post(this.urlValue, {
      body: { body: this.editorContentTarget.value },
      responseKind: "turbo-stream",
    });
    this.#toggleButton(this.previewButtonTarget, this.editorButtonTarget);
    this.#togglePane(this.previewTarget, this.editorTarget);
  }

  #toggleButton(activate, deactivate) {
    // Toggle button styles
    activate.classList.remove("bg-gray-100", "hover:bg-gray-50");
    // ...
  }

  #togglePane(show, hide) {
    // Hide one pane and show the other
    show.classList.remove("hidden");
    hide.classList.add("hidden");
  }
}

We also need to make some changes to our form. We can remove our previous preview button since we no longer need it.

<!-- app/views/posts/_form.html.erb -->
<div class="...">
  <button class="..." data-action="click->preview#editor" data-preview-target="editorButton" type="button">
    Write
  </button>
  <button
    class="..." data-action="click->preview#preview" data-preview-target="previewButton" type="button">
    Preview
  </button>
</div>
<div class="my-5" data-preview-target="editor">
  <%= form.textarea :body,
                rows: 22,
                data: { preview_target: "editorContent" },
                class: "..."
                %>
</div>
<div id="preview" data-preview-target="preview" class="... prose max-w-full" > </div>

And that's all we need to create a magnificent Markdown editor just like the one GitHub has! Well, that's not entirely true. GitHub's editor has a lot of subtle features that make our writing experience more pleasant — we won't add all of them here.

There's one thing we will add, though. What I adore about the GitHub editor is that it lets you add images to your Markdown simply by pasting them into the editor. Let's wrap up this article by adding this advanced feature.

Adding Image Uploads to our Markdown Editor in Ruby on Rails

Remember how we enabled ActiveStorage when we set up our demo application? Well, this is what it's for!

To upload files directly from our editor, we'll be using Rails Direct Uploads. Let's add the necessary JavaScript to our application.

bin/importmap pin @rails/activestorage

Next, we update our trusty preview controller one final time.

import { DirectUpload } from "@rails/activestorage";

export default class extends Controller {
  // Add the upload URL value
  static values = { url: String, uploadUrl: String };

  upload(event) {
    if (!event.clipboardData.files.length) return;

    event.preventDefault();
    Array.from(event.clipboardData.files).forEach((file) =>
      this.#uploadFile(file),
    );
  }

  #uploadFile(file) {
    const upload = new DirectUpload(file, this.uploadUrlValue);
    upload.create((_error, blob) => {
      const url = `/rails/active_storage/blobs/redirect/${
        blob.signed_id
      }/${encodeURIComponent(blob.filename)}`;
      const link = `![${blob.filename}](${url})\n`;
      // Update editor content
      const start = this.editorContentTarget.selectionStart;
      const end = this.editorContentTarget.selectionEnd;
      this.editorContentTarget.setRangeText(link, start, end);
    });
  }
  // ...
}

The upload function will be called whenever a user pastes something in our editor. If the pasted data contains any files, we use the DirectUpload class supplied by ActiveStorage to upload them to our server. Once the upload has finished, we take the resulting blob attributes and convert them into a markdown image link that Commonmarker can convert to HTML. Finally, we use selectionStart and selectionEnd to determine at which position to insert this link.

We now only need to update our HTML to provide the direct upload URL and wire up the paste event to our Stimulus controller. Simple.

<%= form_with(model: post, class: "contents", data:
              { controller: "preview",
                preview_url_value: preview_posts_path,
                <!-- Add the direct uploads URL -->
                preview_upload_url_value: rails_direct_uploads_url
              }) do |form| %>
  <%= form.textarea :body,
                    data: {
                      preview_target: "editorContent",
                      <!-- Wire up the paste event -->
                      action: "paste->preview#upload",
                    },
                    <!-- ... -->
  %>
%>

Now, we did leave some things out. For one, there is no error handling — your image upload may fail, and you should handle that. Also, for a true GitHub-like experience, you'd want to support uploading files via drag-and-drop, as well as allowing users to select files using a file picker. These features work similarly to upload-on-paste — we just left them out in this post for brevity.

Wrap Up

Markdown is arguably the language of the web. Except it isn't a single language, but is comprised of various dialects, the most popular of which are CommonMark and GitHub Flavored Markdown.

In the Ruby ecosystem, Commonmarker is an excellent choice for converting GFM to HTML.

By leveraging the power of Turbo Streams, StimulusJS, and TailwindCSS, we have created a nice-looking Markdown editor that supports live previews and image uploads in no time.

Happy coding!

Completing, Integrating, and Publishing Our Game with DragonRuby

Roy Tomeij — 2025-11-26T00:00:00+00:00

In part one of this series, we started developing a simple Flappy Bird clone game using the DragonRuby game development toolkit. We didn't come very far, though — we stopped after integrating player input to keep our plane afloat.

In this second and concluding part, we'll implement the remaining simple game mechanics. We'll also take a brief look at interfacing with an HTTP server and publishing our game on itch.io.

Scene Management

Let's continue developing the game by adding a condition for terminating it ("game over"). Whenever the plane collides with an obstacle or drops out of the bottom of the screen, the game should abort and give feedback to the player.

For simplicity, let's start with the latter. To keep responsibilities clear in our game code, we'll create a game_over? method that returns true or false (we will later extend it to include arbitrary collisions):

def game_over? args
  args.state.plane_pos.y <= 0 - args.state.plane_pos.h
end

Here, we simply check the plane's y coordinate against the lower boundary of the screen. Additionally, we subtract the sprite's height, so that this check only returns true when the plane has fully disappeared.

When the game is over, we want to display a special screen that just reads "Game Over" (later, we'll add a score). To do this, we have to divide our game logic into two scenes: a :game scene and a :game_over scene. As you might have guessed, we can just make this an additional key to our game state.

In each tick, we will check whether the game is over. If it is, we'll just switch the args.state variable:

  def tick args
+   args.state.scene ||= :game
    args.state.dy ||= 0

    # etc.

+   if game_over?(args)
+     args.state.scene = :game_over
+   end
  end

We've now arrived at a point that warrants a closer look at code organization. Looking at our program right now, it is clear that we can draw a line between code that renders something to the screen and code that calculates changes to the game state. Let's extract both into their own methods (for good measure, I've also added an init method that encapsulates the state initialization code neatly):

GRAVITY = 0.5
FLAP_POWER = 10

def tick args
  init args
  calc args
  render args
end

def init args
  args.state.scene ||= :game
  args.state.dy ||= 0
  args.state.plane_pos ||= { x: 640,
                             y: 360,
                             w: 88,
                             h: 73,
                             anchor_x: 0.5,
                             anchor_y: 0.5 }
end

def render args
  args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
  args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

  args.outputs.sprites << { **args.state.plane_pos,
                            path: "sprites/Planes/planeRed1.png" }
end

def calc args
  args.state.dy -= GRAVITY

  if args.inputs.mouse.down || args.inputs.keyboard.key_down.space
    args.state.dy += FLAP_POWER
  end

  args.state.plane_pos.y += args.state.dy

  args.state.scene = :game_over if game_over?(args)
end

def game_over? args
  args.state.plane_pos.y <= 0 - args.state.plane_pos.h
end

We can now complete our scene management code in the render method. If the current scene is :game, we'll continue painting our plane sprite. If it's :game_over, we'll print a message instead:

def render args
  # rolling background
  args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
  args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

  if args.state.scene == :game
    args.outputs.sprites << { **args.state.plane_pos,
                              path: "sprites/Planes/planeRed1.png" }
  elsif args.state.scene == :game_over
    args.outputs.labels << { x: 640, y: 360, text: "GAME OVER", size_px: 128, anchor_x: 0.5, anchor_y: 0.5 }
  end
end

This completes our minimal scene management implementation. It's not fully complete yet, because the game should reset after a timeout when it's over, but we will leave that out for now. If we run it from our terminal with ./dragonruby, this is what we get:

Collision Detection

Let's now piece together the final component of our game mechanic: obstacles and detecting collisions. We will add a key called walls to our game state, which will hold an array of all currently visible walls. Each wall will have two sprites — one for a rock emerging from below, and one for a stalactite from above. Other than that, we only have to store an x position and update it, as walls move from right to left.

Let's start with preparing the state in init:

  def init args
    args.state.scene ||= :game
    args.state.dy ||= 0
+   args.state.walls ||= []

    # etc.
  end

Next, we'll add some code to calc, to:

update the x position of all visible walls so that they move to the left (in the code below, every wall's x coordinate is diminished by 8)
destroy walls that have moved out to the left. We remove all walls whose x position is smaller than -108 (the wall sprite's width) from the array using reject!.
create new walls from the right. We do this at an interval of 120 frames (that's every two seconds) by adding a new hash to the array with an x position of 1280 (the screen's width).

  def calc args
    args.state.dy -= GRAVITY

    if args.inputs.mouse.down || args.inputs.keyboard.key_down.space
      args.state.dy += FLAP_POWER
    end

    args.state.plane_pos.y += args.state.dy

+   # walls
+   args.state.walls.each { |w| w.x -= 8 }
+   args.state.walls.reject! { |w| w.x < -108 }
+   args.state.walls << {
+     x: 1280
+   } if Kernel.tick_count % 120 == 0

    args.state.scene = :game_over if game_over?(args)
  end

Note that we are using absolute numbers for positions here, just to avoid blowing up the scope of this article. For robust, relative positioning, take a look at the Grid class.

Next, we actually render the walls using more sprites from Kenney's "Tappy Plane" kit:

  def render args
    # ...

    if args.state.scene == :game
      args.outputs.sprites << { **args.state.plane_pos,
                                path: "sprites/Planes/planeRed1.png" }
+     args.state.walls.each do |wall|
+       wall.sprites = [
+         {
+           x: wall.x,
+           y: 0,
+           w: 108,
+           h: 239,
+           path: "sprites/rock.png"
+         },
+         {
+           x: wall.x,
+           y: 720,
+           w: 108,
+           h: 239,
+           anchor_y: 1,
+           path: "sprites/rockDown.png"
+         }
+       ]

+       args.outputs.sprites << args.state.walls.map(&:sprites)
      end
    elsif args.state.scene == :game_over
      # ...
    end
  end

Note that we actually mutate the wall entities stored in the game state by setting a sprites property on them that contains an array of sprite definitions. Thus, every object in args.state.walls contains an array of sprites, which we then use to actually draw the sprites to args.outputs. The reason for this detour will become clear presently.

We will handle the actual collision detection next. Since we have extracted the game_over? method, which is called every tick, this is the logical place to put the code that checks if the plane intersects with an obstacle.

As it turns out, this is pretty straightforward, since DragonRuby provides a couple of convenience methods for collision detection, such as intersect_rect? and any_intersect_rect?.

We can make use of the latter to test whether a collection of objects that respond to x, y, w, and h (such as our walls' sprites) overlaps with another object that also responds to those attributes (such as our plane_pos). All that's needed is a bit of data massaging using flat_map, and this expression returns true whenever our plane intersects with a wall sprite.

def game_over? args
  return true if args.state.plane_pos.y <= 0 - args.state.plane_pos.h

  args.state.walls
    .flat_map { |wall| wall.sprites || [] }
    .any_intersect_rect? args.state.plane_pos
end

With these changes, our game finally behaves as you would expect it to:

Storing and Managing a Score

The final piece of functionality we want to add to the game is tracking a score for our player and displaying it. Again, let's start by adding it to our init method:

  def init args
    args.state.scene ||= :game
+   args.state.score ||= 0
    args.state.dy ||= 0
    args.state.walls ||= []
    args.state.plane_pos ||= { x: 640,
                               y: 360,
                               w: 88,
                               h: 73,
                               anchor_x: 0.5,
                               anchor_y: 0.5 }
  end

Next up, we are going to display it in the top left corner. Let's add that to render:

  def render args
    args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
    args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

+   args.outputs.labels << { x: 10, y: 710, text: "SCORE: #{args.state.score}", size_px: 64 }

    # etc...
  end

To keep things as simple as possible, we are going to increase the score count by 1 every time a wall has moved out of the left screen boundary. The simplest way to do this is to calculate the difference of the wall count before and after their removal in calc. If it has decreased (and the player is still playing, i.e., it's not game over), we increase the score by 1:

  def calc args
    # omitted

    # score
+   wall_count_before = args.state.walls.count
    args.state.walls.reject! { |w| w.x < -108 }
+   if wall_count_before > args.state.walls.count && args.state.scene == :game
+     args.state.score += 1
+   end

    # omitted
  end

Now we can observe how the game score increases with every surpassed wall:

Note: The args.gtk.slowmo! helper method can be really useful when debugging time-critical game mechanics!

Playing Sounds

Before moving on to the final steps (integration and deployment), let's add an audible touch to the game. There are lots of different sounds you might want to add to your game, but in general, they fall into two categories: long-running looping sounds and one-shot sounds.

Let's start with the first category and play a looping engine sound for the plane. In DragonRuby, sound is just another output device and handled with the tick method. However, as sound is time-based, you'll want to only trigger it once, otherwise you'll get hundreds of overlapping samples. That's why we start the engine sound in the init method, like this:

def init args
  # omitted

  if args.state.tick_count == 1
    args.audio[:engine] = {
      input: "sounds/engine.ogg",
      gain: 0.2,
      looping: true
    }
  end
end

Notably, we only start playback at the first tick, and set looping to true. This will ensure that the engine sound keeps playing back, and not overlap.

Let's now turn to some sound effects. First, we are going to add a sound whenever the plane climbs. We already check for input in calc, so we just have to add a one-shot sound:

  def calc args
    # omitted

    if args.inputs.mouse.down || args.inputs.keyboard.key_down.space
      args.state.dy += FLAP_POWER
+     args.outputs.sounds << "sounds/jump.ogg"
    end

    # omitted
  end

As you can see, we just append a sound file path to args.outputs.sounds, which is arguably a very Rubyesque DSL. We can apply the same technique when increasing the score count:

  def calc args
    # omitted

    if wall_count_before > args.state.walls.count && args.state.scene == :game
      args.state.score += 1
+     args.outputs.sounds << "sounds/score.ogg"
    end

    # omitted
  end

Finally, let's add a sound when the plane crashes. For this, we'll have to set a game_over_played boolean flag, because otherwise the sound would start playing at each tick:

  def calc args
    # omitted

    if game_over?(args)
      args.state.scene = :game_over

+     unless args.state.game_over_played
+       args.audio[:engine].paused = true
+       args.outputs.sounds << "sounds/explosion.ogg"
+       args.state.game_over_played = true
+     end
    end
  end

When the game is over, the engine sound is paused and an explosion sound plays. However, to ensure that this happens only once, we set (and subsequently check at every tick) args.state.game_over_played.

Note: If you're wondering where to obtain sounds, it's the same procedure as with sprites. There are bazillions of sound packs available online!

Integration with an HTTP Server to Store and Retrieve High Scores

What would a video game be without a leaderboard? Obviously, the easiest choice would be to keep high scores in a local file. However, DragonRuby also contains a minimal HTTP client that can interact with remote servers.

We are going to use this functionality in the "game over" scene to:

Persist the current score via a POST request.
Retrieve the highest 3 scores via a GET request.

To achieve this, we first need a minimal local web application. We are going to use Roda here to fit the whole app into one file. Install it with gem install roda and then use this config.ru:

require "roda"

$scores = []

class App < Roda
  plugin :json
  plugin :json_parser

  route do |r|
    r.post "score" do
      $scores << r.params["count"]
      $scores.sort.reverse[0..2]
    end
  end
end

run App.freeze.app

If you run this with rackup, it should give you a server running on http://localhost:9292 that you can use to persist the scores. You can POST new scores to /score, which returns the best 3 scores in the response. For demonstration purposes, we are just storing the leaderboard locally, in the global variable called $scores.

First, we need to again add a new variable to the game state, to keep the current state of the leaderboard locally:

  def init args
    args.state.scene ||= :game
    args.state.score ||= 0
+   args.state.hi_scores ||= []
    args.state.dy ||= 0
    args.state.walls ||= []

    # etc.
  end

To send a POST request to the server, DragonRuby has a method called http_post_body, one of the rare cases of asynchronous code in the framework. We have to wait for this "promise" to complete, therefore, we first store it in the game state. Add this to the calc method:

  def calc
    # omitted

    if game_over?(args)
      # omitted

+     payload = "{\"count\": #{args.state.score}}"
+     args.state.post_result ||= args.gtk.http_post_body "http://localhost:9292/score",
+                                                        payload,
+                                                        ["Content-Type: application/json", "Content-Length: #{payload.length}"]
    end
  end

This sends (you have guessed it) the payload as a JSON object ({ "count": YOUR_CURRENT_SCORE }) to the /score route.

Now all that's left is to retrieve the updated high score table once that "promise" has resolved. We do this in the render method and render the leaderboard to the screen if we are in the game_over scene:

  def render
    # omitted

    elsif args.state.scene == :game_over
      args.outputs.labels << { x: 640, y: 520, text: "GAME OVER", size_px: 128, anchor_x: 0.5, anchor_y: 0.5 }

+     if args.state.post_result && args.state.post_result[:complete]
+       if args.state.post_result[:http_response_code] == 200
+         args.state.hi_scores = args.gtk.parse_json args.state.post_result[:response_data]
+
+         args.state.hi_scores.each_with_index do |score, index|
+           args.outputs.labels << { x: 560, y: 400 - index * 90, text: "#{index + 1}. #{score}", size_px: 64 }
+         end
+       end
+     end
    end
  end

After that, we can behold our high score table once the plane has crashed:

We are going to stop here. Just keep in mind that to restart the game, you'd now have to clear the post_result, along with all other relevant game parameters.

Publishing

Now that we've built our first game, we want to tell the world about it! Luckily, DragonRuby comes with built-in support to publish games as HTML (i.e., WASM-based) exports on itch.io, an online marketplace for indie games. Let's walk through the deployment process.

First, create an itch.io account if you don't have one already, and create a new game. Give your game a title, and be sure to select "HTML" as the project type:

Now it's time to fill out the game's metadata for the actual bundled package. Open the file located at mygame/metadata/game_metadata.txt and uncomment the first couple of lines. Then fill it out with your itch.io username, your game's id and title, and add an icon if you want:

devid=my_itch_io_username
devtitle=My Name
gameid=tappy-plane-demo
gametitle=Tappy Plane Demo
version=0.1
icon=metadata/icon.png

Note that devid must match your itch.io username, and gameid must match the project's URL path.

After you've completed this step, it's time to start the actual bundling. Run this command in the terminal:

$ ./dragonruby-publish --package mygame

This creates a builds folder into which DragonRuby compiles binaries of all target platforms:

$ ls builds
tappy-plane-demo-html5-0.1/             tappy-plane-demo-linux-raspberrypi-0.1/
tappy-plane-demo-html5.zip              tappy-plane-demo-linux-raspberrypi.bin*
tappy-plane-demo-linux-amd64-0.1/       tappy-plane-demo-linux-raspberrypi.zip
tappy-plane-demo-linux-amd64.bin*       tappy-plane-demo-mac-0.1/
tappy-plane-demo-linux-amd64.zip        tappy-plane-demo-macos.zip
tappy-plane-demo-linux-arm64-0.1/       tappy-plane-demo-windows-amd64-0.1/
tappy-plane-demo-linux-arm64.bin*       tappy-plane-demo-windows-amd64.exe*
tappy-plane-demo-linux-arm64.zip        tappy-plane-demo-windows-amd64.zip

Depending on your operating system, you can now even go ahead and try the game binary locally. However, we are going to use the HTML export and upload it to itch.io. For this, just edit your game settings and select the tappy-plane-demo-html5.zip file from the builds folder. Make sure to tick the "This file will be played in the browser" checkbox:

Finally, we have to make sure the viewport dimensions are configured properly, and SharedArrayBuffer support is enabled (this is required for DragonRuby's WASM engine to work).

After you have saved these changes, your game should be playable in the browser at the specified URL. Yay!

Deploying to itch.io is only one option, though. DragonRuby helps you deploy to Steam, mobile platforms, and even to Raspberry Pi or virtual reality headsets. For more information, check out the DragonRuby official documentation.

Wrapping Up

In this two-part series, we've taken DragonRuby from a fresh install to a complete, playable game — covering core game architecture, scene management, collisions, scoring, audio, and even online leaderboards and publishing. Along the way, we've seen how DragonRuby’s minimal, Ruby-flavored API makes it possible to build and ship games fast without sacrificing clarity or control.

While our "Tappy Plane" clone is simple by design, it demonstrates the full workflow from idea to release. With these building blocks in place, you can now focus on creative touches — refining gameplay, adding polish, or inventing something entirely new.

Happy coding!

Rendering Samples with Showcase for Ruby on Rails

Roy Tomeij — 2025-11-12T00:00:00+00:00

In parts one and two of this series, we familiarized ourselves with the ins and outs of Showcase.

Now, we'll dive into samples, Showcase's main feature. Samples show how a component can be used in a real application.

Samples in our Ruby App

In our case, we have two samples, one for a small button and one for a large button:

Let's look at how the samples are rendered. We'll add one sample back to our preview file:

<%# test/dummy/app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

<% showcase.sample "Large", description: "This is our larger button" do %>
  <%= render "components/button", content: "Button content", mode: :large %>
<% end %>

If we refresh the page in our browser, it should look like this:

Sample Methods in Showcase for Ruby on Rails

From our understanding of the gem's architecture, the showcase.sample method populates the @samples instance variable of the Showcase::Preview instance, which is then passed to the final view. Let's take a look at the Showcase::Preview#sample method:

# app/models/showcase/preview.rb

class Showcase::Preview
  def sample(name, **options, &block)
    @samples << Showcase::Sample.new(@view_context, name, **options).tap { _1.evaluate(&block) }
  end
end
# => @samples = [Showcase::Sample instance]

And the Showcase::Sample initializer:

# app/models/showcase/sample.rb

class Showcase::Sample
  attr_reader :name, :id, :events, :details
  attr_reader :rendered, :source, :instrumented

  def initialize(view_context, name, description: nil, id: name.parameterize, syntax: :erb, events: nil, **details)
    @view_context = view_context
    @name, @id, @syntax, @details = name, id, syntax, details
    @events = Array(events)
    description description if description
  end

  def description(content = nil, &block)
    @description = content || @view_context.capture(&block) if content || block_given?
    @description
  end
end

There is nothing very surprising in this initializer; most of it is just setting instance variables. Here is the current state of our Showcase::Sample instance:

Showcase::Sample.new(view_context, "Large", description: "This is our larger button")
# => @name         = "Large"
# => @id           = "large"
# => @description  = "This is our larger button"
# => @syntax       = :erb
# => @details      = {}
# => @events       = []
# => @view_context = The view context helper

Now that we have our instance, we can call the evaluate method on it:

# app/models/showcase/sample.rb

class Showcase::Sample
  def evaluate(&block)
    if block.arity.zero?
      consume(&block)
    else
      @view_context.capture(self, &block)
    end
  end
end

About `block.arity`

If you are not familiar with block.arity, it returns the number of arguments that the block expects. If the block does not expect any arguments, block.arity will return 0. In our case, the block is the following:

<% showcase.sample "Large", description: "This is our larger button" do %>
  <%= render "components/button", content: "Button content", mode: :large %>
<% end %>

As we can see, our block does not expect any arguments, so block.arity will return 0. This means that the consume method will be called, with the block as an argument:

# app/models/showcase/sample.rb

class Showcase::Sample
  def consume(&block)
    render(&block)
    extract_source(&block)
  end
end

About the `consume` Method

The consume method calls the render and extract_source methods, passing our block as an argument. Let's have a look at the render method first:

# app/models/showcase/sample.rb

class Showcase::Sample
  def render(&block)
    # TODO: Remove `is_a?` check when Rails 6.1 support is dropped.
    assigns = proc { @instrumented = _1 if _1.is_a?(ActiveSupport::Notifications::Event) }
    ActiveSupport::Notifications.subscribed(assigns, "render_partial.action_view") do
      @rendered = @view_context.capture(&block)
    end
  end
end

The render method uses ActiveSupport::Notifications to subscribe to the render_partial.action_view event. This event is triggered whenever a partial is rendered in Rails. In our case, the block passed to the showcase.sample method will render the components/button partial, so an event will be triggered and the @instrumented instance variable will be set to the ActiveSupport::Notifications::Event published by the rendering of the partial.

This ActiveSupport::Notifications::Event instance responds to the duration and allocation methods. It will be used later to display the duration and allocations of the sample in the view, as shown in the top right corner here:

Calling `extract_source`

Next, the extract_source method is called, with the block as an argument:

# app/models/showcase/sample.rb

class Showcase::Sample
 def extract_source(&block)
    source = extract_source_block_via_matched_indentation_from(*block.source_location)
    @source = @view_context.instance_exec(source, @syntax, &Showcase.sample_renderer)
  end
end

The extract_source method first calls extract_source_block_via_matched_indentation_from with *block.source_location as arguments. If you're not familiar with block.source_location, it returns an array of two elements: the file path and line number where the block was defined. In our case, it will return something like this:

[
  "/path/to/test/dummy/app/views/showcase/previews/components/_button.html.erb",
  4  # The line number where the block was defined
]

The splat operator * before block.source_location is used to unpack the array into two separate arguments: the file path and the line number.

A More Complex Method

Let's have a look at the extract_source_block_via_matched_indentation_from method:

# app/models/showcase/sample.rb

class Showcase::Sample
  def extract_source_block_via_matched_indentation_from(file, source_location_index)
    # `Array`s are zero-indexed, but `source_location` indexes are not, hence `pred`.
    starting_line, *lines = File.readlines(file).slice(source_location_index.pred..)

    indentation = starting_line.match(/^\s+/).to_s
    matcher = /^#{indentation}\S/

    index = lines.index { _1.match?(matcher) }
    lines.slice!(index..) if index
    lines.join.strip_heredoc
  end
end

This is quite a complex method to read, so let's analyze it step by step. File.readlines(file) reads all the lines of the file and returns them as an array. In our case, the result looks like this:

File.readlines(file)
# [
#   "<% showcase.badge :partial, :component %>\n",
#   "<% showcase.description \"Button is our standard element for what to click on\" %>\n",
#   "\n",
#   "<% showcase.sample \"Large\", description: \"This is our larger button\" do %>\n",
#   "  <%= render \"components/button\", content: \"Button content\", mode: :large %>\n",
#   "<% end %>\n"
# ]

The slice(source_location_index.pred..) method call gets lines starting from where the block was defined. The pred method is used to get the previous index, as source_location returns a 1-based index, while Ruby arrays are 0-based:

File.readlines(file).slice(source_location_index.pred..)
# [
#   "<% showcase.sample \"Large\", description: \"This is our larger button\" do %>\n",
#     "  <%= render \"components/button\", content: \"Button content\", mode: :large %>\n",
#   "<% end %>\n"
# ]

If we now view the whole line, the starting_line variable contains the line where the block was defined, and the lines variable contains all the lines after it:

starting_line, *lines = File.readlines(file).slice(source_location_index.pred..)

starting_line
# "<% showcase.sample \"Large\", description: \"This is our larger button\" do %>\n"

lines
# [
#   "  <%= render \"components/button\", content: \"Button content\", mode: :large %>\n",
#   "<% end %>\n"
# ]
# ...

Then, the indentation variable will store the leading whitespace of the starting_line. The regex /^\s+/ matches one or more whitespace characters at the beginning of a string, and to_s converts the match result to a string. In our case, the starting_line doesn't have any leading whitespace, so indentation is an empty string.

indentation = starting_line.match(/^\s+/).to_s
# ""

Then the matcher regex will match lines that start with the same indentation, followed by a non-whitespace character. If our code is properly indented, the index variable will return the index of the block's end keyword:

matcher = /^#{indentation}\S/
index = lines.index { _1.match?(matcher) }
# 1

Finally, lines.slice!(index..) removes all the lines from the index to the end of the array. In our case, it will remove the last line of the block, the end keyword:

lines.slice!(index..) if index
# [
#   "  <%= render \"components/button\", content: \"Button content\", mode: :large %>\n"
# ]

The last step is simply to convert the array of lines back to a string and trim extra white space:

lines.join.strip_heredoc
# "<%= render \"components/button\", content: \"Button content\", mode: :large %>\n"

Phew! That was quite a complicated method, but we managed to get through it by reading carefully, line by line.

Back To `extract_source`

We can now go back to our extract_source method and try to understand the last line of code:

# app/models/showcase/sample.rb

class Showcase::Sample
 def extract_source(&block)
    source = extract_source_block_via_matched_indentation_from(*block.source_location)
    @source = @view_context.instance_exec(source, @syntax, &Showcase.sample_renderer)
  end
end
# source  = "<%= render \"components/button\", content: \"Button content\", mode: :large %>\n"
# @syntax = :erb

Understanding this line requires a solid grasp of the Ruby language. First, we have to know what Showcase.sample_renderer is:

# lib/showcase.rb

module Showcase
  def self.sample_renderer
    @sample_renderer ||=
      begin
        gem "rouge"
        require "rouge"

        formatter = Rouge::Formatters::HTML.new
        @sample_renderer = ->(source, syntax) do
          lexed = Rouge::Lexer.find(syntax).lex(source)
          formatter.format(lexed).html_safe
        end
      rescue LoadError
        proc { _1 }
      end
  end
end

The Rouge Gem in Showcase for Ruby on Rails

By default, Showcase.sample_renderer uses the rouge gem to highlight the syntax of the source code. However, developers using the showcase engine could choose a different syntax highlighter, as the sample_renderer is configurable:

# lib/showcase.rb

module Showcase
  # This line means that the `sample_renderer` can be overridden
  singleton_class.attr_writer :sample_renderer
end

If we remember correctly, the rouge gem is not listed in the showcase.gemspec file, which means that it is required to use the showcase engine:

# lib/showcase.rb

module Showcase
  def self.sample_renderer
    @sample_renderer ||=
      begin
        gem "rouge"
        require "rouge"
        # ...
      rescue LoadError
        proc { _1 }
      end
  end
end

The rescue LoadError idiom means that if rouge is not present in the Gemfile, the sample_renderer will return a simple proc that returns the source code without any syntax highlighting. If it succeeds, it will return a lambda function that takes two arguments: source and syntax, and a highlighted version of the source code.

We can actually test this in the Rails console:

source = "<%= render \"components/button\", content: \"Button content\", mode: :large %>\n"
syntax = :erb
Showcase.sample_renderer.call(source, syntax)
# "<span class=\"cp\">&lt;%=</span> <span class=\"n\">render</span> <span class=\"s2\">\"components/button\"</span><span class=\"p\">,</span> <span class=\"ss\">content: </span><span class=\"s2\">\"Button content\"</span><span class=\"p\">,</span> <span class=\"ss\">mode: :large</span> <span class=\"cp\">%&gt;</span>\n"

The funny cp, n, s2, p, and ss classes are the CSS classes used by Rouge to style the syntax highlighting, as shown in the following picture:

Now that we have a good understanding of the Showcase.sample_renderer, we can go back to the extract_source method and ask ourselves why the last line of code is so complicated:

# app/models/showcase/sample.rb

class Showcase::Sample
 def extract_source(&block)
    source = extract_source_block_via_matched_indentation_from(*block.source_location)
    @source = @view_context.instance_exec(source, @syntax, &Showcase.sample_renderer)
  end
end

In fact, if we make a small experiment and change it to use the Showcase.sample_renderer directly, it will work just as well:

# app/models/showcase/sample.rb
class Showcase::Sample
  def extract_source(&block)
    source = extract_source_block_via_matched_indentation_from(*block.source_location)
    @source = Showcase.sample_renderer.call(source, @syntax)
  end
end

So why does the original code use instance_exec?

As we are in a gem, we need to allow users of the gem to use it in a wide variety of contexts. The instance_exec method allows us to execute the Showcase.sample_renderer lambda in the context of the @view_context. This makes all view helpers available in the lambda, which could be useful if the sample_renderer needed to use any view helper methods, such as sanitize, link_to, or any other method that is available in the view context. However, this is a very niche use case and probably not useful in 99% of situations.

Sample Instance Variables Render the View

We are finally done with our Showcase::Sample instance. Its instance variables now hold all the data necessary to render our view:

# => @name         = "Large"
# => @id           = "large"
# => @description  = "This is our larger button"
# => @source       = "<span class=\"cp\">&lt;%=</span>...<span class=\"cp\">%&gt;</span>\n"
# => @rendered     = "<%= render \"components/button\", content: \"Button content\", mode: :large %>\n"
# => @instrumented = ActiveSupport::Notifications::Event instance

In the view, we can render the "showcase/engine/sample" partial for each of the preview's samples:

<%# app/views/showcase/engine/_preview.html.erb %>

<% if preview.samples.any? %>
  <section>
    <h2 class="...">Samples</h2>
    <%= render partial: "showcase/engine/sample", collection: preview.samples %>
  </section>
<% end %>

If we have a look at the "showcase/engine/sample" partial, it renders a link with the name of the sample:

<%# app/views/showcase/engine/_sample.html.erb %>

<%= link_to sample.name, "##{sample.id}", class: "..." %>

It also renders figures to show the time needed to render the component and its number of allocations:

<%# app/views/showcase/engine/_sample.html.erb %>

<% if event = sample.instrumented %>
  <div class="...">
    <span><%= event.duration.round(1) %>ms</span>
    <span><%= event.allocations %> allocs</span>
  </div>
<% end %>

It renders the actual HTML of the component:

<%# app/views/showcase/engine/_sample.html.erb %>

<% if sample.rendered %>
  <section class="...">
    <%= sample.rendered %>
  </section>
<% end %>

And finally, it renders the source code necessary to render the component:

<%# app/views/showcase/engine/_sample.html.erb %>

<% if sample.source %>
  <details>
    <summary class="...">View Source</summary>

    <section class="...">
      <pre><%= sample.source %></pre>
    </section>
  </details>
<% end %>

We now understand how all values are computed. There is no more magic left for us to uncover in the Showcase gem!

Wrapping Up

In this series on Showcase, we learned how Rails engines work: their main files and how to run them locally. But we also learned much more than that. We broadened our knowledge of Ruby and Rails, learning about block.arity, the view_context helper, ActiveSupport::Notifications, and how to use rouge to highlight code syntax in our Rails applications.

Instead of relying solely on documentation, we can gain a much deeper understanding of Ruby by reading source code. We can even end up adding new features to engines and contributing to open-source, improving our skills and giving back to the community.

Happy coding!

An Introduction to Game Development with DragonRuby

Roy Tomeij — 2025-11-05T00:00:00+00:00

The DragonRuby Game Toolkit is a powerful, cross-platform 2D game engine that allows you to create fun game titles while staying in your favorite developer-friendly language. What's not to love?

In this post, we are going to cover the basics of game development with DragonRuby. We will use a "Flappy Bird" clone to explain the fundamental concepts.

But before we get started, let's address two initial concerns you might have about DragonRuby right off the bat.

Initial Concerns

First of all, DragonRuby is not free. Yes, that's correct, it costs money — at the time of writing, the standard license is a one-time purchase of $48. Given that you get a state-of-the-art 2D graphics engine boxed with integrated publishing/exporting to itch.io and Steam, in my opinion this is more than justified. In addition, there are semi-regular discounts at special occasions like Ruby or gaming conferences and conventions, so watch out for these!

Second, you might have heard that Ruby is "slow". While we can probably agree that this is a relative term in general, it is even more fuzzy in this case. Typically when referring to "Ruby" we mean the CRuby interpreter, which is what you most likely run on the machines that serve your Rails apps. DragonRuby is a whole different runtime though, based on mruby (that is, with an intentionally smaller subset of functionality, but optimized for game development). We'll take a closer look later in this article, but the DragonRuby docs have some videos comparing its performance to other game engines.

Check out DragonRuby's docs page about their overall philosophy and goals.

First Steps: Game Development with DragonRuby

To start out developing a game, download it from the DragonRuby Game Toolkit after you've purchased a standard license. Make sure you select the correct download according to your platform:

Download DragonRuby

Once you unpack the zip file, this is the resulting directory structure:

$ ll
CHANGELOG-CURR.txt
CHANGELOG-PREV.txt
console-logo.png
ctags-emacs
ctags-vim
docs/
dragonruby*
dragonruby-controller.png
dragonruby-httpd*
dragonruby-publish*
dragonruby.png
eula.txt
font.ttf
mygame/
open-source-licenses.txt
README.txt
samples/
VERSION.txt

All you need to get started is the dragonruby executable and the mygame folder, which contains the game logic. Here's a general best practice that might sound counterintuitive at first, but it actually makes great sense: Do not keep this folder in a shared location and refer to it when developing a new game, as this will set you up for trouble.

Instead, always download a fresh copy of the GTK and use the predefined structure. What might seem wasteful actually makes sense, because the moment you package and ship your game, the dragonruby-publish executable expects files to be in certain paths and you'll have to deal with a lot of confusing issues if you rename them. So, even though you might have thought up that very catchy title for your game, it's fine to leave the folder name mygame and just set the game name in the metadata. We'll talk about that later.

If you're curious about the limits of DragonRuby and what functionality and interfacing it provides, check out the samples folder. It contains an abundance of examples and even whole games of different genres.

To round off this section, let's review how DragonRuby interfaces with MRuby, graphics rendering, and hardware I/O. It employs a layered approach where it encapsulates MRuby at the lowest layer, then adds optimizations for its cross-platform targets and the Simple DirectMedia Layer library, with Ruby bindings to get consistent access to low-level resources. The DragonRuby docs have more context on this matter.

A further surprise for a seasoned Ruby developer is that there is no concept of "gems". This is another trait it inherits from MRuby. In fact, there are many third-party libraries, but they are not included via a Gemfile and managed by Bundler. Instead, it's a more manual process requiring you to either clone git repositories manually (e.g., under mygame/lib), or use the built-in download_stb_rb function to download and require external code.

Getting Started with DragonRuby Development

Now let's take our first look at the game. For this, just start the dragonruby executable from a terminal:

$ ./dragonruby
* [Engine] Log started at 2025/8/29 10:26:32
* [Engine] DragonRuby Game Toolkit
* [Engine]   Version: 6.30
* [Engine]   Tier: Standard
* [Engine]   GIT Hash: 4fe7c8c45b2fef5c393a91b5f128c490aa5529ca
* [Engine]   Build Date: Aug 18 2025 19:55:47
* [Engine] Platform: Mac OS X
- [Engine] Process Working Directory: /Users/jrubisch/Documents/_CODE/demo/dragonruby-gtk-macos/dragonruby-macos/
- [Engine] Game Dir: /Users/jrubisch/Documents/_CODE/demo/dragonruby-gtk-macos/dragonruby-macos//mygame
- [Engine] Game ID: hello-SDL
- [Engine] Game Title: Update metadata/game_metadata.txt in your mygame directory to change this title.
- [Engine] Game Version: 1.0
- [Engine] Game Package ID: org.dragonruby.hello-SDL
- [Engine] Game Developer ID: dragonruby
- [Engine] Game Developer Title: DragonRuby LLC

I've shortened the log output a bit to highlight the most salient information: the GTK version and platform, information about our game directory, and metadata. More interestingly, though, a window appears with some educational copy and animation:

DragonRuby start

The Anatomy of a DragonRuby Game

Let's examine this starter demo to get a grasp of the basic building blocks of a DragonRuby game. For this, let's open mygame/main.rb.

def tick args
  args.state.logo_rect ||= { x: 576,
                             y: 200,
                             w: 128,
                             h: 101 }

  args.outputs.labels  << { x: 640,
                            y: 600,
                            text: 'Hello World!',
                            size_px: 30,
                            anchor_x: 0.5,
                            anchor_y: 0.5 }

  ## ... more args wrangling
end

Right off the bat, two things stand out that tell us we're in a whole different Ruby world here:

There's no Ruby class defined. There's not even a main method. Instead, all we get is this tick method.
args seems to be something like a god object.

On a very high level, you can imagine the DragonRuby framework as an infinite loop which calls this tick method 60 times a second — we call this the game loop. Everything you want your game to do has to happen here. The args object is a structure that wraps a good part of the whole DragonRuby API, along with the game state (sprite positions, scores, etc.). Mainly, it covers:

I/O (video, sound, controllers)
Event handling
Capturing and managing game state

We can observe some of this happening here already: args.state is something akin to an OpenStruct where you can add, reference, and modify keys as you go. In this case, we initialize the logo_rect key with screen coordinates and dimensions (x, y, w, h). Note that this key is completely arbitrary. We have to use the conditional assignment operator (||=) here, because, remember, this method is called repeatedly, and we only want to set it in the first iteration.

Next, we append (<<) a hash to args.outputs.labels. You can probably guess that this is what draws "Hello World" to the screen at the specified position.

Drawing and Animating Sprites

A bit further below we find this snippet:

args.outputs.sprites << { x: args.state.logo_rect.x,
                          y: args.state.logo_rect.y,
                          w: args.state.logo_rect.w,
                          h: args.state.logo_rect.h,
                          path: 'dragonruby.png',
                          angle: Kernel.tick_count }

Without exaggeration, this type of interaction is the heart of every 2D computer game. You see, drawing primitives such as lines, points, or rectangles programmatically is far too computationally expensive to do in these 16.67 milliseconds of a game tick. At least as the count of elements in your game starts to grow, you will quickly reach an upper limit of what's possible. That's why pre-drawn graphics known as sprites have been used for this purpose since the invention of video games.

Here, we encounter the simplest case: Just opening an image file (in this case dragonruby.png in the root directory) and rendering it at a specific location with specific dimensions.

This example even goes a bit further by animating it. Notice the Kernel.tick_count method? It reports how many ticks have passed since you started the game, and this will increment by 1 — in every tick, of course. That's why we can use it to rotate our sprite by assigning it to angle.

If you think that you are limited by this technique, you are mistaken: You can go a long way by cleverly organizing and animating your sprites, as we will see later on.

The sprite is positioned at the location designated by args.state.logo_rect. When you are developing a game, it can come in handy to inspect the contents of the state structure. That's where the DragonRuby Console comes into play. While running your game, press ` (the backtick key), and key in args.state.logo_rect.

DragonRuby console

Here, you can inspect and even interact with your game very conveniently.

Checking Inputs and Updating Game State

While the demo game is open, press some arrow keys. You will notice that the rotating logo starts moving:

DragonRuby input

You can find the reason for this in the bottom part of the game:

if args.inputs.keyboard.left
  args.state.logo_rect.x -= 10
elsif args.inputs.keyboard.right
  args.state.logo_rect.x += 10
end

if args.inputs.keyboard.down
  args.state.logo_rect.y -= 10
elsif args.inputs.keyboard.up
  args.state.logo_rect.y += 10
end

We can use args.inputs.keyboard to check whether a certain key has been pressed in the current game tick. left, right, up, down are convenience methods that return true or false depending on the state of the respective arrow keys.

When they are pressed, we update the coordinates of args.state.logo_rect. In the following game tick, this new position will be used to draw the logo sprite, and we've come full circle.

Example Game: A Flappy Bird Clone

To further deepen our understanding of game development, let's build an exciting example game — a ... 🥁 ... Flappy Bird clone.

The first and maybe all too obvious question is: where do we get our sprite images from? In the age of generative AI, the most straightforward answer would probably be to have your favorite chat agent create them. If you're more classically inclined and want to support other indie creators, a safe place to grab some quality assets is Kenney.

As it turns out, they have a pack called Tappy Plane, which contains precisely what we need:

a character (a plane)
obstacles (rocks)
an unobtrusive background

Copy all the sprites into mygame/sprites and we're ready to get going.

Draw and Animate the Background

First, we are going to draw the background, which is just a static sprite. Remove everything inside the tick method and replace it with a simple array concatenation:

def tick args
  args.outputs.sprites << { x: 0, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
end

If you now run dragonruby from the root directory, you'll see the background painted on the canvas.

Now it'd be nice to add some movement to the scene by animating and looping the background.

To achieve this, we just have to adjust our rendering logic a little bit:

def tick args
  args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
  args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
end

As you can see, the first change is that we are now rendering the same sprite twice, albeit with an x offset (the game width — 1280). Then, we use Kernel.tick_count to access how many ticks have passed since the game started, and use it to move the x position to the left and subtract it. The result looks like this:

DragonRuby background

Rendering the Player Sprite

Next, we'll draw the plane. Before we do that, though, we'll instantiate its position in args.state like in the example above. This way, it will "survive" ticks and we can modify the y coordinate by applying gravity and jumping. Once that is done, we can use this position and the sprite image's path to actually display it.

  def tick args
+   args.state.plane_pos ||= { x: 640,
+                              y: 360,
+                              w: 88,
+                              h: 73,
+                              anchor_x: 0.5,
+                              anchor_y: 0.5 }

    args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
    args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

+   args.outputs.sprites << { **args.state.plane_pos,
+                             path: "sprites/Planes/planeRed1.png" }
  end

Note that order is important here: Had we drawn the plane before the background, it would be covered by it and invisible! And voilà, here's our plane hovering in front of the mountains:

DragonRuby plane

Applying Forces

In the final section of this introduction to DragonRuby, we'll touch on game physics. We'll apply gravity to the plane, and listen for input on the keyboard to nudge it upwards. So, first of all, we define GRAVITY as a constant. Then we have to dig up some high school kinetics. We know that gravity is a force that is directed downwards, and that force equals mass times acceleration. So we initialize a new key on args.state called dy (the difference of the y coordinate from one tick to the next) with 0.

Because it's pointing downward, we then subtract GRAVITY from this variable. What this means is that the distance in y direction covered from frame to frame increases — it accelerates. Finally, we add this distance to the plane's current y position, and the result is an avatar that falls out of the game canvas at the bottom:

+ GRAVITY = 0.5

  def tick args
+   args.state.dy ||= 0
    args.state.plane_pos ||= { x: 640,
                               y: 360,
                               w: 88,
                               h: 73,
                               anchor_x: 0.5,
                               anchor_y: 0.5 }

    args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
    args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

+   args.state.dy -= GRAVITY

+   args.state.plane_pos.y += args.state.dy

    args.outputs.sprites << { **args.state.plane_pos,
                              path: "sprites/Planes/planeRed1.png" }
  end

DragonRuby gravity

Checking Inputs

Before we close off this article, let's make sure players have a chance to survive the first few seconds. To do this, we are going to listen for input on the space bar or left mouse button to "flap", i.e., make the plane rise a couple of pixels. To simplify things, we are going to just add a constant value to dy:

  GRAVITY = 0.5
+ FLAP_POWER = 10

  def tick args
    args.state.dy ||= 0
    args.state.plane_pos ||= { x: 640,
                               y: 360,
                               w: 88,
                               h: 73,
                               anchor_x: 0.5,
                               anchor_y: 0.5 }

    args.outputs.sprites << { x: 0 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }
    args.outputs.sprites << { x: 1280 - Kernel.tick_count % 1280, y: 0, w: 1280, h: 720, path: "sprites/background.png" }

    args.state.dy -= GRAVITY

+   if args.inputs.mouse.down || args.inputs.keyboard.key_down.space
+     args.state.dy += FLAP_POWER
+   end

    args.state.plane_pos.y += args.state.dy

    args.outputs.sprites << { **args.state.plane_pos,
                              path: "sprites/Planes/planeRed1.png" }
  end

Now, whenever a player either clicks the mouse or hits the space bar, the value of FLAP_POWER gets added to the y coordinate difference applied in that tick. The result is that with a little practice, you can let the plane hover:

DragonRuby flap

Wrapping Up

In this first part of our two-part series on DragonRuby, we've taken DragonRuby from a fresh download all the way through to building a basic but playable Flappy Bird clone, covering setup, the game loop, state management, rendering sprites, handling inputs, and applying simple physics.

Along the way, we've seen how DragonRuby's unique MRuby-based runtime and procedural architecture keep things both lightweight and powerful, letting you focus on gameplay rather than boilerplate. While our example is intentionally simple, the same principles scale to more complex projects.

In part two, we'll start layering on mechanics, collisions, and scoring to transform this prototype into a real game.

See you next time!

Render a Component Preview In Showcase for Ruby on Rails

Roy Tomeij — 2025-10-15T00:00:00+00:00

In part one of this series, we walked through how to use Showcase in a Rails app.

It's now time to read some Ruby code written by experienced Rails developers. To do this without getting lost, we'll choose one feature of the showcase engine and analyze how it works: rendering a preview of a component.

Let's get started!

About Showcase's Button Component Preview

If we visit http://localhost:3000/docs/showcase/previews/components/button in the browser, we should see the complete documentation for a button component:

According to the showcase documentation, this view is generated by the following code in the test/dummy application:

<%# test/dummy/app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

<% showcase.sample "Basic" do %>
  <%= render "components/button", content: "Button content", mode: :small %>
<% end %>

<% showcase.sample "Large", description: "This is our larger button" do %>
  <%= render "components/button", content: "Button content", mode: :large %>
<% end %>

<% showcase.options do |o| %>
  <% o.required :content, "The content to output as the button text" %>
  <% o.optional :mode, "We support three modes", default: :small, options: %i[ small medium large ] %>
<% end %>

But how does it work? That's what we are going to find out!

Rendering the Title, Badges, and Description

First, we'll modify our preview file to only render the title, badges, and description for now, making things easier to follow. Let's remove everything else from the preview file:

<%# test/dummy/app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

If we refresh the page, we should now see a much simpler page:

To understand how such a view is generated, we can examine our Rails server logs, just as we would in a regular Rails application. If we refresh the button preview page, we should see the following logs in our terminal:

Started GET "/docs/showcase/previews/components/button"
Processing by Showcase::PreviewsController#show as HTML
Parameters: {"id" => "components/button"}

As we can see, the request is routed to the Showcase::PreviewsController#show action, with the parameter id set to "components/button". If we take a look at the routes, we can see that any route starting with previews/ will be routed to the Showcase::PreviewsController#show action:

# config/routes.rb

Showcase::Engine.routes.draw do
  get "previews/*id", to: "previews#show", as: :preview
end

Showcase::PreviewsController#show looks like this:

# app/controllers/showcase/previews_controller.rb

class Showcase::PreviewsController < Showcase::EngineController
  def show
    @preview = Showcase::Path.new(params[:id]).preview_for view_context
  end
end

As we can see, the show action instantiates a new Showcase::Path object with a "components/button" extracted from the params as an argument.

It then calls the preview_for method on this instance, passing the view_context as an argument. The result of this method call is assigned to the @preview instance variable, which will be used later when rendering the view.

About `view_context`

If you are not familiar with view_context, it is a Rails method that provides access to view helpers and other view-related methods. We will use it later to render HTML from models by calling view_context.render.

Let's break down this code. The Showcase::Path initializer looks like this:

# app/models/showcase/path.rb

class Showcase::Path
  attr_reader :id, :segments, :basename

  def initialize(path)
    @id = path.split(".").first.delete_prefix("_").sub(/\/_/, "/")
    @basename = File.basename(@id)
    @segments = File.dirname(@id).split("/")
  end
end

This means that Showcase::Path.new("components/button") will create a new Showcase::Path object with the following attributes:

Showcase::Path.new("components/button")
# => @basename = "button"
# => @id       = "components/button"
# => @segments = ["components"]

The controllers then call the Showcase::Path#preview_for method on this instance, with the view_context as an argument:

# app/models/showcase/path.rb

class Showcase::Path
  def preview_for(view_context)
    Showcase::Preview.new(view_context, id: id, title: basename.titleize).tap(&:render_associated_partial)
  end
end

This method first instantiates a new Showcase::Preview object with the view_context, the id, and the basename as arguments. In our case, the id is "components/button" and the basename is "button".

About the `Showcase::Preview` Initializer

Let's have a look at the Showcase::Preview initializer:

# app/models/showcase/preview.rb

class Showcase::Preview
  attr_reader :id, :badges, :samples

  def initialize(view_context, id:, title: nil)
    @view_context, @id = view_context, id
    @badges, @samples = [], []
    title title
  end

  def title(content = nil)
    @title = content if content
    @title
  end
end

As we can see, the initializer stores those arguments in instance variables. Here is the current state of our Showcase::Preview instance:

Showcase::Preview.new(view_context, id: "components/button", title: "Button")
# => @title        = "Button"
# => @id           = "components/button"
# => @samples      = []
# => @badges       = []
# => @view_context = The view context helper

The Showcase::Preview#render_associated_partial method is then called on the Showcase::Preview instance:

# app/models/showcase/preview.rb

class Showcase::Preview
  def render_associated_partial
    @view_context.render "showcase/previews/#{id}", showcase: self
    nil
  end
end

This method calls the render method on the @view_context. The render method is the one we use every day in our Rails views to render a template or a partial and return some HTML.

In our case, the partial that we want to render is the showcase/previews/components/button partial. The showcase local is passed to the partial, which allows us to access our Showcase::Preview instance in the partial:

<%# test/dummy/app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

In the partial, we call two methods on the showcase local variable. The first method is badge, which adds badges to the preview:

# app/models/showcase/preview.rb

class Showcase::Preview
  def badge(*badges)
    @badges.concat badges
  end
end

# => @badges = [:partial, :component]

The second method is description, which sets the preview description:

# app/models/showcase/preview.rb

class Showcase::Preview
  def description(content = nil, &block)
    @description = content || @view_context.capture(&block) if content || block_given?
    @description
  end
end

# => @description = "Button is our standard element for what to click on"

All of this code exists just to populate the @badges, @description, and @title instance variables of our Showcase::Preview instance.

This instance is returned by the Showcase::Path#preview_for method and assigned to the @preview instance variable in the Showcase::PreviewsController#show action:

# app/controllers/showcase/previews_controller.rb

class Showcase::PreviewsController < Showcase::EngineController
  def show
    @preview = Showcase::Path.new(params[:id]).preview_for view_context
  end
end

Now, just like any controller, the Showcase::PreviewsController will render the corresponding view. By convention, the view should be located at app/views/showcase/previews/show.html.erb. To my surprise, it is located at app/views/showcase/engine/show.html.erb, which isn't very conventional. However, it works because the Showcase::PreviewsController inherits from Showcase::EngineController.

If we have a look at the view, it contains the following code:

<%# app/views/showcase/engine/show.html.erb %>

<%= render "showcase/engine/preview", preview: @preview %>

This view renders a partial called showcase/engine/preview and passes our @preview instance variable.

If we have a look at this view, we can see that it will render the title, badges, and description of the preview in between the Tailwind CSS classes:

<%# app/views/showcase/engine/_preview.html.erb %>

<% if preview.title %>
  <div class="...">
    <h2 class="..."><%= preview.title %></h2>

    <% preview.badges.each do |badge| %>
      <span class="..."><%= badge.to_s.titleize %></span>
    <% end %>
  </div>
<% end %>

<% if preview.description %>
  <div class="..."><%= preview.description %></div>
<% end %>

If we take a step back and reflect on the code architecture, it makes a lot of sense. The Showcase::Preview model is responsible for storing the preview data, such as the title, badges, and description. The trick here is that this data is populated while rendering a preview template:

<%# test/dummy/app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

As users of the showcase engine, it feels like we are writing a view, but in reality, we are populating a model that will be used to render the final view. And look at how clean the interface is! We experience a great developer experience when using this gem!

Next Up: Rendering Samples using Showcase for Ruby on Rails

In this post, we've seen how the title, badges, and description of a preview are rendered in Showcase.

Now that we have a good understanding of how showcase works, we can get our hands dirty with something a bit more meaty in the third and final part of our series: rendering samples!

See you next time!

How to Read Code from the Showcase Ruby on Rails Engine

Roy Tomeij — 2025-10-01T00:00:00+00:00

Reading a lot of code from very senior engineers is probably one of the best ways to level up as a Ruby on Rails developer. By doing so, we can learn new tips and techniques that we can reuse in our jobs. Thanks to open source, we can read code written by the best developers from all over the world, and for free!

However, reading code from a Ruby gem or a Rails engine for the first time without being guided can be daunting. There are so many files; how do we even know where to start?

In this three-part series, we are going to read the source code from the Showcase Rails engine.

We will learn about:

The main files in a Rails engine
How to read source code without getting lost

In this first part, we'll set up Showcase in a Rails application and run it locally.

Let’s get started!

What is Showcase for Ruby on Rails?

Showcase (written mainly by Kasper Timm Hansen, Rails consultant and former Rails Core team member) is a Rails engine that allows us to document a design system. Here is what it looks like:

For a good understanding of how Showcase works before reading its code, we should use it in a project. Let's create a new Rails application and cd into it:

rails new showcase-test
cd showcase-test

According to the Showcase README, we need to add showcase-rails and rouge to our Gemfile:

# Gemfile

group :development, :test do
  gem "showcase-rails"
  gem "rouge", require: false
end

If you are not familiar with the rouge gem, it is a syntax highlighter used to highlight code in the "View source" accordion, as shown below:

As we add new gems to our Gemfile, let's not forget to run bundle install. We then have to mount the engine in our routes to make the engine's routes available to our main Rails application:

# config/routes.rb

Rails.application.routes.draw do
  mount Showcase::Engine, at: "/docs/showcase" if defined?(Showcase::Engine)
  # ...
end

Build a Component

Let’s now create our first component. Every application needs buttons, right? Let’s create one with three different sizes:

<%# app/views/components/_button.html.erb %>

<% classes = { small: "btn--sm", medium: "btn--md", large: "btn--lg" }.fetch(mode) %>
<%= tag.button content, class: [ "btn", classes ] %>

To style our button, we will need CSS, of course. As this is just a demo application, we'll simply write our CSS in the application.css file generated for us:

/* app/assets/stylesheets/application.css */

.btn {
  font-size: var(--btn-font-size);
  padding: 0.5em 1em;
  border-radius: 0.5em;
  color: white;
  background-color: black;
}

.btn--sm {
  --btn-font-size: 0.75rem;
}

.btn--md {
  --btn-font-size: 1rem;
}

.btn--lg {
  --btn-font-size: 1.25rem;
}

Finally, let’s create the preview file to document our button component:

<%# app/views/showcase/previews/components/_button.html.erb %>

<% showcase.badge :partial, :component %>
<% showcase.description "Button is our standard element for what to click on" %>

<% showcase.sample "Small" do %>
  <%= render "components/button", content: "Button content", mode: :small %>
<% end %>

<% showcase.sample "Large", description: "This is our larger button" do %>
  <%= render "components/button", content: "Button content", mode: :large %>
<% end %>

<% showcase.options do |o| %>
  <% o.required :content, "The content to output as the button text" %>
  <% o.optional :mode, "We support three modes", options: %i[ small medium large ] %>
<% end %>

We can now navigate to http://localhost:3000/docs/showcase/previews/components/button to see our first preview:

Gems and engines allow us to use other people's code in our Rails applications. If we use those gems and have absolutely no idea how they work, we have an opportunity to learn something new.

Setting Up Our Ruby on Rails Development Environment

Let's learn about the magic behind gems by reading some source code!

Rails Engines File Structure

The first step to read or contribute to a Rails engine is to clone the repository locally:

git clone git@github.com:bullet-train-co/showcase.git
cd showcase

Let’s open the repository with our text editor and have a look at the different files:

.
├── Gemfile
├── showcase.gemspec
├── app
│   ├── assets
│   ├── controllers
│   ├── helpers
│   ├── jobs
│   ├── mailers
│   ├── models
│   └── views
├── config
│   └── routes.rb
├── lib
│   ├── showcase
│   ├── showcase-rails.rb
│   └── showcase.rb
└── test
    ├── controllers
    ├── dummy
    ├── fixtures
    ├── helpers
    ├── integration
    ├── mailers
    ├── models
    └── views

The first thing we should realize is that the file structure is very similar to a standard Rails application with models, views, controllers, and routes. In fact, we can view a Rails engine as a standalone Rails application that will get merged into our main Rails application!

Our engine also has a /test folder used to test the engine, with a special /dummy folder that is particularly useful. If we have a look at the file structure of the /test/dummy folder, we can see the following tree:

.
├── Rakefile
├── app
│   ├── assets
│   ├── channels
│   ├── controllers
│   ├── helpers
│   ├── jobs
│   ├── mailers
│   ├── models
│   └── views
├── bin
│   ├── rails
│   ├── rake
│   └── setup
├── config
│   ├── application.rb
│   ├── boot.rb
│   ├── cable.yml
│   ├── database.yml
│   ├── environment.rb
│   ├── environments
│   ├── initializers
│   ├── locales
│   ├── puma.rb
│   ├── routes.rb
│   └── storage.yml
├── config.ru
├── lib
│   └── assets
├── log
└── public
    ├── 404.html
    ├── 422.html
    ├── 500.html
    └── favicon.ico

Wait a minute, isn't this the exact file structure of a standard Rails application? Yes, it is! The test/dummy folder contains a real Rails application that we can launch in the browser for development. It is also useful to write tests for our engine in the context of a real Rails application.

Running the Rails Engine Locally

Now that we have a better understanding of the file structure, it's useful to be able to run this test/dummy Rails application on localhost.

As with any other Rails application, the first step is to install dependencies. Before we do so, we have to talk about the showcase.gemspec and the Gemfile. If we open the showcase.gemspec, it currently contains the following code:

# showcase.gemspec

require_relative "lib/showcase/version"

Gem::Specification.new do |spec|
  spec.name        = "showcase-rails"
  spec.version     = Showcase::VERSION
  spec.authors     = ["Daniel Pence", "Kasper Timm Hansen"]
  spec.email       = ["hey@kaspth.com"]
  spec.homepage    = "https://github.com/kaspth/showcase"
  spec.summary     = "Showcase helps you show off and document your partials, components, view helpers and Stimulus controllers."
  spec.license     = "MIT"

  spec.metadata["homepage_uri"]    = spec.homepage
  spec.metadata["source_code_uri"] = spec.homepage
  spec.metadata["changelog_uri"]   = "#{spec.homepage}/blob/main/CHANGELOG.md"

  spec.files = Dir.chdir(File.expand_path(__dir__)) do
    Dir["{app,config,db,lib}/**/*", "MIT-LICENSE", "Rakefile", "README.md"]
  end

  spec.add_dependency "rails", ">= 6.1.0"
  spec.add_development_dependency "tailwindcss-rails"
end

Most of it is just information about the authors, the license, and useful links that you'll find on the RubyGems website. However, the most interesting part is the last two lines:

# showcase.gemspec

spec.add_dependency "rails", ">= 6.1.0"
spec.add_development_dependency "tailwindcss-rails"

The first line indicates that showcase depends on Rails version 6.1 or above.

The second line indicates a development dependency on tailwindcss-rails. This means that showcase uses tailwindcss-rails for styling purposes in development, but you do not need to have tailwindcss-rails installed in your main Rails application to use showcase.

Let’s now have a look at the Gemfile:

# Gemfile

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

# Specify your gem's dependencies in showcase.gemspec.
gemspec

gem "sqlite3"

rails_version = ENV.fetch("RAILS_VERSION", "7.0")

rails_constraint = if rails_version == "main"
  {github: "rails/rails"}
else
  "~> #{rails_version}.0"
end

gem "rails", rails_constraint
gem "sprockets-rails"

# Start debugger with binding.b [https://github.com/ruby/debug]
# gem "debug", ">= 1.0.0"

gem "puma"
gem "rouge", require: false

group :test do
  # Code omitted for brevity
end

The Gemfile should look familiar: It lists additional development dependencies used to run the /test/dummy application.

Those dependencies are from the gemspec, plus the additional ones listed in our Gemfile. This Gemfile also enables us to choose the Rails version we want to use in development by passing the RAILS_VERSION environment variable, and defaults to version 7.0.

Note: While writing this article, I realized there is a small bug when running Showcase with Rails 7.0.X. It is not a very interesting bug to talk about, so we'll just ignore it and use Rails 8.X version instead.

As specified in the Gemfile, we can use the RAILS_VERSION environment variable to specify the Rails version we want to use. To do this, we simply have to prefix all of our commands with RAILS_VERSION=8:

RAILS_VERSION=8 bundle install
RAILS_VERSION=8 bin/rails server
RAILS_VERSION=8 bin/rails console

Let's try to install the dependencies now:

RAILS_VERSION=8 bundle install

With all the dependencies installed, we should now be able to run the Rails server:

RAILS_VERSION=8 bin/rails server

If we now visit http://localhost:3000, we should see the showcase landing page from the test/dummy application:

We should also be able to see our Rails server logs in the terminal:

Started GET "/docs/showcase" for ::1 at 2025-07-24 11:57:42 +0200
Processing by Showcase::EngineController#index as HTML
...

This is going to be very helpful when analyzing how Showcase works, as we'll simply be able to follow the logs!

Next Up: The Button Component Preview

In this post, we learned:

How to use showcase in a real Rails application
What the different files in a Rails engine are used for
How to run the test/dummy Rails application locally

In the next part of this series, we will read some Ruby code written by experienced Rails developers. In order to do this without getting lost, we are going to choose one feature of the showcase engine — the button component preview — and analyze how it works.

Until then, happy coding!

Extend ActiveStorage for Ruby on Rails with Custom Previewers

Roy Tomeij — 2025-08-13T00:00:00+00:00

In part one of this series, we looked at how to extend the ActiveStorage ingest process with custom analyzers.

In this post, we will reverse the procedure and explore how to utilize ActiveStorage previewers to display data.

What Are ActiveStorage Previewers for Ruby on Rails?

Not every uploaded blob is an image. Nonetheless, some non-image blobs can be converted into an image preview. ActiveStorage itself provides built-in previewers for videos and PDFs (via MuPDF or Poppler).

Displaying a preview in your ERB template works exactly the same as creating variants of an image:

<%= image_tag song.recording.preview(resize_to_fill: [640, 160]) %>

Here, we lazily create and display a song preview, taken from the previous article's example use cases. Observe that preview takes the same arguments as variant, which you would use to transform an image.

The Skeleton of an ActiveStorage Previewer

To understand how a previewer is constructed, let's look at an example. Before we dive deep into the code, though, let's examine which previewers are available in the Rails console:

(dev)> ActiveStorage.previewers
=>
[ActiveStorage::Previewer::PopplerPDFPreviewer,
  ActiveStorage::Previewer::MuPDFPreviewer,
  ActiveStorage::Previewer::VideoPreviewer]

These are the two flavors of PDF previewers (depending on the present backend), as well as the video previewer mentioned in the introduction. We are going to study the VideoPreviewer to get a better grasp of what is required to come up with such a solution.

As of today, the implementation looks like this (I've omitted some helper code for clarity):

module ActiveStorage
  class Previewer::VideoPreviewer < Previewer
    class << self
      def accept?(blob)
        blob.video? && ffmpeg_exists?
      end

      def ffmpeg_exists?
        # somehow check if ffmpeg is present
      end
    end

    def preview(**options)
      download_blob_to_tempfile do |input|
        draw_relevant_frame_from input do |output|
          yield io: output, filename: "#{blob.filename.base}.jpg", content_type: "image/jpeg", **options
        end
      end
    end

    private
      def draw_relevant_frame_from(file, &block)
        # invoke a ffmpeg command to extract the first image from the video
      end
  end
end

If you get a feeling of déjà vu, that's understandable, because we encountered a very similar pattern in our previous post. Let's inspect the two important parts:

An accept? method checks whether the previewer is applicable. In this case, it is, if the MIME type is correct and the tooling to transform video (aka ffmpeg) exists on the machine.
The preview method does all the heavy lifting. Specifically, it's used to draw a preview image and store this as a new blob. In the above case, it just uses the first still video image as the preview.

Note: download_blob_to_tempfile is a helper method defined in the ActiveStorage::Previewer base class.

A Use Case for a Custom Previewer

Building on our example from the previous article, we will build a custom previewer for audio waveforms. Let's get started!

Audio data is usually displayed using an abstract waveform or envelope representation (where each bar stands for the loudness at a certain point in time). There are other forms, like spectrograms, but for this demonstration, let's stick with a simple example.

First of all, we are going to produce previews in the PNG format. We'll add ChunkyPNG, a library that helps with PNG manipulation, to our application:

$ bundle add chunky_png

Now, let's start with a minimal WaveformPreviewer scaffold:

# lib/active_storage/waveform_previewer.rb

require "chunky_png"

module ActiveStorage
  class WaveformPreviewer < ActiveStorage::Previewer
    class << self
      def accept?(blob)
        blob.audio? && blob.metadata[:waveform].present?
      end
    end

    def preview(**options)
      waveform = blob.metadata.fetch(:waveform, [])

      # somehow transform the stored waveform into a PNG image
    end
  end
end

The accept? class method checks if we're dealing with an audio blob. Additionally, we check whether waveform data has been stored in metadata, as we did in the previous article. In the preview method, we will have to implement a way of painting an image representing that waveform and provide it as an attachable IO stream. Let's go and do this now.

# lib/active_storage/waveform_previewer.rb

require "chunky_png"

module ActiveStorage
  class WaveformPreviewer < ActiveStorage::Previewer
    # class methods omitted

    def preview(**options)
      waveform = blob.metadata.fetch(:waveform, [])

      width = 640
      height = 240
      center = height / 2

      png = ChunkyPNG::Image.new(width, height, ChunkyPNG::Color::WHITE)

      frame_width = waveform.size / width

      waveform.each_with_index do |v, idx|
        next unless idx % frame_width == 0

        x = idx / frame_width

        y_val = (v * center).to_i
        y1 = center - y_val
        y2 = center + y_val

        png.rect(x, y1, x, y2, 0x4b0082ff, 0x4b0082ff)
      end

      # Write to in-memory buffer
      io = StringIO.new
      png.write(io)
      io.rewind

      yield io: io, filename: "#{blob.filename.base}.png", content_type: "image/png", **options
    end
  end
end

To simplify things, we assume a fixed width and height (640 by 240 pixels). We store a new instance of ChunkyPNG::Image with a white background in the png variable. Now, we have to iterate over each of the datapoints in the waveform and decide whether to draw a line for it or not. Presumably, our stored waveform has more than 640 datapoints, so we calculate 640 frames containing several datapoints. The frame_width is determined by dividing the waveform's size by 640.

When drawing the waveform, we can skip every index that is not an integer multiple of the frame width. For the ones that match, we draw a vertically centered rectangle (a line, really, since it's only one pixel wide) whose height is proportional to the value v.

Finally, we have to wrap our composed PNG image in an in-memory buffer so it's ready for ActiveStorage to be attached. That's what the yield call does: it simply creates a new attachment that's tied to the blob as a preview image.

The result looks something like this:

Beautiful, isn't it? I have one adjustment to propose, though. Drawing a waveform true to size like this looks very technical and might fit an audio editor. But if we consider a simplified version for a social media preview, we can tweak it a bit.

To make it more visually appealing, we are going to do two things:

Make the bars 5 pixels wide (that's why we have to multiply our frame size by 5).
Display only every other frame.

We only need to make a few adaptations to the drawing logic:

# produce bars 5 pixels apart
frame_width = (5 * waveform.size / width).floor

waveform.each_with_index do |v, idx|
  next unless idx % frame_width == 0

  frame_even = (idx / frame_width).even?

  x = idx * width / waveform.size

  y_val = (v * center).to_i
  y1 = center - y_val
  y2 = center + y_val

  png.rect(x, y1, x+6, y2, 0x4b0082ff, 0x4b0082ff) unless frame_even # we skip every other frame for a nice visual effect, and make the bar a bit wider
end

In addition to making the frame width bigger, we have to decide if the frame index is even or odd. We do that by dividing the index by the frame width, and if it's odd, we simply skip drawing the rectangle. To compensate for the diminished width, we increase the width by 6 pixels, so there's a bit of an overlap.

The result is a nice interleaved graph of approximate sound loudness:

To actually activate it, we have to prepend it to the list of previewers in our active_storage initializer:

# config/initializers/active_storage.rb

require_relative "../../lib/active_storage/waveform_analyzer.rb"
require_relative "../../lib/active_storage/waveform_previewer.rb"

Rails.application.config.active_storage.analyzers.prepend ActiveStorage::WaveformAnalyzer
Rails.application.config.active_storage.previewers.prepend ActiveStorage::WaveformPreviewer

Let's now look at a second use case: blurhash image previews.

Blurhash Image Previews

Full disclosure: when writing this article, I assumed that what follows below would also be attainable using a previewer. And it is, however, it also interferes with the normal variant processing logic for images. This makes it unfeasible for a vanilla Rails application.

To understand what's going on, let's look at the image transformation logic employed by ActiveStorage:

When applying transformations (e.g., blob.variant(...)), Rails always uses ActiveStorage::Blob#representation.
This method prefers to create a preview when possible, so any attempt to create a variation will be rerouted to a previewer that returns true on self.accept?(blob).
The only accessible parameter in this method is the blob itself. It is not possible to distinguish whether the developer meant to generate a preview or a variant using an option applied to the representation generation logic.

This limits the usability of ActiveStorage previewers for our use case: preparing a blurhash preview as a placeholder for lazily loading an image. In fact, every call to variant would be overridden by the previewer, so we could not produce scaled down thumbnails, for example.

That doesn't mean that we cannot obtain a similar result though: we just have to use different semantics. We will be able to achieve this using a new, as yet unreleased feature which will be part of Rails 8.1 (introduced in this pull request) — a custom image transformer. Let's try and put this together!

We'll line out our desired API first. We would like to be able to compute a variant from a blurhash:

<%= image_tag image.variant(blurhash: image.blob.metadata["blurhash"]) %>

Rails will now complain because we have just made up the blurhash: ... image processing method, but that is expected:

ActiveStorage::Transformers::ImageProcessingTransformer::UnsupportedImageProcessingMethod (One or more of the provided transformation methods is not supported.)

What we have to do, therefore, is subclass an image processing backend and adapt it for blurhash image calculation. In our case, we can reuse the ImageMagick transformer and insert a conditional that generates our blurhash image if it is present in the transformations hash:

# lib/active_storage/transformers/image_magick_with_blurhash.rb

require "blurhash"
require "chunky_png"

module ActiveStorage
  module Transformers
    class ImageMagickWithBlurhash < ImageMagick
      private

      def process(file, format:)
        if transformations[:blurhash]
          generate_blurhash_png(transformations[:blurhash])
        else
          super(file, format:)
        end
      end

      def generate_blurhash_png(blurhash)
        raise ArgumentError, "Missing blurhash metadata" unless blurhash

        width = 600
        height = 400

        pixels = decode_blurhash(blurhash, width, height)

        png = ChunkyPNG::Image.new(width, height)
        pixels.each_with_index do |(r, g, b), i|
          x = i % width
          y = i / width
          png[x, y] = ChunkyPNG::Color.rgb(r, g, b)
        end

        tempfile = Tempfile.new(["blurhash", ".png"], binmode: true)
        png.write(tempfile)
        tempfile.rewind

        tempfile
      end

      # decode_blurhash method omitted
    end
  end
end

We decode the blurhash, use ChunkyPNG to create an image again, and fill it with the respective pixels. We then need to store it in a temp file so that the ActiveStorage variant processor can pick it up.

I'll save you the dire details of how to produce a PNG from a blurhash, but if you're interested, you can look at this example repository.

That's about it for the image processing part. What's left to do is to register it:

# config/initializers/active_storage.rb

require_relative "../../lib/active_storage/blurhash_analyzer.rb"
require_relative "../../lib/active_storage/transformers/image_magick_with_blurhash.rb"

Rails.application.config.active_storage.analyzers.prepend ActiveStorage::BlurhashAnalyzer

ActiveSupport.on_load(:active_storage_blob) do
  ActiveStorage.variant_transformer = ActiveStorage::Transformers::ImageMagickWithBlurhash
end

Because the default transformer is lazily loaded, this time we have to register it in a load hook, ActiveSupport.on_load(:active_storage_blob). Once that is done, we can give it a go — for example, in the _post partial:

<!-- app/views/posts/_post.html.erb -->

<div id="<%= dom_id post %>">
  <p>
    <strong>Title:</strong>
    <%= post.title %>
  </p>

  <p>
    <strong>Body:</strong>
    <%= post.body %>
  </p>

  <% post.images.each do |image| %>
    <%= image_tag image.variant(resize_to_fit: [600, 400]), width: 600, height: 400 %>
    <%= image_tag image.variant(blurhash: image.blob.metadata["blurhash"]) %>
  <% end %>
</div>

Now we get a scaled-down variant of the original, side by side with the calculated blurhash image:

Writing the necessary JavaScript to swap out the blurhash preview for the real image is left as homework for the reader.

Important note: We are in Rails alpha territory here. While it's pretty certain that this functionality will land in Rails 8.1, there might be breaking changes.

Wrap Up

In this post, we looked at how to make any content stored in an ActiveStorage blob previewable using a custom class. This comes in handy to graphically represent arbitrary media content like audio waveforms, especially when preparing social media previews like open graph images or X (formerly Twitter) cards.

We extended ActiveStorage with a custom waveform previewer based on precomputed audio intensity data. Then we created our own variant transformer capable of converting blurhash data into a downloadable PNG file.

During this deep dive, we learned a lot about ActiveStorage's internals and the interfaces it provides to craft a customized experience. What other analyzers, previewers, and transformers for your media files can you think of?

Happy coding!

Build Custom ActiveStorage Analyzers for Ruby on Rails

Roy Tomeij — 2025-07-30T00:00:00+00:00

In this series, we will take a close look at the architecture of ActiveStorage for Rails.

In this first part, we will examine how ActiveStorage treats uploaded data and how to extend this process. The second part will explore how to augment the presentation of uploaded assets.

But first, let's quickly define what ActiveStorage does.

What Is ActiveStorage for Ruby on Rails?

Without recounting the entire ActiveStorage documentation, in a nutshell, ActiveStorage is an adapter to various forms of storing (mostly) user-generated files in your Ruby on Rails application in a straightforward way. The available storage backends can be divided into:

Local disk storage
Diverse flavors of cloud storage (most prominently Amazon S3 and compatible object storage providers)

Note: For the sake of completeness, there are also adapters to directly store binary data in your database, like active_storage-postgresql.

ActiveStorage allows you to transparently attach files to database records for easy access. You have probably already come across this API:

class User < ApplicationRecord
  has_one_attached :avatar
end

class Post < ApplicationRecord
  has_many_attached :images
end

Here, the class methods has_one_attached and has_many_attached are responsible for wiring up your User or Post records with the respective attachments. But how does this magic work?

Under the hood, ActiveStorage uses two database tables generated when you install it. Here are their entries from the database schema definition:

create_table "active_storage_attachments", force: :cascade do |t|
  t.string "name", null: false
  t.string "record_type", null: false
  t.bigint "record_id", null: false
  t.bigint "blob_id", null: false
  t.datetime "created_at", null: false
  t.index ["blob_id"], name: "index_active_storage_attachments_on_blob_id"
  t.index ["record_type", "record_id", "name", "blob_id"], name: "index_active_storage_attachments_uniqueness", unique: true
end

create_table "active_storage_blobs", force: :cascade do |t|
  t.string "key", null: false
  t.string "filename", null: false
  t.string "content_type"
  t.text "metadata"
  t.string "service_name", null: false
  t.bigint "byte_size", null: false
  t.string "checksum"
  t.datetime "created_at", null: false
  t.index ["key"], name: "index_active_storage_blobs_on_key", unique: true
end

The first table belongs to the join model ActiveStorage::Attachment. It references a polymorphic record (that's why the methods has_(one|many)_attached can be called from any ActiveRecord model) as well as a Blob. As you might correctly assume, this refers to the second table, which is mapped by the ActiveStorage::Blob model. This acts as a data container for all that is needed to upload or download a file to one of the configured services. Let's take it apart a bit:

The key attribute refers to how the blob is called at its actual storage location (in most cases, an S3 bucket).
filename is the original name of the file under which it was uploaded.
content_type is its analyzed MIME type.
metadata is a generic text column that can hold any metadata of the file, in JSON format. It's this column that our custom analyzers will make use of.
service_name is used to identify the service in config/storage.yml to which this file was uploaded.
byte_size and checksum are exactly what you'd expect these attributes to store.

How Does ActiveStorage Treat Uploaded Data?

With these general concepts out of the way, let's examine the ingest process used by ActiveStorage. In other words, once a file is uploaded, what happens next?

The answer lies in the Attachment model's code. In an after_create_commit callback, it enqueues a job to analyze the blob it pertains to later.

The code that's used to perform the analysis reads simple enough:

def analyze
  update! metadata: metadata.merge(extract_metadata_via_analyzer)
end

# ...

private
  def extract_metadata_via_analyzer
    analyzer.metadata.merge(analyzed: true)
  end

  def analyzer
    analyzer_class.new(self)
  end

  def analyzer_class
    ActiveStorage.analyzers.detect { |klass| klass.accept?(self) } || ActiveStorage::Analyzer::NullAnalyzer
  end

We observe that, as indicated before, the analyze method updates the metadata column of the blob. This metadata is extracted by an analyzer, but there's an important detail hidden in how exactly this analyzer is provided. The analyzer_class is retrieved from a list of analyzers that the ActiveStorage module itself keeps. Let's take a brief look at it in the Rails console:

(dev)> ActiveStorage.analyzers
=>
[ActiveStorage::Analyzer::ImageAnalyzer::Vips,
  ActiveStorage::Analyzer::ImageAnalyzer::ImageMagick,
  ActiveStorage::Analyzer::VideoAnalyzer,
  ActiveStorage::Analyzer::AudioAnalyzer]

Listed here are all the standard analyzers that ship with ActiveStorage: two for images (because of the two processing backends, Vips and ImageMagick), one for video, and one for audio data. From this list, the first analyzer that responds to accept?(self) with true is selected. So, for example, ImageAnalyzer checks whether a blob holds an image, and so on. Creating new analyzers, then, only needs a class that extends Analyzer and is prepended to this list. We'll explore how to do this in the remainder of this article.

Two Use Cases for Custom Analyzers

When would you reach for a custom analyzer? I contend that most use cases revolve around needs for enhanced presentation of assets. To back up that claim, I have prepared two prototypical examples:

precomputing audio waveform data
calculating image blurhashes

Extracting and Storing Audio Sample Data

Let's assume that we are building a directory of songs. We might use a Song model, which has an attached recording for this:

class Song < ApplicationRecord
  has_one_attached :recording
end

If we create a new record of this model and attach an audio file, what happens? In a full-stack scenario, we would use an upload form, but for the sake of investigating the analysis process, let's just create one from the Rails console:

(dev)> song = Song.create(title: "Ruby Blues in D Flat")
(dev)> song.recording.attach(io: File.open("/path/to/file"), filename: "ruby_blues.wav")

Apart from the usual SQL log, Rails also informs us that it has enqueued a job to process the data:

Enqueued ActiveStorage::AnalyzeJob (Job ID: 2a923033-b0d2-4bfc-b368-d3eb1344e64b) to Async(default) with arguments: #<GlobalID:0x000000012292f428 @uri=#<URI::GID gid://active-storage-analyzers-previews/ActiveStorage::Blob/1>>

Let's now inspect the respective Attachment and Blob records:

(dev)> song.recording_attachment
=>
#<ActiveStorage::Attachment:0x00000001240c8eb8
  id: 1,
  name: "recording",
  record_type: "Song",
  record_id: 1,
  blob_id: 1,
  created_at: "2025-06-15 15:35:59.470664000 +0000">

(dev)> song.recording_blob
=>
#<ActiveStorage::Blob:0x00000001235c6f60
  id: 1,
  key: "oi7xszss3y6kfer601zvxfpl1muz",
  filename: "ruby_blues.wav",
  content_type: "audio/x-wav",
  metadata: {"identified" => true},
  service_name: "local",
  byte_size: 35765766,
  checksum: "ddUPZtkz3hqVI9wUGOwp4g==",
  created_at: "2025-06-15 15:35:59.461367000 +0000">>

Aha! The file has been correctly identified as being of type audio/x-wav, but apart from that, there's no other metadata being persisted. We're here to change that.

First, let's write our custom analyzer. We'll put it in the lib/active_storage directory, and call it ActiveStorage::WaveformAnalyzer. To draw upon the existing implementation, we'll inherit from ActiveStorage::Analyzer::AudioAnalyzer.

We saw above that the metadata method is responsible for returning the appropriate data, so we'll override it. We have to be careful to call super and merge any new data into what the parent class already provides.

# lib/active_storage/waveform_analyzer.rb

module ActiveStorage
  class WaveformAnalyzer < ActiveStorage::Analyzer::AudioAnalyzer
    def metadata
      super.merge waveform
    end

    def waveform
      rms_values = []

      download_blob_to_tempfile do |file|
        IO.popen([ ffmpeg_path,
          "-i", file.path,
          "-ac", "1",
          "-f", "f32le", "-"
        ], "rb") do |io|
          frame_size = 4 # mono, 4 bytes (float32)
          chunk_size = 512 * frame_size # 512 frames

          while chunk = io.read(chunk_size)
            floats = chunk.unpack("e*") # little-endian float32

            next if floats.empty?

            rms = Math.sqrt(floats.sum { _1 ** 2 } / floats.size)
            rms_values << rms
          end
        end
      end

      # optionally store as Base64 packed string to save space
      # { waveform: [ rms_values.pack("e*") ].pack("m0") }

      { waveform: rms_values }
    end

    def ffmpeg_path
      ActiveStorage.paths[:ffmpeg] || "ffmpeg"
    end
  end
end

The real meat, though, is of course the computation of waveform datapoints. Because ActiveStorage depends on it, we can utilize ffmpeg for our purposes. The full call we pass to IO.popen here reads like this:

ffmpeg -i FILE_TO_ANALYZE -ac 1 -f f32le -

Here, -ac 1 tells ffmpeg to mix the audio down to one channel, while -f f32le specifies "float 32-bit little endian" as the output format. The final dash - instructs ffmpeg to output this to STDOUT, so we can actually pick it up in the block passed to IO.popen.

There's one important signal processing detail to mention: storing each and every sample as metadata wouldn't be very efficient — in fact, it would result in storing the entire audio file as a JSON array. Instead, we perform some data compression here by calculating the root mean square over a frame. That's only a fancy way of saying we're calculating the average of 512 (an arbitrary number, but mostly a power of 2) samples, but we square it and subsequently take the square root, because audio sample data can be both positive and negative.

Our new analyzer isn't yet wired up to be used by the Rails application, so we do that in an initializer:

# config/initializers/active_storage.rb

require_relative "../../lib/active_storage/waveform_analyzer.rb"

Rails.application.config.active_storage.analyzers.prepend ActiveStorage::WaveformAnalyzer

It's important to prepend it to the list here, because, as we've observed above, the first analyzer in the list that accepts audio files will be used.

Let's put it to use in the Rails console again:

(dev)> song = Song.create(title: "Ruby Blues in D Flat")
(dev)> song.recording.attach(io: File.open("/path/to/file"), filename: "ruby_blues.wav")
(dev)> song.reload.recording_attachment.metadata[:waveform]
=>
[0.00012394643736996606,
 0.00087319937227967,
 0.0037783625670793465,
 0.005877352246693453,
 ... etc.]

Now that we have a compressed representation of the audio in our metadata, how can we make use of it? We'll take a look at an idiomatic ActiveStorage method in the second part of this series, but for starters, many JavaScript audio widgets allow you to specify precomputed waveform data, like WaveSurfer does in this example.

Calculating Image Blurhashes in Ruby

Blurhashes are compressed representations of images you can use instead of generic placeholders for an enhanced lazy loading experience. The blurhash Ruby gem provides a straightforward way to encode such strings from images. We add it to our application's dependencies like so:

$ bundle add blurhash

Before we begin writing our custom analyzer, it's important to note that the image processing backend comes in two flavors: Vips or ImageMagick. Since its API is a bit simpler, we'll concentrate on the latter and configure it in config/application.rb:

config.active_storage.variant_processor = :mini_magick

We can now begin our implementation by subclassing the ActiveStorage::Analyzer::ImageAnalyzer::ImageMagick base analyzer:

# lib/active_storage/blurhash_analyzer.rb

module ActiveStorage
  class BlurhashAnalyzer < ActiveStorage::Analyzer::ImageAnalyzer::ImageMagick
    attr_accessor :thumbnail

    def metadata
      read_image do |image|
        build_thumbnail(image)
        super.merge blurhash
      end
    end

    def blurhash
      {
        blurhash: ::Blurhash.encode(
          thumbnail.width,
          thumbnail.height,
          pixels
        )
      }
    end

    def build_thumbnail(image)
      # we scale down the image for faster blurhash processing
      @thumbnail ||= MiniMagick::Image.open(
        ::ImageProcessing::MiniMagick.source(image.path).resize_to_limit(200, 200).loader(page: 0).call.path
      )
    end

    def pixels = @thumbnail.get_pixels.flatten

    protected

    def processor = "ImageMagick"
  end
end

What's going on here? We encounter an already familiar pattern: we populate the database column of the same name with more data using the metadata method. read_image is just a helper method provided by Rails that opens the image as a file, ready for us to use. The build_thumbnail method is actually optional, but very helpful for speeding up the blurhash calculation. We simply use MiniMagick to scale down the image to fit within the bounds of 200x200 pixels. The blurhash method then employs this method to build the compressed string representation from the downscaled image.

Like above, we prepend it to the list of analyzers in our initializer:

# config/initializers/active_storage.rb

require_relative "../../lib/active_storage/waveform_analyzer.rb"
require_relative "../../lib/active_storage/blurhash_analyzer.rb"

Rails.application.config.active_storage.analyzers.prepend ActiveStorage::WaveformAnalyzer
Rails.application.config.active_storage.analyzers.prepend ActiveStorage::BlurhashAnalyzer

Now it's time to test it out. For this, we'll use a test image from picsum.photos:

Let's open a Rails console and attach this image to a post:

(dev)> post = Post.create(title: "Active Storage Analyzers")
(dev)> post.images.attach(io: URI.open("https://picsum.photos/id/128/1200/800"), filename: "picsum_128.jpg")

# we inspect its metadata we will now find a blurhash representation of the image:

(dev)> post.reload.images.first.metadata["blurhash"]
=> "LWDJS1o#D%kD~qbIIUof%2WARkfP"

For reference, converted to an actual preview image, it looks like this:

To make use of this in our application frontend, the blurhash needs to be decoded and presented. Typically, this involves using the official TypeScript library and a bespoke Stimulus controller. In the second part of this series, we'll take a look at implementing a pure ActiveStorage-generated preview.

That's it for this first part!

Wrap Up

In this article, we've taken a few first steps to customize how ActiveStorage handles and processes media data. We've learned that the ActiveStorage::Blob model is where the data describing an attachment is stored. When writing bespoke analyzers, it's necessary to put any results in its metadata column.

We then looked at two examples explaining when and how to write your own custom ActiveStorage analyzers: extracting interleaved waveform data from audio files and calculating image blurhashes. Both implementations demonstrate the diligence that has been put into ActiveStorage's background media processing API: the glue code that is necessary to plug binaries like ffmpeg or ImageMagick into the analysis pipeline is minimal.

In the second and final part of this series, we will reverse this process and examine ways to implement custom ActiveStorage previewers, providing compact graphic representations of the calculated metadata.

Happy coding!

A Deep Dive into Solid Queue for Ruby on Rails

Roy Tomeij — 2025-06-18T00:00:00+00:00

Our previous article in this series established that Solid Queue is an excellent choice if you need a system for processing background jobs. It minimizes external dependencies — no need for Redis! — by storing all jobs in your database. Despite that, it is incredibly performant.

But just being performant is not enough for a production-ready background job system. Rails developers have come to expect a lot over the years. We don't just want to enqueue jobs to run in the background. We want to schedule jobs, run them on a recurring schedule, and we might even want to limit how many jobs can run concurrently. We want more features!

Amazingly, Solid Queue provides all of those features out of the box. Let's dive deeper into Solid Queue and learn how that's possible!

Scheduling Jobs with Solid Queue for Ruby on Rails

First, it's time for a small recap. Solid Queue uses your database — and only your database — to store job data. Everything it does is backed by one database table or another. Scheduling jobs — that is, designating jobs to run at some specific point in the future — is no different. Any scheduled job is stored in solid_queue_scheduled_executions.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
create_table "solid_queue_scheduled_executions", force: :cascade do |t|
  t.bigint "job_id", null: false
  t.string "queue_name", null: false
  t.integer "priority", default: 0, null: false
  t.datetime "scheduled_at", null: false
  t.datetime "created_at", null: false
  t.index [ "job_id" ], name: "index_solid_queue_scheduled_executions_on_job_id", unique: true
  t.index [ "scheduled_at", "priority", "job_id" ], name: "index_solid_queue_dispatch_all"
end

This table is almost identical to the solid_queue_ready_executions table. The only difference is the addition of the scheduled_at column, which tells us when a scheduled job is supposed to be executed. Let's confirm that by looking at what happens when we schedule a job.

MyJob.set(wait_until: Date.tomorrow.noon).perform_later

INSERT
  INTO "solid_queue_scheduled_executions" ("job_id", "queue_name", "priority", "scheduled_at", "created_at")
  VALUES (1, 'default', 0, '2025-04-02 12:00:00.000000', '2025-04-01 12:00:00.000000')
  RETURNING "id"

There are no surprises there. Solid Queue adds a new row to the solid_queue_scheduled_executions table, which contains the data that we'd expect. But how do we go from such a record existing to actually running a job at the right time?

We need a process that continuously polls the solid_queue_scheduled_executions table. That process is called the Dispatcher, and it is responsible for executing scheduled jobs on time. It begins when Solid Queue starts — no additional configuration is required. However, if needed you can start only the dispatcher process by running Solid Queue with a specific configuration.

# Start Solid Queue with this configuration, for example, with bin/jobs -c config/only_dispatcher.yml
production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300

In case you were wondering how the Dispatcher process is supervised, that is the responsibility of the aptly named Supervisor. It keeps track of any running processes within Solid Queue, including worker processes and Dispatchers.

So, how does the Dispatcher actually work? It defines a poll method called within a loop to continuously check for scheduled jobs. The polling code is spread over several classes and modules, but in a heavily simplified form, it looks like this:

# lib/solid_queue/dispatcher.rb
class Dispatcher < Processes::Poller
  # Batch size and polling interval are based on your configuration
  def poll
    job_ids = ScheduledExecution.due.ordered.limit(batch_size).pluck(:job_id)
    jobs = Job.where(id: job_ids)
    Job.dispatch_all(jobs).map(&:id).then do |dispatched_job_ids|
      ScheduledExecution.where(job_id: dispatched_job_ids).delete_all
    end
  end
end

The query to retrieve 'ready' scheduled executions is straightforward.

SELECT "solid_queue_scheduled_executions".*
  FROM "solid_queue_scheduled_executions"
  WHERE "solid_queue_scheduled_executions"."scheduled_at" <= '2025-04-02 12:00:01.000000'
  ORDER BY "solid_queue_scheduled_executions"."scheduled_at" ASC,
    "solid_queue_scheduled_executions"."priority" ASC,
    "solid_queue_scheduled_executions"."job_id" ASC
  LIMIT 500

So, any scheduled job with scheduled_at in the past is ready to be dispatched. As we covered in part one of this series, when Solid Queue dispatches a job, it creates a ReadyExecution record and destroys the corresponding ScheduledExecution record. The ReadyExecution record is then picked up by regular worker processes, and the corresponding job runs.

So far, so good. Scheduled jobs are really not that complicated! Let's look at something more complex: recurring tasks.

Recurring Tasks

Recurring tasks are an oft-requested feature for background job processors. Simply put, they are background jobs that should run on a recurring schedule. They are similar to Cron jobs in that you define a schedule (such as every five minutes, every day at noon, and so on) for when work should occur.

In Solid Queue, you configure your recurring jobs using the config/recurring.yml file. For example, if we wanted to run a CleanupData job every day at noon, this is how we would do it.

production:
  cleanup_data:
    class: CleanupData
    args: []
    schedule: every day at noon

Solid Queue uses Fugit to parse schedule expressions, which is why human-readable schedules such as 'every day at noon' are permitted. When using scheduled tasks, you define the class of the job to be run and any job arguments. The excellent SolidQueue recurring tasks ReadMe provides more details. We're here to learn how it works, so let's look under the hood.

Recurring tasks are represented by the RecurringTask model, which is backed by a corresponding solid_queue_recurring_tasks table. The columns therein correspond to the fields available in the configuration file.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
create_table "solid_queue_recurring_tasks", force: :cascade do |t|
  t.string "key", null: false
  t.string "schedule", null: false
  t.string "command", limit: 2048
  t.string "class_name"
  t.text "arguments"
  t.string "queue_name"
  t.integer "priority", default: 0
  t.boolean "static", default: true, null: false
  t.text "description"
# ...
end

When you start SolidQueue, recurring task records are created according to your recurring tasks configuration file. To create jobs at the right time, we once again need a new process — this time called the Scheduler. The Scheduler is a sibling to the Dispatcher, which we already know about. It works in almost the same way: A new process is spun up when Solid Queue starts, and this process runs an endless loop. The difference between the Scheduler and the Dispatcher is what happens within that loop. Where the Dispatcher queries the solid_queue_scheduled_executions table, the Scheduler queries solid_queue_recurring_tasks — and schedules jobs at the right time. So, how exactly does the Scheduler know what the right time is, and when to schedule the right jobs?

To answer that question, we have to examine the implementation closely. The scheduler class creates a new RecurringSchedule object that defines a schedule method. That method is repeatedly called for each scheduled task. Here's a simplified version:

# lib/solid_queue/scheduler/recurring_schedule.rb
def schedule(task)
  scheduled_task = Concurrent::ScheduledTask.new(task.delay_from_now, args: [ self, task, task.next_time ]) do |thread_schedule, thread_task, thread_task_run_at|
    thread_schedule.schedule(thread_task)
    thread_task.enqueue(at: thread_task_run_at)
  end
  scheduled_task.tap(&:execute)
end

Let's untangle this code. Solid Queue uses Concurrent::ScheduledTask (from the concurrent-ruby library) to spawn a new thread. That thread is scheduled to run at the time specified by the recurring task's schedule. When that thread runs, it first recursively spawns another thread to schedule the next recurring task. Then, it enqueues the 'current' scheduled job.

Let's look at an example of a simple recurring task to get a handle on things.

production:
  my_periodic_task:
    class: CleanupData
    args: []
    schedule: every hour

If we start Solid Queue at 8:30, the variables within the schedule method are assigned the following values. Not verbatim, mind you. We're greatly simplifying here.

Variable	Value
task	my_periodic_task
task.delay_from_now	30 minutes
task.next_time	9:00

So, our background thread is scheduled to run thirty minutes from now, which is 9:00. When that time rolls around, the background thread is executed. It runs thread_task.enqueue(at: 9:00) — so an instance of CleanupData is queued for execution. It also calls itself recursively via thread_schedule.schedule. Because it is now 9:00, the variables for this invocation have changed.

Variable	Value
task	my_periodic_task
task.delay_from_now	59 minutes
task.next_time	10:00

So, the background thread is scheduled to run again at 10:00, and the cycle continues. You may be wondering what happens if the scheduling thread is killed, for example during a re-deploy or system crash. Won't that upset your schedules? Luckily, the answer is no. Cron schedules are static. An expression like 'Every Hour' always resolves to 10:00, 11:00, 12:00, and so on, regardless of when Solid Queue starts. Any interruptions to the scheduling thread don't change that.

Here are some other implementation details to be aware of. First, this pattern of scheduling the next occurrence of a recurring task before executing it is inspired by GoodJob. Second, RecurringTask.enqueue does not create a new Job and ReadyExecution record as you might expect. Instead, it creates yet another record, namely RecurringExecution.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
create_table "solid_queue_recurring_executions", force: :cascade do |t|
  t.bigint "job_id", null: false
  t.string "task_key", null: false
  t.datetime "run_at", null: false
  # ...
  t.index [ "job_id" ], name: "index_solid_queue_recurring_executions_on_job_id", unique: true
  t.index [ "task_key", "run_at" ], name: "index_solid_queue_recurring_executions_on_task_key_and_run_at", unique: true
end

This record is solely to avoid executing recurring jobs multiple times. It has an index on task_key and run_at with unique constraints to serve that purpose. A RecurringTask is only queued if there is no prior RecurringExecution for the same time and the same job.

# app/models/solid_queue/recurring_task.rb
def enqueue(at:)
  active_job = if using_solid_queue_adapter?
    # Create a RecurringExecution and enqueues the job
    enqueue_and_record(run_at: at)
  else
    # ...
  end
  rescue RecurringExecution::AlreadyRecorded
    payload[:skipped] = true
    false
end

Attentive readers will notice that this code snippet points to a limitation in Solid Queue. That is, if you are not using Solid Queue as a backend to run your cron-style tasks — yes, you can do that — Solid Queue can't guarantee that recurring jobs are enqueued only once. If you find yourself in such a situation, you should be aware of that.

You may also be wondering what happens if the Scheduler process dies or is killed — for example, during a deployment. Since recurrences are managed by a thread, won't killing the thread break schedules? Luckily, the answer to that is no.

Concurrency Controls

Let's look at one final feature of Solid Queue, namely concurrency controls. Sometimes, you want to limit how many jobs of a certain kind can run simultaneously. You can do so using Solid Queue with limits_concurrency.

class MyJob < ApplicationJob
  limits_concurrency to: 1, key: ->(user) { user.id }, duration: 15.minutes, group: 'UserOnboarding'
  # ...

Here, we are telling SolidQueue to run a maximum of one instance of MyJob for each user. Let's examine the configuration in more detail.

to: The maximum number of jobs you want running concurrently.
key: A required argument to designate which jobs should be limited together. In our example, jobs with the same User ID are limited to a single concurrent execution. You may use any job arguments as key, but constants such as strings or symbols are also allowed.
duration: The maximum time for which Solid Queue can guarantee concurrency after a job is enqueued. If your jobs run longer than that, concurrency controls will not apply and jobs may overlap. We'll learn why later!
group: You can use this option to limit concurrency across different job classes.

If you want to learn more, I refer you to the concurrency control documentation. Concurrency controls are easily Solid Queue's most sophisticated feature. If scheduled tasks didn't already make your head spin, learning how this feature works surely will.

Let's start with the basics. Like other Solid Queue features, concurrency controls are backed by various models and their corresponding database tables. Two that you need to be particularly aware of are Semaphore and BlockedExecution.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
create_table "solid_queue_semaphores", force: :cascade do |t|
  t.string "key", null: false
  t.integer "value", default: 1, null: false
  t.datetime "expires_at", null: false
  # ...
end

create_table "solid_queue_blocked_executions", force: :cascade do |t|
  t.bigint "job_id", null: false
  t.string "queue_name", null: false
  t.integer "priority", default: 0, null: false
  t.string "concurrency_key", null: false
  t.datetime "expires_at", null: false
  # ...
end

Let's look at Semaphore first. As the name suggests, this is an implementation of the counting semaphore pattern. Whenever Solid Queue enqueues a job with limits_concurrency, it first tries to acquire a semaphore lock based on a concurrency key. This concurrency key is based on the arguments passed to limits_concurrency, namely the job class, the key, and the group name — if any was provided. If the semaphore is available, the job is enqueued. If it is not, a BlockedExecution record is created instead.

Semaphores have a value to support multiple concurrent jobs. You can think of it as the remaining capacity of the semaphore. Acquiring a semaphore means decrementing that value, and releasing it means incrementing it. A semaphore is considered unavailable once its value is zero. Let's look at an example of how the locking mechanism works for a simple job.

class MyJob < ApplicationJob
  # Only three jobs of 'MyJob' should run at the same time
  limits_concurrency to: 3, key: -> { 'MyJob' }, duration: 15.minutes
  # ...
end

Let's see what happens if we try to enqueue this job multiple times in succession.

The first instance of MyJob is enqueued. There is no semaphore yet, so one is created. Its initial value is limit - 1. Because your limit is three, the initial value of the semaphore is two.
The second instance of MyJob is enqueued. Solid Queue attempts to acquire a lock for that job. The job can be enqueued because the value is two, which is greater than zero. The value of the semaphore is decremented to one.
A third instance of our job is enqueued. We repeat the same procedure as before. The value of the semaphore is now zero.
A fourth instance of MyJob is enqueued. Acquiring the semaphore fails because its value is now zero. A BlockedExecution record is created for the job.
The first instance of our job finishes. When it finishes, it releases the semaphore, so the semaphore value is once again one.
On finishing, the first job instance also calls a method to release any blocked jobs.
The fourth instance of MyJob is released and again tries to acquire a lock. The semaphore value is one, so the lock can be acquired and the blocked job queued. The semaphore value is now zero.

The code for releasing a semaphore when a job finishes is straightforward.

# app/models/solid_queue/claimed_execution.rb
def perform
  result = execute
  # ...
ensure
  # This method releases the Semaphore and makes it available for the next blocked job.
  job.unblock_next_blocked_job
end

There is one more detail that we haven't touched on yet. Why do semaphores have an expiry date, and why do we need to set a duration when using limits_concurrency?

Let's consider what happens when a job crashes without releasing its semaphore — for example, when a worker processing that job dies. Unless we add some mechanism to clean up semaphores, the lock held by that job will be retained forever. In the worst case, this would forever block other jobs from being processed.

Semaphores have an expiry that corresponds to the duration given in the job definition to avoid that situation. If a semaphore expires — which happens if no jobs are enqueued — the semaphore will be destroyed. We already know the process responsible for that — it's our friend, the Dispatcher. It instantiates the ConcurrencyMaintenance class, which does two things:

First, it removes any expired semaphores.
Second, it will check if there are any blocked jobs and release them.

Jobs are released one by one, so concurrency limits will still hold. Consider, however, what happens if your job runs longer than the given duration. In that case, the semaphore will be cleaned up, although the job will still run. If another job is then enqueued, those jobs will overlap.

Monitoring Solid Queue for Rails with AppSignal

As we've established, Solid Queue can do a lot. However, with all these moving parts, monitoring becomes crucial. Luckily, AppSignal provides built-in support for Solid Queue, with ready-made dashboards for job execution times, throughput, and failure rates. Simply install AppSignal in your Rails application, and you're good to go.

AppSignal will automatically detect your usage of Solid Queue and create an active job dashboard that contains graphs for important metrics, such as error rate and throughput.

If you ever see jobs that misbehave — either because they run slowly or have too many errors — assign them a status and assignee to effectively resolve the issue.

Obviously, you shouldn't have to look at dashboards all day to figure out if there is a problem. AppSignal Alerts has your back. Simply create a new alert for job metrics, such as failure rate and job duration, and you're all set.

Solid Queue is amazing for adding powerful job processing to your application without hassle. AppSignal does the same when it comes to monitoring!

Wrapping Up

We've covered a lot of ground in our exploration of Solid Queue's advanced features. From scheduled jobs to complex dependency chains, each feature builds on the solid foundation we discussed in part one. As we've seen, it's not easy to build a job processing backend. But by diving deep into the Solid Queue source code and its workings, we've gained an understanding and some appreciation for the challenges involved.

In any case, Solid Queue is a wonderful addition to the Rails ecosystem, due to its excellent database design and process coordination. It provides the tools you need while maintaining its core promise: simplicity and reliability, without external dependencies.

Happy coding!

Set Up Tracing for a Ruby on Rails Application in AppSignal

Roy Tomeij — 2025-05-21T00:00:00+00:00

In this guide, we'll harness AppSignal to detect, diagnose, and remove performance bottlenecks and employ proper tracing in a Ruby on Rails application. From setting up tracing to capturing errors and logging, we’ve got you covered.

We'll ensure our application runs smoother than ever, even under the heaviest loads!

But first, let's quickly touch on how to define tracing and its benefits.

What Is Tracing?

Tracing is the process of following a request and operation through an application. In Ruby applications, tracing captures the execution flow, providing deep insights into the performance of various components.

Benefits of Tracing

Tracing has several benefits, including:

Performance Optimization: Tracing identifies slow parts of an application that need performance improvements.
Better Debugging: Detailed traces allow you to quickly pin down code issues and determine their causes.
Improved Reliability: Tracing tracks application behavior, ensuring more reliable and effective system operation.

A Scenario: Our Lagging Rails App

Let's say it's Black Friday, the biggest shopping day of the year. Your Rails-based e-commerce platform is buzzing with thousands of eager customers, their carts brimming with products, ready to check out. Everything seems perfect — until the system starts lagging.

Transactions fail. Abandoned carts skyrocket. Panic ensues. This is every developer's nightmare. But what if you had a secret weapon? A tool that not only alerts you to issues in real-time but also dives deep into the heart of your application, tracing every request, every database query, and every background job? Enter AppSignal.

How to Set Up Tracing for Ruby on Rails Apps using AppSignal

To demonstrate the power of tracing using AppSignal, I have created a sample Rails e-commerce project that we can integrate with AppSignal. This will help you see firsthand how tracing can identify and resolve these problems.

Prerequisites

Ruby Version: AppSignal is compatible with Ruby 2.5 and above.
AppSignal Account: Create an account on AppSignal (there's a free trial available)

Now you're ready to set up tracing in your Ruby application using AppSignal.

Step 1: Install the AppSignal Gem

Here's the gem:

gem 'appsignal'

Simply run bundle install:

bundle install

Step 2: Initialize AppSignal

Run the AppSignal installation command to set up the necessary configuration files.

bundle exec appsignal install <YOUR-PUSH-API-KEY>

Follow the prompt on your terminal. You can choose a config file or environment variables to configure AppSignal in your app. We will choose the config file option.

This command will generate an appsignal.yml configuration file in your config directory. This file will be pre-filled with your AppSignal push API key and some basic configuration settings.

Step 3: Configure AppSignal

Ensure config/appsignal.yml is correctly configured for your environments. This file contains configuration settings for different environments (development, test, production). Ignore actions that provide little value.

And ta da! You're all set up:

Instrumenting our Ruby on Rails Application

AppSignal can automatically instrument several key components of our Rails application, such as database queries and web requests. We can also add custom instrumentation to trace specific parts of our application.

Custom Rails Instrumentation

To instrument specific parts of our code, we will wrap a couple of lines of code with the Appsignal.instrument method. This method can be used to trace specific blocks of code, such as more complex parts of controller actions or background jobs.

# app/controllers/orders_controller.rb

class OrdersController < ApplicationController
  def index

    @orders = Order.all #=> Candidate for N+1 queries


    Appsignal.instrument('calculate_totals.orders') do
      @orders.each do |order|
        order.total_price = order.line_items.sum { |item| item.quantity * item.product.price }
      end
    end

    render json: @orders.as_json(include: {
      user: { only: [:id, :name, :email] },
      line_items: { include: { product: { only: [:id, :name, :price] } }, only: [:id, :quantity, :price] }
    })

  end
end

Sending a request to this endpoint will result in an N+1 query being reported.

An N+1 issue occurs when our application makes multiple database queries to load associated records for each object in a collection, instead of using a single, efficient query. This can significantly degrade performance, especially when dealing with large datasets.

The fix in this case is to eager load the get all orders query — simply replace this code:

@orders = Order.all

With the following:

@orders = Order.includes(:user, line_items: :product).all

Collecting and Reporting Ruby Errors with AppSignal

AppSignal allows you to capture and report application errors using the Appsignal.set_error method.

Here’s an example of capturing and reporting an error in the order controller index action:

def index
  @orders = Order.all #=> Candidate for N+1 queries
  # Introduce a deliberate error based on order count
  if @orders.size > 10
    raise "Too many orders"
  end
  render json: @orders.as_json(include: {
    user: { only: [:id, :name, :email] },
    line_items: { include: { product: { only: [:id, :name, :price] } }, only: [:id, :quantity, :price] }
  })
rescue => e
  # Report error to AppSignal
  Appsignal.set_error(e)
  render json: { error: e.message }, status: :internal_server_error
end

In this example, we demonstrate custom error handling within a specific controller action. However, it's important to note that AppSignal reports errors by default, so explicit error reporting with Appsignal.set_error is not usually necessary.

For consistent and centralized error handling across all controller actions, it’s recommended you use rescue_from at the controller level. This approach ensures that any unhandled exceptions in your controller actions are properly reported and managed.

Here’s an example of how you can implement this:

class OrdersController < ApplicationController
  # Use rescue_from to handle errors for all controller actions
  rescue_from StandardError, with: :handle_error

  def index
    @orders = Order.includes(:user, line_items: :product).all

    Appsignal.instrument('orders.index.custom_query') do
      # Fake some complexity
      @orders.each do |order|
        order.total = order.line_items.sum { |item| item.quantity * item.product.price }
      end
    end

    render json: @orders.as_json(include: {
      user: { only: [:id, :name, :email] },
      line_items: { include: { product: { only: [:id, :name, :price] } }, only: [:id, :quantity, :price] }
    })
  end

  private

  # Error handling method
  def handle_error(exception)
    Appsignal.set_error(exception)
    render json: { error: exception.message }, status: :internal_server_error
  end
end

On the Error > Issue list tab, we can see a list of all errors, the status of each issue, and how long ago the error occurred:

Clicking on the RuntimeError, we can see the error log and the code line that is triggering this error:

Let's touch on some more advanced tracing techniques before we wrap up.

Advanced Tracing Techniques

We'll implement some advanced tracing techniques on the order endpoint of our e-commerce app.

These techniques will help us manage high-traffic environments efficiently, ensure data privacy, and secure access to trace data.

High Traffic Use Cases

In high-throughput environments, collecting trace data for every request can lead to significant overhead. Sampling helps reduce this by collecting trace data for only a subset of requests.

Adding Metadata to Transactions

You can supply extra context on errors and performance issues using tags and sample data. This can help to add information that is not already part of the request, session, or environment parameters. Read more about how to pass and add additional metadata.

Asynchronous Processing

Asynchronous processing is a common technique used to handle tasks that are too time-consuming to be performed within the request-response cycle, such as sending emails, processing background jobs, and handling large data imports. Ruby on Rails applications typically use background job libraries like Sidekiq, Resque, or Delayed Job to manage these tasks.

AppSignal seamlessly integrates with libraries such as Active Job, DelayedJob, Shoryuken, Sidekiq, and Que to provide insights into the performance and errors of your background jobs.

Security Considerations

Security is a very important piece of any production-ready application. We need to ensure that sensitive data is not included in trace data to maintain user privacy. The values of filter parameters will be replaced with [FILTERED] when transmitted to AppSignal.

Let's modify config/appsignal.yml to include several sensitive request params that need masking.

# config/appsignal.yml
default: &defaults
  push_api_key: "<%= ENV['APPSIGNAL_PUSH_API_KEY'] %>"
  name: "ECommerceApp"
  filter_parameters:
    - password
    - email
    - api_token
    - token

If you use the Rails filter_parameters config option, AppSignal will merge its config with Rails's config, so there's no need to configure it twice.

By implementing these advanced tracing techniques, you can efficiently manage high-traffic environments and ensure the privacy of sensitive data. Read more on filter parameters.

Wrapping Up

We've seen that setting up tracing for a Ruby app using AppSignal involves understanding tracing fundamentals, preparing your application, and following a step-by-step setup process. Advanced methods and debugging techniques can then be used to enhance tracing.

Regularly applying tracing in development and maintenance cycles leads to higher performance and reliability. Get started with AppSignal today for enhanced monitoring and deep performance insights into your Ruby applications.

Happy coding!

An Introduction to Solid Queue for Ruby on Rails

Roy Tomeij — 2025-05-07T00:00:00+00:00

One of the most exciting additions to Rails 8 is undoubtedly Solid Queue, a new library for processing background jobs.

You might not think it's that big of a deal. After all, there are plenty of other queuing systems out there. If you work with Rails, you'll likely know about Sidekiq and Resque — both are exceptionally performant and reliable. There is also GoodJob and the venerable DelayedJob. With all those options available, do we really need another queuing system?

Let's find out together. In this two-part series, we'll dig deep into Solid Queue's internals, discover what makes it unique, and learn more about why it was created in the first place.

Why Solid Queue for Ruby on Rails?

Since Rails 7, the team at 37Signals has been on a quest to reduce the operational overhead needed to launch a new Rails application. As part of this, they made SQLite the new default database for Rails apps - even in production. Furthermore, they started an effort to eliminate additional infrastructure dependencies to take full advantage of this new default.

37Signals had used Resque until then, and Resque requires Redis to function. So does Sidekiq, for that matter. To get rid of Redis, they had to create a queuing system that relies only on your database — and that queuing system turned out to be Solid Queue.

So that's its main selling point: No additional dependencies; just use your database. Very nice! However, as with any queuing system — and especially one that is the new Rails default — Solid Queue needs to satisfy some stringent requirements.

It must provide all the features Rails developers are used to from other background job systems. As the Rails default, it must support all databases that Rails works with. Obviously, it needs to satisfy standard safety requirements — as in, it must never, ever lose jobs. Last but not least, it must be fast enough to be a viable option for large production systems.

That's quite a tall order! So, how does Solid Queue address all those requirements?

Solid Queue From The Top

There are many details to consider, but let's start with a high-level architectural overview. You need to be aware of two significant components: Jobs and Workers.

Job is an ActiveRecord model, and what the user interacts with. Note that that's not necessarily true for other ActiveJob backends — it's just how SolidQueue implements background jobs. If you need to create a new background job, this is the class that you inherit from. Job also defines methods that enable you to enqueue work, such as Job.perform_later.

# app/jobs/my_job.rb
class MyJob < ApplicationJob
  queue_as :default

  def perform
    # Do something later
  end
end

Workers, as the name suggests, are the elements that perform the actual work. These are generally not directly created by the programmer but automatically created based on how you configure your application. For example, to have your application spawn two workers listening to all and two specific queues respectively, you'd use the following configuration file:

# config/queue.yml
production:
  workers:
    - queues: "*"
    - queues: [default, critical]

Workers are spawned as processes, running in the background, waiting for jobs to be assigned to them. As you may have guessed, your database is the missing link between jobs and workers. Whenever Solid Queue does anything, one database table or other is involved. SolidQueue does a lot of things, so a lot of tables are needed.

# lib/generators/solid_queue/install/templates/db/queue_schema.rb
ActiveRecord::Schema[7.1].define(version: 1) do
  create_table "solid_queue_jobs", force: :cascade do |t|
    # ...
  end

  create_table "solid_queue_ready_executions", force: :cascade do |t|
    # ...
  end

  create_table "solid_queue_scheduled_executions", force: :cascade do |t|
    # ...
  end

  create_table "solid_queue_claimed_executions", force: :cascade do |t|
    # ...
  end

  create_table "solid_queue_blocked_executions", force: :cascade do |t|
    # ...
  end

  create_table "solid_queue_failed_executions", force: :cascade do |t|
    #...
  end

  # Lots more tables below...
end

The Life and Death of a SOLID Job

To understand what all those tables do and how they relate to the various features of Solid Queue, let's look at the life cycle of a job. When a user enqueues a job to be executed later — let's say MyJob — a record is created in the solid_queue_jobs table. The record contains all the data required to execute the job — arguments, its name, the queue it is put in, and so forth. If the job is enqueued to run as soon as possible (rather than scheduled to run at some later point in time), an additional record is written to solid_queue_ready_executions.

For example, running MyJob.perform_later results in the following SQL:

INSERT INTO "solid_queue_jobs" ("queue_name", "class_name", "arguments", "priority", "active_job_id", "scheduled_at", "finished_at", "concurrency_key", "created_at", "updated_at")
  VALUES ('default', 'MyJob', '{"job_class": "MyJob","...",}', 0, '...', '2024-12-01 14:00:00', NULL, NULL, '2024-12-01 14:00:00', '2024-12-01 14:00:00')
  RETURNING "id"
INSERT INTO "solid_queue_ready_executions" ("job_id", "queue_name", "priority", "created_at")
  VALUES (1, 'default', 0, '2024-12-01 14:00:00')
  RETURNING "id"

Your workers poll this table for new records. A worker process that finds a new record will first claim it by writing another record to the solid_queue_claimed_executions table — we'll learn why that is necessary later. Only then will the worker actually execute the job. Below is some heavily edited code to illustrate what is happening (much more is happening in the actual code). If you are curious about the nitty-gritty details, I highly recommend you check out the original source code.

class Worker
  def run
    loop do
      break if shutting_down?

      unless poll > 0
        # Polling interval is configurable and defaults to 1ms
        sleep(polling_interval)
      end
    end
  end

  def poll
    # Claim jobs and then execute claimed jobs.
    claim_executions.then do |executions|
      executions.each do |execution|
        # Actually execute the job
      end
    end
  end

  def claim_executions
    # Query the ready executions table and claim a job for execution.
    with_polling_volume do
      SolidQueue::ReadyExecution.claim
    end
  end
end

Once a worker finishes a job, it removes the corresponding records from the solid_queue_jobs, solid_queue_ready_executions, and solid_queue_claimed_executions tables. That's all there is to it — just polling some tables, creating and removing records. Not so tricky, right? It would be if there weren't critical non-functional requirements to consider, too.

On Performance

To achieve production-ready performance, Solid Queue uses ingenious database design. You may have wondered why workers poll solid_queue_ready_executions rather than solid_queue_jobs. The additional table seems redundant at first glance.

Consider that solid_queue_jobs may contain thousands or millions of records, and querying that pile of data takes time. In comparison, solid_queue_ready_executions is tiny, as it only contains records for jobs that must be executed right now! That leads to some serious speedup.

The introduction of additional tables also simplifies queries. Workers only use two different queries for polling. They either poll all queues or specific ones. That, in turn, allows for some nice covering indices.

SELECT job_id
  FROM solid_queue_ready_executions
  WHERE queue_name = "default"
  ORDER BY priority ASC, job_id ASC
  LIMIT 4
  FOR UPDATE SKIP LOCKED

# Indices for polling solid queue ready executions
create_table "solid_queue_ready_executions", force: :cascade do |t|
  t.index [ "priority", "job_id" ], name: "index_solid_queue_poll_all"
  t.index [ "queue_name", "priority", "job_id" ], name: "index_solid_queue_poll_by_queue"
end

All that still wouldn't be enough to achieve truly outstanding performance. Traditionally, queuing systems that rely on polling tables have had a significant problem. One worker would block all others while querying and updating the polling table.

Let's take a look at why. Consider the following query:

SELECT id
  FROM jobs
  WHERE queue = "default"
  AND claimed = 0
  ORDER_BY priority, id
  LIMIT 2
  FOR UPDATE;

The FOR UPDATE statement locks the rows selected by the query. This is necessary to avoid nasty race conditions, such as multiple workers grabbing the same job. But that also means that any worker running this query would block read access to the table. Thus, other workers would have to wait for that query to finish. The polling table becomes a bottleneck that hinders rapid job execution.

Luckily, modern databases (PostgreSQL >= 9.5, MySQL >= 8.0) solve this problem. The SKIP LOCKED statement allows the database to lock only the records that are being updated. The rest of the table remains unlocked and free to be polled concurrently.

SQLite does not support SKIP LOCKED, so worker processes must queue up. In most cases, this shouldn't be an issue. SQLite writes are fast as the database is present on disk. Even so, this is a limitation that you should be aware of.

Whether you're using SQLite or another database, AppSignal provides Solid Queue performance monitoring out of the box! We'll talk more about this in part two of this series.

Safety First

We've spent some time discussing solid_queue_ready_executions, but another table is instrumental for ensuring that Solid Queue functions reliably. A key requirement of any queuing system is that any job being enqueued is executed at least once. In other words, jobs must never be lost—we already alluded to this in the introduction.

Without additional safety measures, this could quickly happen. Imagine that a worker starts working on a job and, in doing so, updates the corresponding job record to claim it. Of course, this is necessary to avoid multiple workers running a job simultaneously.

Imagine that suddenly, this worker process dies without finishing execution. Your machine might crash, and the OS may kill the worker for consuming too much memory — accidents happen, you know. The job it claimed will remain stuck forever because no other workers can grab it. Thus, it will never be executed, and your users will be sad and angry. The end.

That is, unless we add additional safety measures. Solid Queue solves this problem by introducing yet more tables — solid_queue_claimed_executions and solid_queue_processes.

ActiveRecord::Schema[7.1].define(version: 1) do
  create_table "solid_queue_claimed_executions", force: :cascade do |t|
    t.bigint "job_id", null: false
    t.bigint "process_id"
    # ...
  end

  create_table "solid_queue_processes", force: :cascade do |t|
    t.datetime "last_heartbeat_at", null: false
    t.integer "pid", null: false
    #  ...
  end
  # ...
end

We've already mentioned solid_queue_claimed_executions. Let's look at what happens when a worker claims a job. For one, it sets the claimed flag in the solid_queue_jobs table. Additionally, a record is created in solid_queue_claimed_executions. This record contains the job_id of the job being claimed and the id of the worker process that makes the claim.

So, what is the solid_queue_processes table good for? Any worker process will create and periodically update a record in this table by setting last_heartbeat_at. Of course, that alone wouldn't solve our problem.

We need another process to keep track of running processes: the so-called supervisor. This process runs in the background and periodically checks solid_queue_processes. A record with a last_heartbeat_at older than a threshold — which defaults to 5 minutes — indicates that the corresponding worker has met a tragic fate.

If such a record is found, the supervisor jumps into action. First, it removes the record from solid_queue_processes. Then, it marks any jobs previously claimed by the now-deceased worker as up-for-grabs. Thus, other workers can claim them, avoiding the stuck-job situation.

More to Discover in Solid Queue

In this post, we covered a fair bit of Solid Queue's internals. We looked at its high-level architecture and how its most essential feature—enqueuing and executing a job—works under the hood. We also learned the critical role of FOR UPDATE SKIP LOCKED in performance. Finally, we learned how the supervisor process helps avoid stuck jobs.

But there is more to discover. Solid Queue offers many more features we haven't touched on, such as scheduling recurring and sequential jobs. Stay tuned as we continue our deep dive in part two of this series.

Pre-build a Secure Authentication Layer with Authentication Zero for Ruby on Rails

Roy Tomeij — 2025-04-16T00:00:00+00:00

Authentication is a critical element of any web application. The Ruby on Rails ecosystem has no shortage of solutions for this topic, as no authentication layer has been backed into the framework yet.

Devise is a crowd favorite. Although it has ended up as the de-facto standard and sports many built-in features and great plugins (such as integrations with NoSQL databases, encryption, roles, and UID support), it works as a separate entity from your application.

Some have devised a different solution: a configurable generator that equips your app with an authentication scaffold. We'll dive into that option in this post.

But first, let's examine the uses of Authentication Zero.

The Purpose of Authentication Zero

It's best to avoid building authentication yourself, as there are noteworthy security concerns to consider when doing so. Authentication Zero's (not to be confused with Auth0, the other, Ruby-friendly authentication solution) approach is to give you something that follows security and Ruby on Rails best practices. You are then free to adjust some parts to fit your needs.

Yet, that also means that you now have two additional responsibilities:

Not to turn the provided code into a weak authentication system
To keep it up to date

Authentication in a web application is a way to ensure that someone coming back to your application is who they say they are and can access data related to themselves (in short). We usually use a secret linked to a user model or a third-party provider with or without a complimentary second-factor authentication.

The simplest mechanism (a pair consisting of a user identifier and a password) implies:

A User model with username (or email) and password fields.
A Users controller to handle the creation of a user account (with a page to enter details, and an action to create the user and then redirect them elsewhere).
A Sessions controller that shows a login form and handles both login and logout, verifying that the submitted credentials match an existing user.

Don't get fooled, though; these elements don't just cover "a few lines". Each of these three lines implies details related to security, error handling, and also, yes, a friendly User Interface. Authentication Zero gives you all this alongside plenty of advanced features such as email and password validation, email verification, API authentication, logs, rate limiting, lock mechanisms, and more.

Our Project

To demonstrate how Authentication Zero works and how to use it, we are going to integrate it into a sample Ruby on Rails application. We will start from a freshly created application and add the authentication layer with Authentication Zero. We will cover several of the key options we can use with the generator to go from very simple authentication to something using Two-Factor Authentication as well as logging, password-less authentication, and more.

We will also compare Authentication Zero with other solutions — namely Devise and Auth0 — particularly in terms of their features and usage.

Setup for a Ruby on Rails 7.1 Application

Setting up from a barebone Ruby on Rails 7.1 application is pretty standard. Simply add the authentication-zero gem to the Gemfile, run bundle, and then the generator:

rails generate authentication

Let's review the output:

gemfile  bcrypt (~> 3.1.7)
 create  db/migrate/20240512081634_create_users.rb
 create  db/migrate/20240512081635_create_sessions.rb
 create  app/models/current.rb
 create  app/models/session.rb
 create  app/models/user.rb
 create  test/fixtures/users.yml
 create  app/controllers/identity/email_verifications_controller.rb
 create  app/controllers/identity/emails_controller.rb
 create  app/controllers/identity/password_resets_controller.rb
  force  app/controllers/application_controller.rb
 create  app/controllers/home_controller.rb
 create  app/controllers/passwords_controller.rb
 create  app/controllers/registrations_controller.rb
 create  app/controllers/sessions_controller.rb
 create  app/views/home/index.html.erb
 create  app/views/identity/emails/edit.html.erb
 create  app/views/identity/password_resets/edit.html.erb
 create  app/views/identity/password_resets/new.html.erb
 create  app/views/passwords/edit.html.erb
 create  app/views/registrations/new.html.erb
 create  app/views/sessions/index.html.erb
 create  app/views/sessions/new.html.erb
 create  app/views/user_mailer/email_verification.html.erb
 create  app/views/user_mailer/password_reset.html.erb
 create  app/mailers/user_mailer.rb
 # routes (removed for clarity)
 # tests (removed for clarity)

The generator does the following:

Ensures that the bcrypt gem is installed.
Adds User and Session models, plus a Current utility class for tracking the signed‑in user.
Adds controllers for sign‑up, authentication, password management, and sign‑in/out.
Adds views where needed for those workflows.
Adds a mailer for email verification and password‑reset emails.

Just to let you know, we have not used additional parameters when running the generator. This is the most straightforward authentication architecture we can have, with just an email and password to authenticate a user. We will cover additional features later in the article.

We already have a working sign-up and sign-in process with email‑based password resets. The implementation is what is essential here.

First, the ApplicationController has been updated. As it's the base for any controller, it's the first thing we must look at. It now contains two before_action hooks and their methods. The first one (set_current_request_details) heavily uses the Current class to store the user agent and IP address of the current request. The other one is there to check if there is an existing session for the request and also uses the Current class. If no session exists, the request is redirected to the login screen.

So, when we try to get to http://localhost:3000, we are redirected to the login screen. A thing to note: by default, any controller inheriting this one will try to authenticate the user through a session. So, if you do not need that in some controllers or actions, you will have to expressly declare an exception through skip_before_action.

Current Class

Here, the first thing to spot is the reliance on ActiveSupport::CurrentAttributes:

class Current < ActiveSupport::CurrentAttributes
  attribute :session
  attribute :user_agent, :ip_address

  delegate :user, to: :session, allow_nil: true
end

This is an:

Abstract super class that provides a thread-isolated attributes singleton, which resets automatically before and after each request. This allows you to keep all the per-request attributes easily available to the whole system.

Source: Ruby on Rails API docs

Note the delegate definition too: that will tie up nicely to the Session model and its association to the User one.

Session Model

A Session model is a lovely way to handle many things related to a current request's session and its associated data, including a user.

class Session < ApplicationRecord
  belongs_to :user

  before_create do
    self.user_agent = Current.user_agent
    self.ip_address = Current.ip_address
  end
end

User Model

The User model is also fairly standard and uses plenty of good practices, from has_secure_password to token generators, email, and password validations.

class User < ApplicationRecord
  has_secure_password

  # ...

  validates :email, presence: true, uniqueness: true, format: { with: URI::MailTo::EMAIL_REGEXP }
  validates :password, allow_nil: true, length: { minimum: 12 }

  normalizes :email, with: -> { _1.strip.downcase }

  before_validation if: :email_changed?, on: :update do
    self.verified = false
  end
end

Note how the email is validated, formatted properly, and the before_validation hook used to ensure that the email is marked as not verified.

  after_update if: :password_digest_previously_changed? do
    sessions.where.not(id: Current.session).delete_all
  end
end

Finally, note how sessions are destroyed whenever the password changes.

Sessions Controller

The SessionsController is a fairly standard approach, reusing the Current class to facilitate finding sessions.

class SessionsController < ApplicationController
  # ...

  def index
    @sessions = Current.user.sessions.order(created_at: :desc)
  end
end

See how readable this looks:

def create
  if user = User.authenticate_by(email: params[:email], password: params[:password])
    @session = user.sessions.create!
    cookies.signed.permanent[:session_token] = { value: @session.id, httponly: true }

    redirect_to root_path, notice: "Signed in successfully"
  else
    redirect_to sign_in_path(email_hint: params[:email]), alert: "That email or password is incorrect"
  end
end

This one makes great use of Ruby on Rails features to make the session creation very readable, while taking care of setting all the right information in the session (thanks to an Active Record association between the User and Session).

Next up, we'll look at identity controllers.

Identity Controllers

Three controllers have been added in the identity namespace. One handles both sending and verifying the verification email, one handles the password resets (also sending the email and managing the password update itself), and one holds the email address update for a user.

That code is concise and clear, so it's perfect to tweak for your needs. Remember to keep an eye on Authentication Zero's changelog for security issues and other significant updates. You are responsible for porting any security fixes or significant bug fixes to your version of the code generated two weeks, three months, or five years from now. And this is not just a matter of stapling Dependabot onto the repository: you might need to regenerate the code. Here's a trick to avoid conflicts between your custom additions and the generated code: use your own modules alongside the generated code. By relying on this mechanism, you will keep the code clearer and limit any issues when updating it.

The generated views are relatively simple and might not match the CSS engine you have selected (it didn't match the Tailwind one at the time of this post). But that does not matter. The views are the visible side of the iceberg. You can use your favorite Tailwind or Bootstrap template on top of the generated controllers and reuse the blueprint of the generated views for guidance.

Beyond the Base Features

Now that we have seen a primary use case of Authentication Zero and how it works, we can check more advanced features. Here is a short list of a few that are particularly interesting.

Pwned

With so many database leaks and bad password strategies, more than simple validations might be needed. The pwned gem allows you to check any password against the Pwned Password API. Adding this to a Ruby on Rails application is simple (add the gem and a validator on the password field). To add it automatically to your application with Authentication Zero, just add the --pwned flag when using the generator.

This will generate a User model with an additional validator using the Pwned gem:

validates :password, not_pwned: { message: "might easily be guessed" }

Two-factor Authentication

Two-factor authentication (2FA) is all the rage nowadays and should be mandatory for many applications. Authentication Zero proposes two ways to handle this:

Two-factor authentication (with an authenticator app) and recovery codes (--two-factor)
Two-factor authentication using a hardware security key (--webauthn)

Both will generate additional code for the User model, its migration, and controllers and views to handle the different factors.

Password Protect Significant Changes

In many applications, requesting an additional password check before accessing or changing data is legitimate. The generator's --sudo option integrates a require_sudo before the action filter that you can use in controllers to enforce this requirement.

User Event Tracking

The --trackable option ensures that actions (such as logins and logouts) are traced so that you can display a log for security purposes.

This feature relies on an Event model:

class Event < ApplicationRecord
  belongs_to :user

  before_create do
    self.user_agent = Current.user_agent
    self.ip_address = Current.ip_address
  end
end

As you can see, this model is associated with a user and uses the Current class (covered earlier in this post) through a before_create hook. With just this and by relying on similar hooks in the User and Session model, events such as logins (when a Session is created), logouts (when a Session is destroyed), email verifications or changed passwords will be tracked in the database.

The following are tracked:

User agent
IP address
Action
Timestamp
User (since each event is attached to a user)

This feature is primarily for security purposes, providing an audit trail for critical actions related to the security or identity of a user account. The generator also generates a controller and view for a user to see the events. There is no integrated feature to purge it but it'd not be complicated to add one.

Password-less Authentication

Authentication Zero's --passwordless option allows your application to email a magic link to a user so they can sign in to a freshly prepared session.

"Sign-in as" Button

One feature that kept being requested was the ability to masquerade as a user to check how a product looks from their perspective or help them with specific updates. Authentication Zero supports this through its --masqueradable option.

Let's finally take a quick look at how Authentication Zero compares to Devise and Auth0.

Authentication Zero Compared to Devise and Auth0

As mentioned in the introduction to this post, Devise is very popular in the Ruby on Rails ecosystem. Authentication Zero presents a significant interest to many: you get your authentication layer with the knowledge that it relies on good practices. Still, once generated and integrated into your code, any additions might need some complex gymnastics. Adding 2FA two years into a product's lifetime might be tricky. With Devise and Omniauth, it's mostly a matter of configuration.

In Auth0, your application has a very different approach, with almost no authentication layer. Instead, all authentication is done by Auth0, a third-party authentication provider. Integrating key security features such as 2FA involves activating them in Auth0 settings.

Wrapping Up

Authentication Zero can be a great option for your Ruby application, but it comes with a large bag of homework for you and your team:

Customization of UI (granted, that's also the case for Devise)
Backporting any updates that would come later in Authentication Zero
Manual integration of additional features once the code has been integrated

When integrating such vital elements, measure your team's needs and abilities. At the very least, Authentication Zero is an example of elegant Ruby and Ruby on Rails code. The fact that it integrates significant features of modern authentication layers (such as 2FA, passwordless, logs, sudo mode, and masquerading) makes it a promising solution for a team that doesn't want to add complexity from a large library, such as Devise.

Happy coding!

Advanced Queries in ActiveRecord for Ruby on Rails

Roy Tomeij — 2025-02-26T00:00:00+00:00

ActiveRecord: Rails' de facto ORM. We use it every day, and it makes life a lot easier; we type less, read less, have complexity hidden from us, and don't have to specify all kinds of boilerplate we'd otherwise have to.

But its most basic methods, easy-to-read symbol notation, and complete query abstraction aren't always up to the task at hand. As our apps grow, more complex data models and relationships follow, and the limitations of basic querying methods become apparent. Simple, short, and readable queries that once worked well become insufficient and need not just modification but rewriting. This progression calls for a deeper understanding and utilization of ActiveRecord's capabilities beyond the basic finders and associations. Often you need a combination of SQL and ActiveRecord to get the job done.

Here, we'll delve into advanced ActiveRecord queries, shining a spotlight on more complex joins, custom SQL, and strategic employment of database-specific features. This will help us better approach our data handling in Rails and meet the needs of our increasingly complex database relationships.

Complex Joins and Associations in ActiveRecord for Rails

What may begin as a simple one-to-one or one-to-many relationship can quickly expand into a labyrinth of interconnected models, necessitating a deeper dive into complex SQL joins to maintain efficient data retrieval and manipulation.

ActiveRecord supports more complex joins out of the box; you can easily join multiple tables together using pure AR and symbol notation:

# Join our single item from an order to its order, the transaction, and the seller profile, as well as the purchaser (user),
# finding all purchased items by a particular user from a particular seller
  PurchasedItem.joins(order: [:user, transaction: :seller_profile])
               .where(
                 orders: {users: {id: user_id},
                 transactions: {seller_profiles: {id: seller_profile_id}}}
               )

This is great and often suffices. But what happens the moment we realize we need some information from the transaction and the user too? We can't just .select(:id) and hope it'll somehow magically know we want the id of the Transaction and not the PurchasedItem.

This is where we need ActiveRecord to leverage SQL strings.

PurchaseItems.joins(...)
             .where(...)
             .select('desired, purchased_item, columns')
             .select('transactions.id AS transaction_id, transaction.total AS total_order_amount,
                      users.name AS purchaser_name, users.email AS purchaser_email')

Here we've selected whatever columns we wanted from each PurchasedItem (which is of course more efficient than simply selecting all of its columns, AR's default behavior). We've fetched the transaction IDs and total amounts associated with each individual purchased item of a given order, as well as the user's name and email.

We also don't always join exactly the way our models might expect, depending on our use case:

# Here we join on the users table in order to retrieve not the purchaser, but our sales member who was responsible for the sale,
# as well as joining the order on some special identifier our company uses internally
Transaction.joins('JOIN users AS sales_member ON sales_member.id = transactions.sales_member_id')
           .joins('JOIN orders ON orders.proprietary_internal_company_identifier = transactions.proprietary_internal_company_order_identifier')

However, when discovering our team's sales_members via the users table, sometimes a User is more than just a User. This leads us to our next topic: self-referential associations.

Self-Referential Associations

Self-referential associations define both sides of a relationship within the same model. This involves creating a join table that links two instances of a model to each other.

For example, in an organizational context, where users might have managers and sales_members, the User model can be configured to reflect these relationships:

class User < ApplicationRecord
  # Associates sales_members to a user where the user is the manager
  has_many :sales_members, class_name: "User", foreign_key: "manager_id"
  # Links a user to their manager, also a User
  belongs_to :manager, class_name: "User", optional: true
end

This allows us to traverse the hierarchy in both directions, allowing for queries that can retrieve a user's manager or their list of sales_members (e.g., user.manager or manager.sales_members). It also greatly simplifies self joins, so:

# return all users who have managers
User.joins(:manager)

Becomes:

SELECT users.* FROM users
INNER JOIN users ON users.id = users.manager_id

In this relatively simple example, we've simply placed a manager_id column on the User, but there's nothing stopping you from using join tables to create more complex many-to-many associations.

That said, keep in mind that, while powerful and often necessary, self-referential joins can lead to more complex queries. This isn't always avoidable, but if you don't actually need them, creating another model might be a better choice. As always, balance needs with the potential cost.

Database-Specific Features in ActiveRecord for Ruby on Rails

If you're on Rails, there's a pretty good chance you're using PostgreSQL. ActiveRecord, together with PostgreSQL, supports column types like JSON, JSONb, and arrays.

Do you have a particular model with many potentially optional columns? Or one that's likely to grow, where you'll need a lot of model-specific columns? This might be anything from user preferences, to video stats, to a literal need to just store pure JSON (like webhook responses).

If so, JSON or JSONb might be a good choice.

JSON vs. JSONb: Which is the right fit?

If you need to query them often, you should probably go with JSONb as it has more advanced querying options, and supports indexing — specifically, GIN indexing. JSONb uses more space and is slightly slower to write, but is much more efficient to retrieve: the "b" in JSONb stands for "binary". JSON is stored as a string, whereas JSONb is stored as binary, meaning it can be queried more efficiently, and doesn't require parsing when retrieved.

Here's an example of setting up and querying JSONb:

# note you will likely want to set a default to an empty hash, so you don't have to check if the value is NULL in order to attempt to access any keys
add_column :videos, :video_stats, :jsonb, default: {}

# using some ActiveRecord syntax:
Video.where('video_stats @> ?', {filename: "uploaded_video_1080p.mp4"}.to_json)

# alternatively, you might find a string more readable, or simply need it for more involved queries:
Video.where("
  video_stats ->> 'filename' LIKE '%mp4' AND
  (video_stats ->> 'duration')::float < 6.0 OR
  (video_stats ->> 'processed')::boolean = false"
)

You can see the appeal here: if you're hosting or cataloging video, there are all kinds of stats and metadata (relevant probably only to the videos themselves) that you might want to keep track of, for everything from bitrate, to resolution, to compression rate, video and codec type, upload and processed dates, fidelity, filetype, and so on. JSON can be an attractive choice for a model that demands a lot of columns and that may continue to grow as your app does.

Bulk Updates with `update_all`

While we might often retrieve many records at once, we often find ourselves updating records one by one. Sometimes, though, a group of records needs to be updated with the same value. Maybe we've just run a job that's processed a bunch of records in some way, perhaps by making API calls to an external payment service for each.

We could update them one at a time as we process them, but this puts unnecessary strain on our workers and database. Instead, we can simply use update_all, which updates all records it's called on with a given value per column:

# be sure to include `updated_at` if you've got timestamps and their value is important to you, as update_all doesn't do so automatically
@processed_records.update_all(processed: true)

This achieves an atomic update of all records involved in a single transaction. Nice!

We also have insert_all, introduced a few years ago by Rails 6. You will likely not use this as often, but it's still valuable, especially if you need to create a lot of records at once in real time on the main thread.

Maybe you allow your clients to upload their large orders to you. After processing their CSV, you construct their order:

# insert_all accepts an array of hashes.
# if you've got timestamps on a model, you need to include them both, or it will fail
new_orders = [
  {name: "Item 1", completed: false, created_at: Time.now, updated_at: Time.now},
  {name: "Item 2", completed: false, created_at: Time.now, updated_at: Time.now},
  ...,
  {name: "Item 15922", completed: false, created_at: Time.now, updated_at: Time.now},
]

Order.insert_all(new_orders)

Always consider whether or not you need callbacks or validations on the items you're updating.

Because these transactions are atomic, there's no way to trigger callbacks. Sometimes that's exactly what we want, which is perfect. If not, though, we might not be able to use update_all. If you need a callback (or, more rarely for a bulk update, a validation) to do something specific for each individual record, we probably can't.

Let's say you've only got a callback to ensure some other part of our app knows that at least one record of a particular type has been updated (in order to trigger a recount). You can probably do something like update_all on the first n-1 records, then .update on the last. This not only allows us to update everything at once, but it also avoids triggering potentially expensive callbacks on every single item.

Bonus: Got a mass-update scenario where you need to insert a lot of records at once? Check out upsert_all.

ActiveRecord for Ruby on Rails: Words of Caution

It goes without saying, but be mindful of potential SQL injection. Rails does a good job of protecting us from it most of the time, but it does give us enough rope to hang ourselves with. I personally like to often use single quotes for SQL strings to remind myself not to use string interpolation for things like params that users have control over.

Be mindful that JSON/JSONb columns are a single column and not self-documenting like the rest of your schema. The keys you add to any given JSON column are arbitrary, and you will have to document them yourself if you need to keep track of them all without having to hunt around your code for clues, because they're not going to show up in the schema.

Alternatively, you can use ActiveRecord's store_accessor, like this:

class Video < ApplicationRecord
  store :video_stats, accessors: [:filename, :duration, :processed]
  #...
end

This will allow you to both explicitly declare/document all the potential keys that exist in your JSON/JSONb column, and also access them directly:

#  like this:
video.duration

# rather than this:
video.video_stats["duration"]

Just be sure not to accidentally give any of the keys the same name as an existing column on the same model if you're going with this route! You can have JSON keys with the same names as columns, but if you're using this store_accessor method, you'll run into name collisions.

Lastly, as mentioned, always make sure that if you're using update_all, insert_all, or upsert_all, you either don't need any callbacks or validations, or that those requirements are met after the fact. In some cases, this could be as simple as firing off a job after you've updated or created these records. In other cases, there might not be an easy path.

Wrapping Up

Leveraging ActiveRecord's advanced features can help make our Rails apps more flexible, efficient, and maintainable. But as with anything, good things are rarely free; keep in mind that all things being equal, simpler is better. If you can accomplish the same task more simply, you probably should. It's always important to balance complexity with simplicity.

There's always more to learn when it comes to Rails, and ActiveRecord is no exception. The more tools you add to your arsenal, the more effective you'll be, and the better your applications will run.

Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

AppSignal’s Top 5 Ruby Posts in 2024

Roy Tomeij — 2024-12-17T00:00:00+00:00

As the year draws to an end, we're excited to share our top five most-read Ruby articles of 2024, in reverse order:

Season's Greetings ❆ ⛄

Have a wonderful festive break, and we'll see you in the new year!

If you haven’t already, don’t forget to subscribe to our Ruby Magic newsletter, so you never miss an upcoming post.

See you in the new year!

Server-sent Events and WebSockets in Rack for Ruby

Roy Tomeij — 2024-11-27T00:00:00+00:00

In the previous part of this series, we discovered how to create persistent connections in Rack in theory, but now we'll put what we learned into practice.

The web has two formalized specifications for communication over a persistent connection: server-sent events (SSEs) and WebSockets.

WebSockets are widely used and highly popular, but SSEs are far less well-known. Let's explore them first.

Server-sent Events

Server-sent events (SSEs) enable a client to hold an open connection with the server, but only the server can publish messages to the client. It isn't a bi-directional protocol.

SSEs are a JavaScript API, so let's modify our app to serve an HTML page with the required script:

class App
  def call(env)
    req = Rack::Request.new(env)
    path = req.path_info

    case path
    when "/"
      sse_js(env)
    end
  end

  private

    def sse_js(env)
      body = <<~HTML
        <!DOCTYPE html>
        <html>
          <head>
            <meta charset="utf-8">
            <meta name="viewport" content="width=device-width, initial-scale=1">
            <title>SSE - Demo</title>

            <script type="text/javascript">
              const eventSource = new EventSource("/sse")
              eventSource.addEventListener("message", event => {
                document.body.insertAdjacentHTML(
                  "beforeend",
                  `<p>${event.data}</p>`
                )
              })
            </script>
          </head>
          <body>
          </body>
        </html>
      HTML

      [200, { "content-type" => "text/html" }, [body]]
    end
end

The API is encapsulated in the EventSource class and new messages from the server trigger events that we listen for. Next, we need to build the endpoint that sends the events:

class App
  def call(env)
    req = Rack::Request.new(env)
    path = req.path_info

    case path
    when "/"
      sse_js(env)
    when "/sse"
      sse(env)
    end
  end

  private

    def sse_js(env)
        # ...
    end

    def sse(env)
      body = proc do |stream|
        Thread.new do
          5.times do
            stream.write "data: #{Time.now}!\n\n"
            sleep 1
          end
        ensure
          stream.close
        end
      end

      [200, { "content-type" => "text/event-stream" }, body]
    end
end

From a server point of view, this is fairly similar to the streaming bodies example we used in the previous part of this series. It's worth noting the content-type header and the string format written back to the client.

Run the server (make sure you switch back to Puma):

$ bundle exec puma

Open up localhost:9292 on your web browser, and you'll see the time written to the document five times at one-second intervals.

This technique is great when the server just needs to notify the client about updates. The above example is fairly contrived, though, as it uses a loop, so let's look at how we can use this technique in a real application.

Ruby `Queue`s

Ruby provides a Queue data structure for communication between threads. We can use that to publish data back to a client. Let's stick with the same use case of publishing the current time five times at one-second intervals, but now we'll publish from a background thread.

class App
  def call(env)
      # ...
  end

  private

    def sse_js(env)
        # ...
    end

    def sse(env)
      queue = Thread::Queue.new
      trigger_background_loop(queue)

      body = proc do |stream|
        Thread.new do
          loop do
            data = queue.pop
            stream.write "data: #{data}!\n\n"
          end
        ensure
          stream.close
        end
      end

      [200, { "content-type" => "text/event-stream" }, body]
    end

    def trigger_background_loop(queue)
      Thread.new do
        5.times do
          queue.push(Time.now)
          sleep 1
        end
      end
    end
end

In the above example, we spawn another background thread to push the current time to the queue every second. In the SSE thread, we call queue.pop, which blocks until something is added to the queue.

Using this technique, we can use a pub/sub system such as Redis to add data to the queue from a background thread, which is then published to the client.

That's SSEs covered! Next, let's look at WebSockets.

WebSockets

WebSockets are a bi-directional, full-duplex communication protocol that supports both binary and text data for client-server communication. They're widely used in the modern web and underpin Rails' Action Cable framework.

A WebSocket is created using an HTTP connection, but as a protocol, it's completely independent of HTTP.

To create a WebSocket connection, the client must make an HTTP request with these headers:

Connection: Upgrade
Upgrade: websocket

The server will respond with the status 101, meaning Switching Protocols. The TCP connection used for the HTTP request is upgraded to a WebSocket connection.

We won't get into the nitty-gritty of the WebSocket protocol in this post. It's fairly fiddly since it's a binary protocol. If you're curious, Starr Horne has written an amazing article on WebSockets.

Let's look at how to upgrade a TCP socket to a WebSocket connection.

Upgrading from HTTP to WebSockets

As described above, we'll need to send a 101 response. After this, we'll write to the socket using WebSockets' binary protocol for the communication to work.

require 'digest/sha1'

class App
  def call(env)
    req = Rack::Request.new(env)

    key = req.get_header("HTTP_SEC_WEBSOCKET_KEY")
    response_key = Digest::SHA1.base64digest([key, "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"].join)

    body = proc do |stream|
      response = "Hello world!"
      output = [0b10000001, response.size, response]
      stream.write output.pack("CCA#{ response.size }")
    ensure
      stream.close
    end

    [101, { "Upgrade" => "websocket", "Connection" => 'upgrade', "Sec-WebSocket-Accept" => response_key }, body]
  end
end

We have to create a response key to securely create the connection. The UUID used to generate it is a global constant found in the specification. We won't go into the binary format of the string we're writing into the WebSocket connection, but it's all described in 'Building a simple websockets server from scratch in Ruby' by Starr Horne if you're curious.

Demo

Run the server:

$ bundle exec puma

The easiest way to create a connection is to use a WebSocket client. I recommend websocat.

$ websocat ws://127.0.0.1:9292/

You'll see the string Hello world! printed out! The connection is now active. In theory, we can write and receive messages over this socket now. We still need to implement receiving or publishing messages on the server for it to work in practice, but that's for another post.

A final word of warning: always remember that persistent connections come with challenges when using a threaded web server like Puma. A persistent connection ties up a thread and can cause significant performance issues unless you open a sizeable can of worms to implement your own threading mechanism.

Wrapping Up

That concludes our three-part deep dive into Rack! We first looked at how to set up a basic Rack app, before diving into socket hijacking for persistent connections.

Lastly, in this part, we used two specifications provided by our web platform to communicate over persistent connections: server-sent events (SSEs) and WebSockets.

I hope you've found this series useful. Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Rack for Ruby: Socket Hijacking

Roy Tomeij — 2024-11-20T00:00:00+00:00

In the first part of this series, we set up a basic Rack app, learned how to process a request and send a response.

In this post, we'll take over connections from Rack and hold persistent connections to enable pathways such as WebSockets.

First, though, let's look at how an HTTP connection actually works.

HTTP Connections

As this diagram shows, a TCP socket is opened, and a request is sent to a server. The server responds and closes the connection. All communication is in plain text.

Using a technique called socket hijacking, we can take control of a socket from Rack when a request comes in. Rack offers two techniques for socket hijacking:

Partial hijack: Rack sends the HTTP response headers and hands over the connection to the application.
Full hijack: Rack simply hands over the connection to the client without writing anything to the socket.

Partial Hijacking

This is how you do a partial hijack:

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain", "rack.hijack" => body }, []]
  end
end

rack.hijack is a Rack header, set in the same Hash as the HTTP response headers. Rack will look for such headers and process them as per the specification, instead of writing them to the HTTP response.

Run the above app and curl to it. You'll see that it writes the time at one-second intervals.

$ curl -i localhost:9292

Full Hijacking

This is how you'd do a full hijack:

class App
  def call(env)
    headers = [
      "HTTP/1.1 200 OK",
      "Content-Type: text/plain"
    ]

    stream = env["rack.hijack"].call
    stream.write(headers.map { |header| header + "\r\n" }.join)
    stream.write("\r\n")
    stream.flush

    begin
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [-1, {}, []]
  end
end

In this case, we call the proc passed to us using the rack.hijack key, instead of setting one ourselves in the response. This gives us complete control over the socket. At the end, we return an array with the status -1 only because Rack expects an array to be returned. The contents of this array are ignored since we've taken over the socket.

This is a bad practice, rife with gotchas and weird behavior. Don't do it. Samuel Williams, who is a maintainer of Rack, recommends against it as well.

Streaming Bodies in Rack for Ruby

While full hijacking is a terrible idea, partial hijacking is a useful tool. But it still feels hacky, so Rack 3 formally adopted that approach into the spec by introducing the concept of streaming bodies.

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Here we provide a block as the response body rather than an array. Rack keeps the connection open until the block finishes executing.

There's a huge gotcha here when using Puma. Puma is a multi-threaded server that assigns a thread to each incoming request. We're taking over the socket from Rack, but we're still tying up a Puma thread as long as the connection is open.

Puma concurrency can be configured, but threads are limited, and tying one up for long periods is not a good idea. Let's see this in action first.

$ bundle exec puma -w 1 -t 1:1

In two separate terminal windows, run the following command at the same time:

$ curl localhost:9292

One request is immediately served, but the other is held until the first one completes. This is because we started Puma with a single worker and single thread, meaning it can only serve a single request at a time.

We can get around this by creating our own thread.

class App
  def call(env)
    body = proc do |stream|
      Thread.new do
        5.times do
          stream.write "#{Time.now}\n\n"
          sleep 1
        end
      ensure
        stream.close
      end
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Now if you try the above experiment again, you'll see both curl requests are served concurrently because they don't tie up a Puma thread.

Once again, I must warn against this approach, unless you know what you're doing. These demonstrations are largely academic, as systems programming is a deep and complex topic.

Falcon Web Server

Since the threading problem is specific to the Puma web server, let's look at another option: Falcon. This is a new, highly concurrent Rack-compliant web server built on the async gem. It uses Ruby Fibers instead of Threads, which are cheaper to create and have much lower overhead.

The async gem hooks into all Ruby I/O and other waiting operations, such as sleep, and uses these to switch between different Fibers (ensuring a program is never held up doing nothing).

Revert your app to the previous version where we're not spawning a new thread:

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Then remove Puma and install Falcon.

$ bundle remove puma
$ bundle add falcon

Run the Falcon server. We need to explicitly bind it because it only serves https traffic by default.

$ bundle exec falcon serve -n 1 -b http://localhost:9292

The server only uses a single thread, which you can confirm with the command below. You'll need to grab your specific pid from Falcon's logs.

$ top -pid <pid> -stats pid,th

The thread count printed by the above command will be 2 because the MRI uses a thread internally.

Try the earlier experiment again and run two curl requests simultaneously.

$ curl localhost:9292

You'll see they're both served at the same time, thanks to Ruby Fibers!

Falcon is relatively new. Ruby Fibers were only introduced in Ruby 3.0. Since Falcon is Rack-compliant, it can be used with Rails too, but the docs recommend using it with v7.1 or newer only. As such, it's a bit risky to use Falcon in production but it's a very exciting development in the Ruby world, in my opinion. I can't wait to see its progress in the next few years.

We've now learned how to create persistent connections in Rack and how to run them without blocking other requests, but the use cases so far have been academic and contrived. In the next and final part of this series, we'll examine how we can use this technique in a practical way.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

The Basics of Rack for Ruby

Roy Tomeij — 2024-10-30T00:00:00+00:00

Rack is the foundation for every popular Ruby web framework in existence. It standardizes an interface between a Ruby application and a web server. This mechanism allows us to pair any Rack-compliant web server (such as Puma, Unicorn, or Falcon) with any Rack-compliant web framework (like Rails, Sinatra, Roda, or Hanami).

Separating the concerns like this is immensely powerful and provides a lot of flexibility. It does, however, also come with limitations.

Rack 2 operated on the assumption that every request must provide a response and close the connection. It made no facility for persistent connections to enable pathways like WebSockets.

Developers had to make use of a hacky escape hatch to take over connections from Rack to implement WebSockets or similar persistent connections.

This all changed with Rack 3. But first, let's backtrack and take a closer look at Rack itself.

A Barebones Rack App

A basic Rack app looks like this:

class App
  def call(env)
    [200, { "Content-Type" => "text/plain" }, ["Hello World"]]
  end
end

run App.new

env is a hash containing request-specific information such as HTTP headers. When a request is made, the call method is called, and we return an array representing the response.

The first element is the HTTP response code, in this case 200. The second element is a hash containing any Rack and HTTP response headers we wish to send. Finally, the last element is an array of strings representing the response body.

Let's organize this app into a folder and run it.

$ mkdir rack-demo
$ cd rack-demo
$ bundle init
$ bundle add rack rackup
$ touch app.rb
$ touch config.ru

Fill in app.rb with the following:

class App
  def call(env)
    [200, { "content-type" => "text/plain" }, ["Hello World"]]
  end
end

And config.ru with:

require_relative "app"

run App.new

We can run this app using the default WEBrick server by running:

$ bundle exec rackup

The server will run on port 9292. We can verify this with a curl command.

$ curl localhost:9292
Hello World

That's got the basic app running!

Changing Web Servers

WEBrick is a development-only server, so let's swap it out for Puma:

$ bundle add puma

Now try running rackup again. You'll see it has automatically detected Puma in the bundle and started that instead of WEBrick!

$ bundle exec rackup
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 45877
* Listening on http://127.0.0.1:9292
* Listening on http://[::1]:9292
Use Ctrl-C to stop

I recommend starting Puma directly instead of using rackup, as that allows us to pass configuration arguments should we want to. The -w 4 below starts Puma using 4 workers, meaning 4 instances of Puma are started up simultaneously.

$ bundle exec puma -w 4
[45968] Puma starting in cluster mode...
[45968] * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
[45968] *  Min threads: 0
[45968] *  Max threads: 5
[45968] *  Environment: development
[45968] *   Master PID: 45968
[45968] *      Workers: 4
[45968] *     Restarts: (✔) hot (✔) phased
[45968] * Listening on http://0.0.0.0:9292
[45968] Use Ctrl-C to stop
[45968] - Worker 0 (PID: 45981) booted in 0.0s, phase: 0
[45968] - Worker 1 (PID: 45982) booted in 0.0s, phase: 0
[45968] - Worker 2 (PID: 45983) booted in 0.0s, phase: 0
[45968] - Worker 3 (PID: 45984) booted in 0.0s, phase: 0

This basic app demonstrates the Rack interface. An incoming HTTP request is parsed into the env hash and provided to the application. The application processes the request and supplies an array as the response that the server formats and sends to the client.

Rack Compliance In Frameworks

Every compliant web framework follows the Rack spec under the hood and provides an access point to go down to this level.

In Rails, we can send a Rack response in a controller:

class HomeController
  def index
    self.response = [200, {}, ["I'm Home!"]]
  end
end

Similarly, in Roda:

route do |r|
  r.on "home" do
    r.halt [200, {}, ["I'm Home!"]]
  end
end

Every Rack-compliant framework will have a slightly different syntax for accomplishing this, but since they're all sending Rack responses under the hood, they will have an API for you to access that response.

You can find the full, relatively accessible Rack specification on GitHub.

Wrapping Up

As this demo shows, Rack operates under the assumption that a request comes in, is processed by a web application, and a response is sent back. Throwing persistent connections into the mix totally breaks this model, yet Rack-compliant frameworks like Rails implement WebSockets.

In the next post, part two of a three-part series, we'll cover how to take over connections from Rack so we can hold persistent connections to enable pathways such as WebSockets.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Optimize Database Performance in Ruby on Rails and ActiveRecord

Roy Tomeij — 2024-10-30T00:00:00+00:00

In Rails, we're more likely to use SQL databases than other frameworks. Unlike NoSQL databases, which can be scaled horizontally with relative ease, SQL databases like PostgreSQL or MySQL are much less amenable to easy scaling.

As a result, our database usually becomes the primary bottleneck as our business grows. Although SQL databases are very efficient, as our growing customer base puts an increasing load on our servers, we begin scaling our instance counts, workers, etc. But we can't just make copies of our database for each new server we spin up. This makes optimizing database performance critical for any serious Rails project.

In this post, we'll explore strategies for optimizing performance to minimize the load on our database. We'll start with some more basic topics like eager loading and the N+1 query problem, database indexing, select, pluck, and immediate loading. Then, we'll move on to more involved topics like performance profiling, scaling via techniques like database sharding, and background jobs using read replicas.

Let's get going!

Getting ActiveRecord N+1s Out of the Way

No discussion of database performance involving ActiveRecord (or any ORM) is complete without addressing the infamous N+1 problem. While most readers are likely familiar with it, it's still worth revisiting, as N+1 queries can (and do) creep into our projects over time, especially as our codebase evolves and we add or update features.

The N+1 problem generally occurs when iterating through a (potentially large) group of retrieved records. Because we haven't loaded their association/s, each loop means an additional query for each associated model we access inside the loop. This can quickly spiral out of control, resulting in a staggering number of queries and causing serious consequences such as crashed pages, an exhausted database connection pool, and memory running out, ultimately grinding our site to a halt.

There are a few useful tools at your disposal to help identify and resolve N+1 problems. The first is simply looking at your server output; generally, this works pretty well, as N+1s are easy to spot. For a more assisted approach, the Bullet gem is a popular tool that automatically detects N+1s in applications and suggests ways to fix them. Another, arguably better option is prosopite, a less well-known option that generally provides better results with fewer false positives (and false negatives).

Eager loading should be used with care, however; while you're probably safe loading has_one associations, has_many can sometimes be dangerous: what happens when we've got a query of multiple joined, many-to-many tables, and eager load one or more of the many-to-many associations?

We can end up loading way too many records into memory. This can crash our app just as surely as a bad N+1. If you're at this point, and both the N+1 and eager loading approaches are causing you problems, it may be time to reevaluate the query itself. Maybe you're trying to load associations you just want to COUNT (which is something you might be able to build into your query), tighten your pagination limits, or consider offloading the query to a background job if possible (which we'll cover).

Database Performance in Rails: Considerations and Optimization

Sometimes, breaking queries down into a couple of steps can both improve performance and simplify a query. Ever find yourself trying to get a subset of records to complete a mind-bending query? In certain cases, just finding the IDs of the subset you want and then feeding them into a second, less complex query can help reduce the need for complex joins, subqueries, etc.

Have you ever fallen victim to Rails' (usually beneficial) default lazy-loading and evaluation, and wished you could load something immediately rather than the first time it's used? Sometimes we need not only the results of a query, but another aspect of it too, like the number of records we've retrieved. Our seemingly harmless code looks like this:

@users = User.where("email ILIKE ?", search)
@total_users = @users.size

Normally, this is fine, but what if we need to count users before we do anything with them (maybe we're displaying the number of users returned in our live search at the top of the page, followed by the users themselves)? We might add COUNT to our query, but then we need to retrieve the records again.

We can get around this by explicitly calling load on our query — for example:

@users = User.where("email ILIKE ?", search).load
@total_users = @users.size

This simple change can save us a query. With our users already loaded, our display of @total_users above the user list will no longer trigger an expensive COUNT followed by another query to get users.

In the other direction, we have load_async with Rails 7. This allows us to asynchronously query the database with multiple requests at once, rather than being locked to one synchronous query at a time. This can be a total game-changer if you're in a situation that can benefit from it. Be careful though, as this (probably unsurprisingly) uses Ruby threads under the hood, so comes with all the same issues you'd expect from threads. That's not to mention the fact that it (potentially) opens our database up to a greatly increased number of connections, and can exhaust our connection pool if we aren't careful.

A large part of database performance in Rails comes down to how you utilize ActiveRecord. While AR is an invaluable tool, it can also lead to performance issues if not used with care. As you're probably aware, it loads every column of every table involved in a query (SELECT *) by default, regardless of how much of that data we actually use. It's pretty easy to forget this though, especially as our queries evolve, and we can end up loading a lot of unnecessary data. It's important to consider whether or not you're really using all of the columns you're loading, or if a select (or even a pluck) statement might be a better fit for your use case.

You're probably familiar with eager loading in Rails, but one thing that's often overlooked is that Rails makes it easy to introduce optional eager-loading (and WHERE clauses, and SELECTs). If you've ever had slightly different requirements for two very similar queries, you might feel like you're left with two options: write two queries, or write covering both use cases, each retrieving more than you need. This might happen in a before_action, where you typically want the same thing for each action in a particular controller. But one controller might benefit from eager loading, more careful column selection, or a search results page which is just filtering the usual index based on some criteria.

There's another way, though:

class User
  def search_user_posts(before_date: nil, optional_selects: [], optional_eager_loads: [], exclude_inactive: false, search: false)
    optional_before_date_constraint = ["created_at < ?", before_date] if before_date
    optional_joins = {user_posts: [posts: :comments]} if search

    self.posts
        .includes(optional_eager_loads)
        .joins(:shared_joins)
        .joins(optional_joins)
        .select(:columns_shared_by_queries)
        .select(optional_selects)
        .where("your shared WHERE constraints")
        .where(optional_before_date_constraint)
  end
end

Rails is smart enough to know not to execute anything here if there's nothing passed in; where(nil) will behave as though it doesn't exist, as will includes([]), etc. This can really clean up your code and help you tailor performance to your needs.

Finally, rethinking your JOINs and WHEREs can sometimes be beneficial. Are you querying a very large dataset where using a JOIN could significantly reduce the initial result set? Try it out, and don't be afraid to run #explain on your query. It can help you determine if switching to a JOIN might work better and identify any missing indexes on critical queries.

Database Indexing

One of our most important (and nearly universally necessary) tools for database performance is indexing. As you're probably aware, indexes are special data structures (typically B-trees) that a database uses to quickly find records, improving retrieval times from O(n) to O(log(n)). However, it's important to add indexes carefully, as indexing columns unnecessarily can actually harm performance.

The trick is to identify the columns and associations that your application needs to query frequently, and then create indexes on those. This allows the database to quickly locate the relevant data without having to scan the entire table.

You can also use partial indexing. Partial indexes allow you to index a subset of a table's rows based on a specified condition. This is particularly useful when your queries frequently target only a specific portion of the data.

For example, maybe your app has a core of frequent visitors, the majority of whom have their own accounts. They make up the lion's share of your traffic, but represent a small minority of your total users. You could define a partial index on the users table like this:

class AddPartialIndexToUsersOnGuest < ActiveRecord::Migration[7.1]
  def change
    add_index :users, :guest, where: "(guest = false)" # the `where: condition` here makes this a partial index.
  end
end

Here, we've specifically targeted rows where guest is false. This optimizes the performance of any query searching for non-guest users by reducing the total rows covered by the index.

It's important to revisit your indexing profile and database schema periodically. The specific indexing needs of your application may change over time as the user base, codebase, and feature set evolve. Doing so often means you end up reviewing your indexes during comfortable periods where you have time to analyze and think about your database, rather than in a panic when an influx of traffic crashes your app.

All of this brings us to our next topic: performance profiling.

Performance Profiling for Your Ruby on Rails App

So, we've identified some major categories of performance optimization; what now? Do we go through our entire app from top to bottom, optimizing line by line? Probably not.

It's pretty unlikely that you have evenly distributed traffic across your whole site. It's even less likely that you can afford to go through your entire codebase; premature optimization may not actually be the root of all evil, but you also don't have time for it. So how do we know what to focus on?

This is where profiling tools come in. While your local server output can provide valuable performance insights into response times, memory use, and query performance, you are only one user. Your local database likely doesn't reflect the scale or patterns of your production database. Valuable as it is for spotting potential performance issues, your local environment doesn't provide the context you need to make decisions about what genuinely needs your attention.

Application Performance Monitoring tools (APMs) like AppSignal for Ruby are essential for monitoring and maintaining our site's performance. They provide comprehensive insights not just at the database layer, but across the rest of the stack, allowing us to pinpoint specific areas (pages, endpoints, and even specific queries) that are causing headaches. With a good APM tool, we can track the performance of any request over various timescales, monitor background jobs, spot memory leaks, set up custom error and performance alerts, and more.

APMs provide visualization tools that make it much easier to spot and understand complex performance trends, leading to quicker, more informed decisions. They help prioritize optimization efforts by highlighting not only the slowest queries and most resource-intensive parts of an application, but also the endpoints that consume the most total time (total request count * average response time). This means we can focus on what's actually important. Remember that page we were worried about that takes a couple of seconds to load? Turns out, its total impact on our site amounts to practically nothing. Our products page though, clocking in at a reasonable 150ms? That's taking up 15% of our total server time.

To find out more about measuring database performance using AppSignal, check out our post Monitor the Performance of Your Ruby on Rails Application Using AppSignal.

Background Jobs

It isn't always an option, but running queries in background jobs is a powerful tool for improving performance.

You might be asking yourself how this could be, given that we only have one database; what difference does it make if we're merely calling on it from another thread or process? But there are a few main ways background jobs can come to our database's rescue.

First, just by nature of the fact that our background jobs run asynchronously, we're pretty free to run relatively resource-intensive tasks — like setting up bulk inserts (or upserts), for example (note: neither validations nor callbacks are triggered for bulk operations) — and potentially distributing large numbers of queries across periods of time (perhaps especially for write operations), possibly with the use of find_each.

You're likely used to using a job to process a CSV or spreadsheet; those tasks tend to lend themselves to asynchronous processing as well as bulk insertions. If you're doing this on your server's main thread, you're likely doing something wrong. But sometimes other, less obvious things we're currently doing on our main thread can be moved to a background job to improve performance on the main thread and reduce load on the database.

Consider a scenario where your site allows people to make potentially large purchases of relatively inexpensive items. A school, for example, might order tens of thousands of pencils at once, and maybe your system creates a PurchaseItem for every single item purchased. Instead of creating or updating thousands of rows one at a time, we can set up a bulk insert in a background job, create all the individual hashes inside a loop, and use insert_all to get the job done — taking advantage of our background worker to set up our bulk inserts, and then writing to the database once, instead of locking it with thousands of writes.

Read Replicas

Background jobs are great, but eventually, even if we're spacing out those large, numerous database queries, we hit a point where we're asking too much of our database. We're probably okay in terms of write performance (we tend to read from our database far more than we write to it, after all), but we're constantly bombarding it with queries. Our site's response times are steadily rising, and user experience is suffering.

At this point, the next logical step may be to introduce read replicas to offload work from the primary database. Read replicas allow you to distribute the read query load across one or more replicas of your database. This can improve performance across the board, as it greatly reduces the number of reads (and thus the total I/O) of your primary database, leaving it less swamped by requests (most of which are likely reads in the first place).

Ideally, large database operations are performed in background jobs reading from replicas, setting up a bulk insert or upsert, and then making one atomic write to your main database. This means you can potentially raise not only the read performance of your site, but can also boost write performance to some degree, as your primary database is consistently freer to accept write requests.

While read replicas are not painless to set up, they can provide the extra performance our site needs.

Database Sharding At A High Level

Remember how we said SQL databases don't scale horizontally? Technically, they can scale horizontally through database sharding. Sharding involves splitting up your database into multiple databases, which are then distributed across multiple servers (and often across geographical locations).

There are several approaches for doing this, including:

Geographical sharding (based on user locations).
Vertical sharding (splitting into different tables or groups of tables based on access patterns, with each shard storing a different subset of the database’s tables).
Functional sharding (splitting based on business function; like separating out billing from user content).

And more. Each approach has its use cases, and some naturally lend themselves better to some businesses than others.

As you've doubtlessly realized by now, sharding is generally difficult, and depending on your business, sometimes not even possible. It adds complexity, developmental and maintenance overhead to our apps, and potentially a lot of it. The good news is that if you don't have the resources to cover this, you probably don't need it in the first place. The bad news is that even if you do, it still complicates things and means increased costs (presumably balanced by the potential benefits).

But if you've done all you can with query optimization, read replicas, and caching, sharding might just be the next necessary step — and one that can greatly increase your I/O capability.

Words of Caution and Final Words: Database Optimization in Ruby on Rails

We need to be conscientious in how we approach our optimizations. It's possible to over-index (or index the wrong things, which hurts write performance). In the same way that N+1s can get out of control if we're not careful, so can eager loading, and it's not always immediately obvious when we're developing locally (hence the importance of at least approximating your real-world site locally).

Keep an eye on things via an APM, as traffic and user patterns can change and new ones can emerge, not just over time but potentially week to week and season to season, depending on your business.

There's no one-size-fits-all approach, so we have to be mindful of how we tailor our optimizations to our own site. Remember not to waste too much time trying to optimize things until you have an idea of the impact they'll have on your site.

That doesn't mean you shouldn't try to write generally efficient code, just that there's generally no reason to agonize over tweaking things to be absolutely optimal or wasting time on micro-optimizations. Try to keep the time-to-benefit ratio in mind, and how what you're working on will likely fit into your existing site as it is now, and as it evolves.

If you're interested in diving deeper into aspects of database optimization like sharding, you can learn more from the Rails guides. Also, check out Paweł Urbanek's great article on load_async.

Happy optimizing!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

What's New in Ruby on Rails 8

Roy Tomeij — 2024-10-07T00:00:00+00:00

The first Rails 8 beta has officially been released, bringing an exciting set of features, bug fixes, and improvements. This version builds on the foundation of Rails 7.2, while introducing new features and optimizations to make Rails development even more productive and enjoyable.

Key highlights include an integration with Kamal 2 for hassle-free deployments, the introduction of Propshaft as the new default asset pipeline, and extensive ActiveRecord enhancements. Rails 8 also brings several SQLite integration upgrades that make it a viable option for production use.

Let's dive in and explore everything that Rails 8 has to offer!

Effortless Deployments with Kamal 2 and Thruster

Rails 8 makes deploying your applications simple with Kamal 2 and Thruster.

Kamal 2 reduces the need for reliance on managed cloud services and Platform as a Service (PaaS) platforms by enabling quick and easy deployment to cloud VMs, bare metal servers, or VPS environments in just minutes.

With a single command (kamal setup), you can set up a production-ready Rails environment on a standard Linux box, making deployment both easy and cost-effective.

Kamal 2 also integrates with Thruster, a custom proxy built specifically for Rails that enables zero-downtime deployments, HTTP/2 support, automated SSL with Let's Encrypt, Gzip compression, and easy hosting of multiple apps on a single server — all without complex setup.

With Kamal 2 and Thruster, Rails 8 makes deploying apps easier than ever. And if you prefer a different deployment setup, you can opt out using the --skip-kamal flag to maintain your existing workflows.

Leaner Rails Deployments with Solid Adapters

One of the big improvements in Rails 8 is simpler deployments by reducing the number of additional services needed to implement common web application requirements.

Traditionally, if you required features like job queues, caching, and pub/sub messaging, you'd use a combination of a database like PostgreSQL with Redis for auxiliary functions.

With Rails 8, you can handle all these with just SQLite, thanks to three new database-backed adapters: Solid Cable, Solid Cache, and Solid Queue.

Solid Cable is Rails' new default Action Cable adapter in production and means you can drop the common dependency on Redis. It acts as the pub/sub server, relaying messages between the app and connected clients using fast polling through SQLite. Despite polling, Solid Cable's performance is comparable to Redis in most situations.
Solid Cache replaces the need for Redis by using disk storage instead of RAM for caching. This approach allows for much larger, more cost-effective caches that persist longer and handle more requests without sacrificing performance. It also supports encrypted storage and retention policies to meet privacy requirements.
Solid Queue replaces Redis for Active Job background processing, using the FOR UPDATE SKIP LOCKED mechanism for efficient job handling (compatible with PostgreSQL, MySQL, or SQLite). It includes essential features like concurrency control, retries, and recurring jobs, and has proven itself at HEY, where it now manages 20 million jobs per day.

These three adapters are designed around a simple idea: modern SSDs and NVMe drives are fast enough to handle many tasks that previously required in-memory solutions. By tapping into these speedy drives, Rails cuts out the need for separate RAM-based tools like Redis.

SQLite is Ready for Production

Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver.

With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or testing environments.

Here are some key improvements to the SQLite integration in Rails 8:

Full-text search and virtual tables are now supported using create_virtual_table.
The adapter now allows bulk insert fixtures for enhanced data seeding performance.
Transactions default to IMMEDIATE mode to improve concurrency.
Enhanced error handling by translating SQLite3::BusyException into ActiveRecord::StatementTimeout.

A New Era for the Asset Pipeline with Propshaft

Rails 8 also introduces Propshaft as the new asset pipeline default, replacing the long-standing Sprockets system. Sprockets served Rails developers well for over a decade, but it was designed in a different era — before the explosion of JavaScript build tools and modern browser improvements.

Propshaft reflects a simpler, modern approach to managing assets, built around the core needs of today's developers. Its purpose is straightforward: to provide a clear path for assets and apply digest stamps for caching.

Unlike Sprockets, which took on numerous additional tasks, Propshaft focuses only on what's essential, fitting naturally with the new Rails philosophy of keeping asset pipelines lean (while complex JavaScript handling is left to specialized tools like Esbuild or Vite).

Built-In Authentication Made Simple

Rails has been building the key components of authentication over the years, from has_secure_password in Rails 5 to normalizes, generates_token_for and authenticate_by in Rails 7.1.

With Rails 8, all these components come together to give you a straightforward starting point for building a secure, session-based authentication system.

By running a single command, you can set up all the essentials for an authentication system with database-backed sessions and password reset functionality:

bin/rails generate authentication

This command generates key files, including models, controllers, mailers, and views:

app/models/current.rb
app/models/user.rb
app/models/session.rb
app/controllers/sessions_controller.rb
app/controllers/passwords_controller.rb
app/mailers/passwords_mailer.rb
app/views/sessions/new.html.erb
app/views/passwords/new.html.erb
app/views/passwords/edit.html.erb
app/views/passwords_mailer/reset.html.erb
app/views/passwords_mailer/reset.text.erb
db/migrate/xxxxxxx_create_users.rb
db/migrate/xxxxxxx_create_sessions.rb
test/mailers/previews/passwords_mailer_preview.rb

This effectively puts you on the fast track to secure, production-ready authentication. All that's left is to integrate a user sign-up flow customized to your application's needs.

New Script Folder and Generator

Rails 8 introduces a new script folder dedicated to holding one-off or general-purpose scripts, such as data migrations, cleanup tasks, or other utility operations. This addition helps organize these scripts neatly, keeping them separate from your main application logic.

To make script creation easier, a new script generator is available. You can generate scripts with a simple command:

bin/rails generate script my_script

These commands create the corresponding script files, which you can then execute with:

bundle exec ruby script/my_script.rb

This streamlined approach keeps your application organized and makes handling custom scripts more convenient and maintainable.

A Slew of Active Record Improvements

Active Record has also seen major enhancements in Rails 8 to improve performance, simplify migrations, improve troubleshooting, and provide better support for complex database use cases.

Below are some of the key changes introduced in this latest version:

Rails 8 now distinguishes between float4 and float8 in PostgreSQL.
drop_table now supports dropping multiple tables at once.
Support for advanced options when creating a table with PostgreSQL, including inheritance and partitioning.
Bulk inserts of fixtures are now supported to improve data seeding performance.
Migrating a fresh database now starts by loading the database schema before running the migrations.
create_schema and drop_schema operations are now reversible.
Rails 8 now requires MySQL 5.6.4 or later due to advancements like datetime with precision.
Query log tags are enabled by default in development environments to trace SQL statements back to the application code and identify which database is in use.

Wrapping Up

Rails 8 introduces a range of impactful updates, from easier deployments with Kamal and a modern asset pipeline to significant ActiveRecord enhancements and improved production capabilities for SQLite.

These advancements not only boost developer productivity but also align with modern best practices, allowing you to focus on building your application instead of dealing with infrastructure complexities.

For a detailed list of all the new features, optimizations, and changes, check out the official Rails 8 release notes.

If you want to get involved with contributing to Rails, visit the Rails GitHub repository to explore open issues and review the contribution guidelines.

Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Measuring the Impact of Feature Flags in Ruby on Rails with AppSignal

Roy Tomeij — 2024-10-02T00:00:00+00:00

Feature flags are a powerful tool in software development, allowing developers to control the behavior of an application at runtime without deploying new code. They enable teams to test new features, perform A/B testing, and roll out changes gradually.

In Ruby on Rails, feature flags can be managed using diverse tools, the most popular being the Flipper gem. This article will explore implementing and measuring the impact of feature flags in a Solidus storefront using Flipper and AppSignal's custom metrics.

What Are Feature Flags in Rails, Again?

If you are looking for an introduction to the subject, check out the post Add Feature Flags in Ruby on Rails with Flipper.

In a nutshell, though, feature flags are a way to influence how your application behaves at runtime, without having to deploy new code. The simplest type of feature flags are environment variables. Every Ruby on Rails application uses them out of the box. One example is the configuration of application server concurrency using ENV['WEB_CONCURRENCY'].

However, there are other ways to manage feature flags, such as using a persistence layer like ActiveRecord or Redis. A comprehensive way to do this is offered by the Flipper gem.

The following snippet exemplifies how the performance_improvement feature flag is evaluated for a given user:

@categories = if Flipper.enabled?(:performance_improvement, user)
  Category.all.includes(:products)
else
  Category.all
end

Next, we will set up a Solidus storefront to start experimenting with feature flags.

Our Example App: A Solidus Storefront

To measure the impact of feature flags in a somewhat realistic scenario, let's quickly bootstrap a Solidus store:

rails new coder_swag_store && cd coder_swag_store
bundle add solidus
bin/rails g solidus:install

This generator will guide you through the process and ask you a few setup questions.

Choose the starter frontend when queried for the frontend type.
Skip setting up a payment method.
Choose to mount your Solidus application at /, since we are using it as a standalone app.

Afterward, run bin/dev from your terminal and you should be good to go. When you go to http://localhost:3000, you'll see this screen:

Implement Feature Flags with Flipper

Now let's implement two exemplary use cases for feature flags:

A performance improvement.
An attempt at conversion rate optimization.

First of all, though, we have to add the flipper gem along with its active_record storage adapter:

bundle add flipper
bundle add flipper-active_record
bin/rails g flipper:setup
bin/rails db:migrate

This will set up the required database tables to look up Flipper "gates", i.e., concrete conditionals to evaluate when checking a feature flag.

Testing a Performance Improvement

To assess this scenario, we will simulate a slow request in the storefront by adding a sleep 1 call for the unoptimized case:

# app/controllers/products_controller.rb

def index
  @searcher = build_searcher(params.merge(include_images: true))
  @products = @searcher.retrieve_products

  if Flipper.enabled?(:performance_improvement)
    # 🚅
  else
    sleep 1
  end
end

Now, we will use a "percentage of time" strategy to roll out the optimization across a random set of requests. Open a Rails console and key in the following:

Flipper.enable_percentage_of_time(:performance_improvement, 50)

Using the oha load testing tool, we can confirm that indeed half of the requests take one second longer than the others:

$ oha http://localhost:3000/products -z 30s -q 2
Summary:
  Success rate: 100.00%
  Total:        30.0037 secs
  Slowest:      1.2662 secs
  Fastest:      0.1049 secs
  Average:      0.6260 secs
  Requests/sec: 2.0664

  Total data:   2.59 MiB
  Size/request: 44.24 KiB
  Size/sec:     88.47 KiB

Response time histogram:
  0.105 [1]  |■
  0.221 [24] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.337 [8]  |■■■■■■■■■■
  0.453 [0]  |
  0.569 [0]  |
  0.686 [0]  |
  0.802 [0]  |
  0.918 [0]  |
  1.034 [0]  |
  1.150 [6]  |■■■■■■■■
  1.266 [21] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■

Testing Conversion Rate Optimization

When dealing with user-facing features, for example, changes in the UI, it's often advisable to use a "percentage of actors" strategy to roll out flags. This way, every user is consistently offered the same experience.

So to start, we'll create two users for our e-commerce application. Fire up a Rails console and issue the following commands:

Spree::User.create(email: "test1@example.com", login: "test1@example.com", password: "super_safe_password", password_confirmation: "super_safe_password")
Spree::User.create(email: "test2@example.com", login: "test2@example.com", password: "super_safe_password", password_confirmation: "super_safe_password")
Flipper.enable_percentage_of_actors(:conversion_rate_optimization, 50)

This creates two sample users and ensures that the feature flag is consistently enabled for one of them.

To simulate a feature attempting to drive conversion rates up, we'll make the checkout button pulsate:

<!-- app/views/carts/_cart_footer.html.erb -->
<% order = order_form.object %>

<footer class="cart-footer">
  <%= render 'carts/cart_adjustments' %>
  <p class="cart-footer__total flex justify-between mb-3 text-body-20 p-2">
    <%= t('spree.total') %>: <span class="font-sans-md"><%= order.display_total %></span>
  </p>
  <div class="cart-footer__primary-action">
    <%= order_form.button(
      I18n.t('spree.checkout'),
      class: "button-primary w-full #{'animate-pulse' if Flipper.enabled?(:conversion_rate_optimization, spree_current_user)}",
      id: 'checkout-link',
      name: :checkout
    ) %>
  </div>
</footer>

If we log in with both users and arrange the browser windows side by side, we can observe that indeed the effect is active for one (the left) user:

Use AppSignal Custom Metrics to Measure the Impact of Feature Flags

The best feature flag system is useless if there's no way to evaluate its impact. In our example scenario, we simply want to know:

Has the performance improvement led to a significant latency reduction?
Has our pulsating checkout button led to a significantly higher conversion rate?

We will use AppSignal's custom metrics to measure the pay-off of these optimizations.

First of all, create a new application in your AppSignal organization and connect it to your app by following the instructions:

bundle add appsignal
bundle exec appsignal install YOUR_APPSIGNAL_UUID

Measuring Latency with a Measurement Metric

We have verified how effective our improvement is with the oha CLI above, but to make valid judgments we'll install server-side telemetry that reports latency to AppSignal. A measurement metric allows for exactly that: we will send over response times in milliseconds, and add a metric tag indicating whether our performance optimization was active for a specific request.

There's a small gotcha here: because we're employing the "Percentage of Time" metric, we have to capture the flag's state in an instance variable so that the same value is used for execution and for reporting:

# app/controllers/products_controller.rb

class ProductsController < StoreController
  around_action :measure_response_time, only: :index

  # ...

  def index
    @searcher = build_searcher(params.merge(include_images: true))
    @products = @searcher.retrieve_products

    @performance_improvement_enabled = Flipper.enabled?(:performance_improvement)

    if @performance_improvement_enabled
      # 🚅
    else
      sleep 1
    end
  end

  # ...

  private

  # ...

  def measure_response_time
    response_time = Benchmark.realtime { yield }
    Appsignal.add_distribution_value("products_response_time", response_time * 1000, performance_improvement_enabled: @performance_improvement_enabled)
  end
end

Now let's repeat the local load testing from above:

$ oha http://localhost:3000/products -z 30s -q 2

We'll look at charting and evaluating this metric in a bit. Before that, let's turn to our second feature flag.

Tallying Conversions with a Count Metric

We'll use a counter metric to count conversions. This is a great choice if all you want to do is just keep a tally of an event.

To do this, we'll have to open CartsController, and, for demonstration purposes, add an increment_counter call if the checkout button is clicked:

# app/controllers/carts_controller.rb

def update
  authorize! :update, @order, cookies.signed[:guest_token]
  if @order.contents.update_cart(order_params)
    # ...

    Appsignal.increment_counter("checkout_count", 1, optimization_active: Flipper.enabled?(:conversion_rate_optimization, spree_current_user))

    # ...
  else
    render action: :show
  end
end

Now let's test this by manually opening respective browser windows and clicking the "Checkout" button 3 times, and in another case only once. In this way, we can see if the optimization flag is active.

Set Up Custom Dashboards in AppSignal

Our final step is to create informative graphics to make data-informed business decisions. We'll use AppSignal's dashboards to achieve this. Let's go through this step by step:

In the left sidebar, click "Add dashboard" and name it "Feature Flag Evaluation".

Click "Add Graph" and the products_response_time metric. Select "mean" to display only averages and apply the performance_improvement_enabled tag.

Click "Add new Graph" to add a chart for the checkout counts. Again, apply the optimization_active tag.

Now your custom dashboard is ready. In the line graph on the left, you can assert that your performance improvement was effective. On the right, observe how the higher count of checkouts in the optimized case was recorded.

And that's it!

Wrapping Up

We've seen how feature flags offer a flexible and efficient way to manage and deploy new features in a Ruby on Rails application. By using tools like the Flipper gem and AppSignal's custom metrics, developers can not only control feature rollouts, but also measure their impact on performance and user behavior.

This approach ensures that new features are thoroughly tested and optimized before being fully deployed, ultimately leading to a more stable and user-friendly application. Finally, it can lead to more informed business decisions when gauging the effectiveness of alternative approaches.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby’s hidden gems: Sorbet

Roy Tomeij — 2024-09-18T00:00:00+00:00

The debate between static and dynamically typed languages has long been a subject of contention among developers. Each approach offers its own set of advantages and disadvantages, significantly influencing the software development process.

Dynamically typed languages like Ruby provide flexibility by allowing variables to be declared without corresponding types. This approach fosters rapid development and promotes an agile process.

Yet, the absence of strict typing can lead to challenges, such as runtime errors that may be harder to debug and maintain in larger codebases. For example, in a dynamically typed language like Ruby, attempting to divide an array by a string only results in an error when the code is executed, making it potentially harder to identify and fix such issues.

In this article, we explore Sorbet, a type checker for Ruby, which addresses the challenges of dynamic typing in Ruby, enhancing code reliability and maintainability without sacrificing the language's flexibility and expressiveness.

What is Sorbet for Ruby?

Sorbet, implemented in C++, is a Ruby gem designed to harmonize the dynamism of Ruby with the reliability and predictability of static typing. As Ruby projects scale in size and complexity, maintaining code quality and preventing errors becomes increasingly challenging. A primary culprit is the absence of static typing, which often necessitates heavy reliance on extensive testing and runtime checks to ensure code correctness, resulting in more frequent bugs slipping into production.

Developed by Stripe, Sorbet seeks to tackle these challenges by introducing static typing to Ruby. It functions as a type checker and gradual type system for Ruby, enabling the annotation of code with type information and the detection of errors at compile time (rather than runtime).

Key features and benefits of Sorbet include:

Type Declaration: Sorbet allows for the declaration of types for variables, method parameters, and return values. This facilitates early error detection and enhances code readability.
Gradual Typing: Unlike statically typed languages where typing is mandatory, Sorbet permits the incremental introduction of type annotations. This means existing Ruby codebases can transition gradually to a statically typed workflow without needing a complete rewrite.
Instantaneous Feedback: During development, Sorbet provides instant insights into method definitions and usage. Clicking on a method reveals its definition, while hovering over it displays information about its input and return types. This real-time feedback accelerates development and enhances code navigation.
Type Inference: Sorbet employs a straightforward type inference algorithm to deduce types where explicit annotations are absent. This minimizes the need for manual type annotations and facilitates the adoption of static typing in Ruby projects.
Tooling Integration: Sorbet seamlessly integrates with popular Ruby development tools, including editors, IDEs, and build systems. This ensures a seamless developer experience and promotes the adoption of static typing practices within Ruby development workflows.

Getting Started with Sorbet

Before diving into Sorbet's integration into your codebase, it's helpful to explore a Sorbet playground, which provides a sandbox environment to experiment with Sorbet's features. Here's a Sorbet playground that allows you to tweak code and see how Sorbet responds to various changes.

To understand Sorbet and its functionality, before applying it to an existing codebase, let's start a new Ruby project:

Create a new directory: Begin by creating a new directory named sorbet-test where we'll set up our Sorbet testing environment.

mkdir sorbet-test
cd sorbet-test

Set up the Gemfile: If you don't already have a Gemfile in your sorbet-test directory, create one using the following command:

touch Gemfile

Then, add the Sorbet gem to your Gemfile:

source 'https://rubygems.org'

gem 'sorbet', group: :development

Install Sorbet: Install the Sorbet gem using Bundler:

bundle install

Verify the installation: Check that Sorbet is installed correctly by running:

bundle exec srb

This should give an output that indicates that no sorbet/ directory was found and prompts us to initialize our directory with 'srb init'.

Create a Ruby file: Now, let's create a new Ruby file named person.rb in our sorbet-test directory, and add the following code:

class Person
 attr_accessor :name

  def initialize(name)
    @name = name
  end

  def check_name
    if nam == 'John'
      puts 'This person is John'
    else
      puts 'This person is not John'
    end
  end
end

Initialize Sorbet: Initialize your directory with Sorbet by running:

srb init

This will create a sorbet folder with an rbi subfolder in our directory. You'll also find the # typed: false sigil appended to the person.rb file. This sigil indicates to Sorbet what errors to report and which to silence (# typed: ignore being the least, as it causes Sorbet to ignore the file, and # typed: strong being the strictest, as all errors in this file are reported). Read more detailed information about these sigils.

Update typed sigil: Update the # typed sigil in the person.rb file to true:

# typed: true

Run Sorbet: Finally, run Sorbet with bundle exec srb to check your Ruby file for type errors.

The following error should ensue:

person.rb:10: Method nam does not exist on Person https://srb.help/7003
    10 |    if nam == 'John'
               ^^^
  Did you mean name? Use -a to autocorrect
    person.rb:10: Replace with name
    10 |    if nam == 'John'
               ^^^
    person.rb:3: Defined here
     3 | attr_accessor :name
         ^^^^^^^^^^^^^^^^^^^
Errors: 1

Correcting this typo to name and rerunning the command should output the following:

No errors! Great job.

Type Mismatches in Sorbet

Let's see another example of how Sorbet detects a type mismatch.

Add this new method to person.rb file:

def name_length
  puts name.length
end

Within the file, call this method as such:

Person.new(8).name_length

Run Sorbet with bundle exec srb.

We do not get any errors. However, when we run this code, we get the following error:

/.../person.rb:18:in `name_length': undefined method `length' for 8:Integer (NoMethodError)

    puts name.length
             ^^^^^^^

Sorbet should ideally catch this error, but it fails because it lacks the necessary information about the expected name type. To provide Sorbet with this information, we need to add type signatures to our methods.

Adding Type Signatures

We can enable type signatures in our code by extending the T::Sig module and adding signatures to methods. Let's add a signature to the initialize method:

class Person
  extend T::Sig

  sig {params(name: String).void}
  def initialize(name)
    @name = name
  end
end

This signature is basically telling Sorbet that this method takes a parameter name of type String and returns nothing.

Now, running bundle exec srb outputs the following error:

person.rb:24: Expected String but found Integer(8) for argument name https://srb.help/7002
    24 |Person.new(8).name_length
                   ^
  Expected String for argument name of method Person#initialize:
    person.rb:6:
     6 |  sig {params(name: String).void}
                      ^^^^
  Got Integer(8) originating from:
    person.rb:24:
    24 |Person.new(8).name_length

Sorbet successfully detects the error due to the addition of type signatures. Updating the sigil to # typed: strict, requires all methods to possess a type signature.

Sorbet Runtime Support

Although Sorbet is able to carry out its checks successfully, we're not able to run this piece of code. We get the following error:

/.../person.rb:3:in `<class:Person>': uninitialized constant Person::T (NameError)

  extend T::Sig
          ^^^^^

This error indicates that Sorbet's type annotations, represented by the T module, were not found. Our piece of code requires access to the necessary runtime support for Sorbet's type annotations, which allows it to run correctly alongside Sorbet's static type checking.

To resolve this, we need to add the sorbet-runtime gem to our project:

gem 'sorbet-runtime', group: :development

After running bundle install, require sorbet-runtime in our file:

# person.rb
require 'sorbet-runtime'

With sorbet-runtime loaded, our code runs correctly, and Sorbet effectively enforces type safety.

Sorbet IDE Integration

Running bundle exec srb tc frequently to typecheck code can be tedious and time-consuming. To streamline the development process, Sorbet offers a VSCode extension that provides various features to enhance your coding experience. These features include autocomplete, jump to definition, type information and documentation on hover, sig suggestion, quick fixes, autocorrection, static error displays, and more. Find out more about how to install and use the VSCode extension.

For developers using editors other than VS Code, Sorbet does not have official integrations or extensions. However, some editors support the Language Server Protocol (LSP), allowing language servers like Sorbet to integrate with them. JetBrains IDEs, including IntelliJ IDEA, PyCharm, and RubyMine, have built-in support for the LSP. Additionally, there are plugins available for Sublime Text and Atom that enable LSP support. While these solutions may not offer the same level of integration and features as the dedicated Sorbet extension for VSCode, they can still provide basic functionality, such as syntax highlighting, autocompletion, and error checking.

Exploring Tapioca

We've gone through the process of adding types to a new project. However, what about projects already in existence? Typically, these projects come with pre-installed gems, and these third-party services include methods invoked within our project. How do we guarantee that we're passing the correct types of parameters to these methods, or that they're being invoked on appropriate types? The answer to this is by using Tapioca. To understand what Tapioca does, it's important to first understand what Ruby Interface (RBI) files are.

What Are Ruby Interface (RBI) Files?

RBI means Ruby Interface, and RBI files serve as interface files that provide documentation on type information for Ruby code. They contain declarations of types, method signatures, and other type-related metadata. They can either be created manually or autogenerated.

While Sorbet is capable of inferring types to some extent, there are scenarios for which Sorbet lacks sufficient information. For example, when dealing with complex inheritance hierarchies, method overrides, external dependencies, or dynamic method calls, Sorbet may struggle to infer types accurately without explicit type annotations provided by RBI files.

What Does Tapioca Do?

Tapioca, a Ruby gem developed by Shopify, streamlines the creation of RBI files, which are essential for implementing gradual typing in Ruby projects. These RBI files can cover not only the gems used within the application but also the methods available within Rails, along with other DSLs and metaprogramming paradigms.

To integrate Tapioca into an existing Rails project for use with Sorbet, follow these steps:

Add the Tapioca gem to your Gemfile, alongside existing gems such as sorbet and sorbet-runtime.

gem 'tapioca', require: false, group: [:development, :test]

Initialize the project for use with Sorbet by running:

bundle exec tapioca init

This command generates a Sorbet folder structure within your project:

├── config             # Default options to be passed to Sorbet on every run
└── rbi/
  ├── annotations/     # Type definitions pulled from the rbi-central repository
  ├── gems/            # Autogenerated type definitions for your gems
  └── todo.rbi         # Constants which were still missing after RBI generation
└── tapioca/
  ├── config.yml       # Default options to be passed to Tapioca
  └── require.rb       # A file where you can make requires from gems that might be needed for gem RBI generation

The terminal output provides valuable guidance on generating type definitions for DSLs in your application, performing type checking, and upgrading files from # typed: false to # typed: true using tools like Spoom. Take some time to review this information.

Let's illustrate with a simple example involving a Person model in your project. After adding # typed: true to a file, attempting to call Person.all triggers an error:

Method `all` does not exist on `T.class_of(Person)`

With the Sorbet extension in your IDE, Person.all is highlighted in red, indicating an error. This occurs because Sorbet lacks awareness of Person as a model or the available Rails methods for the Person class. To resolve this, we need to generate an RBI file for the Person model. RBI files can be generated using the command:

bin/tapioca dsl

Executing this command generates several RBI files, including person.rbi, which defines methods available to the Person class:

# typed: true

# DO NOT EDIT MANUALLY
# This is an autogenerated file for dynamic methods in `Book`.
# Please instead update this file by running `bin/tapioca dsl Book`.

class Person
  include GeneratedAttributeMethods
  extend CommonRelationMethods
  extend GeneratedRelationMethods

  # Other methods related to Person class are stated here...

  module GeneratedAssociationRelationMethods
    sig { returns(PrivateAssociationRelation) }
    def all; end

    # Additional methods related to associations are stated here...
  end

  # More methods related to Person class are stated here...
end

With the RBI file in place, the error disappears, as Sorbet now recognizes all methods available to Person. Any additions or changes to methods in Person can be reflected in the RBI file by rerunning the bin/tapioca dsl command.

Challenges with Sorbet

Gradually adding types to a codebase often leads to a mixed scenario where some parts are typed, and others aren't. Sorbet helps by detecting errors during runtime with its sorbet-runtime component. This is particularly valuable in identifying situations where types might be incorrect, such as when passing different types from an untyped section of code while expecting a specific type in a typed portion.

# typed: true
require 'sorbet-runtime'

class Person
  extend T::Sig

  def self.happy?
    true
  end

  sig {params(statement: String).returns(T::Boolean)}
  def self.the_truth?(statement)
    statement.length > 9 ? true : false
  end
end

Person.the_truth?(Person.happy?)

Running bundle exec srb doesn't raise an error here because it's unaware of the happy? return type, which prevents it from accurately assessing whether the passed value is not a string. However, Sorbet runtime detects this issue and raises an error during runtime without the execution of the the_truth? method.

Parameter 'statement': Expected type String, got type TrueClass (TypeError)
Caller: person.rb:37
Definition: person.rb:31

This runtime feedback is invaluable for correcting our code and making necessary adjustments. Yet, it's essential to recognize that Sorbet runtime incurs a performance overhead, which may not be suitable for all production environments. According to the Sorbet documentation on runtime:

[...]in some cases, especially when calling certain methods in tight loops or other latency-sensitive paths, the overhead of even doing the checks (regardless of what happens on failure) is prohibitively expensive. To handle these cases, Sorbet offers .checked(...) which declares in what environments a sig should be checked.

Sorbet provides various mechanisms to disable these runtime checks when needed. One such mechanism can be configured in application.rb:

T::Configuration.default_checked_level = :tests # where :tests is the environment, it can also be set to :never

For further details on Sorbet runtime checks, refer to the Sorbet runtime documentation.

Wrapping Up

In this post, we've looked at Sorbet, exploring its fundamental workings, benefits, and some considerations regarding performance. Sorbet stands as a robust tool for type checking in Ruby, offering the flexibility of gradual adoption. Its vibrant ecosystem and ease of implementation make it an invaluable asset for Ruby developers seeking to enhance code reliability and maintainability.

For further exploration, consider diving into the resources provided below:

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby on Rails 7.1: Partial Strict Locals and Their Gotchas

Roy Tomeij — 2024-09-11T00:00:00+00:00

Rails partials have been around for years, but they can be clunky since they're just ERB snippets without a backing object structure.

Recently, libraries like ViewComponent and Phlex have tried to improve the view layer by adding more semantic structure to the templates. These are great libraries and I personally reach for ViewComponent on almost every project I work on. That said, I still feel the humble Rails partial still works great for many use cases.

The Rails team is always trying to improve its offering, so let's look at a new feature introduced in Rails 7.1: strict locals!

Rails Partials and Local Variables

We can pass in arbitrary local variables into Rails partials and they're magically available for use.

<%# app/views/application/_badge.html.erb %>

<ui-badge>
  <p>
    <%= title %>
    <%= tag.i class: icon %>
  </p>
</ui-badge>

This isn't ideal, because it's tricky to ascertain which variables a partial accepts if there's a large block of markup. Also, since variables are dynamically created, a missing variable won't return nil but will cause an error.

Let's say the icon was optional in the above partial:

<%# app/views/application/_badge.html.erb %>

<ui-badge>
  <p>
    <%= title %>
    <%= tag.i class: icon if icon.present? %>
  </p>
</ui-badge>

We render it as:

<%= render "application/badge", title: "New" %>

That will raise an ActionView::Template::Error, with the message undefined local variable or method 'icon' for #<ActionView::Base:0x00000000023078>.

The fact that icon is optional is not obvious to a reader either. Until Rails 7.1, the solution was to inspect the local_assigns hash which contains all injected local variables.

<%# app/views/application/_badge.html.erb %>

<% icon ||= local_assigns[:icon] %>
<ui-badge>
  <p>
    <%= title %>
    <%= tag.i class: icon if icon.present? %>
  </p>
</ui-badge>

This fixes the above error because the icon variable is now always set.

Rails 7.1 attempts to address the shortcomings of local variables in partials with the concept of strict locals.

What Are Strict Locals in Rails?

Starting in Rails 7.1, we can write a magic comment at the top of partials, which defines the local variables it accepts and any default values for them.

<%# app/views/application/_badge.html.erb %>

<%# locals: (title:, icon: nil) %>

<ui-badge>
  <p>
    <%= title %>
    <%= tag.i class: icon if icon.present? %>
  </p>
</ui-badge>

Rails parses this comment and sets the defined variables in the partial. Now it's plainly obvious to a reader which variables the partial accepts and any default values.

The partial should still render as shown above. Try excluding the title variable from the call site and see what happens:

<%= render "application/badge" %>

You get an ArgumentError with the message missing local: :title: a more descriptive error message than before!

When to Use Strict Locals

Strict locals can be fiddly, so it's not always a good idea to use them, especially in legacy apps.

Let's take a use case that's perfect for strict locals. Say we have a partial showing a user's details in a card, but only admins can see when they last signed in:

<ui-card>
  <dl>
    <dt>Name</dt>
    <dd><%= name %></dd>

    <dt>Email</dt>
    <dd><%= email %></dd>

    <% if last_signed_in %>
      <dt>Last signed in at</dt>
      <dd><%= last_signed_in %></dd>
    <% end %>
  </dl>
</ui-card>

This partial takes several variables and has an optional variable, making it a perfect case for strict locals. Adding the following magic comment to the file makes it significantly easier to skim:

<%# locals: (name:, email:, last_signed_in: nil) %>

In the case of a model's associated partial (e.g. posts/_post.html.erb), it's fairly obvious that such a partial accepts a single post variable. I don't think strict locals are necessarily a good idea for these cases.

Strict Locals Gotchas

There are a couple of gotchas to be wary of, which is why I don't recommend using strict locals across the board.

Implicitly Rendering a Collection

When implicitly rendering a collection:

<%= render @posts %>

Rails injects the _post partial with two variables in addition to the post variable. Before Rails 7.1.2, these needed to be manually defined to prevent an error:

<%# app/views/posts/_post.html.erb %>

<%# locals: (post:, post_counter:, post_iteration:) %>
<%# ... %>

This problem was fixed in Rails 7.1.2 and these variables are now optional (but it's good to know they exist if needed)!

Broadcasting Over ActionCable

When broadcasting a partial over ActionCable from an ActiveRecord callback:

class Comment < ApplicationRecord
  after_create_commit -> {
    broadcast_append_later_to(
      post,
      target: dom_id(post, :comments)
    )
  }
end

You need to specify a request_id local variable defaulting to nil, as this is used under the hood by Rails.

<%# app/views/comments/_comment.html.erb %>

<%# locals: (comment:, request_id:) %>
<%# ... %>

Not doing so results in an error when the callback is triggered.

And that's our whistle-stop tour done!

A Few Things to Keep in Mind

Adding strict locals to existing partials is an all-or-nothing proposition for a given file. We can't incrementally add variables to a magic comment as we find them. As shown in the above gotchas, it can be tricky to ascertain all the variables that need to be declared in the magic comment.

Given how much easier it is to reason about code with strict locals, I think they're worth using extensively in new apps (and progressively adding retroactively to legacy apps).

It's important to ensure your test suite in legacy apps covers the partials to which you add strict locals. An omission will cause an exception, so you need to be sure you've tested all use cases before deploying changes.

While I'm not a huge fan of the magic comment approach (as I believe comments should never have functionality), I think strict locals are a great addition to Rails. It's always been fiddly to assign locals to a partial and these do make it easier. Just beware of the gotchas!

Wrapping Up

In this post, we took a quick look at strict locals in Rails 7.1. We saw how partials and local variables work before exploring when to use strict locals and some gotchas.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Squash Your Ruby and Rails Bugs Faster

Roy Tomeij — 2024-08-21T00:00:00+00:00

A bug in software can be disruptive, elusive, maddening, and invasive. Indeed, a developer often needs the tenacity of Edison to find and fix an issue.

But grit isn't the only asset a developer requires. One also needs information to debug code: What are the symptoms and effects of the issue? What is its frequency? Pervasiveness? Provenance? The evidence and artifacts of a bug — a core dump, stack trace, log, or test case — are invaluable.

Let's explore a few techniques and tools readily available to Ruby and Rails developers for gathering and investigating evidence of an issue. Data can't supplant tenacity, but it can help to ease your efforts (and spare you your sleep).

Use Environment Variables to Tune Your Tools in Rails

Rails provides some excellent tools to peer into a running application, including a configurable Logger to capture diagnostics of its own and any that you'd like to add. Rails also provides a verbose query log to pinpoint the origin of database queries and a verbose enqueue log to illuminate where background jobs are queued. The latter two logs are enabled by default in the development environment; you can enable both in other environments with two statements.

##
# In any initializer... enable the verbose query log to see the file
# name and line number of the code initiating each query
ActiveRecord.verbose_query_logs = true

##
# In config/application.rb or any environment initializer, enable the
# queueing log to find where each job is enqueued
config.active_job.verbose_enqueue_logs = true

A lesser-known log option available since Rails 7 annotates each SQL query with comments. If you add the following to config/application.rb or any environment file:

##
# Add to config/application.rb or any environment initializer file
config.active_record.query_log_tags_enabled = true

Your query log is automatically amended with the application name, controller name, action name, or background job name. The following sample output comes directly from Rails's debugging guide:

Article Load (0.2ms)  SELECT "articles".* FROM "articles"
/*application='Blog',controller='articles',action='index'*/

Article Update (0.3ms)  UPDATE "articles" SET "title" = ?, "updated_at" = ? WHERE "posts"."id" = ?
/*application='Blog',job='ImproveTitleJob'*/
[["title", "Improved Rails debugging guide"], ["updated_at", "2022-10-16 20:25:40.091371"], ["id", 1]]

You likely do not want to enable all these features in production. Log generation can be costly in terms of memory and time (finite resources for an application under load).

However, there are exceptions. You may want to briefly enable an on-demand feature, especially when debugging in development and test modes. Rather than alter code (and perhaps redeploy) to enable or disable a diagnostic, use environment variables to control state. In some environments, such as Heroku, changing the setting of an environment variable does not mandate a new deployment.

For example, you can define a set of variables to control logging and verbosity.

Variable Name	Purpose	Values
`COMMENTED_QUERY_LOG`	Enable comments in query log	Non-blank value enables feature
`LOG_ALL`	Enable all logging features	Non-blank value enables feature
`LOG_LEVEL`	Control the threshold for log messages	`debug`, `info`, `warn`, `error`, `fatal`, `unknown` (from most verbose to least verbose order)
`VERBOSE_ENQUEUE_LOG`	Display where jobs are queued	Non-blank value enables feature
`VERBOSE_QUERY_LOG`	Show the origin of each Active Record query	Non-blank value enables feature

For convenience, create a small class to query configuration settings.

##
# Put this in a file such as config/initializers/env_configuration.rb
class EnvironmentConfiguration
  def self.enabled?(envvar)
    Rails.env.development? ||
      ENV.fetch('LOG_ALL', nil).present? ||
      ENV.fetch(envvar.to_s, nil).present?
  end
end

Now use the variables and the code to set logging features in config/application.rb:

##
# Centralize configuration of all logging features in config/application.rb
#
module MyApplication
  class Application < Rails::Application
    # . . .
    ActiveRecord.verbose_query_logs             = EnvironmentConfiguration.enabled?('VERBOSE_QUERY_LOG')
    config.active_job.verbose_enqueue_logs      = EnvironmentConfiguration.enabled?('VERBOSE_ENQUEUE_LOG')
    config.active_record.query_log_tags_enabled = EnvironmentConfiguration.enabled?('COMMENTED_QUERY_LOG')
    config.log_level = (ENV.fetch('LOG_LEVEL', nil).presence || 'info').to_sym
  end
end

Use your shell, a dotenv file, your continuous integration, or hosting and deployment platform to set each option. You can also use a feature on demand. Simply start the application with the environment variable defined on the command line.

LOG_ALL=on LOG_LEVEL=debug bundle exec rails s

Tune Puma for Development

Building on the previous section, let's look at how to use environment variables to tailor your Puma configuration for debugging in the development environment.

Typically, Puma is configured to maximize throughput in production, running multiple workers and many threads per worker. In development, you want the opposite: one worker, one thread, and a very long timeout to allow for interactive debugging. Each of those is a parameter you can tune.

Alter config/puma.rb to reflect the following code.

##
# Control thread count with environment variables.
# Each environment setting, if set, is always a string. Use `Integer` to
# convert the string to an integer value.
#
max_threads_count = Integer ENV.fetch('PUMA_MAX_THREADS', 5)
min_threads_count = Integer ENV.fetch('PUMA_MIN_THREADS, max_threads_count)
threads min_threads_count, max_threads_count

##
# Specify the duration Puma waits before terminating a worker.
# The default is 30 seconds.
#
worker_timeout Integer ENV.fetch('PUMA_WORKER_TIMEOUT', 30)

##
# Set the number of workers
worker_count = Integer ENV.fetch('PUMA_WEB_CONCURRENCY, 2)
workers worker_count
preload_app! if worker_count.positive?

You can now configure the three environment variables to control Puma in each environment. In development, set the values to optimize for interactive debugging.

export PUMA_MAX_THREADS=1
export PUMA_WORKERS=0
export PUMA_WEB_CONCURRENCY=0
export PUMA_WORKER_TIMEOUT=3600

If you want to validate your Puma settings, set the environment variable PUMA_LOG_CONFIG=true and start your application. Puma emits its active configuration on startup.

=> Booting Puma
=> Rails 6.1.7.7 application starting in development
=> Run `bin/rails server --help` for more startup options
Configuration:
. . .
- environment: development
- max_threads: 5
- min_threads: 5
- worker_timeout: 3600
- workers: 0

Coincidentally, the default Puma configuration in the most recent Rails release is similar to what's shown here (thanks to Nate Matykiewicz for the tip).

Run Background Jobs Inline

A Rails application of any complexity typically leverages background jobs to run compute-intensive and (relatively) long-running tasks. Background jobs run asynchronously, disconnected from the request/response cycle. Ideal candidates for "out of band" processing include generating reports, sending emails, and interacting with third-party APIs. But the asynchronous nature of jobs also complicates debugging.

To simplify troubleshooting, run jobs immediately in your local development and test environments. In immediate mode, jobs are not queued, but instead execute instantaneously. Running in the "foreground", you can set breakpoints and inspect state interactively.

Let's look at an example using Delayed Job, a popular and easy-to-manage queueing backend for Active Job. Delayed Job provides a setting to enable queueing. By default, the setting is true and jobs are queued as per usual. However, if set to false, jobs run immediately.

Add the following code to your application in config/initializers/delayed_job.rb:

Delayed::Worker.delay_jobs = ENV.fetch('DELAYED_JOBS_DISABLE_JOB_QUEUES', false).blank?

If DELAYED_JOBS_DISABLE_JOB_QUEUES is set to any value, queueing is disabled. If the environment variable is blank or not defined, queueing is enabled.

Next, in your shell, on the command line, or in your dot files, set DELAYED_JOBS_DISABLE_JOB_QUEUES as needed.

# Set the variable in the current shell
export DELAYED_JOBS_DISABLE_JOB_QUEUES=true

# Set the variable only for the current command
DELAYED_JOBS_DISABLE_JOB_QUEUES=true rails s

# In .env.development, .env.local, and/or .env.test, add the line...
DELAYED_JOBS_DISABLE_JOB_QUEUES=true

Set the environment variable to nothing or delete the environment variable to restore queueing.

# Set the variable to blank
export DELAYED_JOBS_DISABLE_JOB_QUEUES=

# Remove the variable from the environment
unset DELAYED_JOBS_DISABLE_JOB_QUEUES

There are no rules for naming environment variables. Choose names meaningful to you. One helpful convention to categorize variables is to add a package name as a prefix, such as PUMA_ and DELAYED_JOB_. The former indicates variables affecting Puma; the latter connotes variables used by Delayed Job.

Peer Into Network Calls

Like background jobs, Rails applications can also consume application programming interfaces (APIs). APIs provide access to external services, such as GraphQL databases, e-commerce transactions, and public data sources.

Here's another example to conditionally emit HTTP requests and responses from the Net::HTTP library. If the environment variable DEBUG_HTTP is set to any non-blank value, the outgoing request and inbound responses are printed to STDOUT.

def construct_uri(hostname:, path:, port:, params: {}, protocol: 'https')
  target =
    Addressable::URI.new.tap do |uri|
      uri.hostname     = hostname
      uri.path         = path
      uri.port         = port
      uri.query_values = params
      uri.scheme       = protocol
    end

  URI.parse target
end

def debug?
  ENV.fetch['DEBUG_HTTP'].present?
end

def request
  uri = construct_uri(...)

  Net::HTTP.new(uri.host, uri.port).tap do |http|
    http.set_debug_output($stdout) if debug?
    http.use_ssl = uri.scheme == "https"
  end
end

To see the network activity, simply start the application with the environment variable defined on the command line.

DEBUG_HTTP=on bundle exec rails s

The debug? method provides some abstraction from the actual flag implementation. It, and the rest of this code, can form a base class for all your network requests, as shown in this gist taken from production code.

Add Ruby Gems to Your Toolbox

The universe of Ruby gems is vast: it's impossible to cover all the great gems available for debugging. Instead, let's look at a few tools you can add today for an instant boost. You'll likely add each of these to all your projects.

awesome_print and table_print create rich, readable inspections of Ruby data structures, including Active Record models. You can use either gem in code or from the console.

Here is an example of a model emitted by awesome_print in the console:

> ap User.find_by(email: 'george@bigbrother.gov')
#<User:0x00000001163482f0> {
       :account_balance => 0.0,
          :confirmed_at => Mon, 11 Mar 2024 12:29:30.000000000 EDT -04:00,
            :created_at => Mon, 11 Mar 2024 12:29:30.000000000 EDT -04:00,
                 :email => "george@bigbrother.gov",
    :encrypted_password => "992c45d528f9b445de3d6653e14d3e6165171944",
            :first_name => "George",
             :last_name => "Orwell",
         :sign_in_count => 0,
                 :state => "active"
}

Instead of p, use ap to show the enhanced output. awesome_print lists attributes in alphabetical order, one attribute per line, among other niceties.

better_errors replaces the standard Rails error page with an enhanced stack backtrace, parameter list, and an interactive console where you can probe the stack frame and variables at the scene of the exception. You can also tie source code links to your favorite editor. Here is code to tie better_errors to Visual Studio Code:
```
if Rails.env.development?
BetterErrors.editor =
  if (ENV['BETTER_ERRORS_EDITOR'].presence || ENV['EDITOR']) =~ /vscode/i
    proc { |file, line| "vscode://file#{file}:#{line}" }
  else
    :macvim
  end
end
```
Run export BETTER_ERRORS_EDITOR=vscode in your shell and restart the Rails server. Now links to files open automatically in Visual Studio.
Keep the faker and factory_bot gems (or their alternatives) in both the testing and development groups in Gemfile. Both are invaluable if you need to generate data quickly in the console.
```
group :development, :test do
  # other gems
  # ...
  gem 'factory_bot'
  gem 'faker'
end
```

One more idea: If you use AppSignal as your application monitoring tool, it has features to interpret your logs and pinpoint the source of errors. See Debugging in Ruby with AppSignal for more details.

Debugging is a Skill

Debugging code is something of an art, and like any creative endeavor, the more practice you have, the better you become.

Debug while pairing with another developer, especially an experienced developer. If your partner is willing, think out loud together and exchange hunches and insights.

Go forth and hack!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to HTTP Caching in Ruby On Rails

Roy Tomeij — 2024-08-14T00:00:00+00:00

It's 2024, and the HyperText Transfer Protocol (HTTP) is 35 years old. The fact that the vast majority of web traffic still relies on this simple, stateless form of communication is a marvel in itself.

A first set of content retrieval optimizations were added to the protocol when v1.0 was published in 1996. These include the infamous caching instructions (aka headers) that the client and server use to negotiate whether content needs refreshing.

28 years later, many web developers still avoid using it like the plague. A great deal of traffic and server load (and thus, latency) can be avoided, though, by using the built-in caching mechanism of the web. Not only does this result in a greatly improved user experience, it can also be a powerful lever to trim down your server bill. In this post, we'll take a look at:

The basic concepts around HTTP caching.
What cache layers we can leverage.
How web caching is configured and controlled.
How simple it is to cache in Ruby on Rails.

Let's get started!

Concepts

Before we dive deep into the mechanics of web caching, let's get some definitions out of the way.

Fresh Vs. Stale

Let's assume we have a shared cache server in place that stores responses from our app server for reuse. Later, a client issues a request to our app server. The stored response in our cache is now in one of two states:

Fresh: The response is still valid (we'll see what that means in a second) and will be used to fulfill the request.
Stale: The response isn't valid anymore. A new response has to be calculated and served by the upstream server.

HTTP Cache Layers

The HTTP Caching spec defines two types of cache — shared and private.

Shared Caches

Shared caches store responses that are reusable among a group of users. Typically, shared caches are implemented at intermediaries, e.g., Nginx, Caddy, or other reverse proxies. Of course, commercial edge caches like Cloudflare or Fastly also implement shared caches.

Another flavor of shared caches can be implemented in service workers using the Cache API.

We can control which responses are eligible for caching with the Cache-Control header.

Private Caches

Private caches are allotted to a single user. You are most likely to encounter a private cache as a browser component. The main feature here is that the stored responses are not shared with other clients. Any content that the server renders containing personalized data must only be stored in a private cache. In general, this will be the case for any authenticated route in your Rails app, but there might be exceptions.

You can force a response to be cached privately using the Cache-Control: private header.

Controlling Caching Via the `Cache-Control` Header

Although, historically, other headers have been in use (like Expires or Pragma), we will focus on the Cache-Control header here. The specification has an exhaustive description, but we'll focus on the salient parts.

The Cache-Control header sets one or several comma-separated directives that can be further divided into request and response directives.

Request Directives

When requesting a resource from the server, the client (browser) can specify one of the following directives to negotiate how caching should behave:

no-cache - Advises the cache to revalidate the response against the origin server. Typically this happens when you force reload a page or disable caching in the browser's developer tools.
no-store - Indicates that no cache must store any part of this request or any response.
max-age - In seconds, indicates the timespan for which a response from the server is considered fresh. For example, Cache-Control: max-age=3600 would tell the server that any response older than an hour cannot be reused.

Response Directives

Essentially, response directives consist of what's above, with different semantics, plus a few more. A cache must obey these directives coming from the origin server:

no-cache - A bit counterintuitively, this directive does not imply that the response cannot be cached. Rather, each cache must revalidate the response with the origin server for each reuse. This is the normal case for a response from a Rails application.
no-store - No cache of any type (shared or private) must store this response.
max-age - Indicates the period into the future for which the response should count as fresh. So, Cache-Control: max-age=3600 would specify a period of one hour from the moment of generation on the origin server. Afterwards, a cache cannot reuse it.
private - This directive specifies that the response can only be stored in a private cache (i.e., the browser). This should be set for any user-personalized content, especially when a session cookie is required to access a resource.
public - Indicates that a response can be stored in a shared cache.
must-revalidate - An edge case worth noting: HTTP allows the reuse of stale responses when caches are disconnected from the origin server. This directive prevents that and forces a fresh response or a 504 gateway timeout error.

(In)validation

Let's say a response has become stale for some reason (it's expired, for example). Even so, there's a chance that it is still valid, but we have to ask the origin server if this is the case. This process is called validation and is performed using a conditional request in one of two ways: by expiration or based on the response content.

By Expiration

The client sends an If-Modified-Since header in the request, and the cache uses this header to determine if the respective response has to be revalidated. Let's consider this response.

HTTP/1.1 200 OK
...
Date: Fri, 29 Mar 2024 10:30:00 GMT
Last-Modified: Fri, 29 Mar 2024 10:00:00 GMT
Cache-Control: max-age=3600
...

It was retrieved at 10:30 and is stored on the client. The combination of Last-Modified and max-age tells us that this response becomes stale at 11:30. If the client sends a request at 11:31, it includes an If-Modified-Since header:

GET /index.html HTTP/1.1
Host: example.com
Accept: text/html
If-Modified-Since: Fri, 29 Mar 2024 10:00:00 GMT

The server now calculates the content of the response and, if it hasn't changed, sends a 304 Not Modified response:

HTTP/1.1 304 Not Modified
...
Date: Fri, 29 Mar 2024 11:31:00 GMT
Last-Modified: Fri, 29 Mar 2024 10:00:00 GMT
Cache-Control: max-age=3600

Because it's a redirect, this response has no payload. It merely tells the client that the stale response has been revalidated and can revert to a fresh state again. Its new expiry time is now 12:31.

Based On the Response Content

Timing is a problematic matter: servers and clients can drift out of sync or file system timestamps may not be appropriate. This is solved by using an ETag header. This can be an arbitrary value, but most frequently, it is a digest of the response body. Picking up the example from above, we swap Last-Modified for an ETag header:

HTTP/1.1 200 OK
...
Date: Fri, 29 Mar 2024 10:30:00 GMT
ETag: "12345678"
Cache-Control: max-age=3600
...

If this response is stored in a private cache and becomes stale, the client now uses the last known ETag value and asks the server to revalidate it:

GET /index.html HTTP/1.1
Host: example.com
Accept: text/html
If-None-Match: "12345678"

Now, the server will return a 304 Not Modified response if the values of a freshly computed ETag and the requested If-None-Match header match. Otherwise, it will respond with 200 OK and a new version of the content.

I will not go into the details here, but the Rails docs explain weak vs. strong ETags.

Implementation In Rails

We're now ready to implement this knowledge in a Rails app. The relevant module is wired into ActionController and is called ActionController::ConditionalGet. Let's examine the interface it uses to emit the Cache-Control directives discussed above.

`expires_in`

This method sets the max-age directive, overwriting all others. You can pass it ActiveSupport::Duration and options such as public and must_revalidate, which will set the respective directives.

When would you want to use this? Typically, when you need to balance cache effectiveness and freshness. For example:

class ProductsController < ApplicationController
  def index
    # Set the response to expire in 15 minutes
    expires_in 15.minutes, public: true

    @products = Product.all
  end
end

This exemplifies the compromise you might make between the probability that a new product is added, changed, or removed in the course of 15 minutes and somebody seeing a stale response. Every application will have its own limitations here, but it's a good idea to have application monitoring like AppSignal for Ruby built into your production environment. This will enable you to query how often an endpoint is accessed by the same user vs the data manipulation frequency.

`expires_now` and `no_store`

This will set Cache-Control to no-cache and no-store, respectively. Look above for the implications.

`http_cache_forever`

http_cache_forever sets a max-age of 100 years internally and caches the response. It will consult stale?, though (see below), to determine if a fresh response should be rendered. You call it like this:

class StaticController < ApplicationController
  def about
    http_cache_forever(public: true) do
      # Your logic to render the about page goes here
      render :about
    end
  end
end

This renders the about view and allows intermediary shared caches to store it (because public is set in Cache-Control).

`fresh_when` and `stale?`

These methods are siblings and are both concerned with setting appropriate ETag and Last-Modified headers. Thus, they form the heart of Rails' conditional GET implementation. Let's look at each of them now:

class ArticlesController < ApplicationController
  def show
    @article = Article.find(params[:id])
    fresh_when(@article)
  end

  def index
    @articles = Articles.all
    fresh_when(@articles)
  end
end

The show action exhibits the simplest way to enable conditional GET requests on an endpoint. If it's passed an ActiveRecord instance, it will extract the update_at timestamp, reuse it as Last-Modified, and ETag will be computed from the record as a hex digest.

When working with relations, like in the index action, fresh_when will check for the most recent updated_at in the collection using the maximum method.

If desired, you can override this behavior using explicit options. These match the directives discussed above (the only outlier being the template option, which allows you to specify the template used to calculate the ETag digest). This is useful when your controller action uses a different template than the default.

In either case, before rendering a response, fresh_when will check if a stored response is still fresh and return a 304 in this case.

On the other hand, stale? is a predicate method that falls back to fresh_when internally and returns true or false based on its evaluation. You can use it to guard against expensive method calls, analogously to the above examples:

class ArticlesController < ApplicationController
  def show
    @article = Article.find(params[:id])

    expires_in 15.minutes

    if stale?(@article)
      @article.refresh_counters!
      render @article
    end
  end
end

In this example, since it uses the same internal semantics as fresh_when, it would automatically send a 304 Not Modified response unless the article is stale. If it is stale, though (i.e., if 15 minutes have passed since it was generated), the counters are refreshed and return a fresh response. Updating low-priority data in your responses only infrequently, based on a "timeout", is a typical use case.

The `etag` Class Method

How can caches, even private ones, deal with personalized information? The answer is that data identifying the session in some way has to be included in the ETag. This is exactly what the etag controller class method does: it provides the necessary information to differentiate between private responses to individual users.

Continuing with the example above, imagine that some users are administrators who are served "edit" buttons for the articles in the UI. You don't want that information to spill over to an anonymous user, so you can prevent that:

class ArticlesController < ApplicationController
  etag { current_user&.id }

  def show
    @article = Article.find(params[:id])
    fresh_when(@article)
  end

  def index
    @articles = Articles.all
    fresh_when(@articles)
  end
end

Assuming that current_user is a method returning the currently logged-in user, his/her id is now added to the ETag before digesting, preventing the leak of sensitive data to unauthorized visitors.

Takeaways

Given all we have learned above, do you feel confident adding HTTP caching to your controllers? I would cautiously assume that you still have some leftover anxiety regarding the leakage of sensitive data. But all in all, using Turbo Frames and personalizing a UI can provide some valuable insights.

Rigorously Decompose into Turbo Frames

If you are using Turbo in your frontend, you can leverage eager or lazy-loaded Turbo Frames. A lot of applications comprise personalized sections of the UI, while others do not. By detaching the non-personalized parts into separate endpoints that employ HTTP caching individually, you not only gain performance benefits but also a clearer application architecture.

Apply Personalized Bits of the UI After the Fact

Another way to deal with user-dependent data is to apply JavaScript after a page has loaded. This is only viable if the amount of personalized data on a page is small. In essence, you'd query an endpoint for a current user's data in JSON format, then use a Stimulus controller to apply it to the relevant bits of the UI.

Wrapping Up

In this post, we demystified HTTP caching. We looked at some basic concepts, cache layers, and configuration, including how to use the Cache-Control header and validation. We also examined the elegant solution that Rails provides in the form of the ActionController::ConditionalGet module.

Until next time, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Nix for Ruby Developers

Roy Tomeij — 2024-08-07T00:00:00+00:00

A predictable, stable environment (in terms of your operating system, system libraries, build tools, and programming libraries) is essential to each development step: from onboarding, to collaboration, continuous integration, quality assurance, and deployment. Deviation can cause one-off, intermittent, and even catastrophic failures.

However, consistency can be elusive, even with the best intentions, best practices, and tools in place, because:

Developer machines are easily polluted with clashing libraries and binaries.
Programming libraries — Ruby's gems or Python's eggs, for example — are regularly updated on unadvertised and unpredictable schedules.
Docker images can be removed from public repositories without notice.
Continuous integration providers change internals without forewarning or ex post facto notice.

Nix aims to solve some of these issues. Specifically, it is designed to reproduce packages and environments verbatim. Given a set of inputs, it generates the same output every time.

Let's explore how Nix can reproduce an environment for Ruby and Rails development, including a version of Ruby, a collection of gems, a PostgreSQL database, and a Redis cache.

An Overview of Nix

Nix is an entire universe of software. It runs on Linux and macOS, on both Apple Silicon and Intel processors, and has several components:

The Nix Programming Language is a declarative, domain-specific, functional language dedicated to system composition. Portions of the language manipulate Nix directly, while others provide typical programming constructs like flow control and variables.
Nix Packages is an expansive repository of software, from awk to zsh. At the time of writing, the repository contains more than 100,000 entries, each usable with Nix. Each package in the Nix archive explicitly documents all its dependencies down to a specific commit. Further, the nixpkgs archive is itself versioned, providing even greater control over package provenance.
nix-shell provides a virtual, sacrosanct environment contained in a shell. Given a Nix description of a system — a derivation in Nix parlance — you can compose and boot a system in a walled garden isolated from all other virtual and physical systems. For example, you can download, install, build, and run an independent instance of Ruby with nix-shell.
The Nix Store persists immutable data (such as software packages) and all dependencies. On some systems, the Nix Store is the local file system; however, it can also be retained on Amazon S3 and even a remote SSH server.
NixOS is an entire Linux distribution configured entirely by the Nix language and composed from versions in Nixpkgs.

Nix is expansive and each component warrants deep exploration. This introduction focuses on the nix-shell and Nix Packages and presents just enough of the Nix programming language to boot a Rails application. Tony Finn's Nix from First Principles provides another take on Nix.

Nix Versus Docker

Docker and Nix are both capable of building environments. However, Nix provides consistency everywhere, including build tools and source packages.

Docker is not a package manager. While you can connect and coordinate containers using Docker Compose, you cannot use Docker to combine containers into a new container.

It's trivial with Nix to combine disparate Ruby and PostgreSQL derivations into a new environment.

If a Docker image changes without notice, the output of docker build changes too. Pinning a version obviously helps, but if a Docker image is deleted or an image's repository goes offline, a Docker build fails without recourse.

Each package and version stored by Nix can have a unique ID akin to a git SHA. You can think of a complete Nix derivation as a manifest of tens or hundreds of SHAs. A change in any one SHA is detectable and two derivations will not be the same if a single entity has changed.

Get Started with Nix

Let's dive into Nix.

To install Nix natively on Apple MacOS, open the Terminal application in Applications/Utilities and run the following command at the shell prompt:

# Shell command to download and install Nix on your system
curl --proto '=https' --tlsv1.2 -sSf \
  -L https://install.determinate.systems/nix \
  | sh -s -- install

Enter the password for the root user of your system when prompted. (If you are a Mac administrator, you can enter your password too.) You need privileged access as the install creates a new storage volume specifically for Nix and tweaks several system settings, such as excluding the Nix store from Time Machine backups.

The installer summarizes all the modifications it plans to perform and prompts you to proceed. Enter Yes.

# Nix output
Nix install plan (v0.18.0)
Planner: macos (with default settings)

Planned actions:
* Create an encrypted APFS volume `Nix Store` for Nix on `disk1` and add it to `/etc/fstab` mounting on `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.21.2/nix-2.21.2-x86_64-darwin.tar.xz` to `/nix/temp-install-dir`
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Create build users (UID 301-332) and group (GID 30000)
* Configure Time Machine exclusions
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Configuring zsh to support using Nix in non-interactive shells
* Create a `launchctl` plist to put Nix into your PATH
* Configure Nix daemon related settings with launchctl
* Remove directory `/nix/temp-install-dir`

Proceed? ([Y]es/[n]o/[e]xplain): Y

 INFO Step: Create an encrypted APFS volume `Nix Store` for Nix on `disk1` and add it to `/etc/fstab` mounting on `/nix`
 INFO Step: Provision Nix
 INFO Step: Create build users (UID 301-332) and group (GID 30000)
 INFO Step: Configure Time Machine exclusions
 INFO Step: Configure Nix
 INFO Step: Configuring zsh to support using Nix in non-interactive shells
 INFO Step: Create a `launchctl` plist to put Nix into your PATH
 INFO Step: Configure Nix daemon related settings with launchctl
 INFO Step: Remove directory `/nix/temp-install-dir`

Nix was installed successfully!
To get started using Nix, open a new shell or
run `. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh`

The installation takes only a moment or two to finish.

You can easily test it when it's done by running the following sequence of commands in your current shell.

# Load Nix into the current shell environment
source /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh

# Display the free space found on the new volume at /nix
df /nix

# Filesystem   512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
# /dev/disk1s7  976490576    799416 228759728     1%   73715 1143798640    0%   /nix

# Use Nix to install and run the `hello` command
nix run 'nixpkgs#hello'
# Produces `Hello, World!'

The nix run command is something akin to magic. To run the hello utility, Nix:

Downloads all the packages required to build hello, including tools such as make, gcc, and tar. (On the machine used to develop this article, an i9 iMac running MacOS Sonoma, Nix downloaded more than 400 prerequisite packages into /nix/store.)
Downloads the most recent version of hello (as no version number was specified).
Builds and executes the utility, producing Hello, World!.
Terminates and returns to the original shell.

If you run the same command — nix run 'nixpkgs#hello' — again, Nix executes hello immediately, as the assets required for the utility are cached and readily accessible.

You can use Nix to run almost any utility on-demand. Here is another example.

# Launch the `fish` shell (see https://fishshell.com)
nix run `nixpkgs#fish`
martin@iMac ~/p/o/a/martin (ruby-on-nix)>

The penultimate line of the output above is the default fish prompt, showing a user name, machine name, an abbreviated current working directory name, and the current git branch, if any. Press Control-D to exit the shell.

If you're following along, try this: Launch fish via Nix and enter the command which hello. The result should look like this:

strike@iMac ~/p/o/a/martin (ruby-on-nix)> which hello
strike@iMac ~/p/o/a/martin (ruby-on-nix) [1]>

which hello fails in the current environment. ([1] is the return value of the previous command, not the output of the previous command. Shell utilities return non-zero values on failure.) Where is the hello from the previous install? It's in its own encapsulated derivation, completely isolated from the one created for fish.

A one-off Nix derivation such as the one above has its uses, but a Nix derivation is not limited to a sole utility. A derivation may contain any number of packages. Better yet, a derivation can be declared, shared, and re-instantiated in identical form on any machine.

Nix for Ruby Development

Let's create a Nix derivation for Rails development. The derivation must have:

A version of Ruby, such as 3.2.0 (the latest version of Ruby available via Nix at the time of writing). Each Ruby package includes gem and irb.
A database service to persist information. Here, let's use PostgreSQL Version 15.
A key store for caching data, partials, and views. Redis is quite capable.
Tools for JavaScript and asset compilation.

Of course, a plenary Rails environment also needs the rails gem and many others. Ideally, a Nix derivation for Rails development also automatically bundles necessary gems, producing a standalone environment ready for development.

A Nix Derivation for Ruby Development

The code below is a nascent, but working, Nix derivation for Rails development. A Nix derivation is like a Dockerfile or a blueprint to build a working environment from scratch.

let
  nixpkgs =
    import (builtins.fetchTarball {
      url = https://github.com/NixOS/nixpkgs/archive/ee4a6e0f566fe5ec79968c57a9c2c3c25f2cf41d.tar.gz;
    }) { };

  targetRuby = nixpkgs.ruby_3_2;

  myBundler = nixpkgs.bundler.override {
    ruby = targetRuby;
  };

  gems =
    nixpkgs.bundlerEnv {
      inherit (nixpkgs) ruby_3_2;

      name     = "rails-gems";
      bundler  = myBundler;
      gemfile  = ./Gemfile;
      lockfile = ./Gemfile.lock;
      gemset   = ./gemset.nix;
    };

in
  nixpkgs.mkShell {
    buildInputs = [
      targetRuby
      gems
      gems.wrappedRuby
      nixpkgs.bundler
      nixpkgs.bundix
      nixpkgs.nodejs
      nixpkgs.yarn
      nixpkgs.postgresql
    ];
  }

The sample derivation has two sections. The let section defines settings, while the in portion controls the build and enumerates the packages to construct and install in the derivation.

Before discussing how to use the derivation, some commentary:

The setting nixpkgs requires a specific version of the Nix archive. Like a git repository, each iteration of the Nix packages archive is represented by a unique SHA reflecting the state of its contents. If the contents of the archive change, such as by incorporating a new version of Ruby, the SHA changes in tandem. Similar to referencing specific commits in git, you can lock your derivation to a specific iteration of the archive. The particular archive referred to here, ee4a6e0, contained a bug fix required to build the example derivation.
The setting targetRuby specifies a Ruby version. Here, the derivation uses the latest version of Ruby 3.2 available in the archive.
The setting myBundler customizes the Nix build to use the same version of Ruby specified for the shell environment.
Lastly, the setting named gems configures the Nix version of bundler. As with native bundler, the Nix bundler requires a Gemfile and a Gemfile.lock. gemset.nix is original to Nix and is a translation of Gemfile.lock to the Nix syntax.

If you want to know what versions of Ruby are available, point your browser to the Nix Packages Index and search for ruby. The search results include all the Ruby iterations available, such as ruby, ruby_3_2, and ruby_3_3, for Ruby 3.1, 3.2, and 3.3, respectively. Each package in the search result specifies the exact version number, such as 3.1.4 for the ruby package.

If you want to use a specific version of Ruby that is not named in the archive, try nixpkgs-ruby. For instance, to use Ruby 3.2.2, replace the line targetRuby = nixpkgs.ruby_3_2; in shell.nix with this code:

nixpkgs-ruby =
  import (builtins.fetchTarball {
    url = "https://github.com/bobvanderlinden/nixpkgs-ruby/archive/c1ba161adf31119cfdbb24489766a7bcd4dbe881.tar.gz";
  });

targetRuby = nixpkgs-ruby.packages.x86_64-linux."ruby-3.2.2";

Running the Nix Shell

Using the Nix derivation for Rails development proceeds much like development in any shell. The big exception: There is no need to install Ruby or attendant services beforehand. If you have Nix installed, you're ready to go.

Install Nix if necessary.
Clone a Rails project into a new directory or choose an existing codebase. cd to the directory for the project.
Copy and paste the code for the sample derivation to a new file in the project directory named shell.nix. (shell.nix is the customary name for a derivation, but the name is otherwise arbitrary.)

Generate gemset.nix to catalog the gems used within the project.

nix \
  --extra-experimental-features nix-command \
  --extra-experimental-features flakes \
  run 'nixpkgs/nixos-unstable#bundix' -- --lock

This command runs bundix, a Nix program, to convert Gemfile.lock into gemset.nix. Both are gem manifests, albeit the latter expressed natively in the Nix language. For comparison, here is an entry for actioncable from gemset.nix and from Gemfile.lock.

# From gemset.nix
#
actioncable = {
  dependencies = ["actionpack" "activesupport" "nio4r" "websocket-driver" "zeitwerk"];
  groups = ["default"];
  platforms = [];
  source = {
    remotes = ["https://rubygems.org"];
    sha256 = "0ifiz4nd6a34z2n8lpdgvlgwziy2g364b0xzghiqd3inji0cwqp1";
    type = "gem";
  };
  version = "7.1.3.2";
};

## From Gemfile.lock
#
actioncable (7.1.3.2)
  actionpack (= 7.1.3.2)
  activesupport (= 7.1.3.2)
  nio4r (~> 2.0)
  websocket-driver (>= 0.6.1)
  zeitwerk (~> 2.6)

Generate and launch the derivation.

nix-shell --verbose

The --verbose option is not required, but provides good and numerous insights into how nix-shell assembles the environment.

The Downsides: Nix has Some Flaws

Nix is a rich, robust, and burgeoning option for building, reproducing, and sharing environments. Nix is also an emergent tool: its current users are early adopters.

Nix is already a workable solution for many problems, but Ruby support seems to be a work in progress. For example, at the time of writing, no official archive is available for newer versions of Ruby 3.3, and there is no guarantee the latest Nix archive contains every gem in your Gemfile. Documentation for Ruby on Nix is scattered between various GitHub repos, the official Nix website, and user forums. I am grateful to a number of intrepid Nix developers who helped me craft the sample shell.nix, including Evan Travers, Bob van der Linden, and Norbert Melzer.

Nix currently is akin to git's "porcelain": powerful but esoteric. However, much like git evolved into exoteric, user-friendly tools such as git-flow, GitHub Desktop, and Tower to become user-friendly, many developers are building abstractions, wrappers, and utilities to simplify Nix usage. Let's briefly look at a few of these tools now.

Tools to Help with Nix: `devbox`, `devenv.sh`, and `fleek`

A tool to experiment with is devbox. devbox turned a 10-year-old Macbook Pro laptop running macOS Big Sur into a development machine for a modern Rails application in a few minutes. The setup did not include automatic bundling of gems, but it did install Ruby (including bundler and gem), Redis, and PostgreSQL in an isolated and pristine shell environment. devbox eschews a bespoke programming language for configuration and uses JSON. The environment on the MacBook Pro booted from this minimal file named devbox.json:

{
  "$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.10.6/.schema/devbox.schema.json",
  "packages": [
    "postgresql@latest",
    "redis@latest",
    "ruby@3.2.2",
    "libsass@latest"
  ]
}

devenv.sh merits exploration too. It is something of a hybrid, with a JSON-like programming language, YAML configuration, and Docker-like composition of services.

fleek is another Nix-based tool to craft and share configurations for development machines. It also avoids the Nix programming language and instead provides Homebrew-like commands to build an environment.

Wrapping Up

In this post, we've seen how Nix can help reproduce a stable environment for Ruby and Rails applications.

While it has some flaws, Nix and the myriad offshoots in development are worth watching.

Go forth and hack!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Setting Up Custom Metrics with Effective Alerts for a Ruby App in AppSignal

Roy Tomeij — 2024-07-31T00:00:00+00:00

Most of the time, the default application monitoring metrics, graphs, and visualizations provided by AppSignal will do for your Ruby app. However, you might be the kind of user who likes a bit of control over what is measured, how it’s displayed, and how critical information about your app should be relayed.

AppSignal allows you to customize app metrics and dashboards as you wish. In this guide, we’ll learn all about AppSignal's custom metrics, including:

What custom metrics are
The different types of custom metrics you can set up
How to customize graph visualizations
How to set up effective alerts

And more!

But before we dive in, you'll need a couple of things to follow along.

Prerequisites

An AppSignal account: If you don't have one, sign up for a 30-day free trial.
A Ruby application: This app can be based on any supported Ruby framework, like Rails, Sinatra, or just plain Ruby. Additionally, it can be a production or development app. If you don't want to spin up your own app, clone the code for the example Sinatra app we'll be using in this tutorial.

Note: If you're using your own app to follow along with this tutorial, make sure your app is configured to use the latest AppSignal Ruby gem, as the examples used in this tutorial assume this is the case.

What Are Custom Metrics?

Other than measuring your app's error rates, throughput, and performance, you might be interested in measuring custom data specifically tailored for your own app. For example, you could be interested in how many visitors signed up to your app in a particular time period, how your app's websocket layer is performing, and so forth.

For such customized cases, you might be hard-pressed to find a standard measuring tool within AppSignal. Instead, you'll need to use a custom metric. Custom metrics are additional metrics that you define alongside the default ones for deeper context on how your app is running.

Next up, let's learn how to set up our first custom metric.

Setting Up Custom Metrics

You can set up a custom metric for almost any use case within your application. Let's start with a simple example to help you understand how everything fits together.

The first step is to define a custom metric that will be tracked on AppSignal. You can define a custom metric by using the different metric types available:

Gauges
Counters
Distributions

The Gauge Custom Metric

In AppSignal, a gauge custom metric is useful for measuring metrics that increase and decrease over time.

Let's set up a simple gauge custom metric to measure the total number of posts in our example Sinatra app:

# app.rb

require 'sinatra/base'
require 'dotenv/load'
require 'sinatra/reloader'
require 'sinatra/activerecord'
require './models/post.rb'
require 'appsignal/integrations/sinatra'

class App < Sinatra::Base
  ...

  get '/' do
    posts = Post.all

    # set a gauge custom metric
    posts_count = posts.count
    Appsignal.set_gauge("all_posts", posts_count)

    posts.to_json
  end
...

In the code shown above, we use the Appsignal::Helpers::Metrics module and call the set_gauge method, which accepts three arguments:

key: the name of the custom metric. In the example, this would be all_posts.
value - The metric or "thing" to be measured. In the example shown above, this is simply the total post count.
tags - Additional and optional metadata that can be added to a custom metric and are useful for labeling the data being measured however you want. For example, we could easily tag the posts_count metric to account for the environment, as shown below:

# app.rb

...

get '/' do
  posts = Post.all
  posts_count = posts.count
  Appsignal.set_gauge("all_posts", posts_count, environment: "development")

  posts.to_json
end

...

Great, we've just added our first custom gauge metric! But if you were to go back to AppSignal, your new custom metric will not be visible. Instead, you're likely to see the default dashboard, as shown below:

So what do you need to do to make the custom metric appear? You need to add a dashboard. Start with creating a new dashboard:

Then name your new dashboard with a descriptive title and description:

With the custom dashboard added, you'll now need to add a graph for the custom metric:

Then define the new graph:

Here's a breakdown of the fields to set up the new graph:

a. Title - Enter a descriptive title for the new graph.
b. Description - This is optional, but you can enter a description for the new graph.
c. Metrics - Here is where you define the metric that will be measured and displayed by the new graph. This is the name of the custom metric, or the first argument defined in the set_gauge method: all_posts. In this section, you can also define tags (for example, the tag environment is also included as shown).
d. Graph display - This is where you choose the type of graph display for your new graph.
e. Legend label - You can customize the label for the chart legend here.
f. Data format - Define the data type used for the graph display. You can choose from a number of formats, including number, percentage, throughput (in requests/minute or hour), duration (in milliseconds), or file size (in bytes).

Once you've properly defined the new graph's properties, you should get a graph for the custom metric. This is similar to what is shown below:

Moving forward, let's look at the next custom metric type: the counter.

The Counter Custom Metric

A counter custom metric is great for measuring how many times an event occurs. Using the example application, we can apply a counter metric to measure every time the home (root) page is visited.

To begin with, edit the root method to include the code shown below:

# app.rb
...
get '/' do
  ...
  Appsignal.increment_counter("visits_count", 1)
end
...

Here, we use AppSignal's increment_counter method and pass it visits_count as the first argument. The increment step is the integer 1, passed as the second argument. You could also add a tags hash as the third argument, but we'll leave it as is (since this was covered in the previous section).

Now go ahead and follow the steps as outlined for the gauge metric type. Add a custom graph for this counter metric to give you a similar graph to the one shown:

Let's switch gears to the distribution custom metric.

The Distribution Custom Metric

The AppSignal distribution custom metric is useful for measuring something per unit of time: for example, how many seconds it takes to generate a PDF report, or how long a background job takes to execute.

Using the example application, let's modify the main file to include a call to an open API endpoint. Then, we'll use a custom distribution to measure how long the API call takes in milliseconds.

# app.rb
...

require 'httparty'

class App < Sinatra::Base
  ...

  Thread.new do
    while true do
      start_time = Time.now
      url = 'https://openlibrary.org/search/authors.json?q=clavell'
      $response = HTTParty.get(url)
      duration = (Time.now - start_time) * 1000
      Appsignal.add_distribution_value("fetch_books_duration", duration)
    end
  end

  get '/' do
    ...
    $response.to_json
  end

  ...
end

Now, if we go back to AppSignal, we can view the custom distribution as a graph.

Tip: You can follow the steps outlined in the gauge section to set up the custom graph visualization.

Now that you've learned how to create custom metrics and the accompanying graph visualizations, you might have noticed that it's not very convenient to keep going back to the AppSignal dashboards to see what's going on with your app. Instead, it would be very handy if you could get a notification for your custom metrics, right?

Let's learn how to set up notifications for your custom metrics next.

Notification Alerts

By default, whenever an error or performance event occurs, AppSignal will open an incident for that event and place it in the relevant section. For example, if it's an error, you'll find it in the errors list, while performance incidents will be in the performance list.

Additionally, AppSignal sends out incident notifications via email (the default notification channel). You can also set up other notification channels, such as:

Discord
Google Hangouts
Intercom
Microsoft Teams
Slack
Webhook

And more.

But before we set up a notification trigger for one of our custom metrics, it's important to be aware of the various notification options available to you.

To begin with, you can set up a notification for:

Every time - Here, a notification will be sent out every time an incident occurs.
First deploy - This indicates that a notification will be sent the first time an incident occurs after a deployment.
First after close - Here, a notification is sent out whenever an incident re-occurs after the previous one was closed.
Never notify - As the name suggests, in this case, a notification will never be sent, but the error or performance incident will still be tracked on AppSignal.
Every nth hour or day - With this option, you can specify how many alerts will be sent to you within an hour or a day. This option is perfect for keeping a balance between getting notified of important events and having too many notifications (which could easily overwhelm you or your team).

I highly suggest you dig into AppSignal's notification settings documentation to get more information on these options.

Let's see how to set up a notification for one of the custom metrics we made earlier. This will be a trivial example, but it will illustrate the steps you need to go through for your own use case.

Setting Up Notification Alerts for Your Custom Metrics

For this example, we'll use the distribution metric that measured the API call's duration earlier in this post. Let's say we'd like to get an email alert whenever the mean duration exceeds a certain number (in milliseconds).

The steps for setting this up are shown below:

Firstly, begin by hitting the Triggers link in the left-hand menu.

Give your trigger a relevant name, then select the measurement for which this notification is intended. In this example, we are using the fetch_books_duration distribution custom metric. You can also add tags if you wish to.

Next, define the comparison operator and the value to check for. For example, let's say we want to receive an alert whenever the duration exceeds 1600 milliseconds. For this, we would select the comparison operator more than, then, a value of 1600.

Finally, you'll need to define the alert warm-up and cooldown settings. Provide a description for the alert, a link to the dashboard to be included in the alert message (if needed), and, finally, the notification method (with email being the default).

With that done, you'll receive a notification whenever an incident occurs matching the settings you input here.

And that's it!

Wrapping Up

In this article, we learned how to set up custom metrics for a Ruby application with matching dashboards and graph visualizations on AppSignal.

The custom metrics functionality that AppSignal offers can be fine-tuned for very powerful applications. Take a deep dive into AppSignal's custom metrics documentation to discover more possibilities.

Until next time, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

What's Coming in Ruby on Rails 7.2: Database Features in Active Record

Roy Tomeij — 2024-07-24T00:00:00+00:00

Ruby on Rails is currently in major version 7.1 and rolling towards Rails 8, the next comprehensive new release.

Before Rails 8, though, there’s a significant version that will help bridge the gap: Ruby on Rails 7.2.

In this post, we’ll dive into several noteworthy changes in Ruby on Rails 7.2, focusing on the support for database changes in Active Record.

You'll come away with hands-on opportunities to work with these features.

Let's get started!

Active Record Database-Related Features in Ruby on Rails 7.2

The Active Record object-relational mapper (ORM) supports three open-source relational database management systems: MySQL, PostgreSQL, and SQLite. Active Record is most commonly thought of for its ORM facilities, mapping between database operations and database types to Ruby methods and types. However, besides the ORM, Active Record has mechanisms to evolve your database schema definition, capturing incremental changes and a representation of the entire schema in Ruby files or SQL files.

Active Record can connect a single Rails app to multiple databases, and those databases can have distinct roles. The databases might have writer or reader roles or even be “shards” that leverage the Horizontal Sharding feature.

Recently, Beta releases of 7.2 have shipped, allowing developers to preview what’s changed.

As we roll towards Rails 8, we must first pass through 7.2. Let’s examine what's in this release.

What’s New with Composite Primary Keys (CPKs)

Rails 7.1 brought us the first release of composite primary keys (CPKs). What are CPKs? First, let’s talk about unique identifiers in databases more generally. Uniquely identifying rows can take the shape of “natural keys” or “surrogate keys”. Since most databases use surrogate keys, let’s focus on those. Surrogate keys are when “id” integer values are used to uniquely identify a row. This is a surrogate in that it’s not directly related to the data row.

In Rails, the surrogate primary key is usually a column called “id” with an integer type, populated by the database. In PostgreSQL, a sequence object usually provides the value.

What does this have to do with indexes? When we define our id column as the primary key column, this creates a primary key constraint on the table, and constraints have a supporting index. Primary key constraints enforce unique values, and the index helps the lookups performed to enforce that rule to run quickly.

CPKs, or multi-column primary keys, are primary keys where multiple columns uniquely identify a table row. This is a generic mechanism, so it’s up to you as to how you describe your CPK. You may choose to incorporate a surrogate value like the id integer, and combine that with a unique identifier for one of your customers, for example, a customer_id.

You may choose a CPK as a natural key composed of two foreign key columns that point to other tables. This structure is most common with join tables.

With that context in mind, let's arrive at what’s changing in Rails 7.2.

Currently, for the CPK feature, we use the query_constraints keyword on models that relate to other models, and supply a list of foreign key columns that refer to the foreign table primary key columns.

In Rails 7.2, this option changes from query_constraints to foreign_keys. Besides the parameter name change, some interesting discussions are underway about plans to repurpose the original query_constraints name by freeing it up.

If you’d like to play around with single-column surrogate keys, and compare them with composite or multi-column primary keys as a type of natural key, take a look at this Bookshop repo.

Here, we’ve got the models listed in the Rails API documentation for CPKs implemented as a single file Rails app. You can modify the code and database schema design and try out different alternatives.

To try out Rails 7.2 changes, edit the Gemfile portion within the bookshop.rb Ruby file. Use a 7.2 beta version by putting this change into the file, then running bundle install from your terminal:

gem 'rails', '~> 7.2.0.beta1'

Development Containers and Databases in Rails 7.2

Ruby on Rails has always been a productivity-centric framework. One challenge that impacts the developer experience and productivity is creating and maintaining a development environment.

Dev containers intend to fix that. They're used to create reproducible development environments that anyone on your team can run.

Besides reproducibility, another benefit is the isolation of environment dependencies. Some operating system dependencies you use can be particularly challenging to install, or multiple versions might coexist. By isolating dependencies into a container, you can avoid those pitfalls.

Dev containers use Docker, but add a “devcontainer.json” file to the mix. This can describe the text editor configuration and be shared on a team that uses the same editor and configuration.

Let’s generate a new app with a dev container within the bookshop repository. Make sure that Rails 7.2 is the version used by the rails executable, installing the gem if needed.

We'll add --database=postgresql which will configure a Postgres instance in Docker, and install the pg gem for the generated Rails app.

gem install rails -v 7.2.0.beta1 # if needed

rails new myapp --database=postgresql --devcontainer

Open "myapp" in VS Code from the interface or by running code myapp when the code executable is available.

Once in VS Code, if not auto-detected, run "Command-shift-P" and choose “Dev Container: Rebuild in Container”. You'll see “Starting Dev Container”, which starts up Docker as needed.

Click “Show Log” to expand the terminal area, showing log progress. Here, you can tell whether containers are being downloaded and which stage they’re in.

The containers can be viewed and administered using Docker Desktop.

The Rails app, Postgres, and all its gem dependencies should now be running inside the container.

Open a terminal in VS Code and run bin/rails server. Navigate to localhost:3000 in your browser. Since there’s a local port mapping from 3000 into the container, you should see the generated Rails 7.2 app welcome page!

If you run into issues, visit the bookshop repo for more information.

Database Transaction Enhancements, Solid Queue, and Active Job

Transactions are one of the fundamental concepts of relational databases. Databases are designed to support high concurrent access to shared resources, like table row data. For example, multiple clients might try to read and write to the same table row at once. The database consistently processes operations by using transactions with an isolated view of the data.

One of the problems in Rails apps comes when working with relational database transactions and then with another data store like Redis (via background processing with Sidekiq or other Active Job backends).

The issue is one of timing: ensuring data is committed to the relational database before any background work starts.

With Solid Queue — a database-backed queue management system — coming in Rails 8, ensuring transactionally consistent data and operational order is even more important. Transactional consistency errors will become more visible when there's a first-party database-backed queue system.

To prepare for that, what's changed in Rails 7.2? Active Job will now defer enqueuing background jobs until after a database transaction has been committed. This small change ensures no background job processing starts until the database transaction is committed.

Another change in 7.2 is that callbacks will be registered on database transactions.

For example, after_commit can be added within a transaction block to ensure that it runs after the transaction is committed.

Here's the example used in the release notes:

Article.transaction do |transaction|
  article.update(published: true)

  transaction.after_commit do
    # Do work after commit, like send a notification email
  end
end

This approach ensures the code in the after_commit block runs after the article is updated.

New Active Support Instrumentation for Transactions

The Beta 3 version of Rails 7.2 was recently released, and it included new Active Support Instrumentation events we can configure for our applications.

One of these new events is called start_transaction.active_record, and it is triggered when database transactions or savepoints within transactions are started.

Check out the blog post You make a good point! — PostgreSQL Savepoints for more information on savepoints.

When database transactions finish, the event transaction.active_record event is emitted.

What can we do with these events? By creating an Active Support Subscriber that inspects these events and their payloads, we can gain a better understanding of database transaction and savepoint activity within Active Record.

Check out the commit code changes and documentation.

Wrapping Up

In this post, we looked at a few database-related features coming in Ruby on Rails 7.2. We covered the basics of CPKs and an important option name that's changing in 7.2. We also examined how we'll be able to run all of our application dependencies (including our databases) in a dev container.

Finally, we covered how database transactions and database-backed background job systems will tackle the challenge of write operations occurring in the expected order.

Besides those items, there's plenty more to read and learn about Rails 7.2. For example, Ruby 3.1 will become the new default minimum version, there will be support for jemalloc, RuboCop rules, a GitHub CI workflow, and support for progressive web apps (PWAs). To learn more, check out the Rails 7.2 release notes.

Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Auth0 for Ruby on Rails

Roy Tomeij — 2024-07-17T00:00:00+00:00

From custom-made to plug-and-play forms of authentication, Ruby developers have plenty to choose from these days. Yet, as you may know, building your own solution can be costly and dangerous. If Devise is the de facto standard for most teams, an alternative might simplify the lives of most.

This article will cover the setup and use of Auth0 in a Ruby on Rails application, including everything you need to get going properly, from handling roles to relying on multiple providers to authenticate users.

Getting Started

Here's what we need to get started:

An Auth0 account
A Ruby on Rails application (version 7.x onwards)

Auth0 is a third-party authentication service with a free tier that lets you handle up to 7,000 users. That is plenty to get you started, and its pricing is reasonable if you need more advanced features.

Configuring Our Ruby App in Auth0

Since your application will rely on Auth0 to authenticate users through redirects and calls, we must ensure it stays secure.

In our Auth0 account, let's create an application within a tenant. You can create multiple tenants in an account to separate:

Domain names.
Different environments (development, production, etc).
The country or region within which data will be stored.

Once you have created your first tenant, you can build an application. That's where we'll start.

Head to the "Settings" tab in the application panel. You need to copy and paste the following and save it to a safe place:

The domain name: app-name.[eu,us,..].auth0.com
The client's ID
The client's Secret

We must also fill in the following Application URIs. We'll use these values for our local development setup:

Allowed Callback URLs: http://localhost:3000/auth/auth0/callback (the URL Auth0 will redirect to after authentication).
Allowed Logout URLs: http://localhost:3000 (the URL Auth0 will redirect to after someone logs out).

Let's go ahead and configure our app.

Preparing Our Ruby on Rails Application

You can start with a vanilla Ruby on Rails application using the rails new command.

Let's generate one that relies on SQLite3 for the database and Tailwind for the CSS library. Skip the installation of mini-test and name the application "Auth0 article":

cd ~/my_projects
rails new -d sqlite3 -c tailwind -T auth0_article

Create a User model with just a few attributes:

cd auth0_article
bin/rails db:create
bin/rails generate model User email name
bin/rails db:migrate

This builds a simple but efficient base for our application.

Adding the Auth0 Gem to the App

You'll need omniauth-auth0 and omniauth-rails_csrf_protection to use Auth0 in your Ruby on Rails application.

gem 'omniauth-auth0', '~> 3.0'
gem 'omniauth-rails_csrf_protection', '~> 1.0'

Configuring Auth0

We need to create a tiny configuration file (config/auth0.yml) to store credentials in our development environment.

development:
  auth0_domain: <YOUR AUTH0 APPLICATION DOMAIN NAME>
  auth0_client_id: <YOUR AUTH0 APPLICATION CLIENT ID>
  auth0_client_secret: <YOUR AUTH0 APPLICATION CLIENT SECRET>

We can also rely on environment variables here by using some erb:

development:
  auth0_domain: <%= ENV['AUTH0_APPLICATION_DOMAIN'] %>
  auth0_client_id: <%= ENV['AUTH0_APPLICATION_CLIENT_ID'] %>
  auth0_client_secret: <%= ENV['AUTH0_APPLICATION_CLIENT_SECRET'] %>

Of course, relying on Ruby on Rails credentials ensures that things stay more up-to-date.

This file will be used in the Auth0 initializer (config/initializers/auth0.rb), which we will now create:

AUTH0_CONFIG = Rails.application.config_for(:auth0)

Rails.application.config.middleware.use OmniAuth::Builder do
  provider(
    :auth0,
    AUTH0_CONFIG['auth0_client_id'],
    AUTH0_CONFIG['auth0_client_secret'],
    AUTH0_CONFIG['auth0_domain'],
    callback_path: '/auth/auth0/callback',
    authorize_params: {
      scope: 'openid profile'
    }
  )
end

As you can see here, we are using Rails.application.config_for to load up the content of a YAML file as a convenient hash. We could replace this with any secret handling library, including Rails encrypted credentials storage.

Note the following:

The provider name (:auth0)
The three items from the YAML file, read and used through the AUTH0_CONFIG hash.
The callback_path key and value, matching the one we configured in the Auth0 interface.
The authorize_params and the scope key inside it; we will come back to that later.

We need to create the interface and routing elements to allow for authentication.

Setting Up Routes and UI with Tailwind

In this example, we will work with a Ruby on Rails application using version 7.1 of the framework with TailwindCSS.

Use the following command:

rails new -d sqlite3 -c tailwind -T myApp

We can then add two controllers with the index action, preparing to test the authentication process:

bin/rails g controller public index
bin/rails g controller private index

With those two commands, the following are created:

Both public_controller.rb and private_controller.rb controllers, with the index action ready to use
Both related routes
The related views

Let's add the Auth0 controller to handle the callbacks and failures. Create the controller file (bin/rails g controller auth0) and add the following:

# ./app/controllers/auth0_controller.rb
class Auth0Controller < ApplicationController

  # this will happen in case of success
  def callback
    auth_info = request.env['omniauth.auth']
    session[:userinfo] = auth_info['extra']['raw_info']

    redirect_to '/private/index'
  end

  # this will happen in case of failure
  def failure
    @error_msg = request.params['message']
    redirect_to '/public_index'
  end

  # logout route
  def logout
    reset_session
    redirect_to logout_url, allow_other_host: true
  end

  private

  def logout_url
    request_params = {
      returnTo: root_url,
      client_id: AUTH0_CONFIG['auth0_client_id']
    }

    URI::HTTPS.build(host: AUTH0_CONFIG['auth0_domain'], path: '/v2/logout', query: request_params.to_query).to_s
  end
end

Then update the routes to use those three actions:

./config/routes.rb
Rails.application.routes.draw do
  # ..
  get '/auth/auth0/callback' => 'auth0#callback'
  get '/auth/failure' => 'auth0#failure'
  get '/auth/logout' => 'auth0#logout'

  root "public#index"
end

And then we can add Login and Logout buttons in the public and private views, respectively:

# app/views/public/index.html.erb
<div>
  <h1 class="font-bold text-4xl">Public#index</h1>
  <p>Find me in app/views/public/index.html.erb</p>
  <%= button_to 'Login', '/auth/auth0', method: :post, data: { turbo: false }, class: "rounded bg-sky-500 px-2 py-1 text-sm font-semibold text-white shadow-sm hover:bg-sky-400 focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-sky-500" %>
</div>

# app/views/private/index.html.erb
<div>
  <h1 class="font-bold text-4xl">Private#index</h1>
  <p>Find me in app/views/private/index.html.erb</p>
  <%= button_to 'Logout', '/auth0/logout', method: :get, data: { turbo: false }, class: "rounded bg-sky-500 px-2 py-1 text-sm font-semibold text-white shadow-sm hover:bg-sky-400 focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-sky-500" %>
</div>

You can now go to 'http://localhost:3000'. Use the Login button and you'll be redirected to the private page. There, you will find a Logout button.

A couple of things are missing, though:

Data from a user's profile
Identifying a user and handling authorization

Working with Scopes and Data

Let's take a few steps back and bring back the initializer (config/initializers/auth0.rb):

AUTH0_CONFIG = Rails.application.config_for(:auth0)

Rails.application.config.middleware.use OmniAuth::Builder do
  provider(
    :auth0,
    AUTH0_CONFIG['auth0_client_id'],
    AUTH0_CONFIG['auth0_client_secret'],
    AUTH0_CONFIG['auth0_domain'],
    callback_path: '/auth/auth0/callback',
    authorize_params: {
      scope: 'openid profile'
    }
  )
end

The critical piece here is the scope: openid profile. This tells Auth0 we are interested in a few pieces of information, namely:

The provider name (Auth0)
A uid: A unique identifier to match a user
An info hash: Containing a name, URL to a profile picture, and an empty 'email' key
An extra_info hash: Again, with a name and profile picture, but also a pair of given and family names

Read more on the topic of scopes in Auth0's documentation.

The above information is valuable, and you should ensure that, at least upon first login, you copy that data into a table in your application.

To do so, we need to use the content of the request-response in the auth0_controller and, specifically, in the callback action:

def callback
  # all the data sent by auth0
  auth_info = request.env['omniauth.auth']

  # the data that really identifies the user
  session[:userinfo] = auth_info['extra']['raw_info']

  redirect_to '/private/index'
end

Use a breakpoint before the redirect to dig into the hash and experiment.

We usually want the email address too. To get it, you can update the scope:

scope: 'openid profile email'

After reloading the application server and going through the login steps, you will need to give the application access to additional information.

We can now do something like this in the callback action:

def callback
  # all the data sent by auth0
  auth_info = request.env['omniauth.auth']

  # the data that really identify the user
  session[:userinfo] = auth_info['extra']['raw_info']

  user = User.find_by(email: session[:userinfo]['email'] || User.new
  if user.new_record?
    user.name = session[:userinfo]['name']
    user.email = session[:userinfo]['email']
  end
  user.save

  redirect_to '/private/index'
end

This will ensure we have a local, up-to-date profile for the user.

A Word On Sessions

A session is a special hash in a Ruby on Rails application. It has limited storage that is accessible from the controllers and views and unique to each visitor.

We can "open" and "close" sessions. If we need a specific dataset, we can decide that a session is open. If it has no data, then it's closed.

Remember the following lines in our Auth0 controller:

# in the callback method
session[:userinfo] = auth_info['extra']['raw_info']

# in the logout method
reset_session

The first one writes the value in the raw_info key of the hash to the session hash, while the second one just clears up the session hash.

We can then define the following helper method in the ApplicationHelper:

# get the user
def current_user
  User.find_by(email: session[:userinfo]['email']) if session[:userinfo]
end

A Security Concern

Now let's use this as a concern for our controllers. The idea is to define a controller that requires users to be logged in to access it, such as our private_controller.

We can write the following concern file:

# ./app/controllers/concerns/secured.rb
module Secured
  extend ActiveSupport::Concern

  included do
    before_action :logged_in?
  end

  def logged_in?
    redirect_to '/' unless session[:userinfo].present?
  end
end

We can then use it in our controllers like so:

class PrivateController < ApplicationController
  include Secured

  def index
  end
end

If a visitor isn't logged in, but then tries to open up the /private/index page, they will automatically be redirected to the site's root.

Integrating with Other Providers

You can rely on multiple providers through Auth0. However, Google is defined as the default. For many companies, that's enough.

If you need more providers, head to the Authentication menu for the related tenant in Auth0, and then the Social submenu, where you can set up additional providers.

It's also worth noting that you can configure multi-factor authentication (MFA) with Auth0 too.

The process will remain the same except for those configuration steps. Our application will only return visitor data to prove that a visitor has been authenticated.

Opening Up Towards Authorization

Now we can authenticate users and send their information to our application through a callback. We have already seen how to get a user's email address out of a hash and find the relevant user in our database.

From there, we can define authorization policies.

Pundit is a great choice to define and use authorization policies, yet relies on checking a user's role.

Using a role attribute is, in fact, the easiest method. You can fill it in when creating a user in the Rails console or your application's back-end interface, before a user's first login attempt.

Or you can get a list of admins' emails and match a user's email against the list when creating the user.

ADMINS = ['johndoe@example.com']

def find_or_create_user(email:, name:)
  user = User.find_by(email: session[:userinfo]['email'] || User.new
  if user.new_record?
    user.name = session[:userinfo]['name']
    user.email = session[:userinfo]['email']
  end
  if ADMINS.include?(user.email)
    user.role = 'admin'
  else
    user.role = 'normal'
  end

  user.save
end

And that's it!

What We've Covered

In this article, we set up:

A tenant and application in Auth0
Auth0-related gems in a Ruby on Rails application
Routes and views in the application

We then:

Reviewed the concept of a session and saw how to use it
Created base tooling to check if a user is logged in
Added a controller concern to secure controllers behind a login requirement
Saw how to match users to their roles

Auth0 and other authentication providers can help you integrate state-of-the-art authentication in your Ruby on Rails application without writing much code. It's often a much better option than relying on your own implementation of the authentication layer or even using gems like Devise.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Debugging in Ruby with Debug

Roy Tomeij — 2024-07-03T00:00:00+00:00

Debugging is a valuable skill for any software engineer to have. Unfortunately, most software engineers are not trained in it. And that's not just specific to developers going through boot camps; even in universities, we are not often taught and trained to use a debugger.

My teachers and mentors were more interested in getting me to write programs rather than debugging them. If we are fortunate, debugging comes at the end of the semester, in a short, last session.

Luckily, we have tools that can help us with debugging. Since Ruby 3.1, Ruby ships with the debug gem, a powerful debugger.

In this article, we will go through a quick overview of the gem. We'll see how to use it for simple and more advanced cases.

Debugging Without A Debugger: What's the Issue?

Many of us rely on what you might call "printf debugging": we add puts (or its equivalent in the language we're using) to the standard output (STDOUT). We include the current state of an object, variable, or just a string so we know if our program is going into specific branches of its logic tree.

While helpful, this isn't the most optimal way to debug a program. It often leads to many back-and-forth trips between your logs and the code, as you forget to add a puts here and there, or leave in some debugging code.

That method also relies on your own preconceptions about how the code is running and what is going on that's different from what you might expect.

Using a debugger is a very different experience. You add one or more breakpoints in the code where you want to know what's happening. You then run the code and wait for it to hit the breakpoint.

Then, you get a debugging console to check a variable's values at the breakpoint location. You go back and forth in the execution steps.

As we will see later, we can even add conditional breakpoints directly from the debugging console. This makes it easier to avoid exiting the debugging console, so you can add breakpoints you've forgotten about.

Setup

Since Ruby 3.1, a version of the debug gem ships with Ruby. We recommend adding it to your Gemfile so you're using the latest version.

Add debug to your Gemfile and then run bundle install. I recommend adding it to development and test groups for debugging tests too.

Basic Debugging Techniques with Debug for Ruby

Now let's run through some simple debugging methods using debug: using breakpoints, stepping, other commands, moving in the stack, and using a map. We'll then examine the more advanced method of adding breakpoints on the fly.

Breakpoints

Breakpoints are calls that tell the debugger to stop. You can do this in modern IDEs that are integrated into a debugger with a simple click in the sidebar. The standard way is to add binding.break at the line we want to stop at.

require 'debug'

class Hornet
  def initialize
    @colors = [:yellow, :red, :black]
  end

  def show_up
    binding.break   # debugger will stop here
    puts "bzzz"
  end
end

Hornet.new.show_up

By running this little program, we will get the following console output:

[debug] ruby test.rb
[4, 13] in test.rb
     4|   def initialize
     5|     @colors = [:yellow, :red, :black]
     6|   end
     7|
     8|   def show_up
=>   9|     binding.break   # debugger will stop here
    10|     puts "bzzz"
    11|   end
    12| end
    13|
=>#0	Hornet#show_up at test.rb:9
  #1	<main> at test.rb:14
(ruby) @colors
[:yellow, :red, :black]
(rdgb)

As you can see, we can access the instance variable from the breakpoint.

Stepping

Let's dig into a more complex example using stepping.

class Book
  attr_accessor :title, :author, :price

  def initialize(title, author, price)
    @title = title
    @author = author
    @price = price
  end
end

class BookStore
  def initialize
    @books = []
  end

  def add_book(book)
    @books << book
  end

  def remove_book(title)
    @books.delete_if { |book| book.title == title }
  end

  def find_by_title(title)
    @books.find { |book| book.title.include?(title) }
  end
end

# Sample Usage:
store = BookStore.new
book1 = Book.new("Dune", "Frank Herbert", 20.0)
book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

store.add_book(book1)
store.add_book(book2)
store.add_book(book3)

puts store.find_by_title("Hobbit").title

This example app manages books in a bookstore. But at the moment, we cannot be sure which book will be returned when we search titles containing 'Hobbit'. It might well be "The Hobbit", but it's not certain.

To help debug this, we'll jump into the find_by_title method.

Let's add a breakpoint to one of the methods:

def find_by_title(title)
  binding.break
  @books.find { |book| book.title.include?(title) }
end

Then launch the program and get to the breakpoint:

@box [debug] ruby library.rb                                                                                                                          20:25:07
[22, 31] in library.rb
    22|   def remove_book(title)
    23|     @books.delete_if { |book| book.title == title }
    24|   end
    25|
    26|   def find_by_title(title)
=>  27|     binding.break
    28|     @books.find { |book| book.title.include?(title) }
    29|   end
    30| end
    31|
=>#0	BookStore#find_by_title(title="Hobbit") at library.rb:27
  #1	<main> at library.rb:42
(rdbg)

The top part of the console tells us which line and file we are at. We can then query the value of the title variable.

(rdbg) title
"Hobbit"
(rdbg)

We can run the code right in that context to see what's happening:

(ruby) @books.find { |book| book.title.include?(title) }
#<Book:0x00007fd05e4d59f0 @author="J.R.R. Tolkien", @price=15.0, @title="The Hobbit">
(rdbg)

Here might be a good time to reflect on how you want the program you are building and this piece of code to behave. Expressing the code through RSpec tests might be an excellent way to clarify what it should do.

Let's now continue to the next breakpoint by using the continue command.

(rdbg) continue    # command
The Hobbit

In this case, it goes on until the end of the program.

More Commands to Assist Debugging

Of course, we can add more breakpoints to our code to stop at another place. But we can also use commands to move within the stack of our program without restarting it.

Let's add one more breakpoint to the add_book method, just after instantiating the bookstore.

def add_book(book)
  binding.break
  @books << book
end

# [ .. ]

store = BookStore.new
binding.break
book1 = Book.new("Dune", "Frank Herbert", 20.0)
book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

Now, when we run the program, it will stop before the book1 variable is instantiated. The continue command will run the program until the next breakpoint or exit.

Using `next`

Instead of continue, we can use the next command, which will only run the next code line, so we can debug our app in smaller steps. We will need to run next twice to run the line where book1 is defined before we can inspect it.

[30, 39] in library.rb
    30|   end
    31| end
    32|
    33| # Sample Usage:
    34| store = BookStore.new
=>  35| binding.break
    36| book1 = Book.new("Dune", "Frank Herbert", 20.0)
    37| book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    38| book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)
    39|
=>#0	<main> at library.rb:35
(ruby) book1
nil
(rdbg) next    # command
[31, 40] in library.rb
    31| end
    32|
    33| # Sample Usage:
    34| store = BookStore.new
    35| binding.break
=>  36| book1 = Book.new("Dune", "Frank Herbert", 20.0)
    37| book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    38| book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)
    39|
    40| store.add_book(book1)
=>#0	<main> at library.rb:36
(ruby) book1
nil
(rdbg) next    # command
[32, 41] in library.rb
    32|
    33| # Sample Usage:
    34| store = BookStore.new
    35| binding.break
    36| book1 = Book.new("Dune", "Frank Herbert", 20.0)
=>  37| book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    38| book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)
    39|
    40| store.add_book(book1)
    41| store.add_book(book2)
=>#0	<main> at library.rb:37
(ruby) book1
#<Book:0x00007f50fb5f9da0 @author="Frank Herbert", @price=20.0, @title="Dune">

Each next call will run the next line. But it will not step into the code called by Book.new.

Using `step`

In some cases, we may know that an issue lies within a specific call. The step command is great for debugging this.

For example, when we are at line 37, we can use step to follow the execution of the Book object that fills the book2 variable.

(rdbg) step    # command
[2, 11] in library.rb
     2|
     3| class Book
     4|   attr_accessor :title, :author, :price
     5|
     6|   def initialize(title, author, price)
=>   7|     @title = title
     8|     @author = author
     9|     @price = price
    10|   end
    11| end
=>#0	Book#initialize(title="The Hobbit", author="J.R.R. Tolkien", price=15.0) at library.rb:7
  #1	[C] Class#new at library.rb:37
  # and 1 frames (use `bt' command for all frames)

The step command brings us directly to the first line of the initialize method in the Book class. (If you are new to Ruby, the new class method is called the initialize method after it does some internal work). Now we can use the next step from within that method and follow the trail.

next and step are crucial to get familiar with, as they allow us to move forward at different levels and speeds.

Moving In the Stack

We can move up and down (or backward and forwards) in the stack by using the up and down commands. Calling up twice will get us back to line 37:

[2, 11] in library.rb
     2|
     3| class Book
     4|   attr_accessor :title, :author, :price
     5|
     6|   def initialize(title, author, price)
=>   7|     @title = title
     8|     @author = author
     9|     @price = price
    10|   end
    11| end
=>#0	Book#initialize(title="The Hobbit", author="J.R.R. Tolkien", price=15.0) at library.rb:7
  #1	[C] Class#new at library.rb:37
  # and 1 frames (use `bt' command for all frames)
(rdbg) up    # command
# No sourcefile available for library.rb
=>#1	[C] Class#new at library.rb:37
(rdbg) up    # command
=>  37| book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)

We need to call it twice as we skipped over one step, thanks to the next command: the call to the parent class of the Book class: Class itself (and the new method).

Using a Map

When we start to use up , down, next, and step, it's handy to know two more commands:

list: to show where we are in the code
bt (or backtrace): to show the trace of the steps we have followed

For example, when we are at line 37, the bt command displays the following:

(rdbg) bt    # backtrace command
  #0	Book#initialize(title="The Hobbit", author="J.R.R. Tolkien", price=15.0) at library.rb:7
  #1	[C] Class#new at library.rb:37
=>#2	<main> at library.rb:37

Calling down twice brings us to step #0. We can also pass an additional integer to both up and down to move through as many steps as we want to in one go.

Knowing What's Available

A very practical command to know is ls. It will list the variables and methods available to you at your current point in the stack.

For example, on line 37, we see the following:

(rdbg) ls    # outline command
Object.methods: inspect  to_s
locals: book1  book2  book3  store

Using `finish`

We can go to our next breakpoint using continue. However, the finish or fin command will also bring us to the next breakpoint, or to the end of our program.

You can exit more quickly with Ctrl-D or quit.

Adding Breakpoints On the Fly

A more advanced practice is to add breakpoints on the fly while running the debugger.

We have different ways to do that. Let's start with some more simple ways to add a breakpoint:

To a specific line — break <line number> — in the current file.
To the start of a specific method in a specific class: break ClassName#method_name.

(rdbg) break 38    # command
#0  BP - Line  /mnt/data/Code/clients/AppSignal/debug/library.rb:38 (line)
(rdbg) break BookStore#find_by_title    # command
#1  BP - Method  BookStore#find_by_title at library.rb:27

Called on its own, the break command will list the existing breakpoints (the ones added through the debug console):

(rdbg) break    # command
#0  BP - Line  /mnt/data/Code/clients/AppSignal/debug/library.rb:38 (line)
#1  BP - Method  BookStore#find_by_title at library.rb:27

You can also remove breakpoints that are added this way using the del or delete command:

del will remove all breakpoints in one go (confirmation is needed).
del X deletes breakpoints numbered X in the breakpoints list.

Adding Conditions

You can also add conditions when setting a breakpoint. Imagine a method that goes wrong when the book title is "Germinal", but that goes ok if it's "Notre Dame". In this case, we can add a breakpoint on the method, but only if the book title matches.

(rdbg) break BookStore#find_by_title if: book1.title == "Germinal"    # command
#1  BP - Method  BookStore#find_by_title at library.rb:27 if: book1.title == "Germinal"

Integration with IDEs

Many of us rely on modern IDEs and text editors that have support for direct debugging. A good choice is rdbg: it integrates well with many IDEs.

Check the debug README for more details on rdbg.

Recap and Wrapping Up

In this post, we covered the following:

Installing debug
Adding breakpoints from your favorite code editor with binding.break
Looking at the value of variables and objects from a debugger session
Navigating within execution frames from the debugger console (with up, down, and next)
Listing available variables and methods at any point in the console with ls
Adding breakpoints and conditional breakpoints on the fly from the debugger console
Listing and removing breakpoints (with break, delete <number>, and delete)
Ending a debugging session with finish, continue, or quit.

The help command also provides plenty of details on the commands we have seen here and more. You can run help break (for example) to learn more about the break command and its subcommands.

In conclusion, the debug tool will greatly help you with debugging over the years.

Most debuggers use similar commands, so don't hesitate to try others out too (check out our post on pry-byebug, for example).

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Monitor the Performance of Your Ruby on Rails Application Using AppSignal

Roy Tomeij — 2024-06-12T00:00:00+00:00

In the first part of this article series, we deployed a simple Ruby on Rails application to DigitalOcean's app platform. We also hooked up a Rails app to AppSignal, seeing how simple errors are tracked and displayed in AppSignal's Errors dashboard.

In this part of the series, we'll dive into how to set up the following for your Ruby on Rails application using AppSignal:

Performance monitoring
Rails background jobs monitoring, including how to monitor simple API calls
Logging
Notification alerts

Let's get into it!

Monitoring Rails App Performance Using AppSignal

When your uptime monitor shows that your app is up, it may be tempting to think that everything is okay. But, in reality, trouble could be brewing under the hood in the form of slow processes, unoptimized database queries, and long-running service calls.

This is a very important matter when you consider that there's a correlation between slow-loading web pages and a low visitor conversion rate. In a nutshell, those slow-running processes will cost you a lot if left unchecked. But with so many moving parts to a Ruby on Rails app, the important questions are: what should you look for, and where?

That's where AppSignal comes into play. Let's look at how you can use AppSignal to keep track of your Rails app's performance.

Tracking Response Times

From the default dashboard, AppSignal provides two graphs that can give you a quick overview of how slow (or fast) your Rails app is running: the Throughput and Response time graphs.

Throughput - This basically measures how many requests per second your app is currently processing (not to be confused with how many requests per second your app can handle overall). The basic rule of thumb is: the more requests per second, the better. However, if your app server handles too many requests per second, it might be time to scale your server resources to handle this.
Response time - The average time a browser response takes in milliseconds. The more time a response takes, the worse your app is running. A good rule of thumb here is that if your Rails web app has sub-100ms response times, you can consider it fast, while 300ms+ response times can be considered slow.

Now, let's say we want to track our app's response times and throughput over a seven-day window. That's very easy to do — just go to the Graphs sub-menu under Performance located on the left-side menu, then use the time filter buttons on top of the graphs, as shown below:

With that, you can easily see if your app runs slow on some days compared to others.

While on this view, it's also important to check out the Event groups graph, located below the Response time and Throughput graphs:

This graph gives you details on how fast or slow your app is running in the controller layer and the view layer. From here, you can find out what's causing slow response times and fix the issues accordingly.

Let's now switch gears to tracking database queries using AppSignal.

Tracking Database Queries

It is generally true that you can squeeze a lot more speed from your app by optimizing response times and throughput. However, more often than not, slow, unoptimized database queries are the worst offenders when it comes to sluggish app behavior.

Let's take an example of the infamous N+1 query. In the expense tracker app introduced in part 1, the index method in the expenses controller looks like this:

# app/controller/expenses_controller.rb

class ExpensesController < ApplicationController
  ...
  def index
    @expenses = Expense.all
  end
  ...
end

This might look innocent, but if you take a look at the Issues list in your AppSignal dashboard, you'll see something interesting:

An N+1 query has been highlighted in the expenses controller's index method.

If we dig further by clicking on the issue link, we get to an Issue details screen like this:

Notice how AppSignal offers an explanation of the issue, including a link to a more detailed blog post showing you how to deal with it.

Let's move on and learn how AppSignal helps you out with background jobs.

Keeping Track of Rails Background Jobs with AppSignal

Background jobs are a common feature in many production Ruby on Rails applications, with a variety of job queue processing gems and libraries for you to choose from.

AppSignal supports tracking and monitoring various background job processing libraries, including Sidekiq, Que, Delayed Job, Resque, and others.

Let's say we want a feature where users get daily currency exchange rates in their dashboard.

For this to work, we need an API call to a service providing such rates (we'll use one that doesn't require too much upfront configuration or payment, like this one, to fetch currency rates). We also need a background job to queue the call to this service at regular intervals.

The background processing gem we'll use in the expense tracker app is GoodJob, but you're free to use whatever suits you.

Let's create a background job to fetch rates:

# app/jobs/fetch_rates.rb
...
class FetchRates < ApplicationJob
  self.queue_adapter = :good_job

  def perform
    # fetch rates
    response = HTTParty.get('https://open.er-api.com/v6/latest/USD')

    if response['result'] == "success"
      Rate.create(
        base_rate_name: response['base_code'],
        eur: response['rates']['EUR'],
        cad: response['rates']['CAD'],
        aud: response['rates']['AUD'],
        gbp: response['rates']['GBP']
        )
    end
  end
end

And then set GoodJob to run a cron for this job every few minutes (or however you see fit). AppSignal automatically detects the presence of the background service and includes a nifty filter so you can separate it from the default web dashboards:

Without the need for much tooling upfront, AppSignal will also detect API calls and keep track of any issues:

If you want to dive deeper into monitoring Rails background jobs with AppSignal, I recommend you check out the documentation. For now, let's turn our attention to uptime monitoring.

Uptime Monitoring

Another matter that concerns many Rails developers is the continuous availability of their application for end users. Any downtime might mean loss of revenue and have other negative consequences.

Even so, you wouldn't want to poll the uptime status of your app manually. You can set up uptime monitoring using AppSignal.

Setting it up is a breeze. Begin by clicking on the Uptime monitoring link in the left-side menu.

Then, when you click on the create uptime monitor button, you should get a dialog similar to the one shown below:

Complete your uptime monitor by adding the most important settings:

Name - Give your uptime monitor an appropriate name.
URL - Provide the full URL to the uptime route. Beginning with Rails 7.1, the default uptime route is https://your-app-domain/up.
Region - Select the regions where you want to check for uptime.
Notification - Choose the notification channel/s to receive alerts on.

With that done, go ahead and create the monitor!

One thing to note with the AppSignal uptime monitor is that since polling is done almost every minute or so, it's very easy to hit your plan's limits. But there's a very good workaround to this which you can read more about in their docs.

Before moving on to how AppSignal can help with logging, there's another important feature to complete the uptime monitoring layer: free public status pages.

Click on the creating a public status page link as shown below:

Then click on the New status page button, bringing you to this:

Go ahead and fill in all the required information. Note that if you use a custom domain, you'll need to add a CNAME directive pointing to cname.appsignal-status.com in your custom domain's DNS settings.

Logging with AppSignal

Logging is a relatively new feature in AppSignal, but one that is timely and welcome. Using AppSignal, you don't have to run application monitoring and logging as separate services.

To get started with logging, make sure you are running the latest version of the AppSignal gem. If you have an older version of the gem, update it with:

bundle update appsignal

Then create a new initializer and edit it like so:

# config/initializers/appsignal_logging.rb

appsignal_logger = Appsignal::Logger.new("rails")
Rails.logger.broadcast_to(appsignal_logger)

Now you can access your app's logs like so:

Notice that you also get a nice filter to filter log records by severity, as well as access to live logs.

You can go through AppSignal's logging for Ruby documentation here.

Other Notable Features

There are a couple of other notable features you can use in AppSignal: anomaly detection and custom metrics.

Anomaly Detection

Let's say you are concerned with your app running out of memory. You can easily set up a trigger that sends an email notification if your app has a notable drop in available memory.

Go to the Anomaly detection link and create a new trigger, then configure it:

Metric - In my case, I am interested in memory usage, but you could choose any other metric that is specific to your use case.
Trigger details - Here, you'll configure the details relevant to your trigger. In my case, the trigger will fire if the host memory goes below 200MB.

Custom Metrics

AppSignal also gives you the tools to capture and visualize custom metrics. This feature alone warrants an entire article. You can read all about it in How to Monitor Custom Metrics with AppSignal and check out AppSignal's docs too.

Wrapping Up

In part one of this article series, we set up a Rails app hosted on DigitalOcean and monitored it for errors using AppSignal.

In this second and final part, we looked at some of AppSignal's great features for your Rails app, including performance monitoring, uptime monitoring, logging, and more.

There's so much more to AppSignal than could effectively be covered by this article series. I highly encourage you to check it out for your Rails app.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Use Tailwind CSS for Your Ruby On Rails Project

Roy Tomeij — 2024-06-05T00:00:00+00:00

It's hard to overstate the importance of Cascading Style Sheets (CSS) for all websites. Since the first CSS standards were published in late 1996, we have come quite far regarding features and ecosystems.

Several frameworks have appeared and proved popular, one of the most recent being Tailwind CSS.

In this post, we'll first examine Tailwind's utility-first approach before diving into how to use it in a Ruby on Rails application. You will see how Tailwind helps you to build excellent websites without the need for custom CSS and long debugging sessions.

Let's get started!

Tailwind CSS: A Utility-First Approach

Most CSS frameworks (Foundation, Bootstrap, or Bulma, for example) provide ready-to-use components such as buttons and form fields, so you can quickly assemble blocks to shape an interface.

Typically, adding a button with Bootstrap looks like this:

<button class="btn btn-primary">My Button</button>

In this example, a simple button is defined and styled by applying the btn and btn-primary classes. btn-primary sets the right color for our use case. Yet, that interface can't fit our needs, so we add a custom CSS stylesheet to customize every component:

<button class="btn btn-primary admin-button">My Button</button>

Tailwind is a "utility-first" concept. Instead of providing ready-to-use components such as buttons, it has low-level utility classes that you can compose to build custom designs. As such, it encourages a more functional approach to styling, where you apply pre-defined classes directly in your HTML. It aims to minimize the need for custom CSS and promotes design consistency through the constraints of the utility classes.

"Utility-first" means that Tailwind provides atomic, single-purpose classes you can combine to construct complex designs.

Let's have a look at some code to compare Tailwind and Bootstrap. First, here is how Tailwind lets us style a simple button:

<button class="bg-blue-500 hover:bg-blue-600 text-white py-2 px-4 rounded">
  My Button
</button>

There are a series of button element classes to configure:

Background color bg-blue-500: While 'blue' is a pre-picked color, we can set the color shade with the number. The higher the number, the darker the color.
Background color on hover: hover:bg-blue-600.
Text color text-white: No need for a number here, as it's white; there is always a default shade if you don't specify a number, such as with text-red.
Vertical padding py-2: 'p' is padding, 'y' is for the vertical axis, '2' is the spacing value, not in pixels but a scale defined in Tailwind.
Horizontal padding px-4: Same as above, with 'x' for the horizontal axis.
Rounding corners: rounded.

This looks more verbose than the Bootstrap example, but by only adding classes, we can adjust each part of the style. We don't need to create a custom CSS class.

You might not be happy with these colors, but the good news is that you can add custom colors. We will cover that later.

A Word on Scales

CSS is mighty when it comes to spacing (such as margins and padding), and you can work with pixels and rems (root-em, a size relative to the size of the root element). This tends to be difficult, though. Tailwind comes with its own spacing scale that hides complexity while also helping with proportionality.

By default, Tailwind offers values between 0 and 96, with each step proportional to the others. For example, the value 16 has twice as much spacing as 8. Thanks to this, we don't have to do the math to work with rems or pixels. We can define our preferred values and reuse them throughout our design.

Setting Up Tailwind in a Ruby on Rails Environment

Ruby on Rails 7.x directly supports Tailwind in its application generator.

$> cd ~/workspace/ && mkdir tailwind-tryout && cd tailwind-tryout
$> rails new -d sqlite3 -c tailwind -T .

We'll skip the test configuration (-T) to avoid adding unnecessary complexity to this article.

Tailwind has a neat feature that generates the CSS file your application needs. Other frameworks require you to include a whole CSS file defining a framework (even the pieces you don't use). In contrast, Tailwind will scan your project and generate a CSS file that contains only the classes your project needs.

You do need to run a little utility to make that happen. In development mode, you can run a watcher daemon that will keep things up to date as you work: bin/rails tailwindcss:watch.

You can also add the watcher daemon to your Procfile, then use foreman or overmind to start the web and css processes:

web: bin/rails server
css: bin/rails tailwindcss:watch

Let's now use it within a simple landing page:

bin/rails generate controller Landing index

We can then head to http://localhost:3000/landing/index.

Dissecting Our Landing Page

Every landing page needs a title. The generator works since we configured our application to use Tailwind as its CSS framework.

# app/views/landing/index.html.erb
<div>
  <h1 class="font-bold text-4xl">Landing#index</h1>
  <p>Find me in app/views/landing/index.html.erb</p>
</div>

We find something that looks like standard HTML here. We have only two classes for the h1 tag:

font-bold: to control the font weight.
text-4xl: to control the font size.

If we change text-4xl to text-xl and reload the page, the new style will be automatically applied. Looking at the terminal where Foreman is running, you will see that Tailwind has generated a stylesheet in the background again. That's how simple it is to integrate Tailwind into a Ruby on Rails application (this relies on the tailwindcss-rails gem).

Configuring Tailwind for Ruby on Rails

You can edit the config/tailwind.config.js file to adjust Tailwind's settings (e.g., to add additional colors, specify a font to use, adjust spacing, etc).

For example, we could add a "copper" color to our backgrounds and text:

module.exports = {
  content: ["./src/**/*.{html,js}"],
  theme: {
    colors: {
      copper: {
        100: "#FAD9C1",
        200: "#F6C8A4",
        300: "#F2B786",
        400: "#EEA669",
        500: "#E9944C",
        600: "#D17F3E",
        700: "#B96A31",
        800: "#A15524",
        900: "#8A4018",
        dark: "#8A4018",
      },
    },
    fontFamily: {
      serif: ["Times", "serif"],
    },
    extend: {
      spacing: {
        "8xl": "108rem",
      },
    },
  },
};

Note that the shades are helpful but can instead be named. If we only need three shades, for example, we can use 'light', 'medium', and 'dark' instead of numbers in our views.

We can then use the shades in our title:

<h1 class="font-bold text-4xl text-copper-200">Landing#index</h1>
<h2 class="font-bold text-xl text-copper-dark">Subtitle</h2>

You can find details about this in the tailwindcss-rails gem's README and also the Tailwind CSS documentation.

Asset Pipeline

We have seen how bin/rails tailwindcss:watch keeps our stylesheets updated in local development mode. If we need to build the stylesheets just once, we can use bin/rails tailwindcss:build instead.

For production use, you can rely on bin/rails assets:precompile to directly call bin/rails tailwindcss:build.

Learn more about the asset pipeline for Ruby on Rails applications.

Tailwind for Rails in Action

Let's check out a couple of practical uses of Tailwind in some views: a form and a responsive navigation bar.

A Simple Form

Using the Ruby on Rails generator, we create a user resource:

bin/rails g resource user email:string password:string
bin/rails db:migrate

We can then alter the users_controller.rb file and create a view for the form.

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def def new
    @user = User.new
  end
end

# app/views/users/new.html.erb
<div>
  <h1 class="font-bold text-4xl text-blue-500">Users#new</h1>

  <%= form_with model: @user, local: true do |form| %>
    <div class="mb-6">
      <%= form.label :email, class: "block mb-2 text-sm font-medium text-blue-900" %>
      <%= form.text_field :email, class: "bg-gray-50 border border-gray-300 text-gray-900 text-sm rounded-lg focus:ring-blue-500 focus:border-blue-500 block w-full p-2.5" %>
    </div>
    <div class="mb-6">
      <%= form.label :password, class: "block mb-2 text-sm font-medium text-gray-900" %>
      <%= form.password_field :password, class: "bg-gray-50 border border-gray-300 text-gray-900 text-sm rounded-lg focus:ring-blue-500 focus:border-blue-500 block w-full p-2.5" %>
    </div>
    <button type="submit" class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:outline-none focus:ring-blue-300 font-medium rounded-lg text-sm w-full sm:w-auto px-5 py-2.5 text-center">Submit</button>
  <% end %>
</div>

We style each piece individually, adjusting the text color, background color, borders, padding, margins, etc. There is nothing beyond standard Tailwind here, yet we customize the form to fit our needs.

A Responsive Navigation Bar

We can add conditional breakpoints based on a browser's minimum width using any utility class in Tailwind. For example, the following title will change color depending on the window size:

<h2
  class="text-base font-semibold text-gray-900 sm:text-teal-800 lg:text-purple-500"
>
  Additional information
</h2>

By default, the color is a dark shade of gray. When a browser window's width is between 640px and 1024px, it's a shade of teal. If a window's width is above 1024px, it's a shade of purple.

As Tailwind can also handle columns, here is an example to showcase how an element's column width can change based on window size:

<div class="sm:col-span-2 md:col-span-3">
  <label for="region" class="block text-sm font-medium leading-6 text-gray-900"
    >State</label
  >
</div>

The label "State" spans two or three columns in this case.

Here, using Tailwind's grid layout utilities, we define a grid that is:

One column wide by default (grid-cols-1)
Six columns wide above 640px width
Eight columns wide above 768px width

<div class="mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6 md:grid-cols-8">
</div

Breakpoints and their widths:

sm: 640px
md: 768px
lg: 1024px
xl: 1280px
2xl: 1536px

As we've seen, Tailwind simplifies page design and the styling of components.

Tailwind vs. Other Frameworks

Now that we understand how Tailwind can be used, let's review its key differences to other frameworks:

Utility-based: We compose the style of each element using specific CSS classes, each focusing on different parts of the style.
Get what we need: We only get the parts we need to ship our website, making for faster load times; that optimizes build time.
Extensible: We can extend or customize TailwindCSS' defaults through a simple configuration file.
Easy shading of colors: There's no need to figure out how to make lighter or darker shades of a color to handle hover situations, for example.
Simple spacing: The hidden and proportional scales simplify spacing.
Less custom CSS: Since we only assemble classes to style elements, we rely less on custom CSS and can share styles (including complete themes) using HTML files and snippets.
Ruby on Rails friendly: Thanks to the Tailwind gem, everything is integrated into the layouts and the assets pipeline.

Wrapping Up

As we've seen, Tailwind's utility-first approach is a great fit for Ruby on Rails. We don't need to spend time adjusting Tailwind to fit our needs by adding complex custom configurations or additional custom CSS. As we conceive our views and partials, we can use Tailwind utility classes to shape and style them.

If you want to learn more, you can access many ready-to-use templates and components thanks to Tailwind's vibrant community, and products such as TailwindUI (from Tailwind's creators).

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Getting Started: Your Ruby On Rails App Hosted On DigitalOcean With AppSignal

Roy Tomeij — 2024-05-29T00:00:00+00:00

Imagine this: you’ve just finished working on your brand new Rails app and have deployed it to a cloud provider like DigitalOcean. Like any developer, you’re very proud of your work but you still have lots of questions, like:

How well your new app will handle traffic
Whether the optimizations you put in place will actually work, etc.

Your goal is to provide the best user experience. You want to be notified whenever errors or other important events occur so you can take care of them fast.

It would be great to have a setup that automatically monitors your application. Enter AppSignal! In this article, the first part of a two-part series, we'll set AppSignal up so that you can effectively monitor your Rails app hosted on DigitalOcean.

Prerequisites

To follow along, make sure you have:

A local installation of Ruby (this tutorial uses version 3.3.0).
A local PostgreSQL installation (you can use a Docker version or a locally installed version).
A DigitalOcean account to deploy the application.
An AppSignal account (a free 30-day trial is available).

An Introduction To Our Ruby On Rails App

For this tutorial, we'll be using a simple Rails 7 expense tracker app. Users will be able to sign up and create entries for their personal expenses, to track their expenses over time.

We'll deploy this app to DigitalOcean, then configure AppSignal's monitoring solution to keep track of what is going on under the hood.

You can grab the source code or fork the app from here to follow along.

Deploying Your Rails App To DigitalOcean

We'll use DigitalOcean's app platform to deploy our application. The steps below assume that you've already forked the expense tracker app described above and that you have a DigitalOcean account ready to go.

After logging in, we'll create an app and get it up and running. Since we want to control some parts of this process, though, the first step is to create a database for the app.

Creating the App Database

Click on the Databases link on the left-hand menu, then click on the Create Database link to create a new PostgreSQL database:

After completing the database settings in the next screen, take note of the database connection string. You'll use it as an environment variable when creating the app in the next step:

At this point, you'll likely get a warning that your database is open to all incoming connections. Follow the accompanying link to take care of this security warning.

Creating and Deploying the App

Finally, create the app by first clicking on the Apps link on the left side menu:

Then connect the app's code repository resource as shown below:

Continue to the environment variables screen and insert the copied database URL string. Also, add the RAILS_MASTER_KEY as an environment variable since it will be used in the deployment process.

Then, with these environment variables in place, click on the Next button to deploy the app:

With the app successfully deployed, we'll now set up monitoring using AppSignal.

Setting up AppSignal to Monitor Your Rails App

First, log in to your AppSignal account and choose 'Ruby & Rails'.

Now add the AppSignal gem to the Rails app. This nifty gem will collect errors, exceptions, and relevant performance data, and port it over to AppSignal for analysis.

Open up the app's Gemfile and add the gem:

# Gemfile
gem "appsignal"

Then run bundle install.

Finally, you'll need to run the installation script that comes with your account-specific API key attached. When you run this install script, you should get something similar to the below:

 bundle exec appsignal install <redacted>

#######################################
## Starting AppSignal Installer      ##
## --------------------------------- ##
## Need help?  support@appsignal.com ##
## Docs?       docs.appsignal.com    ##
#######################################

Validating API key...
  API key valid!

Installing for Ruby on Rails

  Your app's name is: 'ExpenseTracker'
  Do you want to change how this is displayed in AppSignal? (y/n): n
How do you want to configure AppSignal?
  (1) a config file
  (2) environment variables
  Choose (1/2): 1

Writing config file...
  Config file written to config/appsignal.yml

#####################################
## AppSignal installation complete ##
#####################################

  Sending example data to AppSignal...
  Example data sent!

There are a couple of things to note when you run the script:

You'll get the option to change your app's name as it will appear on AppSignal's dashboard or leave the default option.
You'll also get the option to choose how you want AppSignal configured for your app. In my case, I went with the config file option, creating a config file in config/appsignal.yml, with the below content:

# config/appsignal.yml

default: &defaults
  push_api_key: "<%= ENV['APPSIGNAL_PUSH_API_KEY'] %>"

  name: "ExpenseTracker"

development:
  <<: *defaults
  active: true

production:
  <<: *defaults
  active: true

Here, we define:

A push_api_key which connects the app to AppSignal.
The app's name as it will appear on AppSignal.
The environments in which AppSignal will monitor the app (in this case, both development and production environments).

By the way, if you'd like to turn off AppSignal monitoring on the development environment, just set the active flag to false, like so:

...
development:
  <<: *defaults
  active: false
...

Tip: Both the config file and environment variable options do the same thing. They define how AppSignal will connect to your app, the app name, and which environment will be monitored.

Assuming everything goes to plan, you should see a screen showing that AppSignal is receiving data from your app!

An Introduction to AppSignal's Dashboard for Rails

Now that everything is set up correctly and AppSignal is receiving data from your app, you can access the default app monitoring dashboard view as shown below:

Tip: If you set up monitoring for both development and production environments, you can easily switch between them from the selection shown by the arrow.

From the default view, you have access to the following default charts:

Error rate - This will show the rate of errors occurring in your app per unit of time.
Throughput - Here, you'll get a snapshot of the throughput your app is able to handle in terms of requests per minute.
Response time - This will show your app's response times.
Latest open errors - This section will list the latest errors that will have happened within your app.
Latest open performance measurements - This section lists the latest performance measurements, including method calls, API requests, etc.

Next, we'll see how to set up proper error tracking for a Rails application using AppSignal.

Monitoring Errors with AppSignal

There are more than ten error types that could affect a running Ruby on Rails app. Obviously, some are more common than others. In this section, we'll simulate a few of these errors and see how AppSignal handles them. Let's start with a simple example first.

Since the expense tracker app is using Devise for authentication, let's add a check for whether a user is signed in. Instead of using the recommended user_signed_in? method, let's use the erroneous user_logged_in? method, which should trigger an ActionView::Template::Error:

<!-- app/views/shared/_navbar.html.erb -->
...
<% if user_logged_in? %>
    <%= link_to 'Logout', destroy_user_session_path %>
<% else %>
    <%= link_to new_user_session_path do %>
    Login
    <% end %>
<% end %>
...

Deploy this change, reload the production app, then go into the production environment dashboard in AppSignal and watch how this error shows up:

AppSignal makes it a breeze to get more details on any errors that happen in an application. For example, we can get details about the ActionView template error by clicking on it from the 'Latest Open Errors' dashboard panel.

You can clearly see that the error is caused by an undefined method which points us in the right direction to fix it.

AppSignal's Errors dashboard also gives you extra information through the Logbook and Settings panels:

Logbook - Here, you or your team members can add comments to an error being tracked by AppSignal. The Logbook panel also shows a chronological breakdown of any actions taken regarding the error in question.
Settings - From this panel, you can assign an error to another team member, change alert settings (we'll check these out in detail in the second part of this tutorial), and set error severity.

And that's it for this part of the series!

Wrapping Up

In this tutorial, we deployed a simple Rails application to DigitalOcean's app platform and hooked it up to AppSignal's application monitoring platform. We also went through how errors are monitored and displayed in AppSignal's Errors dashboard.

Obviously, this is just scratching the surface of what's possible with AppSignal. In the second part of this series, we'll dive deeper into performance measurements, anomaly detection, uptime monitoring, and logging.

In the meantime, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Five Things to Avoid in Ruby

Roy Tomeij — 2024-05-22T00:00:00+00:00

As a contract software developer, I am exposed to oodles of Ruby code. Some code is readable, some obfuscated. Some code eschews whitespace, as if carriage returns were a scarce natural resource, while other code resembles a living room fashioned by Vincent Van Duysen. Code, like the people who author it, varies.

Yet, it's ideal to minimize variation. Time and effort are best spent on novel problems.

In this post, we'll explore five faux pas often seen in Ruby code and learn how to turn these idiosyncrasies into idioms.

But first, how can we minimize variance in Ruby?

Minimizing Variance in Ruby with Rubocop and Idioms

Ruby developers can gain efficiency with Rubocop, which unifies style within a single project or across many projects. Rails is a boon to uniformity, too, as it favors convention over configuration. Indeed, disparate Rails codebases are identical in places and, overall, quite similar.

Beyond tools and convention, variance is also minimized by idioms, or those "structural forms peculiar to a language". For example, Ruby syntax is a collection of explicit idioms enforced by the interpreter. Reopening a class is a Ruby idiom, too.

But not all Ruby idioms are dicta. Most Ruby idioms are best practices, shorthand, and common usages Ruby developers share informally and in practice.

Learning idioms in Ruby is like learning idioms in any second spoken (or programming) language. It takes time and practice. Yet the more adept you are at recognizing and proffering Ruby's idioms, the better your code will be.

What to Avoid in Ruby

Now, let's move on to looking at five things to avoid in Ruby:

Verbosity
Long expressions to detect nil
Overuse of self
Collecting results in temporary variables
Sorting and filtering in memory

Verbosity

If you've delved into Ruby, you can appreciate how expressive, compact, fun, and flexible it is. Any given problem can be solved in a number of ways. For example, if/unless, case, and the ternary operator ?: all express decisions — which one you should apply depends on the problem.

However, per Ruby idiom, some if statements are better than others. For instance, the following blocks of code achieve the same result, but one is idiomatic to Ruby:

# Verbose approach

actor = nil

if response == 1
  actor = "Groucho"
elsif response == 2
  actor = "Chico"
elsif response == 3
  actor = "Harpo"
else
  actor = "Zeppo"
end

# Idiomatic approach

actor =
  if response == 1
    "Groucho"
  elsif response == 2
    "Chico"
  elsif response == 3
    "Harpo"
  else
    "Zeppo"
  end

Almost every Ruby statement and expression yields a value, including if, which returns a final code statement value and a matching condition. The version of the if statement in the second code block above leverages this behavior. If response is 2, actor is set to Chico. Assigning the result of an if statement is idiomatic to Ruby. (The same construct can be applied to case, unless, begin/end, and others.)

Another Ruby idiom is present: you need not predefine a variable used within an if statement (or while, for, and others). So, the latter code removes the line actor = nil. In Ruby, unlike other languages, the body of an if statement is not considered a separate scope.

Long Expressions to Detect `nil`

nil represents "nothing" in Ruby. It's a legitimate value and is its own class (NilClass) with methods. Like other classes, if you call a method that's not defined on nil, Ruby throws an exception akin to undefined method 'xxx' for nil:NilClass.

To avoid this exception, you must first test a variable to determine if it's nil. For example, code such as this is common in Rails applications:

if user && user.plan && user.plan.name == 'Standard'
  # ... some code for the Standard plan
end

The trouble is that such a long chain of assertions is unwieldy. Imagine having to repeat the condition user && user.plan && ... every time you have to reference a user's plan name.

Instead, use Ruby's safe navigation operator, &. It is shorthand for the logic "If a method is called on nil, return nil; otherwise, call the method per normal." Using &. reduces the code above to the much more readable:

if user&.plan&.name == 'Standard'
  // ... some code
end

If user is nil and plan is nil, the expression user&.plan&.name is nil.

The example above assumes user and plan represent a custom structure or Rails model of some kind. You can also use the safe navigation operator if a value represents an Array or Hash.

For example, assume a_list_of_values is an Array:

a_list_of_values[index]

If the variable a_list_of_values is nil, an exception is thrown. If you expectantly try a_list_of_values&.[index], a syntax error occurs. Instead, use &. with the Array#at method.

a_list_of_values&.at(index)

Now, if a_list_of_values is nil, the result of the expression is nil.

Overuse of `self`

Ruby uses self in three substantive ways:

To define class methods
To refer to the current object
To differentiate between a local variable and a method if both have the same name

Here's an example class demonstrating all three usages.

class Rectangle
  def self.area(length, width)
    new(length, width).area
  end

  def self.volume(length, width, height)
    area(length, width) * height
  end

  def initialize(length, width, height = nil)
    self.length = length
    self.width  = width
    self.height = height
  end

  def area
    length * width
  end

  def volume
    area = 100
    self.area * height
  end

  private

  attr_reader :length, :width, :height
end

def self.area is an example of the first purpose for self, defining a class method. Given this class, the Ruby code puts Rectangle.area(10, 5) produces 50.

The code self.length = length demonstrates the second application of self: the instance variable length is set to the value of the argument length. (The attr_reader statement at the bottom defines the instance variable and provides a getter method.) Here, the statement self.length = length is functionally the same as @length = length.

The third use of self is shown in the volume method. What does puts Rectangle.volume(10, 5, 2) emit? The answer is 100. The line area = 100 sets a method-scoped, local variable named area. However, the line self.area refers to the method area. Hence, the answer is 10 * 5 * 2 = 100.

Consider one more example. What is the output if the volume method is written like this?:

 def volume
   height = 10
   self.area * height
end

The answer is 500 because both uses of height refer to the local variable, not the instance variable.

Inexperienced Rails developers make unnecessary use of self, as in:

# Class User
# first_name: String
# last_name: String
# ...

class User < ApplicationRecord
  ...
  def full_name
    "#{self.first_name} #{self.last_name}"
  end
end

Technically, this code is correct, yet using self to refer to model attributes is unnecessary. If you combine Rubocop with your development environment, Rubocop flags this issue for you to correct.

Collecting Results in Temporary Variables

A common code chore is processing lists of records. You might eliminate records due to certain criteria, map one set of values to another, or separate one set of records into multiple categories. A typical solution is to iterate over the list and accumulate a result.

For instance, consider this a solution to find all even numbers from a list of integers:

def even_numbers(list)
  even_numbers = []

  list.each do |number|
    even_numbers << number if number&.even?
  end

  return even_numbers
end

The code creates an empty list to aggregate results and then iterates over the list, ignoring nil items, and accumulating the even values. Finally, it returns the list to the caller. This code serves its purpose, but it isn't idiomatic.

A better approach is to use Ruby's Enumerable methods.

def even_numbers(list)
  list.select(&:even?) # shorthand for `.select { |v| v.even? }`
end

select is one of many Enumerable methods. Each Enumerable method iterates over a list and collects results based on a condition. Specifically, select iterates over a list and accumulates all items where a condition yields a truthy value. In the example above, even? returns true if the item is an even number. select returns a new list, leaving the original list unchanged. A variant named select! performs the same purpose, but alters (mutates) the original list.

The Enumerable methods include reject and map (also known as collect). reject collects all items from a list where a condition yields a falsey value. map returns a new list where each item in the original list is transformed by an expression.

Here's one more example Enumerable method in action. First, some non-idiomatic code:

def transform(hash)
  new_hash = {}

  hash.each_pair do |key, value|
    new_hash[key] = value ? value * 2 : nil
  end

  return new_hash
end

And now a more idiomatic approach:

def transform(hash)
  hash.transform_values { |value| value&.*(2) }
end

transform_values is another Enumerable method available for Hash. It returns a new hash with the same keys, but each associated value is transformed. Remember, even integers are objects in Ruby and have methods. value&.*(2) returns nil if value is nil, else value * 2.

Sorting and Filtering in Memory

Here's one last faux pas — this one is specific to Rails and ActiveRecord. Let's examine this code:

# class Student
# name: String
# gpa: Float
# ...
#

class Student < ApplicationRecord
  ...

  def self.top_students
    Student
      .all
      .sort_by { |score| student.gpa }
      .select { |score| student.gpa >= 90.0 }
      .reverse
  end
end

Calling Student.top_students returns all students with a GPA greater than or equal to 90.0 in ranked order, from highest to lowest. Technically, this code is correct, but it isn't very efficient in space or time because:

It must load all records from the students table into memory.
It first sorts all records and then filters based on GPA, performing unnecessary work.
It reverses the order of the list in memory.

Sorting and filtering are best performed in the database, if possible, using ActiveRecord's tools.

def self.top_students
  Student.where(gpa: 90.0..).order(gpa: :desc)
end

The shorthand 90.0.. in the where clause is a Ruby Range expressing a value between 90.0 and Float::INFINITY. If gpa is indexed in the students table, this query is likely very fast and loads only the matching records. If the student records are being fetched for display, more efficiency (in memory) can be gained via pagination (albeit at the possible expense of more queries to fetch the batches).

Wrapping Up and Next Steps

In this post, we've covered five key things to avoid in Ruby and the idioms to use instead.

I highly recommend reading the documentation for Ruby's core classes and modules, including Array, Hash, and Enumerable. The documents are a treasure trove of methods and techniques, and chances are a method exists to solve your problem at hand. Turn knowledge into practice by writing small code samples to learn how each method works.

Add Rubocop into your workflow, even your text editor. Rubocop keeps your code looking nice but can also flag idiosyncratic code. Using Rubocop is one of the best ways to learn to code the Ruby way.

Finally, read other developers' code, especially open-source Ruby projects. To peruse the code of any gem, run bundle open <gem>, where <gem> is the name of a library. If you've included a debugger in your Gemfile, you can even set breakpoints in any gem and step through the code.

Go forth and code!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

P.P.S. Use AppSignal for Ruby for deeper debugging insights.

Creating Forms in Ruby on Rails with Simple Form

Roy Tomeij — 2024-05-15T00:00:00+00:00

Ruby on Rails has changed how we build web applications. Early on, the framework came with some great features to help you get started and build robust applications.

However, it can still be tricky to build and handle forms. Simple Form is a great option. Let's examine what Simple Form is, why we might need it, and some real use cases.

Forms and Ruby on Rails

Ruby on Rails really simplifies building applications. Yet, it also requires your constant attention to keep your codebase streamlined and coherent. One strategy is to avoid writing code by using abstractions, for example. Code that isn't in your actual application is code you don't have to manage. That's why we like abstractions so much, right? Simple Form is just such an abstraction that simplifies building forms for web pages.

Forms are generally complex to build, even using Ruby on Rails. Each field requires attention to define the right type and attach it to the correct attribute. Simple Form eases that pain, as you don't need to find the exact type required for each field.

Let's take a form that gets a user's details, with name, username, and email fields. Consider that we have the User model with similar related attributes. With Simple Form, the form that fills in attributes looks like the following:

<%= simple_form_for @user do |f| %>
  <%= f.input :name, required: false %>
  <%= f.input :username %>
  <%= f.input :email %>
  <%= f.button :submit %>
<% end %>

Note that we are not specifying the types of each field. Simple Form will pick the correct input type for each field based on the column's type.

This saves us time and keeps the code readable and easier to maintain.

Installing Simple Form

To install Simple Form in your project, add the simple_form gem to the Gemfile. Once you have run bundle install, you can use the included generator to set it up correctly for your application.

rails generate simple_form:install

A note on Bootstrap and Zurb Foundation: Both Bootstrap and Zurb Foundation 5 are supported within Simple Form. So, if you are using one of them, you can use the --bootstrap or --foundation parameter with the generator to include additional configurations during the installation. In the case of Bootstrap, this would look like the following:

rails generate simple_form:install --bootstrap

Basic Usage of Simple Form

As we pointed out earlier, Simple Form will generate a complete form mainly based on the data related to your objects. Yet, it's also important to understand that Simple Form will not only generate the appropriate field for each attribute but also provide labels and display error hints above the input fields themselves!

Let's consider a fresh new Ruby on Rails 7.x application. We'll generate a User model and build a form for it using Simple Form.

rails g model User username:string password:string email:string remember_me:boolean
# and follow with running the migration of course
rails db:migrate

Now we can generate a controller with the new, create, and index actions. The aim is to have something we can play with in the form.

rails g controller users

# app/users_controller.rb
class UsersController < ApplicationController
  def new
    render :new, locals: { user: User.new }
  end

  def create
    user = User.create(user_params)
    redirect_to users_path
  end

  def index
    render :index, locals: { users: User.all }
  end

  private

  def user_params
    params.require(:user).permit(:username, :password, :email)
  end
end

# config/routes.rb
Rails.application.routes.draw do
  resources :users, only: [:new, :create, :index]
end

These actions require relevant views. Here is a form that handles a user signing up or being added:

<!-- app/views/users/new.html.erb -->
<%= simple_form_for user do |f| %>
  <%= f.input :username, label: 'Your username please', error: 'Username is mandatory, please specify one' %>
  <%= f.input :password, hint: 'No special characters.' %>
  <%= f.input :email, placeholder: 'user@domain.com' %>
  <%= f.input :remember_me, inline_label: 'Yes, remember me' %>
  <%= f.button :submit %>
<% end %>

In this case, we specify a custom label and error for the username input. If no such customization is passed, it will do its best to guess what the label needs to be based on the attribute's name.

We have set a custom hint for the password field and a custom placeholder for the email field. Notice, also, the ability to specify an "inline label" instead of one located above the field (the usual default in forms).

It's also possible to disable labels, hints, and errors by passing the false value for those attributes: <%= f.input :password_confirmation, label: false %>.

We need to complement this with one more view to make it usable.

<!-- app/views/users/index.html.erb -->
<ul>
<% users.each do |user| %>
  <li><%= user.username %> (<%= user.email %>)</li>
<% end %>
</ul>
<p>
<%= link_to 'New User', new_user_path %>
</p>

You can then start the application using rails s. Head to localhost:3000/users/new to see and use the form. Let's focus on how the form differs from a classic Ruby on Rails form.

A note on validations: Simple Form can work with mandatory fields out of the box. For example, if we were to add the following presence validation in our User model:

class User < ApplicationRecord
  validates :username, presence:true

  # more code
end

This will be used by Simple Form to add a little '*' next to the username field, specifying that it's mandatory. This doesn't manage the errors automatically, so you still have to handle that in the controller action. Return to the appropriate field that has errors and act on them.

About Column Types and Form Fields

As mentioned earlier, we haven't specified a type for each field. The Simple Form README has a complete list of all available input types and the defaults for each column type. Here are the most used ones:

Column Type	Generated HTML element	Comment
`string`	`input[type=text]`
`boolean`	`input[type=checkbox]`
(passwords) `string`	`input[type=password]`	Any column with a name containing 'password'
`text`	`input[type=textarea]`
`integer`, `float`	`input[type=number]`
`datetime`	`datetime select`
`time`	`time select`
`country`	`select`

Booleans

As we can see in the above table, boolean attributes will be represented, by default, as checkboxes. In many cases, that's what we'll want. But if not, there is a way to customize this in Simple Form with the as attribute, which allows you to specify if you will show radio buttons or a dropdown instead.

Here is how to specify radio buttons instead:

<%= f.input :remember_me, input_html: { value: '1' }, as: :radio_buttons %>

This will generate the following HTML:

<div class="input radio_buttons optional user_remember_me">
  <label class="radio_buttons optional">Remember me</label>
  <input type="hidden" name="user[remember_me]" value="" autocomplete="off" />
  <span class="radio">
    <label for="user_remember_me_true">
      <input
        value="true"
        class="radio_buttons optional"
        type="radio"
        name="user[remember_me]"
        id="user_remember_me_true"
      />Yes
    </label>
  </span>
  <span class="radio">
    <label for="user_remember_me_false">
      <input
        value="false"
        class="radio_buttons optional"
        readonly="readonly"
        type="radio"
        name="user[remember_me]"
        id="user_remember_me_false"
      />No
    </label>
  </span>
</div>

It's impressive that such a small piece of Ruby code gets you such a complete piece of HTML!

HTML

From the previous example, we can see how Simple Form generates a lot of HTML for each field. That includes the HTML for the input field and a wrapper div around the label and input field. We can customize those by specifying a custom class and id using the input_html and wrapper_html attributes. Here is an example.

<%= simple_form_for @user do |f| %>
  <%= f.input :username, wrapper_html: { class: 'username' }, input_html { id: 'username' } %>
<% end %>

That's also a way to set attributes for the related HTML, such as maxlength or value. Let's take the password field from our user form. We can limit the size and length of the field with the maxlength attribute.

<%= f.input :password, input_html: { maxlength: 20 } %>

Custom Inputs and Additional Options

Simple Form is a Ruby library that prepares HTML nodes for you. It comes with a whole set of fields, but you can also add your own. To do so, you can create classes inheriting from Simple Form's classes. Let's consider defining a custom social network input field with a little '@' prefix in front of the actual field.

# app/inputs/social_handle_input.rb
class SocialHandleInput < SimpleForm::Inputs::Base
  def input(wrapper_options)
    merged_input_options = merge_wrapper_options(input_html_options, wrapper_options)

    "@ #{@builder.text_field(attribute_name, merged_input_options)}".html_safe
  end
end

Now we can use it in the following way:

<%= f.input :network_handle, as: :social_handle %>

Note that, if your User model doesn't have a network_handle attribute, you must add it through a migration or cheat with an attr_accessor in the model.

i18n Support

As we've mentioned, building forms is often a pain point. But things get even more painful when it comes to websites and applications that support multiple languages. Simple Form, thankfully, follows Ruby on Rails standards. You can use a simple_form key in the local files to define translations for all labels, hints, placeholders, prompts, etc.

Here is a little example:

en:
  simple_form:
    labels:
      user:
        username: 'User name'
        password: 'Password'
    hints:
      user:
        username: 'User name to sign in.'
        password: 'No special characters, please.'
    placeholders:
      user:
        username: 'Your username'
        password: '****'
    include_blanks:
      user:
        age: 'Rather not say'
    prompts:
      user:
        role: 'Select your role'

Value Objects

Sometimes, we might use custom, non-ActiveRecord classes to instantiate the main object a form relies upon. This can be done to compose data from multiple models into a synthetic one for either business logic reasons or to gather several attributes into a single object.

Whatever the reason, you can still rely on Simple Form to build forms for that kind of object. To do so, the object class needs to implement, in the best case, three methods: to_model, to_key, and persisted?.

The to_model method will point to the object itself:

def to_model
  self
end

The to_key allows us to point to the identifier attribute for the object — usually, that means an id attribute named:

def to_key
  id
end

Finally, the persisted? method is there to tell Simple Form if the object is directly persisted or not.

def persisted?
  false
end

If that method isn't present, the f.submit helper isn't usable.

There is a faster way, though — including the ActiveModel::Model module in the class:

# app/models/account.rb
class Account
  include ActiveModel::Model

  attr_accessor :id, :company_name
end

And let's use it through the User model.

# app/models/user.rb
class User < ApplicationRecord
  attr_accessor :network_handle

  def account
    @account ||= Account.new(id: 1, company_name: "Acme Inc.")
  end

  def company_name
    account.company_name
  end
end

We can then add a field for the company name within the user form:

<%= f.input :company_name %>

Of course, this attribute will not get saved in a table. However, the same concept can be applied to compose a value object with attributes pulled from several models.

Wrapping Up

In this post, we've seen that Simple Form is easy to install and integrate into your Ruby on Rails application. Simple Form not only takes care of the details of each field in your form, but also generates complex and complete HTML so your form can be styled and shaped with ease.

I encourage you to dive into Simple Form's documentation to see how powerful it can be.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Debugging in Ruby with pry-byebug

Roy Tomeij — 2024-05-08T00:00:00+00:00

For a software engineer, even the basic use of a debugger can save a lot of pain: adding breakpoints (places in the code the program will stop at and expose the current context) is very easy, and navigating from one breakpoint to another isn't difficult either.

And with just that, you can say goodbye to a program's many puts and runs. Just add one or more breakpoints and run your program. Then you're able to access not only the variables and objects you might have thought of, but also anything accessible from that point in the code.

In this article, we'll focus on pry-byebug, a gem that adds debugging and stack navigation to pry using byebug. We will see how to set up and use pry-byebug, how it integrates with Ruby programs, and a few advanced techniques.

Let's get started!

Set Up `pry-byebug` for Ruby

The setup is really simple. Just add the pry-byebug gem to your Gemfile, and then run bundle install. I'd advise you to add it to the development and test groups to debug tests as well.

Let's now cover some simple debugging using breakpoints.

Basic Debugging with Breakpoints

The whole principle of debuggers is to get rid of printf debugging by giving you direct access to a running program's stack. So, we need to add breakpoints to tell the interpreter where to stop.

Let's take the example of a simple bookstore with software that manages the titles it offers.

class Book
  attr_accessor :title, :author, :price

  def initialize(title, author, price)
    @title = title
    @author = author
    @price = price
  end
end

class BookStore
  def initialize
    @books = []
  end

  def add_book(book)
    @books << book
  end

  def remove_book(title)
    @books.delete_if { |book| book.title == title }
  end

  def find_by_title(title)
    @books.find { |book| book.title.include?(title) }
  end
end

# Sample Usage:
store = BookStore.new
book1 = Book.new("Dune", "Frank Herbert", 20.0)
book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

store.add_book(book1)
store.add_book(book2)
store.add_book(book3)

puts store.find_by_title("Hobbit").title

This program is simple enough. Of course, there is no storage, but that's not needed to demonstrate what we want to do.

The last call will probably list "The Hobbit", but that's not certain and might not be the one we want. To debug this, it's best to "jump" into the find_by_title method.

So, add the following lines to the top of the file:

require 'pry'
require 'pry-byebug'

We will also add a breakpoint to the method using binding.pry:

def find_by_title(title)
  binding.pry
  @books.find { |book| book.title.include?(title) }
end

Let's then launch the program and get to the breakpoint:

From: test-pry-byebug/app.rb:29 BookStore#find_by_title:

    27: def find_by_title(title)
    28:   binding.pry
 => 29:   @books.find { |book| book.title.include?(title) }
    30: end

[1] pry(#<BookStore>)>

The first thing you should see is that we know exactly where we are: there's no need to guess which part of the program that line in STDOUT comes from. Of course, we have only one breakpoint here, but you can easily understand how useful this might be if many breakpoints are used.

We can then check the value of the title variable. There likely isn't much to be surprised by here. The issue is with the use of find (instead of select, or something more complex) to find and return a list of books instead of just one book.

By being right in the context, you can experiment with both find and select and decide on a course of action.

Here might be a good point to reflect on the expected behavior of the program you are building and this piece of code. It might be good to clarify what the code should do through RSpec tests.

Once you are ok with that step or want to see what's happening until the next breakpoint, type continue, and the program execution will start again.

Making Small Steps from Breakpoints

With pry-byebug, you can take smaller steps from breakpoints to see what happens after each line (particularly helpful for multiple calls using a method). Once you have stepped into the program and are at a breakpoint, use the next command to execute the next line.

Let's take the case of adding a breakpoint within the add_book method just before the first three books are created in the bookstore:

def add_book(book)
  binding.pry
  @books << book
end

store = BookStore.new
binding.pry
book1 = Book.new("Dune", "Frank Herbert", 20.0)
book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

Upon execution, the program will first stop just before the book1 variable is instantiated. We might want to know the result of that call. To do so, we don't have to stop the program and add another breakpoint. We just have to type next in the pry console and hit Enter.

From: test-pry-byebug/app.rb:37 :

    32: end
    33:
    34: # Sample Usage:
    35: store = BookStore.new
    36: binding.pry
 => 37: book1 = Book.new("Dune", "Frank Herbert", 20.0)
    38: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    39: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)
    40:
[1] pry(main)> next
From: test-pry-byebug/app.rb:38 :

    33:
    34: # Sample Usage:
    35: store = BookStore.new
    36: binding.pry
    37: book1 = Book.new("Dune", "Frank Herbert", 20.0)
 => 38: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    39: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

We then have access to book1.

[1] pry(main)> book1
=> #<Book:0x00007fad37f964d8 @author="Frank Herbert", @price=20.0, @title="Dune">

This beats a lot of back-and-forth insertion and tweaking puts calls.

Even Smaller Steps

We might want to step into the initialize method (called through new) to see more details. We can do so by using step instead of next.

From: test-pry-byebug/app.rb:8 Book#initialize:

     7: def initialize(title, author, price)
 =>  8:   @title = title
     9:   @author = author
    10:   @price = price
    11: end

[1] pry(#<Book>)> title
=> "The Hobbit"

This allows us to step into a method when it's called rather than skip over it as we do with next.

Free from `puts`

By now, you should see how this approach saves time and frees us from using puts or its equivalent to debug a program. Right from the debugger shell, we can look into the values of objects and variables to figure out why a program behaves like it does. Furthermore, you can save a lot of time from the shell by adding breakpoints on the fly, thus avoiding the "quit, move binding.pry, restart" loop.

Finishing a Debugging Session

You can call continue to go to the next breakpoint or the end of a program. finish executes and finishes the current step, returning just after that step. In the previous example, we end up at the following line:

[2] pry(#<Book>)> finish

From: test-pry-byebug/app.rb:39 :

    34: # Sample Usage:
    35: store = BookStore.new
    36: binding.pry
    37: book1 = Book.new("Dune", "Frank Herbert", 20.0)
    38: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
 => 39: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)

Finally, !!! exits the debugger directly.

Advanced Use Cases of `pry-byebug` for Ruby

Now let's dive into a few more advanced use cases, including rewinding and replaying, adding breakpoints on the fly, and conditional breakpoints.

Rewinding and Replaying

Can we move back and forth between two points? Yes, totally. Given our previous case, we could move up and down from the initialize method to its call. Just use up and down commands after step to get into the method.

From: test-pry-byebug/app.rb:37 :

    34: # Sample Usage:
    35: store = BookStore.new
    36: binding.pry
 => 37: book1 = Book.new("Dune", "Frank Herbert", 20.0)
    38: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    39: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)


[1] pry(main)> step

From: test-pry-byebug/app.rb:8 Book#initialize:

     7: def initialize(title, author, price)
 =>  8:   @title = title
     9:   @author = author
    10:   @price = price
    11: end

[1] pry(#<Book>)> up

From: test-pry-byebug/app.rb:37 :

    35: store = BookStore.new
    36: binding.pry
 => 37: book1 = Book.new("Dune", "Frank Herbert", 20.0)
    38: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
    39: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)


[1] pry(main)> down

From: test-pry-byebug/app.rb:8 Book#initialize:

     7: def initialize(title, author, price)
 =>  8:   @title = title
     9:   @author = author
    10:   @price = price
    11: end

If we need details about the stack we are in, we can call the backtrace method:

[1] pry(#<Book>)> backtrace
--> #0  Book.initialize(title#String, author#String, price#Float) at test-pry-byebug/app.rb:8
    ͱ-- #1  Class.new(*args) at test-pry-byebug/app.rb:37
    #2  <main> at test-pry-byebug/app.rb:37

From: /mnt/data/Code/Pier22/courses/raw-courses/rspec/git-repo/test-pry-byebug/app.rb:8 Book#initialize:

     7: def initialize(title, author, price)
 =>  8:   @title = title
     9:   @author = author
    10:   @price = price
    11: end

The list of frames is in the stack at the top, numbered from 0 to 2. A little cursor shows us which frame we are currently in.

Add Breakpoints On the Fly

To avoid additional debugger exits, we can add breakpoints on the fly from the debugger console, using the break command.

You can add breakpoints in the current file or another file, to a specific line or the start of a specific method.

For example, let's add a breakpoint to the start of the find_by_title method of the BookStore class:

[2] pry(main)> break BookStore#find_by_title

  Breakpoint 2: BookStore#find_by_title (Enabled)

  28: def find_by_title(title)
29:   @books.find { |book| book.title.include?(title) }
30: end

To add it to line 38 of the current file:

[1] pry(main)> break 38

  Breakpoint 1: /mnt/data/Code/Pier22/courses/raw-courses/rspec/git-repo/test-pry-byebug/app.rb @ 38 (Enabled)

      35: binding.pry
    36: book1 = Book.new("Dune", "Frank Herbert", 20.0)
    37: book2 = Book.new("The Hobbit", "J.R.R. Tolkien", 15.0)
 => 38: book3 = Book.new("Hobbit's Journey", "Unknown", 10.0)
    39:
    40: store.add_book(book1)

A call to break lists all the breakpoints in place:

[4] pry(main)> break

  # Enabled At
  -------------

  1 Yes     /mnt/data/Code/Pier22/courses/raw-courses/rspec/git-repo/test-pry-byebug/app.rb @ 38
  2 Yes     BookStore#find_by_title

Note that they are numbered, so you can actually delete them using the --delete argument to break:

[4] pry(main)> break --delete 1

Conditional Breakpoints

You might want conditional breakpoints, too.

Imagine that your code only misbehaves if a user object's role attribute is set to :admin.

Well, you might want to add the following breakpoint, and a condition to trigger it so that the debugger only stops if that condition is set:

[1] pry(main)> break BookStore#add_book if book.title.match /Dune/

  Breakpoint 1: BookStore#add_book (Enabled) Condition: book.title.match /Dune/

Using continue, the debugger executes the addition of Dune without stopping, until the next call to add_book.

This is a great feature that will save you a lot of time.

Wrapping Up

By now, you should have fewer reasons to rely on printf() debugging when facing issues with your Ruby code. You now know how to:

Install pry-byebug and some of its plugins
Add breakpoints from your favorite code editor with binding.pry
Start a debugger session (starting the app)
Look at the value of variables and objects from the debugger session (simply calling the object or variable from the debugger console)
Navigate within the execution frames from the debugger console (with up, down, next, and frame)
Add breakpoints on the fly from the debugger console
Add conditional breakpoints from the debugger console (with break ClassName#method_name if var.nil?, for example)
List and remove breakpoints (with break, break --delete <number>, and break --disable-all)
Conclude your debugging session with finish, continue, or !!!.

Finally, you can also configure pry-byebug through the ~/.pryrc file to add aliases and a few custom behaviors. See the main pry-byebug README for more details.

pry-byebug is a tool that will help you a lot over the years, saving you time and headaches. It's definitely worth practicing its use so that you can confidently open it the next time you're unsure how your code is behaving.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

P.P.S. Use AppSignal for Ruby for deeper debugging insights.

AnyCable for Ruby on Rails: How Does it Improve over Action Cable?

Roy Tomeij — 2024-05-01T00:00:00+00:00

In modern web applications, real-time communication has become more than a feature: it's gradually evolved into a necessity. Users expect instant updates, live interactions, and dynamic content.

In Rails applications, Action Cable has long been the go-to solution, harnessing WebSockets to fulfill these demands. In this article, we introduce:

The basics of WebSockets
How Action Cable enables real-time communication via WebSockets
Why AnyCable was created
How to get AnyCable up and running
The improvements AnyCable brings to the table

Let's get started!

The WebSocket Technology

The WebSocket technology was introduced in 2011 with the publication of RFC 6455, titled "The WebSocket Protocol". This standardized protocol facilitated the establishment of persistent, real-time communication channels between web browsers and servers, allowing for more efficient and continuous data exchange compared to traditional HTTP.

The WebSocket technology enables continuous bi-directional communication between clients and servers, a stark departure from the request-response model of HTTP (where connections terminate after each exchange). This persistence in communication proves pivotal for real-time applications like chat platforms and gaming systems.

Before the introduction of WebSocket technology, many applications often relied on a technique called polling. Polling involves the client repeatedly sending requests to the server at specified intervals to check for updates or new information. While this method allowed for some level of real-time updates, it was less efficient than WebSocket communication. Polling could lead to unnecessary server load and increased latency, as the client had to initiate a new request each time it wanted to retrieve updated data.

WebSocket technology addressed these limitations by establishing a persistent connection between the client and the server, enabling more immediate and efficient data transmission without the need for constant polling. This shift significantly improved the responsiveness and real-time capabilities of applications, providing a seamless and dynamic experience for users, particularly in scenarios demanding instant, up-to-the-moment information.

Both Action Cable and AnyCable harness the power of web sockets to furnish Rails applications with real-time updates, liberating users from the cumbersome need to refresh pages or poll for the latest information.

Action Cable and Real-time Communication

Action Cable enables the incorporation of real-time features into a Rails application via web sockets. This real-time communication happens between the client side (browser) and the Rails server. A few things are important to note:

A connection forms the foundation of this client-server relationship. For every WebSocket connection, Action Cable creates one connection instance.
A channel refers to a communication pathway that allows clients to subscribe to and receive updates about specific topics or categories of information, e.g., BookChannel, MovieChannel, SportsChannel.
A subscriber is a client subscribed to a channel to receive information.
Broadcasting is when the server publishes information that is transmitted by the relevant channel to the subscribed client/s.
A stream is the route or pathway through which broadcasts get to the subscribers. By using stream_from, subscriptions can be made for broadcasting.

When a connection is established between the client and the server, Action Cable creates a connection object for that connection. With this connection, the client can subscribe to one or more channels. Action Cable broadcasts updates to all subscribed clients via their respective channels.

A simple example is a browser view showing a library book list that updates when a book is added.

# create the rails application
rails new book_list
cd book_list
# create the needed controller and model
rails g controller books index
rails g model Book name
rails db:migrate
# change the development Action Cable adapter from Async (the default one) to Redis
rails turbo:install:redis
# start the redis server
redis-server
# start the rails server
rails s

To get a real-time update on the book model, we employ Action Cable for broadcasting the updates. We set up Action Cable to broadcast these updates to a channel called "books". Via Turbo Streams, we create the subscriber to this channel and deliver the updates.

 # app/models/books.rb
class Book < ApplicationRecord
  broadcasts_to -> (book) { :books }
  # Broadcasts all the activities on this model -> create, update, destroy.
end

We can now render the list of books:

# app/views/rooms/_book.html.erb
<div id=<%= dom_id(book) %>> <%= book.name %> </div>

# app/views/books/index.html.erb
<h1>Books</h1>
<%= turbo_stream_from "books" %>
<div id="books">
  <%= render @books %>
</div>

As seen below, different actions trigger a broadcast, and Action Cable broadcasts to the channel specified — "books". The view is also subsequently updated to reflect these updates.

irb> Book.create(name: "The Avatar")
  Rendered books/_book.html.erb (Duration: 0.0ms | Allocations: 11)
[ActionCable] Broadcasting to books: "<turbo-stream action=\"append\" target=\"books\"><template><div> The Avatar </div></template></turbo-stream>"
=> #<Book:0x0000000107c13358>

irb> book_1 = Book.find(1)

irb> book_1.update(name: "The Demo")
  Rendered books/_book.html.erb (Duration: 0.0ms | Allocations: 11)
[ActionCable] Broadcasting to books: "<turbo-stream action=\"replace\" target=\"book_1\"><template><div> The Demo </div></template></turbo-stream>"
=> true

irb> book_1.destroy
[ActionCable] Broadcasting to books: "<turbo-stream action=\"remove\" target=\"book_7\"></turbo-stream>"
=> #<Book:0x0000000107bd2948

In Rails, integrating Action Cable (and, as a result, real-time updates into an application) is quite easy: even a beginner can get up and running in minutes. With Action Cable, we can manage connections, channels, and broadcasting to clients without worrying about the underlying WebSocket details. It also uses the same connection for multiple subscriptions, reducing the overhead of creating new connections for each new real-time update.

Why AnyCable for Ruby on Rails?

Action Cable is built on Ruby and has significant differences from other servers written in Golang and Erlang. Check out a detailed summary of AnyCable vs. Action Cable benchmark findings. Here are the key results:

Action Cable consumes approximately 4 times more memory when handling 20 thousand idle clients without subscriptions or transmissions.
Action Cable requires much higher CPU usage.
Action Cable takes around 10 times longer to broadcast to 10,000 clients. Starting at 1 second for a thousand clients, the time required increases by 1 second for every additional thousand clients.

The AnyCable WebSocket Server was created to combine the beauty of Action Cable with the performance benefits gained from Golang. AnyCable handles WebSockets on a different server called AnyCable-Go, effectively reducing the burden on your primary web application.

In addition to performance advantages, AnyCable offers resumability. If a client becomes disconnected from the server (e.g., due to network issues leading to an offline status), upon reconnection, the client will receive all the messages missed during the disconnection. They are also not required to re-subscribe as sessions are recovered. See a brief demo of the resumability feature.

View detailed information on the recent enhancements to AnyCable.

Integrating AnyCable

Let's replace Action Cable with AnyCable in our BookList example.

Install the gem:

# add the gem to your gemfile
gem "anycable-rails"

# install it
bundle install

Next, run the setup generator.

This generator is interactive and asks questions about:

The type of development environment in use.
How you would like to install the AnyCable-Go websocket server.
If Heroku is being used for deployment.
If JWT is the intended means of authentication.
If you want a Procfile.dev file.

All of this information helps to reduce the amount of manual work you need to do.

rails g anycable:setup

If you did say "yes" to the generation of a Procfile.dev, it would look something like this:

# Procfile.dev

web: bin/rails s
anycable: bundle exec anycable
ws: bin/anycable-go --port=8080

Hence, you do not need to manually run the rest of the commands below.

Run the AnyCable RPC server:

bundle exec anycable

Run the WebSocket server:

anycable-go --port=8080

Restart the Rails server.

In the cable.yml file, you'll find that the adapter changes to any_cable if the ACTION_CABLE_ADAPTER environment variable is not set.

adapter: <%= ENV.fetch("ACTION_CABLE_ADAPTER", "any_cable") %>

In all the $env.rb files, you'll find the Action Cable URL set as:

config.action_cable.url = ActionCable.server.config.url = ENV.fetch("CABLE_URL", "/cable") if AnyCable::Rails.enabled?

In the any_cable.yml file, the Redis channel is set to __anycable__. You can confirm this in the terminal via:

redis-cli PUBSUB CHANNELS
# 1) "__anycable__"

Read more about the AnyCable configuration.

In the Rails server, you'll find a routing error related to the /cable endpoint. You'll also find that, when you filter by websockets in the Network tab of the browser, the Request URL for cable is ws://localhost:3000/cable, which is wrong. We should be attempting to connect to ws://localhost:8080/cable, the value of config.action_cable.url.

To fix this, the action_cable_meta_tag has to be added to the application.html.erb file.

# application.html.erb
<head>
# ...
  <%= action_cable_meta_tag %>
# ...
</head>

This resets the url to the set value. At this point, any broadcasted stream will show up!

Swapping Action Cable for AnyCable in Ruby

To enable resumability, AnyCable utilizes the Action Cable Extended Protocol, which is implemented by the AnyCable server version 1.4 and later. This protocol is inherently supported by the AnyCable JS client.

To integrate this with Turbo Streams, it's necessary to swap out the default Action Cable client for the AnyCable client. However, when using the @hotwired/turbo-rails package, the Action Cable getConsumer function is invoked before the setConsumer function during the initial page load, leading to the consumer not being overridden, and two web socket connections being opened. The @anycable/turbo-stream package was created to handle this (read additional details about this issue).

Here's the procedure for this swap.

Add the @anycable/web and @anycable/turbo-stream packages:

bin/importmap pin @anycable/web @anycable/turbo-stream

This pins the stated packages, along with @anycable/core, @hotwired/turbo, and nanoevents to your importmap.rb file.

Create the anycable client:

// app/javascript/cable.js
import { createCable } from "@anycable/web";

export default createCable();

Refer to this as the new "cable" client:

# config/importmap.rb
pin "cable", to: "cable.js", preload: true

Start the new cable client:

// app/javascript/application.js
// We no longer need @hotwired/turbo-rails
import "@hotwired/turbo";
import { start } from "@anycable/turbo-stream";
import cable from "cable";

start(cable);

With this, we can restart our server, and the consumer/client is replaced with AnyCable.

With the client replaced, resumability can be implemented.

With resumable sessions, the supported adapters are http and redisx (if you use Redis). It is, however, important to note that the broker preset command automatically sets the adapter as http. Hence, when using redisx, it must specified explicitly like so:

$ anycable-go --broker=memory --broadcast_adapter=redisx

There you go! You should have resumable sessions working!

Community Adoption and Feedback

AnyCable has existed for several years and has gained widespread adoption among numerous organizations. Adopters have cited several compelling reasons for its attractiveness, including:

Seamless migration without the need for architectural changes.
No requirement to modify Action Cable channels or connections.
Utilization of a dedicated server for WebSockets.
Ability to handle a substantial number of concurrent connections without encountering any problems.

See a few testimonials from satisfied users.

Wrapping Up

In this post, we've seen that AnyCable offers performance enhancements to Action Cable and can be used as an alternative when dealing with a large number of connections or high traffic.

With recent improvements to AnyCable (such as the introduction of reliable streams), AnyCable has become a solid choice regardless of performance concerns. It supports broadcasting and subscription confirmation, thereby making it consistent and reliable. It can also be used in non-Rails applications, and to power serverless JavaScript applications.

Overall, AnyCable is a reliable choice for modern web applications and supports a variety of servers, allowing for greater flexibility in development.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Should You Use Ruby on Rails or Hanami?

Roy Tomeij — 2024-04-24T00:00:00+00:00

This post was updated on 23 May 2024 to clarify the differences in models and data persistence between Rails and Hanami.

Ruby on Rails is the most popular web framework in the Ruby ecosystem and has a large user base, ranging from freelancers to large established companies. With an active user community and wide-ranging documentation, it can be used to build everything from simple applications to complex web platforms.

That said, a new contestant is taking on Rails’ dominance for the full-stack Ruby framework title: Hanami. It is a fast, modular Ruby framework with improved performance and maintainability compared to Rails.

In this article, we'll explore the strengths and weaknesses of each framework in terms of performance, features, testing, and more. So whether you are looking to build a customer-facing web app, an internal tool, or a massively scalable API, you should leave being better informed about what to use for your next project.

Let's get started with a brief introduction to each framework.

Introducing Ruby on Rails

Ruby on Rails is the most well-known Ruby web application development framework with a mission to enhance developer productivity (by making a bunch of assumptions about how apps should be built, often referred to as the "Rails way").

Some of these assumptions include making sure developers do not spend time configuring stuff, or what is termed as "convention over configuration", and an emphasis on using DRY ("don't repeat yourself") principles. DRY principles encourage a developer to avoid repeating code over and over again, but instead use single and focused representations of app functionality to ensure maintainability and organization.

Introducing Hanami

While Rails is very well-known in the Ruby community, Hanami is less so. It's a fairly new modern Ruby framework trying to take on Rails' dominance of the full-stack web framework space.

Hanami is built from the ground up to have a small memory footprint and a focus on modularity, which, in turn, makes for a very fast and nimble framework.

Obviously, these brief introductions won't really give you all the information you need to decide on the framework that suits you best. For that, we'll need to dive deeper into each, starting with how they are structured.

Structure and Architecture of Rails and Hanami

Rails and Hanami are somewhat similar in that both are Ruby frameworks. However, how each is built and their application architecture is where you'll find most of the differences.

For starters, with Rails, you have fewer files or abstractions (the building blocks you use to build your app), which tend to grow in size as you develop your app. On the other hand, Hanami takes abstractions to a whole new level, with lots of files that tend to be smaller in size.

The diagrams below illustrate the point more clearly. Let's start with Rails.

Now compare the Rails abstraction diagram with the Hanami one below.

As you can see, each framework generally follows the model-view-controller (MVC) structure. However, Hanami takes abstraction to the next level.

Here's a simplified outline of how each framework is organized:

Routes - Each framework has a routes definition with your app's endpoints.
Controllers vs. actions - In Rails, you get controllers with one or a couple of actions in them. Controllers receive and respond to route requests by directing them to the relevant actions. In Hanami, there are no controllers. Instead, you go straight to self-contained actions (each action has its own self-contained class).
Models and persistence - The Rails model layer takes care of data validation and database communication, including any queries your app may have. Persistence is handled by the Ruby Object Mapper (ROM). In Hanami, the model layer is more abstract, separated into entities and repositories. Entities handle domain logic and are database-agnostic, whereas repositories (repos) are used to communicate with the database. Hanami 2.0 onwards ships without a pre-configured persistence layer, allowing you to choose an Object Relational Mapper (ORM) that suits your needs or program ORM free.
View rendering - As with the persistence layer, Ruby on Rails views tend to contain everything you need for rendering data to the outside world in one place. This includes all the HTML structure, view helpers, and view logic. When it comes to view rendering in Hanami, things are more abstract. For starters, you have views that utilize any view helpers the app has and also render the template. The template handles the actual HTML structure, while the parts take care of any presentation logic.

So, what does all this mean when it comes to building an app? With fewer abstractions, Rails is a good choice for getting your app off the ground and is more beginner-friendly (as we shall see later on in this article). With Rails, you can build very robust monolith apps, but your code will get more and more complex as you scale. On the other hand, Hanami's more complex structure can be daunting to learn, but it will allow you to build massively scalable applications and could make for much better code organization.

Next, let's take a look at each framework's ecosystem.

Ecosystem and Community

As we mentioned earlier, Rails has a more established framework than Hanami. Rails has been around for longer and, as such, has a bigger and more mature community.

The differences in ecosystem and community are summarized below:

Documentation - It doesn't matter what you're trying to build, if you use Rails, you have access to very well-done documentation. You are guided through all parts of an app build, from authentication, to data persistence, view presentation, and everything in between. Additionally, a wide range of third-party tutorials cover almost everything imaginable in terms of building an application.

That said, things are not so established on the Hanami side of things. Being a relatively newer framework, Hanami has a comparatively limited documentation base. The Hanami team has done an impressive job with the official guides, but compared to the Rails documentation base, it doesn't come close.

Community - Here, as with the documentation, Rails is the clear winner. Rails has been around much longer than Hanami, so it's been adopted the most. It doesn't matter whether you are a beginner or advanced developer, you'll find a Rails community on relevant subreddits, Slack groups, Discords, and so forth. On the other hand, Hanami is still growing, meaning it has a much smaller community.
Gems and libraries - Since both Hanami and Rails are Ruby frameworks, it could be argued that gems that work in Rails will work in Hanami. Although this is technically true, you'll often find that Rails has a more established base of gems and libraries for all sorts of specialized functions.

That said, because of Hanami's focus on abstractions and specialization, it has some very advanced gems that could take your Rails apps to the next level. For example, the default dry-rb gems in Hanami could bring better code organization and abstraction when used in Rails apps.

Moving on, let's compare each framework's learning curve and adoption.

Ease of Use, Adoption, Governance, and Learning Curve

For the very reasons outlined in the previous section, Ruby on Rails easily beats Hanami in terms of ease of use, industry adoption, and learning curve, specifically:

Learning - Imagine you're a complete beginner who's looking to learn a new programming language. Your very first thought will be to check out online learning resources. If a framework has widely available learning materials covering the app-building process from start to finish, you'll likely choose that language over others. And because Rails has a more established documentation base, it trumps Hanami as the beginner's choice of framework. In addition, Rails is much more friendly for beginners to pick up since it makes so many assumptions under the hood (compared to Hanami, which offers a more abstract way of building apps). Hanami's abstractions can be daunting even for established Rails developers.
Industry adoption - Without including the adoption of other popular and more established frameworks like Python, React, C#, and others, if we consider the adoption of Ruby frameworks, Rails easily eclipses Hanami. The Rails homepage lists some big-name organizations using the framework. On the other hand, as the new kid on the block, Hanami is not so widely adopted. We'll have to wait and see whether that will change in the future.
Job market prospects - In terms of job prospects, you'll find more job openings for Rails developers than Hanami developers.
Governance - Another important aspect that should not be overlooked is governance. A changing landscape guides how open-source frameworks evolve and advance. However, more than that, the core team members of these frameworks often wield immense powers and can dictate what goes into a framework, how it develops, etc.

A good example is January's announcement by David Heinemeier Hansson (DHH), the creator of Rails. He said that, in the future, he would push for Rails to offer first-class support for building full-stack progressive web applications and native notifications. These features would make Rails very attractive to mobile developers.

In reaction, many developers, some outside the Ruby ecosystem and even others who had dropped Rails over the years, overwhelmingly responded in the positive, saying they would gladly pick up the framework or learn it if DHH kept his word. This example just goes to show you that governance is a very important consideration when deciding what framework to pick.

Let's now switch gears and look into something more technical, application performance.

Performance: Ruby on Rails vs. Hanami

As a developer, you will be concerned about your app's reliability, responsiveness, and how it will utilize server resources once deployed in production. These are fundamental concerns that can greatly affect your choice of framework.

You could use many methods to run performance tests on your app, one of the most popular being Apache JMeter. Let's use the benchmark numbers to compare Hanami and Rails.

Here's a quick benchmark test showing the number of requests each framework can handle per second:

Hanami beats Rails hands down, with the ability to handle 3 times more requests than Rails can handle.

This next screenshot shows the average latency values for each framework:

Again, Hanami beats Rails with an average latency time that's about 3 times shorter.

If you're looking for a Ruby framework that is blazingly fast (let's say, you need to work on a fast and hugely scalable API), then you'll be hard-pressed to find anything better than Hanami in the Ruby landscape.

Testing

When it comes to testing code, both frameworks are very much comparable since you can test either using the versatile RSpec library.

RSpec is included via the hanami-rspec gem for Hanami apps, whereas you need to install the rspec-rails gem to use it in your Rails app. The example below shows a very basic test spec that is included with your new Hanami app:

# spec/requests/root_spec.rb

RSpec.describe "Root", type: :request do
  it "is successful" do
    get "/"

    expect(last_response).to be_successful
    expect(last_response.body).to eq("Hello from Hanami")
  end
end

Running it with $ bundle exec rspec spec/requests/root_spec.rb should result in a passing test (assuming you've not edited the default view template):

.

Finished in 0.01812 seconds (files took 0.47734 seconds to load)
1 example, 0 failures

On the Rails side, you need to handle some things by yourself. Firstly, you need to add the RSpec gem to the development/test block in the Gemfile as shown below, then run bundle:

# Gemfile
...
group :development, :test do
    gem 'rspec-rails'
end

The next step is to run the RSpec install with the command bundle exec rails generate rspec:install to get it ready for your Rails project.

Finally, before writing and running any tests, you'll likely need to install another gem to define test data for your Rails app: FactoryBot.

And assuming you've set up your test suite properly and defined a few tests, you can run a test just as you would in Hanami, with bundle exec rspec spec/models/user_spec.rb. You can get your test results just as you would with a Hanami app:

Finished in 0.04421 seconds (files took 0.88106 seconds to load)
1 example, 0 failures

Finally, let's see how the frameworks compare when it comes to deployment.

Deployment

Nowadays, there are lots of options for deploying Ruby apps to production.

To begin with, you could go with a Platform-as-a-Service (PaaS) provider like Heroku, or Fly for a more seamless experience. You can also do a bit of DevOps: set up a Docker installation on a VPS and deploy your app there.

Whichever option you choose, you can expect deployment to be relatively similar in both cases. The only caveat is the lack of extensive documentation and tutorials for deploying Hanami apps.

As we pointed out before, being a newer framework, Hanami still doesn't have extensive documentation covering the deployment process and the challenges that might crop up. If you don't mind hacking around this, then you should be okay.

Wrapping Up

In this article, we've looked at two Ruby frameworks — Ruby on Rails and Hanami — comparing them in terms of features, architecture, performance, and more. Ultimately, we have to answer the question: "Which framework should I use, and why?". We can summarize as follows:

If you're a beginner Ruby developer, start with Rails: it's easier to pick up and learn. If you're a more advanced Ruby developer, develop a Hanami skillset, as it will help you develop even more robust Ruby applications.
If you need to develop an app that should be fast, say an API serving several clients, Hanami's small memory footprint, blazing fast responses, and low latency will serve you well. However, if you just need to develop a monolithic full-stack app to validate a SaaS idea, Rails will definitely get you there faster.

Ultimately, what you end up choosing is really up to you. It isn't a bad idea to acquire skills in both frameworks. That way, you can decide on the best framework to go for, depending on the needs of your project.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

P.P.S. Did you know AppSignal has an integration for Rails and Hanami?

Handling Exceptions in Grape for Ruby

Roy Tomeij — 2024-04-17T00:00:00+00:00

Grape is a popular Ruby framework for building RESTful APIs. Exception handling plays a crucial role in ensuring the stability and reliability of any application, including those made with Grape.

This article will explore the basics of Grape exception handling, including customizing exceptions. We'll also touch on some best practices, and how to integrate your app with AppSignal for enhanced error monitoring and management.

Let's get started!

Basics of Grape Exception Handling

In this tutorial, we’ll see how to handle exceptions in a Grape API built in Rails. I have made a demo job board API for this, and you can check out the source code on GitHub.

Raising an Exception

You can raise an exception in Grape by using error!. For example, in the job API mentioned above, we have a show route that returns a job based on the ID. We can return a 404 error when the record is not available, like this:

get do
  if job
    present job
  else
    error!('404 Not Found', 404)
  end
end

When you raise an exception, you’ll want to handle it in a “unique” way — you most likely will not want to send the raised exception to your users.

In Ruby, we have a default mechanism for exception handling. It works by wrapping code that might raise an exception in a begin block. The rescue block is used to handle the exception that has been raised.

begin
    #... process, may raise an exception
rescue =>
    #... error handler
else
    #... executes when no error
ensure
    #... always executed
end

So, here is what that will look like in a typical scenario:

begin
    File.readlines('input.txt').each { |line| values << Float(line) }
rescue Errno::ENOENT
    p 'file not found'
rescue ArgumentError
    p 'file contains unparsable numbers'
else
    print values
end

The `rescue_from` Method

When you raise exceptions, or they happen without your direct involvement, you’ll want to handle them properly. By default, Grape provides a rescue_from method. This allows you to specify a block of code that gets executed when defined exceptions are raised.

So, to “rescue” or handle the 404 error we raised before any other one that arises in the jobs resource, we can use the rescue_from method. The method is added above the jobs resource.

# Rescue 404 errors
rescue_from :all do |error|
  error!({ error: error.message }, 404)
end

# Jobs resource
resource :jobs do
  desc 'Return list of jobs'
  get do
    ...
  end

  ...
end

We can also specify the content type to be used:

rescue_from :all do |error|
  error!({ error: error.message }, 404, { 'Content-Type' => 'text/error' })
end

This way of handling an exception is too generic — we are rescuing every form of exception and returning an error with a 404 status code. That is misleading if our API users expect to get a 400 status code.

We can instead specify the exception we want to handle:

rescue_from ActiveRecord::RecordNotFound do |error|
  error!({ error: error.message }, 404, { 'Content-Type' => 'text/error' })
end

rescue_from :all do |error|
  error!({ error: error.message }, 500, { 'Content-Type' => 'text/error' })
end

When we encounter an ActiveRecord::RecordNotFound error, we’ll return an error message with a 404 status code. Otherwise, we’ll return an error message with a 500 status code.

This shows that we can improve on what we currently have, but what if we want an error handler that rescues from all errors? That's where customizing exceptions comes in.

Customizing Exceptions in Grape for Ruby

Depending on the type of error encountered, this error handler should be able to return an error message alongside the correct status code.

First, create a file called exceptions_handler. Then, we’ll move our current exception handlers into the file:

# frozen_string_literal: true

module V1
  module ExceptionsHandler
    extend ActiveSupport::Concern

    included do
      rescue_from ActiveRecord::RecordNotFound do |error|
        error!({ error: error.message }, 404, { 'Content-Type' => 'text/error' })
      end

      rescue_from :all do |error|
        error!({ error: error.message }, 500, { 'Content-Type' => 'text/error' })
      end
    end
  end
end

Our ExceptionHandler module uses ActiveSupport::Concern , allowing us to access functionalities like included and class_methods. In the snippet above, we have the error handlers in the included block, so wherever this module is included, they will be available as they’re defined.

We can go ahead and remove the error handler from the files where we had them previously. Then we can include the ExceptionsHandler module in our API entry file — api.rb:

# frozen_string_literal: true

module V1
  class API < Grape::API
    include ExceptionsHandler

    mount V1::Jobs
  end
end

Let’s create a base error class for our errors. This class will be responsible for returning the error response.

module V1
  module Exceptions
    class BaseError < StandardError
      attr_reader :status, :message
      def initialize(message: nil, status: nil)
        @status = status || 500
        @message = message || "Something unexpected happened."
      end

      def body
        Rack::Response.new({ error: message }.to_json, status)
      end
    end
  end
end

The class accepts two keyword parameters: a message string and a status. If none is passed, we’ll use the default.

In the body method, we return a Rack response. By default, the rescue_from handler must return a Rack::Response object, call error!, or raise an exception.

We can go ahead and make use of it in the ExceptionsHandler:

included do
  rescue_from ActiveRecord::RecordNotFound do |error|
    error!({ error: error.message }, 404, { 'Content-Type' => 'text/error' })
  end

  rescue_from :all do |error|
    Exceptions::BaseError.new(message: error.message).body
  end
end

When we call the /error endpoint, we’ll see oops returned as the response. At this point, we can create a class for NotFound errors.

module V1
  module Exceptions
    class NotFound < BaseError
      def initialize(message: nil)
        super(
          status: 404,
          message: message || "Oops, we could not find the record you are looking for."
        )
      end
    end
  end
end

The NotFound class only accepts message. Since it inherits from BaseError, we need not return a Rack::Response again. We can go ahead and use it in the ExceptionsHandler like this:

included do
  rescue_from ActiveRecord::RecordNotFound do |error|
    Exceptions::NotFound.new(message: error).body
  end

  rescue_from :all do |error|
    Exceptions::BaseError.new(message: error.message).body
  end
end

Now, if we attempt to raise an error like this manually:

raise Exceptions::NotFound.new(message: "Something unexpected happened.......")

This will work fine, but the status code will be 500, because it returns the response in the BaseError class (as the BaseError class handles the error). To fix that, we’ll need to modify the ExceptionHandler to explicitly use the NotFound class to handle the error instead.

So, whenever an error corresponding to ActiveRecord::RecordNotFound and V1::Exceptions::NotFound is encountered, use Exceptions::NotFound. Otherwise, use Exceptions::BaseError.

included do
  rescue_from ActiveRecord::RecordNotFound do |error|
    Exceptions::NotFound.new(message: error).body
  end

  rescue_from V1::Exceptions::NotFound do |error|
    Exceptions::NotFound.new(message: error.message).body
  end

  rescue_from :all do |error|
    Exceptions::BaseError.new(message: error.message).body
  end
end

You can see that we’ll need specificrescue_fromblocks as we create more error classes. We can improve this by using case statements:

module V1
  module ExceptionsHandler
    extend ActiveSupport::Concern

    included do
      rescue_from :all do |error|
        case error.class.name
        when 'ActiveRecord::RecordNotFound', 'V1::Exceptions::NotFound'
          Exceptions::NotFound.new(message: error.message).body
        else
          Exceptions::BaseError.new(message: error.message).body
        end
      end
    end
  end
end

Et voilà!

Best Practices and Tips

While there are tons of best practices that you can employ for exception handling, here are a few quick tips to follow:

Group related exceptions: As we saw in the code above, grouping related exceptions allow us to have maintainable code. As the number of exceptions we want to handle increases, we can add them to our list.
Use helpers like error! to quickly raise exceptions. This simplifies your exception handling.
Make use of exception monitoring tools like AppSignal.

AppSignal Integration: Grape for Ruby

AppSignal helps you to monitor and track errors in your applications. Integrating AppSignal with your Grape API gives you valuable insights into exceptions. This guide shows you how to integrate AppSignal with your Grape API. Whenever an error occurs in your API, you’ll see it in your AppSignal dashboard, like so:

Wrapping Up

Exception handling is a critical aspect of developing robust APIs. In this tutorial, we’ve seen how to properly handle exceptions in a Grape API. We also briefly looked at some best practices and AppSignal's integration for Grape.

Exception handling is an ongoing process — one you’ll need to improve on consistently.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

P.P.S. Did you know that AppSignal offers an Active Record integration? Find out more.

Good Database Migration Practices for Your Ruby on Rails App using Strong Migrations

Roy Tomeij — 2024-03-20T00:00:00+00:00

One great feature that comes with modern web frameworks is the ability to manage database schema migrations. However, schema migrations are not 100% safe and remain a recurring cause of issues within projects I have encountered over the last 15 years.

This article will review the issues surrounding poorly managed schema migrations and then look into Strong Migrations, a gem that can help you avoid most problems. Finally, we will discuss a few good practices around database management.

Let's get started!

Issues with Schema Migrations in Ruby on Rails

Schema migrations are changes to a table or database schema within an RDBMS: adding, renaming, removing, and updating a table (or a column within another table), index, or view. Some schema migrations are inherently risky (such as removing a column or table). Generally, all carry a risk, especially alongside any related code changes.

Schema migrations are also made worse with larger tables. Some changes, like updating columns and altering indexes, take longer to apply and might imply database locks, causing performance issues.

We also have to talk about how migrations are brought to production via related code changes. Migrations that add, update, or remove a schema element (table or column) and are used in the code are the most problematic. This is because many developers don't build good practices into those cases early enough. The typical scenario is as follows:

A developer adds a Model and related table.
They merge the code change.
The code change gets deployed, but the migration hasn't run yet.
The application starts and crashes.

The reason for the crash is simple: as an application starts, Ruby on Rails will load the schema but won't find the new table in the database.

How Can We Fix Problems with Schema Migrations?

Looking at the above example, the solution is to run a database migration before an application starts. The deployment tooling should take care of that. However, in the case of a column or table removal, a migration must run after an application starts. So the easiest thing is to separate the change into two deployments in the correct order and make sure the tooling or team runs the migration when appropriate. Sometimes, we may have to split the change into more deployments.

Let's take the cases we listed before: adding, removing, or updating a column.

In the first case of adding a column, we need to deploy and run a migration before the code change that will start to use it.
In the second case (removing a column), it's the opposite: we need to remove code that's using the column before removing the column itself.
Updating a column is trickier. Such a change will enforce a lock. And it's even worse if you change the column type. This is the suitable strategy:

Create a new column.
Write to both old and new columns.
Backfill existing data from the old column into the new one.
Move the reads to the new column.
Stop writing to the old column.
Drop the old column.

These three strategies should be used for any project, especially once a database fills up and users rely on the product.

Yet, just like linters remove some cognitive load by taking care of this kind of toil, it's best to rely on a tool to automate this.

Here's where the Strong Migrations library comes into play. It adds a layer of protection around migrations to ensure you can avoid issues upon deployment in any environment.

Getting Help from Strong Migrations

The folks at Instacart have open-sourced many libraries. Among them sits Strong Migrations, aimed at "catching unsafe migrations in development". Once installed, it will:

Detect potentially dangerous operations.
Prevent them from running by default, and tell you why.
Provide instructions on safer ways to do things.

A migration will be deemed dangerous if it's likely to cause the issues we discussed earlier: locks and potential errors due to timing. Check out the Strong Migrations README on GitHub for more information.

We will now cover a few cases here.

Installation

This is straightforward: use bundle add strong_migrations, followed by bundle install, and rails generate strong_migrations:install. The last command will create an initializer to configure a few things when the application starts.

Adding a Column or a Table

There is no help from Strong Migrations on this one, but remember it's good practice to separate a table addition from the code using it. Please ensure a schema change happens before any code using it runs and expects to find it.

Renaming a Column

Here's where we start to see what Strong Migrations can do for us. Taking the example of a migration that renames a column from "address" to "location", we get the following message when we try to run it locally in our development environment:

== 20231116202001 RenameUserAddress: migrating ================================
rails aborted!
StandardError: An error has occurred; this and all later migrations have been canceled:

=== Dangerous operation detected #strong_migrations ===

Renaming a column that's in use will cause errors
in your application. A safer approach is to:

1. Create a new column
2. Write to both columns
3. Backfill data from the old column to new column
4. Move reads from the old column to the new column
5. Stop writing to the old column
6. Drop the old column

article/db/migrate/20231116202001_rename_user_address.rb:3:in `change'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

So, yes, the Strong Migrations gem does not replace you. Instead, it reminds you that what you are doing is dangerous and that you should not do it. It also tells you how you should do things instead.

Removing a Column

Now, let's say that we have gone through the first five steps of the list that Strong Migrations generated for us, and it's time to remove the old column.

Is it as simple as creating a migration and using the remove_column method? No, it's not.

Again, Strong Migrations will print the following text:

== 20231116202636 RemoveUserAddress: migrating ================================
rails aborted!
StandardError: An error has occurred; this and all later migrations have been canceled:

=== Dangerous operation detected #strong_migrations ===

Active Record caches attributes, which causes problems
when removing columns. Be sure to ignore the column:

class User < ApplicationRecord
  self.ignored_columns = ["address"]
end

Deploy the code, then wrap this step in a safety_assured { ... } block.

class RemoveUserAddress < ActiveRecord::Migration[7.0]
  def change
    safety_assured { remove_column :users, :address, :string }
  end
end

article/db/migrate/20231116202636_remove_user_address.rb:3:in `change'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

So we know what to add to the model and how to change our schema migration. It's tailored to your use case, including the table's name and column. Once that is done, the migration will go through without an issue.

Backfilling Data

The remaining piece on the database strategy side is backfilling data in new columns from old ones. While relying on the Data Migrate gem might be tempting, we can also backfill data in a regular migration. A safe backfilling migration will consider a table's size, the task's complexity, and avoid transactions.

By default, Ruby on Rails uses a transaction around each migration. If we include backfilling in the migration doing the schema change, this might take some time and potentially cancel our whole schema migration.

So, instead, let's use a separate migration, avoid using a transaction, batch the process, and throttle the work.

class BackfillCountryCodeColumn < ActiveRecord::Migration[7.1]
  disable_ddl_transaction!

  def up
    User.unscoped.in_batches do |relation|
      relation.update_all country_code: 'fr'
      sleep(0.01)
    end
  end
end

Notes:

unscoped ensures that we don't rely on the "default scope" defined in the model to work with all rows.
in_batches forces the block to run in batches of 1000 entries.
sleep gives some reprieve to the database in between batches.

An alternative method is to use a script or a rake task to handle this. Whatever solution you pick, remember that backfilling a column or table with data might be pretty intensive for the database. Estimate how much data will be handled and consider the techniques used in the above example, if needed.

Adding Strong Migrations to an Existing Ruby on Rails Project

Adding Strong Migrations to an existing project is a good idea. However, you might run into issues when setting up a new project instance on a workstation or within a new environment. A failsafe will kick in and prevent any prior migrations from running.

So, what shall we do? You don't have to go through all the migrations and alter them accordingly. Instead, you should follow the Ruby on Rails guide and clean up old migrations.

I've not seen many projects follow that advice, but it's the "good thing" to do: db/schema.rb or db/structure.sql holds a snapshot of the current database schema and can be used to load it. Tasks such as db:setup and db:prepare (and ones based on either of them) use that snapshot to load schema in a database. Only pending migrations are run; old ones are barely a line in one table, a mere memory.

You can remove old migration files if you need them: git (or Mercurial, or Svn) will keep a trace of them. Once a migration has been applied to all existing environments, it can be deleted as the schema.rb (or structure.sql) file will serve for new environments.

Here are the steps you should follow:

Before introducing Strong Migrations, ensure all environments are up to date in terms of migrations.
Remove old migration files.
Add and install the Strong Migrations gem.
Start to create new migrations.

Doing so will prevent you from going through years of migrations to set them in line with Strong Migrations. It will also lighten the burden of setting up a new environment.

You can then devise a periodic removal of migration files, keeping the last 5, 10, or any from the previous month, for example.

Some Additional Advice on Migrations

Let's finish up by sharing some general tips on migrations when it comes to:

Production environments and databases
Avoiding downtime
Backups

Production Environments and Databases

Considering your production environment and its database is key when creating and running any migration. A developer environment is rarely bursting at the seams when it comes to data. Consequently, most migrations will run smoothly with barely any waiting time. Production environments have a lot more data that's more diverse. This will tend to complicate things when it comes to writing and applying a migration.

You should follow strategies like backfilling. Batching and throttling are also good practices when handling large volumes of data. It's never too early to start using those to develop good habits. Once a migration is out there and running, we should monitor performance metrics: response time and slow queries. Compare these metrics with previous trends, and look into unexpected changes.

Avoiding Downtime

One reason that developers tend to be scared about making deployments around the end of the day or the week is the fear of the unknown.

By following the good practices pointed out by Strong Migrations, we can reduce the amount of risks and unknowns, thus giving us more confidence in our ability to deploy at any time.

Backups

If things go wrong, don't panic; you should have a backup ready. Most managed solutions available for PostgreSQL, MySQL, and similar RDBMS offer backups at regular intervals. Some also provide point-in-time recovery, allowing us to rewind to better times.

Wrapping Up

In this post, we first explored some potential issues related to schema migrations in Rails, before seeing how we can fix these issues ourselves. We then introduced Strong Migrations. As we have seen, Strong Migrations ensures that risky changes such as removals and changes to a schema element are done correctly to avoid risk.

Finally, we touched on a few additional tips regarding migrations.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Turbo Streaming Modals in Ruby on Rails

Roy Tomeij — 2024-03-13T00:00:00+00:00

In part one of this series, we used Hotwire's Stimulus and Turbo Frames to present modals in Rails.

Now, we'll dive into another method we can use to present modals: Turbo Streams.

What Are Turbo Streams in Ruby on Rails?

Turbo Streams is a subset of Turbo. It allows us to make fine-grained, targeted updates to a page. By default, it contains seven CRUD actions, but we're free to add more actions within our applications.

Now, we'll create a show_remote_modal action which renders and presents the <dialog> from our previous post.

Creating a Custom Action

Create a folder to place all custom Stream Actions in:

$ mkdir app/javascript/stream_actions
$ touch app/javascript/stream_actions/index.js

And a file for the Action:

$ touch app/javascript/stream_actions/show_remote_modal.js

Import the Stream Actions into the application:

// app/javascript/stream_actions/index.js

import "./show_remote_modal";

// app/javascript/application.js

// ...
import "stream_actions";

If you're using import maps, you'll need to update the config and restart the server:

# config/importmap.rb

# ...
pin_all_from "app/javascript/stream_actions", under: "stream_actions"

Change the global remote modal container to an HTML element instead of a Turbo Frame:

<%# app/views/layouts/application.html.erb %>

<!DOCTYPE html>
<html>
  <%# ... %>

  <body>
    <%= yield %>

    <remote-modal-container></remote-modal-container>
  </body>
</html>

The custom Stream Action can be implemented as:

// app/javascript/stream_actions/show_remote_modal.js

Turbo.StreamActions.show_remote_modal = function () {
  const container = document.querySelector("remote-modal-container");
  container.replaceChildren(this.templateContent);
  container.querySelector("dialog").showModal();
};

In the above snippet, this refers to StreamElement, which is the custom element underpinning <turbo-stream>. The templateContent getter is defined by this element.

Using the Action with a Rails Helper

Since this is a custom Action, we'll need to manually create a Rails helper to use it.

$ bin/rails generate helper TurboStreamActions

# app/helpers/turbo_stream_actions.rb

module TurboStreamActionsHelper
  def show_remote_modal(&block)
    turbo_stream_action_tag(
      :show_remote_modal,
      template: @view_context.capture(&block)
     )
  end
end

Turbo::Streams::TagBuilder.prepend(TurboStreamActionsHelper)

We can now use this helper in our views.

<%# app/views/support/tickets/new.html.erb %>

<%= turbo_stream.show_remote_modal do %>
  <dialog id="contact_form_modal" aria-labelledby="modal_title">
    <header>
      <h2 id="modal_title">
        Contact us
      </h2>
      <form method="dialog">
        <button aria-label="close">X</button>
      </form>
    </header>

    <%= form_with(url: support_tickets_path) do |form| %>
      <%= form.label :message, "Your message" %>
      <%= form.text_area :message, autofocus: true %>

      <%= form.button "Close", value: nil, formmethod: :dialog %>
      <%= form.button "Send" %>
    <% end %>
  </dialog>
<% end %>

Remember to remove the data-controller attribute: we don't need it anymore. In fact, we can delete the controller itself.

$ rm app/javascript/controllers/remote_modal_controller.js

We'll also need to change the template's name so it renders as a Turbo Stream.

$ mv \
    app/views/support/tickets/new.html.erb \
    app/views/support/tickets/new.turbo_stream.erb

Turbo Streams are disabled by default for GET requests, so we'll need to manually enable them for the link:

<%# app/views/support/show.html.erb %>

<%# ... %>

<%= link_to new_support_ticket_path, data: { turbo_stream: true } do %>
  Show contact form
<% end %>

<%# ... %>

Refresh the page and click Show contact form. It should still work as before, but now it's rendered using a custom Stream Action!

Wrapping Up

In this two-part series, we explored three different methods to present modals using Hotwire: Stimulus, Turbo Frames, and Turbo Streams. More importantly, the modals were presented with accessibility as the main consideration.

The web should be usable by everyone and it's important for us, as web developers, to put in the effort to make websites accessible.

Basecamp's accessibility guide is publicly available and a fantastic resource to learn the ropes.

I also recommend checking out the docs for Stimulus and Turbo to familiarise yourself with all their features and the APIs used in this series.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Active Record or Sequel: Which Best Fits The Needs of Your Ruby App?

Roy Tomeij — 2024-03-06T00:00:00+00:00

When it comes to choosing an object-relational mapping (ORM) library for your Ruby application, Active Record is usually the favorite choice. It's an easy-to-use ORM library that allows for lots of data wrangling without resorting to SQL. All the same, you might wonder: "Is Active Record the only Ruby ORM library I can use?"

In this article, we'll compare some Active Record features to its lesser-known but powerful cousin, Sequel. There are too many points of comparison to cover everything (such as how each library handles CRUD operations, table joins, associations, database replication and sharding, etc). Instead, we'll scratch the surface of a few database operations — namely, filtering, database locking, and transactions — and show how each library handles them.

By the end, you'll have a better idea about how to use each library's strengths to the fullest.

But first, let's learn what object-relational mapping is all about.

What Is Object-Relational Mapping (ORM)?

Object-relational mapping is a way to pass data between an application and its data store, usually a relational database like SQLite, PostgreSQL, or MySQL. Without ORM techniques, you would be forced to write raw SQL queries for all operations to get data in or out of your Ruby app's database. But with an ORM, you get access to class objects and methods, making it much easier to read/write data to your database without using raw SQL.

Let's introduce the two ORM libraries we're looking at, starting with Active Record, and then moving on to Sequel.

Introducing Active Record and Sequel for Ruby

Let's quickly examine what Active Record and Sequel can do.

Active Record

Active Record is the most well-known Ruby object-relational mapping library. It defines class objects that are directly mapped to tables in a database and provides convenient methods for manipulating data using these classes.

The library was first described by the pioneering computer scientist Martin Fowler in his book Patterns of Enterprise Architecture. It comes bundled with Rails, which could explain why it's more popular than Sequel. That said, Active Record is suitable for:

Object-oriented database operations
Performing validations on models before persisting them in the database
Representing models, their data, and relationships

Sequel

On the other hand, Sequel is the lesser-known of the two. It includes a powerful DSL for manipulating data within a variety of databases.

Sequel can:

Represent database records as Ruby objects
Handle simple and even complex associations in a concise way
Do database sharding, database replications, and more

One standout feature that makes Sequel so powerful is its dataset. A dataset is Sequel's way of directly representing an SQL query and making it readily available for a developer to use.

Of course, a lot more could be said of each of these ORM libraries, but we'll leave it at that. Let's dive straight into our first example to see how Active Record and Sequel deal with record filtering.

Filtering Records

Filtering data is the process of refining data results by applying specific filters. By using filters, you can fetch exactly what you want from a dataset. Let's see how each ORM library achieves this, starting with Active Record.

Using Active Record

Let's say you want to filter data to return all orders of products priced under $20. How would you achieve this using Active Record?

You can apply conditions to your Active Record query. Active Record has many methods that act as conditions for filtering data, including where, limit, and where.not. While it's not possible to get into all of the conditions in this article, we'll sample a few and see how they stack up against Sequel's filter methods.

For Active Record, you'll first need to set up the proper model associations like so:

# models/order.rb
class Order < ActiveRecord::Base
  belongs_to :product
end

Then the Product model:

# models/product.rb

class Product < ActiveRecord::Base
  has_many :orders
end

Now for the Active Record query:

product_orders_under_20 = Order.joins(:product).where('products.price < ?', 20)

In this example, we've used a joins to include results from the products table and then applied a where condition to filter for the products that fit our desired criteria.

Using Sequel

Sequel's use of SQL representations in the form of datasets makes filtering a breeze. Let's consider how Sequel would handle the example we used for Active Record.

Just like we did with Active Record, you need to define some models first:

# models/product.rb

class Product < Sequel::Model
  one_to_many :orders
end

The Order model:

# models/order.rb

class Order < Sequel::Model
  many_to_one :product
end

And this is how you'd run the query with Sequel:

product_orders_under_20 = DB[:orders].join(:products, id: :product_id).where { price < 20 }

Next up, let's look at database locks.

Database Locking

Imagine you have a content management system where a user with the editor role can edit other users' posts. Let's say there are several editors in the organization and it just so happens that two editors end up working on the same post at the same time. Such a scenario raises several questions, such as which editor's work is saved, assuming both commit changes at the same time.

This challenge is solved through the use of database locks.

Using Active Record

Active Record allows for two types of locks: optimistic locks and pessimistic locks. In optimistic locking, you need to include a version column in the respective database table. This is so that a record is checked against a matching version to ensure that another user or process doesn't modify it. If it is, an error is raised.

The example below shows how optimistic locking happens:

invoice1 = Invoice.find(1)
invoice2 = Invoice.find(1)

invoice1.total_amount = 100
invoice1.save # 100

invoice2.total_amount = 500
invoice2.save # raises ActiveRecord::StaleObjectError

When it comes to pessimistic locks, Active Record can implement them at the row level if your database allows for it. There are several ways of achieving pessimistic locking with Active Record, but we won't go into the details now. Instead, the example below should give you an idea of how this is done:

Invoice.lock.find(1) # applies a pessimistic lock on the record so that an UPDATE is done

Using Sequel

On the other hand, Sequel implements optimistic locking through a plugin.

You first add the plugin to the model like so:

# invoice.rb

class Invoice < Sequel::Model
  plugin :optimistic_locking
end

Then test this with two concurrent queries:

invoice1 = Invoice.find(1)
invoice2 = Invoice.find(1)

invoice1.update(status: 'Sent') # record will be updated

invoice2.update(status: 'Void') # raises Sequel::NoExistingObject error

For pessimistic locking, Sequel implements the lock when the records (or record) are fetched using the following methods: lock_style, lock, and for_update. Again, we won't provide details on this. You can read the documentation or Jeremy Evans' Pessimistic Locking blog post to learn more.

Here's an example using the for_update method to implement a pessimistic lock:

dataset = DB[:invoices]
invoice1 = dataset.where(id: 1).for_update # will apply a pessimistic lock on invoice1

Next, let's take a look at database transactions.

Database Transactions

To understand the concept of database transactions, consider a peer-to-peer lending app where Person A sends some money to Person B who can then withdraw the money and use it. Here, the app needs to first withdraw money from Person A's account and then send that amount over to Person B.

In such a scenario, you wouldn't want the app to send money to Person B if there's an error with the withdrawal from Person A's account. The ideal solution would be to wrap the two processes in a "database transaction", so if one of the processes fails (usually the first one), then the database can be rolled back to its original state before any of the transactions happened.

Using Active Record

With Active Record, implementing these queries without the use of database transactions might look something like this:

person_a = Person.find_by(name: 'A')
person_b = Person.find_by(name: 'B')

withdrawal = person_a.accounts.first.withdraw(250)
person_b.accounts.first.account_total # 100

person_b.find(B).accounts.first.deposit(withdrawal)
person_b.accounts.first.account_total # 450

Although this could work, you also need to account for something going wrong with the withdrawal and take steps to take care of it. Let's now use a database transaction to improve this example:

a_account = Person.find_by(name: 'A').accounts.first
b_account = Person.find_by(name: 'B').accounts.first

Account.transaction do
  a_account.withdraw(250)
  b_account.deposit(250)
end

Here, we use ActiveRecord::Transactions to wrap the two queries in a transaction so that the database can be rolled back if something goes wrong.

What about Sequel?

Using Sequel

Just like in Active Record, if you want to implement transactions in Sequel, you'll need to be explicit (although there are situations where this is enabled by default).

With the same example we used for Active Record, this is how to wrap our queries using a transaction in Sequel:

DB.transaction do
  Person.first(name: "A").accounts.first.withdraw(250)
  Person.first(name: "B").accounts.first.deposit(250)
end

From this example, we see that Sequel's dataset feature allows for complex transactional queries to be built using a few commands.

When To Choose Active Record Vs. Sequel for Ruby

When should you use Active Record, and when might Sequel be a better choice? Here's a quick summary of each:

Active Record is very beginner-friendly, in that, it conveniently hides complex SQL queries under the hood. On the other hand, if you are an advanced user who's comfortable using SQL, then this may be a bit limiting. Additionally, if you're looking for more powerful features like proxying queries, you'll likely need to add complementary gems on top of Active Record.
Sequel is very powerful, especially for those who master its dataset feature. That said, you can expect a steeper learning curve compared to Active Record. If you get through that, though, you will be rewarded with a flexible ORM that is perfect for almost anything you could throw at it.

Wrapping Up

In this article, we've compared two Ruby object-relational mapping libraries, Active Record and Sequel. We explored how each handles filtering, database locking, and transactions. Finally, we touched on when you might want to use one ORM over the other.

Until next time, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

P.P.S. Did you know that AppSignal offers an Active Record integration? Find out more.

Hotwire Modals in Ruby on Rails with Stimulus and Turbo Frames

Roy Tomeij — 2024-02-21T00:00:00+00:00

Modals are widely used on the web, but they are rarely built with accessibility in mind. When a modal is displayed, the background is dimmed visually but it's still visible to screen readers and keyboard-only users.

In this post, the first of a two-part series, we'll look at presenting accessible modals in Rails using two different approaches powered by Hotwire's Turbo and Stimulus libraries.

But first, let's see what we need to do to make modals accessible.

How Can We Make Modals Accessible?

To make modals accessible, we need to:

Hide the background from screen readers.
Implement "focus trapping" so keyboard users cannot focus on elements outside the modal.
Shift focus to the first focusable element in the modal (if there is one).
Dismiss the modal with the Esc key.

We could accomplish the first two by setting inert="true" on the background elements, but this attribute can't be set on an ancestor of the modal. We can do the last two with some more custom JavaScript. As you can imagine this is rather tedious.

The HTML <dialog> element gives us most of the above list for free, and is supported for 94.25% of global users, meaning it's ideal in most cases.

Let's look a bit more closely at the <dialog> element.

The `<dialog>` Element

According to MDN:

The HTML <dialog> element is used to create both modal and non-modal dialog boxes. Modal dialog boxes interrupt interaction with the rest of the page being inert, while non-modal dialog boxes allow interaction with the rest of the page.

This means that a <dialog> isn't necessarily a modal. It can be presented within the document flow as well.

Presenting a `<dialog>`

To present a dialog as a modal, we need to use JavaScript:

<dialog>Lorem ipsum ....</dialog>

const modal = document.querySelector("dialog");
modal.showModal();

When presenting a dialog this way, the background is made inert, focus is trapped in the modal, and it can be dismissed using the Esc key.

To present a <dialog> in a non-modal context, and hence make it visible by default, we use the open attribute:

<dialog open>Lorem ipsum ....</dialog>

We won't get the accessibility features using this method, though, as the dialog hasn't been presented "modally".

Dismissing a `<dialog>`

A modally presented <dialog> can be dismissed using JavaScript:

const modal = document.querySelector("dialog");
modal.showModal();
modal.close();

It can also be closed using a <form> with the dialog method. This is a great way to implement a "close" button.

<dialog>
  <header>
    <h2>A modal dialog</h2>
    <form method="dialog">
      <button type="submit">Close</button>
    </form>
  </header>

  Lorem ipsum...
</dialog>

That covers the basics of the <dialog> element, so let's look at using it with Hotwire.

Modal Presentation Using Stimulus in Ruby on Rails

Stimulus is a JavaScript library under the Hotwire umbrella. It allows us to attach pieces of JavaScript logic to HTML elements encapsulated in controllers. This post assumes a basic familiarity with its API.

Let's start with a Rails controller and view. As an example use case, we'll use a support page which has a button to display some contact details in a modal. Generate a controller and action using:

$ bin/rails generate controller support show

Amend the created route to a more user-friendly path:

# config/routes.rb
Rails.application.routes.draw do
  # ...

  get '/support', to: "support#show"
end

Sketch out a button and dialog in the newly generated view file:

<%# app/views/support/show.html.erb %>

<button>
  Show contact details
</button>

<dialog aria-labelledby="modal_title">
  <header>
    <h2 id="modal_title">
      Contact details
    </h2>
    <form method="dialog">
      <button aria-label="close">X</button>
    </form>
  </header>

  <p>
    Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
    incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
    nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
  </p>
</dialog>

Run the Rails server and navigate to the /support path. You should see a button there, but it doesn't do anything at the moment. Let's create a Stimulus controller and wire it up.

The Stimulus Controller in Rails

Use the generator to create a Stimulus controller.

$ bin/rails generate stimulus modal

When the button is clicked, we need to get a reference to the <dialog> and call showModal() on it. To keep the controller generic, we'll pass in the element's id as a param.

// app/javascript/controllers/modal_controller.js

import { Controller } from "@hotwired/stimulus";

// Connects to data-controller="modal"
export default class extends Controller {
  show(event) {
    const dialog = document.getElementById(event.params.dialog);
    dialog.showModal();
  }
}

We can now decorate the button with the required data- attributes:

<%# app/views/support/show.html.erb %>

<button
  data-controller="modal"
  data-action="modal#show"
  data-modal-dialog-param="contact_details_modal">
  Show contact details
</button>

<dialog id="contact_details_modal" aria-labelledby="modal_title">
  <%# ... %>
</dialog>

Refresh the page and the button should now work!

There's an opportunity to remove some boilerplate code here. The modal controller should always be attached to a button that shows the modal. We can remove the data-action from the markup and set it in the controller:

// app/javascript/controllers/modal_controller.js

// ...
export default class extends Controller {
  connect() {
    this.element.dataset.action = "modal#show";
  }

  show(event) {
    // ...
  }
}

Styling the Modal

The dialog looks fairly basic, so let's add some styles:

/* app/assets/stylesheets/application.scss */

dialog {
  width: 80vw;
  margin: auto;

  &::backdrop {
    background: red;
    opacity: 0.2;
  }

  header {
    display: flex;
    align-items: center;

    h2 {
      flex: 0 1 100%;
    }
  }
}

The ::backdrop pseudo-element is a really great way to style the modal's background!

Try using the Tab key to navigate around the page and you'll see that the focus is trapped within the modal. It's worth reviewing the page with a screen reader as well.

This is the simplest way to modally present a <dialog>. Next, we'll look at a server-driven way to present modals: Turbo Frame.

Turbo Frame Powered Modals for Ruby

Turbo Frames are a subset of the Turbo library which is also under the Hotwire umbrella. It allows us to scope navigation to specific parts of a page, updating them in isolation from the rest of the page.

We can use Turbo Frames to present a modal rendered from the server.

Setting Up

Let's add another button to display a modal contact form. We'll need a controller and action to render the form:

$ bin/rails generate controller support/tickets new create

Replace the auto-generated routes with:

Rails.application.routes.draw do
  # ...

  namespace :support do
    resources :tickets, only: [:new, :create]
  end
end

We'll also need a global Turbo Frame to render modals, so let's put it in the main application layout:

<%# app/views/layouts/application.html.erb %>

<!DOCTYPE html>
<html>
  <%# ... %>

  <body>
    <%= yield %>

    <%= turbo_frame_tag :remote_modal %>
  </body>
</html>

Add a link to show the contact form:

<%# app/views/support/show.html.erb %>

<button
  data-controller="modal"
  data-modal-dialog-param="contact_details_modal">
  Show contact details
</button>

<%= link_to new_support_ticket_path, data: { turbo_frame: :remote_modal } do %>
  Show contact form
<% end %>

<%# ... %>

And you're done!

Rendering and Presenting the Form

Clicking the Show contact form link will expect a <turbo-frame> with id="remote_modal" in the response and update the global Turbo Frame with its content. Fill in the view with the form:

<%# app/views/support/tickets/new.html.erb %>

<%= turbo_frame_tag :remote_modal do %>
  <dialog id="contact_form_modal" aria-labelledby="modal_title">
    <header>
      <h2 id="modal_title">
        Contact us
      </h2>
      <form method="dialog">
        <button aria-label="close">X</button>
      </form>
    </header>

    <%= form_with(url: support_tickets_path) do |form| %>
      <%= form.label :message, "Your message" %>
      <%= form.text_area :message, autofocus: true %>

      <%= form.button "Close", value: nil, formmethod: :dialog %>
      <%= form.button "Send" %>
    <% end %>
  </dialog>
<% end %>

We've now got a text area in the modal which is a focusable element. For accessibility, it should be focused by default when the modal is presented. The autofocus attribute is used to accomplish this.

Refresh the page and try clicking Show contact form. Nothing will happen visually, but on inspecting the HTML, you'll notice the <dialog> has been rendered in the remote_modal Turbo Frame. We haven't presented it yet, which is why it's invisible.

We could render it with the open attribute, but that would defeat the purpose as it'd be presented in a non-modal context without the modal accessibility features.

Let's create another Stimulus controller to present the form:

$ bin/rails generate stimulus remote_modal

// app/javascript/controllers/remote_modal_controller.js

import { Controller } from "@hotwired/stimulus";

// Connects to data-controller="remote-modal"
export default class extends Controller {
  connect() {
    this.element.showModal();
  }
}

And hook it up to the dialog:

<%# app/views/support/tickets/new.html.erb %>

<%= turbo_frame_tag :remote_modal do %>
  <dialog
    id="contact_form_modal"
    aria-labelledby="modal_title"
    data-controller="remote-modal">
    <%# ... %>
  </dialog>
<% end %>

Refresh the page and try viewing the contact form again. This time it should work. We've just rendered a modal from the server!

While this is quite convenient, it's not ideal to create a new Stimulus controller just to present a modal. There's another method we can use as well: Turbo Streaming Modals. We'll take a look at those in part two of this series.

Up Next: Turbo Streaming Modals

In this post, we explored two different methods to present modals using Hotwire: Stimulus and Turbo Frames.

In part two, we'll look at another way to present modals: using Turbo Streams.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Full-Text Search for Ruby on Rails with Litesearch

Roy Tomeij — 2024-02-14T00:00:00+00:00

In this post, we'll turn to the last piece of the puzzle in LiteStack: Litesearch.

As an example, we will equip a prompts index page with a search bar to query a database for certain prompts. We will generate a couple of fake records to test our search functionality against.

Let's get to it!

What Is Litesearch?

Litesearch is a convenience wrapper built around FTS5, SQLite's virtual table-based full-text search module.

We'll dive into the mechanics a bit later. For now, we will assume that Litesearch is a Ruby module providing a simple API to perform text searches against a SQLite database. This works in standalone mode, but we will focus on the ActiveRecord integration, of course.

Configure the `Prompt` Model for Litesearch

The single addition we must make to our prompt model is a search index schema definition. To do this, we have to include the Litesearch::Model module in our model and call the litesearch class method to add fields to the schema:

  class Prompt < ApplicationRecord
    include AccountScoped
+   include Litesearch::Model

    # ...

+   litesearch do |schema|
+     schema.fields [:title]
+   end

    # ...
  end

You can also target associations like so, and change the tokenizer used for indexing:

litesearch do |schema|
  schema.field :account_name, target: "accounts.name"
  schema.tokenizer :porter
end

Note: Currently, ActionText fields are not supported.

Let's quickly try this out in the Rails console:

> Current.account = Account.first
> Prompt.search("teddy")
  Prompt Load (7.4ms)  SELECT prompts.*, -prompts_search_idx.rank AS search_rank FROM "prompts" INNER JOIN prompts_search_idx ON prompts.id = prompts_search_idx.rowid AND rank != 0 AND prompts_search_idx MATCH 'teddy' WHERE "prompts"."account_id" = ? ORDER BY prompts_search_idx.rank  [["account_id", 1]]
=>
[#<Prompt:0x0000000105f80fb0
  id: 1,
  title: "A cute teddy bear",
  prompt_image: <...>,
  account_id: 1,
  created_at: Fri, 12 Jan 2024 10:47:08.604031000 UTC +00:00,
  updated_at: Fri, 12 Jan 2024 10:47:41.321896000 UTC +00:00,
  content_type: "image/jpeg",
  search_rank: 1.0e-06>]

Remember to set Current.account, because our prompt model is scoped to an account, otherwise we get an empty result set.

Impressive! By changing only 4 lines of code, we already have a crude working version of full-text search.

Add a Typeahead Search Bar to Our Ruby on Rails Application

Next up, we'll combine a few of the techniques we've reviewed to implement snappy typeahead searching. Before we do that, though, let's generate more sample data. I will use the popular faker gem to do that:

$ bundle add faker --group development
$ bin/rails console

Drop into a Rails console and create 50 sample prompts. I'm re-using the first prompt's image data here. Also, note that I'm again setting the Current.account first.

> Current.account = Account.first
* 50.times do
*   Prompt.create(title: "a #{Faker::Adjective.positive} #{Faker::Creature::Animal.name}", content_type: "image/png", prompt_image: Prompt.first.prompt_image)
> end

To prepare our user interface for reactive searching, we will wrap the prompts grid in a Turbo frame. This frame will be replaced every time the search query changes.

  <!-- app/views/prompts/index.html.erb -->
  <h1>Prompts</h1>

+ <%= turbo_frame_tag :prompts, class: "grid" do %>
- <div id="prompts" class="grid">
    <% @prompts.each do |prompt| %>
      <%= link_to prompt do %>
        <%= render "index", prompt: prompt %>
      <% end %>
    <% end %>
+ <% end %>
- </div>

  <%= link_to "New prompt", new_prompt_path %>

The PromptsController needs to be updated to filter prompts if a query parameter is passed in:

  # app/controllers/prompts_controller.rb
  class PromptsController < ApplicationController
    # ...

    def index
      @prompts = Prompt.all
+
+     @prompts = @prompts.search(params[:query]) if params[:query].present?
    end

    # ...
  end

Next, let's rig up the search bar in the prompt index view. For this, we'll use a shoelace input component:

  <!-- app/views/prompts/index.html.erb -->
  <h1>Prompts</h1>

+ <section>
+   <sl-input name="search" type="search" placeholder="Search for a prompt title" clearable>
+     <sl-icon name="search" slot="suffix"></sl-icon>
+   </sl-input>
+ </section>

  <div id="prompts" class="grid">
    <% @prompts.each do |prompt| %>
      <%= link_to prompt do %>
        <%= render "index", prompt: prompt %>
      <% end %>
    <% end %>
  </div>

  <%= link_to "New prompt", new_prompt_path %>

To implement typeahead searching, we must add a bit of custom JavaScript to app/javascript/application.js:

  // app/javascript/application.js

  // Entry point for the build script in your package.json
  import "@hotwired/turbo-rails";
  import "./controllers";
  import "trix";
  import "@rails/actiontext";

  import { setBasePath } from "@shoelace-style/shoelace";

  setBasePath("/");
+
+ document
+   .querySelector("sl-input[name=search]")
+   .addEventListener("keyup", (event) => {
+     document.querySelector(
+       "#prompts"
+     ).src = `/prompts?query=${encodeURIComponent(event.target.value)}`;
+   });

This tiny JavaScript snippet does little more than place a keyup listener on our search field, and update the Turbo Frame's src attribute afterward. The input's value is added as the query parameter. Turbo Frame's default behavior performs the rest of the magic: reloading when the src attribute changes, with the updated content fetched from the server.

Here's what this looks like:

Excursus: Highlighting Search Results Using a Turbo Event in Rails

Currently, Litesearch doesn't feature a native highlighting solution like pg_search, but it is pretty easy to build this ourselves using the before-frame-render event:

  // Entry point for the build script in your package.json
  import "@hotwired/turbo-rails";
  import "./controllers";
  import "trix";
  import "@rails/actiontext";

  import { setBasePath } from "@shoelace-style/shoelace";

  setBasePath("/");

  document
    .querySelector("sl-input[name=search]")
    .addEventListener("keyup", (event) => {
      document.querySelector(
        "#prompts"
      ).src = `/prompts?query=${encodeURIComponent(event.target.value)}`;
    });

+ document
+   .querySelector("turbo-frame#prompts")
+   .addEventListener("turbo:before-frame-render", (event) => {
+     event.preventDefault();
+
+     const newHTML = event.detail.newFrame.innerHTML;
+
+     const query = document.querySelector("sl-input[name=search]").value;
+     if (!!query) {
+       event.detail.newFrame.innerHTML = newHTML.replace(
+         new RegExp(`(${query})`, "ig"),
+         "<em>$1</em>"
+       );
+     }
+
+     event.detail.resume();
+   });

This leverages a nifty, somewhat hidden Turbo feature: intercepting rendering. The Turbo before-render and before-frame-render events support pausing rendering and mangling returned HTML from the server. Here, we use this to wrap each occurrence of a search query in an  tag:

Under the Hood: Litesearch for Ruby on Rails Explained

We've covered the basics of activating and configuring Litesearch for your LiteStack-powered Ruby on Rails application. As you might have guessed, there's a lot more potential hidden here.

So let's briefly examine how Litesearch wraps around and leverages SQLite's built-in full-text search module, FTS5.

Virtual Tables in SQLite

First, let's discuss the notion of virtual tables in SQLite. Since there's no direct counterpart in the PostgreSQL or MySQL realm, it pays off to learn about these.

From the vantage point of a user issuing an SQL statement against the database, a virtual table is a transparent proxy that adheres to the interface of a table. In the background, however, every query or manipulation invokes a callback of the virtual table structure instead of writing to disk.

In short, a virtual table is something you reach for when you want to access "foreign" data without leaving the domain of your database connection. Apart from full-text search, other examples include geospatial indices or accessing a different file format, such as CSV.

SQLite's FTS5 Full-Text Search Extension

At its core, SQLite's full-text search engine is a virtual table.

The table definition used by Litesearch in ActiveRecord mode looks like this:

"CREATE VIRTUAL TABLE #{name} USING FTS5(#{col_names}, content='', contentless_delete=1, tokenize='#{tokenizer_sql}')"

name is the index name (it defaults to "#{table_name}_search_idx"), and col_names are the fields we set in our Litesearch schema definition.

We will now briefly look at tokenizers.

Tokenizers

To allow for efficient indexing, a full-text search engine employs a helper utility to split the payload into tokens: a tokenizer. FTS5 has three built-in tokenizers you can choose from:

unicode61 (default): All punctuation and whitespace characters (i.e. ",", "." etc.) are considered separators. Text is split at those characters, and the resulting list of connected characters (usually, words) are the tokens. In the wild, you might encounter the remove_diacritics option. This option specifies how to treat glyphs added to letters, like "á", "à", etc. The default is to remove these "diacritics", so these characters are regarded as equivalent.
ascii: Similar to unicode61, but all non-ASCII characters are always considered token characters. There is no remove_diacritics option.
porter: A tokenizer that employs porter stemming for tokenization. This essentially means that you can do similarity searches, i.e., "search", "searchable", and "searching" will be considered related.

FTS5 Search Interface

To enable a convenient experience, Litesearch exposes a search class method. Essentially, this method joins the model's table to the associated search index and issues a MATCH query. Results are then ordered according to the search rank and returned:

def search(term)
  self.select(
    "#{table_name}.*"
  ).joins(
    "INNER JOIN #{index_name} ON #{table_name}.id = #{index_name}.rowid AND rank != 0 AND #{index_name} MATCH ", Arel.sql("'#{term}'")
  ).select(
    "-#{index_name}.rank AS search_rank"
  ).order(
    Arel.sql("#{index_name}.rank")
  )
end

Currently, Litesearch doesn't expose more of FTS5's search syntax, but you can learn more about it in FTS5's documentation.

Wrapping Up

This concludes our series on LiteStack. In this post, we discovered Litesearch, the full-text search engine built into LiteStack. We learned how to configure an ActiveRecord model to expose search fields and other options to an SQLite text search index.

We then flexed our Hotwire muscles to build a simple reactive search interface into our UI.

Finally, we explored some of the inner workings of full-text search in SQLite to get a better understanding of what powers it, its benefits, and its limitations.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Deep Dive Into RSpec Tests in Ruby on Rails

Roy Tomeij — 2024-02-07T00:00:00+00:00

In our last post, we looked at the basics of RSpec and explored how well it works with Behavior Driven Development in Ruby.

Now, we will look at specific types of RSpec tests for different parts of a Ruby on Rails application.

Let's dive straight in!

Unit, Controller, and Integration Tests in Ruby on Rails

A Ruby on Rails application is composed of several layers. As the framework is built around Models, Views, and Controllers, we might think of those three as the only layers of an application. Yet, often, that's barely enough to describe a Ruby on Rails application.

Mailers, Jobs, and Helpers are secondary layers we don't want to miss. When it comes to testing, it's important to remember that these components are largely impacted by how our code has been designed. Using SOLID principles — particularly Single Responsibility — helps keep our code straightforward and more testable.

So, we usually aim to keep things simple in views, controllers, models, jobs, and mailers: we want to assemble and use the bricks of code we have designed and tested separately already. Once we enter the realm of views and controllers, testing becomes much more complex, and slower too.

Categories of Ruby Tests

Usually, we split tests into these categories:

Unit tests (models and other plain old Ruby objects — POROs): Ensure class methods are working as expected.
Controller tests: Test the expected outcome of controller actions (rendering the right template, redirections, and flash messages).
View tests: (Used rarely) to test specific elements or content in views.
Helper tests: For when you use a complex helper method.
Feature or system tests: Simulate user interactions in the browser — this is where Capybara will help.
Request tests: Test an application's response to various HTTP verbs without browser overhead. This is faster than feature tests.
Mailer and jobs tests: Specific tests are needed for this secondary layer to ensure emails are sent properly with the right content and jobs trigger the appropriate work.
Routing tests: Complex routing is a code smell. Testing routes will help you avoid trouble when you have too many routes or routes that are too complex.

What Kinds of Tests Should We Write?

Do we need to write all of these tests? It depends on your culture and team, but these might be good starting points:

Unit tests: Write focused and (as much as possible) database-dependent tests for your models' public methods and other POROs you craft into your application; they should run fast.
Controller tests: Write tests that help ensure controller actions are routable and respond as expected for each context they could be in. These should not aim to test the whole response or complex scenarios made of multiple requests.
Request tests: Test request scenarios from a machine client (think API controllers), typically: "for a given HTTP request context (HTTP verb, path, and parameters), what HTTP response should we get?"
System tests: Test scenarios of human user requests. Typically, this is where we want to test complex scenarios with multiple clicks.
Mailer and job tests: These should only test that a mailer and job are doing the specific "mailer" and "job" work properly. In particular, for jobs, most of the "work" should be done through classes and methods tested separately.

Controller tests are often replaced by a good layer of request and system tests, as they ultimately serve the same purpose. Too many tests will cause fatigue in your team or slow down your whole test suite for little benefit. It's up to you. The point is to test your controller layer: figure out which way is best in your case.

Let's see how unit, controller, and integration (both request and system) tests look.

Using RSpec and Ruby on Rails for Testing

As pointed out in our previous post, we can use the rspec-rails gem to integrate RSpec into a Ruby on Rails application's code base.

Unit Tests

Unit tests are at the base of the testing pyramid, so there will be numerous unit tests in a codebase. They need to be fast.

Remember: tests (with RSpec) are focused on testing code behavior, not implementation. That should help to write fast tests. In the case of Ruby on Rails projects, models are a big source of unit tests. As models are related to database access (either read or write access), those tests can have a performance impact. So be careful about reading and writing to the database for those tests. In some cases, you won't be able to avoid calling create, save, update, or find, but you can in most. You should be able to rely on new instead of create, for example.

Here is what an RSpec unit test for a Ruby on Rails model looks like:

# spec/models/user.rb
require 'rails_helper' # a file generated with rspec-rails containing configuration for rspec in a rails context

RSpec.describe User, type: :model do
  describe '#valid_email?' do
    subject(:user) { User.new(email: email) }

    context 'when email is valid' do
      let(:email) { Faker::Internet.email }

      it { expect(user.valid_email?).to be(true) }
    end

    context 'when email is not valid' do
      let(:email) { 'bob@example' }

      it 'returns false' do
        expect(user.valid_email?).to be(false) }
      end
    end
  end
end

While those two examples are written a bit differently, they basically do the same thing. One has no description and reads pretty nicely still; the other one has a description but carries a bit of a duplication, don't you think?

Note that we use User.new to instantiate a user. This call doesn't require a read or write to the database. Since our valid_email? method only works with the instance's attributes, it won't slow the test down either.

Those two are good unit tests: simple, focused, and fast.

Unit tests are not just for Ruby on Rails models. They are also good for any classes you build around models and the rest of an application's architecture. We have specified the test type in the first line (RSpec.describe User, type: :model), but we can totally avoid that if we write tests for a plain Ruby class.

Controller and Request Specs

Up until recently, controller tests were the main way to test controllers. Nowadays, we tend to rely on request specs instead. Similarly to controller tests, they are designed to test an application from a machine client. They allow us to test one, or multiple, controller action/s. They are tests, so we need to define a context, then our expectation. As they are functional tests, we have to define a context composed of a given HTTP verb (get, post, put, ...), a path (/, /users, ...), parameters, and (potentially) a body. The expectation is then focused on the response (HTTP status code, response/s header/s, and body).

With request specs, it's not a matter of UI or Javascript.

A simple request spec will look like this.

require "rails_helper"

RSpec.describe "User management", type: :request do
  it "does not render the incorrect template" do
    get "/users/new"
    expect(response).to_not render_template(:show)
  end
end

It's simple, but you can see the context in the part doing the request: get "/users/new". In turn, the expectation itself is centered on the response. We see a new matcher here, allowing us to test if a specific template is rendered. Another one (redirect_to) allows us to test that we are properly redirected.

Request specs can also be used to test more complex scenarios.

require "rails_helper"

RSpec.describe "User management", type: :request do

  it "creates a user and redirects to the user's page" do
    get "/users/new"
    expect(response).to render_template(:new)

    post "/users", params: { user: { first_name: "John", last_name: "Doe", email: "john@example.org" } }

    expect(response).to redirect_to(assigns(:user))
    follow_redirect!

    expect(response).to render_template(:show)
    expect(response.body).to include("User John Doe (john@example.org) created.")
  end
end

However, request specs are better adapted to test behavior from a machine client's perspective. So, request specs are best used to test an API backend's controller parts.

require "rails_helper"

RSpec.describe "User management", type: :request do

  it "creates a user and returns the user's details" do
    headers = { "CONTENT_TYPE" => "application/json" }
    post "/users", params: { user: { first_name: "John", last_name: "Doe", email: "john@example.org" } }

    expect(response.content_type).to eq("application/json; charset=utf-8")
    expect(response).to have_http_status(:created)
    expect(response.body).to include() # TODO
  end
end

While we haven't included any in this example, we can also express different contexts to test behaviors. The RSpec vocabulary and grammar still applies fully in that kind of spec. We merely use a bit more to handle the parts specific to making an HTTP request and its response.

System Specs

System specs are more complete integration tests than request ones. They are used to test an application in a real or headless browser, and require a driver for the browser (the default is Selenium; this can be changed).

System specs are similar to request specs in concept, but with improved grammar to focus on a user's actions in the browser rather than the requests being made. Here is an example.

require "rails_helper"

RSpec.describe "Account management", type: :system do
  it "enables me to create an account" do
    visit "/account/new"

    fill_in "Name", with: "Acm'e Ltd."
    fill_in "Address", with: "6 Av. Elysian fields"
    click_button "Create Account"

    expect(page).to have_text("Account successfully created.")
  end
end

Notice that here, the context (verb, path, parameters) is defined underneath but still there: we visit a page, fill out a form (which sets parameters), and then click a button (trigger a POST request). Finally, the expectation is defined as the content of the page rendered. Indirectly, it's still about testing the response's content, although the browser interprets that.

Because system specs use and drive the complete stack, they are a lot slower than other specs. As such, they should be limited in number and potentially run at particular times within the CI pipeline.

Complimentary Test Types in Ruby

As mentioned, Ruby on Rails applications are composed of a few more components with specific roles: mailers, jobs, serializers, and decorators. Those require tests as well — ones closer to unit than integration tests. It can be useful to test out specifics regarding mailers and jobs.

Mailer Specs

Thanks to ActiveMailer, Ruby on Rails has a simple way to abstract interactions with a third party that sends emails. Thus, instead of testing email sends, you can focus on testing what matters: behavior. In the case of emails, what matters is the subject, from and to addresses, and the body of the response.

require "rails_helper"

RSpec.describe Notifications, type: :mailer do
  describe "notify" do
    let(:mail) { Announcements.account_renewal }

    it "prepares the email headers properly" do
      expect(mail.subject).to eq("Renewal")
      expect(mail.to).to eq(["company@example.org"])
      expect(mail.from).to eq(["bot@example.com"])
    end

    it "renders the body" do
      expect(mail.body.encoded).to match("A customer has renewed their account.")
    end
  end
end

Job Specs

Similarly, job specs allow you to test specific behavior around jobs: if they have been enqueued, performed, etc. This includes testing arguments using Delayed Job.

We can be tempted to test ActiveJob's behavior. We do know that calling perform and perform_later bill queues a job; what we want to know is if, and when, that happens.

So, you should test if a job is queued in contexts requiring it. Then, test that a job properly calls upon code that has its own unit tests.

describe Building do
  subject(:building) { Building.new(name: 'Bank') }

  describe '#setup' do
    it 'triggers the preparation job' do
      ActiveJob::Base.queue_adapter = :test
      expect { building.setup }.to have_enqueued_job.with('setup').on_queue('low')
    end

   # alternative
   it 'triggers the preparation job' do
     ActiveJob::Base.queue_adapter = :test
     expect(Building::PreparationJob).to have_been_enqueued.with('setup').exactly(:once)
   end
  end
end

This expresses and tests a job's enqueued behavior with certain parameters when a specific method is called.

A Note on Concerns in Ruby on Rails

Ruby on Rails has a way to factorize code used in multiple models or controllers through a disguised Ruby module called a concern. Concerns are a great way to avoid duplication of code. Make sure you test the content of concerns. Proper unit tests should also cover models. In the case of controllers, those are tested through request and system tests.

Some Thoughts on Testing with RSpec for Ruby

It's worth reiterating that your first layer of testing should be unit tests. Tests for models and other POROs should represent the vast majority of your tests in a code base.

Tests should run fast and avoid database and external services access as much as possible. To test the controller layer, request and system specs are best, respectively, for machine or human-driven activity on controller actions. They are slower than unit tests due to their inherent complexity, so there should be less of them than unit tests.

Finally, components like mailers and jobs should be used wisely and in their specific context, not to test the behavior of the lower library (like ActiveMailer and ActiveJob).

Wrapping Up

As we saw in part one of this series, we can factorize a lot of code and avoid duplication by making use of RSpec's basics: before and after hooks, let, and a proper structure built with describe and context.

In this part, we took a deep dive into testing with RSpec for Ruby, focusing on unit, controller, and integration tests.

We have explored how it's usually easy to keep unit tests small, while minimizing request and system tests is more difficult. So, after you have written (and made green) system and request specs, don't hesitate to spend some time slimming them down. This will keep them readable and easier to maintain.

Happy testing!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Behaviour Driven Development in Ruby with RSpec

Roy Tomeij — 2024-01-24T00:00:00+00:00

RSpec is a library for writing and running tests in Ruby applications. As its landing page states, RSpec is: "Behaviour Driven Development for Ruby. Making TDD productive and fun". We will return to that last part later.

This post, the first of a two-part series, will focus on introducing RSpec and exploring how RSpec especially helps with Behaviour Driven Development in Ruby.

Let's dive in!

The History of RSpec

RSpec started in 2005 and, through several iterations, reached version 3.0 in 2014. Version 3.12 has been available for almost a year. Its original author was Steven Baker, but the mantle has passed to several maintainers through the years: David Chelimsky, Myron Marston, Jon Rowe, and Penelope Phippen. Many contributors have also been part of the project.

When you read RSpec's good practices, you see this phrase early on: "focus on testing the behavior, not the implementation". That's not a strategy specific to RSpec; it's good advice for anyone writing tests. In more practical terms, you should focus your tests on how the code you are testing behaves, not how it works.

For example, to test that a user's email address is valid, you should test that the validate_email method returns false when the email address is invalid. You are not testing a specific implementation but rather how that code reacts (i.e., behaves) when handling different strings that should be email addresses.

Behavior interests us: how the code acts and defines how our application will work. Furthermore, this approach simply lets us know whether things work (or not), and we can then be more relaxed to either fix or refactor the implementation. Our tests will tell us if the code's behavior has changed; we will know if our changes have been successful or not in the most direct way possible.

The inner workings of the code don't interest us so much. Of course, we don't want a bad implementation, but measuring a good implementation is much harder than knowing if the code does what it's expected to do or not.

Let's look at how to install RSpec next.

Installing RSpec for Ruby

RSpec comes as a gem, rspec-core, so you can add it to a test group in your Gemfile. It's probably best you also add rspec. It's a meta gem that includes rspec-core, rspec-expectations, and rspec-mocks. You can also add the rspec-rails one in a Ruby on Rails project.

Tests usually live in the spec/ folder at the root of your project, and you can launch them by providing a path to a file or a directory: rspec spec/models/*rb, for example.

Now let's turn to how the RSpec DSL is set up to help with behavior testing.

RSpec DSL: How it Helps with Testing Behavior

RSpec's whole Domain Specific Language (DSL) is completely worked around behavior testing, giving you a direct way to describe the behavior you expect from your code within different contexts. A few parts could be smoother, but overall, tests in RSpec read directly as English, much like a good piece of Ruby code.

Tests in RSpec are not written as classes, with methods taking center stage as tests. Instead, tests are written as Ruby blocks (ever used do .. end?), which, thanks to the method name we pass to the block as an argument, makes things very easy to read.

The primary method used in RSpec tests is describe. describe will contain one or more tests and can even contain more describe calls. The second method is it. The it blocks are called examples and contain the actual assumptions; they are where the testing happens. Finally, RSpec relies on "expectations" within the it blocks. Using the expect method, we define how the subject of our test is expected to behave.

Now we can start writing some simple tests.

Simplest Tests in RSpec for Ruby: `describe` and `it`

Let's imagine a User class. We want a name method that will output first and last names together if both are present (or just one, if one is missing). Those are the different behaviors we want to test.

Here is how the simplest of those contexts look expressed as an RSpec test.

# first call to describe, as topmost one, its description or title is used to tell what we are testing, here the User class.
# this title can be a string or a class name
describe User do

  # we are adding a second describe to regroup the tests focused on the `name` method
  describe '#name' do

    # our first example ! Note the description focusing on the behavior
    it 'returns the complete name' do

      # we define a user variable by instantiating a user
      user = User.new(first_name: 'John', last_name: 'Doe')

      # and here comes the expectation
      expect(user.name).to eq('John Doe')
    end
  end
end

Look at the general structure: we start by describing the focus of our test in the User class, the method we are testing (name), and then present an expected behavior.

Note how the expectation is written: we expect the value returned by user.name to equal 'John Doe'. The eq method is a matcher. It allows us to match the tested value (on the left) and the expected one (on the right). The expect part is always followed with to or not_to to dictate how the matcher that follows will be used.

Handling Multiple Contexts

While this first test shows us how it's done, it needs to catch up to what we want. It only handles one case if both first and last names are present. Let's see how we can test another case if the first name is absent.

describe User do

  describe '#name' do
    it 'returns the complete name when both first and last name are present' do
      user = User.new(first_name: 'John', last_name: 'Doe')
      expect(user.name).to eq('John Doe')
    end

    it 'returns only the last name when the first name is missing' do
      user = User.new(last_name: 'Doe')
      expect(user.name).to eq('Doe')
    end
  end
end

This looks more realistic, but we still need to test another case. The description is also relatively verbose and repetitive. We could add another layer of description between the describe '.name' call and the example for each one. Thankfully, though, RSpec gives us a more obvious synonym for describe to express what we need to express for different contexts: context.

describe User do

  describe '#name' do
    context 'when both first and last name are present' do
      it 'returns the complete name' do
        user = User.new(first_name: 'John', last_name: 'Doe')
        expect(user.name).to eq('John Doe')
      end
    end

    context 'when the first name is missing' do
      it 'returns only the last name' do
        user = User.new(last_name: 'Doe')
        expect(user.name).to eq('Doe')
      end
    end
  end
end

Thanks to this, the whole file reads even more quickly and gives us, without much thinking, an understanding of exactly which behavior we are testing within different contexts.

Defining the Subject of Tests

We can use the subject method to make the subject of our tests obvious.

describe User do

  describe '#name' do
    subject { user.name }
    context 'when both first and last name are present' do
      let(:user) { User.new(first_name: 'John', last_name: 'Doe') }

      it 'returns the complete name' do
        expect(subject).to eq('John Doe')
      end
    end

    context 'when only the first name is present' do
      let(:user) { User.new(first_name: 'John') }

      it 'returns the complete name' do
        expect(subject).to eq('John')
      end
    end
    # ... other contexts
  end
end

This is especially handy to avoid repetition and add clarity.

Handling Complex Setup (Before, After Hooks) in Ruby on Rails

In many cases, we need a bit more to prepare a context. Let's take, as an example, a class method on a Ruby on Rails model named latest_three. It's expected to return the last three users created in the database. If we have less than that, we should get whatever users we have. By omitting the topmost describe, here is how a test might look.

# note that we are using the '::method_name' here to refer to a class method, '#method_name' is reserved to refer to an instance method's name
describe '::latest_three' do
  context 'when more than three users are present' do
    it 'returns three users' do
      3.times { User.create(first_name: Faker::Name.first_name, last_name: Faker::Name.last_name) }

      expect(User.latest_three.size).to eq(3)
    end
  end

  context 'when no users are present' to
    it 'returns an empty collection' do
      expect(User.latest_three.empty?).to be(true)
    end
  end
end

If you are unfamiliar with Faker, it's a Ruby library used to generate fake data such as names and dates through handy methods.

These two tests look ok, but the creation of the data doesn't belong to the example. It's important for the specific context, though: we need that data created before the example is run. To do so, we can use a before block. Those blocks are run before the tests that follow them (in each block's context), thus giving us a perfect opportunity to set up our data.

describe '::latest_three' do
  context 'when more than three users are present' do
    before do
      4.times { User.create(first_name: Faker::Name.first_name, last_name: Faker::Name.last_name) }
    end

    it 'returns three users' do
      expect(User.latest_three.size).to eq(3)
    end
  end

  context 'when no users are present' to
    it 'returns an empty collection' do
      expect(User.latest_three.empty?).to be(true)
    end
  end
end

Once again, I hope this shows how well thought-through the RSpec DSL is. Doesn't it read nicely and give us a good understanding of each context and the behavior we expect?

If we were tempted to destroy data or execute some other form of cleanup after a context, we could do so through an after block. This is especially useful if you are writing tests using a database without the comfort of built-in automatic database cleanup between test runs.

Avoiding Repetition with `let` in RSpec for Ruby

We still lack a few more concepts to be able to write real-world tests. Let's take the case of email validation again.

describe '#valid_email?' do
  context 'when email does not contain an @' do
    subject(:user) { User.new(email: 'bob') }

    it { expect(user.valid_email?).to be(false) }
  end

  context 'when email does not have a tld' do
    subject(:user) { User.new(email: 'bob@appsignal') }

    it { expect(user.valid_email?).to be(false) }
  end

  context 'when email is valid' do
    subject(:user) { User.new(email: 'bob@example.org') }

    it { expect(user.valid_email?).to be(true) }
  end
end

There are several repetitions, to the point of blurring the actual tests. We notice that the only thing that changes, in the setup of each context, is the actual value of the email address. Couldn't we use a variable to do this?

We can use let blocks. They allow us to define a memoized helper method. The value is cached across multiple calls in the relative context. let's syntax is similar to the one we saw for the subject block: first, we pass a name for the helper, then a block to be evaluated. That block is lazy-evaluated. If we don't call it, it will never be evaluated.

describe '#valid_email?' do
  subject(:user) { User.new(email: email) }

  context 'when email does not contain an @' do
    let(:email) { 'bob' }

    it { expect(user.valid_email?).to be(false) }
  end

  context 'when email does not have a tld' do
    let(:email) { 'bob@appsignal' }

    it { expect(user.valid_email?).to be(false) }
  end

  context 'when email is valid' do
    let(:email) { 'bob@example.org' }

    it { expect(user.valid_email?).to be(true) }
  end
end

Note that the subject is moved up in the structure too: it will be evaluated within each context and thus use each context's email value. Here we can see the purpose of subject within a describe with multiple contexts: we define the subject of the test early to make it obvious. We can then focus on expressing each different context we want to check the subject behavior in.

A Note on `let`

let is lazily defined. In the above example, email won't be instantiated and set until it's called. Once it is invoked, though, it's set. In effect, it's just like a memoized helper method.

Yet, in some cases, you might want to set the value associated with a let before the examples run. To do so, you can use let!. With let!, the defined memoized helper method is called within an implicit before hook for each example. In other words, the value associated with the let! is eagerly defined before the example is run.

Let's create a user in our context before we run our example:

describe "#count_users" do
  let(:account) { Account.create(name: 'Acc Ltd') }
  let!(:user) { User.create(name: 'Jane', account: account) }

  it "counts the users in the account" do
    expect(account.count_users).to eq(1)
  end
end

This prevents us from additional setup or even a call to user within the before hook to get the value memoized.

`let` Vs Instance Variables in RSpec for Ruby

Some developers might be tempted to rely on instance variables through a describe or context and their before hooks:

describe "#count_users" do
  before do
    @account = Account.create(name: 'Acc Ltd')
    @user = User.create(name: 'Jane', account: @account)
  end

  it "counts the users in the account" do
    expect(@account.count_users).to eq(1)
  end
end

This is not very practical. It adds dependencies and state sharing between contexts, weakens isolation, and is more difficult to debug.

The additional issue is that, if you were to make a call to an instance variable that has not been initialized, you'd get a nil value in return. That's in contrast to the exception you'd get if you were to call a local variable that doesn't exist (raising a NameError exception).

So, when writing tests with RSpec, let is preferred, and let! is to be used when you need an eager evaluation. Other methods to handle variables are not recommended.

Matchers in RSpec for Ruby

If describe, context, and it are very important to the structure of RSpec tests, the key part to making actual tests is matchers.

We have only seen a few, mainly be() and eq(). Those two are the simplest ones and are very handy. Here is a list of the others you should know about as a start:

eq: test the equality of two objects (actually, their equivalence, the same as ==); expect(1).to eq(1.0) # is true
eql: test the equality of two objects (if they are identical, not just equivalent); expect(1).to eql(1.0) # is false
be: test for object identity; be(true), be(false) ...
be_nil: test if an object is nil
be <= X: test if a number is less or equal to a value (X); also works with <, >, >=, ==
be_instance_of: test if an object is an instance of a specific class; expect(user.name).to be_instance_of(String)
include: test if an object is part of a collection; expect(['a', 'b']).to include('a')
be_empty: test if a collection is empty; expect([]).to be_empty
start_with, end_with: test if a string or array starts (or ends) with the expected elements; expect('Brian is in the kitchen').to start_with('Brian'), expect([1, 2]).not_to start_with('0')
match: test if a string matches a regular expression; expect(user.name).to match(/[a-zA-Z0-9]*/)
respond_to: test if an object responds to a particular method; expect(user).to respond_to(:name)
have_attributes: test if an object has a specific attribute; expect(user).to have_attributes(age: 42)
have_key: test if a key is present within a hash; expect({ a: 1, b: 2 }).to have_key(:a)
raise_error: test if a block of code raises an error; expect { user.name }.to raise_error(ArgumentError)
change: test that a block of code changes the value of an object or one of its attributes; expect { User.create }.to change(User.count).by(1), expect { user.activate! }.to change(user, :is_active).from(false).to(true)
to_all: test that all items in a collection match a given matcher; expect([nil, nil, nil]).to all(be_nil)
match_array: test that one array has the same items as the expected one (the order isn't of importance); expect([1, 3, 2]).to match_array([2, 1, 3])

You can already write most of the tests you'll ever need with those matchers. To read more about matchers, you can check out RSpec's documentation.

A Few Thoughts

As you have seen, we have yet to write a line of actual code; we just wrote tests. That might be the most crucial point of this article: RSpec's DSL and structure allow you to write your test first from the behavior point of view. When you start to work on a new class, you can first express the behavior as an RSpec example within a given context. Then, simply rely on the guard rails to make your implementation a reality.

That's actually how TDD works. We are not writing tests just for the sake of tests. Instead, we write tests to express the behavior we want to see from the code. In effect, those tests are merely a transcription (through RSpec DSL) of the behavior expected for a feature.

Wrapping Up

To summarize what we have covered in this article:

RSpec is a library that gives us a powerful DSL to express and test the behavior of code
describe is the main element to structure tests in each file
context is equivalent to describe, but is used to separate different contexts for testing code behavior
it allows us to define examples: the blocks within which tests happen
expectations define the actual tests with matchers
let and let! allow us to define memoized helpers through custom-named blocks to avoid repetitions; let! is eagerly loaded
subject allows us to clearly define what is being tested and can be named

In the next post, we will look at specific types of tests for different parts of a Ruby on Rails application.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Speed Up Your Ruby on Rails Application with LiteCache

Roy Tomeij — 2024-01-17T00:00:00+00:00

In this series, we have looked at the "musts" (databases) and "shoulds" (asynchronous jobs, websockets) of a web application. Now we turn to one of the "coulds" (that is nonetheless recommended for scaling businesses): caching. In particular, we mean caching HTML fragments and other snippets of data, as referred to in Rails Guides. We are not concerned with HTTP or SQL query caching.

In this part, we'll see how to speed up our Rails app using LiteCache. But first, we'll touch on Russian doll caching and how it comes in handy.

Russian Doll Caching in Ruby on Rails

At first glance, it might seem counterintuitive to employ the same technology for the main database as well as the cache. After all, what speed improvements will that engender? This is where a technique called "Russian doll caching" comes in, which we will briefly shine a light on now.

In one sentence, the gist of this caching approach boils down to:

The fastest database query is no query.

What do I mean by that? Let's turn to the Rails Guides definition of fragment caching first:

Fragment Caching allows a fragment of view logic to be wrapped in a cache block and served out of the cache store when the next request comes in.

In other words, when you wrap a piece of view code in a cache helper, the rendered HTML fragment is put into the cache store, with a unique, expirable cache key. This key is constructed to expire whenever either the entities of which the cache key is composed, or the underlying view template, change.

An Example

That may sound very unwieldy. Let's look at an example in the context of our app:

  <!-- app/views/prediction/_prediction.html.erb -->

+ <% cache prediction do %>
    <div id="<%= dom_id(prediction) %>">
      <%= turbo_stream_from prediction %>

      <% if prediction.prediction_image.present? %>
        <%= image_tag prediction.data_url %>
      <% else %>
        <sl-spinner style="font-size: 8rem;"></sl-spinner>
      <% end %>
    </div>
+ <% end %>

Here, we have wrapped our _prediction partial in a cache block. This generates a cache key in the fashion of:

view path                     template digest                  identifier
|                             |                                |
views/predictions/_prediction:9695008de61cf58325bbf974443f54bc/predictions/3

As we can see, this key comprises:

The view path, containing the cache block.
A digest of the template (i.e., if you change the partial, the cache entry will be invalidated).
A unique identifier — in other words, our prediction — see the ActiveRecord documentation.

Stored with this key is a timestamp (the updated_at column of our prediction) and the HTML fragment, in our case also the complete image's data URL. Whenever this partial is rendered again, and a matching cache key is found, rendering is bypassed. Instead, the stored HTML is returned.

Making Use of Russian Doll Caching

What's that "Russian doll" piece about, though? To answer this, let's jump a layer higher into a view that renders this _prediction partial:

  <!-- app/views/prompts/_prompt.html.erb -->
+ <% cache prompt do %>
    <sl-card class="card-header card-prompt" id="<%= dom_id prompt %>">
      <div slot="header">
        <!-- header content omitted -->
      </div>

      <%= turbo_stream_from :predictions %>
      <div class="grid">
        <div>
          <%= image_tag prompt.data_url %>
        </div>
        <div id="<%= dom_id(prompt, :predictions) %>">
          <%= render prompt.predictions %>
        </div>
      </div>

      <div slot="footer">
        <%= prompt.description.presence || "No Description" %>
      </div>
    </sl-card>
+ <% end %>

We can apply the same technique to the _prompt partial, which, in turn, renders a _prediction partial for every prediction associated with it. This will result in a single HTML fragment comprising all the child fragments. We just saved one SQL query for each prediction!

There's a catch, though: When the list of predictions associated with a prompt changes (for example, a new one is added), the top fragment doesn't know about this and will serve stale content (without the newly added image).

In other words, we have to expire its cache key and construct a fresh fragment. This is where the timestamp stored with each cache entry comes in handy. We can invalidate the cache simply by updating this timestamp. In ActiveRecord, luckily, there's a shorthand for this:

  # app/models/prediction.rb
  class Prediction < ApplicationRecord
    # callbacks omitted

-   belongs_to :prompt
+   belongs_to :prompt, touch: true

    # methods omitted
  end

Using the touch flag on the belongs_to association, every time a child record (a prediction) is updated, deleted, or added, the updated_at timestamp of the parent (a prompt) is refreshed. This marks the cache fragment as stale, and it is reconstructed upon the next render cycle.

The term "Russian doll caching" in this context refers to the fact that all the valid child fragments can still be pulled from the cache, thus speeding up the rendering process.

Now that we have reviewed how fragment caching works and what benefits it yields, let's discuss how to enable and configure LiteCache.

LiteCache Configuration in Rails

In config/environments/development.rb, add this configuration snippet:

config.cache_store = :litecache, {
  path: Litesupport.root.join("cache.sqlite3")
}

This will resolve the root database path depending on your environment (in this case, db/development) and create a cache.sqlite3 file there. There are more configuration options worth considering, though:

size: the total allowed size of the cache database (the default being 128MB)
expiry: cache record expiry in days (the default is one month)
mmap_size: how large a portion of the database to hold in memory (default: 128MB)
min_size: the minimum size of the database's journal (default: 32KB)
sleep_interval: duration of sleep between cleanup runs (default: one second)
metrics: boolean flag to indicate whether to gather metrics

The ones you will likely want to tweak to your liking are size, expiry, and (potentially) sleep_interval.

Note: To enable caching in development, you have to run:

$ bin/rails dev:cache

Although LiteCache has many benefits, it comes with some drawbacks too. Let's first look at some optimizations, and then a few of LiteCache's limitations.

Optimizations and Limitations of LiteCache for Your Ruby App

LiteCache connects to the SQLite database in a way that's optimized for use as a cache store. First, it's important to reiterate that LiteCache is configured to use a separate cache database, so all these optimizations only affect an isolated environment.

With this disclaimer in place, let's look at how LiteCache optimally utilizes SQLite.

First, LiteCache sets the pragma statement synchronous to 0, so there is no sync after a commit to the database. This results in a tremendous speedup at the expense of data safety. In very rare cases, such as a power loss or operating system crashes, data loss might occur. However, considering that cache entries are seen as ephemeral in most cases, this is a sensible tradeoff. Needless to say, you can also override this setting in your configuration.

LiteCache also uses a least recently used (LRU) eviction policy with a special index, but it delays updating it. Instead, it will buffer the updates in memory, and flush them as a single transaction every few seconds.

What about limitations? At the time of writing this article, one crucial piece of the ActiveSupport::Cache interface isn't implemented yet: The ability to do multiple reads, writes, and deletes against the cache database. Why is that so important? Because other cache backends like Redis demonstrate how significant speedups can be achieved by batching these operations. Indeed, the implicit performance gain built into rendering a collection of partials is the most impressive feat of Rails fragment caching.

Speaking of performance speed, let's now quickly look at some benchmarks before we wrap up.

Benchmarks: LiteCache Vs. Redis

The benchmarks for LiteCache are impressive, with a small caveat. While LiteCache outperforms a local Redis installation for every read operation, it seems like there's still room for improvement, especially for large write payloads.

Considering the increased benefits of caching large HTML fragments, this is a worthwhile limitation that will hopefully be tackled in the future.

Up Next: Built-In Full-Text Search with LiteSearch

In this post, we illuminated Russian doll caching as a technique to speed up Rails applications by avoiding unnecessary database calls.

Through practical examples, we’ve seen how nested cache fragments operate in harmony — each layer independent, yet interconnected — thus ensuring efficient rendering.

We also delved into the practicalities of configuring LiteCache to your liking and looked at important built-in optimizations. With that in mind, the missing support for multiple read and write operations is bearable, and perhaps is a feature that might soon arrive on the horizon.

In the next and final post of this series, we will take a tour through the latest addition to LiteStack: A SQLite-based full-text search engine called LiteSearch.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Keep your Ruby Code Maintainable with Draper

Roy Tomeij — 2024-01-10T00:00:00+00:00

Design patterns can help to simplify your codebase so you don't need to reinvent the wheel.

In this post, we'll go into how to use Draper. But first, we will start with an overview of the decorator pattern and how to use it with Ruby's standard library.

Let's get started!

The Decorator Pattern for Ruby

According to Refactoring.Guru:

Decorator is a structural design pattern that lets you attach new behaviors to objects by placing these objects inside special wrapper objects that contain the behaviors.

Decorator can also be called a wrapper. The idea is that you initialize an instance of a decorator class by passing the object you want to decorate to that decorator class initializer.

The new returned object can expose the same methods the original object did, plus a few more, or alter the original ones. In most cases (and in Draper's case), we use this pattern to delegate calls to the embedded object.

Let's dig into the Ruby standard library to look at the SimpleDelegator class, itself a child of the Delegator class. It's a great way to see the decorator's side of the pattern.

class Company
  def founded_on
    Date.new(1981, 9, 10)
  end
end

class CompanyDecorator < SimpleDelegator
  def founding_year
    founded_on.year
  end
end

decorated_company = CompanyDecorator.new(Company.new)
decorated_company.founding_year  #=> 1981
decorated_company.__getobj__  #=> #<Company: ...>

One thing to note: the decorated_company object has direct access to the public methods of the object it decorates.

And here lies the principle behind the decorator: the founded_on method, which can be an attribute pulled from the database, provides a key value for the company.

To display the year the company was founded, we might be tempted to use company.founded_on.year. This can work, yet we will slowly riddle the codebase with such calls. That is not practical; if we were to change the math or presentation of that value, we would need to change every single instance of that piece of code.

This kind of presentation code doesn't belong in the Company class; in Ruby on Rails, it doesn't belong in models either. If you have seen a few Rails projects, you probably have seen views littered with such presentation code. Decorators, delegators, or presenters are a better place for such presentation logic.

Using Decorators in Ruby on Rails Views, Models, and Controllers

Most, if not all, web applications need to display attributes of objects in different ways: names, dates, times, prices, numerical values, and coordinates. These uses are all over the place. Sometimes, we copy this kind of code in every view, use helper methods, or fill the model with methods that are just presenting data. Delegators are handy in moving all this logic into focused places.

Helper methods are usually spread around in different places. They might be grouped, but only carry a little meaning beyond the simple method's name. While models sound more appropriate, adding presentation methods tends to fatten them for little purpose. Any business logic in models or ActiveRecord-related code (associations, scopes, etc.) will be muddied by the presence of those methods.

In short, in Ruby on Rails, decorators allow us to keep responsibilities limited in the model, the view, the decorator, and the controller. Here is how it could work (note that we have simplified the code a bit to stay concise):

# app/models/company.rb
class Company < ApplicationRecord
  def founded_on
    Date.new(1981, 9, 10)
  end
end

# app/decorators/company_decorator.rb
class CompanyDecorator < SimpleDelegator
  def founding_year
    founded_on.year
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def show
    company = Company.find(params[:id])
    decorated_company = CompanyDecorator.new(company)

    render :show, locals: { company: decorated_company }
  end
end

# app/views/companies/show.html.erb

<h1><%= company.name %></h1>
<h2>Founded in <%= company.founding_year %></h2>

Using Draper in Your Ruby App

Draper is a bit more advanced than SimpleDelegator. To install Draper, simply add the gem to the application's Gemfile:

gem 'draper'

And run bundle install.

Decorators are usually written in the app/decorators folder; you can also use namespaces within that folder. Namespaces and the decorator's name should mirror the related models for the best results.

Draper includes a couple of Ruby on Rails generators too. So when generating resources (rails generate resource company), the related decorator will be created automatically.

If you prefer to do it by hand, you can also call the decorator generator with the model's name: rails generate decorator Company. By following this, you can rely on a bit of magic within your code to call decorate on the Company class instance and get the CompanyDecorator instance for that company.

decorated_company = Company.find(params[:id]).decorate
# equivalent to
company = Company.find(params[:id])
decorated_company = CompanyDecorator.new(company)

Draper also works with collections through the decorator's decorate_collection class method or the decorate method on a collection of objects.

decorated_companies = CompanyDecorator.decorate_collection(filtered_companies)
# or
decorated_companies = Company.recent.decorate

The decorator is similar to the one we saw with SimpleDelegator; it relies on Draper::Decorator instead. Methods of the decorated object are not directly accessible. Instead, we must call them through the object or model alias.

class CompanyDecorator < Draper::Decorator
   def founding_year
    object.founded_on.year
  end
end

Then, if we want to directly call public methods of the decorated object with the decorator instance, we can define a list of such methods by using the delegate method.

class CompanyDecorator < Draper::Decorator
  delegate :name

  def founding_year
    object.founded_on.year
  end
end

That way, we can use decorated_company.name. This approach is also handy to limit the exposure of the object that's being decorated from the view.

Example Draper Use Case: Rails View, Controller, and Model

A classic Draper use case relates to a User, Account, or Post model: they are often part of the core domain. The sheer quantity of business logic and presentation logic in those models can cause senior developers to frown and grab a new cup of coffee.

Let's take a User model use case: always a good magnet for plenty of excess fat.

class User < ApplicationRecord
  # table would have the following columns
  #
  # first_name, String
  # last_name, String
  # date_of_birth, DateTime
  # company_id, Uuid
  # street_name, String
  # street_number, String
  # city, String
  # zipcode, String
  # country, String

  def age
    DateTime.current.years_ago(date_of_birth.year).year # Note: this is Ruby on Rails specific
  end

  def company_name
    company.name
  end
end

None of those methods add any value to the model. They are misplaced or purely related to the presentation of data.

The view might look something like the following.

# app/models/users/show.html.erb

<h1><%= "#{user.first_name} #{user.last_name}" %></h1>
<ul>
  <li>Born <%= user.age %> years ago</li>
  <li>Address: <%= "#{user.street_number} #{user.street_name}" %></li>
  <li>City: <%= "#{user.zipcode} #{user.city} - #{user.country}" %></li>
  <li>Company: <%= user.company_name %></li>
</ul>

The controller is something like this:

class UsersController < ApplicationController
  def show
    user = User.find(params[:id])

    render :show, locals: { user: user }
  end
end

Of course, another view, within the cart display (for example), would make use of an address partial:

# app/views/carts/_address.html.erb
<ul>
  <li><%= "#{user.street_number} #{user.street_name}" %>
  <li><%= "#{user.zipcode} #{user.city} - #{user.country}" %></li>
</ul>

A decorator can make things cleaner.

class UserDecorator < Draper::Decorator
  delegate :company_name

  def age
    ((Time.now - object.date_of_birth.to_i) / 60 / 60 / 24 / 365).to_i
  end

  def address_line
    "#{object.street_number} #{object.street_name}"
  end

  def city_line
    "#{object.zipcode} #{object.city} - #{object.country}"
  end

  def name
    "#{object.first_name} #{object.last_name}"
  end
end

The controller then looks like the following:

def show
  user = User.find(params[:id])

  render :show, locals: { user: user.decorate }
end

And the view is simpler.

# app/models/users/show.html.erb

<h1><%= user.name %></h1>
<ul>
  <li>Born <%= user.age %> years ago</li>
  <li>Address: <%= user.street_line %></li>
  <li>City: <%= user.city_line %></li>
  <li>Company: <%= user.company_name %></li>
</ul>

As you can see, it's not so different either.

Decorators: When Not to Use Them in Your Ruby on Rails App

Decorators can be overused and end up poisoning your code base. You might stuff too much in them, causing complex queries and an overhead. Decorators could also end up mixed in with helper methods and undecorated classes. All in all, you may overuse Draper.

Decorators, like any class, need to be focused and stick to one responsibility: presentation logic. If a method doesn't fit, maybe it belongs somewhere else. Complex queries might appear when the company_name method or something similar pulls data from a model and its associates.

A similar issue can emerge when you decorate too many objects at once. After all, you basically double the number of objects if you decorate a whole collection. So, be aware of this and decorate when needed.

Ruby on Rails developers tend to use a lot of helper methods. Some helper methods make sense but lack the object-oriented angle, in my opinion, and end up causing confusion. Relying on decorators will help limit the number of helper methods, but if you do use some, keep it light.

Testing Ruby code is almost a standard, it's definitely a good practice, and it also applies to Decorators. This will help, just like for any other class.

Preparing a Whole View with Decorator

There is a form of Decorator that is a bit more advanced: the view model, which can be done with Draper, SimpleDelegator, or even a plain old Ruby object (PORO). The concept is as follows: you create classes to prepare the data for a view from one or more objects. If we're talking just one object, we have something similar to what we have seen. If it's more than one, then it gets more crunchy.

The idea is to gather the whole responsibility of "data presentation" into one class, even for different objects at once. This is a case where we might cause complex queries, so don't hesitate to use includes and preload methods with queries to eager load associations as needed.

That way, instead of passing several objects to the view from the controller, you can prepare just one, and then access the data — ready to be used — from properly named methods.

Wrapping Up

In this post, we ran through the basics of Decorator and how to use it, before diving into how to use Decorators in Draper for a Ruby on Rails application. We finally touched on when to avoid using decorators and how to prepare a whole view with Decorator.

Draper is one of those gems you'll wish you had discovered before. It gives wings to Decorators. It's also the kind of nicely integrated library that will not only help you keep your code clean by setting clear limits between responsibilities, but also provide you with custom grammar and magic that fits the Ruby on Rails ecosystem like a glove.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

AppSignal’s Top 5 Ruby Posts in 2023

Roy Tomeij — 2023-12-20T00:00:00+00:00

As the year draws to an end, we're excited to share our top five most-read Ruby articles of 2023!

Season's Greetings ❆ ⛄

Have a wonderful festive break, and we'll see you in the new year!

If you haven’t already, don’t forget to subscribe to our Ruby Magic newsletter, so you never miss an upcoming post.

See you in the new year!

Stream Updates to Your Users with LiteCable for Ruby on Rails

Roy Tomeij — 2023-12-13T00:00:00+00:00

So far in this series, we have been exploring the capabilities of SQLite for classic HTTP request/response type usage. In this post, we will push the boundary further by also using SQLite as a Pub/Sub adapter for ActionCable, i.e., WebSockets.

This is no small feat: WebSocket adapters need to handle thousands of concurrent connections performantly. The emergence of alternatives to ActionCable — read AnyCable — bears witness to the fact that this is a pressing concern for modern web applications. We'll take a look at how SQLite performs under these conditions.

But first, let's set up our app to broadcast streaming updates to users via Turbo Streams.

Configure Your Ruby on Rails App to Use LiteCable for Websockets

Configuring your application to use LiteCable for WebSocket connections is as easy as specifying the adapter in config/cable.yml:

development:
  adapter: litecable

test:
  adapter: test

staging:
  adapter: litecable

production:
  adapter: litecable

Preparing Our Rails App for Live Updates

Before we dive deep into how to broadcast model updates using Turbo Rails, we need to rework the mechanics of creating a prediction.

First, we'll create an empty prediction in GenerateImageJob to display a placeholder in our _prompt partial. This has the added benefit of forwarding the actual prediction's SGID to the webhook. Note, though, that we also have to pass the account's SGID, because the incoming webhook doesn't have any session information about the currently active user.

  # app/jobs/generate_image_job.rb

  class GenerateImageJob < ApplicationJob
    include Rails.application.routes.url_helpers

    queue_as :default

    def perform(prompt:)
+     empty_prediction = prompt.predictions.create

      model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
      version = model.latest_version
      version.predict({prompt: prompt.title, image: prompt.data_url},
        replicate_rails_url(host: Rails.application.config.action_mailer.default_url_options[:host],
-         params: {sgid: prompt.to_sgid.to_s}))
+         params: {prediction: empty_prediction.to_sgid.to_s,
+         account: prompt.account.to_sgid.to_s}))
    end
  end

In parallel, in our ReplicateWebhook, we can locate and simply update the prediction. Note that we have to set the Current.account because Prompt is scoped to an account and would otherwise end up empty (due to the way AccountScoped is set up).

  # config/initializers/replicate.rb

  class ReplicateWebhook
    def call(prediction)
      query = URI(prediction.webhook).query

-     sgid = CGI.parse(query)["sgid"].first
+     prediction_sgid = CGI.parse(query)["prediction"].first
+     account_sgid = CGI.parse(query)["account"].first

-     prompt = GlobalID::Locator.locate_signed(sgid)
+     located_prediction = GlobalID::Locator.locate_signed(sgid)
+     Current.account = GlobalID::Locator.locate_signed(account_sgid)

-     prompt.predictions.create(
+     located_prediction.update(
        prediction_image: URI.parse(prediction.output.first).open.read,
        replicate_id: prediction.id,
        replicate_version: prediction.version,
        logs: prediction.logs
      )
    end
  end

This change entails that the created prediction is empty, i.e., has no prediction image (obviously). Let's cater for this by adding a conditional to our _prompt.html.erb partial. When the image is missing, we display a spinner:

  <!-- app/views/prompts/_prompt.html.erb -->

  <p>
    <strong>Generated images:</strong>
    <% prompt.predictions.each do |prediction| %>
+     <% if prediction.prediction_image.present? %>
        <%= image_tag prediction.data_url %>
+     <% else %>
+       <sl-spinner style="font-size: 8rem;"></sl-spinner>
+     <% end %>
    <% end %>
  </p>

Great, we're done preparing our app to deliver live updates. Let's implement Turbo-Rails model broadcasts to finish this proof of concept.

Delivering Prediction Updates Live with Turbo-Rails

To test the WebSocket capabilities of LiteStack, we are going to use Turbo::Broadcastable. We'd like to show the spinner and the generated image once it has been created.

The way to do that is quite idiomatic: We tie this to after_create_commit and after_update_commit model callbacks invoking one of Turbo::Broadcastable's broadcast methods. Before we can do that, though, let's separate out a model partial for Prediction:

<!-- app/views/predictions/_prediction.html.erb -->
<%= turbo_stream_from prediction %>

<div id="<%= dom_id(prediction) %>">
  <% if prediction.prediction_image.present? %>
    <%= image_tag prediction.data_url %>
  <% else %>
    <sl-spinner style="font-size: 8rem;"></sl-spinner>
  <% end %>
</div>

Observe that I added a turbo_stream_from tag to the partial, containing the stream identifier and subscribing to the channel. We can now simply call render from the prompt partial and add another turbo_stream_from to listen for changes to the prediction list:

  <!-- app/views/prompts/_prompt.html.erb -->

  <p>
    <strong>Generated images:</strong>
-   <% prompt.predictions.each do |prediction| %>
-     <% if prediction.prediction_image.present? %>
-       <%= image_tag prediction.data_url %>
-     <% else %>
-       <sl-spinner style="font-size: 8rem;"></sl-spinner>
-     <% end %>
-   <% end %>
+   <%= turbo_stream_from :predictions %>
+   <div id="<%= dom_id(prompt, :predictions) %>">
+     <%= render prompt.predictions %>
+   </div>
  </p>

Now we're ready to set up model broadcasts. In the Prediction class, we add two model callbacks, invoking two Turbo Stream actions.

  # app/models/prediction.rb

  class Prediction < ApplicationRecord
+   include ActionView::RecordIdentifier

+   after_create_commit -> { broadcast_append_later_to :predictions,
+     target: dom_id(prompt, :predictions) }
+   after_update_commit -> { broadcast_replace_later_to self }

    belongs_to :prompt

    def data_url
      encoded_data = Base64.strict_encode64(prediction_image)

      "data:image/png;base64,#{encoded_data}"
    end
  end

What's Happening Here?

First, when a prediction is created, we append it to the predictions list. This will show our loading spinner once GenerateImageJob has run.

Then, every update to the record will trigger a replace of the prediction partial. Once the prediction is updated in ReplicateWebhook, the image returned from Replicate displays.

Here's what this looks like (note that I'm using Shoelace components for styling purposes):

Benchmarks: LiteCable Vs. Redis

So far, this article has shown that it's possible to run ActionCable with LiteCable as its adapter. This is a nice proof of concept, but we're here to check how LiteStack compares to other adapters as well.

Luckily, the official LiteStack benchmarks include measurements for LiteCable against Redis, which I am going to quote here.

Here's a small but important caveat: All these measurements were performed on the same machine. In typical production setups with managed Redis, you'll have to factor in additional network latency.

Let's look at the requests per second metric first. This captures how many Pub/Sub requests the Redis and SQLite processes are able to serve.

Requests	Redis requests/second	LiteStack requests/second
1,000	2611	3058
10,000	3110	5328
100,000	3403	5385

Note that LiteStack is able to process more requests per second, but tapers off with higher loads. Though not representative, this might be an issue for loads of 1M requests and beyond — but that's when you'll typically reach for faster solutions like AnyCable over stock ActionCable anyway.

Furthermore, there are some latency tests included in the benchmarks.

Requests	Redis p90 Latency	LiteStack p90 Latency	Redis p99 Latency	LiteStack p99 Latency
1,000	34	27	153	78
10,000	81	40	138	122
100,000	41	36	153	235

Allowing for some inaccuracy of measurement, both perform equivalently in this regard, maybe with the exception of the 99 percentile. Here, SQLite's locking model interferes with the amount of concurrent requests.

Again keep in mind, though, that you'll have to add a couple of milliseconds of latency once Redis runs on a different machine (LiteCable always runs on the same machine by design).

Limitations of SQLite for Rails

It is fair to assume that once you hit a certain level of Pub/Sub activity, you'll reach the ceiling of what's possible with a single SQLite database. That's the moment when you'll have to think about sharding, and here other technologies like Redis have a head start — though it will be interesting to see what LiteFS will have to offer.

Continuous monitoring of your app's WebSocket performance metrics using tools like AppSignal is your friend here. Reusing the ActionCable consumer on the client side is also advisable, as it will prevent wasting Pub/Sub connections.

LiteCable is tailored for vertical scaling by a tight integration of components. If you extract maximum performance from the SQLite engine, the limits of this approach are pushed a lot further. Once you observe that your latencies start to explode, though, I would suggest researching options like AnyCable, which inherently provide better strategies for horizontal scaling.

Up Next: Speed Up Rails App Rendering with LiteCache

In this post, we explored using SQLite as a Pub/Sub adapter for ActionCable to enable real-time updates in a Rails application via WebSockets. Configuring LiteCable was straightforward, requiring just a simple adapter specification. Leveraging Turbo::Broadcastable model callbacks made our implementation clean, tying broadcasts to creation and updates.

Though powerful, LiteCable is not designed to scale across multiple processes or servers. But for single-machine deployments, it unlocks real-time features in Rails without requiring a separate Redis instance.

Our next post will look at the next puzzle piece in LiteStack: the ActiveSupport cache store it provides. We'll test out how it can help us to lower server response times, and look at some benchmarks again.

See you then!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Use Shoulda Matchers with RSpec for Ruby on Rails

Roy Tomeij — 2023-12-06T00:00:00+00:00

When writing tests in Rails, you should avoid repetition and have the right amount of tests to satisfy your use case.

This article will introduce you to shoulda-matchers with RSpec for testing functionality in Rails. At the end of the post, you should feel confident about using shoulda-matchers in your Rails application.

Let's get going!

Getting Started

Go ahead and clone the repository of this starter Rails app.

The starter-code branch has the following gems installed and set up:

Shoulda Matchers for Ruby on Rails

According to the shoulda-matchers documentation:

Shoulda Matchers provides RSpec- and Minitest-compatible one-liners to test common Rails functionality that, if written by hand, would be much longer, more complex, and error-prone.

Let's see how shoulda-matchers will look before installing and using them. Our repository has an Author and Book model. We'll add name validation to the Author model without shoulda-matchers.

RSpec.describe Author, type: :model do
  describe "validations" do
    it "is invalid with invalid attributes" do
      expect(build(:author, name: '')).to_not be_valid
    end
  end
end

In the above, we build an author record without a name, and we expect it to be invalid. If we validate the name's presence in our Author model, this spec should pass.

Note: While we’ll cover shoulda-matchers with RSpec in this post, you can use other frameworks like Minitest instead.

Installation of shoulda-matchers Gem for Ruby on Rails

Add the shoulda-matchers gem to the test group in your Gemfile. It should look like this:

group :test do
  gem 'shoulda-matchers', '~> 5.0'
end

Then run bundle install to install the gem. Next, place the code snippet below at the bottom of the spec/rails_helper.rb file.

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    with.test_framework :rspec
    with.library :rails
  end
end

Here we specify the test framework and library we’ll be using.

Now we'll dive into our Active Model spec.

Active Model Spec in Rails

Your Active Model spec might consist entirely of validations similar to the spec above, which shoulda-matchers handles for you. You’ll want to test validating the presence or length of certain attributes. For example, in the sample app we have above, it’s important to validate the name presence for the author model.

describe "validations" do
  it { should validate_presence_of(:name) }
  it { should validate_length_of(:name).is_at_least(2)}
  it { should validate_length_of(:name).is_at_most(50)}
end

Here, we validate the presence and length of name. You can see that these validations are one-liners compared to the initial spec we created when we didn’t use shoulda-matchers. The opposite of presence is absence, so we can validate that an attribute is absent like this:

it { should validate_absence_of(:name) }

Here's another validation spec:

it { should validate_numericality_of(:publication_year).is_greater_than_or_equal_to(1800) }

In the above, we test whether publication_year is a numerical value and if it’s greater than or equal to 1800. We can modify the comparison to look like this:

it { should validate_comparison_of(:publication_year).greater_than(1800) }

This assumes we intend to make use of validate_comparison_of.

You can also test for validate_exclusion_of (and its opposite, validate_inclusion_of) like this:

it { should validate_exclusion_of(:username).in_array(['admin', 'superadmin']) }
it { should validate_inclusion_of(:country).in_array(['Nigeria', 'Ghana']) }

Let's say you need to validate a password confirmation:

it { should validate_confirmation_of(:password) }

You’ll want to validate that an attribute has been accepted where necessary. This comes in handy when dealing with terms_of_service, for example:

it { should validate_acceptance_of(:terms_of_service) }

Next up, let's turn our attention to the Active Record spec.

Active Record Spec in Rails

In some cases, you’ll want to validate an attribute's uniqueness. This one-liner handles that:

it {  should validate_uniqueness_of(:title) }

You can take it a bit further using scope:

it {  should validate_uniqueness_of(:title).scoped_to(:author_id) }

This will check that you have a uniqueness validation for the title attribute, but scoped to author_id.

We can also test the relationship between authors and books. Let's say an author is supposed to have many books.

describe "association" do
  it { should have_many(:books)}
end

This spec will pass if we have the relationship specified in the author model. Then, for the book model, we can have a belongs_to spec:

it { should belong_to(:author) }

There are also one-line specs for other associations you might want to test:

it { should have_one(:delivery_address) }
it { should have_one_attached(:avatar) }
it { should have_many_attached(:pictures) }
it { should have_and_belong_to_many(:publishers) }
it { should have_rich_text(:description) }

If you want, you can test that there are specific columns in your database:

it { should have_db_column(:title) }

You can take it further to test for the column type:

it { should have_db_column(:title).of_type(:string) }

There is also the option of testing for an index:

it { should have_db_index(:name) }

Even if you have a composite index:

it { should have_db_index([:author_id, :title]) }

You can use implicit_order_column in Rails v6+ to define the custom column for implicit ordering:

self.implicit_order_column = "updated_at"

Here, we specify that we want the updated_at column to handle ordering. So when we run Book.first, Rails will use the updated_at column instead of the id. By default, Rails uses the id to order records.

shoulda-matchers has a one-liner test for this:

it { should have_implicit_order_column(:updated_at) }

If we have an enum for our model (like enum status: [:published, :unpublished]), we can write this test:

it { should define_enum_for(:status) }

We can specify the test values:

it { should define_enum_for(:status).with_values([:published, :unpublished]) }

If you have a read-only attribute, you can also test for that:

it { should have_readonly_attribute(:genre) }

And you can test for accepts_nested_attributes_for:

it { should accept_nested_attributes_for(:publishers) }
it { should accept_nested_attributes_for(:publishers).allow_destroy(true) }
it { should accept_nested_attributes_for(:publishers).update_only(true) }

The above tests depend on the use case defined in your model. You can check the Rails API Documentation if you’re unsure how accept_nested_attributes_for works.

There are also options for testing that your records are serialized when you use the serialize macro:

it { should serialize(:books) }
it { should serialize(:books).as(BooksSerializer) }

Here, we test that books is serialized. We specify the exact serializer that we expect to use with as.

Finally, let's turn to the Action Controller spec.

Action Controller Spec in Rails

Moving on to params, let's use config.filter_parameters to filter parameters that we don’t want to show in our logs:

RSpec.describe ApplicationController, type: :controller do
  it { should filter_param(:password) }
end

You can see from the above that this spec is for the ApplicationController. For params that will be used in other controllers when creating a record (like the BooksController), we can have a spec that looks like this:

RSpec.describe BooksController, type: :controller do
  it do
    params = {
      book: {
        title: 'Tipping Point',
        description: 'Tipping Point',
        author: 1,
        publication_year: 2001
      }
    }
    should permit(:title, :description, :author, :publication_year).
      for(:create, params: params).
      on(:book)
  end
end

This will test that the right parameters are permitted for the BooksController action. The params hash we create matches part of the request to the controller. The test checks that title, description, author, and publication_year are permitted parameters for book.

What if the action needs a query parameter to work?

RSpec.describe BooksController, type: :controller do
  before do
    create(:book, id: 1)
  end

  it do
    params = {
      id: 1,
      book: {
        title: 'Tipping Point',
        description: 'Tipping Point',
        author: 1,
        publication_year: 2001
      }
    }
    should permit(:title, :description, :author, :publication_year).
      for(:update, params: params).
      on(:book)
  end
end

In the above, we use the before block to create a new book record with the id as 1. Then we include the id in the params hash.

If you have a controller action that simply redirects to another path, you can have a spec that looks like this:

describe 'GET #show' do
  before { get :show }

  it { should redirect_to(books_path) }
end

This checks that we are redirected to the books_path when the request gets to the show action.

We can modify the above spec to also test for its response:

describe 'GET #show' do
  before { get :show }

  it { should redirect_to(books_path) }
  it { should respond_with(301) }
end

We’ve modified it to test for the status code. If we’re not sure of the exact status code but we have a range of numbers, we can use the following:

describe 'GET #show' do
  before { get :show }

  it { should redirect_to(books_path) }
  it { should respond_with(301..308) }
end

We can use a rescue_from matcher to rescue from certain errors, like ActiveRecord::RecordInvalid:

it { should rescue_from(ActiveRecord::RecordInvalid).with(:handle_invalid) }

This assumes we have (or will have) a method called handle_invalid that will handle the error.

There are matchers for callbacks we tend to use in our controllers:

it { should use_before_action(:set_user) }
it { should_not use_before_action(:set_admin) }

it { should use_around_action(:wrap_in_transaction) }
it { should_not use_around_action(:wrap_in_transaction) }

it { should use_after_action(:send_admin_email) }
it { should_not use_after_action(:send_user_email) }

You can test if the session has been set or not.

it { should set_session }
it { should_not set_session }

You’ll want to use should_not set_session in your destroy action.

Finally, here's how you can write a spec for your routes:

it { should route(:get, '/books').to(action: :index) }
it { should route(:get, '/books/1').to(action: :show, id: 1) }

And that's it!

Wrapping Up

In this article, we’ve seen what a spec that does not use shoulda-matchers looks like. We then explored how to use shoulda-matchers for your Rails project. It simplifies specs — instead of a spec spanning multiple lines, shoulda-matchers span just one line.

While it’s helpful to use shoulda-matchers, you should know that they cannot replace every spec you’ll need to write (mostly just specs to do with business logic).

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Keep Your Ruby Code Maintainable with Money-Rails

Roy Tomeij — 2023-11-29T00:00:00+00:00

When working with money in an application, ensuring everything is accounted for is important.

In this post, we will explore some common methods and best practices of handling money in your Ruby app, and see how you can use money-rails to write maintainable money-handling code.

Let's get started!

Use Cases for Storing Money

There are several use cases where your Ruby app might need to handle money — for example, an e-commerce company that sells products, a SaaS maintaining users' subscriptions, etc.

In this post, we will walk through some examples of how to handle money in an e-commerce app.

Storing Money in a Ruby Database

First, let’s start with strategies for storing money in your database: using a float column, the numeric type, and the integer type.

Using a Float Column

The most obvious way to store money is to use a float column. However, because of the way floating-point numbers are stored, they do not guarantee exactness. Instead, they only approximate the actual value. While this might be sufficient for some applications, there are much better ways to store money.

Using the `numeric` Type

The numeric type can store arbitrary-precision numbers in a database.

A numeric column can store a large range of numbers (up to 131,072 digits before a decimal point and up to 16,383 digits after a decimal point) and is therefore perfectly suited to work with money.

But one important point to note here is that calculations on numeric types are very slow compared to integer/floating-point numbers.

Numeric types should be the go-to data type when you need arbitrary-precision numbers, but, depending on business logic, they are rarely needed unless it's for very specific use cases. Let’s see if we can do better in an e-commerce scenario.

Using the `integer` Type

We need to store the prices of products, with the lowest unit being cents. A product can be priced at $19.99 but never $19.9954321. This means that, instead of working at the dollar level, we will always have an integer value as the price at the cents level. We can easily store an integer without any approximation in the database using one of the integer types (e.g., smallint, integer, bigint).

But please note that this does constrain the represented values (as compared to the numeric type) — the maximum integer value on Postgres 2147483647 will be $21,474,836.47. This should normally be enough for any e-commerce product, but it’s good to know that there’s a limit. If you require higher precision (for example, a 10th of a cent), this is possible by just storing the amount as a mill instead of cents. Keep in mind that this will further decrease the range of values that can be represented in an integer column.

But an integer's big advantage over a numeric type is that calculations are very fast. As long as you can work within the integer (or bigint) range and have clear constraints set, using an integer/bigint is the recommended storage option.

Sorting out storage is just the first step to representing money in our Ruby application, though. You might also need to handle different currencies, formatting the values based on the location of users (e.g., USD is formatted as $1,234.56, whereas the same amount in Euros is formatted as €1.234,56).

While it is possible to implement this manually, a gem like money-rails can handle it out of the box for you.

Enter `money-rails` for Ruby on Rails

Money-Rails integrates the money gem in Rails. It can handle automatic conversion between the database representation of money to Money instances. These instances can then be used to access all helpers available from the money gem (we will get to an overview of those helpers later in this post).

Additionally, money-rails also has helpers for migrations and configurable validation options. Let’s start by creating a product with a price. We'll use the monetize helper to do that:

# db/migrate/20230803062226_create_products.rb
class CreateProducts < ActiveRecord::Migration[7.0]
  def change
    create_table :products do |t|
      t.string :title, null: false
      t.monetize :price, null: false

      t.timestamps
    end
  end
end

t.monetize :price adds price_cents and price_currency columns to the table. In addition to this, we need to mark the price_cents column that money-rails will control in the model.

# app/models/product.rb
class Product < ApplicationRecord
  monetize :price_cents
end

With just a few lines, we now have access to a lot of helpers. We can create a new product by simply passing a price attribute. The price will automatically be converted to a cent value for the database. When accessing the price later, we will have instances of Money.

irb> product = Product.new(title: "Foo", price: 19.99)
# => #<Product:0x0000000108ff9e30 id: nil, title: "Foo", price_cents: 1999, price_currency: "USD", created_at: nil, updated_at: nil>

irb> product.price
# => #<Money fractional:1999 currency:USD>

You can also pass the attributes directly:

irb> product = Product.new(title: "Foo", price_cents: 1999, price_currency: "EUR")
# => #<Product:0x0000000108ff2130 id: nil, title: "Foo", price_cents: 1999, price_currency: "EUR", created_at: nil, updated_at: nil>

irb> product.price.format
# => "€19,99"

Currency in the `money` Gem for Ruby

Currencies are consistently represented as instances of Money::Currency. This not only includes the name of the currency, but a lot of additional information like the name, symbol, iso code, etc.:

irb(main):042:0> product.price.currency
# => #<Money::Currency id: usd, priority: 1, symbol_first: true, thousands_separator: ,, html_entity: $, decimal_mark: ., name: United States Dollar, symbol: $, subunit_to_unit: 100, exponent: 2, iso_code: USD, iso_numeric: 840, subunit: Cent, smallest_denomination: 1, format: >

Check out the money gem's currency documentation.

Let's say you only have a single currency in your application and don’t need to track currency inside your database. First, set a default currency for money in an initializer:

# config/initializers/money.rb
MoneyRails.configure do |config|
  # set the default currency
  config.default_currency = :usd
end

In the migration, disable the currency column:

# db/migrate/20230803062226_create_products.rb
class CreateProducts < ActiveRecord::Migration[7.0]
  def change
    create_table :products do |t|
      t.string :title, null: false
      t.monetize :price, null: false, currency: { present: false }

      t.timestamps
    end
  end
end

There are many other Money-Rails migration helpers that allow null values for amounts or configure default amounts/currencies.

Validations with `monetize`

You can configure validations on money using the monetize Active Record helper. To do this, update your model to opt in for validations:

class Product < ApplicationRecord
  monetize :price_cents,
           numericality: {
             greater_than_or_equal_to: 0,
             less_than_or_equal_to: 1000
           }
end

Now, if you try to save a product with an invalid price, validation will fail:

irb> product = Product.new(title: "Foo", price: -10)
irb> product.save!
# => raise_validation_error: Validation failed: Price must be greater than or equal to 0 (ActiveRecord::RecordInvalid)

irb> product.errors
# => #<ActiveModel::Errors [#<ActiveModel::Error attribute=price, type=greater_than_or_equal_to, options={:allow_nil=>nil, :value=>-10, :count=>0}>]>

Localization

Money can be configured to either use the location information provided by i18n or to localize based on the amount’s currency. Use the locale_backend to configure this — usually in an initializer like this:

# config/initializers/money.rb
Money.locale_backend = :currency

In the above example, we set it to :currency. This uses the formatting rules defined for each currency:

irb> product = Product.new(title: "Foo", price_cents: 15_999_00, price_currency: "USD")
irb> product.price.format
# => "$15,999.00"

irb> product.price_currency = "EUR"
irb> product.price.format
# => "€15.999,00"

To configure a consistently formatted amount regardless of the amount’s currency, use the :i18n locale backend:

irb> Money.locale_backend = :i18n
irb> I18n.locale = :en

irb> Product.new(title: "Foo", price_cents: 15_999_00, price_currency: "USD").price.format
# => "$15,999.00"
irb> Product.new(title: "Foo", price_cents: 15_999_00, price_currency: "EUR").price.format
# => "€15,999.00"

When using the locale backend i18n, it is also possible to configure the formatting using the translation files:

# config/locale/en.yml
en:
  number:
    currency:
      format:
        delimiter: ","
        format: "%u%n"
        precision: 2
        separator: "."
        significant: false
        strip_insignificant_zeros: false
        unit: "$"

Note that if you are using rails-i18n, configuration is automatically available for many locales.

Computations

Money instances are usable as regular numerical values with all operators, so you can compare values and perform arithmetic operations (using +, -, *, /). This makes it much more intuitive to perform operations on objects containing money.

# Comparisons - both value and currency must be equal for equality
irb> p1 = Product.new(price_cents: 1000, price_currency: "USD")
irb> p2 = Product.new(price_cents: 1000, price_currency: "USD")
irb> p1.price == p2.price
# => true

Since mathematical operators work as expected, you can easily compute sum/average or price discounts.

# Compute invoice total by summing all line item amounts
irb> item = LineItem.new(amount_cents: 1000, amount_currency: "USD")
irb> discount = LineItem.new(amount_cents: -250, amount_currency: "USD")
irb> total = item.amount + discount.amount
# => #<Money fractional:750 currency:USD>

# Find the average price of a product
irb> average = Product.sum(&:price) / Product.count
# => #<Money fractional:1000 currency:USD>

If you need to access the underlying value from a Money instance, you can do that using amount or cents:

irb> Product.new(price_cents: 1000, price_currency: "USD").amount
# => BigDecimal(10)

irb> Product.new(price_cents: 1000, price_currency: "USD").cents
# => 1000 (Integer)

Currency Exchange

If you are working with multiple currencies, you can automatically convert two different currencies based on exchange rates using exchange rate stores.

The default implementation is just an in-memory store. It stores the exchange rates you provide using Money.add_rate.

To use it, first add a rate. Then, you can use the exchange_to method to convert a money object to another currency.

irb> Money.add_rate("USD", "EUR", 0.92)
irb> Product.new(price_cents: 1000, price_currency: "USD").price.exchange_to("EUR")
# => #<Money fractional:920 currency:EUR>

If a conversion rate is not present, it raises a Money::Bank::UnknownRate exception.

irb> Product.new(price_cents: 1000, price_currency: "USD").price.exchange_to("EUR")
# => No conversion rate known for 'USD' -> 'INR' (Money::Bank::UnknownRate)

So, if you are using this in production, you must provide the exchange rate before you perform any conversions.

You can provide a custom exchange rate store to handle this automatically. There are also some exchange rate implementations already available as gems. For example, you can drop in the eu_central_bank exchange rate store to fetch the latest exchange rates from the European Central Bank.

Once the gem is installed, you can set it as the default bank in the initializer:

# config/initializers/money.rb
require 'eu_central_bank'
eu_bank = EuCentralBank.new
Money.default_bank = eu_bank

# Update rates from the EU Central Bank
# Ideally this should be performed periodically in a cron job.
# If you need conversions rarely, you can call it before using exchange_to.
eu_bank.update_rates

Once set up, you can simply exchange any currency, and it should work:

irb> Product.new(price_cents: 1000, price_currency: "USD").price.exchange_to("EUR")
# => Money.from_cents(92, "EUR")

Wrap Up

In this post, we saw how to store and work with money in a Rails application.

Money-Rails greatly simplifies the whole process and provides helper methods that make working with money safer and more maintainable in the long run.

Many other options are available to customize this for your Ruby on Rails app. Check out the full details in the money and money-rails guides.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Shaping the Future of Ruby and Kafka Together with rdkafka-ruby

Roy Tomeij — 2023-11-22T00:00:00+00:00

Hello there! My name is Maciej Mensfeld, and some of you might recognize me from my involvement in RubyGems Security, OSS commitments, or perhaps from Karafka: a multi-threaded, efficient Kafka processing framework tailored for Ruby and Rails.

While I generally pen my thoughts on my personal blog, today's post is unique. This article results from a collaborative effort with the brilliant people over at AppSignal. To set the record straight, I don't work for AppSignal. However, I felt that AppSignal's blog would be the most appropriate platform for this piece due to its nature.

Karafka Embraces `rdkafka` Gem

We're happy to stand behind the rdkafka-ruby gem, a brainchild of AppSignal. Their incredible contributions have not only transformed the landscape of Ruby Kafka but have also played a pivotal role in the evolution of the Karafka ecosystem.

This library is a beacon of modernity in the Kafka space for Ruby. It integrates with librdkafka, the production-ready C client using the ffi gem that provides outstanding performance and stability.

Following the sunset of ruby-kafka, it rapidly became the go-to low-level driver for Kafka interactions in Ruby. The migration of Karafka and Racecar (leading Ruby Kafka frameworks) to rdkafka-ruby further solidified its dominance.

AppSignal's Use Case and the OSS Community

While AppSignal deploys rdkafka-ruby on a vast scale, it's essential to recognize that its use cases might only encompass some of the needs of the Ruby OSS community. This is especially true in realms like data consumption and crafting higher-level functionalities.

Recognizing these gaps and driven by my passion for the Karafka ecosystem and Kafka, I introduced a fork named karafka-rdkafka to expand this library further. However, the vision of having two autonomous gems, each with its developmental journey, raised concerns about the overarching benefit to the Ruby Kafka OSS community. It was this realization that led me to extend an "adoption offer" to AppSignal's talented development team, particularly to Thijs.

Merging Visions for a Unified Future

Today, I'm pleased to share that our discussions bore fruit, carving a path that supports the collective interests of the Ruby and Kafka open-source community. The rdkafka-ruby gem will now thrive under the care of the Karafka community, with some guiding principles:

Openness: rdkafka-ruby will eternally stand as a fully open, MIT-licensed primary binding for librdkafka.
Low-level Commitment: While it offers direct APIs based on librdkafka, rdkafka-ruby will maintain its "layer 1" stature, focusing on low-level functionalities and leaving high-level details to libraries built on top of it.
Alignment: rdkafka-ruby and karafka-rdkafka will harmonize their developmental paths. Unless karafka-rdkafka requires any framework-specific elements that would not be configurable, both will be merged into one.
Maintainability - rdkafka-ruby will consistently support every official version of both Kafka and Ruby, ensuring smooth operations for its users.

Such a strategic shift promises stability and continued evolution to the entire ecosystem. Whether you're a rdkafka-ruby enthusiast valuing its flexibility or someone invested in high-level frameworks like Karafka and WaterDrop, you can rest assured that rdkafka-ruby remains the heartbeat of these platforms.

`rdkafka-ruby` and Karafka Instrumentation with AppSignal

While transitioning the rdkafka-ruby ownership, feedback from AppSignal users highlighted another need. It became clear that an official instrumentation and monitoring solution for the Karafka ecosystem was essential. So, in response to this demand, I took steps to provide just that. This wasn't just about meeting a need; it was about fortifying the Karafka community with robust tools to ensure efficiency, stability, and clarity in their operations.

In the realm of Kafka with Ruby and Rails, the choice of monitoring tools is crucial. AppSignal emerges as the premier choice for Karafka applications instrumentation, and here's why:

AppSignal offers insights into the performance, stability, health, and resource utilization of Karafka consumers, ensuring they operate optimally.
It provides comprehensive error notifications for both consumers and producers, adeptly capturing asynchronous errors related to their operations.
With official support from Karafka's creator, AppSignal's integration promises continuous updates and refinements, ensuring it remains the go-to tool for Karafka monitoring.
For developers seeking simplicity, AppSignal integration is a breeze. It comes packaged with Karafka, eliminating the need for extra gems or tedious setups.

Choosing AppSignal isn't just about monitoring; it's about optimizing and ensuring the reliability of your Karafka and Kafka-based applications.

The Importance of Monitoring Your Consumers and Producers

Monitoring producers and consumers is not just a 'good-to-have' but a foundational pillar of any robust Kafka-based system. Here's an in-depth look into why this is so essential:

Ensuring Data Consistency and Integrity

Accurate Data Flow: Kafka's architecture is fundamentally about ensuring that messages are transported from producers to consumers. Effective monitoring verifies that this data flow is accurate and consistent.
Detecting Anomalies: In a distributed system, there's a possibility of message loss, duplication, or even out-of-order processing. Monitoring tools can promptly detect such issues, allowing for timely interventions.
Audit and Compliance: For sectors where data integrity is not just an operational requirement but a legal one (like finance or healthcare), monitoring ensures that systems remain compliant with regulatory standards.

Identifying and Resolving Performance Bottlenecks

Understanding System Dynamics: Monitoring offers insights into the dynamics between producers and consumers. By visualizing message flow rates, system administrators can pinpoint areas of congestion or under-utilization.
Proactive Issue Resolution: Imagine a scenario where a producer continuously pushes messages, but the consumer lags in processing them. This disparity could go unnoticed without monitoring until it snowballs into a significant issue. AppSignal highlights these discrepancies and provides actionable intelligence to resolve them.
Optimizing Resources: Resources, be they computational or human, are finite. Through monitoring, you can prevent wastage and ensure the highest levels of efficiency.

Enhancing System Reliability and Availability

Immediate Failure Detection: Kafka systems are often the backbone of critical business operations. Any downtime or failure can lead to substantial financial and reputational losses. Monitoring tools act as vigilant guards, immediately detecting failures or disruptions.
Insights into System Health: Beyond immediate failures, monitoring provides a continuous pulse check on the Karafka ecosystem's health. Whether tracking memory usage, gauging disk I/O, or measuring network latency, these insights ensure the system remains in its prime operational state.
Future-Proofing: By observing trends and patterns over time, monitoring can aid in capacity planning, ensuring that the Kafka setup is reliable today and geared up for future demands. In essence, neglecting to monitor Kafka's consumers and producers is more than just an operational oversight; it's a missed opportunity to harness the true power and efficiency of the Karafka and Kafka ecosystem.

Getting Started with Karafka and AppSignal

The beauty of integrating Karafka with AppSignal lies in its simplicity. Thanks to the AppSignal integration being inherently part of the Karafka gem, there's no need for extra installations or gems. All the necessary components are pre-packaged in Karafka. Remember one key detail: when configuring, ensure the errors listener is subscribed before the metrics listener for optimal monitoring accuracy.

# Require both listeners
require 'karafka/instrumentation/vendors/appsignal/errors_listener'
require 'karafka/instrumentation/vendors/appsignal/metrics_listener'

# Create them
errors_listener = ::Karafka::Instrumentation::Vendors::Appsignal::ErrorsListener.new
metric_listener = ::Karafka::Instrumentation::Vendors::Appsignal::MetricsListener.new

# Subscribe with the error listener first to track producers and consumers errors
Karafka.monitor.subscribe(errors_listener)
Karafka.producer.monitor.subscribe(errors_listener)
# And subscribe to Karafka monitor to get all the exciting metrics
Karafka.monitor.subscribe(metric_listener)

Monitoring Karafka and Kafka with AppSignal

Here are some of the key metrics and insights offered by the Karafka-AppSignal integration (the following graphs are automatically created for you when you set up the integration):

Consumption Metrics

Understanding consumers' and producers' health and behavior is foundational to any Karafka setup. With AppSignal, you can monitor aspects such as message processing times, batch sizes, error rates, and more. This data allows developers to ensure that consumers and producers operate at their peak efficiency.

Message Throughput and Lag

The core function of Kafka is the flow of messages. AppSignal offers insights into message throughput, clearly showing the volume of messages being processed. Moreover, tracking message lag is crucial; it highlights the time difference between when a message is produced and when it's consumed. High lags can be indicative of potential issues in the pipeline.

Performance Monitoring of Consumers

AppSignal allows you to monitor the performance of each of the consumers' jobs. Tracking performance provides invaluable insights, ensuring that each consumer operates efficiently and processes data at optimal speeds.

Error and Exception Tracking

AppSignal's robust error tracking system ensures that any anomalies, whether application-level or system-level, are immediately flagged. This not only helps in prompt rectification but also aids in understanding patterns that lead to such issues.

Wrapping Up

In summary, the Karafka AppSignal integration isn't just a monitoring tool; it's a magnifying glass into the intricate world of Karafka.

While the aspects we've covered above are some of the core metrics and insights available, the platform offers much more. Dive deeper to understand how to best optimize and enhance your Karafka and Kafka operations.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Handle Incoming Webhooks with LiteJob for Ruby on Rails

Roy Tomeij — 2023-11-15T00:00:00+00:00

In parts one and two of this series, we only dealt with the pure CRUD aspects of using SQLite as a production database.

In this post, we will explore the world of queue mechanisms, using SQLite as the pub/sub adapter for ActiveJob.

Let's make use of LiteJob to handle incoming webhooks in our Rails application.

Our Ruby on Rails Use Case

Our use case is the image generation that we are currently doing synchronously in the PromptsController:

model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
version = model.latest_version
version.predict({prompt: prompt_params[:title], image: @prompt.data_url}, replicate_rails_url)

Clearly, there are some long-running processes here that should be deferred to a background job: the loading of the model, and the prediction itself.

Configuration

In your environment configuration files, ensure you reference LiteJob as the queue adapter:

# config/environments/development.rb
# config/environments/production.rb
Rails.application.configure do
  # ...

  config.active_job.queue_adapter = :litejob

  # ...
end

Now we're all set to continue our investigation. Don't forget to restart your development server, though!

If you like, you can add more ActiveJob configuration in config/litejob.yml — for example, queue priorities:

queues:
  - [default, 1]
  - [urgent, 5]
  - [critical, 10, "spawn"]

Please refer to the README for further details.

That's set up, so let's move on to generating our images asynchronously.

Generating Images Asynchronously in Our Ruby on Rails Application

We'll start by scaffolding our job with the usual Rails generator command:

$ bin/rails g job GenerateImage

Let's move the image generation code from PromptsController into the perform method of this job:

class GenerateImageJob < ApplicationJob
  include Rails.application.routes.url_helpers

  queue_as :default

  def perform(prompt:)
    model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
    version = model.latest_version
    version.predict({prompt: prompt.title, image: prompt.data_url}, replicate_rails_url(host: Rails.application.config.action_mailer.default_url_options[:host]))
  end
end

Note that we have to include the URL helpers module here to access replicate_rails_url (as opposed to controllers, where this happens automatically). Furthermore, we also need to explicitly specify the host — we grab it off ActionMailer's default_url_options, in this case. Look at Andy Croll's excellent 'Use Rails URL helpers outside views and controllers' post for more sophisticated uses.

To make this work with the incoming webhook, make sure you also reference your ngrok URL in your app's configuration:

# config/environments/development.rb
config.action_mailer.default_url_options = { host: "YOUR_NGROK_URL", port: 3000 }

As the final step on the requesting side of our call to Replicate, we have to actually call the job from PromptsController:

  # app/controllers/prompts_controller.rb

  def create
    # ...
-   model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
-   version = model.latest_version
-   version.predict({prompt: prompt_params[:title],
-     image: @prompt.data_url}, replicate_rails_url)

    respond_to do |format|
      if @prompt.save
+       GenerateImageJob.perform_later(prompt: @prompt)
+
        format.html { redirect_to prompt_url(@prompt), notice: "Prompt was successfully created." }
        format.json { render :show, status: :created, location: @prompt }
      else
        format.html { render :new, status: :unprocessable_entity }
        format.json { render json: @prompt.errors, status: :unprocessable_entity }
      end
    end
  end

Persisting Replicate.com Predictions

Now it's time to take a look at the receiving end of our prediction request, i.e., the incoming webhook.

Right now, generated predictions are handed back to us, but we don't do anything with them. Since images are purged from Replicate's CDN periodically, we want to store them locally.

Let's start by generating a new child model, Prediction. We'll prepare it to store the generated image as a binary blob again:

$ bin/rails g model Prediction prompt:references replicate_id:string replicate_version:string prediction_image:binary logs:text
$ bin/rails db:migrate

On top of that, we create columns to store some metadata returned from Replicate.com: id, version, and logs — see the API docs.

Finally, we also have to register this new association in the Prompt model:

  # app/models/prompt.rb

  class Prompt < ApplicationRecord
    include AccountScoped

    belongs_to :account
+   has_many :predictions, dependent: :destroy

    # ...
  end

Now we just have to process the incoming predictions in our webhook, but we face an issue. We have to find a way to identify the specific Prompt instance the prediction was created for. We can circumvent this problem by adding a query param to the incoming webhook URL:

  # app/jobs/generate_image_job.rb

  class GenerateImageJob < ApplicationJob
    include Rails.application.routes.url_helpers

    queue_as :default

    def perform(prompt:)
      model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
      version = model.latest_version
-     version.predict({prompt: prompt.title, image: prompt.data_url}, replicate_rails_url(host: Rails.application.config.action_mailer.default_url_options[:host]))
+     version.predict({prompt: prompt.title, image: prompt.data_url}, replicate_rails_url(host: Rails.application.config.action_mailer.default_url_options[:host], params: {sgid: prompt.to_sgid.to_s}))
    end
  end

This will effectively send our incoming webhook to YOUR_NGROK_URL/replicate/webhook?sgid=YOUR_RECORD_SGID, making it easy to identify the prompt.

Obtain the `sgid` in Ruby

Sadly, the replicate-rails gem's default controller doesn't pass the query parameters to the webhook handler, but it does pass the exact webhook URL.

So we can flex our Ruby standard library muscles to obtain the sgid from it:

# config/initializers/replicate.rb

# ...

require "open-uri"

class ReplicateWebhook
  def call(prediction)
    query = URI(prediction.webhook).query

    sgid = CGI.parse(query)["sgid"].first

    prompt = GlobalID::Locator.locate_signed(sgid)

    prompt.predictions.create(
      prediction_image: URI.parse(prediction.output.first).open.read,
      replicate_id: prediction.id,
      replicate_version: prediction.version,
      logs: prediction.logs
    )
  end
end

# ...

Let's dissect this:

First, we pluck the exact webhook URL off the incoming prediction object and parse it, returning the query string.
Next, we invoke CGI.parse to convert the query string into a hash. Accessing the "sgid" key returns a one-element array, which is why we have to send first to it.
Then, we can use the obtained sgid to locate the prompt instance.
Finally, we append a new prediction to our prompt using the properties returned from Replicate.com. We also download the prediction image and save it in SQLite. Note that we can skip resizing because the generated image will already have the correct dimensions.

There's one final bit we have to tend to: we have to make sure our webhook handler works in an idempotent way. Webhooks can arrive duplicated or out of order, so we have to prepare for this case.

Luckily, we can use the ID returned by replicate to ensure idempotency. All we have to do is add a unique index to the replicate_id column. This will make it impossible to add a duplicate prediction to our database:

$ bin/rails g migration AddUniqueIndexToPredictions

class AddUniqueIndexToPredictions < ActiveRecord::Migration[7.0]
  def change
    add_index :predictions, :replicate_id, unique: true
  end
end

$ bin/rails db:migrate

Keep in mind that you typically would still want to store your assets and attachments in object storage buckets like Amazon S3 or DigitalOcean spaces. To get a glimpse of what SQLite can do, we have opted to store and render them directly from the database here.

To complete our picture — pun intended — we again have to find a way to display our stored predictions as children of a prompt. We can do this using a data URL:

  class Prediction < ApplicationRecord
    belongs_to :prompt

+   def data_url
+     encoded_data = Base64.strict_encode64(prediction_image)
+
+     "data:image/png;base64,#{encoded_data}"
+   end
  end

  <!-- app/views/prompts/_prompt.html.erb -->
  <div id="<%= dom_id prompt %>">
    <p>
      <strong>Title:</strong>
      <%= prompt.title %>
    </p>

    <p>
      <strong>Description:</strong>
      <%= prompt.description %>
    </p>

    <p>
      <strong>Prompt image:</strong>
      <%= image_tag prompt.data_url %>
    </p>

+   <p>
+     <strong>Generated images:</strong>
+     <% prompt.predictions.each do |prediction| %>
+       <%= image_tag prediction.data_url %>
+     <% end %>
+   </p>
  </div>

A Look Behind the Scenes of LiteJob for Ruby on Rails

Let's quickly look into how LiteJob uses SQLite to implement a job queueing system. In essence, the class Litequeue interfaces with the SQLite queue table. This table's columns, like id, name, fire_at, value, and created_at, store and manage job details.

The push and repush methods of the Litequeue class add jobs to the queue, interfacing with their respective SQL statements. When it's time for a job's execution, the pop method in the same class retrieves and removes a job based on its scheduled time. The delete method allows for specific job removal.

The Litejobqueue class is a subclass of Litequeue, which, upon initialization, is tasked with creating the worker/s. These workers contain the main run loop that oversees the actual job execution, continuously fetching and running jobs from the queue.

Limitations of LiteJob

Now that we've broken down how to use LiteJob for asynchronous job processing, let's look at some of the potential drawbacks ensuing from this approach.

While an SQLite-based architecture provides simplicity and reduces the need for external dependencies, it also introduces several limitations.

Firstly, LiteJob's reliance on SQLite inherently restricts its horizontal scaling capabilities. Unlike other databases, SQLite is designed for single-machine use, making it challenging to distribute workload across multiple servers. This can certainly be done using novel technologies like LiteFS, but it is far from intuitive.

Additionally, even on the same machine, concurrency contends with your web server (e.g., Puma) because LiteJob spawns threads or fibers that compete with those handling the web requests.

On the other hand (and crucially, for the majority of smaller apps), you might never need such elaborate scaling concepts.

LiteJob's Performance in Benchmarks

On a more positive note, LiteJob seems to allow for a lot of overhead before you have to scale horizontally: at least the benchmarks put me in a cautiously euphoric mood. Granted, the benchmarks don't carry a realistic payload, but they demonstrate that LiteJob makes very efficient use of threads and fibers.

Up Next: Streaming Updates with LiteCable

In this installment of our series, we delved deeper into the intricacies of using SQLite with LiteJob to handle asynchronous image generation and incoming webhooks in Rails applications.

To recap, while the SQLite-based architecture of LiteJob offers simplicity and reduced external dependencies, it also presents challenges in horizontal scaling. However, for smaller applications, LiteJob's efficiency with threads and fibers suggests it can handle a significant workload before necessitating more complex scaling solutions.

In the next post of this series, we'll look at providing reactive updates to our users using Hotwire powered by LiteCable.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Deep Dive Into LiteDB for Ruby on Rails

Roy Tomeij — 2023-11-01T00:00:00+00:00

In the second post of our series covering LiteStack (an alternative way to build Rails applications entirely based on SQLite), we'll explore the database's concepts of flexible typing and type affinity.

We'll not only discover how SQLite's data handling differs from other SQL databases, but also how we efficiently process and store binary data, like images, directly in a database column.

Note: LiteDB is essentially SQLite, but fine-tuned for usage in Rails. We'll use LiteDB and SQLite interchangeably in this post.

Flexible Typing In LiteDB for Ruby on Rails

The first thing to always keep in mind is that SQLite is flexibly typed. This means that, in contrast to most other SQL databases (where the data type of a value is defined by its container — the column data type), SQLite is very forgiving. Or, as the SQLite docs state more formally:

Datatype of a value is associated with the value itself, not with its container.

This means, for example, that you can store strings in INTEGER columns, or exceed the length defined by VARCHAR(n). In the development process, this is usually not an issue. It boils down to a bit of a vendor lock-in, though: Using an SQLite database in development and a PostgreSQL one in production can become a nightmare.

Note: SQLite version 3.37.0 introduced the option of STRICT tables.

SQLite for Rails: Type Affinity

Type affinity in SQLite is a concept that determines how data is stored and converted. Before explaining what this means, let's take a look at SQLite's storage classes. These are:

NULL
INTEGER (7 different data types depending on size)
REAL (any floating point value)
TEXT (any string)
BLOB (raw binary data)

These are equivalent to data types for the purpose of a general discussion, but have a broader scope. When you create a table, you do not declare storage classes for each column, but type affinities.

A certain type affinity determines what storage class(es) will be used to store a certain column. Because this is a rather elusive topic to discuss theoretically, let's look at an example. Say we create the following table:

CREATE TABLE employees (
    id INTEGER PRIMARY KEY,
    name TEXT,
    department TEXT,
    salary NUMERIC
);

INTEGER, TEXT, and NUMERIC are the columns' type affinities. For example, suppose I enter an integer number into the department column. It will be converted into TEXT, because the TEXT affinity only uses the TEXT, NULL, and BLOG storage classes:

-- stored as 1, 'Anna Lee', '123', 85000
INSERT INTO employees (id, name, department, salary) VALUES (1, 'Anna Lee', 123, 85000);

On the other hand, if I enter the string "67000" as salary, the NUMERIC affinity will convert it to INTEGER, because it is a well formed number literal.

-- stored as 2, 'Mike Johnson', 'Customer Support', 67000
INSERT INTO employees (id, name, department, salary) VALUES (2, 'Mike Johnson', 'Customer Support', '67000');

If I were to enter "eighty-five thousand", it would just default to the TEXT storage class.

-- stored as 3, 'Jane Smith', 'Engineering', 'eighty-five thousand'
INSERT INTO employees (id, name, department, salary) VALUES (3, 'Jane Smith', 'Engineering', 'eighty-five thousand');

The rules are rather complex, so check out the SQLite official docs on datatypes. The main takeaway here is that, somewhat counterintuitively, the column type affinities you declare need not reflect the actual data types of the stored values.

SQLite also lacks boolean, datetime, and JSONB datatypes, and UUIDs. Let's take a look at these now.

No Boolean Datatype

Boolean values are commonly represented as 0 and 1 instead of TRUE and FALSE.

No Datetime Datatype

The SQLite docs recommend using the built-in conversion functions and that you store dates and times:

As a TEXT string in the ISO-8601 format. Example: '2018-04-02 12:13:46'.

As an INTEGER number of seconds since 1970 (also known as "unix time").

As a REAL value that is the fractional Julian day number.

No JSONB Datatype

Especially for developers relying on the convenience of the PostgreSQL JSONB datatype, it's important to point out that SQLite doesn't use a binary format to store JSON:

Experiments have been unable to find a binary encoding that is smaller or faster than a plain text encoding.

Most notably, SQLite's JSON implementation doesn't support BLOB storage.

No Universally Unique Identifiers (UUIDs)

Another characteristic that comes as a surprise to most developers is that SQLite lacks the concept of UUIDs. Not completely, as they can be compiled and loaded as an extension, but they are not built into the database per default.

If you are interested in exploring the advantages and tradeoffs of flexible database types further, the SQLite docs have a decent write-up.

Enough with the theoretical deliberations and dwelling on what SQLite doesn't have, though. Let's get back to building! Let's see how we can use SQLite to store image prompts.

Storing Image Prompts Directly in SQLite

The SQLite documentation claims that the database is excellently suited to store small blobs of binary data (e.g., thumbnails). We probably still would use a CDN in production, but it's interesting enough to give it a try.

For starters, let's add a content_type column to our prompts table. It will come in handy later:

$ bin/rails g migration AddContentTypeToPrompts content_type:string
$ bin/rails db:migrate

With this in place, the transcoding into a data URL can be done in the model itself:

  # app/models/prompt.rb
  class Prompt < ApplicationRecord
    include AccountScoped

    belongs_to :account
    has_rich_text :description

-   validates :title, :prompt_image, presence: true
+   validates :title, :prompt_image, :content_type, presence: true
+
+   def data_url
+     encoded_data = Base64.strict_encode64(prompt_image)
+
+     "data:image/#{content_type};base64,#{encoded_data}"
+   end
  end

We can now start to simplify our controller code a bit. If we assign both prompt_image and content_type to @prompt from the uploaded file, we can scrap the helper methods prompt_image and prompt_image_data_url:

  # app/controllers/prompts_controller.rb

  class PromptsController < ApplicationController
    # ....

    def create
+     prompt_image = prompt_params.delete(:prompt_image)
+
      @prompt = Prompt.new(prompt_params)
      @prompt.account = Current.account
+     @prompt.prompt_image = prompt_image.read
+     @prompt.content_type = prompt_image.content_type

      model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
      version = model.latest_version
-     version.predict({prompt: prompt_params[:title],
-       image: prompt_image_data_url}, replicate_rails_url)
-
+     version.predict({prompt: prompt_params[:title],
+       image: @prompt.data_url}, replicate_rails_url)

      # ...
    end

      # ...

    private

      # ...

-
-     def prompt_image_data_url
-       encoded_data = Base64.strict_encode64(prompt_image.read)
-
-       "data:image/#{prompt_image.content_type};base64,#{encoded_data}"
-     end
-
-     def prompt_image
-       @prompt_image ||= prompt_params[:prompt_image]
-     end
  end

All that's left to do now is to actually display the image in our view. For this, we can use the same data_url helper method from the model:

  <!-- app/views/prompts/_prompt.html.erb -->
  <div id="<%= dom_id prompt %>">
    <p>
      <strong>Title:</strong>
      <%= prompt.title %>
    </p>

    <p>
      <strong>Description:</strong>
      <%= prompt.description %>
    </p>

    <p>
      <strong>Prompt image:</strong>
-     <%= image_tag prompt.prompt_image %>
+     <%= image_tag prompt.data_url %>
    </p>
  </div>

There is a tiny blemish here, though. We are potentially storing and transferring way too much image data. StableDiffusion needs only a 768 pixel wide (or long) image to work, so we will scale it down beforehand. To accomplish this, we'll install the ImageProcessing library and use libvips to transform our image.

Note: you have to install libvips on your operating system for this to work. Instructions are available in the README.

The image_processing gem is already installed in an off-the-shelf Rails 7 app, so we can start writing our transformation logic. We'll do this in a before_save callback in our Prompt model:

  # app/models/prompt.rb
+ require "vips"

  class Prompt < ApplicationRecord
    include AccountScoped

    # ...

+   before_save :scale_prompt_image

    # ...

+   private
+
+   def scale_prompt_image
+     image = Vips::Image.new_from_buffer(prompt_image, "")
+     pipeline = ImageProcessing::Vips.source(image)
+
+     self.prompt_image = pipeline.resize_to_fit(768, 768).call.read
+   end
  end

The resize_to_fit method will scale an image to fit the specified dimensions while preserving the aspect ratio. For example, if we upload an image with 1024x768 pixels, it will be scaled down to 768x576.

This completes the preprocessing logic of our image generation pipeline.

Overall Notes on SQLite for Rails and Its Performance

Although some traditional data types like boolean and datetime are absent, SQLite excels in handling binary data, making it an excellent choice for certain applications. If you are interested in how SQLite compares to other Relational Database Management Systems (RDBMS's) in terms of performance, have a look at the benchmarks.

A Quick Note On Litestream

As a final side-note, I would like to touch on Litestream, a performant and safe way to replicate your SQLite databases to S3-compatible object storage in a streaming manner. Not only does it provide a very cost-effective backup solution, but it is also a very simple way to restore your database.

Up Next: Asynchronously Handling Image Predictions with LiteJob

In this article, we've delved into SQLite's flexible typing and type affinity, harnessing its power to process and store binary data. We have looked into optimizing image storage and leveraged SQLite's unique data-handling features.

Next, we'll consider how to handle the actual generation and persistence of prediction images. We have only implemented a webhook stub so far, and created the predictions in a peculiar way in our PromptsController. We will look into performing this step asynchronously with LiteJob.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Advanced Usages of Action Policy for Ruby on Rails

Roy Tomeij — 2023-10-18T00:00:00+00:00

In part one of this series, we looked at some basic usages of Action Policy. Now we'll leverage Action Policy for more advanced authorization use cases.

First up, let's explore applying pre-checks.

Pre-checks

Let's say we want users with "editor" status to have access to a post's show, update, and destroy actions, like so:

# app/policies/posts_policy.rb

class PostPolicy < ApplicationPolicy
    def show?
      user.editor || user.reader || user.id == record.user_id
    end

    def update?
      user.editor || (user.id == record.user_id)
    end

    def destroy?
      user.editor || user.id == record.user_id
    end
end

We can refactor this policy code using an Action Policy pre-check that extracts common rules into pre-checks:

# app/policies/posts_policy.rb

class PostPolicy < ApplicationPolicy
  pre-check :allow_editors

  def show?
    user.reader || user.id == record.user_id
  end

  def update?
    user.id == record.user_id
  end

  def destroy?
    user.id == record.user_id
  end

  private

  def allow_editors
    allow! if user.editor
  end
end

Scopes

Another Action Policy feature that deserves our attention is scoping. Scopes filter data depending on any authorization rules you've set.

Using our blog app as an example, let's say we want to apply the following rules to the posts index action:

List all posts for all users with the "editor" role
List only posts that a user has created if they have the "author" role

Without using any Action Policy authorization, the posts controller index action looks like any generic index action:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController
  ...

  def index
    @posts = Post.all
  end

  ...
end

We'll need to utilize Action Policy scoping to refactor this action and apply the outlined access rules. Scoping rules are defined within a policy class and applied in the respective controller using the authorized_scope or authorized methods.

First modify the Post policy, like so:

# app/policies/posts_policy.rb

class PostPolicy < ApplicationPolicy
  ...

  relation_scope do |scope|
      if user.editor?
        scope
      end
    end
  end

  relation_scope(:own) do |scope|
      scope.where(user: user)
    end
  end

  ...
end

Then the posts controller's index action, as shown below:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

  ...

  def index
    @posts = authorized_scope(Post, type: :relation, as: :own)
  end
  ...

end

Caching in Action Policy for Ruby and Rails

Action Policy is a performant authorization library, partly thanks to its efficient use of caching.

It has several cache layers for you to leverage, including memoization and external caching.

Memoization

Consider a situation where the same rule is called several times on an object instance. For example, let's say we need to load all comments associated with a post within the index action:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

  ...

  def index
    @posts = authorized_scope(Post.includes(comments), ...)
  end
  ...

end

Now, imagine there's an "edit" link for every comment a post has within the index view. As you can see, this is a resource-intensive undertaking since we need to check if the current user is allowed to first access the post index, and then edit a post's comments. If a post has multiple comments, this would mean loading the authorization policy multiple times.

In a situation like this, Action Policy will re-use the required policy instance — specifically, the record.policy_cache_key — as one of the identifiers in the local store.

This kind of caching works well for short-lived requests, but if you need to cache resource-intensive rules that will be persisted across requests, using an external cache store is a better option.

Using an External Cache Store

If you need to run access control rules that utilize complex database queries, for example, you can use an external cache store such as Redis. Your rules cache will be made available across requests. You just need to remember to explicitly define which rules to cache within the policy class:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
  cache :index?

  def show?
    # some complex database heavy rules...
  end

  ....
end

And configure the cache store for your app:

# config/application.rb

Rails.application.configure do |config|
  config.action_policy.cache_store = :redis_cache_store
end

Quick tip: There's a lot more to this. Dig into the Action Policy caching documentation for more information.

Aliases in Action Policy for Rails

An alias is an alternative way to name policies so that they make more sense.

Let's check out an example using our blog app. Consider this Post policy:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
  alias_rule :destroy?, :update?, to: :manage_post?

  ...

  def manage_article?
    user.editor || user.id == record.user_id
  end

  ...
end

Here, we combine the update? and destroy? rules into one alias called manage_post?.

Then we use it in the controller, like so:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

  ...

  def update
    authorize! :manage_post?, @post
    ...
  end

  # DELETE /posts/1 or /posts/1.json
  def destroy
    authorize! :manage_post?, @post
    ...
  end
  ...
end

Finally, let's quickly look at how to handle unauthorized access.

Handling Unauthorized Access in a Ruby and Rails App

If a user tries to access a resource they shouldn't, an ActionPolicy::Unauthorized error is raised.

You need to explicitly handle this error in your app's ApplicationController, like so:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  rescue_from ActionPolicy::Unauthorized do
    redirect_to root_path, alert: 'Access denied.'
  end
end

And that's it!

Wrapping Up

In this post, we explored advanced Action Policy features, including pre-checks, scopes, caching, aliases, and finally, handling unauthorized access.

From basic rules to complex conditional scenarios, Action Policy has everything you need to handle almost any authorization scenario. Use this library in your next project and see how powerful it is.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Expressive Ruby and Rails: Communicate Effectively With Your Code

Roy Tomeij — 2023-10-11T00:00:00+00:00

Ruby is an expressive language. This is no accident; Matz very consciously designed Ruby as an intuitive language to more or less read like English. It's safe to say that he succeeded. Methods are named very carefully, and do what they say they do; they also tend to have inverse methods which do the opposite.

In this post, we'll look at why expressive code is important and its impact on your productivity as a developer. Then, we'll explore how to best use some of Ruby's methods.

Let's get started!

The Importance of Expressiveness in Ruby

Expressive code goes beyond readability ("What does this code do?"), and clearly communicates intent: it clues us into the developer's considerations and concerns, and the edge cases they were accounting for. It makes clear not just what the code does, but why it was written how it was.

Why is this important? It saves us from spending a lot of time familiarizing ourselves with the context of the code we're working on, clues us in to what we should be looking out for when we modify it, and keeps us from potentially bulldozing carefully-coded functionality and measures meant to safeguard our application.

Ruby takes to heart the principle that good code should be self-documenting; it largely obviates the need for comments. This is likely one of the reasons we often associate commented code in Ruby with bad code: because, in many cases, it is. Working with external APIs or unfamiliar gems is one thing, but generally, for most of our core functionality, writing idiomatic Ruby means we don't need to write (or read) comments. Usually, any time I find myself with comments all over the place, it's a sign I need to refactor.

Ultimately, expressive code saves us time, and our company money.

The Impact of Expressive Coding on Productivity

What problems do we face if we aren't conscientious about how we write and structure our code? The answer is at once straightforward and easy to forget in the moment. The most obvious issue — a lack of clarity — in addition to being its own problem, can lead to a host of other headaches:

Opacity

In addition to being confusing, inexpressive code also turns the thought process of the developer who coded it into something of a black box, rather than a window into their decision-making.

As touched on above, this can force the developer maintaining our code to delve into the implementation in greater detail than they should to get context. This only serves to derail us from our task at hand, and makes our lives harder.

Loss of Trust

When a particular feature or area lacks expressiveness, it can be frustrating, but it is also an opportunity to improve our app. But when a lack of expressiveness is a pattern across our entire codebase, we lose faith in our code as a reliable source of truth to communicate the potential values it may be handling.

This inevitably leads to constant defensive programming, which can serve to further muddy the waters. We may wind up accounting for potential values that aren't even real possibilities, magnifying the existing problem and further eroding trust. In addition, we might add verbosity to our code, and send potentially misleading signals to other developers (and even our future selves) about what we're accounting for and trying to accomplish.

On the other hand, a lack of trust can lead to bad assumptions when the right method is actually being used. Consider a scenario where someone has just spent time cleaning up a section of code and spots something like if @user.address.present?.

Rather than stopping and thinking: "hey, there must be a reason present? was used," they might think: "hey, this particular field is optionally set by the user... so either they set it and it's a string, or it wasn't set, so it's nil." So they change it to if @user.address. Except, what happens when the user has a value set, and then later removes it and submits the form? We'll now have an empty string (which now, without present?, evaluates as truthy).

Type Issues

Constantly having to check for potential data types and the possibility of nil values can be genuinely exhausting and really slow us down. It not only wastes time by forcing us to work out the possible data types of inputs, but it can also break our focus and flow, and add unnecessary cognitive overhead.

In some ways, we can think of Ruby's expressiveness as an answer to a lack of static typing. By employing clear variable names and self-explanatory methods that do what they say in a fixed, predictable manner, we can automatically infer the type (or potential types) of data we're working on in any given line with little effort.

Maintenance and Extensibility Issues

Inexpressive code can be a nightmare to maintain, and it often leads to tangled code, quickly becoming tough to modify without breaking something. It's even harder to extend with new features and functionality, and can become a bug-prone black box that's slow and challenging to develop for.

Hurting the Bottom Line

We pile up technical debt, which costs us increasingly painful amounts of time, loses our business money, and potentially hurts our reputation if this leads to missed deadlines or buggy releases.

On top of all this, the less expressive our code is, the more painful turnover becomes; onboarding new developers takes longer, as it will be a while before they're productive and know the codebase, and losing developers who do know it has a greater impact. And, of course, even when they're up and running, they and everyone else working on the application will simply be less productive than they could be.

Now, let's focus on how we should best use methods in Ruby and Rails to avoid some of these issues.

Comparing Similar Ruby and Rails Methods

As we touched on early in this post, Ruby and Rails come together to offer us a wide variety of methods that accomplish an array of everyday tasks and answer basic questions we might need to ask as programmers. At first glance, there might appear to be a lot of overlap, and we may think that many of these methods can simply be used interchangeably without issue.

Of course, as we know, these methods were named very carefully. Both Ruby and Rails were designed with close attention to what these methods do and don't do.

As a result, methods (excluding aliases) each behave slightly differently, with different intended purposes. Their correct use can help us provide specific context to clue in anyone reading our code to what's going on. Conversely, failing to do so can have the opposite effect. Let's take a look at some methods that sometimes get used interchangeably, to our detriment — starting with nil?, empty?, blank?, and none?.

`nil?`

This method does exactly what you'd expect: it tells us whether a value is nil. I generally like to use it to avoid doing things like !@user, but it can also indicate that a value is not a boolean, if that's not already clear. While it's difficult to misuse, it's easy to use another, technically valid but misleading method (such as blank?) in its place. Don't do this!

If a value can only be nil or truthy, nil? should always be used. Doing otherwise can mislead anyone modifying the code and potentially lead them to code for non-existent edge cases, which is both a waste of time, and adds unnecessary complexity and verbosity.

`empty?` (Nearly Opposite Method: `any?`)

empty? is useful when you want to check whether a collection array, string, hash, or other type of collection contains no elements. It's a straightforward and intuitive method that returns true if the collection contains no elements, and false otherwise.

For example, "".empty? and [].empty? both return true, because the string and the array do not contain any elements. Similarly, {}.empty? returns true because the hash does not contain any key-value pairs.

When you use empty?, you're communicating to other developers that the object you're checking should be a string or collection of some sort, and you're interested in whether it contains anything.

`blank?` (Opposite Method: `present?`)

blank? is often used to check if a string is empty. However, as you're likely aware, it checks for more than just empty strings; it will return true if a value is nil, false, or an empty array or hash. Essentially, it asks and answers !object || object.empty?, so we expect that the potential values are probably either nil, or an empty string, array, or hash — this is what you're communicating to other developers when you use this.

It is often used in place of empty? or nil? when it really shouldn't be.

`none?` (Opposite Method: `all?`)

This asks whether an array or hash is empty or only contains falsy values, and it's not too often misused. Sometimes, though, we might see someone use blank? instead.

This is a mistake. In addition to the fact that blank? asks: "is this object falsy or empty?", none? asks not only: "is this array or hash empty?", but also: "...or does this array have no truthy values?". [false, nil].blank?, for example, will return false, but [false, nil].none? will return true (likewise, any? would return false).

So not only have we introduced ambiguity, we've introduced a bug waiting to happen — and sadly, this one is not a feature.

`size` Vs. `length`

`size`

Put simply: size is for collections (ActiveRecord Collections, arrays), and should almost always be used for them. It never makes sense to call on a string.

`length`

length is generally for strings, and except in specific cases, shouldn't be used for collections (and probably never for arrays).

`boolean?` Methods in Ruby

Every Ruby boolean? method answers a yes or no question: "Is this value truthy or not?". Most Rails developers are aware of this and generally use the question mark to good effect, adding it to the end of the boolean methods that they write. But many may not be aware that Rails itself does, too.

Any Rails object automatically responds to a column with a ? on the end. So, a verified column on your user table can be called with @user.verified?. Setting up a standard for your team to only use these methods for boolean columns can help provide even more context for other (especially new) team members. They won't have to check the schema to see if verified is maybe a timestamp or even a string rather than a boolean.

And let's not forget the methods we write ourselves; generally speaking, any method we write that answers a yes-or-no question should return a boolean method, and end with a question mark.

For example, say we have a method that tells us whether or not a value is outside a particular limit. We can define something like this:

class Vendor
  def use_exceeds_limit?
    get_time_slot_use_count(1.month) > time_slot_tier.limit
  end

We can then conditionally email to let users who have exceeded the limit know they'll be charged at a higher rate for the month:

class DailyLimitCheckJob < ApplicationJob
  def perform(vendor)
    VendorMailer.alert_vendor_limit_exceeded(vendor) if vendor.use_exceeds_limit?
  end
end

So, great, we know that methods that end in a ? return a boolean. But what about other methods?

Note the method get_time_slot_use_count. It may be slightly more explicit than it needs to be, but it also tells us exactly what we're getting: an Integer. If it were get_time_slot_usage, we might wonder if it wasn't a Float, or even an ActiveSupport::Duration (with a return value like 173.hours). Being explicit helps us know what we'll get without looking at the actual implementation elsewhere in the Vendor class.

So, we can also apply this principle elsewhere: we probably don't need it for strings like name or title, but if we're assigning variables to dynamically hold parts of a query, we probably do want to name it something like query_string, rather than query.

The same is true of arrays and hashes: generally, a plural name (say, products) indicates that we're dealing with a collection. We often don't need to explicitly specify "hash", either; consider pagination_options. We know options in contexts like these are typically hashes. But sometimes we have products as a key-value pair collection; we should probably name this products_hash.

Further Resources

For those seeking a deeper dive into the world of Ruby, a number of resources are available.

Check out the books The Ruby Programming Language and Ruby Best Practices, the official Ruby docs, and Rails Guides.

Wrapping Up

In this post, we've walked through the pitfalls of inexpressive code, including a lack of clarity, a decrease in trust, and other problems. In contrast, well-crafted, expressive code can help avoid issues like technical debt, make our lives easier, and save money.

The design philosophy of Ruby prioritizes expressiveness and intuitiveness. Properly leveraging the tools Ruby provides to communicate your intent can eliminate the need for extensive comments and lead to substantial savings in both time and resources.

To fully unlock the power Ruby and Rails affords us, it's critical to be conscientious in your variable and method naming, and to use the right method for the job. Expressive code makes you — and everyone around you — more productive.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Using Action Policy for a Ruby on Rails App: The Basics

Roy Tomeij — 2023-10-04T00:00:00+00:00

To keep your app secure, you need to control who and what can access it. Access control can be categorized into authentication — "who" to allow — and authorization — "what" they can access.

Authentication is a subject for another day, but when it comes to user authorization, you generally have two ways to go about it: using a role-based or resource-based strategy.

In this two-part series, we'll take a deep dive into using the Action Policy gem for a Ruby on Rails blog application.

In this part, we'll cover the basics of Action Policy.

Let's get started!

Prerequisites

Ruby (we're using version 3.2.2)
Rails (using version 7.0.7)
Some experience using Ruby

Let's get into it by first defining resource-based authorization.

What Is Resource-Based Authorization?

Where role-based authorization focuses on setting user permissions according to predefined user roles, resource-based authorization enforces access by setting rules on the actual resources within an application. Each resource is associated with a policy that explicitly defines what a user can do on that resource.

Even though this article is focused on resource-based authorization, knowing the differences between the two authorization strategies means you are better equipped to know what each can do.

When to Use Role-Based Authorization

You should use role-based authorization for:

Simple applications - If you're working on an app with a straightforward permissions system and fewer user roles, then a role-based authorization strategy could work for you.
Well-defined user groups - If your app has well-defined user groups such as "admins", "editors", "writers", and so forth, use role-based authorization as it handles access control at the user role level.

When to Use Resource-Based Authorization

Resource-based authorization is great for:

Dynamic or complex access control - When your application's authorization needs evolve frequently or are determined by dynamic conditions, then a resource-based authorization system makes for a better choice.
Fine-grained control - Useful for when you need to allow or deny access to resources based on multiple conditions (for example, a Rails helpdesk app where users submit support tickets based on a list of dynamic categories). Assuming support staff are assigned tickets by category, this is a scenario where resource-based authorization would really shine.
Object-oriented control - Because resource-based authorization happens at the object level, defining complex object-oriented rules is easier when you use resource-based authorization techniques.

That said, whether you choose between role-based and resource-based authorization will depend on your application's unique characteristics.

Let's now turn our attention to the Action Policy gem.

The Action Policy Gem for Ruby and Rails

Action Policy is a flexible, extensible, and performant authorization framework for Ruby and Rails apps. It uses multiple caching strategies out of the box, making it very fast, especially if your authorization rules require database queries.

Another feature that makes this gem ideal for building resource-based rules is its ability to be customized. It provides several Ruby classes and modules that can be combined in many ways. You can pretty much set fine-grained access control rules beyond Rails controller, anywhere in your app.

Check out the Action Policy project homepage to learn more.

Now let's go over the Rails app we'll be building today.

Create the Rails App

Moving forward, we'll reference a Rails 7 blog application where users can Create, Read, Update, and Delete (CRUD) posts.

We'll progressively define an action policy for the Post model that offers resource-based access control. You'll learn how to use Action Policy to make this access control strategy work.

Go ahead and generate a new Rails application:

rails new blog_app

Note: We'll use Pico CSS styles for our example app, but you can use whatever you like.

Then quickly scaffold a Post resource that we'll use throughout the rest of the tutorial:

rails g scaffold Post title body

Now run the migration with rails db:migrate.

Setting up User Authentication

As much as this article is about user authorization, there's something important we need to cover: user authentication. Without it, any authorization policies we try to define later on will be useless. But there is no need to write authentication from scratch. Let's use Devise.

Installing Devise

Begin by adding the Devise gem to your Gemfile:

# Gemfile

gem "devise"

Then install it and generate a User model:

bundle install
rails generate devise:install
rails generate devise User
rails db:create
rails db:migrate

Also, remember to add config.action_mailer.default_url_options = { host: 'localhost', port: 3000 } to the app's development configuration.

Next up, let's implement a few basic user roles to test different user access scenarios for the Post resource.

Defining Basic User Roles

First, add the role column to the user table. Run this command to generate a migration:

rails g migration AddColumnRoleToUser role:integer

Note: We use an integer data type for the role, so we can use an enum — a quick and easy way to implement roles.

Now run the migration with rails db:migrate, then open up the User model and edit it:

# app/models/users.rb

class User < ApplicationRecord
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

         enum :role, { reader: 0, author: 1, editor: 2 }
end

With that done, let's shift our focus to installing Action Policy.

Setting up Action Policy for Ruby and Rails

Open the app's Gemfile and add the line below:

#  Gemfile

gem "action_policy"

Then run the command bundle install to install Action Policy.

Finalize the installation by running:

rails g action_policy:install

This should give us a base — the ApplicationPolicy class under app/policies:

# app/policies/application_policy.rb

class ApplicationPolicy < ActionPolicy::Base
end

Basic Usage of Action Policy

Action Policy's core foundation is a policy class, ApplicationPolicy, where you can define global configurations from which all policies can inherit. A good recommendation is to place all policies under app/policies, and to separate policies according to the resources in your app. For example, if you have a Post resource, the corresponding policy should be PostPolicy, CommentPolicy would accompany a Comment resource, and so forth.

One more thing to consider when working with Action Policy: rule definitions must happen within public methods in policy classes. Using a private method will raise an error.

Moving on, let's use this information to create a PostPolicy where we'll progressively build various levels of user access.

Creating Our First Policy

Create a new file named post_policy.rb under app/policies:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
    def show?
      true
    end
end

Here, we define a simple rule on the Post resource declaring that anyone can access a Post.

Associating Posts to Users

To work with this new post policy, we need to modify the Post resource we generated earlier so that it's associated with a logged-in user on creation (by adding a user_id column to the posts table):

rails g migration AddColumnUserIDToPost user_id:integer

Then run the migration with rails db:migrate.

Next, we need to modify the posts controller to account for this change. First, let's add user_id to the allowed post_params:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

    ...

    private

    def post_params
      params.require(:post).permit(:title, :body, :user_id)
    end
end

Then modify the create action:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

    ...

    def create
        @post = Post.new(post_params)
        @post.user_id = current_user.id # Devise gives us the logged in user as a helper method `current_user`

        respond_to do |format|
            if @post.save
                ...
            end
        end
    end

    ...
end

Next, let's secure the PostsController.

Implementing Authorization in Controllers

We need to modify the PostsController with a callback to only allow access to logged-in users:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

  before_action :authenticate_user!, except: :show

  ...
end

Next, let's use the new Post policy to add some basic user access controls:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
    ...

    def update?
    # post can only be updated by an author role or the post's author
      user.author? || (user.id == record.user_id)
    end

    def destroy?
    # a post can only be deleted by its author
      user.id == record.user_id
    end
end

Here, we define an access rule on the posts controller's update and destroy actions. We specify that a post's author (or a user with the "author" role) can update a post, but only a post's author can delete a post.

Quick tip: Action Policy can reference the current_user provided by Devise and assign it to a user within the policy.

Next, we need to use the access rule in the posts controller like so:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController
  before_action :set_post, only: %i[ show edit update destroy ]
  ...

  def update
    authorize! @post # Add this line
    respond_to do |format|
      if @post.update(post_params)
        ...
      end
    end
  end

  def destroy
    authorize! @post # Add this line
    @post.destroy
    ...
  end

  ...
end

With the access rule applied to the posts controller, our next course of action is to ensure the views are protected.

Protecting Views with Authorization

In the screenshot below, a user with the email <user2@example.com> and "reader" role is logged in. As you can see, this user can view the post created by <user@example.com>, and they even have access to the edit and delete links for this post. This is not ideal — the edit and delete links should only be available to the post's author.

Our first task will be to remove access to these links for users that aren't authors of a post. Since we've already defined this access rule, we only need to apply it to the show view using Action Policy's nifty allowed_to? method:

<!-- app/views/posts/show.html.erb -->

<%= render @post %>


<div>
  <% if allowed_to?(:update?, @post) %>
    <%= link_to "Edit this post", edit_post_path(@post) %>
  <% end %>

  <% if allowed_to?(:destroy?, @post) %>
    <%= button_to "Destroy this post", @post, method: :delete %>
  <% end %>

  <%= link_to "Back to posts", posts_path %>
</div>

With this rule applied, we can now refresh the post's show page and see what we get:

The same user who had access is now limited in what they can do while on a post's view.

Wrapping Up

In the first part of this two-part series, we've learned some of the basics around the authorization gem Action Policy.

In the second and final part, we'll explore some more advanced use cases.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to LiteStack for Ruby on Rails

Roy Tomeij — 2023-09-27T00:00:00+00:00

In this series of posts, we will look at LiteStack, a one-stop-shop solution that hosts and processes all your production data on a single machine. LiteStack (as the name suggests) makes use of SQLite to provide:

a database using the LiteDB adapter
an ActiveJob backend (LiteJob)
an ActionCable backend (LiteCable)
an ActiveSupport::Cache store (LiteCache)

In this first post, we'll introduce the basics of LiteStack and set up an example Rails application.

Let's begin!

An Introduction to SQLite

SQLite itself has been the go-to embedded database of many industries for decades. For example, it's widely used in native app development, testing environments, caching, and others.

Recently, though, it has attracted a lot of experimentation and extensions. One of the most popular extensions is Litestream, which can recover stream changes to an S3-compatible bucket. This means you get a replica of your production database at a very cheap price point and can recover from failure anytime.

Incidentally, this has made using SQLite as a production database for your Rails app a feasible option. Combined with a full-fledged development stack like LiteStack, it promises to make apps hosted on a single machine a reality. We are here to test this hypothesis and point out any obstacles in the way.

To do this, we need an example app that's complex enough to surface potential difficulties, but simple enough to fit this series.

Our Example Rails Application

We will write an app that transforms children's drawings using StableDiffusion on replicate.com.

As an example, here's a cute teddy bear drawn by my daughter, with a couple of StableDiffusion interpretations:

As a rough sketch, our app will cover the following steps:

The user uploads an image with a textual prompt (we'll show some advanced SQLite techniques here).
The user chooses an image style (e.g., "cartoon", "oil painting", "photorealistic", "3D rendering").
The processing happens in the background (this kicks off a LiteJob-powered job).
While the processing is underway, we show a placeholder image and update the logs sent over by the server. Once completed, we update it to show the actual image. This allows us to explore LiteCable as we replace the image via Turbo Streams.
We store the image prediction.
We use LiteCache to wrap computationally costly views.

These steps provide a scaffold for this series. The remainder of this post, though, will be concerned with setting up the app.

We start by creating a new Rails app called skAItch, using esbuild as our JavaScript bundler and SASS as the CSS preprocessor:

$ rails new -d sqlite3 --skip-action-mailbox -j esbuild -c sass skaitch
$ cd skaitch
$ bin/rails g action_text:install
$ bin/rails db:migrate

Install LiteStack

Next, we install LiteStack using the shipped generator:

$ bundle add litestack
$ bin/rails g litestack:install

After that is done, we complete the setup and start the development server:

$ bin/setup
$ bin/dev

Authentication and Tenants

Subsequently, we need a way to authenticate our users to associate prompts with them. Rather than using an incumbent like Devise, I chose to use a different approach. The authentication-zero gem can flexibly generate an authentication system, as opposed to including it as an engine. Conveniently, it comes with options such as:

authentication by token (for APIs)
two-factor auth
multitenancy
rate limiting
an OmniAuth interface
passwordless auth

I've chosen to add the --tenantable option because it's always a good idea to automatically scope your database records to accounts. Authentication-zero provides this with the AccountScoped model concern.

$ bundle add authentication-zero --group development
$ bin/rails g authentication --tenantable
$ bin/rails db:migrate

Let's add a first user via database seeds:

# db/seeds.rb
User.create(
  email: "julian@example.com",
  password: "mypassword123",
  password_confirmation: "mypassword123",
  verified: true
)

Prompt Scaffold

Now it's time to start writing our actual application logic. We start by defining the central model of our app: the Prompt.

We want our prompt to have a title, a description, and a reference to the account that created it. Furthermore, to test SQLite's "file system" capabilities, we would like it to store the prompt image in binary form:

$ bin/rails g scaffold Prompt title:string description:rich_text prompt_image:binary account:references
$ bin/rails db:migrate

A prompts resources entry is also added to config/routes.rb. Note that authentication-zero adds a before_action authenticating the user to ApplicationController by default.

  # config/routes.rb
    Rails.application.routes.draw do
+     resources :prompts

      get  "sign_in", to: "sessions#new"
      post "sign_in", to: "sessions#create"
      get  "sign_up", to: "registrations#new"
      post "sign_up", to: "registrations#create"
      resources :sessions, only: [:index, :show, :destroy]
      resource  :password, only: [:edit, :update]
      namespace :identity do
        resource :email,              only: [:edit, :update]
        resource :email_verification, only: [:show, :create]
        resource :password_reset,     only: [:new, :edit, :create, :update]
      end
      root "home#index"
    end

Furthermore, we include the AccountScoped concern in our Prompt model, which allows us to scope the stored records by logged-in account. We also validate that a prompt title and image are present.

  # app/models/prompt.rb
  class Prompt < ApplicationRecord
+   include AccountScoped

    belongs_to :account
    has_rich_text :description

+   validates :title, :prompt_image, presence: true
  end

As the final step to enable multitenancy, we have to connect the prompt to an account when it is created. We do this in the PromptsController:

  # app/controllers/prompts_controller.rb
  class PromptsController < ApplicationController

    # other actions omitted

    def create
      @prompt = Prompt.new(prompt_params)
+     @prompt.account = Current.account

      respond_to do |format|
        if @prompt.save
          format.html { redirect_to prompt_url(@prompt), notice: "Prompt was successfully created." }
          format.json { render :show, status: :created, location: @prompt }
        else
          format.html { render :new, status: :unprocessable_entity }
          format.json { render json: @prompt.errors, status: :unprocessable_entity }
        end
      end
    end

    # ...
  end

Connecting our Rails Application to Replicate.com

Replicate.com is a leading platform for running AI predictions on high-performance graphics cards. It features an API to create predictions, train and store models, etc. To use it, you have to obtain an API token from https://replicate.com/account/api-tokens.

Note: Running predictions on Replicate is subject to a charge.

Fortunately for us, there are official and unofficial clients that interface with the API. One such wrapper is the replicate-rails gem, which we will install now:

  # Gemfile

  # ... other gems

+ gem 'replicate-rails', require: 'replicate_rails'

$ bundle install

To store our API key securely, we are going to use Rails credentials:

$ EDITOR=vim bin/rails credentials:edit

  # config/credentials.yml.enc
  secret_key_base: YOUR_SECRET_KEY_BASE

+ replicate:
+   api_token: YOUR_REPLICATE_API_TOKEN

Editing and saving this file will encrypt your credentials, and it can only be opened by providing the correct RAILS_MASTER_KEY.

Now, we have to put it to use. As suggested in replicate-rails' README, we authenticate against Replicate in an initializer. We also define a webhook handler which (in our case) is just a class put into the same file. Note that I added a binding.irb breakpoint here for a first test of our functionality.

# config/initializers/replicate.rb
Replicate.client.api_token = Rails.application.credentials[:replicate][:api_token]

class ReplicateWebhook
  def call(prediction)
    binding.irb
  end
end

ReplicateRails.configure do |config|
  config.webhook_adapter = ReplicateWebhook.new
end

Replicate-rails also ships a default webhook controller, which calls the above handler. We only have to mount it in our config/routes.rb:

  # config/routes.rb
  Rails.application.routes.draw do
+   mount ReplicateRails::Engine => "/replicate/webhook"

    resources :prompts

    # ... more routes omitted
  end

To test the webhook locally, you have to set up a tunnel, for example, with Ngrok. The steps to set it up are beyond the scope of this article, but it's pretty simple. Please refer to the Ngrok docs.

You do, however, have to tell Rails that it may listen to your tunnel's URL. To enable this, add it to the allowed_hosts in config/application.rb:

  # ...requires omitted

  # config/application.rb
  module Skaitch
    class Application < Rails::Application
      # Initialize configuration defaults for originally generated Rails version.
      config.load_defaults 7.0

      # Configuration for the application, engines, and railties goes here.
      #
      # These settings can be overridden in specific environments using the files
      # in config/environments, which are processed later.
      #
      # config.time_zone = "Central Time (US & Canada)"
      # config.eager_load_paths << Rails.root.join("extras")
+     config.hosts << "YOUR_NGROK_URL"
    end
  end

Running a Prediction

Now we'll test creating a prediction. The only missing bit is to hook it up in our PromptsController. The general workflow is as follows:

We retrieve a model (in our case, the stable-diffusion-img2img model) from Replicate.
We grab a specific version (in our case, the latest one) to run the prediction against.
We run the prediction, specifying:
- A text prompt describing the image (in our case, our prompt's title).
- An image prompt that we have to provide as a Base64-encoded Data URL.
- A webhook to ping when the prediction completes (we point it at the webhook route provided by replicate-rails).

# app/controllers/prompts_controller.rb
  class PromptsController < ApplicationController

    # other actions omitted

    def create
      @prompt = Prompt.new(prompt_params)
      @prompt.account = Current.account

+     model = Replicate.client.retrieve_model("stability-ai/stable-diffusion-img2img")
+     version = model.latest_version
+     version.predict({prompt: prompt_params[:title], image: prompt_image_data_url}, replicate_rails_url)

      # respond_to omitted
    end

    # ...

    private

    # other private methods omitted

+   def prompt_image_data_url
+     encoded_data = Base64.strict_encode64(prompt_image.read)
+
+     "data:image/#{prompt_image.content_type};base64,#{encoded_data}"
+   end
+
+   def prompt_image
+     @prompt_image ||= prompt_params[:prompt_image]
+   end
  end

If you go to https://YOUR_NGROK_URL/prompts/new and upload an image together with a title, you will now trigger a prediction. When done, it will call back via the provided webhook. Because we set a binding.irb breakpoint there, the controller action stops in a REPL, and we can take a look around:

> prediction.succeeded?
=> true
> prediction.output
=> ["https://replicate.delivery/SOME_SIGNED_PATH/out-0.png"]

As we can see, our first run of an image-to-image generation succeeded. The prediction model returns a URL to the created image, which we will want to store locally (they are deleted at Replicate periodically). We will take a closer look at this next time.

Up Next: LiteDB Deep Dive

In this opening post, we introduced the LiteStack environment as an interesting alternative to host all of your Rails app's components on a single machine. We have furthermore set up an example app that talks to Replicate.com for AI image generation.

In the next part of this series, we will explore the first and central element of LiteStack — LiteDB — in more depth. We will look at some traits that make it uniquely powerful, common pitfalls, limitations, and tradeoffs.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Sidekiq for Ruby on Rails

Roy Tomeij — 2023-09-20T00:00:00+00:00

Sidekiq allows Ruby developers to maintain fast and responsive web applications by moving time-consuming tasks into the background.

With multithreading at its core, Sidekiq can process many jobs at once. This makes Sidekiq an important part of Ruby or Rails applications that handle heavy loads or perform tasks like sending emails or processing files. Without background processing, long-running tasks would block your application's main thread, resulting in slow response times and a poor user experience.

In this post, we'll focus on effectively utilizing Sidekiq to manage and process background jobs.

Let's get started!

The Basics of Sidekiq

In essence, Sidekiq is a full-featured background processing framework for Ruby. It lets you run background tasks concurrently, crucial for responsive and reliable web applications. Whether sending emails, resizing images, or processing CSV files, time-consuming tasks can be done behind the scenes using Sidekiq.

Sidekiq's architecture relies heavily on multithreading and the concept of "workers". However, as of Sidekiq 6.3.0, the Sidekiq::Worker module has been deprecated in favor of Sidekiq::Job as the terminology "worker" can be confusing:

Are you talking about a process? A thread? A type of job? I encourage developers to stop using the term worker.

Redis, a lightning-fast in-memory database, plays an essential role here, acting as the queue system that stores these background jobs.

Note: While Sidekiq does rely on Redis for storage, we won't be getting into the details of Redis configuration and usage in this article.

Installation and Setup of Sidekiq for Rails

To begin, you'll need to run Ruby version 2.5 or later and Redis server version 4 or later.

Now you're ready to install the Sidekiq Ruby gem.

Start by adding Sidekiq to your application's Gemfile:

gem 'sidekiq'

Next, install it in the directory of your project:

bundle install

Now that Sidekiq is installed, you'll need to configure it. In a Rails application, you configure Sidekiq as your ActiveJob adapter. Open config/application.rb and add the following line inside the application class definition:

config.active_job.queue_adapter = :sidekiq

You can further configure Sidekiq by creating a Sidekiq initializer to point to a Redis instance, for example. This is a special script in Rails that runs when your application starts up.

In config/initializers, create a file named sidekiq.rb. Inside this file, you can specify various Sidekiq configurations. For instance:

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/1' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/1' }
end

This configuration tells Sidekiq to connect to a local Redis server. Of course, in a production environment, you'd replace the local URL with that of your actual Redis server (this is a great place for an environment variable).

And that's it! With Sidekiq installed and configured, you're now ready to start creating and processing jobs in the background of your Rails application.

Setting Up a Sidekiq Job

Here's how you set up and initialize a basic Sidekiq job:

require 'sidekiq'

class SomeNameForAJob
  include Sidekiq::Job

  def perform(name, count)
    # Actual implementation of the task to run in the background goes here
  end
end

To actually queue a job to run, you call the perform_async method on the job class, like this:

SomeNameForAJob.perform_async('some_name', 42)

This line adds a job to the default queue that executes as soon as a job becomes available. The arguments 'some_name' and 42 are passed to the perform method SomeNameForAJob upon execution.

From here, the Sidekiq client pushes the job into a queue in Redis, and the Sidekiq server pulls that job out of the queue when it's ready to process it.

Creating Your First Sidekiq Job

In Sidekiq, jobs are represented by job classes, and the tasks they need to perform are defined in a perform method. Let's create a basic Sidekiq job:

class HelloNameJob
  include Sidekiq::Job

  def perform(name, times)
    times.times do
      puts "Hello, #{name}!"
    end
  end
end

In this code snippet, we've created the class HelloNameJob. This takes two arguments: name and times. When performed, it prints a greeting to the console a specified number of times. This is, of course, a simple example. In a real-world application, the perform method can contain any code you want to run in the background.

Now execute your job. Simply call the perform_async method on the class and pass in any arguments your perform method expects. Here's how we can schedule the HelloNameJob we've just created:

HelloNameJob.perform_async('Jeff', 5)

This will enqueue a job to print "Hello, Jeff!" five times.

If you need a job to be performed at a particular time, you can use the perform_in or perform_at methods. For example:

HelloNameJob.perform_in(5.minutes, 'Meredith', 3)

HelloNameJob.perform_at(2.days.from_now, 'Jeff', 2)

In these examples, 'Meredith' will receive her greeting three times in five minutes, while 'Jeff' will have to wait two days to receive his greeting two times.

Advanced Sidekiq Usage

Sidekiq is not restricted to offloading work to the background. It also offers advanced features that allow you to configure your jobs, including job retries and job prioritization.

Automatic Job Retries

One of Sidekiq's most powerful features is automatic job retries. By default, if a job fails due to an unhandled exception, Sidekiq will retry the job with an exponential backoff. You can customize the number of retries by specifying sidekiq_options in your class like this:

class ThisJob
  include Sidekiq::Job

  sidekiq_options retry: 10

  def perform(args)
  end
end

In this example, if a job fails, Sidekiq will retry it ten times before giving up.

Job Prioritization

Job prioritization is another advanced feature that Sidekiq provides. You can control the priority of your jobs by assigning them to different queues and setting the priority of each queue. Here's how you can specify a queue when defining a job:

class ThisJob
  include Sidekiq::Jobs

  sidekiq_options queue: 'critical'

  def perform(args)
  end
end

In this example, jobs for ThisJob will be placed in the critical queue. When starting your Sidekiq server, you can then specify the order in which queues should be processed.

These are just a couple of examples of how Sidekiq's advanced features can be leveraged to fine-tune your background job processing. With Sidekiq, you have the tools to ensure that your jobs are processed efficiently, reliably, and in a way that best suits your application's needs.

Monitoring and Scaling with Sidekiq

Sidekiq comes with standard features for monitoring and scaling your job processing capabilities. A web-based dashboard gives you a real-time view of your job queues. You can see the number of processed and failed jobs, as well as detailed information about current and scheduled jobs.

To use the Sidekiq dashboard, mount it in your Rails routes file. In config/routes.rb, add:

require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'

After adding these lines, you can access the dashboard by navigating to '/sidekiq' on your server.

Monitoring Sidekiq with AppSignal

For more comprehensive monitoring, Sidekiq integrates well with application performance monitoring tools like AppSignal. AppSignal's Sidekiq dashboard provides detailed insights into your Sidekiq jobs, including failed and retried jobs, job duration, and Redis memory usage.

This kind of information is invaluable for identifying bottlenecks as you scale your application. Beyond that, AppSignal's alerts help you stay on top of anomalies like queue length, so you know when your queues have grown.

Integrating AppSignal into your Sidekiq setup with Rails is simple. AppSignal's Ruby gem inserts into the Sidekiq server middleware without any extra configuration. This also works with ActiveJob!

You can even use AppSignal for a Sidekiq application that doesn't run on Rails, using the setup mentioned in the docs.

A Sidekiq Use Case

Two of the most common use cases for Sidekiq are asynchronous emailing and scheduled report generation. Let's take a closer look at one of these examples.

Emails, particularly those with large attachments or numerous recipients, can take a significant amount of time to send. By handling email sending in a background job, your application can continue to respond to user requests while an email is being processed.

Here's a simple example of a Sidekiq job for sending emails in a Rails application:

class ResetPasswordJob
  include Sidekiq::Job

  def perform(user_id)
    UserMailer.reset_password_email_email(user_id)
  end
end

In this example, ResetPasswordJob uses the UserMailer class to send a password reset email to a user. To enqueue an email to send, you call ResetPasswordJob.perform_async(user_id) from somewhere else in your application.

However, in a typical Rails application, you often don't need to create a separate Sidekiq job for sending emails. Rails provides the Action Mailer framework, which is used to send emails and integrates nicely with Active Job for background processing.

Here's an example of how you might use Active Job and Action Mailer to send a password reset email:

class UserMailer < ApplicationMailer
  def reset_password_email(user_id)
    mail(to: User.find(user_id).email, subject: 'Password Reset')
  end
end

# elsewhere in your application
UserMailer.reset_password_email(user_id).deliver_later

In this example, calling deliver_later on the mailer automatically enqueues the email to be sent as a background job. Behind the scenes, Active Job uses Sidekiq to manage these jobs.

But if you want more control over your jobs — e.g., you want to customize retry behavior or queue prioritization — or if you're dealing with tasks that don't integrate with Active Job as easily as Action Mailer, then it's appropriate to create separate Sidekiq jobs.

This flexibility, along with Sidekiq's performance and ease of use, makes it a valuable tool for handling various background tasks in Rails applications.

Wrapping Up

In this post, we walked through an introduction to Sidekiq, exploring its basics, diving into installation and setup, and creating and executing useful jobs.

We also touched on Sidekiq's more advanced features, looked at real-world use cases, and discussed its crucial role in improving the performance and scalability of Ruby applications.

Armed with the knowledge from this introduction, you're now equipped to start using Sidekiq more effectively in your own Rails apps.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to RuboCop for Ruby on Rails

Roy Tomeij — 2023-09-06T00:00:00+00:00

Good code has a lot to do with how readable it is. As developers, we more often read code than write it. As my Perl teacher told us many times: the flexibility of Perl's syntax was its best and worst trait at the same time. Ruby's syntax was influenced partly by Perl and is also quite flexible.

Whatever language you pick, set some guidelines to avoid overusing a language's flexibility. Style guides for Ruby abound on the web, and it's not difficult to pick a style nowadays. But there is not much point in having copious debates on style and whether a proposed change follows a guide. Style enforcement is best left to a tool.

Such tools are called linters and static code analyzers. The de facto standard for Ruby in that category is RuboCop. Here follows an introduction to RuboCop: what it is, how it helps developers, how to use it, and some key practical use cases.

What Is RuboCop for Ruby?

First, RuboCop is a great tool to lint a code base and ensure a specific coding style is followed (such as indentation, spacing, naming conventions, etc.). It's heavily configurable on that level through many rules, called "cops". Each can be activated (or not) and tailored (for example, the number of characters per line).

Ensuring a specific coding style is followed helps reduce developers' cognitive load in writing and reading code. It's not a matter of code looking "nice"; it's rather a matter of code looking similar to the rest of the code base.

The second side of RuboCop is its ability to analyze code statically for quick insights into code complexity and potential code issues, such as the misuse of variables.

As RuboCop is a little utility, it's easily run within a CI pipeline and in any developer's IDE. Thus, RuboCop's feedback can be integrated into the key places where it matters: when code is being written and before it's merged into the code base.

How to Use RuboCop for Ruby

Style is a matter of taste, so it is entirely subjective. Remember the main point behind using a linter: it's just there to help follow a chosen style because it gives us more capacity to do what matters in any project.

By default, RuboCop will enforce the style defined in the Ruby Community Style Guide. We can tailor it to our specific tastes and context, but let's rely on this basic set of rules to learn how to use RuboCop.

Simple Use of RuboCop

Adding RuboCop to a project is as simple as adding the rubocop gem to the development and test group in the Gemfile.

group :development, :test do
  gem 'rubocop', require: false
end

There are also several complimentary gems you can use, including:

Those last two will be useful for many of you.

Once the gem, or gems, is installed, we can run rubocop from within the root folder of the Ruby application.

$> rubocop *

This will print out a report of all "offenses" and how many can be automatically fixed.

For example:

$> rubocop *
Inspecting 807 files
...CCC....................
Offenses:

app/controllers/api/messages_controller.rb:46:25: C: Naming/MethodParameterName: Method parameter must be at least 3 characters long.

80 files inspected, 20 offenses detected, 7 offenses autocorrectable

Let's dissect the only offense we have kept in this example.

RuboCop's documentation explains that this issue:

Checks method parameter names for how descriptive they are. It is highly configurable.

This is followed by an explanation of the configuration that can be made for this specific rule (we will cover that soon).

What will be of interest to most are the examples of good and bad styles, as checked by this rule:

# bad
def bar(varOne, varTwo)
  varOne + varTwo
end

# With `AllowNamesEndingInNumbers` set to false
def foo(num1, num2)
  num1 * num2
end

# With `MinNameLength` set to number greater than 1
def baz(a, b, c)
  do_stuff(a, b, c)
end

# good
def bar(thud, fred)
  thud + fred
end

def foo(speed, distance)
  speed * distance
end

def baz(age_a, height_b, gender_c)
  do_stuff(age_a, height_b, gender_c)
end

We can apply this advice and rewrite the code to match the required style. RuboCop can often fix the issue it has found (we will look at this a bit later).

Configuring Rules in RuboCop

By default, out of the box, RuboCop comes with a default set of pre-configured rules. The documentation will tell you Rubocop's default rules.

Configuring RuboCop for a project is done through the .rubocop.yml file at the project's root. This can be tailored further in sub-folders, but I wouldn't recommend it. And, of course, it's also possible to have system-wide configuration files within the user's home folder: ~/.rubocop.yml or ~/.config/rubocop/config.yml.

Here is a simple example of such a configuration:

Layout/LineLength:
  Max: 99

Let's reuse the previous rule: Naming/MethodParameterName.

Naming/MethodParameterName:
  MinNameLength: 3
  AllowedNames:
    - as
    - at
    - in
    - ip
    - id
    - to

There are a lot of things that can be tailored in the RuboCop configuration file. The documentation can help you pick the rules you want to use, but a few other options are available to give you a starting point beyond the default rules.

The Ruby style guide matches RuboCop's default configuration, for example. But you can also find other guides like the Relaxed Ruby Style, and some companies - such as Shopify - share theirs.

Arguments in RuboCop

RuboCop can be run without arguments for a basic check, but we can add on a few arguments for different effects:

-a : to auto-correct offenses, but only when it's safe to do so.
-A: to auto-correct offenses, even when it's not safe to do so.
-E: to display extra details for each offense listed.
-S: to display style guide URLs in the offense messages.

Those are the main options we might be interested in. -E and -S are great to add whenever you run RuboCop to check the code base. They both give more details for developers to figure out what to do. -a and -A are handy to quickly take care of the "small" issues, but beware of the automatic side. It might do well enough for minor style offenses, such as spaces, but other things might not be that simple.

Bending the Law

In some rare cases, you might want a RuboCop rule or rules to be disabled around one or several lines in a file. This is generally frowned upon, as it creates a precedent many developers won't hesitate to replicate. Yet it might still be useful to know how to do this hack for certain cases.

The way to disable a rule and then re-enable it is to use a pair of comments:

# rubocop:disable Layout/ClassStructure, Style/AndOr
[...]
# rubocop:enable Layout/ClassStructure, Style/AndOr

This will disable and enable two rules: Layout/ClassStructure, and Style/AndOr.

In the previous example, you can also disable whole sections (or departments) of rules by using the department name: Layout or Style.

Finally, you can also disable all the rules in one go.

# rubocop:disable all
[...]
# rubocop:enable all

RuboCop in Practice

Now that we have seen what it is and how to use it, let's review how we might use RuboCop in our day to day.

Local Uses

As discussed before, a linter's purpose is to ensure a standard style is used throughout a code base. Standards are how we help each other work together without putting up complex conversion and compensation schemes. When building a house, standards are the way every carpenter knows how to work together. The actual details will be a bit different from one house to another, but, using the standard, the work different carpenters do will match.

RuboCop helps software engineers work together by keeping their code up to a certain standard style.

The simplest way to use RuboCop is to run it before making a commit or once in a while before pushing commits to a remote repository. As described in the first part of this post, using rubocop * will run a check of the code base and display any offenses for you to fix.

You can then use the -a or -A option on another run to get some or all offenses fixed by RuboCop.

$> rubocopo -a

You can also fix them yourself, of course.

Within an IDE

Some IDEs and text editors have RuboCop plugins, so you can get notices about style offenses while working directly on code. The only issue with these is the configuration, which can be fiddly.

Integration with CI

It's important to run RuboCop within CI pipelines.

Usually, we run linters such as RuboCop as part of a pipeline's first steps to notify the author of code changes about any offense quickly. We also consider such infractions important, so they will block the Pull or Merge request in its tracks. A merge should not be possible if the style is not correct.

If your project relies on GitHub's Actions, you can find a RuboCop action. You can build similar steps for any CI/CD pipeline out there. The principle is simple: run RuboCop against the code base as an early block within the pipeline; if there is an infraction, it will return a non-zero code, and the CI will treat that as a failure. The output can be used more easily as an artifact.

The -f option will allow you to tailor the output format. This can be practical when running RuboCop in the CI pipeline: in markdown, HTML, or GitHub. I'd advise starting with markdown or HTML, but you should also consider the fairly readable default format.

Adopting RuboCop in a New Project

Many projects start without a coding style or diverge from one for some reason. Getting such code bases back in the fold is often challenging and hard work.

The first run of RuboCop in such a context will usually output a fairly long list. Trying to solve all offenses at once is foolish. Instead, RuboCop allows us to generate a to-do list.

Running RuboCop with the --auto-gen-config option will generate the todo file. This file will disable any rule that is not respected. It's located within the project folder .rubocop_todo.yml.

Disabling all the rules that are not respected rather than fixing them might seem counterproductive. But we don't stop here. This step ensures we don't face a mountain of work. We can remove a rule from the ignore list and fix all offenses against it, working our way, one rule at a time, through the to-do list and the code base.

This approach — of ignoring infractions to one or more rules within multiple files and then fixing the infractions one by one in many files — is doable, but will require a lot of back and forth between files. You might end up changing several files and several lines multiple times over the course of the process.

Another method is to use a "per file" approach. Using the following syntax within the to-do list file, you will ignore infractions to all cops in the files listed.

AllCops:
  Exclude:
    - "app/models/user.rb"
    - "app/controllers/users_controller.rb"

You can work your way through those files one by one: removing the file from the list, running RuboCop against it, and then fixing all offenses.

Whichever approach you pick, you must remember this is a step towards improvement through continuous learning and effort. You need a strategy and vision that works well with that: include enough Slack time within the team schedule, or set specific times aside to work on this issue.

Making RuboCop Easier to Use with Standard Ruby

RuboCop and coding style guides have helped us determine standards for each project and team. We should only debate what code is doing, not how it looks.

Some Ruby developers have made the point that we can even forego adjusting RuboCop's configuration. The idea is simple: after so many years working with Ruby, we more or less know how Ruby code should look, so a style that works for most will be quite okay.

This approach is known as Standard Ruby. It can also be completed with plugins, including one for Ruby on Rails projects.

Wrapping Up

RuboCop is, alongside Bundler, a good friend to any Ruby developer. As we have seen, it's easy to run it locally to get a list of offenses against a project's style standard.

We have also seen how RuboCop can fix a lot of those offenses on its own, and how to work our way through a long list of offenses if we start from a large backlog.

RuboCop is a tool in your toolbox; code style is only meant to help a team work together and shouldn't be a perpetual topic of debate. Yet, we have also seen that if you find it tedious to figure out a style for your project, you should look into projects such as Standard Ruby.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Secure Your Ruby App with JSON Web Tokens

Roy Tomeij — 2023-08-23T00:00:00+00:00

If a web application involves users, as a matter of course, their data should be protected and secured.

Securing a web application can mean several things. In this post, we'll discuss a subset of web security that involves authentication using JSON Web Tokens (JWTs) and the Ruby on Rails web application framework.

Let's get started!

What is a JSON Web Token?

A JSON Web Token is an internet standard defined by the Internet Engineering Task Force (IETF) as a: "compact, URL-safe means of representing claims to be transferred between two parties".

Here, "claims" refers to assorted pieces of information about a subject. A claim is represented as a name/value pair where the name is always a string, and the value can be any JSON value.

The Basic Structure of JSON Web Tokens

Delving into the intricacies of JWTs is out of the scope of this post. That said, it's worth knowing the structure of JWTs.

A JWT consists of three parts, separated by a period: the header, payload, and signature.

An example JWT could look like the following:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.
eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.
SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

For the sake of readability, each part of the token starts on a new line. In practice, the parts are joined.

This token is sent from a server to a client. The client sends the token to the server to identify itself and have a request processed.

The first part, the header, contains information about the algorithm used to generate the token and the type of token. If decoded, we get something like:

{
  "alg": "HS256",
  "typ": "JWT"
}

The second part, the payload, contains a set of claims about the user. In most cases, this would be the client in a client-server setup, and it could look like:

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

The signature (the last part of a signed JWT) validates the token. It is generated by encoding the header and the payload using Base64url Encoding — RFC 4648 and then concatenating them with a period separator.

Essentially, what happens with the signature bit is this:

HMAC_SHA256(
  secret,
  base64urlEncoding(header) + '.' +
  base64urlEncoding(payload)
)

HMAC_SHA256 is a type of keyed hash algorithm constructed from the SHA-256 hash function that hashes the signature. The choice of the cryptographic algorithm comes from the "alg": "HS256" in the header. If the token is unsigned, it'll only have the header and payload without the signature.

JWTs Vs. Other Authentication Methods for Your Ruby App

JSON Web Tokens, as the name implies, are token-based. On the other end of the spectrum, we have session-based authentication: a more traditional way of authenticating users. The flow of session-based authentication is quite different from that of token-based authentication.

The flow of session-based authentication might look like the following:

A user or client sends a request which contains user credentials.
The server authenticates the user, stores a session, and returns a session ID stored as a cookie in the browser.
The client sends the cookies along with its subsequent requests to the server.
The server inspects the session information presented and, if valid, authenticates the user and returns the requested information to the client.

While session-based authentication is mostly used for client-server connections, token-based authentication is often used with server-server connections (e.g., between two APIs).

One important difference to note, however, is that with session-based authentication, the authentication state is handled on the server — while tokens are managed on the client.

Why Use JWTs for Authentication?

Aside from being relatively simple to implement, there are a few other advantages of using JSON Web Tokens for authentication, such as:

They are stateless, meaning that a session store is not necessary. The token itself contains all of the user information, so there is no need to query a database or authentication server for information on each request.
JWTs are generally more performant than most traditional methods of authentication (as long as the server doesn't do any lookups against a database or store to authenticate a user), making them quite efficient.
They also offer solid security guarantees, in that signed JWTs provide safeguards so an attacker or client can't modify the tokens to gain access to protected data.

JWT Best Practices for Ruby Apps

It goes without saying that the secret keys used to sign JWTs should be long, random, and have complex character combinations. This ensures that the keys are adequately safe and it's hard for attackers to brute-force them.

The secret keys Rails generates are safe for the most part, but safety guarantees are nullified if you accidentally commit keys and reveal them.

It's also important to use Transport Layer Security (TLS) when transporting tokens between parties on a network. TLS can mitigate a man-in-the-middle attack (both token and session-based authentication methods are prone to such attacks).

Implementing JWT authentication in a Rails App

Let's take a look at a token-based authentication flow:

Unlike session-based authentication, a stateful authentication technique where we use sessions to keep track of an authenticated user, token-based authentication with JWTs is stateless; there's no need to store any information about a user's authentication state on the server. This simplifies application design.

In this post, we'll assume that our application is split into a frontend and a backend. Authentication occurs on the backend, so we will be building a Rails API backend with authentication.

The sample code in this post is based on Rails 7.0.5 and Ruby 3.2.2.

Using the `jwt` and `bcrypt` Ruby Gems

We'll need two gems for our application: jwt and bcrypt.

jwt is a Ruby implementation of the RFC 7519 OAuth JSON Web Token standard. bcrypt is a Ruby binding for the OpenBSD bcrypt() password hashing algorithm.

You can follow along with the sample code in this code repo.

Note: jwt is not the only solution for working with JWTs; another well-known gem is devise-jwt, which provides JWT authentication for Devise and Rails. But we'll focus on jwt in this post.

Let's get started.

Building our Rails API

The first thing we need is an API application. We'll create one with:

rails new jwt_rails_api --api

The --api option here preconfigures a smaller stack of Rails for API applications only.

Inside the Gemfile, we can add our first dependency, jwt. Our second gem, bcrypt, is already in the Gemfile of a newly-generated Rails application — we only need to uncomment it.

We need bcrypt to securely hash user passwords in the database. It's important to note that we won't use bcrypt directly. We'll leverage Active Model's has_secure_password class method, which depends on bcrypt.

Ignoring the default gems that come with a new Rails application, our Gemfile should look something like:

gem 'jwt', '~> 2.7'
gem "bcrypt", "~> 3.1.7"

Now is a good time to install our gems with bundle install.

Generate `User` and `Product` Models

Next, we'll generate two models: User and Product. User will be the model to represent users and we'll authenticate it to allow access to products, represented by the Product model.

rails g model User username:string password_digest:string
rails g model Product name:string description:text

After running our migration with rails db:migrate, our setup with models is complete, and our schema, found in db/schema.rb, should now look similar to this:

ActiveRecord::Schema[7.0].define(version: 2023_05_28_224534) do
  create_table "products", force: :cascade do |t|
    t.string "name"
    t.text "description"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end

  create_table "users", force: :cascade do |t|
    t.string "username"
    t.string "password_digest"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end
end

It's important to point out at this stage that we're deliberately ignoring potential issues such as database constraints, validations, ensuring uniqueness, etc. For the sake of this post, we'll disregard handling errors, among others. Our purpose here is to demonstrate JWTs in action as a form of stateless authentication to secure our Ruby application. In a production application, you'd want to make sure all of these are covered.

Build a `jwt` Gem Wrapper

The next step is to build a wrapper around the jwt gem we installed earlier. We'll use this wrapper to encode and decode claims from the server to the client. For this, we'll create an app/lib folder.

The reason we're not using the lib folder that comes with Rails is that it's not autoloaded. Everything under app is autoloaded and eager-loaded by default, making for a simpler setup in our case.

Our wrapper class is found in app/lib/json_web_token.rb and looks like this:

class JsonWebToken
  JWT_SECRET = Rails.application.secrets.secret_key_base

  def self.encode(payload, exp = 12.hours.from_now)
    payload[:exp] = exp.to_i

    JWT.encode(payload, JWT_SECRET)
  end

  def self.decode(token)
    body = JWT.decode(token, JWT_SECRET)[0]

    HashWithIndifferentAccess.new(body)
  end
end

The main methods here are encode — to encode user information — and decode — to later decode user information in the server. Note how we're delegating the encoding and decoding tasks to the jwt gem through JWT.encode and JWT.decode.

At this point, you can already test this class in your Rails console:

data = {"name"=>"AppSignal"}

JsonWebToken.encode(data)
# => "eyJhbGciOiJIUzI1NiJ9.eyJuYW1lIjoiQXBwU2lnbmFsIiwiZXhwIjoxNjg1NDI0MjI5fQ.zWJyFHH8Pa6phBOU99XgtRntyfZQSOTX4TdwOxFY9gY"

JsonWebToken.decode(JsonWebToken.encode(data))
# => {"name"=>"AppSignal", "exp"=>1685424262}

Notice how the result of JsonWebToken.encode(data) is split into three parts by a period, producing the header, payload, and signature. We have the signature bit because we signed our payload with a secret key that Rails provides through Rails.application.secrets.secret_key_base.

Back to the `User` Model

Now will be a good time to visit our User model at app/models/user.rb. All we need to do here is add the has_secure_password class method:

class User < ApplicationRecord
  has_secure_password
end

has_secure_password securely hashes our users' passwords in the database.

Create a Sample User and Product in Rails

Now we can hop into the Rails console to generate a sample user and product in our database and test the security of our application:

User.create(username: "emi", password: "password")
Product.create(name: "Rad Ruby", description: "A book collection of Ruby tips")

You can place the same piece of code in your seeds.rb file to save some typing in case you reset your database.

Using JWTs in Rails Controllers

In the next steps, we'll implement security using JWTs inside our controllers. A good place to start is the ApplicationController at app/controllers/application_controller.rb:

class ApplicationController < ActionController::API
  before_action :authenticate

  rescue_from JWT::VerificationError, with: :invalid_token
  rescue_from JWT::DecodeError, with: :decode_error

  private

  def authenticate
    authorization_header = request.headers['Authorization']
    token = authorization_header.split(" ").last if authorization_header
    decoded_token = JsonWebToken.decode(token)

    User.find(decoded_token[:user_id])
  end

  def invalid_token
    render json: { invalid_token: 'invalid token' }
  end

  def decode_error
    render json: { decode_error: 'decode error' }
  end
end

Here, we create an authenticate method to decode JSON Web tokens that users send us. If we can successfully verify the token, we return the User object that represents the user making the request. We're mostly interested in the happy path here and will bypass a lot of checks.

Defining the authenticate method in the ApplicationController and setting it up as a before_action secures every controller inheriting from it. A request to any other controller will need a valid JWT to access those controllers (because every other controller will inherit this main controller).

Next, we need an AuthenticationController to which users can send requests and get a signed JSON Web Token from our server. This controller should be placed at app/controllers/authentication_controller.rb and may look like this:

class AuthenticationController < ApplicationController
  skip_before_action :authenticate

  def login
    user = User.find_by(username: params[:username])
    authenticated_user = user&.authenticate(params[:password])

    if authenticated_user
      token = JsonWebToken.encode(user_id: user.id)
      expires_at = JsonWebToken.decode(token)[:exp]

      render json: { token:, expires_at: }, status: :ok
    else
      render json: { error: 'unauthorized' }, status: :unauthorized
    end
  end
end

When a request hits this controller, because it's a user asking for a token, we don't want to initially authenticate them. The purpose of this controller is to respond with a token the user can use to access the rest of the resources on our server. Hence the need for skip_before_action :authenticate on the second line.

In the login action (the one that users hit for a token), we grab the username and password from the parameters that come with the request to this controller. If we can authenticate the user — that is, verify if their username and password match what we have stored in our database — then we present them with a signed token and information about when that token expires.

In our case, we won't use the expiry period of the token. But in a production application, that could be used to revoke access to a resource.

We'll go through all these steps using curl later on.

Testing Our Ruby Application with a Protected Resource

We partly covered the flow in the earlier 'Implementing JWT authentication in a Rails App' section's flow diagram. To completely cover everything and test out all the steps in the flow diagram, we need a resource to protect.

We have a Product model already. We now need a controller for the product model and routes to access the product and tokens.

Let's create a controller with rails g controller Product index so we have something like:

class ProductsController < ApplicationController
  before_action :authenticate

  def index
    @products = Product.all

    render json: @products
  end
end

Of course, we need a route to access these controllers. Our config/routes.rb should look something like:

Rails.application.routes.draw do
  post 'login', to: "authentication#login"
  get 'products', to: "products#index"
end

Now we'll test with curl to see if everything works as expected. Note that we already have a user to authenticate and a product resource to access.

Let's try getting a JWT with a user that doesn't exist:

curl -H "Content-Type: application/json" -X POST -d '{"username":"manny","password":"password"}' http://localhost:3000/login

We should get the following response:

{"error":"unauthorized"}

Now try the same with a user that we created earlier:

curl -H "Content-Type: application/json" -X POST -d '{"username":"emi","password":"password"}' http://localhost:3000/login

This should give us a signed JSON web token that could look like this:

{"token":"eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoxLCJleHAiOjE2ODU0NTEyMTR9.1UEYAbmFOSF93yp9pJqNEzkdHr3rVqutPNZWRIPDYkY","expires_at":1685432077}

Let's keep this token for a second and try accessing a product resource with a bad token (I changed a random character in the token):

curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoxLCJleHAiOjE2ODU0NTEyMTR9.1UEYAbmFOSF93yp9pJqNEzkdHr3rVqutPNZWRIPZYkY" http://localhost:3000/products

And we should get:

{"decode_error":"decode error"}

However, if we make the same request to access the product resource with the valid token we got from the server previously:

curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoxLCJleHAiOjE2ODU0NTEyMTR9.1UEYAbmFOSF93yp9pJqNEzkdHr3rVqutPNZWRIPDYkY" http://localhost:3000/products

We're granted access to the product resource:

[{"id":1,"name":"Rad Ruby","description":"A book collection of Ruby tips","created_at":"2023-05-29T19:33:30.826Z","updated_at":"2023-05-29T19:33:30.826Z"}]

That's it! We've successfully secured our Ruby application with a JSON web token!

Wrapping Up

In this post, we discussed JSON Web Tokens and how they work. We first covered the basics of JWTs, including their structure and some best practices. Then we implemented a simple JWT authentication using the jwt gem.

I hope you've found this post helpful. Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Advanced Usages of Devise for Rails

Roy Tomeij — 2023-08-02T00:00:00+00:00

In part one of this series, we introduced Devise using an example app to explore modules, helpers, views, controllers, and routes.

In this part, we'll explore more advanced usages of Devise, specifically the use of OmniAuth, API authentication, and Authtrail.

Let's dive straight in!

Authentication with OmniAuth for Ruby

Nowadays, almost every web application you come across will offer you the option to log in via a wide selection of authentication providers, ranging from social networks like Twitter and Facebook to Google, GitHub, and many more.

In many cases, this convenient multi-provider authentication is powered by a library called OmniAuth. OmniAuth is a flexible and powerful authentication library for Ruby that allows you to integrate with multiple external providers.

It provides a simple and unified API for connecting to various OAuth providers. OmniAuth is particularly useful in situations where you want to give your users the option to sign up or log in using their social media accounts. With OmniAuth, you can easily add social login functionality to your Rails application.

When OmniAuth is used alongside the Devise gem, it becomes even easier to manage user authentication and authorization. You can take advantage of Devise's built-in authentication features, while using OmniAuth for external provider sign-in options.

Getting Started with OmniAuth and Devise

As mentioned, OmniAuth allows you to integrate with a number of third-party authentication providers. For the purposes of this article, we'll use GitHub.

Install OmniAuth Gems

In your app's Gemfile, add the following lines:

# Gemfile

gem 'omniauth'
gem 'omniauth-github' # each provider will have their own gem to work with Omniauth

And if you're using the OmniAuth 2.0+ gem, you'll also need to add:

#  Gemfile

gem 'omniauth-rails_csrf_protection'

The omniauth-rails_csrf_protection gem ensures that any GET requests to the OAuth flow are disabled. It also inserts a Rails CSRF token verifier before the OAuth request phase. These two actions are meant to mitigate cross-site forgery attacks aimed at the OAuth authentication flow.

Next, run bundle install to install the gems.

Create a New GitHub OAuth App

Now we need to create a new OAuth app on GitHub. This app will act as a user with authentication permissions which can be easily revoked, if needed.

The first step is to go to the settings page under your GitHub account profile. Then, on the left-hand menu, click on "Developer settings". You'll see a screen like the one below, where you can create a new OAuth app:

When you click on "Register new application", you'll get a screen like this one:

Fill in the form details as follows:

Application name - Give your new OAuth app an appropriate name.
Homepage url - For now, use http://localhost:3000/. In production, you would use your app's actual homepage url.
Application description - Not necessary, but you can still have one if you have many apps and need to differentiate between them.
Authorization callback url - This is required input and will generally follow an OAuth callback url format like http://<app-url>/users/auth/<application-provider>/callback. That said, some OAuth providers like Google may not follow this format, so you need to take note of that.

With that done, click on "Register application". In the following screen, generate a new app secret and note it down someplace safe (as it will only be shown to you once).

Configure the Devise Initializer

Open up the Devise initializer config/initializers/devise.rb and navigate to the OmniAuth section specific to GitHub. It will likely be commented out, so uncomment it and edit it with your new GitHub OAuth app's ID and secret:

# config/initializers/devise.rb

...
  config.omniauth :github, ENV['GITHUB_APP_ID'], ENV['GITHUB_APP_SECRET'], scope: 'user,public_repo'
...

Create the OmniAuth Callbacks Controller

If you have already generated Devise controllers, you'll find the OmniauthCallbacksController ready for you to customize accordingly. If yours isn't there, just create one manually, and edit it as follows:

# app/controllers/users/omniauth_callbacks_controller.rb

class Users::OmniauthCallbacksController < Devise::OmniauthCallbacksController
  # You can easily add more OmniAuth providers here
  ...

  def github
    @user = User.from_omniauth(request.env["omniauth.auth"])
    sign_in_and_redirect @user # sign_in_and_redirect is an OAuth method
  end
  ...
end

Some things to note about the above code:

from_omniauth is a method we'll implement inside our User model.
sign_in_and_redirect is a method inside of OAuth.

Add a Migration to Modify the User Model

We now need to add some columns to the User model, specifically the provider column and a uid column:

bundle exec rails g migration AddOmniauthToUsers provider:string uid:string

Run bundle exec rails db:migrate to finalize this step.

Make the Devise Model Omniauthable

Here, we'll need to edit our User model by adding the Devise Omniauthable module to it:

# app/models/user.rb

class User < ApplicationRecord
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable,
         :omniauthable
  ...
end

After this, add the from_omni_auth method. This will be called from the Users::OmniauthCallbacksController that we just set up:

# app/models/user.rb

class User < ApplicationRecord
  ...

  def self.from_omniauth(auth)
    where(provider: auth.provider, uid: auth.uid).first_or_create do |user |
      user.provider = auth.provider
      user.uid = auth.uid
      user.email = auth.info.email
      user.password = Devise.friendly_token[0,20]
    end
  end

  ...

end

Now we're left with just one more thing: adding the login links to our Devise views.

Setup the Login Links

By default, Devise will automatically add the appropriate provider's login link for you in the user registration and login views. This link will use the GET method, but we know that OmniAuth 2.0+ prefers POST requests. So, we need to disable the link and insert our own using POST requests:

# app/views/users/sessions/new.html.erb

# notice we use a button_to which results in POST requests by default

<%= button_to "Sign in with Github", user_github_omniauth_authorize_path %>

With that, we've successfully set up a Ruby on Rails 7 app with Devise and GitHub OAuth authentication. You can grab the full source code of the companion app here.

Next, we'll get into another advanced use case: using Devise to authenticate an API call.

API Authentication with Devise for Ruby

Today, it's not uncommon for users to expect to be able to connect to your app via an API. In this section, we'll look at how we can safely authenticate such user requests using Devise.

Where browser-based authentication is generally cookie-based, most API authentication happens via tokens called JSON Web Tokens (or simply JWTs), which are passed around in the headers.

Tip: For the purposes of this section, we'll assume we're working with a Rails API-only app. To follow along, create one with rails new app_name --api

The JWT-based Authentication Flow

As we mentioned, API authentication is based on JWT tokens, and it's important that we understand how a JWT-based authentication flow happens. Basically, it follows a general sequence of steps as outlined below:

A user client makes a call to the API app.
The API app responds with a JSON Web Token (JWT), an authentication token that can be used in place of cookies.
Subsequent requests by the user client are made using this token in the Authorization header.
The user can then hit the Devise 'session destroy' action, which results in the destruction of the token and the user being logged out.

Now, let's put this flow into action, starting with what is termed cross-origin resource sharing (CORS).

Setting Up CORS

CORS sets up our API app to allow requests from external sources. CORS is a HTTP-based security policy that defines how external requests will be handled by your application. By default, CORS will block any request that originates from a domain different to the one that made the initial request (in other words, a request coming from a different "origin").

To handle CORS properly, we'll use the nifty gem rack-cors. In the Gemfile, uncomment the line below, then run bundle install:

# Gemfile
gem 'rack-cors'

Also, open up a corresponding CORS initializer file and modify it as below:

# config/initializers/cors.rb

Rails.application.config.middleware.insert_before 0, Rack::Cors do
  allow do
    origins "*"

    resource "*",
      headers: :any,
      methods: [:get, :post, :put, :patch, :delete, :options, :head],
      expose: %w[Authorization Uid]
  end
end

Some important notes about what we've just done:

origins "*" - Simply means that our API app can now receive requests from any other source.
expose: %w[Authorization Uid] - By default, the rack-cors gem doesn't expose the authorization and UID headers, but we need them since we'll be passing authorization tokens through.

With that done, let's install Devise and the accompanying Devise-JWT gem.

Add Devise and Devise-JWT Gems to Your Rails App

The devise-jwt gem is an extension of Devise that will allow us to work with JWT tokens. Add the gems to Gemfile, then run bundle install:

# Gemfile

gem 'devise'
gem 'devise-jwt'

Run the Devise install generator bundle exec rails g devise:install.

Generate and Configure the Models

We need to set up two models: the normal Devise user model (bundle exec rails g devise User), and a model that we'll use for the revocation strategy (in other words, how a user will sign out of the API):

bundle exec rails g model jwt_revocation

We modify the normal Devise user model for API authentication by adding the JWT token authenticatable module and defining the token revocation strategy to use our second model, JwtDenylist:

# app/models/user.rb
class User < ApplicationRecord
  devise :database_authenticatable, :registerable,
  :jwt_authenticatable, jwt_revocation_strategy: JwtDenylist
end

Next, we configure our second model by referencing the revocation strategy and the revocation table to be used:

# app/models/jwt_denylist.rb

class JwtDenylist < ApplicationRecord
  include Devise::JWT::RevocationStrategies::Denylist
  self.table_name = 'jwt_denylist'
end

In the following section, we'll outline token revocation and why we need it.

The Importance of Token Revocation

Why is token revocation important? Because JWT tokens are stateless. The server knows nothing about them other than signing them. In such a scenario, the server has no way to sign out a user by revoking the corresponding token. Since there's no way to revoke individual tokens, we need to build one out and tell the server to use it.

When revoking a token, what's really happening under the hood is that a unique piece of the token, the jti (JWT ID), is extracted and used according to the defined revocation strategy.

Of course, this raises another question of what a token revocation strategy is. In a nutshell, it's a definition of how token revocation will be handled by your server. There are three basic revocation strategies:

JTIMatcher strategy - Here, a unique column called "jti" is added to the user model, which also acts as the revocation table. Whenever a user makes a request, the jti in the header is matched against the stored tokens and access is allowed only when a match is found.
Denylist strategy - For this one, the jti and the token expiry (exp of revoked tokens) are stored in a database table. For every request made by a user, a check is made against this table comparing the user's current token jti against the revoked ones in the database. If a match is found, that user's requests are denied.
Allowlist strategy - In a way, this strategy is similar to the first strategy, except that here the table storing the JWT IDs is in a one-to-many relationship with another table storing the user tokens. Whenever a request is made, the user's jti stored in the Allowlist table is matched against what is stored in the matching table with tokens. Access is only allowed if a match is found.

Obviously, this is a very simplified overview of token revocation, and you can learn more about it here.

Set Up a Signing Key for JWT Tokens

Since we'll be using secure tokens to authenticate users and their requests, we need a way of signing them. This is where our secret key comes in. It is recommended that you generate a new key different from the Rails secret key, the secret_key_base.

Run bundle exec rake secret to generate a unique key, then edit the Devise initializer to include this key:

# config/initializers/devise.rb
Devise.setup do |config|
  ...
  config.jwt do |jwt|
    jwt.secret = ENV['DEVISE_JWT_SECRET']
  end
  ...
end

Finally, let's set up the controllers.

Controller Set Up

The final step to implement Devise for API authentication is to set up our controllers. For the sake of simplicity, we'll set up two controllers: one to handle registration and the other one for sessions.

Let's create these manually, starting with the registrations controller:

# app/controllers/users/registrations/registrations_controller.rb

class Users::RegistrationsController < Devise::RegistrationsController

 respond_to :json

 private

 def respond_with(resource, _opts={})
  successful_registration && return if resource.persisted?
  failed_registration
 end

 def successful_registration
  render json: { message: "You've registered successfully", user: current_user }, status: :ok
 end

 def failed_registration
  render json: { message: "Something went wrong. Please try again" }, status: :unprocessable_entity
 end

end

Here's what's going on with this controller:

We configure it to respond to requests with JSON.
We specify a respond_with action that returns the result of a successful or a failed registration.

And now for the sessions controller:

# app/controllers/users/sessions/sessions_controller.rb

class Users::SessionsController < Devise::SessionsController

 respond_to :json

 private

 def respond_with(resource, _opts={})
  render json: { message: "Welcome, you're in", user: current_user }, status: :ok
 end

 def respond_to_on_destroy
  successful_logout && return if current_user
  failed_logout
 end

 def successful_logout
  render json: { message: "You've logged out" }, status: :ok
 end

 def failed_logout
  render json: { message: "Something went wrong." }, status: :unauthorized
 end

end

Just like the registrations controller, here we specify that the controller will respond with JSON. We also define a respond_with action for when a user successfully logs in and a respond_to_on_destroy to handle a user signing out.

And with that, you should have a working API authentication flow powered by Devise and JWT tokens!

Our final section will take a quick look at how you can track user logins using Devise and Authtrail.

Tracking Devise Logins with Authtrail

Let's say you want to send your app users a notification email whenever someone logs in to their account, with details such as the IP address and the timestamp of the login. How can you accomplish this?

You'll need to track user logins, then use that information in the notification emails you send your users. But first, you'll need a way of tracking user logins. This can be accomplished using a nifty gem called Authtrail, which also pairs well with Devise.

Installing Authtrail

The first step is to install the gem with bundle add authtrail. Additionally, since you'll be storing user-identifiable information such as emails and IP addresses in your app database, it's highly recommended that you encrypt this data in production using a combination of Lockbox and Blindindex gems.

Next, run the Authtrail generator to create its initializer and an accompanying table migration that will store the login data:

# Notice here we run the generator with the encryption flag set to 'lockbox', you can also indicate --encryption=none for no encryption

bundle exec rails g authtrail:install --encryption=lockbox
bundle exec rails db:migrate

How Authtrail Works

Whenever a user tries to log in, a new Authtrail record is created with the following important details:

The login email address used
Whether the login was successful or not
The reason a login failed (if the login was a failure)
The user's IP address, referrer, and a lot more

You can then use this information as you wish. For example, you can send a notification email to a user, including the email and IP address information, to let them know that a login attempt was made on their account.

Go through Authtrail's documentation to see all the possibilities available to you.

Wrapping Up

In this series, we took a deep dive into the Devise gem.

First, we got to grips with the basics of Devise, including how its modules, helpers, views, controllers, and routes work. In this second and final part, we explored how to use Devise with OAuth, Authtrail, and for API authentication.

Hopefully, this series will serve as a helpful guide for all things Devise authentication.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Metaprogramming in Ruby

Roy Tomeij — 2023-07-26T00:00:00+00:00

You've heard of metaprogramming: code you write that generates other code dynamically. Without it, Rails as we know it wouldn't (and couldn't) exist. But there's a good chance you've never done it yourself, and it's not hard to see why; even a brief excursion into the realm of metaprogramming can leave you beset with strange and foreign methods, unfamiliar syntax, and downright mystifying blocks of code.

It's true: metaprogramming is a relatively advanced topic. If you want to really dig in and leverage it on a deep level, it will take time, effort, and a different way of thinking than you're used to. But there's good news: You don't need to wade too deeply into the metaprogramming waters to discover useful methods and techniques to make your life and workflow a little easier.

In this post, we'll take a look at metaprogramming methods like send, define_method, and method_missing and show how they can solve problems we sometimes run into, even in normal Rails applications. But first, let's briefly define metaprogramming and explore when it might come in handy.

When Is Metaprogramming Useful?

Maybe you've got a couple of models in your app that, while being very similar, aren't identical — they have different attributes and methods. But both need to be acted on in a job, and you don't want to write what is essentially the same code more than once.

Maybe once (or twice), you hacked together a ~~kludge~~ masterpiece for a tight deadline on a new feature that got the job done, but you modified an existing model rather than creating a new one the way you would have if you'd had more time. Now that feature needs to be expanded, requiring you to do it the "right" way. This can be intimidating, especially if your codebase references your current implementation all over your app.

Or maybe you rely on a third-party gem that isn't well-maintained, and you discover far too late that it's got a bug in it — right in the middle of a line of code full of metaprogramming.

I've had all these things happen, and I solved them by leveraging some light metaprogramming, all without having to be an expert, and without too much effort. You can, too.

What Is Metaprogramming in Ruby?

Metaprogramming is a set of techniques that Ruby offers us to write code that dynamically writes other code for us. Rather than working on data, metaprogramming works on other code. This is great because it means we can write more dynamic, flexible, and adaptable code if the situation calls for it.

It can help DRY up portions of our code, dynamically define methods we might need, and write useful and reusable pieces of software (like gems!) to make our lives easier.

In fact, Rails leverages metaprogramming on a large scale and to great effect. Any time anyone talks about Rails' magic, they're really talking about its use of metaprogramming.

Caveats

Despite how useful metaprogramming is, it isn't without its drawbacks.

One issue is readability: a little metaprogramming here and there can go a long way and likely won't cause many headaches, but lots of it will hurt the readability of your code. Many Rails engineers probably are not very familiar with it, so relying on it too heavily can prove frustrating both to your future self and to others who need to work on your code.

Maintainability is also a concern; heavy use of metaprogramming can be mind-bending for even experienced Ruby developers. This means you should document your code. Ruby is an incredibly expressive, intuitive, and readable language, but if you're doing things others might find confusing, you should leave comments. You should also use access modifiers - e.g., private, protected - the way you would with other methods.

Another potential source of trouble: monkeypatching. While monkeypatching (dynamic modification of a class — typically, a Rails core class) can be useful, it should also be used sparingly and carefully. If we're not mindful, we can modify behavior in ways that, while perhaps solving one of our problems, can easily create dozens of others. For example, rather than fixing a small bug, adding a new, uniquely-named method, or simply extending the capabilities of a class, we might modify the behavior of existing methods used not just by our own application but by our gems (and even Rails itself), with potentially disastrous consequences. Check out our post Responsible Monkeypatching in Ruby for more on monkeypatching.

Lastly, metaprogramming can make it hard to find things you're looking for, especially if you didn't write the code in the first place. If you find yourself trying to figure out how it's possible CompanySpecificClass#our_special_method exists on an object but isn't defined anywhere, metaprogramming could be the culprit (in this case, probably the use of define_method).

Basic Ruby Metaprogramming Techniques

Let's explore some of the methods we mentioned earlier: send, define_method, and method_missing.

First, we'll take a look at send, what it does, and how it can help us DRY up our code.

Using `send`

Remember the example we mentioned earlier (under the header 'When Is Metaprogramming Useful?') of a couple of similar models with different attributes and methods? In this case, both models need to be acted on in a job in a more or less identical manner.

Essentially, invoking .send allows us to dynamically pass methods (along with arguments) to an object, without necessarily knowing what that object (or method) might be at the time we write the code.

First, a word of caution: send can actually call both public and private methods. If you want to be very explicit (and careful), you can use public_send to ensure it's clear you're calling a public method. I've never done this, but I've also never used it to call any private or protected methods; if you do, you might want to use public_send and send for each use case.

So how does it work? Pretty simply. You just pass the method's name to the object you want to call like this: my_object.send(:method_name). If you need to pass parameters/arguments, you can do so: my_object.send(:method_name, argument1, argument2...). send accepts parameters the same way other methods do, so you can use named arguments, too.

So rather than doing something verbose like this:

def sell_or_refund_or_return_item(item, action)
  if action == "sell"
    item.sell
  elsif action == "refund"
    item.refund
  elsif action == "void"
    item.void
  elsif action == "return"
    item.return
  end
end

We can simply pass our method as an argument and call it with send:

def process_item(item, action)
  return unless item.respond_to?(action)

  item.send(action)
end

Note: For the sake of simplicity and brevity, we're not raising an error if our object doesn't respond_to? the action passed into our above methods. But in a real app, we'd probably want to raise an error to bring our attention to the fact something in our codebase is calling a non-existent method/attribute on the object.

`send`: An Example Scenario

Let's consider the following example. We've got a Purchase model, representing a given purchase from an online store. That purchase could be anything from a single item of one type to many different items.

In the instance below, we're interested in people who bought a particular subscription, which may contain tickets to events (purchase_events), and/or vouchers for products (purchase_products).

Products and events are different things, but there's a lot of overlap, like price, status (sold, returned), etc. In this example, we will refund all instances of a particular item from a subscription, because customers were unable to redeem their purchase (an event was rained off, a company we source from ran out of a collectible, etc.).

class RefundAllInstancesOfItemJob
  def perform(item)
    purchases = Purchase.joins(:subscriptions)
                        .includes(:purchase_events, :purchase_products)
                        .where(subscriptions: { id: item.subscription_id, status: "active" })

    association, foreign_key, foreign_key_value = item.is_a?(PurchaseEvent) ? [:purchase_events, :event_id, item.event_id] : [:purchase_products, :product_id, item.product_id]

    purchases.find_each do |purchase|
      purchase.send(association)
              .where(purchase_id: purchase.id)
              .where(foreign_key => foreign_key_value)
              .each { |item| item.update(status: "refunded") }
    end
  end
end

What have we done here? Ordinarily, we might have ended up writing some if/else logic and duplicating most of this code or even two separate jobs. Instead, we've dynamically sent the appropriate association and foreign key to objects within our query, keeping our code DRY.

We've laid out the keys and associations here explicitly, but they could be passed in as arguments if we set up our call to the job differently. Our customers have now been refunded for the particular item(s) in this subscription cycle! (Let's assume there's a callback on those models that refunds the user when the item is marked "refunded").

Using `define_method`

Now, let's look at define_method and how it can help us with our earlier example where we moved over from storing data on one model where it didn't really belong to a new dedicated model.

Let's say we currently have a Presentation model. When we first added video capabilities to our app, we did it because our biggest customer had to play a single, long video about their business to their shareholders. We had a tight deadline, so rather than create a new, dedicated model, we just tacked on a few columns to our Presentation table, and it got the job done.

Now, though, because we've advertised our app's recording and streaming capabilities, other clients are interested — and they need more than one video per Presentation. Unfortunately, we're still in a bit of a time crunch and have to roll this out fast.

We decide the quickest way to accomplish this without changing tons of code is to:

Create a new VideoAsset model
Move existing columns over from Presentation
Set a boolean current_asset which will update when our clients select the video they want

class Presentation
  has_many :video_assets

  def current_video_asset
    video_assets.find_by(current_asset: true)
  end

  # Replace our dropped columns as new instance methods on Presentation
  # We've used the safety operator (&) on send the way we would any other method with a potential nil return value
  ["status", "playback_id", "asset_id"].each do |column_suffix|
    define_method(column_suffix) do
      current_video_asset&.send(column_suffix)
    end
  end
end

What's going on here? We're dynamically defining methods on our Presentation model to replace the columns we dropped and moved over to VideoAsset. The existing calls in our codebase to @presentation objects will now first find the current VideoAsset associated with our Presentation and then call the corresponding column. We don't have to go around updating controllers and views everywhere!

It's worth noting that the methods defined above could also have been accomplished using ActiveSupport's delegate helper. delegate allows you to use this pattern more often, abstracting away the metaprogramming implementation:

class Presentation
  has_many :video_assets
  delegate :status, :playback_id, :asset_id, to: :current_video_asset

  def current_video_asset
    video_assets.find_by(current_asset: true)
  end
end

Using `method_missing`

Finally, let's look at method_missing. It does pretty much what you'd expect, given its name — it allows you to account for methods that don't exist but are called on an object or class. Let's take a look at turning methods we've placed in our classes into methods that test for truthiness.

class User
  def method_missing(method_name, *args)
    if method_name.ends_with?("?")
      regular_method = method_name.to_s.sub("?", "")

      result = self.send(regular_method, *args)

      result.present? && result != 0
    else
      super
    end
  end
end

So, what's happening here?

Well, we've basically recreated an ActiveRecord feature for our User model. Any column that exists on a table in Rails has an identically-named method ending in a ? added to instances of its corresponding class. We've done something similar with our User class — any undefined instance method called on a user object ending in a ? will be tried against a method of the same name without its question mark, and turn the result into a boolean.

So, for example, @user.purchase_totals? will (rather than return a decimal representing the total amount of money a user has spent in our app) simply return true if the number is nonzero — otherwise, false.

Note the use of present? here. It accounts for things like empty strings and arrays, which otherwise would return true if we'd used a double-bang to assess if it was truthy or not.

If there's no match, we default to calling super, resulting in the expected behavior (a NoMethodError being thrown).

More Advanced Metaprogramming Techniques in Ruby: An Overview

We've covered some useful methods that can be used sparingly to help us out in everyday situations. But what about more advanced uses of metaprogramming? What else is it good for?

DSLs: As mentioned, ActiveRecord, the ORM (and DSL) that ships with Rails heavily leverages metaprogramming. That's where all those automatic methods on our objects come from. Every column we add to a table becomes two methods on our instances — column and column= (and column? for those who were paying attention!). It isn't magic that's making this happen, it's metaprogramming — in fact, the use of the same method_missing method we talked about earlier! Metaprogramming is also responsible for Rails' ability to automatically create methods like find_by_first_name and find_by(first_name: "name").
Gems: If you're building a gem, chances are you'll want it to be flexible, adaptable, and to work with more than just one specific Rails stack. Most of the gems you regularly use have at least some degree of metaprogramming, and many use a lot to achieve the level of flexibility they offer.
Dynamic APIs: Metaprogramming can help us design dynamic APIs by allowing us to define methods on the fly based on runtime information, rather than hardcoding every possible method. For example, a RESTful API might expose resources with dynamic paths and attributes based on the data model of the underlying application (this may sound familiar — Rails' routing system does this). We can generate methods for each resource at runtime based on the database schema and dynamically respond to HTTP requests.
Frameworks: As mentioned, Rails wouldn't be Rails without metaprogramming. While it's a particularly magical framework, almost any framework you build will need plenty of metaprogramming. If you're going to build your own (presumably far more lightweight) framework, you'll need to leverage metaprogramming.

Wrapping Up

We've covered some real ground here! We learned about dynamic method definition with define_method and method_missing, and about dynamic method invocation with send and public_send.

We also talked about metaprogramming's pitfalls, including its issues with readability, maintainability, and searchability. Finally, we touched on more advanced use cases like writing gems, DSLs, and frameworks.

Further Learning

There are a lot of resources out there for taking Ruby metaprogramming further.

RubyMonk offers a short but free course.

A couple of other paid courses include:

Ruby Metaprogramming — Complete Course from Udemy
Chris Oliver's comprehensive Advanced Ruby: Behind the Magic. In addition to digging deep into Rails, this also has a section on metaprogramming and DSLs.

An oft-recommended book is Metaprogramming Ruby 2: Facets of Ruby.

I hope you found this post useful. Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Delegate Methods in Ruby

Roy Tomeij — 2023-07-19T00:00:00+00:00

Delegation in programming refers to handing off a task from one part of a program to another. It's an essential technique in object-oriented programming, empowering clean, maintainable code by ensuring that each object or method is responsible for a specific task or behavior.

Understanding and using delegation is key to mastering Ruby and other object-oriented languages. Delegation promotes separation of concerns, making your code more modular and easier to understand, test, and refactor.

In this article, we'll dive into three ways to achieve delegation in Ruby: using explicit delegation, the Forwardable module, and ActiveSupport::Delegate (for Rails).

Let's get started!

Delegation in Ruby

Let's begin by discussing explicit delegation — calling a method within a method. Then we'll explore the built-in Forwardable module and how it can streamline delegation. Lastly, we'll cover ActiveSupport::Delegate, a Rails-specific delegation tool with some handy features.

Explicit Delegation

In its simplest form, delegation can be achieved by explicitly calling a method within another method. This approach is often employed when the delegated method belongs to a separate object or when the delegation is simple enough not to warrant the use of more advanced techniques.

Implementing explicit delegation in Ruby is straightforward. Let's take an example of a Printer class that uses an HP object to print text:

class HP
  def print(text)
    # Actual code to print
  end
end

class Printer
  def initialize(printer_model)
    @printer_model = printer_model
  end

  def print(text)
    @printer_model.print(text)
  end
end

In the example above, the print method of the Printer class explicitly delegates the formatting task to the HP class by calling its format method. This delegation outsources the printing responsibility to another class in a simple and readable way.

Explicit delegation is common in real-world applications, as it helps promote the separation of concerns and maintainable code. For instance, you might find this technique used in applications that require communication between different services, where one service calls another to perform a specific task.

Additionally, explicit delegation can be useful when building adapters or wrappers around third-party libraries, as it allows you to isolate the interaction with the external code.

In summary, explicit delegation is a straightforward and effective way to delegate tasks between objects in Ruby. By calling a method within another method, you can easily delegate responsibilities, making your code more maintainable and promoting the separation of concerns.

Ruby's Forwardable Module

The Forwardable module is a built-in Ruby library that provides a more streamlined and flexible approach to delegate methods compared to explicit delegation. By including the Forwardable module in your class, you gain access to methods like def_delegator and def_delegators, making delegation a breeze.

To get started with the Forwardable module, include it in your class and use the def_delegator and def_delegators methods to delegate one or multiple methods, respectively. Let's revisit our Printer example with a new Formatter class and implement the Forwardable module:

require 'forwardable'

class Formatter
  def format(text)
    # Perform formatting and return the formatted text
  end
end

class Printer
  extend Forwardable

  def_delegator :@formatter, :format

  def initialize(formatter)
    @formatter = formatter
  end

  def print(text)
    formatted_text = format(text)
    puts formatted_text
  end
end

In the example above, we've used def_delegator to delegate the format method from the Formatter class. If you need to delegate multiple methods at once, you can use def_delegators. For example, if the Formatter class had additional methods such as capitalize, you could delegate them like this:

class Printer
  extend Forwardable

  def_delegators :@formatter, :format, :capitalize

  # rest of code
end

When using the Forwardable module, there are a few important potential drawbacks to consider:

First, ensure that the target object is initialized before using delegation. Otherwise, you may encounter errors.
Second, the overuse of delegation can lead to potential confusion in code readability, as it may become unclear which methods belong to the class and which are delegated.
Finally, there may be a slight performance overhead when using Forwardable compared to direct method calls. However, this overhead is generally negligible in most applications.

For more in-depth information about the Forwardable module, see the Forwardable documentation.

`ActiveSupport::Delegate` for Rails Applications

ActiveSupport::Delegate is a delegation utility provided by Rails, offering a concise syntax for delegating methods to associated objects. While it shares similarities with Ruby's built-in Forwardable module, ActiveSupport::Delegate offers additional options and features tailored for Rails applications.

To use ActiveSupport::Delegate, simply call the delegate method in your class and specify the method(s) to be delegated along with the target object. Let's revisit the Printer and Formatter example and implement delegation using ActiveSupport::Delegate:

class Formatter
  def format(text)
    # Perform formatting and return the formatted text
  end
end

class Printer
  delegate :format, to: :@formatter

  def initialize(formatter)
    @formatter = formatter
  end

  def print(text)
    formatted_text = format(text)
    puts formatted_text
  end
end

ActiveSupport::Delegate offers additional options and features that can be useful in various scenarios. For instance, you can use the :prefix option to prepend a prefix to the delegated method. If you'd like to delegate multiple methods at once, simply list them before the to: option:

class Printer
  delegate :format, :bold, :italic, to: :@formatter, prefix: true

  # ...
end

This generates methods like formatter_format, formatter_bold, and formatter_italic in the Printer class, delegating to the respective methods in the Formatter class.

ActiveSupport::Delegate is commonly used in real-world Rails projects to delegate methods between associated models or objects. For example, if you have a User model that belongs_to an organization, you could delegate the name method from the organization model to the User model like this:

class User < ApplicationRecord
  belongs_to :organization
  delegate :name, to: :organization, prefix: :organization
end

This allows you to access the organization's name through a user object: user.organization_name.

In conclusion, ActiveSupport::Delegate is a powerful tool for Rails applications that enables concise and expressive delegation of methods. By leveraging its additional options and features, you can create maintainable and well-organized code that adheres to the principles of object-oriented programming in the context of Rails projects.

For more in-depth information about ActiveSupport::Delegate and all the available options, see the Rails ActiveSupport documentation.

Comparison of Delegation Techniques

Explicit delegation is the most straightforward approach to delegation, requiring manually defined methods that call upon the desired methods from the delegated object. Although this technique is simple and doesn't require any external libraries, it may be too verbose when delegating a growing number of methods.

The Forwardable module, built into Ruby, offers a more streamlined way to delegate methods. By using the def_delegator and def_delegators methods, you can delegate one or multiple methods concisely.

ActiveSupport::Delegate, part of Rails' ActiveSupport library, provides a declarative way to handle method delegation. With a clean syntax and additional options like prefixing, it's an expressive tool in a Rails context. Compared to explicit delegation, ActiveSupport::Delegate is more expressive and less verbose, particularly when delegating multiple methods. However, it requires Rails, so it's not suitable for non-Rails Ruby projects.

Choosing a delegation technique depends on your specific needs and application. While explicit delegation offers simplicity and clarity, the Forwardable module provides a more streamlined approach, and ActiveSupport::Delegate offers a powerful, Rails-specific solution. Understanding these options lets you choose the technique that best fits your project and coding style.

Wrapping Up

From explicit delegation to using the built-in Forwardable module or the Rails-specific ActiveSupport::Delegate, each technique offers its unique benefits and potential drawbacks. The technique you should choose depends on your specific context, the nature of your project, and your personal coding style.

Understanding these techniques not only broadens your Ruby programming skills but also equips you to write more modular, readable, and maintainable code. With this knowledge, you are better prepared to tackle complex problems, with eloquent design.

Happy delegating!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Devise for Ruby on Rails

Roy Tomeij — 2023-07-12T00:00:00+00:00

With over 20,000 GitHub stars and lots of integrations, the Devise gem is one of the most popular gems in the Ruby landscape. So why would we term it one of Ruby's "hidden" gems? Well, as popular as it is, most developers only scratch the surface of the library's capabilities.

In this two-part series, we'll take a deep dive into Devise.

In this first part, we'll learn some of the basics, including:

What Devise is and why you should use it in the first place, including some situations where it's advisable not to use it.
How to install Devise and use it in your project.
How to customize the library for your project.

In part two, we'll look at more advanced usages of Devise, including:

OmniAuth and using Devise for API-only applications.
Integrating Devise with an authorization library.

Let's get started!

Pre-requisites

In this tutorial, we'll use a simple Ruby on Rails 7 application featuring users and tasks. Users can register, log in, and log out, with access to create, read, update, and delete actions for tasks depending on their assigned role.

We'll use this app to progressively build ever more complex features and, in the process, show off Devise's powerful features in action.

Let's kick off with a brief introduction to Devise.

Introducing Devise for Ruby

Devise is an authentication library built on top of Warden, a Rack-based authentication framework.

Warden handles user sessions using secure session strings to verify the identities of logged-in users. It also handles users who are not logged in to ensure they cannot access restricted resources.

But since Warden is purely Rack-based, it does not add controller actions, views, helpers, or any other configuration options necessary for building a proper user authentication solution. Devise, on the other hand, does.

Another notable feature of Devise is its modularity. The library comes with around 10 modules which allow you to specify exactly how you want authentication handled in your application. You don't need to use all 10 modules. Instead, you activate and use only what you need for your app. We'll go over these modules later, which include the Registerable module, Omniauthable, Trackable, and others.

With that in mind, let's start building our Tasks app and install Devise.

Getting Started: Installing Devise

Generate a new Rails 7 app using bundle exec rails new tasks_app. Alternatively, just pull our example app's code from the repo.

Ensure you're in the app's root directory, then run bundle add devise. This will add the Devise gem to your app's Gemfile. After this, run bundle exec rails g devise:install to install Devise and generate its initializer file (we'll take a closer look at this file when we go through Devise's modules).

When you run this command, the generator will also give you a few setup suggestions for Devise to work as expected. Just follow the given instructions, and you should be good to go.

Next, let's generate a user model that Devise will work with.

Generating a Devise User Model

Devise needs a user model. What you call this model is totally dependent on your preferences and your app's use case, but it's usually either User or Admin.

Go ahead and generate a Devise user model with bundle exec rails g devise User. This will generate a User model under app/models/user.rb, a migration file to create the user table, and a route for accessing the user resource:

# app/config/routes.rb

Rails.application.routes.draw do
  devise_for :users

  root "home#index"
end

Run bundle exec rails db:migrate to run the migration and set up the user table. With that done, we now have a user model we can use for all things authentication.

Next, let's set up a Task model to use moving forward.

Generating a Task Model

Remember, using Devise, our simple app needs to:

Enable a user to register.
Enable a user to sign in and out of the app.
Enable a user to create a Task that will belong to them.
Allow (or disallow) a user from interacting with a Task, depending on their role.

With that in mind, let's now generate the Task model that users will interact with:

bundle exec rails g scaffold Task user:references title body:text status:integer

Now we should have a Task model associated with the user who created it, a title, a body, and a status (showing whether it's done or not).

After that, run bundle exec rails db:migrate to create the tasks table. Also ensure that you relate a Task to a User:

# app/models/user.rb

class User < ApplicationRecord
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  has_many :tasks, dependent: :destroy # add this line
end

When you run the scaffold generator, the relation belongs_to :users will automatically be added to the Task model. All the same, edit the Task model to use an enum for the status as shown below:

# app/models/task.rb

class Task < ApplicationRecord
  belongs_to :user

  enum :status, { draft: 0, underway: 1, done: 2, archived: 3 }
end

With that, we are now ready to dive deeper into Devise. Let's start with a brief overview of Devise's modules.

Modules in Devise for Ruby

As we mentioned in the introduction, one of Devise's key features is its modularity. But what does "modularity" actually mean, in this case? It simply means separating the authentication process into different parts to be managed independently.

The latest version of Devise (at the time of writing) contains 10 modules:

Database authenticatable - This module will take a user-supplied password and convert it into a secure hash which is then stored in the database. The module also handles verification when a user signs in.
Omniauthable - Enables OmniAuth authentication.
Lockable - This module will lock an account depending on the number of failed logins. The account is made accessible again after a certain time period or via email.
Trackable - Tracks the number of logins an account has made, the IP addresses used, and the login timestamps.
Confirmable - Sends an email with confirmation instructions when an account is registered and will also check if a user's account is confirmed when they log in.
Registerable - Allows users to register an account on your app, and handles account editing and destruction.
Recoverable - Handles password resets and account recovery.
Timeoutable - Expires user sessions after a specified amount of time has elapsed.
Validatable - This lets you define custom validation rules for user-supplied emails and passwords during account creation.
Rememberable - Uses a cookie to remember a user during authentication.

For more information, Devise's documentation on modules will serve you well. For now, we'll switch to Devise helpers and filters.

Devise Helpers and Filters

One of the reasons to use an authentication library like Devise is to manage access to controller resources and associated views. Devise comes loaded with a set of convenient helpers, which include:

user_signed_in? - Checks whether there is a currently logged-in user.
current_user - Allows you to reference the currently logged-in user. For example, you can use a code snippet like current_user.email to fetch the currently logged-in user's email.
user_session - For the currently logged-in user's session.
destroy_user_session_path - Destroys a logged-in user's session and redirects to a specified path or the root path.
new_user_session_path - Responds with a user login view.
edit_user_registration_path - Gives the currently logged-in user access to a view to edit their registration details.
new_user_registration_path - Responds with a view that has a registration form for registering new users.

That's about it for the helpers. To manage controller access, Devise gives you a nifty before_action filter you can use like this:

# app/controllers/posts_controller.rb

class PostsController < ApplicationController

  before_action :authenticate_user! # if your model is called "User"
  before_action :authenticate_member! # if your model is called "Member"

  ...
end

With that, let's now shift focus to how Devise integrates with Rails' strong parameters.

Devise and Ruby on Rails' Strong Parameters

Strong parameters is a well-known Ruby on Rails feature that prevents the mass assignment of request parameters to objects. Strong parameters require that request parameters be explicitly declared before any assignment is done, usually at the controller level.

Devise also mirrors this functionality by requiring parameters to be explicitly defined under appropriate Devise-specific controller actions:

sign_up - By default, this is found in the Devise controller Devise::RegistrationsController#create. The keys allowed by default are email, password, and password_confirmation.
sign_in - This is found in the controller Devise::SessionsController#new, with the default allowed authentication keys email and password.
account_update - Found in Devise::Registrations#update — the authentication keys email, current_password, password, and password_confirmation are allowed.

The most convenient way to add your own authentication keys is to use a before_action filter in ApplicationController, like here, where we add a username key that is required when a user signs up:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  before_action :configure_permitted_parameters, if: :devise_controller?

  protected

  def configure_permitted_parameters
    devise_parameter_sanitizer.permit(:sign_up, keys: [:username])
  end
end

An interesting use case for Devise's strong parameters is when you need to pass an array of keys to the Devise sanitizer. For example, let's say we have a freelancer marketplace app where a user can be a "contractor" for hire, or an "employer" who can hire other contractors, or a contractor and employer at the same time.

Without going into too many technical details, we could achieve this functionality by allowing users to update their account using two select boxes for the respective roles — "contractor" and "employer" — so that a user can choose either or both options.

A quick way of achieving this is to use an array with the roles as keys, but Rails' strong parameters only allow for the following scalar values: String, Symbol, NilClass, Numeric, TrueClass, FalseClass, Date, Time, DateTime, StringIO, IO, ActionDispatch::Http::UploadedFile, and Rack::Test::UploadedFile. As you can see, arrays, hashes, and other objects are, by default, not permitted.

To use an array that permits multiple roles for our user, we can modify our code as shown below:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  before_action :configure_permitted_parameters, if: :devise_controller?

  protected

  def configure_permitted_parameters
    devise_parameter_sanitizer.permit(:account_update) do |user_params|
      user_params.permit({ roles: [] },
      :email, :password, :password_confirmation, :current_password)
    end
  end
end

The Devise documentation on helpers covers the topic in more detail.

Next, we'll learn how to customize Devise's views and controllers.

Customizing Devise Views

Because Devise is built as a Rails engine, almost all of its components are pre-packaged. Good examples of this include the views for logging in, signing up, updating account details, resetting your password, etc. At some point in your journey to build an app, you may need to customize these built-in views to suit your needs.

The first step is to generate views using the command bundle exec rails g devise:views. When you do that, the respective views are generated and placed in the app/views/devise folder:

Let's use a practical example of customizing Devise's views. In the previous section, we learned how to add extra authentication keys to the Devise sanitizer. Let's now extend that example to add a username field to the user registration view.

Since the username field is not available to our default user model, we'll add it using a bundle exec rails g migration add_column_username_to_user username migration. Run the command bundle exec rails db:migrate to start the migration and add the column to the user table.

Next, open up the new user registration file and edit it as follows:

# app/devise/registrations/new.html.erb

<h2>Sign up</h2>

<%= form_for(resource, as: resource_name, url: registration_path(resource_name)) do |f| %>
  ...

  <!-- username field added -->

  <div class="field">
    <%= f.label :username %><br />
    <%= f.text_field :username, autofocus: true %>
  </div>
  ...
<% end %>

Generating Devise views this way is fine if you're working with all views. But say you only want to customize a couple of views. How can you do this? Well, you simply pass the generator command a list of the views you want by indicating the view flag bundle exec rails g devise:views -v sessions registrations. In this case, this only generates views associated with the login/logout process and those that handle sign-ups.

Moving on, let's dig further into Devise customization by learning how to customize the library's controllers and routes.

Customizing Devise Controllers and Routes

Customizing Devise's views can only get you so far. If you want real customization, you need to go into the library's controllers and routes.

Let's say we want to send an email to our admin notifying them of every new user registration on the app. (This is not the ideal way to go about things, but let's go with this example to illustrate what we want).

To begin with, we need to modify the signup action of the Devise::RegistrationsController. Normally, Devise's controllers are not directly accessible for editing, so, just like we did with the views, we can generate them like so:

bundle exec rails generate devise:controllers users

The last bit of that command is the scope - in this case, users. If you want it to be something else, you can change it accordingly. When you are done, the generated controllers should be in a structure similar to this:

Next, open up the routes file and modify Devise's routes to reflect the changes in the controller structure:

# config/routes.rb

Rails.application.routes.draw do
  resources :tasks

  devise_for :users, controllers: {
        sessions: 'users/sessions',
        registrations: 'users/registrations'
      }

  root "tasks#index"
end

Now open up the newly generated Users::RegistrationsController and modify it to send an email to the admin whenever a new user signs up:

# app/controllers/users/registrations_controller.rb
class Users::RegistrationsController < Devise::RegistrationsController
  ...

  def create
    super do |resource|
    # use a custom mailer to send an email to the admin
    AdminNotifierMailer.user_signup_notification(resource).deliver_later
  end

  ...
end

As you can see, with access to Devise's controllers and actions, you can modify your app's authentication flow in any way you want.

Coming Up Next: Advanced Use of Devise

In the first part of this two-part series, we covered the basics of the Devise gem. We learned about the Devise's different modules, as well as how to customize its views and controllers.

In part two, we'll dive into more advanced usages of Devise, including API authentication and how to use OmniAuth with Devise.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Keep Your Ruby App Secure with Bundler

Roy Tomeij — 2023-06-28T00:00:00+00:00

This article covers the use of bundler features to secure Ruby applications. In this day and age, we have to be more and more careful about software supply chain security.

We'll show you how to start this journey by relying on a Gemfile and bundler to manage your project's dependencies.

By the end of the post, you will better understand how bundler audit and bundler outdated work. Both can help you monitor the security state of your project's dependency tree.

Let's dive in!

An Introduction to Bundler for Ruby

The history of Bundler is linked to RubyGems. RubyGems, first released in 2004 by Chad Fowler, is a package manager that makes it possible to distribute and manage Ruby libraries, applications, and their dependencies.

Bundler, on the other hand, is a dependency manager. It was first released in 2009 by Carl Lerche.

Bundler helps developers manage dependencies between different Ruby libraries and applications. It uses a Gemfile to define the libraries and versions that a project depends on, then installs and loads those libraries in the correct order. Bundler provides a simple way to manage dependencies and ensure that all the required libraries are installed and loaded correctly.

Supply Chain Security

RubyGems and Bundler can only solve some security issues. For example, in 2013, 2018, and 2020, several security issues emerged directly related to RubyGems. Each time, gems were compromised, potentially exposing thousands of projects and products to malicious actors.

Nowadays, the security of the software supply chain has become more high profile, due to major incidents like 2020's SolarWinds hack, EventStream in 2018, and Equifax, Maersk, Merck, and FedEx in 2017.

Let's see how Bundler can help us avoid some security issues.

Bundler's Security-Related Features for Ruby

Bundler comes with several features that allow us to secure a Ruby project:

Source validation: ensures gems are installed from a specific, trusted source.
With a dependency graph, we can see a whole list of dependencies for each gem, helping us to identify potential security risks.
A Gemfile.lock file records the specific versions of each gem used in the project. As it's packaged with the project, it ensures that the same versions of gems are used across different environments.
Vulnerability detection: Thanks to an integration with the Ruby Advisory Database (RDB), Bundler can detect known security vulnerabilities in gems.
An audit command lets you list known security vulnerabilities related to gems a project depends on.
Checksum verification: Bundler verifies the checksum of each gem before installing it.
Gem signing and signature verification: Gems can be signed cryptographically by a developer.

But, alas, signing gems is complicated and requires a lot of steps. As a consequence, not all gems are signed, thus putting many projects at risk.

Yet, in the meantime, we can rely on two features in particular to ensure that a project doesn't rely on packages that are too old or that have been identified as carrying a security risk.

These features are bundler audit and bundler outdated. Let's look at bundler audit first.

About Bundler-audit

bundle audit is a command-line tool that helps you identify security vulnerabilities in the Ruby libraries that your project depends on. It is a plugin for the Ruby dependency manager Bundler and is included with Bundler version 1.10 and above.

bundle audit works by analyzing your Gemfile.lock file, which lists all the dependencies for your project and their versions. It then compares this information with a database of known vulnerabilities in Ruby gems (maintained by Rubysec).

If a vulnerability is found, bundle audit will provide you with information about the vulnerability, including its severity level and which gem versions have been affected. It will also suggest actions you can take to address the vulnerability, such as upgrading to a patched version of the gem or removing it entirely.

bundle audit is a useful tool for ensuring the security of your Ruby projects, especially if you are using third-party libraries or dependencies. By regularly running bundle audit as part of your development workflow, you can stay up-to-date on any vulnerabilities that may affect your project and take proactive steps to address them.

Usage of Bundler-audit in Ruby

The basic use of Bundler-audit is through the bundle audit command:

$> bundle-audit

Name: actionpack
Version: 3.2.10
Advisory: OSVDB-91452
Criticality: Medium
URL: http://www.osvdb.org/show/osvdb/91452
Title: XSS vulnerability in sanitize_css in Action Pack
Solution: upgrade to ~> 2.3.18, ~> 3.1.12, >= 3.2.13

$>

As you can see, it outputs the name and version of a package used by the project we are testing, which contains a medium-level security issue. It also outputs the URL of the security advisory and a possible solution.

Keeping Bundler-audit's Database Up to Date

As the audit is basically comparing a list of gems used in the project with a list of known security issues in a local copy of the security advisory database, you need to keep that database up to date.

To do so, you just need to run the bundle audit update command. This will fetch the latest version of the database. You should only have to run the audit again to check for any new security risks.

Integration In a CI Pipeline

As with other tools aimed at keeping your project safe, it's best to ensure an audit runs automatically on a regular basis (if not after every commit and push to the Git repository).

You can rely on a git post commit hook to run the audit locally if the Gemfile.lock changes. Here is an example:

# .git/hooks/post-commit
if git diff --name-only HEAD~1 | grep -q Gemfile.lock; then
    echo "Gemfile.lock has changed, running audit..."
    bundle audit update && bundle audit
fi

This is a bit radical, but it works. You can rely on the same approach within the CI pipeline. As the bundle audit command will return a non-zero exit status, the CI pipeline will consider it to have failed. Unfortunately, we don't have a proper audit report out of the box. Instead, you have to direct the output of the bundle audit command towards a file and save it as an artifact of the build for the CI pipeline.

Here is how to direct the command's output to a file:

$> bundle audit update && bundle audit > /tmp/bundle-audit-report.txt

As each CI service handles artifacts differently, check the relevant documentation to see how to do this.

Bundler-audit in Summary

bundle audit is not a complete solution, yet it still lets your team know if your project relies on unsafe gems (without costing you an arm and a leg). Your team should put tools such as git hooks or CI tasks in place, and relevant integrations to make any issues visible.

All things considered, I'd say it's not too much trouble. It probably requires a few hours to get started and have the basic information spit out as a comment in your favorite git hosting solution, or as a notice in a Slack channel.

Yet bundle audit is only one part of what you can do. It only tells you if there is a security issue related to a gem relied on by a project. It does not tell you if a gem is outdated. To handle that, bundler outdated is the command to use.

Bundler Outdated for Ruby Gems

Auditing for security risks is very important, yet listing outdated gems is also a big issue. The longer you wait to update a dependency, the higher the risk that your code breaks.

You should aim to work in small iterations, deployed frequently, to limit the amount of change contained in each deployment. Equally, you should aim to update any gem you use as soon as it's updated, or as close to that as possible.

Usage of Bundler Outdated

Just like bundler audit, bundler outdated is very simple to use. Just call it at the root of your Ruby project. It will compare the Gemfile.lock content to the current gem releases listed in it.

$> bundle outdated
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies......

Gem           Current    Latest    Requested   Groups
addressable    2.8.1      2.8.4
capybara       3.38.0     3.39.0    >= 0        test
devise         4.9.0      4.9.2     >= 4.6.0    default

$>

As you can see, three gems are outdated in this project. Let's see how we should read this table:

The first column contains the name of the gem.
The second column shows the currently installed version.
The third column shows the latest version available.
The fourth column shows the request version (from the Gemfile).
The fifth column shows the group the gem is part of (in the Gemfile).

Here, we can see we don't have much to worry about for addressable and devise, as the difference in releases is within the patch versions (third part of the release number). The version of capybara is only behind one minor release, which implies a few more changes, but this rarely means breaking changes.

Still, the logical thing to do here is to update all three as soon as possible.

Usage In a CI Pipeline

As bundle outdated will return a non-zero exit status for outdated gems, a CI task it's based on is considered failed. So you can have a simple and direct flag in your CI pipeline in case of outdated dependencies.

Yet you might want to rely on a pair of flags to improve your use of bundle outdated. The command can filter its output to only list gems that have pending patch-level updates:

$> bundle outdated --filter-patch
...
Gem           Current    Latest    Requested   Groups
addressable    2.8.1      2.8.4
devise         4.9.0      4.9.2     >= 4.6.0    default

In the same manner, you can list only gems with pending minor version updates:

$> bundle outdated --filter-minor
Gem           Current    Latest    Requested   Groups
capybara       3.38.0     3.39.0    >= 0        test

And in the same way, you can list only gems with pending major release updates:

$> bundle outdated --filter-major
Gem                Current    Latest    Requested   Groups
sidekiq-scheduler   4.0.3      5.0.1      >= 0       default

Those three flags let you easily tailor a CI task to only warn you that a patch-level update is available but not block the build (or the reverse), for example.

A complimentary option allows you to only list outdated gems within the default group:

$> bundle outdated --group default
...
Gem                Current  Latest  Requested          Groups
devise             4.9.0    4.9.2   >= 4.6.0           default
faker              3.1.1    3.2.0   >= 0               default

As gems within test and development groups don't impact the production release, they could be considered less of a priority to keep up to date, for example. Thus you can use the flag only to raise a warning or block the CI pipeline if gems within the default group are outdated.

Wrapping Up

Maintaining the security of your Ruby application is not just the result of one action. Rather, it's a cumulative effect of several actions put together. As we've covered in this post, Bundler provides us with two commands that can help you to improve your application's security:

bundle audit quickly lets you know if any gem you add or use in your project could pose a security threat and what to do about it.
bundle outdated allows you to flag gems that should be updated. It complements bundle audit, and if you keep on top of it, you avoid having an update bundle that's too large.

These two commands are easy to use and integrate with local feedback loops and CI pipelines.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Lambdas in Ruby

Roy Tomeij — 2023-06-21T00:00:00+00:00

Lambdas are a powerful programming construct used in many languages. These functions allow developers to write code that is more concise, efficient, and easier to maintain, making Lambdas an essential part of any developer's toolkit.

In this article, we'll explore how you can use Lambda functions in Ruby to simplify your code and make it more powerful.

What Is a Lambda Function?

Lambdas are expressions whose value is a function. For this reason, they are also called anonymous functions. The really nice part about this is that we can reference them using a variable in our code, just like a string, integer, or any other object.

Everything in Ruby is an object, and Lambdas are no exception. Specifically, a Lambda is an instance of the Ruby Proc class. They behave the same way as a Proc, except that a Lambda validates the number of parameters. Its use case is targeted towards specific functions, albeit those without a proper name. In fact, the Ruby Proc class has a lambda? method which returns true if argument handling is rigid (as is the case for a Proc generated by a Lambda).

Side-note: Because of its popularity, it is also important to clarify that AWS Lambda is a different concept to a Lambda in Ruby. AWS Lambda is a cloud-computing service that implements functions without managing the underlying server infrastructure. You can simply write the function in any number of programming languages, including Ruby.

How to Define and Invoke a Lambda Function in Ruby

The canonical form of a Lambda expression is as follows:

a_lambda = lambda { puts "Hello world!" }

More commonly in Ruby, you will see the -> shorthand notation used to define a Lambda expression.

a_lambda = -> { puts "Hello world!" }

If you simply puts a_lambda, here is what you get.

puts a_lambda
=> #<Proc:0x0000000108f77bb0 10.rb:6 (lambda)>

Lambdas can use a block definition notation as well.

a_lambda = lambda do
  puts "Hello world!"
end

The Lambda is invoked as shown below, and the output is as expected:

a_lambda.call
=> Hello world!

The mechanism to invoke a Lambda is the same as it is with a Proc. However, it is different from the way in which traditional functions are called. Note that, if you simply reference a_lambda in your code, it will not invoke the function. This is equivalent to referencing a variable but doing nothing with it.

Lambdas can also accept parameters. The following Lambda returns the square of a number:

a_lambda_with_params = lambda {|val| val**2 }

The call method can take parameters to match your Lambda signature:

three_squared = a_lambda_with_params.call(3)
puts "Three squared is #{three_squared}"

This outputs the following:

Three squared is 9

Ruby offers other syntax options for invoking a Lambda. All of the following statements are equivalent:

val1 = a_lambda_with_params.call(3)
val2 = a_lambda_with_params.(3)
val3 = a_lambda_with_params[3]
val4 = a_lambda_with_params===3

The last few notation alternatives seem quite obtuse, but they work nonetheless.

Now consider that we want to calculate the square of a set of numbers. We might typically write Ruby code such as this:

numbers = [1, 2, 3, 4, 5]
squares = numbers.map { |n| n**2 }
puts squares.join(", ")

In this code, what is the expression passed to the map function? It is a block that accepts a single parameter. Thus, it is equivalent to an anonymous function or Lambda. We can rewrite our code as follows:

numbers = [1, 2, 3, 4, 5]
square_lambda = lambda { |n| n**2 }
squares = numbers.map &square_lambda
puts squares.join(", ")

This outputs the following:

1, 4, 9, 16, 25

Notice the ampersand that precedes the Lambda expression passed to map. Why is this needed? Well, the ampersand tells Ruby that this is the special block parameter. If we omit this, Ruby thinks that square_lambda is a 'regular' parameter. Using that approach, we would get the following error.

in `map': wrong number of arguments (given 1, expected 0) (ArgumentError)

One of the main benefits of a Lambda is the convenience and ease of using the function only once. We could also define a traditional method here, but if this is the only place it would be used, a Lambda function provides a cleaner solution for such a 'disposable' function.

Another primary benefit of Lambdas is that they provide a portable mechanism to execute code anywhere in your program. This supports some interesting use cases that we'll explore next.

Lambda Use Cases in Ruby

There are a number of use cases that work well as Lambda functions in Ruby.

First Class Functions

Lambda functions can be passed to other functions for use. This concept is referred to as a first-class function. Consider a use case where you need to execute code at a later point in time, once a certain condition is met.

# Example: Use a lambda to defer execution of code until later
def do_something_later(&a_lambda)
  @deferred_action = a_lambda
end

# Later, when the condition is met, call the deferred action
if some_condition_is_met
  @deferred_action.call
end

Callbacks

Callbacks are another related use case for which Lambdas are useful. Lambdas can be used to implement logic invoked at certain points in the application lifecycle, or a specific object’s lifecycle.

A good example of this pattern is found in MongoDB's Mongoid framework. You can set a :default at the field level, and Lambdas allow you to defer execution until runtime.

Consider a document with a last modified timestamp. The first implementation runs when the class is loaded. However, execution of the Lambda is deferred until the document is created — the outcome you likely want in your application.

# Static implementation at class load time
field :last_modified, type: Time, default: Time.now

# The implementation below defers the lambda execution until document creation time
field :last_modified, type: Time, default: ->{ Time.now }

The Visitor Pattern

The visitor pattern in Ruby is another use case that can take advantage of the flexibility of Lambdas. If the visitor logic is not known a priori or otherwise fits the Lambda use cases, visitors can be implemented as Lambda functions.

Consider the following code snippet for a visitor to a hierarchy of nodes representing mathematical expressions.

class PrintVisitor < Visitor
  NODE_TYPES = {
    IntegerNode => ->(node) { puts "IntegerNode: #{node.value}" },
    FloatNode => ->(node) { puts "FloatNode: #{node.value}" },
    ExpressionNode => ->(node) {
      puts "ExpressionNode:"
      node.left.accept(self)
      node.right.accept(self)
    }
  }
  def visit(node)
    handler = NODE_TYPES[node.class]
    handler.call(node)
  end
end

This implementation uses Lambda functions to avoid defining separate methods for each element subclass. Instead, the Visitor class defines a single visit method that can handle any element subclass by using a Lambda function to call the appropriate method.

This may be a bit over-engineered, but if you consider cases where the visitor logic is not predefined, Lambdas become much more useful.

Functional Programming Paradigms

Lambdas in Ruby enable developers to adopt a functional programming paradigm by creating anonymous functions that can be used as arguments in other functions. This approach allows for creating complex functionality by combining smaller, reusable functions. The use of Lambdas in Ruby code results in concise, readable code. They also leverage key functional programming concepts, such as closures and higher-order functions.

Lambdas can support functional programming features such as currying if you want to take them to that level. Currying permits functions to be partially applied and reused in diverse contexts.

Metaprogramming

Lambdas are used in metaprogramming to generate code dynamically. Consider the following code that adds a method:

# Example: Use a lambda for dynamic code generation
def create_method(name, block)
  define_method(name, &block)
end

a_lambda = -> { puts "Hello, world!" }
create_method(:hello, a_lambda)

hello   # Output: "Hello, world!"

Lambdas as Closures

A powerful capability of Lambdas is their ability to create a closure. This encapsulates the function as well as the runtime environment, including variables from the surrounding scope. Variables are closed (or bound) to their values in a Lambda. This is why it is called a closed expression or closure. The execution context is stored in an instance of a Ruby Binding object.

Consider the following example that creates a Lambda which implements an incremental counter:

# Example: Use a lambda as a closure to capture runtime variables
def create_counter(start)
  lambda { start += 1 }
end

counter = create_counter(0)
puts counter.call   # Output: 1
puts counter.call   # Output: 2

The Impact of Lambdas on Performance

Performance can be a consideration if you have extremely tight operational requirements. In a tight loop, for example, you will see that Lambdas perform a bit slower than other Ruby code. It is easy to understand why. A Proc object is created to implement a Lambda (as opposed to the runtime engine delegating control to a method).

In the vast majority of use cases, you will likely not notice much difference in performance (considering that computers can execute thousands of instructions in the time it takes to perform a single database query). Nonetheless, this is something to keep in mind.

Wrapping Up

In this post, we explored some of the fundamentals of Lambdas and Lambda use cases for Ruby.

Lambdas provide a powerful fundamental programming construct for Ruby developers. They can be convenient one-time use functions, implementation mechanisms for callbacks, and used for several other use cases. Lambdas can also come in handy on projects that adhere to functional programming paradigms.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Setting Up Business Logic with DCI in Rails

Roy Tomeij — 2023-06-14T00:00:00+00:00

In our last post, we examined the most common ways to organize business logic in Ruby on Rails. They all have advantages and drawbacks, and essentially, most do not leverage the full power of Object Oriented Programming in Ruby.

This time, we will introduce another alternative that more naturally fits the mental models we apply when reasoning about the behavior of our applications: DCI.

Enter DCI (Data, Context, Interaction) for Rails

DCI is an often overlooked paradigm that can circumvent a lot of the issues outlined in the last post, while still adhering to MVC.

More importantly, though, it's a code architecture style that simultaneously lets us consider an application as a whole and avoids introducing objects with unclear responsibilities.

Without further ado, here's the definition of DCI from Wikipedia:

The paradigm separates the domain model (data) from use cases (context) and roles that objects play (interaction). DCI is complementary to model–view–controller (MVC). MVC as a pattern language is still used to separate the data and its processing from presentation.

One of the main objectives of DCI is to improve a developer's understanding of system-level state and behavior by putting their code representation into different modules:

slowly changing domain knowledge (what a system is)
rapidly changing system behavior (what a system does)

Moreover, it tries to simplify the mental models of an application by grouping them into use cases.

Let's break this down into its individual parts: data, context, and interaction.

Data

In DCI terms, Data encompasses the static, descriptive parts of what we call the Model in MVC. It explicitly lacks functionality that involves any interaction with other objects. In other words, it's devoid of business logic.

class BankAccount
  attr_reader :balance

  def increase(by)
    @balance += by
  end

  def decrease(by)
    @balance -= by
  end
end

Consider the simple BankAccount class above: it allows you to query the account balance, and increase or decrease it. But there is no such concept as a transfer to another account.

'Wait a second!' I hear you say! Isn't that a description of an anemic model? And isn't that the most horrific anti-pattern of all time 👻? Bear with me for a moment.

Context

The point of DCI is not to strip models of any domain logic, but to dynamically attach it when it's needed, and tear it down afterward. Context realizes this.

A context is responsible for identifying a use case and mapping data objects onto the roles that those play.

The nice thing, by the way, is that they align nicely with our mental models of everyday processes. People don't carry every role around with them all the time either.

Real-World Examples

A school classroom, for example, is composed of people, but some are teachers and some students.

Similarly, some people are passengers on public transport, and some are conductors. Some even play multiple roles at once that may change over time — e.g.:

I'm a passenger.
If it's a long ride, I might also assume the role of book reader.
Suppose someone calls me on the phone. I'm simultaneously a conversation participant.
If I travel with my daughter, I'm also her dad and have to look after her.

And so on.

Back to Our `BankAccount` Example in Rails

Let's continue with the BankAccount example from above, and expand it with a requirement to transfer money from one person to another. Consider the following module, which just defines a method to transfer the money:

module MoneyTransferring
  def transfer_money_to(destination:, amount:)
    self.decrease amount
    destination.increase amount
  end
end

The key notion in this snippet is the reference to self, as we shall see in a moment.

The beauty of applying this pattern to Ruby is the ability to inject modules at run time. A context can then map source and destination roles:

class Transfer
  delegate :transfer_money_to, to: :source

  def initialize(source:)
    @source = source
    @source.include(MoneyTransferring)
  end
end

So, now BankAccount is equipped with the ability to transfer_money_to another account in the Transfer context.

Interaction

The final part of DCI — interaction — comprises everything the system does.

It is here that the use cases of an application are enacted through triggers:

Transfer.new(source: source_account)
  .transfer_money_to(destination: destination_account, amount: 1_000)

The source and destination roles are mapped to their respective domain models, and a transfer takes place. In a typical Rails app, this would happen in a controller or a job — sometimes even in model callbacks.

A critical constraint of DCI is that these bindings are guaranteed to be in place only at run time. In other words, the Transfer object will be picked up by the garbage collector afterward, and no trace of the mapped roles remains with the domain models.

If you flip this around, this ensures that DCI roles are generic, making them both easier to reason about and test. In other words, the Transfer context makes no assumptions about the kind of objects its roles are mapped to. It only expects increase/decrease methods. The fact that they are BankAccounts with an attached state is irrelevant! They could equally be other types of objects (e.g., Wallets/StockPortfolios/MoneyBox). The context does not care. Only through its enactment in a certain use case are the roles associated with certain types. As the snippet above shows, it's succinct and readable.

Case Study Using DCI in a Rails Application

I want to conclude this article with an example from a real-world app where I used DCI to organize parts of the business logic. I will attempt to show how regular Rails MVC can enact a DCI use case. Note that I'm using Jim Gay's surrounded gem to strip away some of the boilerplate.

Here's a Checkout context that includes methods to create and fulfill a Stripe::Checkout:Session object:

# app/contexts/checkout.rb
class Checkout
  # ...

  role :payable do
    def create_session
      Stripe::Checkout::Session.create({
        line_items: line_items, # provided by model
        metadata: {
          gid: to_gid.to_s
        },
        success_url: polymorphic_url(self)
        # ...
      })
    end

    def fulfill
      # ...
    end
  end
end

Imagine that the class method role :payable is just a wrapper around the manual .include(SomeModule) we did above; create_session and fulfill are called the RoleMethods of this context. Note that in this case, the create_session method only relies on self, to_gid (present on any ActiveRecord::Base subclass), and a line_items accessor.

We can now write a test to enact this context:

class CheckoutTest < ActiveSupport::TestCase
  # VCR/Stub setup omitted

  test "created stripe checkout session includes gid in metadata" do
    @quote = quotes(:accepted_quote)

    session = Checkout.new(payable: @quote).create_session

    assert_equal @quote, GlobalID::Locator.locate(session.metadata.gid)
  end
end

This looks good! Now let's look at two separate use cases for this context.

Use Case 1: Quote Checkout

In my app, a Quote is a bespoke offer to a certain customer. It typically contains only one line item, which Stripe will use to create a price:

# model
class Quote
  def line_items
    [{price: price_id, quantity: 1}] # Stripe::Price
  end
end

Now a Stripe Checkout session can be created in a controller as follows:

# controller
@checkout_session = Checkout.new(payable: @quote).checkout_session

This is then used in the view to send the customer to a Stripe Checkout form via a simple link:

<!-- view -->
<%= link_to @checkout_session.url, target: "_top", class: "btn btn-primary", id: dom_id(@quote, :checkout_button) do %>
  <%= t(".checkout") %>
<% end %>

Use Case 2: Review Checkout

Another payable in my app is a Review, which contains many chapters, each with its own line item:

# model
class Review
  def line_items
    chapters.map do |chapter|
      {price: chapter.price_id, quantity: 1} # Stripe::Price
    end
  end
end

Apart from exchanging the payable, the code for enacting the checkout use case stays exactly the same:

# controller
@checkout_session = Checkout.new(payable: @review).create_session

<!-- view -->
<%= link_to @checkout_session.url, target: "_top", class: "btn btn-primary", id: dom_id(@review, :checkout_button) do %>
  <%= t(".checkout") %>
<% end %>

Takeaways

DCI certainly isn't a grab-all solution to code organization, but compared to some other approaches, it feels like a natural inhabitant of a Rails app's ecosystem. What enthralls me the most is that it provides a clear structure for separating descriptive structure ("what the app is") from behavior ("what the app does") without compromising the SOLID principles for OOP design. This separation makes it a breeze to refactor key parts of such an app.

The actual business logic design also feels more streamlined because DCI attempts to closely reflect our mental models of the software we build.

That said, like any design pattern or paradigm out there, it's not a hammer that fits every nail. You might find that this separation of behavior from data is something that diminishes code cohesion in your app, for example.

If you want to try it on for size, I recommend using DCI for integrating third-party APIs, or with fringe concerns that don't directly touch your app's core functionality. That's because those areas typically don't change very often and are thus ideal playgrounds for experimenting with new tools.

Wrapping Up and References

In part one of this two-part series, we examined common approaches for building business logic in your Rails application, including fat models, service objects, jobs, and event sourcing.

In this second and final part, we turned our attention to DCI specifically, exploring each individual part: Data, Context, and Interaction. We showed how to use the DCI paradigm in a real-world Rails application.

Happy coding!

References:

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Use Sinatra to Build a Ruby Application

Roy Tomeij — 2023-05-31T00:00:00+00:00

In this article, we'll introduce Ruby on Rails' lesser-known but powerful cousin Sinatra. We'll use the framework to build a cost-of-living calculator app.

By the end of the article, you'll know what Sinatra is and how to use it.

Let's go!

Our Scenario

Imagine this: you've just landed a job as a Ruby developer for a growing startup and your new boss has agreed to let you work remotely for as long as you like.

You start dreaming of all the cool cities where you could move to begin your digital-nomad life. You want to go somewhere nice but, most importantly, affordable. And to help you decide, you hit upon an idea to build a small app that shows cost-of-living data for almost any city or country you enter.

With so many languages, frameworks, and no-code tools available today, what will you use to go from idea to app?

Enter Sinatra!

Overview of Sinatra

Compared to Ruby on Rails, a full-stack web framework, Sinatra is a very lean micro-framework originally developed by Blake Mizerany to help Ruby developers build applications with "minimal effort".

With Sinatra, there is no Model-View-Controller (MVC) pattern, nor does it encourage you to use "convention over configuration" principles. Instead, you get a flexible tool to build simple, fast Ruby applications.

What Is Sinatra Good For?

Because of its lightweight and Rack-based architecture, Sinatra is great for building APIs, mountable app engines, command-line tools, and simple apps like the one we'll build in this tutorial.

Our Example Ruby App

The app we are building will let you input how much you earn as well as the city and country you'd like to move to. Then it will output a few living expense figures for that city.

Prerequisites

To follow along, ensure you have the following:

Ruby development environment (at least version 3.0.0+) already set up.
Bundler and Sinatra installed on your development environment. If you don't have Sinatra, simply run gem install Sinatra.
A free RapidAPI account since we'll use one of their APIs for our app project.

You can also get the full code for the example app here.

Before proceeding with our build, let's discuss something very important: the structure of Sinatra apps.

Regular (Classical) Vs. Modular Sinatra Apps

When it comes to structure in Sinatra apps, you can have regular — sometimes referred to as "classical" — apps, or "modular" ones.

In a classical Sinatra app, all your code lives in one file. You'll almost always find that you can only run one Sinatra app per Ruby process if you choose the regular app structure.

The example below shows a simple classical Sinatra app.

# main.rb

require 'sinatra'
require 'json'

get '/' do
# here we specify the content type to respond with
  content_type :json

  { item: 'Red Dead Redemption 2', price: 19.79, status: 'Available'  }.to_json
end

This one file contains everything needed for this simplified app to run. Run it with ruby main.rb, which should spin up an instance of the Thin web server (the default web server that comes with Sinatra). Visit localhost:4567 and you'll see the JSON response.

As you can see, it is relatively easy to extend this simple example into a fairly-complex API app with everything contained in one file (the most prominent feature of the classical structure).

Now let's turn our attention to modular apps.

The code below shows a basic modular Sinatra app. At first glance, it looks pretty similar to the classic app we've already looked at — apart from a rather simple distinction. In modular apps, we subclass Sinatra::Base, and each "app" is defined within this subclassed scope.

# main.rb

require 'sinatra/base'
require 'json'
require_relative 'lib/fetch_game_data'

# main module/class defined here
class GameStoreApp < Sinatra::Base

  get '/' do
    content_type :json

    { item: 'Red Dead Redemption 2', price: 19.79, status: 'Available'  }.to_json
  end

  not_found do
    content_type :json

    { status: 404, message: "Nothing Found!" }.to_json
  end

end

Have a look at the Sinatra documentation in case you need more information on this.

Let's now continue with our app build.

Structuring Our Ruby App

To begin with, we'll take the modular approach with this build so it's easy to organize functionality in a clean and intuitive way.

Our cost-of-living calculator app needs:

A root page, which will act as our landing page.
Another page with a form where a user can input their salary information.
Finally, a results page that displays some living expenses for the chosen city.

The app will fetch cost-of-living data from an API hosted on RapidAPI.

We won't include any tests or user authentication to keep this tutorial brief.

Go ahead and create a folder structure like the one shown below:

.
├── app.rb
├── config
│   └── database.yml
├── config.ru
├── db
│   └── development.sqlite3
├── .env
├── Gemfile
├── Gemfile.lock
├── .gitignore
├── lib
│   └── user.rb
├── public
│   └── css
│       ├── bulma.min.css
│       └── style.css
├── Rakefile
├── README.md
├── views
│   ├── index.erb
│   ├── layout.erb
│   ├── navbar.erb
│   ├── results.erb
│   └── start.erb

Here's what each part does in a nutshell (we'll dig into the details as we proceed with the app build):

app.rb - This is the main file in our modular app. In here, we define the app's functionality.
Gemfile - Just like the Gemfile in a Rails app, you define your app's gem dependencies in this file.
Rakefile - Rake task definitions are defined here.
config.ru - For modular Sinatra apps, you need a Rack configuration file that defines how your app will run.
Views folder - Your app's layout and view files go into this folder.
Public folder - Files that don't change much — such as stylesheets, images, and Javascript files — are best kept here.
Lib folder - In here, you can have model files and things like specialized helper files.
DB folder - Database migration files and the seeds.rb will go in here.
Config folder - Different configurations can go into this folder: for example, database settings.

The Main File (`app.rb`)

app.rb is the main entry point into our app where we define what the app does. Notice how we've subclassed Sinatra::Base to make the app modular.

As you can see below, we include some settings for fetching folders as well as defining the public folder (for storing static files). Another important note here is that we register the Sinatra::ActiveRecordExtension which lets us work with ActiveRecord as the ORM.

# app.rb

# Include all the gems listed in Gemfile
require 'bundler'
Bundler.require

module LivingCostCalc

  class App < Sinatra::Base

    # global settings
    configure do
      set :root, File.dirname(__FILE__)
      set :public_folder, 'public'

      register Sinatra::ActiveRecordExtension
    end

    # development settings
    configure :development do
      # this allows us to refresh the app on the browser without needing to restart the web server
      register Sinatra::Reloader
    end

  end

end

Then we define the routes we need:

The root, which is just a simple landing page.
A "Start here" page with a form where a user inputs the necessary information.
A results page.

# app.rb

class App < Sinatra::Base
...

  # root route
  get '/'  do
    erb :index
  end

  # start here (where the user enters their info)
  get '/start' do
    erb :start
  end

  # results
  get '/results' do
    erb :results
  end

  ...
end

You might notice that each route includes the line erb :<route>, which is how you tell Sinatra the respective view file to render from the "views" folder.

Database Setup for the Sinatra App

The database setup for our Sinatra app consists of the following:

A database config file — database.yml — where we define the database settings for the development, production, and test databases.
Database adapter and ORM gems included in the Gemfile. We are using ActiveRecord for our app. Datamapper is another option you could use.
Registering the ORM extension and the database config file in app.rb.

Here's the database config file:

# config/database.yml

default: &default
  adapter: sqlite3
  pool: 5
  timeout: 5000

development:
  <<: *default
  database: db/development.sqlite3

test:
  <<: *default
  database: db/test.sqlite3

production:
  adapter: postgresql
  encoding: unicode
  pool: 5
  host: <%= ENV['DATABASE_HOST'] || 'db' %>
  database: <%= ENV['DATABASE_NAME'] || 'sinatra' %>
  username: <%= ENV['DATABASE_USER'] || 'sinatra' %>
  password: <%= ENV['DATABASE_PASSWORD'] || 'sinatra' %>

And the ORM and database adaptor gems in the Gemfile:

# Gemfile

source "https://rubygems.org"

# Ruby version
ruby "3.0.4"

gem 'sinatra'
gem 'activerecord'
gem 'sinatra-activerecord' # ORM gem
gem 'sinatra-contrib'
gem 'thin'
gem 'rake'
gem 'faraday'

group :development do
  gem 'sqlite3' # Development database adaptor gem
  gem 'tux' # gives you access to an interactive console similar to 'rails console'
  gem 'dotenv'
end

group :production do
  gem 'pg' # Production database adaptor gem
end

And here's how you register the ORM and database config in app.rb.

# app.rb

module LivingCostCalc

  class App < Sinatra::Base
    # global settings
    configure do
      ...
      register Sinatra::ActiveRecordExtension
    end

    # database settings
    set :database_file, 'config/database.yml'

    ...
  end
end

Connecting to the Cost-of-Living API

For our app to show relevant cost-of-living data for whatever city a user inputs, we have to fetch it via an API call to this API. Create a free RapidAPI account to access it if you haven't done so.

We'll make the API call using the Faraday gem. Add it to the Gemfile and run bundle install.

# Gemfile

gem 'faraday'

With that done, we now include the API call logic in the results method.

# app.rb
...
  get '/results' do
    city = params[:city]
    country = params[:country]

    # if country or city names have spaces, process accordingly
    esc_city = ERB::Util.url_encode(country) # e.g. "St Louis" becomes 'St%20Louis'
    esc_country = ERB::Util.url_encode(country) # e.g. "United States" becomes 'United%20States'

    url = URI("https://cost-of-living-prices-by-city-country.p.rapidapi.com/get-city?city=#{esc_city}&country=#{esc_country}")

    conn = Faraday.new(
      url: url,
      headers: {
        'X-RapidAPI-Key' => ENV['RapidAPIKey'],
        'X-RapidAPI-Host' => ENV['RapidAPIHost']
      }
    )

    response = conn.get

    @code = response.status
    @results = response.body

    erb :results
  end

  ...

Views and Adding Styles

All our views are located in the "views" folder. In here, we also have a layout file — layout.erb — which all views inherit their structure from. It is similar to the layout file in Rails.

# views/layout.erb

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>Cost of living calc app</title>
    <link
      rel="stylesheet"
      href="css/bulma.min.css"
      type="text/css"
      rel="stylesheet"
    />
    <link rel="stylesheet" href="css/style.css" rel="stylesheet" />
  </head>
  <body>
    <!-- navbar partial -->
    <%= erb :'navbar' %>
    <!-- //navbar -->

    <div><%= yield %></div>
  </body>
</html>

We also add a local copy of Bulma CSS and a custom stylesheet in public/css to provide styling for our app.

Running the Sinatra App

To run a modular Sinatra app, you need to include a config.ru file where you specify:

The main file that will be used as the entry point.
The main module that will run (remember that modular Sinatra apps can have multiple "apps").

# config.ru
require File.join(File.dirname(__FILE__), 'app.rb')
run LivingCostCalc::App

Deploying Your Sinatra App to Production

A step-by-step guide for deploying a Sinatra app to production would definitely make this tutorial too long. But to give you an idea of the options you have, consider:

Using a PaaS like Heroku.
Using a cloud service provider like AWS Elastic Cloud or the likes of Digital Ocean and Linode.

If you use Heroku, one thing to note is that you will need to include a Procfile in your app's root:

web: bundle exec rackup config.ru -p $PORT

To deploy to a cloud service like AWS's Elastic Cloud, the easiest method is to Dockerize your app and deploy the container.

Monitoring Your Sinatra App with AppSignal

Another thing that's very important and shouldn't be overlooked is application monitoring.

Once you've successfully deployed your Sinatra app, you can easily use Appsignal's Ruby APM service. AppSignal offers an integration for Rails and Rack-based apps like Sinatra.

When you integrate AppSignal, you'll get incident reports and dashboards for everything going on.

The screenshot below shows our Sinatra app's memory usage dashboard.

Wrapping Up and Next Steps

In this post, we learned what Sinatra is and what you can use the framework for. We then built a modular app using Sinatra.

You can take this to the next level by building user authentication functionality for the app.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Manage Your Ruby Logs Like a Pro

Roy Tomeij — 2023-05-17T00:00:00+00:00

Logs are essential to any application's development. Most Ruby logs are verbose and chunky, so digging for exactly what you need can be difficult. Even though they contain useful information, you might not get as much value as you should from logs if you don't know how to use them effectively.

In this article, we'll explore:

The importance of logging
How to use Ruby's inbuilt logging utilities — puts and Logger
Log levels and their differences
Customizing your logs using JSON
Integrating a logging library, using Lograge as an example

Let's get started.

Prerequisites

To follow along with this tutorial, you should have the following:

A local installation of Ruby (we'll be using Ruby 3.2.0 for the tutorial).
Some experience with Ruby.

Let's start by answering the question: "Why do we need to log, anyway?"

Why We Log

Before we dive in, it helps to know exactly why we log in the first place.

Logs provide us with information on what's happening inside our Ruby app. This is very useful for tasks like debugging errors and collecting critical app metrics. Log data depends on the type of app we have and the environment it is running on.

With that out of the way, let's introduce Ruby's inbuilt logging utilities.

Ruby’s Inbuilt Logging Utilities: `puts` and Logger

Ruby comes with two logging utilities. One is puts, a method provided by the IO class (other methods include print, printf, and write).

The other inbuilt logging utility is the Logger class.

Let's explore each in more detail, starting with puts.

Ruby's `puts` Method

Ruby's puts method — which is actually syntactic sugar for STDOUT.puts — is a way of writing Ruby program messages to the STDOUT output stream (usually a terminal output). puts will print out the given message string and add a new line to the end of the output.

The example below shows a simple use case where we use puts to output each line's length from a given text file:

# line_lengths.rb
ARGV.each do |f|
  File.open(f).each_line do |line|
    puts line.length
  end
end

# MyFile.txt
Ruby is an awesome language to learn.
It's very simple to get started.

Then we run the above Ruby script and pass in the text file as the argument:

ruby line_lengths.rb MyFile.txt

And once the script has run, each line's length is written out to the console:

# output
38
33

This simple example provides a glimpse into the world of the input/output (IO) streams at the heart of logging. Basically, anytime you run a Ruby program, you spawn the following IO streams:

STDIN - This stream accepts commands into your Ruby program — for example, a key pressed on a user's keyboard.
STDOUT writes to an output — for example, a terminal displayed on a user's screen.
STDERR writes to an output just like the STDOUT one, but with a subtle difference that we'll cover in the next section.

Each of these is integral to logging and understanding how they work forms the basis of properly applying logs in Ruby apps.

The Limitations of `puts` in Ruby

Even though puts is a convenient and easy logging method for simple Ruby programs, you will very quickly need a proper logging library instead.

Here are a couple of its limitations:

Automatic timestamps aren't included by default - Unless you manually customize puts, you will not get any timestamps in your log output.
Output is not categorized into appropriate log levels - We'll cover this in more detail later, but the puts method has no way to categorize output messages according to standard log levels.

These features are basic requirements for any logging solution. The fact that they are missing from the puts method by default (coupled with the manual customization needed to have meaningful output messages) means that our ability to properly log using the method is limited.

So let's now consider a more robust solution, the Ruby Logger class.

Logging with Ruby Logger

The Logger class is a sophisticated logging utility that can create single or multiple event logs from a Ruby application. Logger comes bundled in Ruby, so no gem installation is needed to get it up and running. You only need a simple require statement somewhere at the top of your script:

require 'logger'

After that, instantiate a new Logger object as below:

require 'logger'

logger = Logger.new(STDOUT)
logger.info("This is an info level log message")

Note: The Logger class can accept a number of arguments, but the first one is almost always the log stream output, followed by other arguments like the retention policy (how long logs should be kept, how much the log files should grow in size before they are rotated, etc.)

You will get output similar to this:

# output
I, [2023-01-11T17:45:49.067481 #67559]  INFO -- : This is an info level log message

From the printed log message, we have:

The log level's shortened format - Indicated by the leading "D". In this case, it shows that this is an info level log entry.
A timestamp - A full timestamp of the log entry.
A process ID - The ID of the server process running the program.
The log level type in full.
A program name - An optional argument that can be added to a Logger entry (we'll see how in a later code example).
The log message - The message output that will be printed in the log entry (also optional).

Each of these log entry components is important, and together they give context to app events that lead to the creation of log entries.

That said, one aspect of log entry that deserves more coverage is the log level. Let's take a look at it next.

Understanding Log Levels

A log level setting tells the Logger a log message's severity. The severity level depends upon the event that resulted in that log output.

There are six levels of severity, ranging from the least to the most severe:

# example.rb
require 'logger'

logger = Logger.new(STDOUT)

logger.add(Logger::DEBUG, 'Least severe - debugging info')
logger.add(Logger::INFO, 'Next level - informational log')
logger.add(Logger::WARN, 'Warning level - warning log info')
logger.add(Logger::ERROR, 'Non-fatal error log info')
logger.add(Logger::FATAL, 'Fatal error log info')
logger.add(Logger::UNKNOWN, 'Most severe log level')

And here's the output:

# output
D, [2023-01-11T18:19:47.919567 #69315] DEBUG -- : Least severe - debugging info
I, [2023-01-11T18:19:47.919622 #69315]  INFO -- : Next level - informational log
W, [2023-01-11T18:19:47.919645 #69315]  WARN -- : Warning level - warning log info
E, [2023-01-11T18:19:47.919661 #69315] ERROR -- : Non-fatal error log info
F, [2023-01-11T18:19:47.919675 #69315] FATAL -- : Fatal error log info
A, [2023-01-11T18:19:47.919689 #69315]   ANY -- : Most severe log level

Log levels can help filter out app and server events that warrant your attention. To put this into context, consider these differences between log levels:

debug - This is the default log level and the most information-rich. When this log level is enabled, your log entries will include pretty much every request happening in your app. This is particularly useful for a debugging session.
info - A level above debug. Just like the previous log level, enabling this level will result in some very information-rich log entries of your app's events. info log entries will tell you what's going on with your app — for instance, running processes and method calls.
warn - Any log entries at this level usually mean that your app encountered a problem, but nothing so severe that your app can't continue to function as expected (for example, you may have used an outdated method call for a gem or library.)
error - At this level, your app has encountered an error that stops it from proceeding with the action that caused the error. However, this does not stop other parts of the app from working as expected. For example, if your app references a recently moved file, an error log entry will likely be generated.
fatal - Your app has encountered an error that stops it from working any further. This could be caused by anything from running out of memory on the production server or a crashed background job.
unknown - The actual cause of an error cannot be reliably traced.

Now that we have more information on log levels, let's see how to practically use them in a Ruby app.

Using Log Levels in a Ruby Application

In a development environment, the default log level is set to debug. As mentioned earlier, if your app's log level is set to debug (or even info), your output will include everything happening in the app. You will have very information-rich log entries that make for easier debugging in development.

However, in production, it isn't wise to use the debug or info log levels.

For starters, a production app will likely have too many processes, users, and method calls going on, so could very easily generate thousands (if not millions) of log entries. If these entries are being saved to log files on a production server, you can quickly run out of disk space.

Secondly, any attempt to sift for important error and fatal entries is extremely inefficient in production simply because you will have too much information to go through. So, you should only log entries when something "important" happens or when your app malfunctions in such a way that the user experience is affected.

It therefore makes more sense to set the log level at something more manageable like the error level, for example.

Although setting the log level can reduce the amount of information in the log entries, it doesn't help format the entries in any way. To do that, you need to customize the log output.

Customizing Log Output in Ruby

Customizing log output involves changing the format of how log entries look and the information they contain when printed out.

By default, log entries printed to the terminal will be in black and white text. But for some logs, a little color can go a long way. You can colorize default log output using ANSI color codes.

In Ruby, this is possible using a gem like Colorize or Rainbow.

You can format the printed output from your Ruby program using either of these gems. This example code uses Rainbow to colorize some sample log outputs:

# example.rb
require 'logger'
require 'rainbow'

logger = Logger.new(STDOUT)
logger.debug("This is a #{Rainbow('debug message!').blue}")
logger.info("This is a #{Rainbow('info message!').green}")
logger.warn("This is a #{Rainbow('warning message!').orange}")

And here's the colorized log output:

The other way to format log entries goes beyond just customizing the look and feel. First, consider the simple Logger entry below:

# output
D, [2023-01-23T15:29:35.924904 #11021] DEBUG -- my_prog: This is a debug level log entry

It's good to note that you are not limited to using the log entry as is. It is entirely possible to customize a log message by tweaking the log entry components. Let's see how.

The first step is to call Logger.new, and then add some log entries:

# example.rb
logger = Logger.new(STDOUT)

logger.info('This is an informational level log')
logger.error('This is just an error log')

Which gives us this output:

# output
I, [2023-01-23T16:24:34.315029 #12604]  INFO -- : This is an informational level log
E, [2023-01-23T16:24:34.315112 #12604] ERROR -- : This is just an error log

It's also possible to customize a program name in the log entry, like so:

# example.rb
logger = Logger.new(STDOUT)

logger.progname = 'sending emails'

logger.info('This is an informational level log')
logger.error('This is just an error log')

And here's the subsequent output:

# output
I, [2023-01-23T16:45:55.156600 #14183]  INFO -- sending emails: This is an informational level log
E, [2023-01-23T16:45:55.156660 #14183] ERROR -- sending emails: This is just an error log

By customizing the log entry's program name, you can add even more contextual information to your logs.

Now, let's look at how to customize all of the log entry's components all at once:

# example.rb
logger = Logger.new(STDOUT)

logger.formatter = proc do |severity, datetime, progname, msg|
  date_format = datetime.strftime("%Y-%m-%d")
  "date=[#{date_format}] severity=#{severity.ljust(5)} pid=##{Process.pid} message='#{msg}'\n"
end

logger.debug("This is a formatted log message!")

# output
date=[2023-01-12] severity=DEBUG pid=#75142 message='This is a formatted log message!'

Here, we use the logger entry formatter proc, which takes four arguments — severity, datetime, the program's name, and the message — to format the log entry and output a string formatted in the way specified.

Great — but we can still do better! So far, everything we've done has produced log outputs in an unstructured format. But log entries can build up quickly in a production app that has thousands of users and millions of processes. Parsing unstructured logs can take up significant time and is very inefficient.

What if there was a way to quickly and efficiently search through log files? Enter JSON.

Logging into JSON

The Javascript Object Notation format — or JSON — is a structured text format that uses key/value pairs.

It's not the only format that has structured text — there are others like YAML and even TOML. Both YAML and TOML function best as configuration files, though, while JSON's structure makes it really good for logging.

Here's some code that outputs a JSON-formatted log entry:

# example.rb
require 'logger'
logger = Logger.new(STDOUT)
logger.formatter = proc do |_severity, datetime, _progname, msg|
  %({timestamp: "#{datetime}", message: "#{msg}"}\n)
end
logger.info 'Hello, world!'

# output
{timestamp: "2023-02-13 17:47:13 +0300", message: "Hello, world!"}

At first glance, it's not easy to see why one would even want to do something like this. But with JSON formatted logs, it's possible to:

Easily parse logs as they are now structured in a particular format (compared to the mass of unstructured entries characteristic of default logs). You can search for particular pieces of information using JSON keys tied to specific values.
Forward these logs to a proper log processing tool that can process log outputs of almost any size.

So we've seen how useful Logger can be. But that does not mean the library is perfect.

Logger does have its imperfections, and we'll look at these next.

Limitations of Logger

If you use Logger to write output to log files frequently, one challenge you'll likely face is unreadable characters messing up your log files. These are caused by colorization ANSI codes not being parsed correctly in the log output files.

For example, consider this example code:

# example.rb
require 'logger'
require 'rainbow'

msg = Rainbow("this is red").red + " and " + Rainbow("this on yellow bg").bg(:yellow) + " and " + Rainbow("even bright underlined!").underline.bright

logger = Logger.new('colorized.log')

logger.debug(msg)

If you try to open the resulting log file, you'll notice it contains a bunch of unreadable characters in the log message section:

# output
# Logfile created on 2023-01-23 23:09:12 +0300 by logger.rb/v1.5.3
D, [2023-01-23T23:09:12.920635 #31599] DEBUG -- : ^[[31mthis is red^[[0m and ^[[43mthis on yellow bg^[[0m and ^[[4m^[[1meven bright underlined!^[[0m

One way to fix this problem is by using a string regex to escape these characters. For example, the Colorize gem contains a convenient uncolorize block:

# example.rb
...
def uncolorize
  self.scan(REGEXP_PATTERN).inject("") do |str, match|
    str << (match[3] || match[4])
  end
end
...

Another limitation of Logger, especially for busy production apps, is that default Logger entries can end up as information-rich multi-line entries (even when the log level has been set to something above the debug and info levels, like the error level).

If you're in charge of such an app, debugging can easily become a messy affair as you have to go through a mass of unstructured text to get what you want.

Considering these limitations, let's seek a better logging solution — one that has the features we've been lacking in most of the solutions we've outlined so far.

Integrating a Third-Party Logging Library

You can choose from a number of third-party logging libraries, including Logging — based on Java's log4j library — and Lograge. Lograge is a feature-rich logging library meant to simplify the often messy and verbose Rails logs characteristic of the default application logger.

Also worth mentioning is httplog, a specialized logging library for tracking third-party API calls that might not be captured in regular logs.

For this article, we'll look at Lograge.

Lograge for Logging in Rails

According to Lograge's GitHub page, Lograge is an attempt at bringing sanity to Rails' unusable and unparsable log output.

Lograge replaces the default Rails logger and simplifies output to single lines, with all the important information included. The resulting output is much easier to read through than the Rails logger.

Getting Started with Lograge

To get started with Lograge, open up your project's Gemfile and add the following line:

# Gemfile

gem "lograge"

Then run bundle install to finalize the installation.

Logging in Rails: Using Lograge with AppSignal

With Lograge installed, you're halfway through setting up a capable logging solution for your app. The next obvious question is what you'll use to read and parse the logs. SSHing into your server to read through log tails is highly inefficient, so what to do?

Well, AppSignal has a new logging feature that is perfect for this.

To get started, install the latest version of the AppSignal for Ruby gem by adding it to your Gemfile:

# Gemfile

gem 'appsignal'

Then run bundle install. If you already have the gem installed in your project, just run bundle update appsignal to get the latest version.

Next, add an initializer with the following information:

# config/initializers/lograge.rb

Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.keep_original_rails_log = true
  config.lograge.logger = Appsignal::Logger.new(
    "rails",
    format: Appsignal::Logger::LOGFMT
  )
end

And that's it! You can now conveniently access your app's logs from AppSignal's logging dashboard:

Wrapping Up

In this article, we took a deep dive into the world of Ruby logs. We learned about puts, the Logger class, and how to customize log messages using JSON. We finally explored logging using Lograge and AppSignal.

Obviously, logging in Ruby goes much deeper. Use this post as a stepping stone to learn more. You might also find our Making the Most of Your Logs in Rails and Audit Logging in Ruby and Rails posts useful.

Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Organize Business Logic in Your Ruby on Rails Application

Roy Tomeij — 2023-05-10T00:00:00+00:00

With its strong emphasis on convention over configuration, Ruby on Rails has counteracted many architectural considerations that caused bikeshedding when building web applications.

Still, one area that has continuously piqued developers' interest is how to handle business logic, i.e., code that epitomizes "what an app does." Another way to phrase this question is: Where do we put all the transactional code?

In this first part of a two-part series, we'll explore the most well-known methods to organize your business logic in a Ruby on Rails application.

Let's get going!

Where to Put Transactional Code in Rails

There has been an ongoing debate about this topic in the Rails community, with two extremes being advocated for, and a couple of intermediate solutions.

On one hand, there's the "Golden Path", which fully embraces MVC and is most prominently followed by 37Signals.

On the other hand, there is Domain Driven Design (DDD), practiced by companies like Shopify, and consultancies like Arkency.

These alternative approaches look pretty heavy-handed. The one you should choose depends very much on the following:

how many business domains your app spans (DDD)
organizational structure
app architecture/infrastructure (microservices/monolith, k8s/cloud, etc.)

Let's take a closer look at some of the familiar alternatives, and their pros and cons, starting with fat models.

Fat Models

Fat models are models equipped with methods for processing their data, possibly resulting in side effects and updating other records. Consider this example, simplified from Jorge Manrubia's post about rich models:

# app/models/recording
class Recording < ApplicationRecord
  def copy_to(bucket, parent: nil)
    copies.create! destination_bucket: bucket, destination_parent: parent
  end
end

One problem with fat models is that unless you are really disciplined in your code hygiene, they can tend towards an assortment of code smells:

Feature Envy: Models reaching out to other models, either to query their internals (a "tell, don't ask" violation) or to actually mutate them, are a red flag 🚩. To achieve low coupling, objects should be confident just sending messages to other objects without querying them for the results.
Single Responsibility Principle Violations: Models encoding interactions with other models may break the rule of single responsibility. Keep rigorously asking yourself if the code you are adding to a class really belongs to the representation of the object it describes.
Logic in Callbacks: An enduring controversy, model callbacks are really a double-edged sword. A powerful tool, they can simplify a lot of imperative code, but can also hide complexity and lead to bugs that are hard to track down. Here is a rule of thumb you can follow: only use model callbacks to prepare or post-process self. Never trigger any jobs, mailers, or other services from them.
Tendency towards God Objects: Rich models act as attractors for functionality. Wait a few development cycles, and I'll take any bet there'll be at least one model accumulating a few dozen methods. Even splitting up such an object into model concerns merely covers up the mess 🌶️.

The points listed above can be summarized as a question of perspective. To paraphrase Jim Gay, is a look from within an individual model actually a good vantage point to design what your app does?

This has been picked up by a recent advance in Rails development: namely, service objects.

Rails Service Objects

Really another title for the Command Pattern, service objects encapsulate a "unit of work", or "action", that usually involves several steps of transactional logic. Following the above example, you would typically formulate something like this:

# app/services/recording_copier.rb
class RecordingCopier
  def call(source, destination)
    source.copies.create! destination_bucket: destination
  end
end

There are multiple problematic facets of this pattern, but the focal point is:

This object does not encapsulate an instance state.

This is a sign that service objects do not actually describe any concept of our domain, as Jason Swett has noticed.

Others - like Avdi Grimm - have observed that service objects run counter to the accepted practice of representing your domain objects with nouns, and sends messages to these objects with verbs. Classes named using the nominalization of a verb indicate that you have identified a message you want to send but can't come up with a receiver, i.e., a matching role to send it to. Objects with such fuzzy responsibilities can be the hardest to refactor.

First and foremost, though, I always found this concept a bit confusing, because Rails already has a built-in primitive for this: jobs.

Jobs in Rails

In my consulting practice, I've observed that jobs are generally underused because their limits and requirements can seem daunting. Let's first rewrite our example as a job:

# app/jobs/copy_record_job.rb
class CopyRecordJob < ApplicationJob
  def perform(source, destination)
    source.copies.create! destination_bucket: destination
  end
end

As you can see, there's almost no syntactical difference between a job and a service object. What, then, sets them apart?

A couple of things, actually:

The code above isn't idempotent. If you run it twice, it will create two copies, which is probably not what you intended. Why is this important? Because most backend job processors like Sidekiq don't make any guarantees that your jobs will run exactly once.
Jobs cannot return a value or indicate when they are done out of the box (although you can do this manually, via callbacks, for example).

There are several workarounds for this, like the magnificent Acidic Job gem or Sidekiq Pro/Enterprise features around enhanced reliability and unique jobs. Still, if they occur, bugs related to missing jobs and/or job idempotency are hard to track down and even harder to fix.

Event Sourcing

Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states.

Source: Event Sourcing by Martin Fowler

Essentially, event sourcing boils down to a publish/subscribe algorithm with integrated versioning. It's a high-level Domain Driven Design concept that I will not discuss in detail here.

That's not to say it's not an interesting pattern. You should use it if you have advanced reporting requirements, for example. If you want to learn more about it, look at Rails Event Store.

Up Next: DCI (Data, Context, Interaction) for Rails

We looked at a few of the most popular approaches to organize your business logic in a Ruby on Rails application: fat models, service objects, jobs, and (briefly) event sourcing.

In the next and final part of this series, we will look at a convenient alternative called DCI (Data, Context, Interaction). DCI caters particularly well to the mental models we employ as engineers when we reason about application behavior.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Integrate and Troubleshoot Inbound Emails with Action Mailbox in Rails

Roy Tomeij — 2023-05-03T00:00:00+00:00

If you’ve ever looked at the Request for Comments (RFCs) around sending and receiving emails, you’ll see the technical complications involved when hitting send in your email inbox.

Thankfully, many existing tools provide the Simple Mail Transfer Protocol (SMTP) service for us developers — from a Postfix server you manage to a fully scalable sending service such as SendGrid, Amazon SES, or Postmark. However, moving between providers for deliverability or pricing reasons means rewriting or refactoring our apps to meet the peculiarities of each service.

Rails helps us out here by providing Action Mailbox. In this post, we'll dive into how you can use Action Mailbox to integrate and troubleshoot inbound emails.

But first, let's quickly define what Action Mailbox is.

What Is Action Mailbox for Rails?

Action Mailbox uses conceptual compression for receiving email in Ruby on Rails. Conceptual compression means it encapsulates all the small differences between every SMTP service and writes your inbound processing code just once. You can even write a provider for a new service.

The key concept of ActionMailbox is routing based on email recipients. By setting up an inbound email provider, mail going to a domain will be routed into your app. You can look at the recipient address to determine how each mail message should be processed.

If you go through the rails conductor action to send a test email, as I have here:

Then your inbound email will have recipients from the To, CC, BCC, and X-Original-To fields.

> inbound_email = ActionMailbox::InboundEmail.last
> inbound_email.mail.recipients
["to@john.onrails.blog", "cc@john.onrails.blog", "bcc@john.onrails.blog", "original@john.onrails.blog"]

Each address is tested to determine where it will be routed, but the mail message is only routed once.

One key aspect of development is actually testing email from your system. Rails has a set of development pages under the routes /rails/conductor/ that allow you to input emails locally into your development setup.

You can enter the email manually, like I did in the above example, or you can upload an email complete with all the headers.

A great way to get a complete email (with headers, a message body, and attachments) is to use an email client like Thunderbird. Save the individual email in a .eml, open the file with a text editor, and copy the complete contents into the conductor page.

Now you can test more complicated email processing.

Posting and Commenting Demo for a Rails App

Let’s put together a small demo to show how this all works. I’m a great admirer of 37Signals, and I especially like their blogging with Hey World. But they don’t allow comments, so let’s create a clone that includes comments for each blog post.

Follow along with the code in this post.

Create a new app (I’m using Tailwind CSS, but you can pick what makes sense to you). I’ll also add Action Text for the Post and Comment models.

$ rails new BlogWorld -c tailwind
$ cd BlogWorld
$ bin/rails action_text:install
$ rails g scaffold Post title:string author:string content:rich_text
$ rails g scaffold Comment author:string content:rich_text post:references

The scaffolding gives us a quick way to see the Posts and Comments. Add the association in post.rb, and the post will display related comments:

class Post < ApplicationRecord
  has_many :comments
  has_rich_text :content
end

In posts/_post.html.erb, let's add the comments partial:

<div class="ml-12 pl-4 my-4 border-l-2 border-green-500">
  <%= render post.comments %>
</div>

We now have a sparse post and comment view. Set up an inbound email for posting to the blog:

$ bin/rails action_mailbox:install
$ bin/rails g mailbox Post

This will generate the ApplicationMailbox. We’ll set up a route so that anything for blog@ goes to our Post Mailbox and creates a post.

class ApplicationMailbox < ActionMailbox::Base
  routing /blog@/i => :post
end

You can test this quickly by going to http://localhost:3000/rails/conductor/action_mailbox/inbound_emails and sending some emails to your service. If you send something to blog@whatever.com, the email should be delivered to an inbox on our app. If you email any other address, the message will bounce.

Receive Email with Post Mailbox

Let’s set up the Post Mailbox to receive the email and post it to the blog. Each Mailbox has access to the original inbound_email and mail objects. The InboundEmail is a wrapper around the mail class used throughout rails.

For our purposes, we’re interested in who the email is from, its subject, and its body copy. We can extract these and create a Post record that will show up on our blog's front page.

class PostMailbox < ApplicationMailbox
  def process
    Post.create title: mail['subject'].to_s, author: mail['from'].to_s, content: mail.body.to_s
  end
end

Send another email to your blog address and then refresh the index page. You should see the post!

Add Comments for a Post in Action Mailbox

Now to add comments for a post. First, any email commenter needs to refer to the correct post when sending an email. A simple way to do this is to encode the post ID in the inbound email address (like comment+123@whatever.com, where the 123 in the email address refers to a Post element).

Generate the CommentMailbox:

$ bin/rails g mailbox Comment

Add a route in Action Mailbox to send any emails with comment+123 to the CommentMailbox:

routing /^comment\+\d+@/i => :comment

In the _post.html.erb, add a link to generate the email address, so someone can open their email app and send an email:

<div class="ml-12 pl-4 my-4 border-l-2 border-green-500">
  <%= render post.comments %>
  <%= mail_to "comment+#{post.id}@whatever.com", class: "rounded-lg py-3 px-5 bg-gray-100 inline-block font-medium" %>
</div>

The incoming email will be routed to the CommentMailbox and parsed into a comment attached to the correct blog post.

class CommentMailbox < ApplicationMailbox
  def process
    Comment.create author: mail["from"].to_s, content: mail.body.to_s, post: post
  end

  def post
    return @post unless @post.nil?
    email = mail.recipients.reject { |address| address.blank? }.first
    match = email.match(/^comment\+(.*)@/i)
    token = match[1]

    begin
      if token
        @post = Post.find_by_id(token)
      else
        bounced!
      end
    rescue RecordNotFound
      bounced!
    end
  end
end

The process method creates a comment from the email body and sender email. It references the Post queried in the post method. This method gets the first recipient's email address and uses a regular expression to pull out the post ID.

If the Post doesn’t exist or a token can’t be parsed, the email bounces, which stops the processing.

Now go to the Rails conductor form and send a comment to the address for each Post. The comment will appear underneath the post on the index page!

A More Complex Example Using Action Mailbox

Emails are actually really complicated. Imagine you've got an application monitoring tool set up, you deploy something like this to your app, and you start seeing errors in your APM dashboard.

You may see a parsing error, or that posts/comments have a lot of weird formatting errors.

Your app receives HTML emails, and you take the raw body source and post it to the website. The mail gem allows us to see if the incoming email has an HTML body, and we can pull whatever parts from the message we need.

Let’s change the CommentMailbox and PostMailbox to check for multipart emails and pull out the HTML part, falling back to text if that’s the only thing left.

Each email either has no parts or multiple parts. The preferred order is to see if there is an HTML part and use it, and if not, try to get and use the text part. If there aren’t parsed HTML or text sections, we’ll use the email body as before.

The PostMailbox is now a little more complicated:

class PostMailbox < ApplicationMailbox
  def process
    post = Post.new title: mail["subject"].to_s, author: mail["from"].to_s
    post.content = if mail.html_part
      mail.html_part.decoded
    elsif mail.text_part
      mail.text_part.decoded
    else
      mail.decoded
    end
    post.save
  end
end

The CommentMailbox also has a different process method:

def process
  comment = Comment.new author: mail["from"].to_s, post: post
  comment.content = if mail.html_part
    mail.html_part.decoded
  elsif mail.text_part
    mail.text_part.decoded
  else
    mail.decoded
  end
  comment.save
end

Now we can handle emails coming from someone’s phone.

Adding Action Mailbox to Your Rails App

Thanks to Action Mailbox, we can consider emails as another I/O avenue for our Rails app. We can write code independent of email service providers using conceptual compression. I’ve even been able to move email providers with minimal work since I don’t have to worry about the underlying infrastructure.

APM tools like AppSignal also provide a convenient dashboard to monitor all your outgoing ActionMailers and keep an eye on deliverability.

Here’s an example, showing one of my apps that sends and receives lots of emails:

This gives you more visibility into what’s happening inside your app.

Wrapping Up

In this post, we first defined the capabilities of Action Mailer for Rails. We then set up a demo project where we integrated inbound emails and parsed them to create posts for a blog.

I hope you've found this useful. Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Load Code in Ruby

Roy Tomeij — 2023-04-19T00:00:00+00:00

There are many ways to load code and access file-related constants in Ruby. We can create a clear architecture by separating and handling concerns into classes and pulling in only the classes we depend on.

Many full-stack frameworks like Rails and Hanami offer a built-in method to access the classes we want, as long as we stick with a certain convention. How does this work?

In this post, we will explore three different options for loading code: using load, require, and autoload. We will also look into the Ruby gem Zeitwerk. Zeitwerk is the default code-loading mechanism for many new projects, and a lot of established projects are switching to this great gem. Rails and Hanami both utilize Zeitwerk as their code-loading tool.

Before we dive into the code loading options on offer in Ruby, let's take a quick look at $LOAD_PATH to get to grips with how code loading works.

`$LOAD_PATH` in Ruby

Before we dive deeper into this topic, let's summarize the $LOAD_PATH in Ruby, as it is essential in understanding how loading works.

The global variable $LOAD_PATH or shorter $: references all directories with Ruby source files registered in an array. So when you prepend bundle exec with your program name, Ruby adds all the program gems to the load path.

Then, when we load or require a file, Ruby iterates over this array and searches for the file name. This can add significant time to the booting process, as we might have to search for a given file in many places on the hard disk.

Keeping that in mind, we'll now explore some Ruby code-loading options in detail.

Options for Loading Code in Ruby

We have three main options for loading code:

Using the load method. This loads and parses a Ruby program in a specified file every time you call the method.
The require method. With this, we load and parse a given file only once.
The third option is autoload. We declare upfront that Ruby should require a specified file when we use a constant that doesn't exist yet.

`load` a File in Ruby

As mentioned, with load we parse and execute a Ruby file. If the filename we set as an argument is a relative path, i.e., it starts with ./ or ../, the file will be loaded relative to the current working directory.

Remember: This doesn't mean it's relative to the file where the load is located — it's instead relative to the file that started the Ruby process. If you want to know what the working directory is, run Dir.pwd. Otherwise, Ruby searches for a file with the given name in the $LOAD_PATH.

Let's see how load works:

# foo.rb
$x = 4 # define global Variable

class Foo
  def bar
    puts "I'm doing it now"
  end
end

Then we load the file with load "foo.rb".

What happens when we load this file? The global variable $x is created and set to the integer 4. The constant Foo is also created, and we can instantiate an object from it. So far, this seems just like the more familiar require call.

But what happens when we load this file again? If we have changed our global value $x, it resets to 4.

In older Ruby versions, we also might get a warning that the constant Foo is being redefined.

Another problem with load is that every reload takes some time to run as the Ruby VM is eval-ing the file. require, on the other hand, loads and parses a file just once, during the first run.

Wrapping the File with `load`

Right now, every time we load or require a file, we pollute the global namespace. To avoid this, we can use the load method with the wrap parameter set to true. This will create an anonymous module where all the constants and methods defined in a loaded file are placed. Here is an example:

# file to be loaded foo.rb

class Foo
  puts self.name # => Here we will see where our Foo class is located in
  def bar
    puts "I'm doing it now"
  end
end

###############################
# Now in another file run this:

load "foo.rb", true

If you run this, Ruby will output something like this: #<Module:0x00007fa3430682e8>::Foo.

Why is this useful? For one thing, we are not polluting the global namespace. But since Ruby creates an anonymous module, we can only access the stuff we load with some trickery.

Nevertheless, wrapping the file is still useful if we want to configure something in our application or set some things up. This way, we can achieve what we want without leaving anything behind.

Accessing the Wrapped File Constants

There are two ways to access the constants we create in the wrapped file. The first and most obvious is to set a module name as the argument for the wrap parameter.

class Foo
  def bar
    puts "I'm doing it now"
  end
end

###############################
# Now in another file run this:

module Parent
end
load "./code_loading/file.rb", Parent

Parent::Foo.new

Here is how we can have access to the anonymous Ruby module:

class Foo
  def bar
    puts "I'm doing it now"
  end
end

throw :wrapper, Module.nesting.last


############################################
# When we load the file we catch the  error
# and make the module accessible

mod  = catch :wrapper do
  load "./code_loading/file.rb", true
end

mod::Foo.new

There is a hidden problem with wrap. If the author of the loaded file does something like this, it will break the sandbox:

class ::Foo
  # stuff
end

Now Foo is in the global namespace, and there is no way to prevent this pollution.

Ruby Code Loading with `require`

Just like the load method, require is a method in the Kernel module. The big difference between these two methods is how they behave on multiple calls.

While load always reevaluates a file, require doesn't. Its response is also different. If you call require for the first time and the method actually loads something, it returns true, otherwise, it returns false. load, on the other hand, always returns true.

Some pseudo-code for require would look something like this:

def require(file_path)
  if !already_loaded_files? file_path
    eval(File.read(find_file_in_load_path(file_path)))
    mark_file_as_loaded! file_path
    true
  else
    false
  end
end

require accepts absolute paths for files as well as just a simple name. If we need a simple file, require also searches in the $LOAD_PATH. This, of course, means that we still have the same disadvantages we had with load.

Simpler File Searching with `require_relative`

require_relative always looks for a file that is relative to the current file you are working on. require_relative doesn't iterate over a directory, and no real search is involved. Ruby looks into the path specified and tries to load the file. If it isn't there, we get a LoadError.

Unfortunately, as of now, require doesn't have a wrap parameter like load does, so require always pollutes the global namespace. But there has been some recent discussion on the Ruby issue tracker to add something like wrap to the require method.

`autoload` Code Loading in Ruby

Our third option for code loading in Ruby is autoload. With autoload, we tell the Ruby VM upfront what constants might be accessed and where to find and load them.

A big advantage of this approach is that we only load the files we really need. Therefore, the booting process can be really fast as we just pay for what we use. When we actually need the constant, autoload requires the file with the method require.

Here's how we set up an autoload:

### lib/services/foo.rb
module Services
  class Foo
    def bar
      puts "bar"
    end
  end
end

### lib/services.rb
module Services
  autoload :Foo, "#{__dir__}/services/foo.rb"

  # Now we have access to the Foo constant in this module
  # and we will load the file when we actually need it.

  # Foo.new.bar => loads the file and prints "bar"
end

The most popular gem for managing loading files is Zeitwerk, and it works with autoload. Now let's see how Zeitwerk works and how to set it up.

Code Loading with Zeitwerk

Zeitwerk takes a directory and makes every file underneath it available to load. The convention is that every new sub-directory is a new module, and every file defines a class with the same name as the file.

Zeitwerk offers an option for code reloading that is useful during development. Rails uses this mode in an active development environment.

How does it work under the hood? Here is some very, very simplified pseudo-code:

module Zeitwerk
  class Loader
    def initialize()
      @root_dir = []
    end

    def push_dir(dir)
      @root_dirs << dir
    end

    def setup
      # scan root dirs and define autoloads for each file
      @root_dirs.each do |dir|
        Dir.foreach(dir) do |file|
          next if file == "." || file == ".."

          autoload file.class_name.to_sym, File.expand_path(file, dir)
        end
      end
    end
  end
end

loader = Zeitwerk::Loader.new
loader.push_dir(__dir__)
loader.setup

Now, with the last call to loader.setup, we get access to every constant defined in one of the files relative to the current file. Of course, this simplified code doesn't highlight many of the other great features that Zeitwerk has, like eager loading and handling namespaces. That's outside of the scope of this post, and I'll leave it to you to dig deeper and explore all Zeitwerk has to offer.

How to Set Up `Zeitwerk` with Roda

Since Rails and Hanami use Zeitwerk as their default, let's see how we can configure Zeitwerk for Roda:

### in config.ru

loader = Zeitwerk::Loader.new
loader.push_dir(__dir__)
loader.push_dir("#{__dir__}/lib")

if ENV['RACK_ENV'] == 'production'
  loader.setup
  Zeitwerk::Loader.eager_load_all
  run App
else
  loader.enable_reloading
  loader.setup
  run ->(env) {
    loader.reload
    App.call(env)
  }
end

If we configure our app like this, we have code reloading during development and eager loading in production. This way, the boot time during development is fast, as only the files that are immediately needed are loaded. In production, we load everything upfront and have a faster response time.

Wrapping Up

In this post, we first took a quick look at $LOAD_PATH in Ruby to lay the foundations of how code loading works. We then explored three options to load code in some detail: load, require, and autoload (which works well with the Zeitwerk gem).

So, how should we load our code? Most of the time, we should use require and Zeitwerk to automate that. If you have special requirements and want to put loaded code into a namespace, the load method with a wrap parameter is a good option.

Happy code loading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Audit Logging in Ruby and Rails

Roy Tomeij — 2023-04-12T00:00:00+00:00

An audit log is a documented record of events and activity within an information technology system. Audit logging is critical for application owners and administrators to track activity within their systems.

In this post, we'll first dive into what auditing entails and what to consider when audit logging. We'll then explore some options for implementing audit logs, including PaperTrail, Audited, AuditLog, AppSignal, and a custom implementation.

Let's get started!

Auditing and Audit Logs: An Introduction

Auditing provides an understanding of system behavior using verifiable, well-defined events. It has a different goal than application logging, which records general-purpose messages often used to diagnose or troubleshoot issues within an application.

Auditing is typically a foundational requirement for many systems, yet it is often one of the last concerns of development teams. The use of simple, effective patterns upfront can change this trend.

One motivation for teams to tackle this capability early on is the feedback it provides in terms of metrics for testing and evaluating systems. The observability of audit events can highlight system issues or even missed requirements.

Audit logs typically have a defined structure that includes the following event information:

Timestamp
Event type
Description
User or system process that initiated or requested the event
Impacted entities in the system
Any additional relevant detail

Any system, service, or device can have its own audit log. A collection of audit logs spanning a particular dimension is considered an audit trail of related activity. An audit trail could be specific to a user, application, or system entity. For example, the AWS CloudTrail service records all activity within an AWS account.

Audit logs support, among others, the following key functions and use cases:

Compliance and regulatory requirements
Security reviews and investigations of potential security breaches
Determining what users make particular changes in a system and when

Design Considerations for Audit Capabilities

Audit logging solutions typically involve a centralized log management system. Each application instance can directly send audit events to the centralized repository, or the solution may include agents running on compute nodes that facilitate this communication.

Distributed architectures can make auditing challenging because there are many sources of audit events throughout the infrastructure. Centralized solutions aim to funnel all of the disparate audit events through a central pipeline for processing.

Audit management systems provide storage and search capabilities for the collection of audit events. There are several persistence options, although many Ruby and Rails-based solutions use database persistence to store audit events as a natural extension of ActiveRecord capabilities.

As an alternative, cloud data streaming services can also be used. For example, AWS Kinesis is designed to handle extremely large volumes of streaming data and is well suited for the audit use case. Another option is to store audit data in AWS S3 buckets and then query it using AWS Athena.

What Constitutes an Audit Event in Ruby and Rails?

Countless events occur within a system, so one of the most important decisions is determining what exactly to audit. Audit events can be based on functional requirements (e.g., a user submits an order) or non-functional requirements (e.g., a security-relevant event occurs, such as a failed login attempt). Both types are important and serve different purposes. Certainly, security auditing is of foremost concern in today’s operating environment.

The level of detail is another key consideration. A single business transaction may involve updates to multiple database tables. While a change in an individual database entity could be the subject of an audit event, by itself this may not contain enough information about the overarching event taking place. It is often beneficial to audit the business function in addition to, or even in place of, individual database rows. Business events typically occur within the service or controller level code.

Some use cases require all of the detail you have. For example, requirements may dictate that all user HTTP requests must be audited. You may need to know which users queried for particular information in comprehensive security audits. As is often the case in software engineering, let your requirements be your guide in making these design decisions.

Ruby Audit Log Implementation Options

Most Ruby gems in this space are focused on the Rails use case, as it has ORM capabilities to identify changes to ActiveRecord instances and store them as audit events. This enables basic audit capabilities without much development effort. Keep in mind, however, the level of detail of your audit events. You may need to add code to specifically audit business events in your service or controller classes.

AppSignal also has a new logging feature as part of its integration for Ruby. Applications can log messages and structured event data from their applications which are stored on managed AppSignal servers. Once you've signed up, you can use AppSignal dashboards to search, filter, and view your logs.

PaperTrail

The PaperTrail gem is a popular choice for audit logging in Ruby. It is focused on tracking changes to ActiveRecord model classes using a versions table. It allows you to see how a model looks at any point in its lifecycle. Several configuration options determine what type of model changes result in creating an audit event, or version. For example, you can choose to audit all changes or only create audit events where designated attributes are modified.

If you have an application that requires maintaining a history of changes over time, you will find this gem particularly useful. There is a good deal of overlap between auditing and versioning capabilities, and PaperTrail provides a nice solution for both. A versions method is added to each model class.

You can easily see the changes between versions as well, as shown in this code snippet from the PaperTrail documentation:

widget = Widget.create name: 'Bob'
widget.versions.last.changeset # reads object_changes column
# {
#   "name"=>[nil, "Bob"],
#   "created_at"=>[nil, 2015-08-10 04:10:40 UTC],
#   "updated_at"=>[nil, 2015-08-10 04:10:40 UTC],
#   "id"=>[nil, 1]
# }
widget.update name: 'Robert'
widget.versions.last.changeset
# {
#   "name"=>["Bob", "Robert"],
#   "updated_at"=>[2015-08-10 04:13:19 UTC, 2015-08-10 04:13:19 UTC]
# }
widget.destroy
widget.versions.last.changeset
# {}

As long as your controller has a current_user method, it will track who is responsible for changes using the following callback:

class ApplicationController
  before_action :set_paper_trail_whodunnit
end

One drawback is the lack of a well-defined pattern for custom audit events. A workaround is to create a database entity that captures relevant audit information, although that somewhat defeats the purpose of this gem’s paradigm.

Audited

The Audited gem is another implementation similar in scope to PaperTrail. It now only focuses on ActiveRecord support. An opt-in approach is used to denote which model classes should be audited, as shown below.

class User < ActiveRecord::Base
  audited
end

Model classes get a revisions method similar to PaperTrail’s versions. Audited models also get an audits method that provides change-tracking information.

However, this choice suffers from the same drawback as PaperTrail: a lack of support for custom audit events.

AuditLog

AuditLog takes a different approach. Rather than centering the design around model classes, it uses explicit calls to create audit events, as shown in the code below.

AuditLog.audit!(:update_password, @user, payload: { ip: request.remote_ip })
AuditLog.audit!(:sign_in, @user, payload: { ip: request.remote_ip })
AuditLog.audit!(:create_address, nil, payload: params)

If you want to audit individual model entities, you can use code similar to the following to accomplish this.

class TicketsController < ApplicationController
  def index
    audit! :list_tickets, nil
  end

  def create
    if @ticket.save
      audit! :create_ticket, @ticket, payload: ticket_params
    else
      render :new
    end
  end

  def update
    if @ticket.save
      audit! :update_ticket, @ticket, payload: ticket_params
    else
      render :edit
    end
  end
  ...
end

PaperTrail and Audited can do this automatically for you. However, in this code sample, you can also see the use of action-based auditing in the index method. This equates to an audit event capturing a view tickets operation.

AuditLog also provides a simple web interface to view audit information. While your operational requirements may dictate a more comprehensive reporting capability, this is a nice feature to have right out of the box.

Audit Logging with AppSignal

AppSignal provides a managed logging mechanism that offloads the storage of log data and provides a comprehensive set of search and filter capabilities. Each application has a default log source, or you can create additional sources as shown below.

The following code shows an example of creating an audit log event from a sample blog application. The logger constructor takes a group name as a parameter that can be used to identify event sources. This code example comes from an articles controller class where new articles are created.

logger = Appsignal::Logger.new("articles")
logger.info("Created new article #{@article.id}", { article_id: @article.id, date: DateTime.now })

AppSignal dashboards can be used to view log data, both at the aggregate level as well as detailed audit log information.

You can drill down into specific log events and view the structured data included with the event, as shown below.

Custom Audit Logging Implementation

You can also roll your own implementation if you have specific requirements that don’t fit one of these solutions. In most cases, you will need to implement additional reporting capabilities using the stored audit data.

For this, you may choose to:

Use another Ruby gem.
Push the data to a data warehouse or cloud service for analysis and long-term storage.

Choosing an Audit Logging Implementation

Here are a few key considerations to help you choose between audit solutions:

What types of events do you need to audit? Are they primarily database-driven events? If so, a solution like PaperTrail is a great choice. If you have higher-level business events, then you need support for custom audit events similar to the pattern implemented by AuditLog.
Do gems with higher activity levels give you more confidence? If so, PaperTrail and Audited are two great choices. AuditLog provides more flexibility, but it has less of an installed user base.
What volume of audit events do you need to capture? Will the storage of audit events in the same application database add significant overhead to your application? If so, consider using AppSignal for minimal impact to your application's performance.
Remember that you also need query and reporting capabilities. The out-of-the-box capability here varies widely across the different options. AppSignal provides a comprehensive search and filter capability over log data.

Wrapping Up

Audit logging is a fundamental capability for almost all business applications. We've seen that it is beneficial to establish a design pattern early on for the observability of a system as development progresses.

There are some great implementations available for Ruby and Rails developers: PaperTrail, Audited, AuditLog, and AppSignal. Evaluate your requirements using the criteria discussed above to determine the best choice for your needs.

Happy audit logging!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Use the rodauth-omniauth Gem in Ruby

Roy Tomeij — 2023-04-05T00:00:00+00:00

Setting up authentication for your Ruby app is important to ensure it is secure. In this post, we'll explore the rodauth-omniauth gem for Ruby to implement authentication in your app via third-party providers.

First, let's quickly define authentication before exploring the benefits of OmniAuth and setting up the rodauth-omniauth gem for a Rails app.

Let's dive straight in!

What Is Authentication?

Authentication is the act of verifying that someone or something is who or what they say they are to grant them access to requested resources. So if user A wants access to an application, they have to provide identification that is acceptable and confirmable by the system to be granted access.

Authentication can take many forms — for example:

Providing a password that matches an account's password as stored in an app's database.
Using facial recognition or fingerprints.
The use of a code sent to a registered channel, like an email or phone number, or an authenticator app.

There are various descriptions of authentication — we have single-factor authentication, two-factor authentication, and multi-factor authentication.

What Does OmniAuth Bring to the Table?

Authentication can be tiring for a user, especially when they have to constantly remember their various usernames and passwords for various applications. One of the solutions OAuth (Open Authentication) provides is to allow one service provider to authenticate a user using their identity with another service provider. This means that as a user, you only need to remember one username and password. Then every other application you need access to can delegate your authentication to your preferred OAuth provider.

Also, you do not need to give your password to the application that requires authentication. Instead, you are redirected to your provider, where you can be authenticated, and then redirected to the application you intend to access (thereby keeping your credentials safe and secure). Your provider, on the other hand, then shares the relevant information about you with the requesting application. Some common OAuth service providers are Google, Facebook, Twitter, and GitHub.

OmniAuth is a library that standardizes multi-provider authentication for web applications. It uses a strategy pattern for the different providers, and these strategies are generally released individually as Ruby gems. The rodauth-omniauth gem:

Offers login and registration via multiple external providers using OmniAuth, together with the persistence of external identities.

Source: rodauth-omniauth GitHub repo

Setting Up rodauth-omniauth for a Rails Application

Let's start by creating a new Rails project titled "rodauth_test_app".

$ rails new rodauth_test_app
$ cd rodauth_test_app

It is important to note that the rodauth-omniauth gem is an extension to the rodauth gem, and as such, we need to have already set up Rodauth to use it. For a Rails app, a quick way to do that is to run the following commands:

$ bundle add rodauth-rails
$ rails generate rodauth:install
$ rails db:migrate

To have access to the views rodauth provides, you can run either of the following commands:

$ rails g rodauth:views # to have access to all the views that rodauth provides
$ rails g rodauth:views login # to access only the login page which is what we need

The rails rodauth:routes command gives us a list of the routes handled by the RodauthApp, making it possible for us to get to the different pages we're interested in.

Among the tables created during our installation, the most important one to us is the Accounts table. This is the table where all user accounts are stored and will be the reference for the AccountIdentities table we will create.

Creating an Account Identities Table

A person can have a driver's license, international passport, and voter's card. All these means of identification point to the same person and can be used interchangeably.

Similarly, a user can have several means of identification (account identities), and this user may choose to log in at any time using any of these identities. That is why we'll create an AccountIdentities table to store all the identities of a specific user account, identified by a unique email address.

With a quick look in our db folder at the ..._create_rodauth.rb file, we find the schema for the accounts table as follows:

create_table :accounts do |t|
  t.integer :status, null: false, default: 1
  t.string :email, null: false
  t.index :email, unique: true, where: "status IN (1, 2)"
  t.string :password_hash
end

To create a table to store our account identities, we run the following command:

$ rails g migration CreateAccountIdentities

We find the generated file in the db/migrate folder. Let's go ahead and define its schema.

class CreateAccountIdentities < ActiveRecord::Migration[7.0]
  def change
    create_table :account_identities do |t|
      t.references :account, null: false, foreign_key: { on_delete: :cascade }
      t.string :provider, null: false
      t.string :uid, null: false
      t.index [:provider, :uid], unique: true
      # timestamps are not included -> to be explained later
    end
  end
end

Within this table, we ensure the following:

Every identity must belong to an account.
When an account is deleted, all associated identities also get deleted.
The provider must be declared (e.g., Google, Twitter, Facebook, GitHub).
A uid must be present.
The combination of the provider and the uid must be unique to ensure that all account identities are unique for each provider.

It is also very important to have a homepage — a root route — to redirect users to after they are successfully authenticated. To ensure that, create a home controller with an index method that serves as the homepage.

$ rails g controller home index

In our config/routes.rb file, we add the root route as follows:

root "home#index"

Setting the OAuth Provider

We'll use GitHub as the OAuth provider for this app, so install the GitHub OAuth gem and the rodauth-omniauth gem.

$ bundle add rodauth-omniauth omniauth-github

Having installed these gems, enable the omniauth feature and register your OAuth provider in the Rodauth configuration. The config file can be found at app/misc/rodauth_main.rb.

configure do
# ...
  enable :omniauth
  omniauth_provider :github, ENV["GITHUB_CLIENT_ID"], ENV["GITHUB_CLIENT_SECRET"], name: :github
  # name is only important here if it's different from the provider e.g. :google_oauth2 with name: :google
# ...
end

To get the client id and client secret for this app, we need to create the OAuth app and set the callback URL to {root_url}/auth/{provider}/callback. In our case, this translates to https://localhost:3000/auth/github/callback.

At this point, we're all set to start our server and visit our homepage.

$ rails s

Visiting /login, as confirmed from our rodauth routes, leads to the login page. It is on this page that we add our Login via GitHub button.

# app/views/rodauth/_login_form_footer.html.erb
# ...
  <li><%= button_to "Login via GitHub", rodauth.omniauth_request_path(:github), method: :post, data: { turbo: false } %></li>
# ...

Authentication and Redirection with Rodauth

On clicking the Login via GitHub button, we are redirected to the GitHub authentication page, assuming that the providers are properly configured. On this page, we are required to provide some details for authentication and after a successful authentication, we're redirected to the root URL of our application.

Between authentication and redirection to the homepage, Rodauth does the following:

Creates an account, along with an account identity, if no account with that email previously existed.
Assigns an identity to an account if an account with that email already exists.
Sets the status of a newly created account to "verified", because it assumes that the external provider verified the email address.
Returns an error response during the callback phase if an account associated with the external identity already exists and is unverified (e.g., it is created through normal registration), as only verified accounts can be logged into.

Accessing Account Identities

With rodauth-omniauth version 0.3.3 and above, you can access the identities related to an account without any further setup, as shown below:

> Account.first
=> # <Account:0x00007f236b455a70 id: 1, status: "verified", email: "biodun9@gmail.com", password_hash: nil>

> Account.first.identities
=> [#<Account::Identity:0x00007fe90a0612d8
  id: 1,
  account_id: 2,
  provider: "github",
  uid: "52589264"]

This is possible because during the installation of rodauth-rails, rodauth-model is automatically configured within the account model, defining an identities one-to-many association on the account model.

class Account < ApplicationRecord
  include Rodauth::Rails.model # the rodauth-model is included here
  enum :status, unverified: 1, verified: 2, closed: 3
end

For versions below 0.3.3, this isn't the case due to a bug. So to access identities, you can either choose to upgrade your version of rodauth-omniauth, or define the relationship manually like this:

# ... app/models/account_identity.rb
class AccountIdentity < ApplicationRecord
  belongs_to :account
end

# ... app/models/account.rb
class Account < ApplicationRecord
  has_many :identities, class_name: "AccountIdentity"
end
# ...

Adding Extra Columns

During the identities table migration, the timestamps column was missing. This is because that column is not originally included in the implementation provided by the gem.

However, we can manually add that column (as well as any other columns) and any logic required to configure the gem.

$ rails g migration add_timestamps_to_account_identities

class AddTimestampsToAccountIdentities < ActiveRecord::Migration[7.0]
  def change
    add_timestamps :account_identities, null: true
    # allowing this field to be nullable due to already existing records
  end
end

Within the configuration, we have to implement the logic to add these columns when creating an account identity.

# ... app/misc/rodauth_main.rb
configure do
# ...
  # omniauth_identity_insert_hash handles account identity creation
  omniauth_identity_insert_hash do
    super().merge(created_at: Time.now)
  end

  # omniauth_identity_update_hash handles account identity updates
  omniauth_identity_update_hash do
    super().merge(updated_at: Time.now)
  end
# ...
end

On deleting the previous account identity and creating a new one, we get this:

> AccountIdentity.first
=> #<AccountIdentity:0x00007f28533b83c8
 id: 1,
 account_id: 1,
 provider: "github",
 uid: "52589264",
 created_at: Sat, 18 Feb 2023 09:04:10.069443000 UTC +00:00,
 updated_at: Sat, 18 Feb 2023 09:04:10.069430000 UTC +00:00>

If it becomes necessary to access the auth hash or any other additional information, this gem provides us with some helper methods, like omniauth_provider, omniauth_email, and many more. Hooks are also available to implement the logic required at certain phases like omniauth_before_callback_phase, omniauth_on_failure, etc. A comprehensive list of these helper methods and hooks can be found in the rodauth-omniauth documentation.

A Note on Compatibility and Versioning

When installing the omniauth-google_oauth2 gem, a versioning error is encountered.

Bundler could not find compatible versions for gem "omniauth":
  In Gemfile:
    omniauth-google_oauth2 was resolved to 0.1.5, which depends on
      omniauth (~> 1.0)

    rodauth-omniauth was resolved to 0.3.1, which depends on
      omniauth (~> 2.0)

This is because the omniauth version dependencies for both gems are not compatible. omniauth-google_oauth2 depends on omniauth ~> 1.0, while the rodauth-omniauth gem depends on omniauth ~> 2.0. A more compatible omniauth-google gem would be omniauth-google-oauth2.

Any OAuth provider gem used alongside rodauth-omniauth must be compatible with OmniAuth 2.0.

rodauth-omniauth: Now and Future Plans

In summary, the rodauth-omniauth gem, though powerful, cannot be employed independently as it is a Rodauth extension. However, when employed with Rodauth, it saves you time that would have been spent manually implementing OmniAuth logic.

The gem handles:

Routing a request to a third-party provider.
The response (by creating the required accounts/identities or validating existing ones).
Outputting errors when necessary.

If the login is successful, it also redirects the request to the root URL.

In the future, the plan is for the rodauth-omniauth gem to offer services like allowing users to connect or remove external identities when signed in. So you will have the option to connect your Google account to an application (even if you're signed in with GitHub) or disconnect an already existing identity from an application while signed in.

Wrapping Up

In this post, we looked briefly at the usefulness of OmniAuth before setting up rodauth-omniauth for a Rails application. We also defined authentication and explained why it is so important for your Ruby app.

I hope you've found this introduction to the rodauth-omniauth gem useful.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Diving into Custom Exceptions in Ruby

Roy Tomeij — 2023-03-29T00:00:00+00:00

Customizing exceptions is usually not a common concern during software development. But if you catch an error in an observability tool and can't correctly and quickly identify the problem, you may need more information and details about an exception.

This article will show you how to customize exceptions in Ruby and mitigate potential future problems due to a lack of error information.

Let's dive straight in!

A Quick Side-Note

This post is a natural progression to Debugging in Ruby with AppSignal, so we recommend reading that first for an overview of debugging and an introduction to custom exceptions.

How to Rescue an Exception with Ruby

Ruby has a class called Exception, from which error-handling classes inherit. In this section, we'll better understand Ruby's structural exception flow before we create our custom exception.

Exception is the main exception class in Ruby, and rescue Exception will catch all exceptions inherited from Exception. It isn't best practice to use this in our app, as it will catch all exceptions used internally by Ruby.

The easiest way to rescue an exception in Ruby is to create a begin ... rescue block of code like the example below:

begin
  call_some_method()
rescue Exception => e
  ...
end

Since Exception will catch all exceptions, we should try to use something more specific to filter and get only the error we want. One example is the StandardError class, a standard rescuer used to catch exceptions related to application code errors by ignoring Ruby's internal exceptions.

To catch a StandardException, you can use the example below:

begin
  call_some_method()
rescue StandardException => e
  ...
end

Or, if no exception class is stated, Ruby will return the exceptions handled by the StandardException class.

begin
  call_some_method()
rescue => e
  ...
end

To create a custom exception with Ruby, you'll probably create a class that inherits from StandardError. We'll cover that in the next few sections.

To learn more about exceptions and find the best one to inherit, check Ruby's exceptions list.

Adding Additional Information to the Ruby Exception

Creating a new exception without sending relevant information will not effectively help in debugging — identifying and fixing — the problem. You need to add the error details. Here, we will see how to pass information from objects to the custom exception.

Let's create a new custom exception just to receive the data we want to log. As mentioned in the previous section, the custom exception needs to inherit from StandardError.

You can use a simple Rails project to test this implementation. Create a new folder called exceptions inside the app folder and a class called custom_exception.rb in your Rails project.

# app/exceptions/custom_exception.rb

class CustomException < StandardError
  def initialize(code, name, msg, details)
    @code = code
    @name = name
    @msg = msg
    @details = details
  end
end

In this exception class, we can define all the information that needs to be shown in the logs and we use four attributes for our data:

a code to enumerate the exception
a name
an informational message
internal information details about the exception already provided by Ruby

And now we create a begin...rescue block to raise the CustomException and pass the parameters to the exception attributes.

# app/models/employee.rb

class Employee
  def calculate_fees
    begin
      nil.object # Any code that throws an exception.
    rescue => exception
      raise CustomException.new(
        333,
        "My customized exception",
        "A new exception occurred in the application",
        exception)
    end
  end
end

This looks useful, so let's continue the implementation and use this data to customize the exception log.

Extending an Exception in Ruby

Extending an exception allows us to elevate customization to a more specific and reusable level. Just as Ruby has several exceptions for different types of errors, we can also follow this line of programming in our systems.

Next, let's understand how to override our exception and use it in a context that demands an extension.

Finding information in logs is a painful activity. Developers often blame themselves for not including more information about errors or how to search and filter. If you are not using any monitoring tools that provide this, including meaningful data could save you in the foreseeable future.

Pro-tip: Check out the section on 'Debugging Exceptions in Ruby with AppSignal' in our post Debugging in Ruby with AppSignal for information about how to set up and track custom exceptions in AppSignal.

In the example below, we will customize the display of exceptions in the log. Initially, you might think that it's just a light and pretty way to show data. However, the keywords used in the data are key to finding an error and helping whoever will be monitoring the application.

Add the colorize gem to your project by including the code below in your Gemfile.

# Gemfile

gem 'colorize'

Run the bundle to install the colorize gem and launch your Rails application.

$ bundle install
$ rails s

Now, in your exception class, import the gem. Print the attributes in a formatted way, so it's easy to see and quickly find an important error.

# app/exceptions/custom_exception.rb

require 'colorize'

class CustomException < StandardError
  def initialize(code, name, msg, details)
    ...
    log_exception
  end

  def log_exception
    print_separator

    STDERR.puts "🚨🚨🚨 #{self.class} 🚨🚨🚨".colorize(:red)

    self.instance_variables.each do |attr|
      # All attributes will be printed.
      STDERR.puts "#{attr}: #{self.instance_variable_get(attr)}".colorize(:red)
    end

    print_separator
  end

  def print_separator
    85.times { print "-".colorize(color: :black, background: :red) }
    STDERR.puts "\n"
  end

  def to_s
    @name.colorize(:red)
  end
end

Check out the result of this visual customization and test some options to create your format and style.

Wrapping Up and Next Steps

In this post, we covered how to customize and highlight a Ruby exception in your logs. Many other customizations are possible, but what's most important is that you handle the exceptions relevant to your application. And remember, don't apply the same formatting style to all exceptions, as Rails already does.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Authorization Gems in Ruby: Pundit and CanCanCan

Roy Tomeij — 2023-03-22T00:00:00+00:00

Today, many web applications will feature pages that are publicly available — like a homepage — and more secure ones where a user has to log in to get access. The process of user registration, logging in, and tracking user session state is called "authentication".

At the same time, when dealing with logged-in users, it's necessary to separate actions and resources that are available to them depending on their user roles. For example, "admins" generally have more access than normal users. This process of separating authenticated user access is termed "authorization".

In this post, we'll explore two of the most popular authorization libraries in Ruby so far: Pundit and CanCanCan.

Let's dive in!

Setup and Prerequisites

In this article, we'll use a simple Rails 7 app featuring users and posts. Users will be assigned an "editor" or "writer" role. Such a scenario is perfect for showcasing how authorization works.

Check out the code repos for our example app with:

Even though this article focuses on authorization, the companion subject of authentication cannot be ignored.

We won't get into the details of setting up authentication as that's outside the scope of this post. You can follow the installation instructions in the Devise gem documentation (we'll pair Devise with our authorization gems).

One more thing — whether you deal with Pundit or CanCanCan, the actual work of defining your app's user roles does not happen automatically when you install either of the authorization gems. You will need to set it up manually.

Let's do that.

Defining User Roles

Let's assume you've already installed the Devise gem and set up a user model. The next step is to decide what roles the app's users will have. In our case, we'll set out the following roles:

Writer - This user role will be able to create, edit, update, and delete their own posts. At the same time, a writer can also view other writers' posts.
Editor - A user with the editor role can edit, update, view, and delete any user's posts, but they cannot create their own posts.

Add a column to store a user's role (using a migration to modify the user's table):

bundle exec rails generate migration add_column_role_to_users role:integer

Then run the migration:

bundle exec rails db:migrate

And then modify the user model to include the roles we've just defined:

# app/models/user.rb

class User < ApplicationRecord
  # User role
  enum role: %i[writer editor]

  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

end

While we are on this, let's go ahead and set up a Post model with the attributes of title and body, as well as a reference to the user who writes the post:

bundle exec rails g scaffold Post title body:text user:references

We add the foreign key user_id into the Post model to associate every post that's created with a particular user. Since we have already set up user authentication using Devise, we can simply modify the create method of the Posts controller to automatically set the user_id to the currently logged-in user:

# app/controllers/posts_controller.rb
...

def create
    @post = current_user.posts.new(post_params) # automatically assign a post to the current user on creation

    respond_to do |format|
        if @post.save
        format.html { redirect_to post_url(@post), notice: 'Post was successfully created.' }
        format.json { render :show, status: :created, location: @post }
        else
        format.html { render :new, status: :unprocessable_entity }
        format.json { render json: @post.errors, status: :unprocessable_entity }
        end
    end
end
...

We can also modify the User model to make sure it's associated with the Post model:

# app/models/user.rb

class User < ApplicationRecord
  has_many :posts

  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  enum role: %i[writer editor]
end

Finally, let's seed the database with some users, each with a different role:

# db/seeds.rb

User.create(email: 'writer1@example.com', password: 'example', password_confirmation: 'example', role: 0) # this creates a user with the writer role
User.create(email: 'writer2@example.com', password: 'example', password_confirmation: 'example', role: 0) # second writer
User.create(email: 'editor@example.com', password: 'example', password_confirmation: 'example', role: 1) # this creates a user with the editor role

Then seed the database:

bundle exec rails db:seed

So far, our app now has:

Authentication set up using Devise.
Two defined user roles of "writer" and "editor".
A Post model.

With that, we now have everything we need to work properly with Pundit.

Authorization in Your Ruby App with Pundit

Pundit is an authorization library built around object-oriented architecture and plain Ruby classes. It gives you tools to build a solid authorization layer that can scale with your app.

Installing Pundit in Your Ruby App

Add the gem to your app's Gemfile:

# Gemfile

gem 'pundit'

Then in the terminal, run the command:

bundle install

Alternatively, run the following command:

bundle add pundit

Since authorization mostly deals with granting or denying access to controller resources, the next step is to add Pundit's authorization module to the application controller:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  include Pundit::Authorization
end

And finally, generate a base policy class all other policies will inherit from:

bundle exec rails g pundit:install

Which gives you the following base policy class:

# app/policies/application_policy.rb

class ApplicationPolicy
  attr_reader :user, :record

  def initialize(user, record)
    @user = user
    @record = record
  end

  def index?
    false
  end

  def show?
    false
  end

  def create?
    false
  end

  def new?
    create?
  end

  def update?
    false
  end

  def edit?
    update?
  end

  def destroy?
    false
  end

  class Scope
    def initialize(user, scope)
      @user = user
      @scope = scope
    end

    def resolve
      raise NotImplementedError, "You must define #resolve in #{self.class}"
    end

    private

    attr_reader :user, :scope
  end
end

With that, Pundit is now properly set up and ready to go. Additionally, we have authentication and user roles set up.

Next, let's use policies to implement rules that define how each user role will access the Post resource.

Configuring Pundit Policies

In Pundit lingo, a "policy" is a plain Ruby class where you define all the rules for how a user role interacts with different resources.

These policies come with some notable features:

Each policy is named after an existing model, suffixed with the word "Policy". For example, a policy defining how the Post model is accessed is called PostPolicy.
An attr_reader: this takes two arguments - the first is a user, specifically, the currently logged-in user — current_user — and the second argument is the model that you'd like to define authorization rules for, in our case, post.
Query methods that will map to the controller methods of the resource that has authorization rules set up. For organizational purposes, it's best to have all policies under the app/policies folder.

Since we know what access rules we need to define for the different user roles in our app, let's start with the writer role.

Defining a Pundit Policy for a Role

Let's use the writer role to see how this can be done. To begin with, we can outline the writer role's access to the Post resource as follows:

Create their own posts
Edit and update their own posts
View (or read) their own posts as well as other user's posts
Delete their own posts

With that in mind, go ahead and generate a new policy to control how posts are accessed:

bundle exec rails g pundit:policy post

This gives us the following generic policy class that inherits from the base policy we generated earlier:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
  class Scope < Scope
    # NOTE: Be explicit about which records you allow access to!
    # def resolve
    #   scope.all
    # end
  end
end

Let's now add access rules for the writer role accordingly (these also override any rules that are inherited from the base policy class):

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
  ...
  def create?
    @user.writer? # a writer is able to create a post
  end

  def edit?
    @user.writer? # a writer is able to edit a post
  end

  def update?
    @user.writer? # a writer can update a post
  end

  def delete?
    @user.writer? # a writer can delete a post
  end
end

Here, we define what a writer can do when creating, editing, updating, and deleting posts which are corresponding actions on the posts' controller. If you have noticed, these rules apply to posts in general and not necessarily to a writer's own posts (we'll get to that in the section on scopes).

For now, let's see how we can use this policy.

Using a Policy in Pundit

To use a policy, call Pundit's authorize method on the controller's method where you want to check access rules. You instantiate the relevant policy class and, more specifically, the action that should be called based on where the authorize method has been called.

For example, let's call authorize on the post controller's create method:

# app/controllers/post_controller.rb
...

def create
    @post = current_user.posts.new(post_params)
    authorize @post

    respond_to do |format|
        if @post.save
        format.html { redirect_to post_url(@post), notice: 'Post was successfully created.' }
        format.json { render :show, status: :created, location: @post }
        else
        format.html { render :new, status: :unprocessable_entity }
        format.json { render json: @post.errors, status: :unprocessable_entity }
        end
    end
end
...

To test this, log in as an editor and try to create a post. Doing this will result in the following error:

Though this does what we want, showing such an error page is not good for the user experience. In the next section, you will learn how to rescue the NotAuthorizedError and serve up something more user-friendly.

Rescuing From Pundit's `NotAuthorizedError`

First, we'll need to edit ApplicationController as follows:

# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
  include Pundit::Authorization

  rescue_from Pundit::NotAuthorizedError, with: :user_not_authorized

  private

  def user_not_authorized
    flash[:alert] = "You are not authorized to perform this action."
    redirect_back(fallback_location: root_path)
  end
end

Here, we are basically telling Pundit to use the user_not_authorized method whenever the NotAuthorizedError is raised. It will simply redirect the unauthorized user to a specific page and also provide a relevant flash message explaining what has happened:

We now have a simple authorization system capable of handling a very generalized permissions case.

But what if we want more fine-grained permissions? For this, we need to utilize Pundit's scopes.

Pundit's Scopes

Pundit scopes are similar to ActiveRecord scopes. In the latter, you can use scopes to fetch records according to specific criteria. However, with Pundit's scopes, you manage access to specific resources according to certain rules you set.

Let's say we want editors to be able to view and edit posts that are in "draft" status, and at the same time, allow writers to create, view, edit, update and delete posts that only belong to them.

We can start by editing the post policy to look like this:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
  class Scope < Scope
    def resolve
      if user.editor?
        # an editor can only access posts in "draft" status
        scope.where(published: false)
      else
        # can access a post if they are the author
        scope.where(user: user)
      end
    end
  end
  def show?
    @user.writer? || @user.editor?
  end
  def create?
    @user.writer?
  end
  def edit?
    @user.writer? || @user.editor?
  end
  def update?
    @user.writer? || @user.editor?
  end
  def delete?
    @user.writer?
  end
end

Then we'll authorize access to the resource in the posts controller, like so:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  ...
  def index
    @posts = policy_scope(Post)
  end
  # GET /posts/1 or /posts/1.json
  def show
    @post = policy_scope(Post).find(params[:id])
  end
  ...
end

Of course, scoping with Pundit can go way deeper than we've shown, but we'll leave it at that in this post. If you want, check out the Pundit documentation to see how you can use scopes in a more advanced way.

For now, let's go through how you can use the library with Rails' strong parameters.

Using Pundit with Rails' Strong Parameters

By combining Pundit's authorization rules with Rails' strong parameters, you can achieve lockdown access to a resource's attributes. Let's say you want editors to be the only ones with access to an excerpt field of the Post model. How would you go about it?

First, add an aptly-named block to the relevant policy:

# app/policies/post_policy.rb

class PostPolicy < ApplicationPolicy
...
  def permitted_attributes
    if user.editor?
      [:title, :body, :excerpt]
    else
      [:title, :body]
    end
  end
end

Then, modify the permitted params block in the controller:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  ...

  private

  def post_params
    params.require(:post).permit(policy(@post).permitted_attributes)
  end
end

With that, you have made the excerpt attribute available to editors only.

We'll now shift gears to look at the other authorization library, CanCanCan.

Introducing CanCanCan for Your Ruby App

CanCanCan is an authorization library that uses an "ability" class to define who has access to what in a Rails app. Actual access control is achieved using an authorization module and various view helpers.

Installing CanCanCan

Installation is as easy as running the command below:

bundle add cancancan

Just like Pundit, with CanCanCan, you can define all access rules within a plain Ruby class object called an "ability" class. Let's do that next with the following generator command:

bundle exec rails g cancan:ability

Which generates the class object below:

# app/models/abilty.rb

class Ability
  include CanCan::Ability

  def initialize(user)
  end
end

Let's learn more about how the ability class can define access rules for our example Rails app next.

Defining and Checking CanCanCan Abilities

We'll use the same user roles as in the Pundit example: writers and editors. A writer can create, edit, update, destroy their own posts, and view other writers' posts; an editor can do everything except create a post of their own.

To use CanCanCan, first define what each user or role can access in the ability class, following this format:

# app/models/ability.rb

can actions, subjects, conditions

As an example:

# app/models/abilty.rb

class Ability
  include CanCan::Ability

  def initialize(user)
    can :update, Post, user: user # With CanCanCan, the update action covers both the edit and update actions
  end
end

Then, in the controller, check if an access rule exists for a particular action — using our example, the edit action:

# app/controllers/posts_controller.rb

...
def edit
    authorize! :edit, @post
end
...

With this in place, if we visit the edit post view as another writer, we get the error below:

And just like we did with Pundit, let's rescue this error and show the user a better error page.

Handling CanCanCan’s “Access Denied” Errors

Whenever a resource is not authorized, CanCanCan will raise a CanCan::AccessDenied error. The easiest way to catch this exception is by modifying the ApplicationController as follows:

# app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  rescue_from CanCan::AccessDenied do |exception|
    respond_to do |format|
      format.json { head :forbidden }
      format.html { redirect_to root_path, alert: exception.message }
    end
  end
end

Doing this makes for a good user experience. In the screenshot below, the unauthorized user is redirected to the home page and shown a relevant flash message:

You can even customize the error message shown to the user:

#config/locales/en.yml
en:
  unauthorized:
    update:
      all: "You're not authorized to %{action} %{subject}."
      writer: "You're not authorized to %{action} other writer's posts."

If your app serves XML as a response, or you just want to dive deeper into handling the CanCanCan::AccessDenied exception, check out CanCanCan's documentation. For now, let's see how we can combine CanCanCan's various abilities to create a more robust authorization layer.

Combining Multiple CanCanCan Abilities

You can define multiple access rules for a resource in the ability class. Taking the writer and editor roles, for example, we can do this:

# app/models/ability.rb

class Ability
  include CanCan::Ability

  def initialize(user)
    can :update, Post, user: user # only a post's author/owner can update or edit a post
    can :read, Post # any user can read a post or list of posts (access both show and index actions)
    can :destroy, Post, user: user # only a post's author/owner can delete it

    return unless user.editor?
    cannot :create, Post # an editor role cannot create a post
    can :update, Post # an editor can update any post
  end
end

The question is, why would you want to do this?

With CanCanCan, you can define all access rules in one ability file. This has both advantages and disadvantages.

Having all your rules in one place is very convenient for handling access rules since all rules are defined in one place.

However, if your app deals with many user roles or you have several resources that need authorization, the ability class can easily become too big and complex to handle. One way to handle this is by reorganizing the ability class to use method definitions, like so:

# app/models/ability.rb

class Ability
  include CanCan::Ability

  def initialize(user)
    anyone_abilities

    if user.writer?
      writer_abilities
    elsif user.editor?
      editor_abilities
    end

  end

  private

  def anyone_abilities
    can :read, Post
  end

  def writer_abilities
    can :update, Post, user: Current.user # we define Current.user in the application controller so that the ability class (which is a model) is able to pick up the currently logged in user
  end

  def editor_abilities
    cannot :create, Post # an editor role cannot create a post
    can :update, Post # an editor can update any post
  end

end

There's a lot more to CanCanCan than can be effectively covered in this article. Do check out the detailed CanCanCan documentation to learn more.

To wrap up, let's briefly touch on the features of each library and give reasons why you might choose one over the other.

Feature Comparison: Pundit vs. CanCanCan for Your Ruby App

File organization - With Pundit, you can easily organize your app's authorization across multiple policy files. But with CanCanCan, authorization rules will live in one ability file. Working with multiple ability files is still possible, but this is not the default implementation style.
Tests - Because permissions code will mostly reside within a single class compared to Pundit's multiple ability classes, writing tests for CanCanCan could be easier than for Pundit.
Helpers - Both libraries provide a number of view helpers to check for permissions in the view layer. CanCanCan gives you the can? method to use in your views, while Pundit gives you relatively similar functionality with the policy helper.
Devise integration - As you can see from the examples we have used in the article, both libraries integrate with Devise very well.

Wrapping Up

In this article, we looked at two of the most popular authorization gems in the Ruby and Rails ecosystem: Pundit and CanCanCan.

Both libraries offer a rich set of features for managing permissions in your Rails app. Because of this, it is nearly impossible to tell you which gem to choose — both can manage even the most complex permissions setups.

We encourage you to try out Pundit and CanCanCan in your app and see which one fits your needs best.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Generalized User-local Container for UI State in Kredis

Roy Tomeij — 2023-03-15T00:00:00+00:00

In our last post, we persisted and restored a collapsed/expanded UI state with Kredis. However, the great pain point with this is that we have to invent a lot of Kredis keys.

This time, we'll see how we can avoid this by developing a generalized solution for any DOM node whose attribute states we want to track on the server side.

Let's dive straight in!

The Challenge

Remember that we had to concoct a Kredis set key (open_department_ids) on the user model in the previous post. Now, following our example, imagine a user might be associated to the products they bought via has_many. Assume that each of these products has a details page which also has some ephemeral state attached.

All of a sudden, the complexity increases tremendously: we now have N kredis_sets to cater for, prefixed with the product ID. Or we could handle it on the Product model, storing the state for each user ID. Either way, it is going to get cumbersome, and we'll likely have to resort to metaprogramming to smooth it out a bit.

This, you'll agree with me, is not a delightful challenge. It would be very beneficial for the app's architecture if we could abstract that away.

And this is precisely what we are going to attempt. We'll use an accordion on the product details page to illustrate this case and also its resolution.

Preparation

To start, we'll add two more text columns, description and specs, to our Product model:

$ bin/rails g migration AddDescriptionAndSpecsToProducts description:text specs:text
$ bin/rails db:migrate

Again, in the Rails console, we'll provide some seed data:

Product.find_each do |product|
  product.update(description: Faker::Lorem.paragraph, specs: Faker::Markdown.ordered_list)
    description: Faker::Lorem.paragraph,
    specs: Faker::Markdown.ordered_list
  )
end

Finally, let's amend the _product partial to include those two new properties in an "accordion", again making use of the <details> element.

  <!-- app/views/products/_product.html.erb -->

  <div id="<%= dom_id product %>">
    <p>
      <strong>Name:</strong>
      <%= product.name %>
    </p>

    <p>
      <strong>Department:</strong>
      <%= product.department_id %>
    </p>

+   <p>
+     <details>
+       <summary>Description</summary>
+
+       <%= product.description %>
+     </details>
+   </p>
+
+   <p>
+     <details>
+       <summary>Specs</summary>
+
+       <%= simple_format product.specs %>
+     </details>
+   </p>
  </div>

In our home index view, let's link to the products:

  <!-- app/views/home/index.html.erb -->

  <!-- ... -->

        <ul>
          <% child_dep.products.each do |prod| %>
-           <li><%= prod.name %></li>
+           <li><%= link_to prod.name, prod %></li>
          <% end %>
        </ul>

  <!-- ... -->

Here's the detail of the view this produces:

Observing UI Changes with JavaScript

Now we have to rewrite our UIState Stimulus controllers to listen for arbitrary DOM changes. We do this by placing a MutationObserver on its element, which looks for attribute changes after it connects. In its callback (mutateState), the following happens:

All attributes on the element, except for the data-controller attribute itself, are collected.
Their respective values are stored on an attributes object with their name as the key.
This object is appended to a new FormData object, along with a unique key that identifies the exact instance of the UI state (we'll get to that in a moment).
This is then sent over to the server as before, in the form of a PATCH request.

  // app/javascript/controllers/ui_state_controller.js

  import { Controller } from "@hotwired/stimulus";
- import { patch } from "@rails/request.js";
+ import { get, patch } from "@rails/request.js";

  export default class extends Controller {
    static values = {
-     departmentId: String,
+     key: String,
    };

-   async toggle() {
-     const body = new FormData();
-     body.append("open", this.element.open);
-     body.append("department_id", this.departmentIdValue);
-
-     await patch("/ui_state/update", {
-       body,
-     });
-   }

+   async connect() {
+     this.mutationObserver = new MutationObserver(this.mutateState.bind(this));
+
+     this.mutationObserver.observe(this.element, { attributes: true });
+   }
+
+   disconnect() {
+     this.mutationObserver?.disconnect();
+   }
+
+   async mutateState() {
+     const body = new FormData();
+
+     const attributes = {};
+     this.element
+       .getAttributeNames()
+       .filter((name) => name !== "data-controller")
+       .map((name) => {
+         attributes[name] = this.element.getAttribute(name);
+       });
+
+     body.append("key", this.keyValue);
+     body.append("attributes", JSON.stringify(attributes));
+
+     await patch("/ui_state/update", {
+       body,
+     });
+   }
  }

Persisting UI State with Unique Kredis Keys

To generalize persisting UI state, we have to accomplish two things:

Generate unique Kredis keys regarding the user, resource (in our case, the product), and the location in the view, respectively.
Use these keys to persist the attributes object received from the client.

Using Rails Helpers

So first, we have to solve the problem of coming up with unique keys. If you look at the requirements carefully, they read a lot like the built-in Rails fragment cache helper.

In short, this helper will build a cache key based on the template it's called from, and any cacheable Ruby objects passed to it. To quote the documentation:

views/template/action:7a1156131a6928cb0026877f8b749ac9/projects/123
      ^template path   ^template tree digest             ^class   ^id

We don't need the cache helper though, because we are not actually storing a fragment, we just need a key. Thankfully, the method it calls internally to generate one, cache_fragment_name, is also exposed.

Let's check out how we can make use of this. If we add the following line to our products/_product.html.erb partial:

<%= cache_fragment_name([current_user, product]) %>

Something like this is rendered:

["products/_product:0e55db8bd12c9b268798d6550447b303",
  [#<User id: 1, email: "julian@example.com", ...>,
  #<Product id: 68, name: "Durable Wool Wallet", ...>]
]

Clearly, this nested array needs some massaging, but we are already very close. What's still missing is a reference to the specific line in the template, but we can obtain this from the call stack's first entry representing a template. Let's extract this to a helper and convert it to a string:

# app/helpers/application_helper.rb

module ApplicationHelper
  def ui_state_key(name)
    key = cache_fragment_name(name, skip_digest: true)
      .flatten
      .compact
      .map(&:cache_key)
      .join(":")

    key += ":#{caller.find { _1 =~ /html/ }}"

    key
  end
end

Note: We are skipping the view digesting as including the caller location makes it redundant.

Called as ui_state_key([current_user, product]), this yields a string of the following format:

"users/1:products/68:/usr/app/ui-state-kredis/app/views/products/_product.html.erb:23:in `_app_views_products__product_html_erb___2602651701187259549_284440'"

Now there's one final piece to solve: because we will include this key in our markup (to be picked up by the Stimulus controller above), we'll want to obfuscate it. Otherwise, it would be easy for a bad actor to exchange the user ID and/or the product global ID using the browser's developer tools. A simple digest will do in our case, because we do not intend to decrypt it back.

  # app/helpers/application_helper.rb

  module ApplicationHelper
    def ui_state_key(name)
      key = cache_fragment_name(name, skip_digest: true)
        .flatten
        .compact
        .map(&:cache_key)
        .join(":")

      key += ":#{caller.find { _1 =~ /html/ }}"

-     key
+     ActiveSupport::Digest.hexdigest(key)
    end
  end

That's it, now we can obtain a digest of our UI fragment that will be unique to the location in the template and its MTIME, as well as user and resource:

"d45028686c0171e1e6e8a8ab78aae835"

Attach Stimulus Controllers to Your Rails App

Equipped with this, we can now attach the Stimulus controllers along with the respective keys to our <details> elements:

  <!-- app/views/products/_product.html.erb -->

  <div id="<%= dom_id product %>">
    <p>
      <strong>Name:</strong>
      <%= product.name %>
    </p>

    <p>
      <strong>Department:</strong>
      <%= product.department_id %>
    </p>

    <p>
      <details
+       data-controller="ui-state"
+       data-ui-state-key-value="<%= ui_state_key([current_user, product]) %>"
      >
        <summary>Description</summary>

        <%= product.description %>
      </details>
    </p>

    <p>
      <details
+       data-controller="ui-state"
+       data-ui-state-key-value="<%= ui_state_key([current_user, product]) %>"
      >
        <summary>Specs</summary>

        <%= simple_format product.specs %>
      </details>
    </p>
  </div>

We can now set out to generalize the UIStateController as well. For this, we obtain a new Kredis.json key using the transmitted key param. Whenever a UI state update occurs, all the attributes sent over as form data will be parsed and stored therein.

  # app/controllers/ui_state_controller.rb

  class UiStateController < ApplicationController
+   include ActionView::Helpers::SanitizeHelper

+   before_action :set_ui_state

    def update
-     if ui_state_params[:open] == "true"
-       current_user.open_department_ids << params[:department_id]
-     else
-       current_user.open_department_ids.remove(params[:department_id])
-     end
+     @ui_state.value = JSON.parse(params[:attributes]).deep_transform_values { sanitize(_1) }

      head :ok
    end

    private

    def ui_state_params
-     params.permit(:department_id, :open)
+     params.permit(:attributes, :key)
    end

+   def set_ui_state
+     @ui_state = Kredis.json params[:key]
+   end
  end

Rehydrating the UI

Since our UI state is now safely stored in an ephemeral Kredis key, the only piece of the puzzle that's missing is how to put the UI back together. This process is called rehydration. In this example, we'll use server-side rehydration.

For this, we will revisit the ui_state_key helper from above, and extend it with the rehydration logic.

   # app/helpers/application_helper.rb
   module ApplicationHelper
+    def remember_ui_state_for(name, &block)
+      included_html = capture(&block).to_s.strip
+      fragment = Nokogiri::HTML.fragment(included_html)
+
+      first_fragment_child = fragment.first_element_child
+
+      # rehydrate
+      ui_state = Kredis.json ui_state_key(name)
+
+      ui_state.value&.each do |attribute_name, value|
+        first_fragment_child[attribute_name] = sanitize(value, tags: [])
+      end
+
+      # add stimulus controller and create unique key
+      first_fragment_child["data-controller"] = "#{first_fragment_child["data-controller"]} ui-state".strip
+      first_fragment_child["data-ui-state-key-value"] = ui_state_key(name)
+
+      first_fragment_child.to_html.html_safe
+    end

     def ui_state_key(name)
       key = cache_fragment_name(name, skip_digest: true)
         .flatten
         .compact
         .map(&:cache_key)
         .join(":")

       key += ":#{caller.find { _1 =~ /html/ }}"

       key
     end
   end

The remember_ui_state_for helper takes a name argument that it will pass on to the ui_state_key helper from above. Before it does that, though, it will capture the HTML from the block passed to it and extract its first fragment child with Nokogiri.

Afterward, the actual rehydration logic starts:

the ui_state is read back from Kredis
in case such a key already exists, each attribute is put back on the HTML element obtained in the previous step. To prevent any XSS vulnerabilities, the attributes' values are also sanitized
finally, the Stimulus controller is attached to the element, along with the ui_state_key.

Any change to the attributes would now again be transmitted by the MutationObserver, and any page reload would invoke this helper again, rehydrating them upon the element.

Note: An additional security mechanism would be safelisting the allowed attributes, which I have excluded from this example for simplicity.

In our product partial, instead of wiring up the Stimulus controller manually, we must now use the new view helper:

  <!-- app/views/products/_product.html.erb -->

  <div id="<%= dom_id product %>">
    <p>
      <strong>Name:</strong>
      <%= product.name %>
    </p>

    <p>
      <strong>Department:</strong>
      <%= product.department_id %>
    </p>

    <p>
-     <details
-       data-controller="ui-state"
-       data-ui-state-key-value="<%= ui_state_key([current_user, product]) %>"
-     >
-       <summary>Description</summary>
-
-       <%= product.description %>
-     </details>
+     <%= remember_ui_state_for([current_user, product]) do %>
+       <details>
+         <summary>Description</summary>
+
+         <%= product.description %>
+       </details>
+     <% end %>
    </p>

    <p>
-     <details
-       data-controller="ui-state"
-       data-ui-state-key-value="<%= ui_state_key([current_user, product]) %>"
-     >
-       <summary>Specs</summary>
-
-       <%= simple_format product.specs %>
-     </details>
+     <%= remember_ui_state_for([current_user, product]) do %>
+       <details >
+         <summary>Specs</summary>
+
+         <%= simple_format product.specs %>
+       </details>
+     <% end %>
    </p>
  </div>

This is what the end result looks like:

It turns out that this approach is perfect for HTML elements or custom web components that reflect their state in one or more attributes (e.g. <details>, Shoelace, Lit, etc.).

Wrapping Up

In part one of this series, we described what Kredis can and cannot do, and developed an example use case for storing ephemeral UI state using a bespoke Redis key.

In this second and final part, we identified the drawbacks of this approach. We developed a generalized solution to apply to any DOM node whose state of attributes we want to track on the server side.

Finally, let me point out that these considerations inspired me to develop a library called solder, which I would be happy to receive feedback on!

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Making the Most of Your Logs in Rails

Roy Tomeij — 2023-03-01T00:00:00+00:00

Most people only realize the necessity of logs when they need them the most. But when your application breaks, user complaints start flooding in, and you have no clue how to fix it, it's too late to add some log messages that might have helped.

Good logs pay for themselves tenfold. They make it a breeze to diagnose those tricky bugs, and if you do logs right, they can alert you of issues even before your users notice. But what does it mean to 'do logging right'?

Logging is easy to start with but hard to master. In this post, we'll dive into how you can use your Rails application's logs to their full potential.

Logging in Rails

Let's talk about the basics first. When you start a new Rails application, logging is already set up for you. Rails will initialize a new instance of ActiveSupport::Logger that you can use anywhere within your application.

Rails.logger.debug('Hello logs!')

I, [2019-01-04 05:14:33 #123] INFO -- Main: Hello Logs!

The Rails logger writes to the standard output or log/<environment>.log and will automatically log incoming requests or executed queries in addition to any log messages you write explicitly.

There are numerous ways in which you can configure Rails logs, as outlined excellently in the documentation for Rails.

To make the most of our logs, we are especially interested in setting log levels as well as log formatting.

But before tackling that, let's talk about why we should write logs.

Good Logging, Bad Logging

The purpose of logs is to inform you about system events so you can react to them. For example, when an error happens, a log message should tell you about it in a way you can understand.

How well you understand a log message depends on how descriptive and contextual it is. A descriptive log message provides relevant information about what happened. A contextual log message includes information about a system's state when the system wrote it.

Let's consider a simple example to see why we need both. Here is some code that calls an external API and returns its response when a user performs a request.

def call_external_api(user_id, payload)
  Rails.logger.info('Method entered')
  client = Client.new(user_id)
  response = client.request(payload)
  if response.ok?
    Rails.logger.info('Response success')
    return response
  else
    Rails.logger.info('Response failure')
    return response
  end
  rescue ClientError => e
    logger.debug('An error occurred)
end

Now let's imagine a situation where a customer reports an issue, and you look into the logs for help. This is what you might see:

I, [2022-07-10T10:40:34.827604 #100038]  INFO -- : [42f2e074-d53f-41e4-adca-13f59c6383ec] Processing by UserController#action as HTML
I, [2022-07-10T10:40:34.828084 #100038]  INFO -- : [42f2e074-d53f-41e4-adca-13f59c6383ec] Method entered
I, [2022-07-10T10:40:34.933263 #100038]  INFO -- : [42f2e074-d53f-41e4-adca-13f59c6383ec] Response success
I, [2022-07-10T10:40:34.933803 #100038]  INFO -- : [42f2e074-d53f-41e4-adca-13f59c6383ec] Completed 200 OK in 106ms (Views: 0.2ms | Allocations: 2135)
I, [2022-07-10T10:40:35.959454 #100038]  INFO -- : [458a209f-6b0e-4d29-a9b0-449d6ff3d29a] Started GET "/action" for 127.0.0.1 at 2022-07-10 10:40:35 +0200
I, [2022-07-10T10:40:35.959975 #100038]  INFO -- : [458a209f-6b0e-4d29-a9b0-449d6ff3d29a] Processing by UserController#action as HTML
I, [2022-07-10T10:40:35.960081 #100038]  INFO -- : [458a209f-6b0e-4d29-a9b0-449d6ff3d29a] Method entered
I, [2022-07-10T10:40:36.043971 #100038]  INFO -- : [458a209f-6b0e-4d29-a9b0-449d6ff3d29a] Response failure
I, [2022-07-10T10:40:36.044316 #100038]  INFO -- : [458a209f-6b0e-4d29-a9b0-449d6ff3d29a] Completed 200 OK in 84ms (Views: 0.1ms | Allocations: 1297)
I, [2022-07-10T10:42:49.701879 #103892]  INFO -- : [e02c3a8f-c081-40aa-90fb-9d1474126403] Started GET "/action" for 127.0.0.1 at 2022-07-10 10:42:49 +0200
I, [2022-07-10T10:42:49.703400 #103892]  INFO -- : [e02c3a8f-c081-40aa-90fb-9d1474126403] Processing by UserController#action as HTML
I, [2022-07-10T10:42:49.703745 #103892]  INFO -- : [e02c3a8f-c081-40aa-90fb-9d1474126403] Method entered
I, [2022-07-10T10:42:49.775846 #103892]  INFO -- : [e02c3a8f-c081-40aa-90fb-9d1474126403] An error occurred

These log messages provide some information, but not nearly enough. We are no closer to finding out what's responsible for the error the customer has reported. These log messages lack descriptiveness and context. They are noise. How can we improve them?

Log Levels in Rails

The default Rails logger provides the log levels DEBUG, INFO, WARN, and ERROR. Using them, you can group log messages into different categories of relevance. This not only helps you filter log messages, but also provides some context.

It can be challenging to decide when to use which log level. Here are some rules of thumb:

DEBUG: Use this log level for detailed information about actions happening within the system. You may use debug statements when methods are entered or exited, or anywhere you think it may add value during debugging.
INFO: Whenever your system changes its state, or some relevant event occurs, reach for the INFO level. Examples of common info messages in the context of Rails applications are received requests, requests sent to external APIs, or jobs starting and finishing.
WARN: Use this log to signify that something unexpected has happened. It's not a problem yet (because your application can handle it), but if it keeps happening, it might warrant your attention.
ERROR: Use this log level when an error happens. Errors are invalid application states that you must resolve as soon as possible.

You can change the kinds of log messages your application writes by modifying the respective environment file. Normally, Rails will discard DEBUG messages in production, but you can change that.

# config/production.rb
Rails.logger.level = :debug

Let's apply the log-level suggestions above to our code sample.

def call_external_api(user_id, payload)
  Rails.logger.debug('Method entered')
  client = Client.new(user_id)
  response = client.request(payload)
  if response.ok?
    Rails.logger.info('Response success')
    return response
  else
    Rails.logger.warn('Response failure')
    return response
  end
  rescue ClientError => e
    logger.error('An error occurred')
end

When debugging, we can now identify root causes for issues by looking into WARN and ERROR messages. Sadly, our log messages haven't gotten any more descriptive.

Descriptive Log Messages

Descriptive log messages leave no room for interpretation. They provide the details necessary to give the reader instant knowledge about what happened.

When you read messages such as 'An error occurred', you are left wondering what the error is. You want to avoid such confusion when writing your log messages. The most important thing when writing descriptive log messages is to put yourself in the shoes of the log reader. Will they get all the information they need when reading your logs?

Let's change the message 'An error occurred' to the error itself, including its message.

logger.error { "#{e}: #{e.message}" }

That's an improvement. Let's check the other log messages in our example. 'Method entered' doesn't tell us which method was called, 'Response success' leaves out the actual response, and 'Response failure' provides no information about the nature of the failure. Let's change that.

def call_external_api(user_id, payload)
  Rails.logger.debug('Calling external api')
  client = Client.new(user_id)
  response = client.request(payload)
  if response.ok?
    Rails.logger.info { "Request success, received #{response.body}" }
    return response
  else
    Rails.logger.warn { "Request returned #{response.code}, #{response.body}" }
    return response
  end
  rescue ClientError => e
    logger.error { "#{e}: #{e.message}" }
end

Note: You might have noticed we use the block syntax when performing string interpolation with our logs. Using it avoids unnecessary computation when the application's log level is higher than the log messages' level. For example, log.debug("Some #{concatenation}") will always perform the string concatenation, but log.debug { "Some #{concatenation}" } will only do so when the log level is set to debug.

I, [2022-07-30T11:19:13.742108 #122944]  INFO -- : [29d1ced1-3d05-4fbf-8b75-2a50e87ad749] Started GET "/users" for 127.0.0.1 at 2022-07-30 11:19:13 +0200
I, [2022-07-30T11:19:13.743058 #122944]  INFO -- : [29d1ced1-3d05-4fbf-8b75-2a50e87ad749] Processing by UsersController#index as HTML
D, [2022-07-30T11:19:13.743248 #122944] DEBUG -- : [29d1ced1-3d05-4fbf-8b75-2a50e87ad749] Calling external api, user 1, payload {:search=>"name"}
I, [2022-07-30T11:19:13.844355 #122944]  INFO -- : [29d1ced1-3d05-4fbf-8b75-2a50e87ad749] Request completed. Received {"userId"=>1, "id"=>1, "title"=>"title", "completed"=>false}
I, [2022-07-30T11:19:13.844836 #122944]  INFO -- : [29d1ced1-3d05-4fbf-8b75-2a50e87ad749] Completed 200 OK in 102ms (Views: 0.2ms | Allocations: 2195)
I, [2022-07-30T11:20:09.501016 #123422]  INFO -- : [598eb218-d3c9-4e92-9236-13281cc660af] Started GET "/users" for 127.0.0.1 at 2022-07-30 11:20:09 +0200
I, [2022-07-30T11:20:09.502518 #123422]  INFO -- : [598eb218-d3c9-4e92-9236-13281cc660af] Processing by UsersController#index as HTML
D, [2022-07-30T11:20:09.502848 #123422] DEBUG -- : [598eb218-d3c9-4e92-9236-13281cc660af] Calling external api, user 1, payload {:search=>""}
W, [2022-07-30T11:20:09.574884 #123422]  WARN -- : [598eb218-d3c9-4e92-9236-13281cc660af] Request failed with 400: Empty search
I, [2022-07-30T11:20:09.575415 #123422]  INFO -- : [598eb218-d3c9-4e92-9236-13281cc660af] Completed 200 OK in 73ms (Views: 0.2ms | Allocations: 2106)
I, [2022-07-30T11:20:41.229494 #125518]  INFO -- : [77b70a25-268c-43ea-a3da-8788086165f2] Started GET "/users" for 127.0.0.1 at 2022-07-30 11:20:41 +0200
I, [2022-07-30T11:20:41.230735 #125518]  INFO -- : [77b70a25-268c-43ea-a3da-8788086165f2] Processing by UsersController#index as HTML
D, [2022-07-30T11:20:41.231039 #125518] DEBUG -- : [77b70a25-268c-43ea-a3da-8788086165f2] Calling external api, user 1, payload {:search=>"long name"}
E, [2022-07-30T11:20:41.301237 #125518] ERROR -- : [77b70a25-268c-43ea-a3da-8788086165f2] Request timeout exceeded
I, [2022-07-30T11:20:41.301878 #125518]  INFO -- : [77b70a25-268c-43ea-a3da-8788086165f2] Completed 200 OK in 71ms (Views: 0.3ms | Allocations: 2085)

Our logs read much better now. There is little doubt about what each log message signifies.

Further Context for Log Messages

Our new log messages provide a clear picture of what happened. Because this is a simple example, we also have a good understanding of why certain things happened. In reality, it's usually not that easy.

Providing additional information about the context in which the system wrote each message can significantly help with debugging.

For example, when logging requests or responses, it may be helpful to know who performed the request. Attaching a stack trace for errors might be useful so we can gather additional information about why the error occurred.

def call_external_api(user_id, payload)
  Rails.logger.debug { "Calling external API. user_id: #{user_id}, payload: #{payload}" }
  client = Client.new(user_id)
  response = client.request(payload)
  if response.ok?
    Rails.logger.info { "Request completed. Received #{response.parsed_response}. user_id: #{user_id}, payload: #{payload}" }
    return response
  else
    Rails.logger.warn { "Request failed with #{response.code}: #{response.parsed_response}. user_id: #{user_id}, payload: #{payload}" }
    return response
  end
  rescue ClientError => e
  logger.error("#{e.message} (#{e.class}), #{ e.backtrace.drop(1).map { |s| "\t#{s}" } }")
end

Note: You have likely noticed that creating log messages with contextual information can be cumbersome when you use Rails' default logger. Luckily, custom loggers such as Ougai or MrLogaLoga make this a lot easier.

Structured Logging

Once you've made your logs readable for humans, you are ready to take things to the next level. You now need to make your logs readable by machines so you can use them for monitoring and data analysis.

My go-to format is JSON, but depending on which tools you use, you may prefer other log formats such as Logstash. You can change the format of your log messages by creating a custom log formatter.

# lib/json_log_formatter.rb
class JsonLogFormatter < ::Logger::Formatter
  def call(severity, time, progname, msg)
    json = { time: time, progname: progname, severity: severity, message: msg2str(msg) }
      .compact_blank
      .to_json
    "#{json}\n"
  end
end

# application.rb
config.log_formatter = JsonLogFormatter.new

{"time":"2022-08-07T18:57:20.261+02:00","severity":"INFO","message":"Request completed" }

Rather than writing custom formatters yourself, you can use one of the many gems that provide custom log formatters. I recommend Lograge, which not only offers out-of-the-box formatting but also cleans up Rails' somewhat verbose request formatting so it's much easier to read.

Rails Logging with AppSignal

Now that your logs are nice and readable, it would be nice to make them more accessible. After all, you don't want to SSH into some machine and tail or grep the logs there, right?

Luckily, AppSignal has recently launched a logging feature in beta! This allows you to inspect and analyze Rails logs directly in the AppSignal web interface.

You can access our logging beta through the 'Logging' tab in the left-hand side menu:

If you follow the steps in our Ruby logging docs, you can configure the Rails Logger to use the AppSignal::Logger class by placing the code below in the config/environment.rb file before initialization:

# Use AppSignal's logger
Rails.logger = Appsignal::Logger.new("rails")

# Initialize the Rails application.
Rails.application.initialize!

You can also ingest structured logs in JSON format or with Lograge.

Once you've set everything up, you should see your Rails logs show up on AppSignal!

Logging and Error Reporting

You're reading this on the AppSignal blog, so you might wonder: why should I care about logging if I'm already using a nice error reporting platform?

It's a good question. Error tracking tools provide detailed information about application errors. They capture the full stack trace of any application error and provide lots of contextual information out of the box.

However, while most tools allow you to capture events other than errors, sending these tools an arbitrary amount of messages is generally infeasible. You won't be able to replace the detailed, historical information that well-written logs provide.

In reality, well-written logs augment and support error tracking tools. You should likely use both.

Wrap Up

In this post, we looked into how you can get the most out of your logs. We saw that getting started with logging in Rails is easy, but writing useful logs can be challenging.

Logs that lack descriptiveness or context will end up just filling up your disk while providing little value. However, logs that use the correct log levels and provide readers with the information they need are a great asset.

Once you've mastered writing great log messages, you can take things further and format them in a way that allows for log analysis and easy filtering.

We also explored how you can use AppSignal's new logging feature to access your logs easily. Finally, we touched on how logs should augment error tracking tools rather than replace them.

Happy logging!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Storing Ephemeral UI State with Kredis for Rails

Roy Tomeij — 2023-02-22T00:00:00+00:00

Kredis (Keyed Redis) is a recent addition to the Rails developer's toolkit. It strives to simplify storing and accessing structured data on Redis.

In this first part of a two-part series, we'll start by going into how Kredis works. We'll then run through an example use case for storing ephemeral UI state using a bespoke Redis key.

Let's get started!

An Introduction to Kredis for Rails

Kredis is a Railtie that provides convenient wrappers to streamline its use in three ways:

Ruby-esque API: For example, collection types like Kredis.list or Kredis.set emulate native Ruby types (and their respective API) as much as possible.
Typings: Especially handy for collections, Kredis can handle type casting the elements from/to standard data types (e.g., datetime, json).
ActiveRecord DSL: Probably the library's biggest asset, it allows you to easily connect any Redis data structure with a specific model instance.

Here's an example from the README:

class Person < ApplicationRecord
  kredis_list :names
  kredis_unique_list :skills, limit: 2
  kredis_enum :morning, values: %w[ bright blue black ], default: "bright"
  kredis_counter :steps, expires_in: 1.hour
end

person = Person.find(5)
person.names.append "David", "Heinemeier", "Hansson" # => RPUSH people:5:names "David" "Heinemeier" "Hansson"
true == person.morning.bright?                       # => GET people:5:morning
person.morning.value = "blue"                        # => SET people:5:morning
true == person.morning.blue?                         # => GET people:5:morning

Kredis' major benefit is the ease it provides to store ephemeral information associated with a certain record, but independent of the session. Typically, when you need to persist data in Rails, you have a few options — of which the two most common ones are:

ActiveRecord: In most cases, this requires adding a column or otherwise patching your data model. A migration is needed, plus the optional backfilling of old records.
Session: The default key/value store of every Rails app and requires no or little setup. The downside is that data stored in it doesn't survive a login/logout cycle.

Kredis brings a third option to the table. Little setup is required, apart from invoking the DSL in the model. But unless your Redis instance goes down, your data is stored across sessions, and even devices. So a good use case for Kredis is uncritical information that you want to share across device borders, e.g., in a web app and a companion mobile app.

Case Study: Persist and Restore a Collapsed/Expanded UI State Using Kredis

A typical instance of a good use case for Kredis is when persisting UI state, such as:

Sidebar open/closed state
Tree view open/closed state
Accordion collapsed/expanded state
Custom dashboard layout
How many lines of a data table to display

Exemplarily, we will take a look at how to manage the collapsed/expanded state of a <details> element.

Let's start out with a fresh Rails app, add kredis to the bundle, and run its installer:

$ rails new ui-state-kredis
$ cd ui-state-kredis

$ bundle add kredis
$ bin/rails kredis:install

Note: This will create a Redis configuration file in config/redis/shared.yml.

For the rest of this article, I will assume that you have a local running Redis instance. On macOS with Homebrew, this is as easy as running:

$ brew install redis

Please consult the official "Getting Started" guide for information on how to install Redis on your operating system.

User Authentication

We are going to use a User model as the entity to store UI state information. To avoid bikeshedding here, let's just use what Devise provides out of the box:

$ bundle add devise
$ bin/rails generate devise:install
$ bin/rails generate devise User
$ bin/rails db:migrate

We then create an example user in the Rails console:

$ bin/rails c

User.create(
  email: "julian@example.com",
  password: "mypassword",
  password_confirmation: "mypassword"
)

Our Example App: An Online Store

To illustrate how Kredis can help persist the state of a complex tree structure, let's pretend we are running an online department store. To this end, we will scaffold Department and Product models. We include a self join from department to department, to create a two-level nested structure:

$ bin/rails g scaffold Department name:string department:references
$ bin/rails g scaffold Product name:string department:references
$ bin/rails db:migrate

We have to permit null parents, of course, to allow for our tree structure roots:

  class CreateDepartments < ActiveRecord::Migration[7.0]
    def change
      create_table :departments do |t|
        t.string :name
-       t.references :department, null: false, foreign_key: true
+       t.references :department, foreign_key: true

        t.timestamps
      end
    end
  end

Our Department and Product models are defined as such:

class Department < ApplicationRecord
  belongs_to :parent, class_name: "Department", optional: true
  has_many :children, class_name: "Department", foreign_key: "department_id"
  has_many :products
end

class Product < ApplicationRecord
  belongs_to :department
end

Finally, we use faker to generate some seed data:

$ bundle add faker
$ bin/rails c

5.times do
  Department.create(
    name: Faker::Commerce.unique.department(max: 1),
    children: (0..2).map do
      Department.new(
        name: Faker::Commerce.unique.department(max: 1),
        products: (0..4).map do
          Product.new(name: Faker::Commerce.unique.product_name)
        end
      )
    end
  )
end

Scaffolding a Storefront

We'll create a very simple HomeController that will act as our shop's storefront.

$ bin/rails g controller Home index --no-helper
      create  app/controllers/home_controller.rb
       route  get 'home/index'
      invoke  erb
      create    app/views/home
      create    app/views/home/index.html.erb
      invoke  test_unit
      create    test/controllers/home_controller_test.rb

We perform a self join on the departments' children to retrieve only those which actually have subdepartments (or, in other words, are our tree's roots):

# app/controllers/home_controller.rb

class HomeController < ApplicationController
  def index
    @departments = Department.joins(:children).distinct
  end
end

In the index view, we set up a nested tree view using two levels of <details> elements for our departments:

<!-- app/views/home/index.html.erb -->

<% @departments.each do |dep| %>
  <details>
    <summary><%= dep.name %></summary>

    <% dep.children.each do |child_dep| %>
      <details style="margin-left: 1rem">
        <summary><%= child_dep.name %></summary>

        <ul>
          <% child_dep.products.each do |prod| %>
            <li><%= prod.name %></li>
          <% end %>
        </ul>
      </details>
    <% end %>
  </details>
<% end %>

Right now we have a tree view of departments with intentionally silly product names that we can explore by opening and closing:

We'd like to persist the disclosure state of the individual categories, which we will tend to next.

Persisting UI State of Categories in Kredis

Here is what we are going to do, step by step:

Add a kredis_set called open_department_ids to the User model. The reason we are using a set here is that it doesn't allow duplicates, so we can safely add and remove our departments.
Create a UIStateController that will receive the following params:
- the department_id
- the open state of that department
It will then add or remove this department to the kredis_set for the currently logged-in user.
Create a Stimulus controller which will listen for the toggle event on the details element and send over the respective payload.

Let's get into it!

Adding said Kredis data structure to the User model is as easy as calling kredis_set and passing an identifier:

  # app/models/user.rb

  class User < ApplicationRecord
    # Include default devise modules. Others available are:
    # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
    devise :database_authenticatable, :registerable,
           :recoverable, :rememberable, :validatable

+   kredis_set :open_department_ids
  end

Next, we generate a UIStateController to receive the UI state updates. Note that we have to configure the generated route to be a patch endpoint:

$ bin/rails g controller UIState update --no-helper --skip-template-engine
      create  app/controllers/ui_state_controller.rb
       route  get 'ui_state/update'
      invoke  test_unit
      create    test/controllers/ui_state_controller_test.rb

  Rails.application.routes.draw do
-   get 'ui_state/update'
+   patch 'ui_state/update'
    get 'home/index'
    resources :products
    resources :departments
    devise_for :users
    # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html

    # Defines the root path route ("/")
    root "home#index"
  end

Our first encounter with Kredis' API is in the controller. We can see that it tries to conform to Ruby developers' expectations as closely as possible, so you can add to the set using <<, and delete using remove.

# app/controllers/ui_state_controller.rb

class UiStateController < ApplicationController
  def update
    if ui_state_params[:open] == "true"
      current_user.open_department_ids << params[:department_id]
    else
      current_user.open_department_ids.remove(params[:department_id])
    end

    head :ok
  end

  private

  def ui_state_params
    params.permit(:department_id, :open)
  end
end

What's happening here is that we toggle the presence of a specific department_id in the set based on the open param being handed over from the client. To complete the picture, we must write some client-side code to transmit these UI state changes.

We are going to use @rails/request.js to perform the actions, so we have to pin it:

$ bin/importmap pin @rails/request.js
Pinning "@rails/request.js" to https://ga.jspm.io/npm:@rails/request.js@0.0.8/src/index.js

In a new Stimulus controller that we'll attach to a specific <details> element, we append the department ID and its open state to a FormData object, and submit it:

// app/javascript/controllers/ui_state_controller.js

import { Controller } from "@hotwired/stimulus";
import { patch } from "@rails/request.js";

export default class extends Controller {
  static values = {
    departmentId: Number,
  };

  async toggle() {
    const body = new FormData();
    body.append("open", this.element.open);
    body.append("department_id", this.departmentIdValue);

    await patch("/ui_state/update", {
      body,
    });
  }
}

We edit our view code as proposed, and listen for the toggle event of each <details> element to trigger the UI state updates:

  <!-- app/views/home/index.html.erb -->

  <% @departments.each do |dep| %>
-   <details>
+   <details
+     data-controller="ui-state"
+     data-action="toggle->ui-state#toggle"
+     data-ui-state-department-id-value="<%= dep.id %>"
+   >
      <summary><%= dep.name %></summary>
      <% dep.children.each do |child_dep| %>
-       <details style="margin-left: 1rem">
+       <details style="margin-left: 1rem"
+         data-controller="ui-state"
+         data-action="toggle->ui-state#toggle"
+         data-ui-state-department-id-value="<%= child_dep.id %>"
+       >
          <summary><%= child_dep.name %></summary>

          <ul>
            <% child_dep.products.each do |prod| %>
              <li><%= prod.name %></li>
            <% end %>
          </ul>
        </details>
      <% end %>
    </details>
  <% end %>

Rehydrate the DOM Manually

The only component missing to go full circle is rehydrating our DOM to the desired state once the user refreshes the page. We do this manually by adding the open attribute to the <details> node (if its department ID is present in the Kredis set):

  <!-- app/views/home/index.html.erb -->

  <% @departments.each do |dep| %>
    <details
      data-controller="ui-state"
      data-action="toggle->ui-state#toggle"
      data-ui-state-department-id-value="<%= dep.id %>"
+     <%= "open" if current_user.open_department_ids.include?(dep.id) %>
    >
      <summary><%= dep.name %></summary>

      <% dep.children.each do |child_dep| %>
        <details style="margin-left: 1rem"
          data-controller="ui-state"
          data-action="toggle->ui-state#toggle"
          data-ui-state-department-id-value="<%= child_dep.id %>"
+         <%= "open" if current_user.open_department_ids.include?(child_dep.id) %>
        >
          <summary><%= child_dep.name %></summary>

          <ul>
            <% child_dep.products.each do |prod| %>
              <li><%= prod.name %></li>
            <% end %>
          </ul>
        </details>
      <% end %>
    </details>
  <% end %>

Finally, here's the result. Note that the open/closed state of individual tree nodes is preserved over 2 levels:

Up Next: A Generalized User-local Container for UI State

In the first part of this two-part series, we introduced Kredis and explored how to persist and restore a collapsed/expanded UI state with Kredis.

We used the example of an online department store to highlight how Kredis can persist a complex tree structure's state, before finally manually rehydrating the DOM.

However, this does mean that we have to invent a lot of Kredis keys. Next time, we'll dive into how we can address this with a generalized user-local container for UI state.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

What's New in Rails 7.1

Roy Tomeij — 2023-02-15T00:00:00+00:00

Rails 7 was a welcome release that brought a lot of significant features and changes. On the backend, Rails 7 introduced asynchronous query loading and Zeitwerk for autoloading. The frontend saw Hotwire becoming the default solution for new Rails apps.

Rails 7.1 will add to these notable features. In this post, we'll discuss some noteworthy additions that are likely to be shipped.

A New API for Async Queries in Rails

Building on an earlier feature from Rails 7, Rails 7.1 will make it possible to run some queries asynchronously. Rails 7 introduced ActiveRecord::Relation#load_async, which schedules a query in a background thread pool, allowing us to do stuff like Post.where(published: true).load_async.

In Rails 7.1, we'll be able to run a lot more queries in background threads. Aggregate methods will run concurrently. Assuming you run two or more independent queries on a job or controller, query results may return quicker if your application is set up accordingly.

For this to work as intended, there are two configuration options worth paying attention to:

config.active_record.async_query_executor
config.active_record.global_executor_concurrency

Among the methods you can run asynchrously in Rails 7.1 are async_sum, async_pluck, and async_count_by_sql.

Resetting Singular Associations

Rails 7.1 allows resetting the cache on singular associations. Currently, you can only clear the cache of has_many associations with a class like:

class Teacher < ActiveRecord::Base
  has_many :students
  has_one :classroom
end

teacher = Teacher.first

We can only do teacher.students.reset to clear the caches of the results returned by teacher.students. Subsequent requests need to hit the database again for a fresh results set in case some data goes stale.

With Rails 7.1, we'll get the reset method on a has_one association. Using our example class above, Rails 7.1 will allow us to do teacher.classroom.reset_teacher to clear the cache for the associations between teacher and classroom.

Disabling Methods Generated By `ActiveRecord#enum`

ActiveRecord#enum generates a bunch of methods if you create an enum Rails. Rails 7.1 will provide an option to opt out of these generated methods. Here's a simple example:

class Payment < ActiveRecord::Base
  enum :status, %i[succeeded failed], instance_methods: false
end

Rails won't generate auxiliary methods with instance_methods: false. Currently, we expect to have methods like:

payment = Payment.first

payment.succeeded?
payment.failed?

payment.succeeded!
payment.failed!

Support for Common Table Expressions

Rails 7.1 will have in-built support for Common Table Expressions (CTEs). This ensures that code will be more succinct but, more importantly, that we won't have to use Arel::Nodes for complex queries.

With Rails 7.1, we'll have a new .with method to write queries similar to the one below:

Post.with(
  posts_with_comments: Post.where("comments_count > ?", 0),
  posts_with_tags: Post.where("tags_count > ?", 0)
)

Support for Async Bulk Record Destruction

As mentioned, Rails 7.1 will introduce several ways to run code asynchronously. One such new addition to async code executions is the destroy_association_async_batch_size configuration.

With this new configuration, Rails applications can set a maximum number of records to be destroyed in a single background job by the dependent: :destroy_async association.

The default behavior, where all dependent records are destroyed in a single background job when the parent record is destroyed, will remain unchanged. However, if the number of dependent records exceeds the new configuration, they will be destroyed in multiple background jobs.

Other Rails 7.1 Updates

`ActiveRecord::Relation#explain` Accepts Options

Rails 7.1 will allow you to pass database systems that support EXPLAIN options to ActiveRecord::Relation#explain. An example query might look like this:

Customer.where(id: 1).joins(:orders).explain(:analyze, :verbose)

Active Record `regroup`

Active Record will allow for "regrouping" queries with a new regroup method that can be used like so: Post.group(:title).regroup(:author). This generates SQL equivalent to SELECT posts.* FROM posts GROUP BY posts.author.

The same can be achieved in current versions of Rails with more verbose code:

 Post.group(:title)..unscope(:group).group(:author)

New `stub_const method` for Testing

A new stub_const method for ActiveSupport::TestCase will be added that stubs a constant for a yield's duration.

For example:

# World::List::Import::LARGE_IMPORT_THRESHOLD = 5000

stub_const(World::List::Import, :LARGE_IMPORT_THRESHOLD, 1) do
  assert_equal 1, World::List::Import::LARGE_IMPORT_THRESHOLD
end

assert_equal 5000, World::List::Import::LARGE_IMPORT_THRESHOLD = 5000

Take note, however, that stubbing a constant will affect its value across all threads in a multi-threaded setup. This means that if multiple concurrent threads rely on the same constant, simultaneous and conflicting stubbing may occur.

Password Challenge via `has_secure_password`

Rails 7.1 has improved the functionality of has_secure_password by adding a password_challenge accessor and a corresponding validation. The validation will verify that a password_challenge matches the stored password_digest.

With this, implementing a password challenge becomes as straightforward as a password confirmation. This also enables reusing the same error-handling logic in both the view and the controller.

For instance, instead of writing separate code in the controller, you will simply use the existing logic for password confirmation.

password_params = params.require(:password).permit(
  :password_challenge,
  :password,
  :password_confirmation,
).with_defaults(password_challenge: "")

if current_user.update(password_params)
  # perform some work
end

Saving Attachments Returning the Blob

With Rails 7.1, when you save attachments to a record, the attach method will return the attached blob or blobs. This enables the direct use of blob methods on the attachment. However, if the record fails to save, attach will return false.

Here's an example demonstrating its use:

@user = User.create!(name: "Josh")
avatar = @user.avatar.attach(params[:avatar])

# You can now directly call blob methods as follows:
avatar.download
avatar.url
avatar.variant(:thumb)

Storage of CSRF Tokens Outside of Sessions

Rails has introduced a new configuration option to address the excessive creation and eviction of millions of sessions for just storing a CSRF token when sessions are not stored in cookies.

This option allows the use of a lambda function to store the CSRF token in a custom location, thus enabling the storage of CSRF tokens outside of sessions.

You can also create custom strategy classes for storing CSRF tokens.

class CustomStore
  def fetch(request)
    # Return the token from a custom location
  end

  def store(request, csrf_token)
    # Store the token in a custom location
  end

  def reset(request)
    # Delete the stored session token
  end
end

class ApplicationController < ActionController:x:Base
  protect_from_forgery store: CustomStore.new
end

Validity Checking for PostgreSQL Indexes

Creating indexes as shown below may lead to an invalid index:

add_index :account, :active, algorithm: :concurrently

With Rails 7.1, you can verify an index's validity as shown here:

connection.index_exists?(:users, :email, valid: true)
connection.indexes(:users).select(&:valid?)

`ActiveRecord::QueryMethods#select` Accepts a Hash

ActiveRecord::QueryMethods#select in Rails 7.1 now accepts a hash of options. This is best demonstrated with an example:

# You can now write selects like this:
Post.joins(:comments).select(
  posts: { id: :post_id, title: :post_title },
  comments: { id: :comment_id, body: :comment_body}
)

# In place of this:
Post.joins(:comments).select(
  "posts.id as post_id, posts.title as post_title,
  comments.id as comment_id, comments.body as comment_body"
)

Number of Processors Match the Puma Worker Count

By default, newly generated Rails applications will have Puma workers that are capped at the total number of physical processors on the host machine. This default setting can always be changed in the puma.rb file.

The puma.rb file for newly-generated Rails applications will now look like the following:

if ENV["RAILS_ENV"] == "production"
  worker_count = ENV.fetch("WEB_CONCURRENCY") { Concurrent.physical_processor_count }
  workers worker_count if worker_count > 1
end

`preload` and `eager_load` Associations to Be Unscoped

Rails 7.1 will add the ability to unscope preloaded and eager loaded associations in a manner similar to how Active Record's includes, select, and joins methods work.

This feature allows for the use of aggregate functions on has_many associations previously loaded through eager_load or preload in existing queries.

An example usage could look like:

query.unscope(:eager_load, :preload).group(:id).select(:id)

Default Dockerfiles for New Rails Applications

Docker files are to be added as a default option for new Rails applications. The files include:

Dockerfile
.dockerignore
bin/docker-entrypoint

These files serve as a starting point for deploying an application in a production environment and are not intended for use during development. However, if desired, you can skip these files by using the --skip-docker option.

Default Health Controller

Rails 7.1 introduces a new endpoint for load balancers and uptime monitors. The endpoint, named Rails::HealthController#show, is mapped to the "/up" path in newly generated Rails applications. This allows load balancers and uptime monitors to easily track an app's availability.

Note that monitoring the database, Redis, and internal network connections to microservices that an application relies on must be managed separately.

New `Rails.env.local?` for Environment Checks

In Rails 7.1, a new local? method can be used to simplify environment checks.

You'll be able to replace code like:

if Rails.env.development? || Rails.env.test?
end

With:

if Rails.env.local?
end

New `ActiveRecord::Persistence#update_attribute!` Method

Rails has added a new method, ActiveRecord::Persistence#update_attribute!, which functions similarly to update_attribute but uses save! instead of save.

Here's how you could use this new method:

class Apm < ActiveRecord::Base
  before_save :check_name

  def check_name = throw(:abort) if name == "abort"
end

monitor = Apm.create(name: "App Signal")
# => #<Apm name: "App Signal">

monitor.update_attribute!(:name, "AppSignal")
# => #<Apm name: "AppSignal">

monitor.update_attribute!(:name, "abort")
# raises ActiveRecord::RecordNotSaved

Templates Capable of Defining Accepted Locals

Templates will be enhanced with the option of required arguments that have default values.

Currently, templates accept any locals as keyword arguments. With 7.1, Rails templates will define specific accepted locals through a magic comment.

This improvement provides greater control and customization options for template behavior and functionality.

A partial in Rails could now look like:

<%# locals: (title: "Default title", comment_count: 0) %>

<h2><%= title %></h2>
<span class="comment-count"><%= comment_count %></span>

Instead of:

<% title = local_assigns[:title] || "Default title" %>
<% comment_count = local_assigns[:comment_count] || 0 %>

<h2><%= title %></h2>
<span class="comment-count"><%= comment_count %></span>

Wrapping Up

As you can see, Rails 7.1 promises a lot of further improvements on Rails 7.

For more information on features, updates, and bug fixes, check out the Rails 7.1 release notes.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Guide to Rails View Helpers

Roy Tomeij — 2023-02-01T00:00:00+00:00

Views in Rails don't do much besides showcasing what we want. Sure, they might render slightly different results depending on who's watching (an admin or a logged-in user has a different experience than a guest user, for example), but they don't do much processing on what they're given.

Or at least they shouldn't. Often though, Rails projects wind up with a lot of logic in their views.

In this post, we'll explore how to use Rails helpers to keep our views clean and readable.

But first, let's see why too much logic in our views can become problematic.

The Problem with Logic-heavy Rails Views

Logic-heavy views are bad for a lot of reasons. We might end up with logic-heavy views because of the pressure of a crushing deadline, or they might result from inevitable legacy code buildup. New features get introduced along with new edge cases, and it seems harmless to update <%= @post.name.titlecase %> to <%= @post.name.titlecase.gsub("#", "") %>.

But before you know it, this can snowball, even if you don't keep tacking on little additions to this particular line of code. If you keep doing things like this elsewhere in the view, you end up with an ugly, difficult-to-read mess.

So what do we do? Should we litter our controllers with dozens of instance variables like @normalized_titlecased_post_name = @post.name.titlecase.gsub("#", "")? Bloat our models with tons of methods intended only for use in our views? Probably not.

Enter Rails helpers.

What Are Helpers in Rails?

A helper is a method intended to abstract logic out of our views, leaving them more readable, less cluttered, and not directly responsible for processing what they've been passed by our controllers.

Rails already ships with a lot of useful helpers to begin with. Some you're likely already familiar with, like form_with, link_to, and image_tag, to name a few.

Most of the helpers you write yourself will naturally be much more specific to your app than what Rails includes, but you will likely still find many Rails helpers very useful.

If you haven't already, familiarize yourself with the helpers Rails provides — otherwise, there's a decent chance you'll end up trying to solve a problem that's already been solved.

I once wrote a method early in my Rails career to convert a number to a dollar amount, only to discover later that Rails already had a solution — number_to_currency!

Organizing Your Helpers in Rails

By default, a new Rails application has one helper — ApplicationHelper — which all other helpers should inherit from. You've likely noticed that any time you create a controller via rails g controller, Rails automatically creates a new (pluralized) helper module for you as well.

While this form of organization (having a helper module for each controller) is both useful and important, it's also a bit deceptive. The methods defined in any of these helpers are available in all your views. So if you define my_generic_method() in, say, UsersHelper, and an identically-named my_generic_method() in TransactionsHelper, they won't only be available in their respective views — one will simply overwrite the other and almost certainly break something.

So you should be fairly thoughtful when naming your helper methods. Don't assume there's an invisible User:: namespace or user_ prefix in front of the methods in your UsersHelper, because there isn't, real or implied.

Any helper methods generic enough to be used all over (most of these probably won't be model-specific) can go in your ApplicationHelper. Methods specific to view folders (e.g., /views/users) belong to helpers of the same name.

Again, this is strictly for organizational purposes, given all helper methods are available to all views. But, as you might imagine, it will definitely confuse other developers (and likely your future self) if you end up finding User-specific methods in your ZooAnimalsHelper.

Note: Before Rails 4, this was not the case. You can turn this behavior off by setting config.action_controller.include_all_helpers = false in your application.rb.

Do's and Don'ts for Helpers

Helpers in Controllers

You can include helpers in controllers. In modern Rails versions (5+), you can simply access them in your controllers via helpers (e.g. helpers.number_to_currency).

It's often tempting, and you can absolutely abuse this — I won't pretend I never have. But generally, you shouldn’t. It's okay to do this occasionally, and if you must, do use the helpers method rather than include on the entire helper module in your controller. include dumps every method defined in that helper into the controller, potentially conflicting with your controller methods.

If you keep finding you want to share a helper across different layers of your app, consider whether it might be better off elsewhere, like in a model.

Helpers and Instance Variables

Helpers can access any instance variables available for the view they're called in. The instance variables can be referenced inside your helpers without being passed explicitly as method arguments, but this is not advised.

Instead, pass your instance variables to your helpers as regular arguments. Doing otherwise tightly couples your controllers and views (and helpers), making reuse, refactoring, testing, and debugging difficult.

On rare occasions, you might want to reference instance variables that aren't passed as method arguments inside your helpers. You could have a helper method that's so specific, it only belongs in one spot, or the opposite — it only leverages an instance variable you have everywhere.

This is still somewhat iffy, but ultimately it's up to you. I will say though that if you're not very certain, err on the side of caution. As with anything else, you'll get a feel for things as you see what does and doesn't work over time.

Never Modify Objects with Helpers

This is probably obvious, but a helper should never modify the state of a passed object or (even worse) modify that object in the database. Remember, they're responsible for helping the View display things cleanly, not changing anything.

Never Make Database Calls with Helpers

Helpers should also never be responsible for making calls to the database — no part of the view should. Not only does doing so badly violate the separation of concerns, it's also incredibly confusing to other developers (where is this query coming from?), and can make tracking down problems difficult. I once spent hours completely baffled by where an N+1 was being introduced, only to discover some queries inside a loop embedded in the view.

Don't Let Helpers Become a Dumping Ground

One final word of caution: don't, as is so often the case, let your helpers become a dumping ground for one-off methods and little snippets of code that either serve little purpose, or might be better off as another kind of method. They should be relatively simple methods, and only concerned with the view; business logic really belongs in our models.

Generating HTML with Rails Helpers for Views: An Example

Rails helpers can work with and generate HTML for our views. Remember Rails' built-in helpers? A content_tag will let us generate our own HTML.

Let's say we have a zoo and want to display a list of exhibit times for a given group of animals a customer selects. We might have something like this in our view:

<div>
  <% @animals.map do |animal| %>
    <div class="time-heading">
      <%= animal.name %> (<i><%= animal.species.name %></i>): <%= animal.exhibit_time.in_time_zone(animal.exhibit_time.time_zone).strftime("%m/%d/%Y %Z") %>
    </div>
  <% end %>
</div>

There are a few issues with this. For one, it's a bit hard to read. We're also calling things like in_time_zone and strftime to format and display the exhibit time, along with a lot of method chaining.

It's also in no way reusable. Let's say we need to display this (or something similar) in different places on our site. If we then later decide it needs a style or content update, we have to change it everywhere. This is annoying for obvious reasons (in addition to the fact that it can be tricky to find every example if they're all slightly different).

Let's see what handling this in a helper might look like:

# helpers/animals_helper.rb
module AnimalsHelper
  def display_exhibit_times(animals, dom_class: nil, style: nil)
    exhibit_strings = animals.map { |animal|
      "#{animal.name} (<i>#{animal.species.name}</i>): #{animal.exhibit_time.formatted_start_time}".html_safe
    }

    exhibit_strings.map { |exhibit_string|
      content_tag(:div, "#{exhibit_string}", class: dom_class, style: style)
    }.join
  end
end

Admittedly, this may not be the prettiest example, but it's considerably easier to read and, more importantly, removes the logic from our view entirely. It's also reusable and flexible; notice the two named arguments we've passed in — dom_class and style. Because content_tag allows us to pass in HTML class and style information, we can pass this in any time we call this method to customize it in different places as needed.

Another thing you may have noticed is the transformation of the time formatting into a call from exhibit_time. This is just to point out that not all methods used in the view have to be helpers; this might be a model method, or a mixin shared across models with times and dates for easier formatting. (That isn't to say it might not also make sense to use a helper here instead; it might.)

Note: .html_safe is needed to translate the  tag to HTML. .join is required to return multiple content tags for use in the DOM.

Our view now becomes:

<%= display_exhibit_times(@animals, dom_class: "time-heading") %>

Isn't that better?

A Second Example

Let's consider another example. We need to email our customers their tickets, and we'd like to provide them with an online menu for a dining area at our zoo.

In both cases, we'll probably want to render QR codes. Let's use RQRCode, a gem that can do just that for us:

<div>
  <%= RQRCode::QRCode.new(@ticket.qr_code).as_svg(color: "black", shape_rendering: "crispEdges", module_size: 5, standalone: true, use_path: true) %>
</div>

This isn't ideal for a number of reasons. For one thing, we're instantiating objects in our view. For another, we're passing every single required option, when most of them will probably be the same throughout our app.

Let's move this to a helper:

# helpers/application_helper.rb
def render_qr_code(code, options = {})
  qr_code = RQRCode::QRCode.new(code)

  defaults = {
    color: "black",
    shape_rendering: "crispEdges",
    module_size: 3,
    standalone: true,
    use_path: true
  }

  qr_code.as_svg(defaults.merge(options)).html_safe
end

Our view becomes:

<div>
  <%= render_qr_code(@ticket.qr_code, module_size: 5) %>
</div>

Much cleaner, and now we can reuse it for our restaurant!

One More Trick: The `tag` Helper

Do you ever find yourself interpolating conditional class names in your elements?

Say we're listing our animal exhibits on our main page, and want to highlight certain exhibits if they're currently featured:

<div class="border-black margin-standard <%= 'bg-featured text-bold' if @exhibit.featured? %>">
  <%= @exhibit.name %>
</div>

As of Rails 6.1, you can set conditional classes using the tag helper.

<%= tag.div class: ["border-black margin-standard", "bg-featured text-bold": @exhibit.featured?] do %>
  <%= @exhibit.name %>
<% end %>

The classes "bg-featured" and "text-bold" will only be included if it's a featured exhibit. This can make things easier to read, especially if you've got multiple classes with multiple conditions!

There Isn't Always a 'Perfect' Solution

So you've been using helpers for a while now and are comfortable with them. You've refactored your views, pulled methods out of your models and your controllers, and everything's cleaner, easier to read and test. You've noticed you're introducing fewer bugs into your application.

And yet — you still have a bit of logic in some of your views, a few calls to helpers in your controllers, and a few methods that don't belong in the right place.

That's completely fine. This is normal and, at least in my opinion, inevitable. Some devs claim there's always a solution — be it a helper or model method, presenter/decorator class, etc., but personally, I'm not convinced.

Suppose you continually find you want to use a particular method across multiple layers of your app. In that case, it might be a good candidate for a concern, model method, or part of a separate module. But sometimes, there isn't an absolutely perfect solution for every use case or a perfect place to put code. There's certainly no such thing as a 'perfect' app. And that's okay.

Wrapping Up

In this article, we explored how to make your Rails views cleaner and more readable using helpers. We looked at how to organize your helpers, some do's and don'ts, and how to generate HTML for your views using helpers.

We finally argued that even if you master helpers, you'll likely still never achieve the 'perfect' Rails app. However, you'll certainly have an app that's easier to maintain.

I hope you've found this post useful.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Calling Ruby Methods in C: Avoid Memory Leaks

Roy Tomeij — 2023-01-25T00:00:00+00:00

Memory leaks are a pain for gem users. They are hard to track and can lead to expensive infrastructure costs.

Memory leaks within a C extension are even worse. You'll see a lot of tools and articles about finding leaks in Ruby. However, you don't have the same access to internals in C.

A naive usage of rb_funcall can cause memory leaks: it's much better to use rb_protect instead. So, if you are a C extension writer, please read on for the sake of developers who will use your gem.

Let's get started!

The Issue with `rb_funcall` and C

rb_funcall can be a great tool when you need to interact between Ruby and the C parts of your library but only need to write a little C.

However, when you run rb_funcall, you are no longer in C where everything is straightforward. You can be left in muddy waters if the called function:

Completely changes its definition during runtime
Raises a call

Number 1 is the easiest one to catch. You'll likely end up with a segfault, and if your test suite is complete enough, you should catch that before publishing.

However, the latter can cause memory leaks and make your codebase way harder to read. Let's take a look at that now.

Raise in Ruby Causing C Memory Leaks

Ruby's raising mechanism jumps between parts of the code from one scope to the first parent that catches an error. This is implemented in the MRI using longjmp and setjmp.

If you are interested in how this is built, read the Evaluator chapter in the Ruby Hacking Guide. In a nutshell, when you use a begin..ensure block, you setjmp(), and when you raise within this block, you longjmp() to the saved position.

So if a function is raised with rb_funcall, the C code called after it never executes.

The example below illustrates a potential leak. If json_parse raises, it will leak.

VALUE rb_create_geometry_hash(VALUE self, VALUE wkt) {
    // Alloc
    GEOSWKTReader* reader = GEOSWKTReader_create();
    GEOSGeoJSONWriter* writer = GEOSGeoJSONWriter_create();

    // C processing
    GEOSGeometry* geom = GEOSWKTReader_read(reader, StringValuePtr(wkt));
    char* geojson = GEOSGeoJSONWriter_writeGeometry(writer, geom, -1);

    // Ruby processing
    VALUE rb_geojson = rb_str_new_cstr(geojson);
    VALUE result = rb_funcall(self, rb_intern("json_parse"), 1, rb_geojson);

    // Free
    GEOSWKTReader_destroy(reader);
    GEOSGeom_destroy(geom);
    GEOSGeoJSONWriter_destroy(writer);
    GEOSFree(geojson);

    return result;
}

Of course, the example above is a bit silly — you could invert the freeing and Ruby processing parts. However, this is not always possible, and longer function bodies can become more intertwined.

Using `begin..ensure` in Ruby

If you're using Ruby, you could instead write the above example using begin..ensure:

def create_geometry_hash(wkt)
    reader = GEOSWKTReader.new
    writer = GEOSGeoJSONWriter.new

    begin
        json_parse(writer.write(reader.read(wkt)))
    ensure
        reader.close
        writer.close
    end
end

This API is also available in C with rb_rescue and rb_ensure:

static VALUE try_ruby_processing(VALUE args) {
    char* geojson = (char*)args;
    // Ruby processing
    VALUE rb_geojson = rb_str_new_cstr(geojson);
    VALUE result = rb_funcall(self, rb_intern("json_parse"), 1, rb_geojson);
}

struct to_free {
    GEOSWKTReader* reader;
    GEOSGeoJSONWriter* writer;
    GEOSGeometry* geom;
    char* geojson;
};

static VALUE ensure_free(VALUE args) {
    struct to_free data = (struct to_free)args
    GEOSWKTReader_destroy(data.reader);
    GEOSGeom_destroy(data.geom);
    GEOSGeoJSONWriter_destroy(data.writer);
    GEOSFree(data.geojson);

}

VALUE rb_create_geometry_hash(VALUE self, VALUE wkt) {
    // Alloc
    GEOSWKTReader* reader = GEOSWKTReader_create();
    GEOSGeoJSONWriter* writer = GEOSGeoJSONWriter_create();

    // C processing
    GEOSGeometry* geom = GEOSWKTReader_read(reader, StringValuePtr(wkt));
    char* geojson = GEOSGeoJSONWriter_writeGeometry(writer, geom, -1);

    return rb_ensure(
        try_ruby_processing, (VALUE)geojson
        ensure_free, (struct to_free){ reader, writer, geom, geojson }
    );

    return result;
}

However, this is a bit cumbersome, and if you want to add a rescue block to the party, it gets way less readable. I suggest reading Peter Zhu's 'A Rubyist's Walk Along the C-side (Part 8): Exceptions & Error Handling' if you want to use the begin..rescue..ensure..end API in C.

Using `rb_protect` for C

There is another option. First, let's see how it could look in Ruby:

def create_geometry_hash(wkt)
    reader = GEOSWKTReader.new
    writer = GEOSGeoJSONWriter.new

    err = nil
    result = nil
    begin
        result = json_parse(writer.write(reader.read(wkt)))
    rescue => e
        err = e
    end

    reader.close
    writer.close

    raise err if err

    result
end

This looks strange in Ruby, but is a workflow very well suited to C. The MRI has an API for that, rb_protect, and the C function looks like this:

VALUE ruby_call(VALUE rb_geojson) {
    return rb_funcall(self, rb_intern("json_parse"), 1, rb_geojson);
}

VALUE rb_create_geometry_hash(VALUE self, VALUE wkt) {
    int state;

    // Alloc
    GEOSWKTReader* reader = GEOSWKTReader_create();
    GEOSGeoJSONWriter* writer = GEOSGeoJSONWriter_create();

    // C processing
    GEOSGeometry* geom = GEOSWKTReader_read(reader, StringValuePtr(wkt));
    char* geojson = GEOSGeoJSONWriter_writeGeometry(writer, geom, -1);

    // Ruby processing
    VALUE rb_geojson = rb_str_new_cstr(geojson);
    rb_protect(ruby_call, rb_geojson, &state);

    // Free
    GEOSWKTReader_destroy(reader);
    GEOSGeom_destroy(geom);
    GEOSGeoJSONWriter_destroy(writer);
    GEOSFree(geojson);

    if (state) rb_jump_tag(state);

    return result;
}

The above method will re-raise a Ruby error after having freed everything.

Note that we could also choose to ignore the error by using an empty rescue block in Ruby:

    ...

    if (state) rb_set_errinfo(Qnil);

    return result; // => nil
}

Warning: If you do not raise the error, the rb_set_errinfo(Qnil) step is important so you don't keep information available about an error that users should not know about.

Or, you can conditionally choose to raise an error, like rescue My::Error:

    ...

    if (state) {
        if (rb_obj_is_kind_of(rb_errinfo(), rb_define_class_under(rb_mMy, "Error", rb_eStandardError))) {
            rb_jump_tag(state);
        } else {
            rb_set_errinfo(Qnil);
        }
    }

    return result;
}

You can actually consider rb_errinfo() as the same as the $! global variable.

This is all great, but when it boils down to one rb_funcall only, we can simplify that API.

The overall idea behind using the rb_protect API when there is a function to raise is to enhance readability. You don't need to check if the function can raise or not, you assume it can, and use the state to work with that.

The `rb_protect_funcall` Proposal

Let's isolate rb_funcall, as it's the only dangerous method to use. Here's an API that will do that:

VALUE rb_protect_funcall(VALUE recv, ID mid, int* state, int n, ...);

This API is the same as rb_funcall, with a state from rb_protect. Hence the usage is pretty straightforward:

VALUE rb_create_geometry_hash(VALUE self, VALUE wkt) {
    int state;

    // Alloc
    GEOSWKTReader* reader = GEOSWKTReader_create();
    GEOSGeoJSONWriter* writer = GEOSGeoJSONWriter_create();

    // C processing
    GEOSGeometry* geom = GEOSWKTReader_read(reader, StringValuePtr(wkt));
    char* geojson = GEOSGeoJSONWriter_writeGeometry(writer, geom, -1);

    // Ruby processing
    VALUE rb_geojson = rb_str_new_cstr(geojson);
    rb_protect_funcall(self, rb_intern("json_parse"), &state, 1,  rb_geojson);

    // Free
    GEOSWKTReader_destroy(reader);
    GEOSGeom_destroy(geom);
    GEOSGeoJSONWriter_destroy(writer);
    GEOSFree(geojson);

    if (state) rb_jump_tag(state);

    return result;
}

This API is not yet available in Ruby, and may never be. You can take it from RGeo (MIT LICENSE).

A Real-World Example

If you want to see a real-world example, I encourage you to read the RGeo codebase as we recently switched to going full rb_protect. We even have some functions, such as rgeo_convert_to_geos_geometry, that propagate this state for simpler usage. This function is a good place to start digging around.

Feel free to open an issue on RGeo to discuss the choices we made further.

Wrapping Up

In this post, we warned against using rb_funcall with C as it can cause memory leaks. We explored using begin..ensure or rb_protect instead.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Parse Arguments in Your Ruby C Extension

Roy Tomeij — 2023-01-18T00:00:00+00:00

Ruby is a wonderful language, made for humans first and machines second. It is easy to read and write. There are plenty of ways to write anything, and you can often guess its standard library by typing the name of the method you would have chosen yourself.

Because of this, Ruby's arguments are very flexible, which lets us express our APIs very clearly. But this comes with a drawback: Ruby is quite hard to parse for C extension developers!

In this article, we'll go through two ways to set up a complex Ruby API that is written in C:

with rb_define_method and parsing it with rb_scan_args
using a Ruby interface

Let's get started!

C and Ruby: An Introduction

As mentioned, Ruby is hard to parse for C extension developers.

For example:

def this(is, a = "quite", *convoluted, yet: 1, possible:, &example)
  # And we could have omitted the block, yet still passed it as an argument!
end

The beauty of C, the language Ruby is written in, stems from its simplicity, including in its function parameters:

<data-type> <variable-identifier>
... for variadic arguments.

These will help you maintain a codebase that is not too hard to understand.

Here's the most complex way to define a C function:

int printf(const char*, ...);

When you code a C extension for your Ruby codebase, you'll start to understand where the complexity begins. But don't worry — Ruby MRI developers have us covered.

Simple Method Definition in a Ruby C Extension

We'll start with a method you'll have to use at some point, rb_define_method.

You can also follow along with the code examples in this repo.

Here is rb_define_method's signature:

void rb_define_method(VALUE klass, const char *name,
                      VALUE (*func)(ANYARGS), int argc);

And according to Ruby's extension.rdoc:

argc is the number of arguments. if argc is -1, the function will receive 3 arguments: argc, argv, and self. if argc is -2, the function will receive 2 arguments, self and args, where args is a Ruby array of the method arguments.

In a nutshell:

// argc is -1.
VALUE func(int argc, VALUE* argv, VALUE self);
// argc is -2.
VALUE func(VALUE self, VALUE args);
// argc is N (here N=2).
VALUE func(VALUE self, VALUE arg1, VALUE arg2);

So if your API only consists of methods with fixed parameter lengths, or only one variadic parameter (def foo(*bar)), read no further — you are done! If you want a richer way to call your API, please, be my guest.

By the way, if you want more involved examples on rb_define_method, I suggest you read Peter Zhu's article on Defining Methods.

Using Ruby C API Internals

So, let's go back to our use case: parsing complex arguments. Fortunately, some tools can help us.

But first, let's see the limitations of solely using rb_define_method.

Drawbacks of `rb_define_method`

No Mention of Block Arguments

One limitation is that rb_define_method never mentions block arguments. Those are not considered, as it doesn't really matter to Ruby anyway if you pass a block. You can still ensure that a block is passed by using rb_block_given_p or rb_need_block. There's more on that topic in Peter Zhu's article.

Args Can Vary

Another important limitation is that args can vary, but the method call itself is not so constrained. Therefore, if you want your API to be like def foo(bar, *baz), you'll have to parse your arguments. There are a few methods to help you down that path. rb_check_arity is one you can use, along with the -1 version of rb_define_method.

Here's the function signature:

rb_check_arity(int argc, int min, int max)

Keyword Arguments

One last limitation we'll go through is the use of keyword arguments. And I kept the best for last, as that will be the core of this article.

We have to retrieve keyword arguments before we can parse them correctly. Fortunately, the Ruby C API comes with a method for this, rb_scan_args.

Here's the rb_scan_args signature:

rb_scan_args(int argc, VALUE *argv, const char *fmt, ...)

You pass rb_scan_args the argc and argv given by rb_define_method — a string that says how the arguments should be parsed (fmt) and the receiver for those arguments. There you go, all of Ruby's args complexity parsed in one line! Well, almost.

You can refer to extension.rdoc for a formal representation of how fmt should be written, although we'll partially cover it in the examples below.

Write and Parse a Function: An Example

For the rest of this article, let's consider that we want to write this function:

def voronoi_diagram(envelope, *polygons, tolerance:, only_edges: false)
end

Parsing this using rb_scan_args will look like this:

VALUE voronoi_diagram(int argc, VALUE *argv, VALUE self) {
  VALUE envelope;
  VALUE polygons;
  VALUE kwargs;
  rb_scan_args(argc, argv, "1*:", &envelope, &polygons, &kwargs);

  // Actual method
}

The "1*:" gibberish means:

1: one required positional argument
*: any amount of positional arguments not required
:: keyword arguments at the end

Parse Keyword Arguments

Now we've constrained the method, unfortunately, we are not done yet. Our current API is def voronoi_diagram(envelope, *polygons, **kwargs).

Finally, we need to parse those keyword arguments using rb_get_kwargs.

int rb_get_kwargs(VALUE keyword_hash, const ID *table,
                  int required, int optional, VALUE *values);

You have to choose some required and optional arguments. Once that's done, you use table to tell Ruby the name of those arguments, and you store the result in an array (values).

VALUE voronoi_diagram(int argc, VALUE *argv, VALUE self) {
  VALUE envelope;
  VALUE polygons;
  VALUE tolerance;
  VALUE only_edges;

  VALUE kwargs;
  rb_scan_args(argc, argv, "1*:", &envelope, &polygons, &kwargs);

  ID table[2];
    table[0] = rb_intern("tolerance");
    table[1] = rb_intern("only_edges");
  VALUE *values;
  rb_get_kwargs(kwargs, table, 1, 1, values);

  tolerance = values[0];
  only_edges = values[1] == Qundef ? Qfalse : values[1];

  // Actual method
}

There you have it! A complex Ruby method, parsed using only C. However, if this is too convoluted for you, there is another option.

Using a Ruby Interface

Another way to handle the problem is actually to use Ruby's syntax directly and do the parsing at the Ruby stage.

def voronoi_diagram(envelope, *polygons, tolerance:, only_edges: false)
  c_voronoi_diagram(envelope, polygons, tolerance, only_edges)
end

With this, you can directly use the third form of rb_define_method, for a C method that looks like this:

VALUE c_voronoi_diagram(VALUE self, VALUE envelope,
                        VALUE polygons, VALUE tolerance,
                        VALUE only_edges) {
  // Actual method
}

And there you go — you completely avoid the problem with an elegant solution that is actually used for some methods in a Ruby implementation (with the Primitive class).

Although the class used by the MRI is quite complex and generates C itself, we can get inspired by it.

Let's create an object and plug our methods to avoid having a visible c_voronoi_diagram for users of our API:

void Init_ext() {
  VALUE primary_mod = rb_define_module("Hidden")
  rb_define_method(primary_mod, "voronoi_diagram", func, 4)
}

def voronoi_diagram(envelope, *polygons, tolerance:, only_edges: false)
  Hidden.voronoi_diagram(envelope, polygons, tolerance, only_edges)
end

Check out a real use case of this class in RGeo's codebase.

Parsing Arguments: Which Method Should I Use?

This repo showcases the two ways to set up a complex Ruby API that is written in C — to recap:

with rb_define_method and parsing it with rb_scan_args
using a Ruby interface

When we compare both solutions in terms of performance, they are roughly equal (Ruby parsing is 1.02 times faster on average on my M1). That doesn't make much of a difference.

In the RGeo lib, our first design choice was to have an API that only uses variadic length arguments. No keywords, no blocks. This can be very limiting, and we are now using the Primitive way to allow more convoluted arguments.

Benefits of Using a Ruby Interface

My advice is to use a Ruby interface for multiple reasons, including that:

the code size is smaller
changes are easier to make

Overall, this makes the experience of reading your codebase simpler.

Engaging Ruby users in reading the source code of the gems they use is important to me. As Ruby is easy to read, gems should be as well.

Benefits of Using `rb_define_method`

The C version, however, gives us a taste of some very useful internal methods. For instance, rb_check_arity is still really useful. Methods for handling blocks are great as well, and you might not need to use the Ruby facade for blocks.

It's all about finding the best choice for your use case.

Wrapping Up

In this post, we explored two methods to parse arguments in your Ruby C extension — using rb_define_method (also looking briefly at its limitations) and parsing it with rb_scan_args and using a Ruby interface.

If you want to read more about C extensions, I recommend:

And my final word of advice for you: check out RGeo. It is a widely used C extension codebase in active development. Most of my examples come from here.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Create a Business Language for a Rails Application

Roy Tomeij — 2023-01-11T00:00:00+00:00

As web developers, we tend to approach problems with traditional low-risk solutions. When all you have is a hammer, everything looks like a nail. When you need complex input from the user, you use a form and JSON representation (even if, in retrospect, it is not the most efficient solution).

In this post, we'll take a different approach. We'll leverage some tooling to create a business language that extends the functionality of a Rails application.

Let's get started!

Background

A few years ago, my team implemented a feature that enabled users to input a set of complex conditions. Our system presented results in accordance with these conditions.

We took the conservative road and implemented a state-of-the-art form with multiple input widgets, drag and drop, and all the UI sugar. Then the form state was serialized into a JSON object and sent to the backend, where it was unpacked, conditions applied, and results sent back.

It worked, but not without multiple problems:

The form was difficult to maintain, and bugs kept creeping in.
It was also very complex; most users could only use 10% of its capabilities.
Any change to JSON representation had to be implemented in two places: in the form frontend and the backend.

I will now discuss a possible alternative approach that can solve at least some of the problems outlined above and that we had not considered at the time.

A Problem to Solve in a Rails App

First, let me describe the problem we will solve in more detail. This is a very simplified version of the actual requirement I talked about above.

We'll build a backend for a promo mobile app, with a 'Coupons' section. I'm sure you are familiar with this concept from some real-world mobile applications as well.

At any given moment, you'll usually have a small number of coupons available — let's say, 10 to 30. This number is too large to fit on the screen, so to increase the conversion (usage) rate, it is important to personalize the order of the coupons. The ones most likely to pique a user's interest should go first, based on data about the available user.

So, when adding a new coupon to the system, the operator fills in the 'targeting info', i.e., the cohort this should interest. This might be based on science, intuition, or stereotypes. It can be very simplistic ("women aged 18–35") or quite complex ("women aged 50+ interested in health, or men interested in sports or the outdoors, or people with children aged 10+"). In fact, any combination of data assertions we have should be possible with any logical operators.

Using the aforementioned JSON representation, we could represent a complex condition with something like this:

{
  "operator": "or",
  "conditions": [
    {
      "operator": "and",
      "conditions": [
        { "property": "gender", "operator": "in", "value": ["woman"] },
        { "property": "age", "operator": "gt", "value": 18 },
        { "property": "age", "operator": "lt", "value": 35 }
      ]
    },
    {
      "operator": "and",
      "conditions": [
        { "property": "gender", "operator": "in", "value": ["man"] },
        {
          "property": "interests",
          "operator": "intersect",
          "value": ["sports", "outdoors"]
        }
      ]
    },
    {
      "property": "child_birth_year",
      "operator": "between",
      "value": [2004, 2012]
    }
  ]
}

Even though this is legible, it is hard to follow. And the form supporting it is tricky to understand as well (actually, we haven't even come up with a satisfactory solution for a form that mixes "and" and "or" logical operators).

A Solution: Design a New Business Language

An alternative solution to this problem is to get rid of the form and the JSON representation. Instead, we'll design a whole new, specific business language to apply here and then implement it in a Rails application. As a result, our condition will look like this:

(gender(woman) and age > 18 and age < 35) or (gender(man) and (interest(sports) or interest(outdoors))) or has_child_aged(10, 18)

As this is much terser, it is easier to understand and debug. It could be really hard for a regular user picked from the depths of the Internet to use. However, in cases like this, the actual users can be trained with access to fast customer support.

We could call our business language a domain-specific language (DSL) because this is more or less the term's original meaning. However, in recent years, especially in the Ruby community, the meaning of DSL has somewhat changed. When we talk about DSLs, we usually mean bending the Ruby language to look like something else while it is still actually Ruby. Think about RSpec:

describe "a subject" do
  let(:number) { 15 }
  it { expect(NumberDoubler.call(number)).to eq(30) }
end

Even though we introduce specific words defined as Ruby methods (describe, let, it), this is still Ruby.

It is important to understand that the business language example above is not Ruby. Users won't be able to access the filesystem, the network, or make infinite loops. All the language constructs only allow talking about users and their properties or combining expressions with logical operators.

It is a completely new language, which needs to have a grammar, parser, etc. This is what we are going to do next.

Implement a Business Language in Your Rails App with Parslet

To achieve our goals, we will use a gem called Parslet, for:

Constructing parsers in the PEG (Parsing Expression Grammar) fashion.

Source: Parslet website

PEG is relatively simple (compared to other techniques) and quite fast, especially for small input.

This post is not going to be a step-by-step Parslet tutorial. The official Parslet documentation does a great job of going through the process. But let's briefly go over the building blocks of language interpreting with Parslet, which consists of three steps.

In step one, we define a parser:

class CouponLang::Parser < Parslet::Parser
  rule(:lparen)     { str('(') >> space? }
  rule(:rparen)     { str(')') >> space? }
  # [...]
end

tree = CouponLang::Parser.new.parse("age > 18 or age < 30")

The result of the parse method is an intermediary tree, which is a hash-like structure representing language tokens.

This tree is fed to a transform:

ConjunctionExpr = Struct.new(:left, :right) do
  def eval(user)
    left.eval(user) && right.eval(user)
  end
end

# [...]

class CouponLang::Transform < Parlet::Transform
  rule(:and => [subtree(:left), subtree(:right)]) { ConjunctionExpr.new(left, right) }
  # [...]
end

tree = CouponLang::Parser.new.parse("age > 18 or age < 30")
ast = CouponLang::Transform.new.apply(tree)

The result of the transform is an abstract syntax tree (AST). The final step is to evaluate the AST, passing a user as a context:

user = User.find(params[:user_id])
coupons = Coupon.active(Time.now)
proritized, others = coupons.partition do |coupon|
  tree = CouponLang::Parser.new.parse(coupon.priority_condition)
  ast = CouponLang::Transform.new.apply(tree)
  ast.eval(user)
end

The last code listing shows how this can be used in a wider context, relevant to our original requirements.

However, there is one important limitation to mention. As you can see, all active coupons are fetched upfront, and we apply the conditions on them one by one. It is safe, because, as stated before, only a handful of coupons are active at the time. However, this makes it impossible to answer the question: "What users would coupon X prioritize?"

If you really need to answer this kind of question, you have to fetch all the users and apply the conditions to them, user by user, in the application layer.

Of course, there are more efficient solutions.

For example, you can write another transform which, instead of evaluating the conditions, translates them into an SQL query which you can then execute.

A Complete Example

I'm not going to lie to you: getting the code parser for the language right might be a tedious task. I spent a few hours getting the example for this article working. For brevity, I show an example that only implements the age and has_children functions for the user params.

This is the parser:

module CouponLang
  class Parser < Parslet::Parser
    rule(:lparen) { str("(") >> space? }
    rule(:rparen) { str(")") >> space? }

    rule(:space) { match('\s').repeat(1) }
    rule(:space?) { space.maybe }
    rule(:sep) { space | any.absent? }

    rule(:or_) { str("or") >> space }
    rule(:and_) { str("and") >> space }

    rule(:gt) { str(">").as(:gt) >> space? }
    rule(:lt) { str("<").as(:lt) >> space? }
    rule(:comparison_op) { gt | lt }
    rule(:comparison) { int.as(:left) >> comparison_op >> int.as(:right) }

    rule(:integer) { match("[0-9]").repeat(1).as(:int) >> space? }
    rule(:string) { match["a-z"].repeat(1).as(:string) >> space? }

    rule(:age_fun) { str("age").as(:age_fun) >> space? }
    rule(:has_children_fun) { str("has_children").as(:has_children_fun) >> sep }

    rule(:int) { age_fun | integer }
    rule(:bool) { comparison | has_children_fun | bool_in_parens }
    rule(:bool_in_parens) { lparen >> expr >> rparen }

    rule(:expr) { infix_expression(bool, [or_, 1, :left], [and_, 1, :left]) { |left, op, right| {op.to_s.strip.to_sym => [left, right]} } }
    root(:expr)
  end
end

And this is the transform:

module CouponLang
  class Transform < Parslet::Transform
    UserAgeFun = Class.new do
      def eval(user) = user.age
    end

    HasChildrenFun = Class.new do
      def eval(user) = user.has_children?
    end

    IntLit = Struct.new(:int) do
      def eval(user) = int.to_i
    end

    InfixOp = Struct.new(:left, :op, :right) do
      def eval(user) = left.eval(user).public_send(op, right.eval(user))
    end

    LogicalOr = Struct.new(:left, :right) do
      def eval(user) = left.eval(user) || right.eval(user)
    end

    LogicalAnd = Struct.new(:left, :right) do
      def eval(user) = left.eval(user) && right.eval(user)
    end

    rule(:age_fun => simple(:_)) { UserAgeFun.new }
    rule(:has_children_fun => simple(:_)) { HasChildrenFun.new }
    rule(:or => [subtree(:left), subtree(:right)]) { LogicalOr.new(left, right) }
    rule(:and => [subtree(:left), subtree(:right)]) { LogicalAnd.new(left, right) }
    rule(:left => subtree(:left), :gt => simple(:_), :right => subtree(:right)) { InfixOp.new(left, :>, right) }
    rule(:left => subtree(:left), :lt => simple(:_), :right => subtree(:right)) { InfixOp.new(left, :<, right) }
    rule(:int => simple(:int)) { IntLit.new(int) }
  end
end

It's important to thoroughly test the language you have created. Otherwise, some unpleasant surprises will be waiting for you when you want to change it in the future. An example of a spec for the parser looks like this:

RSpec.describe CouponLang::Parser do
  it "parses expression with parentheses" do
    expect(described_class.new.parse_with_debug("has_children and (age > 12)")).to eq(and: [{has_children_fun: "has_children"}, {left: {age_fun: "age"}, gt: ">", right: {int: "12"}}])
  end
end

And for the transform:

def parse_and_eval(code, user)
  tree = CouponLang::Parser.new.parse(code)
  ast = CouponLang::Transform.new.apply(tree)
  ast.eval(user)
end

RSpec.describe CouponLang::Transform do
  it "evaluates conditions with parentheses" do
    user_with_children = User.new(birth_year: 1980, child_birth_years: [2008, 2012])
    user = User.new(birth_year: 1981)
    code = "age > 65 or (has_children and age > 40)"

    expect(parse_and_eval(code, user_with_children)).to eq(true)
    expect(parse_and_eval(code, user)).to eq(false)
  end
end

Of course, for a full Rails integration, you must also validate whether a user puts a valid CouponLang code in a new coupon form. Parslet returns nil when it cannot parse the text, so it is as simple as this:

class Coupon < ApplicationRecord
  validate :correct_code

  def correct_code
    CouponLang::Parser.new.parse(code).present?
  end
end

Extra Parsing Tips

We are ready to ship the CouponLang to our internal users, with great flexibility for defining conditions. The road to arrive here has not exactly been without bumps, so here are a few additional tips:

Start Small

Writing a parser is difficult. Start with the simplest possible language, make a parser for it, write tests, commit, and add another feature.

For the CouponLang, I first started with a language that could only evaluate logical conditions with true and false values (like false and true). Then I added parentheses — (true or false) and true. Only after this was I ready to replace true/false literals with comparisons and functions.

Test Subparsers

I haven't shown it, but Parslet allows you to test only a subset of parsers. With our language above, you could use CouponLang::Parser.new.comparison.parse("11 > 12") to just check if the comparison part is fine. This is very useful for testing and debugging.

Check the Tree is Transform-friendly

Unless you are quite experienced in writing PEGs with Parslet, you may end up with a parser that works, but a tree that's not transform-friendly. As a result, you will have to change it. Be prepared for that.

Be Careful with Backward Compatibility

When changing the parser breaks backward compatibility, chances are that the coupons already saved in the database will stop working. You should consider running every coupon from the production database against the new parser to check whether they will work after the change. It's a one-time operation, but important to save a lot of headaches.

When You Should Consider Using a Parser

You may think that my coupon example is quite unusual, even though it comes from a real-world problem. It's true that what I show here is not for everyday Rails development. You need a good reason to implement such a powerful (and not very user-friendly) tool as Parslet.

And the reason is simple — it is still easier than trying to do it any other way.

A few examples of when a parser might be worth considering include:

For very fine access control to some resources - e.g., when you need to make a document accessible only for "all managers, members of team alpha or beta that have worked here for at least 3 months, and Jason from HR".
When modeling game-like conditions - Want to wield the Epic Battleaxe of Doom? Sure, you just need to be at least level 32, have at least 180 strength, and have finished the quest "Waiting for Ragnarok" or "The Abyss of Destiny".

Remember that you are not limited to languages that return a boolean as a result. You can, for example, create a language that evaluates numbers and thus gives multiple rankings for users. Or restaurants. Or dogs.

If you are brave enough, you could also build a language that behaves much more like a "real" programming language and executes something with variables and conditionals. For example, if you build a system that reacts to certain events (like a notification that a database is down):

if(notification.severity <= 2) {
  user = sample_from_groups('sre)
  if(user.has_phone_number and notification.severity == 1) {
    send_sms_to(user)
  } else {
    send_email_to(user)
  }

  send_slack_notification('alerts, notification.payload)
}

This, however, is much more complicated than the example I showed in this article. You might want to check out this attempt to parse Java source code with Parslet.

Wrapping Up

In this post, we saw how you can leverage tooling to build a programming language that extends your Rails application's functionality. Even if you don't need it in your current project, it's worth knowing that it is possible without too much effort.

We also explored when it's worthwhile to consider using this approach.

If this piqued your curiosity about creating languages, I hope you will have a lot of fun experimenting.

Happy parsing (and interpreting)!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

AppSignal’s Top 5 Ruby posts in 2022

Roy Tomeij — 2022-12-21T00:00:00+00:00

With the holidays approaching, the AppSignal team is taking a mini winter break from publishing Ruby posts. We hope to bring you a lot more of them in 2023. 🍵 ❄️

In the meantime, let's look at the top 5 most liked Ruby blog posts from 2022!

State Machines in Ruby: An Introduction

A state machine can hold all possible states of something and the allowed transitions between these states. For example, the state machine for a door would have only two states (open and closed) and only two transitions (opening and closing). In this post, we'll look at how to set up a state machine in Ruby and use the state machines gem.

An Introduction to Ractors in Ruby

In this post, we'll dive into ractors in Ruby, exploring how to build a ractor. You'll send and receive messages in ractors, and learn about shareable and unshareable objects.

An Introduction to Polymorphism in Ruby on Rails

This article will give you a greater understanding of polymorphism, specifically in Ruby on Rails. To accomplish this, we’ll dive into:

The use of polymorphism in the real world
Polymorphism in programming by way of OOP
How you can incorporate it into your Rails application to help maintain high-quality code

Get Started with Hotwire in Your Ruby on Rails App

Hotwire is a completely new way of adding interactivity to your app with very few lines of code, and it works blazing fast by transmitting HTML over the wire. That means you can keep your hands clean from most Single Page Applications (SPA) frameworks. You can also keep your rendering logic centralized on the server, while still maintaining quick page load times and interactivity.

5 Tips to Design Ruby on Rails Transactions the Right Way

This article offers a set of good practices for working with transactions. The tips are pretty simple, but they will help make your transactions bulletproof, readable, and relatively safe.

Honorable Mention - Test and Optimize Your Ruby on Rails Database Performance

AFK Time!

We hope you've learned something new while reading our blog this year. We know we have! In 2023, our team will continue working hard to publish new blog posts to spread knowledge across the Ruby community and bring you the best content and resources to help you improve your skills and build performant Ruby applications.

Wherever you are in the world, we hope you enjoy a happy and healthy end of year. We hope that 2023 will be an amazing year for you, with no bugs and packed with events and adventures that are almost as sweet as a stroopwafel.

A Guide to Memoization in Ruby

Roy Tomeij — 2022-12-20T00:00:00+00:00

Memoization is a caching technique to make your Ruby application run more efficiently and faster.

In this post, we'll look at the benefits of memoization and when to use it in your Ruby application. We'll also look at some memoization mistakes to avoid.

Let's first start by looking at code optimization — what it is and some of the different optimization techniques available.

What Is Code Optimization?

Code optimization is the process of improving code quality to make a piece of code or program more efficient and functional. Its advantages include — but are not limited to:

Less memory consumption during an expensive calculation
Much faster execution
Sometimes, less space in terms of codebase size

The need to achieve some of the goals mentioned above arises at one time or another during an app's life cycle. If you're at a loss about where to begin with code optimization, try profiling!

What Is Profiling?

Profiling refers to analyzing a program to measure its space and time complexities. Via profiling, we can obtain information such as:

The frequency and duration of function calls
The percentage of program execution time spent in a function in comparison with other functions
The call stack of every function
How many database calls are made for an HTML page to successfully load and how long it takes

This information can guide us to where code optimization is highly required.

Code Optimization Methods for Ruby and Rails

A few Ruby and Rails code optimization techniques include:

Getting rid of N+1 queries - This can help improve the speed of an app. The Bullet or Prosopite gems can give a lending hand here. The N+1 Dilemma — Bullet or Prosopite? entails a brief comparison of both.
Using static code analyzers - These reduce memory consumption and codebase size, as we're alerted to code duplication, unused variables or method arguments, and the like. Examples include Rubocop and RubyCritic.
Caching - Stores the content generated during the request-response cycle and reuses it when responding to similar requests, to improve the speed of an application. In Rails, we have page caching, fragment caching, action caching, low-level caching, and many more.
Choosing appropriate data structures - Some data structures perform better in certain situations than others. As a result, a good way to optimize code is to use the most appropriate data structures for each situation, considering their space and time complexities.
Memoization - Improves speed by reducing the number of times certain computations are carried out.

Let's now turn our focus to memoization.

An Introduction to Memoization in Ruby

Memoization is the act of caching a method's result so that the next time that method is called, the previous result is returned (as opposed to carrying out a recomputation). This helps save time when running a program.

Let's look at an example involving anagrams.

In the class below, we create a dictionary to pass in any word as an argument, and find the anagram.

class Dictionary
  def words
    puts "creating my dictionary"
    words = File.readlines('/usr/share/dict/words')
    dictionary = Hash.new {|h,k| h[k] = []}
    words.each do |word|
      word = word.chomp
      dictionary[word.chars.sort.join("")] = word
    end
    dictionary
  end

  def check(word)
    words[word.chars.sort.join("")]
  end
end

Testing the class above, we obtain:

dictionary = Dictionary.new
dictionary.check('rasp')
# creating my dictionary
=> "spar"
dictionary.check('kame')
# creating my dictionary
=> "make"

We can see that every time we call the check method, we create the dictionary again. This is definitely not optimal, as the dictionary does not change.

What if we created the dictionary once and used it whenever it was needed? We can; using memoization. Memoization allows us to cache the dictionary if it has been previously created.

def words
  @words ||= begin
    puts "creating my dictionary"
    words = File.readlines('/usr/share/dict/words')
    dictionary = Hash.new {|h,k| h[k] = []}
    words.each do |word|
      word = word.chomp
      dictionary[word.chars.sort.join("")] = word
    end
    dictionary
  end
end

Testing the above, we obtain:

dict.check('eat')
#creating my dictionary
=> "tea"
dict.check('kame')
=> "make"
dict.check('live')
=> "vile"

As we can see, the dictionary is created once and the cached version is used after that.

Let's take a benchmark to see how much faster our program has become because of memoization. Name the memoized version memoized_check, and the non-memoized version check:

require "benchmark"

dictionary = Dictionary.new
puts Benchmark.measure { 10.times { dictionary.check('rasp') } }
puts Benchmark.measure { 10.times { dictionary.memoized_check('rasp') } }

We get the following result:

5.771061   0.044656   5.815717 (  5.836218)
0.563966   0.000016   0.563982 (  0.564909)

This shows us that the unmemoized version takes 5.83 seconds while the memoized one takes 0.56 seconds, making it about ten times faster.

Memoization Mistakes to Avoid in Your Ruby Application

Let's now look at some mistakes to avoid when using memoization.

Ignoring the False Or Nil Return

In cases where a method's computation returns a false or nil, each call leads to a recomputation whenever the method is called (even though memoized). This is because the comparison is made using an or — and in Ruby, both nil and false are false values.

2 || 4+5
=> 2
nil || 4+5
=> 9
false || 4+5
=> 9

Memoization using ||= doesn't consider false/nil return values, so such cases should be handled.

def do_computation
  puts "I am computing"
  nil
end

def check
  @check ||= do_computation
end

Calling the check method, we get the following results:

check
# I am computing
=> nil
check
# I am computing
=> nil

To deal with this, we can ascertain if the variable has been defined. If so, we return early before proceeding to the computation.

def check
  return @check if defined?(@check)
  @check ||= do_computation
end

This results in:

check
# I am computing
=> nil
check
=> nil

Passing Parameters to a Method

Another common mistake is assuming memoization will work differently when parameters are passed to a method.

Let's say that a method's result is memoized using ||=, but that result depends on parameters. If these parameters change, the result does not change.

def change_params(num)
  @params ||= num
end

Let's see what happens here:

change_params(4)
=> 4
change_params(8)
=> 4

The result does not change just because we changed the parameters, and frankly, this is what is expected, considering how memoization works in its basic form.

To deal with cases like this, you have to be comfortable with:

storing the parameters as keys in a hash
using a gem or module that takes all the different cases to be handled into consideration

Using a hash:

def change_params(num)
  @params_hash ||= {}
  if (@params_hash.has_key?(num))
    @params_hash[num]
  else
    puts 'creating a new key-value pair'
    @params_hash[num] = num
  end
end

Trying this out, we have:

change_params(4)
creating a new key-value pair
=> 4
change_params(8)
creating a new key-value pair
=> 8
change_params(4)
=> 4
change_params(8)
=> 8

Another way to rewrite this would be as the following:

def change_params(num)
  @params_hash ||= {}
  @params_hash[num] ||= num
end

Or you can use the gem Memoist. It handles caching the method's results, considering the arguments passed. It also provides a way to flush the current value or the entire memoization cache for an object.

When To Memoize — and When Not To

To decide when to memoize, take note of the following:

Expensive Operations

Let's say that an expensive operation will most definitely be called multiple times within a class and return the same result.

We can move this into an instance variable initialized upon object instantiation (the expensive operation is also done at this time).

attr_reader :result

def initialize
  @result = do_expensive_calculation
end

result is available throughout the lifecycle of the class instance, and the expensive calculation is only done once.

In cases like this, we don't need a separate method to memoize the do_expensive_calculation value.

An expensive calculation that might not happen — but if it does, might happen more than once (and return the same value) — is a good candidate for memoization. This means we only do_expensive_calculation when needed and then cache the result.

def expensive_calculation
  @expensive_calculation ||= do_expensive_calculation
end

Analysing Potential Performance Improvements

Memoization may be seen as necessary only after we've carried out a performance analysis. We need to accurately ascertain that the implemented memoization actually improves performance (as we did within the Dictionary class).

Without this, we could add unnecessary complexities to the code base. Be sure that the space and code complexities incurred from memoization are low compared to the gain in run-time.

Changing Parameters

If the parameters used for computation are constantly changing, memoization is not a good option.

Memoization is more suitable for pure functions whose return values are the same for the same set of arguments.

def calculation(a, b)
  a + b + Time.now.to_i
end

In the example above, let's say we do cache the method result. Every other time we call the method, our cached value is wrong because Time.now changes.

Wrapping Up

In this post, we explored how memoization, like every caching technique, comes with its advantages and disadvantages. We first looked at a range of code optimization techniques before diving into an example involving memoization. Then we touched on some mistakes to look out for. Finally, we explored when memoization is beneficial and when to avoid it.

When memoization is carried out on methods available to class instances, the memoized result is only available for the lifecycle of that object. Where this result is the same for multiple class instances (e.g., multiple web requests), class-level memoization is usually a go-to.

However, this might add more complexities in terms of cache invalidation. Using a cache store might be a better alternative to caching and allow for better optimization.

Before you decide to employ it, you must ascertain that the pros of memoization outweigh the cons for your specific use case.

Happy memoizing!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Database Performance Optimization and Scaling in Rails

Roy Tomeij — 2022-12-07T00:00:00+00:00

Web applications usually rely heavily on databases, for the most part. And as applications grow, databases grow too. We keep scaling web servers and background workers to keep up with the heavy load. But eventually, the database needs to keep up with all the new connections from these processes.

One way to tackle this is to grow a database with an app using vertical scaling. This means adding more CPU power and memory to the database server. But this is usually slow. You might have to copy all your data to the new server and then put the application down to change the database servers that it communicates with.

This is also usually a one-way operation. You can't keep adding/removing CPU power or memory from the database server based on your load.

This post will cover some alternative methods to fine-tune and scale a database under heavy load and improve the performance of your Rails app. We will focus on:

Splitting schema and data between multiple databases
Using read-only replica databases

Let's get going!

Disclaimer: Scaling Isn't Always Needed

As always, with any post about performance optimization and scaling, I would like to put up a standard disclaimer: make sure you have an issue before trying to solve it.

For example, on one of our apps in production, we tackled scaling and optimization only when we started processing more than 3 million background jobs a day.

This will, of course, be different for different apps. But usually, a good indicator of whether you need to optimize is if your database is always running at high CPU or memory usage.

A Separate Database for Your Rails Application

Sometimes, simply using a separate database server for a part of your app is a good solution. For example, your app might do two different things that don’t have a big overlap. Alternatively, maybe there's one very database-heavy feature, but it is only used rarely or by a small section of users. You should go for a separate database server if — and only if — there is a clear distinction between the parts of your app that will use different databases.

For example, a separate database is useful when you're audit logging in a high-frequency app (or have other very high-volume data that isn't necessarily accessed frequently). Let's see how we can set this up.

First, set up the database.yml to configure the second database. Let’s call the second database log. We'll add a log_default entry for the secondary database with a common configuration across all environments:

log_default: &log_default
  host: localhost
  port: <%= ENV["DB_PORT"] || 5432 %>
  adapter: postgresql
  encoding: unicode
  user: <%= ENV["DB_USER"] || "postgres" %>
  password: <%= ENV["DB_PASSWORD"] || "postgres" %>
  migrations_paths: db/log_migrate

The important bit here is the migrations_paths option that tells Rails where to find migrations related to this database.

Then, update database.yml to configure access to the log database for development, test, and production environments:

development:
  primary:
    <<: *default
    database: <%= ENV["DB_NAME"] %>
  log:
    <<: *log_default
    database: <%= ENV["LOG_DB_NAME"] %>

test:
  primary:
    <<: *default
    database: my_app_test
  log:
    <<: *log_default
    database: my_app_log_test

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  log:
    adapter: postgresql
    encoding: unicode
    migrations_paths: db/log_migrate
    url: <%= ENV["DATABASE_LOG_URL"] %>

As you can see, we move the configuration for each environment one level deeper. The configuration for our primary database lives inside the primary key for each environment. The second database's configuration (that we call log) lives inside the log key. primary is a special keyword that tells Rails to use this database as the default.

The next step is to set up ActiveRecord to use this new database. We first create a base class that establishes a connection to this database (instead of a primary one) using establish_connection:

class LogRecord < ActiveRecord::Base
  self.abstract_class = true
  establish_connection :log
end

And then, we inherit from LogRecord instead of ApplicationRecord for all the records that should access the log DB:

class AuditLog < LogRecord
end

Note that you keep using the rails generators for generating models and migrations by appending the --database log option to target the second database. The migration task now automatically migrates both databases, but if you need to only migrate one, you can use db:migrate:primary or db:migrate:log tasks.

This works great if there is a clear distinction between two parts of the app. But what if you don’t have a clear idea of the database that's creating issues? There are still a couple of options left. Let's check them out.

Using Read-Only Replicas for Rails

Our second scaling option is to access data from read-only replicas while still writing to the primary database. This can improve performance for users who only browse parts of the app without performing any writing operations. Depending on your app's use case, this can be 80% of your users or 10% of them. So, evaluate the behavior of your users or your application use case before going down this route.

For an app with a majority of read-only users (like Twitter, for example), you can gain huge improvements in performance. New read-only replicas can be added/removed at will without affecting the primary database, which opens up options for auto-scalability.

Let's see how to set this up.

Setting up a Read-Only Replica

As usual, we will start with modifications to the database.yml config to include the replica:

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  primary_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_REPLICA_URL"] %>

Again, the primary key is a special key indicating that this database is the default. We can use any other key for the replica database configuration. Let’s choose primary_replica for sanity.

Then update the ApplicationRecord to configure a connection to multiple databases:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  connects_to database: { writing: :primary, reading: :primary_replica } if Rails.env.production?
end

Note that we add configuration for a production environment only in the above example. If you want to, you can set it up for all environments, after modifying your development setup to support replicas.

Finally, to use the read-only replica, you need to tell Rails when to use the primary database and when to use the replica. Rails provides a basic implementation for automatic role-switching based on the HTTP verb of the request out of the box. To enable it, add the following to your production.rb:

# Use Read-Only Databases on GET requests
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = ActiveRecord::Middleware::DatabaseSelector::Resolver
config.active_record.database_resolver_context = ActiveRecord::Middleware::DatabaseSelector::Resolver::Session

This tells Rails to use the replica for all GET or HEAD requests and the primary one otherwise. Notice the option that sets the delay to two seconds? That tells Rails to keep using the primary database for GET and HEAD requests if it is within two seconds of a write request in the same session.

Gotchas

Does all of this sound too simple? Well, we aren’t done yet. Depending on how your application is structured, you need to keep a few points in mind.

Firstly, ensure you do not write anything in a read-only action like index or show. If there are legitimate reasons for writing during those operations, you can manually switch the connection:

ActiveRecord::Base.connected_to(role: :writing) do
  # Your code here
end

A good example where this can be useful is if you are managing sessions in a database. If you do that, make sure you use the writing role. For example, you can override the save_record method in your session with authlogic:

class UserSession < Authlogic::Session::Base
  # Your session configuration

  def save_record(alternate_record = nil)
    ActiveRecord::Base.connected_to(role: :writing) do
      super
    end
  end
end

Another use case for automatic role-switching is when an app exposes a GraphQL API. All operations on a GraphQL API are POST, but conventionally, all queries are read-only, and mutations are read-write. In this case, to allow Rails to select the correct database, we can use a custom tracer in the schema.

First, create a tracer that switches roles based on the GraphQL document:

module Tracers
  class DatabaseRoleTracer
    EVENT_NAME = "execute_multiplex"

    # Execute on read only database if the operation has only queries.
    def trace(event, data)
      return yield unless Rails.env.production?
      return yield unless event == EVENT_NAME

      multiplex = data[:multiplex]
      ActiveRecord::Base.connected_to(role: role(multiplex)) do
        yield
      end
    end

    private

    def role(multiplex)
      if multiplex.queries.all?(&:query?)
        ActiveRecord::Base.reading_role
      else
        ActiveRecord::Base.writing_role
      end
    end
  end
end

And then use the tracer in your GraphQL Schema:

class MyAppSchema < GraphQL::Schema
  tracer Tracers::DatabaseRoleTracer.new

  # Your code here
end

Let’s get to the final alternative. What can you do if your application still has a huge load on your primary database? The answer: database sharding.

Database Sharding in Your Rails Application

This is the most complex of all the methods for scaling databases discussed in this post. So reach for it only if you need it. It will add considerable complexity to your application and needs to be done right to provide any real benefit.

There are two strategies to shard your database:

Distribute different tables on different nodes (vertical sharding).
Have the same schema across all nodes, but distribute data depending on certain parameters (horizontal sharding).

Vertical sharding is similar to our “multiple databases” strategy in terms of setup and pros and cons, so we will not discuss it again.

Let's discuss horizontal sharding and where it could be helpful.

One example is when you have a SaaS platform where a lot of data is associated with each user, but there is no overlap between the data of two users. In this case, splitting the data across nodes based on the user's id is the most logical way to go. Every signed-in user will have to access only a single database, so we won't have to reach out to multiple databases at the same time.

Setting up Horizontal Sharding

Let’s start with the configuration inside database.yml again:

production:
  primary:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_URL"] %>
  primary_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_REPLICA_URL"] %>
  primary_shard_one:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_SHARD_ONE_URL"] %>
    primary_shard_one_replica:
    adapter: postgresql
    encoding: unicode
    url: <%= ENV["DATABASE_SHARD_ONE_REPLICA_URL"] %>

Then modify the ApplicationRecord to connect to different shards:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  connects_to shards: {
    default: { writing: :primary, reading: :primary_replica },
    shard_one: { writing: :primary_shard_one, reading: :primary_shard_one_replica }
  }
end

Now we can use the shard option with ActiveRecord::Base.connected_to to switch shards. Like the automatic database role selector, Rails also provides an automatic shard switcher that can be activated inside production.rb like this:

Rails.application.configure do
  config.active_record.shard_selector = { lock: true }
  config.active_record.shard_resolver = ->(request) { Tenant.find_by!(host: request.host).shard }
end

The shard_resolver lambda is the most interesting part. The above implementation relies on the assumption that our application is accessed from different domains/subdomains and distinguishes between the shards. Modify it to fit your application needs (you might need to access cookies to identify a user before switching the shard).

As with vertical scaling, this strategy is usually a one-way street. Once you shard a database, it is very hard to “un-shard” it (since several databases could have the same ids for different objects). But this lays down the foundation to really scale your app to millions or billions of users when fitting everything on the same server is not an option.

If you want to read more about database sharding, see this write-up by Linode.

Measure Your Rails Database Performance with AppSignal

It can be tricky to keep an eye on the performance of your database without any other tools. Using AppSignal, you can easily track how your databases perform. See our AppSignal for Ruby page for more information.

You can also check out our post Test and Optimize Your Ruby on Rails Database Performance for more information on the metrics you can measure in AppSignal and how a dashboard might look.

Wrap Up

In this post, we discussed three major strategies to scale your database servers horizontally:

Splitting schema between multiple databases
Using read-only replica databases
Splitting data between multiple databases

All strategies have their pros and cons depending on your application use case and scale.

Rails has really upped its database game in its last few releases. This post has shown how easy it is to set up a Rails app to use multiple databases.

As always, if your application allows it, I suggest starting small with optimizations. The easiest optimization to start with is a read-only replica, then only move on when you need more scale.

Happy scaling!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A First Look at Hanami 2 for Ruby

Roy Tomeij — 2022-12-06T00:00:00+00:00

As of today (06/12/2022), Hanami 2.0.1 has been released. Read more about the enhancements, bug fixes and gems in release 2.0.1.

Hanami 2 was released on 22 November, concluding four years of work on this version. It brings a breath of fresh air into Ruby's web development community. Version 2.0 is not just an incremental upgrade. One could say it's a project written anew, with bright ideas from version one rebuilt on top of a solid dry-rb libraries ecosystem.

Since there is no clear upgrade path from version 1.3, in this article, we will focus on some key aspects of the release rather than comparing the two versions.

We will take a close look at the following:

Slices
Dependency management
Performance

Let's get started!

Slices in Hanami 2

Hanami's go-to solution for setting boundaries between different contexts of a growing application is slices. By providing separate namespaces, slices clarify which code belongs to which area and when the boundaries are crossed.

Note that, as useful as they are, slices are also completely optional. If your app is simple or you don't yet know what slices will emerge, you can just put your code under a "main" slice in the app directory.

A new slice can be created within an existing Hanami application using a CLI generator:

> hanami generate slice accounts
Updated config/routes.rb
Created slices/accounts/
Created slices/accounts/action.rb
Created slices/accounts/actions/.keep

This produces a very bare-bones structure under the slices subdirectory, with an Accounts::Action superclass and a directory where account management actions will be created. These classes will, by convention, inherit from Accounts::Action. By doing so, they gain behaviors specific to the application's user account area.

You may wonder what "actions" are, exactly. They are Hanami's take on organizing web endpoints. In Rails, a controller class groups a few actions (represented by public methods). However, in Hanami, you create one class per action. So an action is a standalone unit of code that can be tested in isolation and quite clearly states what it does.

Let's have a look at an example action class:

class Accounts::Actions::UpdatePassword < Accounts::Action
  include Accounts::Deps["commands.change_password"]
  before :require_authenticated

  params do
    required(:current_password).filled(:str?)
    required(:password).filled(:str?, min_size?: 8)
  end

  def handle(req, res)
    if !req.params.valid?
      res.status = 422
      res.body = req.params.errors.to_h.join(", ")
    elsif change_password.call(
        req.session[:current_user],
        req.params[:current_password],
        req.params[:password])
      res.status = 201
      res.body = "password updated"
    else
      res.status = 422
      res.body = "cannot update password"
    end
  end
end

Here we have all the information in one place:

The action depends on a change_password command
It requires an authenticated user context
It needs two parameters to be passed (and one of them must be at least 8 characters long)
Finally, we have a handle method that holds the core logic of the endpoint: checking the validity of the params, calling a dependency, and setting an appropriate response

You might be wondering what this syntax means:

include Accounts::Deps["commands.change_password"]

This brings us to our next section about dependencies.

Dependencies

Accounts::Deps is a dependency mixin for the accounts slice. Hanami makes a dependency between different classes a first-class citizen.

Instead of having dependencies hidden in the code, you can (and should, if you want to follow Hanami philosophy) define them on top of the class. Not only does it clarify what you rely on, but it also makes the class easier to test.

In the example above, the "change password" command is defined in slices/accounts/commands/change_password.rb as an Accounts::Commands::ChangePassword class. The dependency mixin will make an instance of it available as change_password in the class where you use it. This is all based on dry-system.

In tests, you can stub the dependency easily using a built-in stub method:

Accounts::Container.stub("commands.change_password", FakePasswordChanger.new)

If, for example, your original command uses a slow-by-design hashing algorithm (such as bcrypt), you can swap it with something faster in tests.

Great Performance in Hanami

Speaking of performance, Hanami is fast. It boots really fast because providers are only booted when needed, not eagerly at the application start. As a result, a hanami-reloader gem (used to reload an application after changes are made to the code in development) doesn't need a sophisticated and fragile reloading mechanism. Instead, it just reboots the whole app in milliseconds. Simple yet effective.

But there's more on this front. Hanami 2.0 includes a brand-new router, which makes it faster than Sinatra, Grape, and Rails in benchmarks. That might not be the most important indicator for a database-heavy web application, but it's worth noting.

Hanami actions themselves can be very speedy as well. As an anecdote, a few days before the final release, Peter Solnica from the core team tweaked a logger to display the time it takes to process a request in microseconds because he was getting sub-millisecond times.

What Isn't in Hanami 2 (Yet)?

Before you start to rewrite your main application in Hanami, you should keep a few things in mind.

Version 2.0 is a bit stripped-down. As of today, the framework does not have a default view and database layer. However, people report that adding the former is easy with the hanami-view gem, which is almost ready. The persistence layer can be implemented using ROM.rb.

Official support for the missing pieces is planned in the Hanami 2.1 release.

My Personal Experience with Hanami 2

I have had a test Hanami application since the beta 1 release. It is a plain old server-rendered discussion forum - think something like phpBB. The app checks how much effort it takes to add common features to a Hanami application, such as user authentication, file uploads, etc.

My experience so far has been positive. Pre-2.0, I reported some issues to the maintainers. They were super helpful and the issues were quickly fixed.

In general, even though the Hanami ecosystem lacks any "plug-and-play" solutions such as Devise, you can use many existing libraries not tightly coupled to Ruby on Rails. For authentication, you can use Warden, OmniAuth or Rodauth. For uploads there is Shrine. The pagination is built into ROM. Integration with exception catchers such as Rollbar is easy.

So far, I haven't hit any obstacles that would block me from progressing.

Wrapping Up

I am excited about the future and waiting to see what Hanami 2.1 will bring in terms of views and persistence.

The Ruby community hasn't really had more than one feature-complete full-stack web framework since Merb, so it is really refreshing to see a change in that area.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

System Notifications with Noticed and CableReady in Rails

Roy Tomeij — 2022-11-23T00:00:00+00:00

This article has been partly inspired by an upcoming chapter of the author's Advanced CableReady book, and tailored to fit this guest post for AppSignal.

Notifications are a typical cross-cutting concern shared by many web applications.

The Noticed gem makes developing notifications fantastically easy by providing a database-backed model and pluggable delivery methods for your Ruby on Rails application. It comes with built-in support for mailers, websockets, and a couple of other delivery methods.

We'll also examine the merits of using the CableReady gem for triggering system notifications in your Ruby on Rails application.

Let's get into it!

Prerequisites and Requirements

Every now and then, you might need to trigger system notifications from your app. You can achieve this with the Notifications API.

For example, let's say your application allows users to upload large files which take a long time to be transcoded. You may want to send users a notification once their video upload completes. This means they can switch to a new task in the meantime and don't have to keep your application open for several minutes.

Luckily, it's easy to implement and flesh out system notifications with these two prerequisites:

Noticed has support for custom delivery methods (Noticed exposes a simple API to implement any transport mechanism you like — for example, posting to Discord servers).
CableReady has a notification operation arrow in its quiver.

The list of requirements for CableReady and Noticed is short — you simply need the following:

an ActionCable server running
an ActiveJob backend (typically used by Noticed)

Note: The entire sample code for this project is available on GitHub. You can also follow along below — we will guide you step-by-step.

A CableReady Primer for your Ruby on Rails Application

Let's start by making ourselves familiar with CableReady. Released in 2017, it can be thought of as the "missing ActionCable standard library".

Before the advent of Turbo, the only way to craft real-time applications in Ruby on Rails was through ActionCable. ActionCable is the native Rails wrapper around WebSockets, providing both a server-side and client-side API to send messages through a persistent connection in both directions at any time.

The downside to this approach was (and is) that you have to write a lot of boilerplate code to make it happen.

This is where CableReady helps by providing an abstraction layer around a multitude of DOM operations to be triggered on the server. A few examples include:

DOM mutations (inner_html, insert_adjacent_html, morph, etc.)
DOM element property mutations (add_css_class, remove_css_class, set_dataset_property, etc.)
dispatching arbitrary DOM events
browser history manipulations
notifications (which we will make use of)

The CableReady Server and Client

How does CableReady work its magic? In a nutshell, it consists of a server-side and client-side part.

On the server side, the module CableReady::Broadcaster can be included in any part of your application that calls for it. This could be in a job, a model callback, or a plain controller. But first, an ActionCable channel has to be in place. To cite CableReady's official documentation:

class ExampleChannel < ApplicationCable::Channel
  def subscribed
    stream_from "visitors"
  end
end

Note that visitors is called a stream identifier. It can be used to target either a broad audience or only specific clients subscribed to your channel. To conclude the example, we can include the broadcaster module in a model and send a console.log to the client after sign-up:

class User < ApplicationRecord
  include CableReady::Broadcaster

  after_create do
    cable_ready["visitors"].console_log(message: "Welcome #{self.name} to the site!")
    cable_ready.broadcast # send queued console_log operation to all ExampleChannel subscribers
  end
end

On the client side, the logic is simple. Create a subscription to the aforementioned channel. Then, in its received hook, call CableReady.perform on all operations passed over the wire:

import CableReady from "cable_ready";
import consumer from "./consumer";

consumer.subscriptions.create("ExampleChannel", {
  received(data) {
    if (data.cableReady) CableReady.perform(data.operations);
  },
});

CableReady vs. Turbo for Rails

Summing up, when should you use CableReady, and when should you avoid it?

With the introduction of Turbo, the web development community received a powerful toolbox to craft server-rendered reactive applications. Being essentially a frontend technology with powerful server-side bindings for Rails, it fits well into the standard Model-View-Controller (MVC) stack. Thus, it can cover most of your typical app's requirements.

CableReady, on the other hand, is the Swiss army knife of real-time Rails development and should be used with care. It is a powerful abstraction that can seem very inviting to use pervasively. But if you imagine that every part of your DOM can be mutated from any location in your app, you'll understand that this can lead to race conditions and hard-to-track-down bugs.

There are cases like the one at hand, though, where CableReady makes perfect sense because it allows for more fine-grained control over the DOM.

Asked for a simple TLDR, I would respond that Turbo is for application developers, while CableReady is for library builders. But as we will see, there are gray areas between the two.

Noticed — Simple Notifications for Ruby on Rails Applications

The second library we will apply to deliver system notifications is Chris Oliver's Noticed gem. At its heart, it is built upon an ActiveRecord model that models a single notification to a recipient. It holds common metadata, such as:

who a notification was sent to (the recipient)
when the notification was read
any parameters the notification is associated with (typically a reference to another model)

If you are familiar with how the ActiveStorage/ActionText meta-tables work, this is very similar.

Adjacent to this, Noticed employs POROs (Plain Old Ruby Objects, i.e., objects without any connection to Rails or other frameworks), which serve as blueprints for actual notifications. These are, somewhat misleadingly, also called Notifications and carry logic about how to render and distribute them. Here is an example from the README:

class CommentNotification < Noticed::Base
  deliver_by :database
  deliver_by :action_cable
  deliver_by :email, mailer: 'CommentMailer', if: :email_notifications?

  # I18n helpers
  def message
    t(".message")
  end

  # URL helpers are accessible in notifications
  # Don't forget to set your default_url_options so Rails knows how to generate urls
  def url
    post_path(params[:post])
  end

  def email_notifications?
    !!recipient.preferences[:email]
  end
end

We will see this at work shortly. Of special interest are the deliver_by invocations, as they determine which delivery methods this notification should use:

deliver_by :database stores a Notification record (of the model mentioned above) for later access
deliver_by :action_cable sends it via a defined ActionCable channel and stream (default Noticed::NotificationChannel)
deliver_by :email specifies a mailer used to send the notification. The example displays how to factor in any preferences the recipient(s) might have set.

Our goal for the remainder of this article is to implement a custom delivery method that will send our system notifications.

A Custom Delivery Method: System Notifications via the Notifications API

Before we set out to do this, let's clear the ground by creating a new Rails application. Because integrating with CableReady is easier this way, I opted for the esbuild JavaScript option over importmaps:

$ rails new noticed-cableready --javascript=esbuild

Note: At the time of writing, the current Rails version is 7.0.4. If you have an existing Rails application, you can skip the next step, but make sure you have a User model or similar concept to act as a notification recipient.

1. Prepare Recipients

Noticed needs a User model to act as recipients, so to be concise, pull in Devise and generate a User model.

Afterward, open the Rails console and create a sample user:

$ bundle add devise
$ bin/rails generate devise:install
$ bin/rails generate devise User
$ bin/rails db:migrate

$ bin/rails c
irb(main):001:1* User.create(
irb(main):002:1*   email: "julian@example.com",
irb(main):003:1*   password: "mypassword",
irb(main):004:1*   password_confirmation: "mypassword"
irb(main):005:1> )

2. Add Noticed

Next, let's add Noticed to our bundle and generate the database model.

$ bundle add noticed
$ bin/rails generate noticed:model
    generate  model
       rails  generate model Notification recipient:references{polymorphic} type params:json read_at:datetime:index
      invoke  active_record
      create    db/migrate/20221026184101_create_notifications.rb
      create    app/models/notification.rb
      invoke    test_unit
      create      test/models/notification_test.rb
      create      test/fixtures/notifications.yml
      insert  app/models/notification.rb
      insert  db/migrate/20221026184101_create_notifications.rb

🚚 Your notifications database model has been generated!

Next steps:
1. Run "rails db:migrate"
2. Add "has_many :notifications, as: :recipient, dependent: :destroy" to your User model(s).
3. Generate notifications with "rails g noticed:notification"

We do as told, run db:migrate and add the polymorphic has_many association to our User model:

# app/models/user.rb
class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  has_many :notifications, as: :recipient, dependent: :destroy
end

The final piece before we can put this to the test is to create a blueprint PORO:

$ bin/rails generate noticed:notification TestNotification

# app/notifications/test_notification.rb
class TestNotification < Noticed::Base
  deliver_by :database

  def message
    "A system notification"
  end
end

We tweak it a bit so that it just uses the database delivery method and a placeholder message for the moment. You would typically add required params here, like a model id, to construct the message and the URL to link to — see the Noticed README for details.

Using only the Rails console, we can demonstrate how to deliver a notification based on this PORO now:

$ bin/rails c
irb(main):001:0> TestNotification.deliver(User.first)
  User Load (0.3ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT ?  [["LIMIT", 1]]
Performing Noticed::DeliveryMethods::Database (Job ID: 32f27ac7-fb2e-42e4-9c5e-a3290d2d1297) from Async(default) enqueued at  with arguments: {:notification_class=>"TestNotification", :options=>{}, :params=>{}, :recipient=>#<GlobalID:0x00000001111e1718 @uri=#<URI::GID gid://noticed-cableready/User/1>>, :record=>nil}
  TRANSACTION (0.1ms)  begin transaction
  Notification Create (0.6ms)  INSERT INTO "notifications" ("recipient_type", "recipient_id", "type", "params", "read_at", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?)  [["recipient_type", "User"], ["recipient_id", 1], ["type", "TestNotification"], ["params", "{\"_aj_symbol_keys\":[]}"], ["read_at", nil], ["created_at", "2022-10-27 07:25:28.865982"], ["updated_at", "2022-10-27 07:25:28.865982"]]
  TRANSACTION (0.3ms)  commit transaction
Performed Noticed::DeliveryMethods::Database (Job ID: 32f27ac7-fb2e-42e4-9c5e-a3290d2d1297) from Async(default) in 29.93ms
=> [#<User id: 1, email: "julian@example.com", created_at: "2022-10-26 18:51:40.370635000 +0000", updated_at: "2022-10-26 18:51:40.370635000 +0000">]

As we can see, Noticed performs a database insert, so now we can grab all Notifications for a specified user:

irb(main):002:0> User.first.notifications
  User Load (0.6ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT ?  [["LIMIT", 1]]
  Notification Load (0.8ms)  SELECT "notifications".* FROM "notifications" WHERE "notifications"."recipient_type" = ? AND "notifications"."recipient_id" = ?  [["recipient_type", "User"], ["recipient_id", 1]]
=>
[#<Notification:0x00000001113e3b38
  id: 1,
  recipient_type: "User",
  recipient_id: 1,
  type: "TestNotification",
  params: {},
  read_at: nil,
  created_at: Thu, 27 Oct 2022 07:25:28.865982000 UTC +00:00,
  updated_at: Thu, 27 Oct 2022 07:25:28.865982000 UTC +00:00>]

This is enough for us to construct a simple index view that lists all delivered notifications for the current user:

$ bin/rails g controller Notifications index

# app/controllers/notifications_controller.rb
class NotificationsController < ApplicationController
  def index
    @notifications = current_user.notifications
  end
end

<!-- app/views/notifications/index.html.erb -->
<h1>Notifications</h1>

<ul>
<% @notifications.each do |notification| %>
  <% instance = notification.to_notification %>
  <li>
    <p>Sent at: <%= notification&.created_at.to_s %></p>
    <p>Message: <%= instance.message %></p>
  </li>
<% end %>
</ul>

After spinning up the app with bin/dev, we can log in and browse to http://localhost:3000/notifications:

3. Installing CableReady

To make use of CableReady, we need to install it. Luckily, this is quickly done:

$ bundle add cable_ready
$ yarn add cable_ready@4.5.0

Let's generate a NotificationChannel to deliver our messages:

$ bin/rails g channel Notification

This will add all the missing ActionCable (JavaScript) dependencies and scaffold the respective channel files, specifically app/channels/notification_channel.rb and app/javascript/channels/notification_channel.js.

For the server-side channel, we inherit from Noticed::NotificationChannel:

# app/channels/notification_channel.rb
class NotificationChannel < Noticed::NotificationChannel
end

Before we continue, we need to ensure our ActionCable is authenticated for Devise users. I won't go into details about that here. The necessary boilerplate looks like this:

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = find_verified_user
    end

    protected

    def find_verified_user
      if (current_user = env["warden"].user)
        current_user
      else
        reject_unauthorized_connection
      end
    end
  end
end

Check out the StimulusReflex docs for other options.

On the client side, we need to add the tiny bit of setup code mentioned above:

// app/javascript/channels/notification_channel.js
import CableReady from "cable_ready";
import consumer from "./consumer";

consumer.subscriptions.create("NotificationChannel", {
  received(data) {
    if (data.cableReady) CableReady.perform(data.operations);
  },
});

Note: Redis is required for ActionCable to work, so the rest of this article assumes you have a local server running.

4. Delivery Method Implementation

To broadcast system notifications, let's generate a new delivery method:

$ bin/rails generate noticed:delivery_method System

The scaffolded class looks like this:

# app/notifications/delivery_methods/system.rb
class DeliveryMethods::System < Noticed::DeliveryMethods::Base
  def deliver
    # Logic for sending the notification
  end

  # You may override this method to validate options for the delivery method
  # Invalid options should raise a ValidationError
  #
  # def self.validate!(options)
  #   raise ValidationError, "required_option missing" unless options[:required_option]
  # end
end

Let's continue by drafting how we envision our deliver method to work.

  class DeliveryMethods::System < Noticed::DeliveryMethods::Base
+   include CableReady::Broadcaster
+
    def deliver
-     # Logic for sending the notification
+     cable_ready[channel].notification(
+       title: "My App",
+       options: {
+         body: notification.message
+       }
+     ).broadcast_to(recipient)
    end

+   def channel
+     @channel ||= begin
+       value = options[:channel]
+       case value
+       when String
+         value.constantize
+       else
+         Noticed::NotificationChannel
+       end
+     end
+   end
-   # You may override this method to validate options for the delivery method
-   # Invalid options should raise a ValidationError
-   #
-   # def self.validate!(options)
-   #   raise ValidationError, "required_option missing" unless options[:required_option]
-   # end
  end

We have borrowed the channel method in part from the built-in ActionCable delivery method. It allows us to pass in a channel via a class method option. Otherwise, it falls back to the provided Noticed::NotificationsChannel.

Then we use CableReady's notification method to broadcast the respective instance to the recipient.

To put it into action, we have to connect it to our notification PORO:

  class TestNotification < Noticed::Base
    deliver_by :database
+   deliver_by :system, class: "DeliveryMethods::System", channel: "NotificationChannel"

    def message
      "A system notification"
    end
  end

5. Putting It to Work

Now all that's left to do is try it out. We can simply run this again from the Rails console:

irb(main):001:0> TestNotification.deliver(User.first)

Assuming you are still logged in, the browser will first ask you for permission to receive notifications on behalf of the app:

Once you have confirmed this, you get this beautiful pop-up notification from your browser:

Wrapping Up

Taking a fast lane tour through CableReady and Noticed, we have demonstrated how to integrate a native browser API into your app. The result is a simple, coherent way to deliver system notifications to your users.

This use case is also meant to illustrate how easy it is to mix CableReady into your use case. If you think ahead one step, decoupling the drafted delivery method into a library is not hard.

I hope this inspires you to look for manifestations of vertical reactive problem domains in your app and give CableReady a try.

Read more by picking up my new book Advanced CableReady.

Use the coupon code APPSIGNAL-PROMO and save $10!

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Scale Ruby on Rails Applications

Roy Tomeij — 2022-11-09T00:00:00+00:00

Today we will dive into some strategies you can use to scale Ruby on Rails applications to a huge user base.

One obvious way of scaling applications is to throw more money at them. And it works amazingly well — add a few more servers, upgrade your database server, and voila, a lot of the performance issues just go poof!

But it is often also possible to scale applications without adding more servers. That's what we will discuss today.

Let's get going!

Use AppSignal for your Rails Application

Before we dive into scaling and performance optimization, you first need to identify if you need to do this, what the bottlenecks are in your application, and what resources can be scaled.

One easy way to do this is to use AppSignal's performance monitoring and metrics for Ruby.

The performance dashboard helps you pinpoint the exact controller actions and background jobs that are slow on average.

For example, here's how a performance dashboard might look for ActiveRecord:

This gives a good starting point to any scaling journey — whether you decide to add more servers or optimize performance through code.

Now let's move on to one of the simplest techniques you can use to scale your Rails app — caching.

Caching in Ruby on Rails

Caching allows you to stop computing the same things again and again.

For example, let's say you run a social media platform and there's a very popular post. Caching can immediately help you regain all the CPU cycles you spend rendering that post for every user. And that's only a part of what caching can help you do.

Let's look at all the possible resources that can be cached.

Caching Views

Rendering views can sometimes be an expensive operation, especially when that view has a lot of data to be rendered. Even when the operation isn't expensive, using a pre-rendered view will give you a lot of performance as opposed to rendering that same view a million times.

Rails supports this out of the box using the cache view helper. For example, this is how it can cache each post when rendering a list:

<% @posts.each do |post| %>
  <% cache post do %>
    <%= render post %>
  <% end %>
<% end %>

For this, Rails automatically caches each post under a specific key that depends on the HTML content of the template, post id, and update timestamp.

To read more about this technique, check out the posts Fragment caching in Rails and Rails collection caching.

One thing to keep in mind, though, is that the cache keys don’t include nested template content. So if you are nesting the cache calls deeper than one level, there might be stale results. Read more about this in Russian doll caching in Rails.

Caching Responses

In addition to caching views/fragments, you can also choose to cache the full response of GET requests. This is supported through the If-None-Match and If-Modified-Since headers sent by the browsers.

When an If-None-Match header is present on the request, the server can return a 304 Not Modified response with no content if there are no changes to the response. The server-computed Etag is compared with the value inside that header.

Similarly, if the If-Modified-Since header is present without an If-None-Match, the server can return a 304 Not Modified response with no content (as long as the response hasn’t changed since that date).

Rails provides easy ways to do this inside controller actions. You can simply write:

class PostsController < ApplicationController
  def show
    @post = Post.find(params[:id])
    fresh_when last_modified: @post.updated_at.utc, etag: @post
  end
end

Rails will send all the required headers to support caching, handle incoming headers, and respond with 304 when the data hasn’t changed. The server can skip rendering the full views again unless things change. You can read more about advanced configuration for this strategy in Client-side caching in Rails: conditional GET requests.

Caching Values

Finally, it is also possible to cache raw values (anything that can be serialized to the cache store). This is usually useful to cache the results of resource-intensive or slow operations and avoid performing them again.

Identifying a value that can benefit from this caching depends greatly on the application, but usually, looking at your slowest events can help point you in the correct direction.

Finally, when you identify what to cache, the API that Rails provides for this is very simple to use:

Rails.cache.fetch(cache_key_with_version, expires_in: 12.hours) do
  perform_the_slow_computation
end

The above code will perform_the_slow_computation only once and then cache the value under the cache_key_with_version key. The next time the same code is called, Rails will first check if we already have a cached value and use that instead of triggering perform_the_slow_computation again.

The most important part of this caching strategy is to compute a good cache key that depends on all the inputs used in the value's computation. This is to ensure we don’t keep using a stale value.

Cache Stores

Now that we know what to cache and the techniques Rails provides to store things in the cache, the next logical question is — where do we cache this data? Rails comes with several in-built cache store adapters. The most popular cache stores for production use cases are Redis and Memcached. There are a couple of other options as well — the file store and memory store. A full discussion of these stores can be found in the post Rails' built-in cache stores: an overview.

File and memory stores can be great for development use to get things up and running quickly. However, they are usually unsuitable for production, especially if you're working in a distributed setup with multiple servers. Redis and memcached are both suited for production use. Which one you use usually depends on the application.

Background Workers in Ruby on Rails

Most applications need background jobs for mailers, regular clean-ups, or any other time-consuming operation that doesn't require a user to be present. Chances are, you already have a background worker set up already.

Whenever you find yourself doing anything that takes more than a second to do inside a controller action, see if you can move it to a background worker instead. This could range from a user-facing operation like searching for data inside a large table to an API method that ingests a large amount of data.

Example Implementation

To run custom jobs, Rails provides the Active Job framework. Let's see how we can use it to move a very complex filtering logic to a background job. First, let’s create our background job:

# jobs/filter_huge_dataset_job.rb
class FilterHugeDatasetJob < ApplicationJob
  queue_as :default

  def perform(user, filters)
    # search your data
  end
end

We can run this job from the controller like this:

# controllers/huge_datasets_controller.rb
class HugeDatasetsController < ApplicationController
  def index
    FilterHugeDatasetJob.perform_later(@current_user, filters)
  end
end

We need to render a loading indicator on our template while we wait for our job to compute data and deliver the results.

But how can we get the results from our job to the view? Turbo makes this really easy. For example, inside the view, we can subscribe to turbo-stream events on a specific notification channel using turbo_stream_from.

Using this, let’s write our templates:

# view/huge_datasets/index.html.erb
<%= render "index", user: @current_user %>

# view/huge_datasets/_index.html.erb
<%= turbo_stream_from user, :huge_datasets %>
<%= render "filters" %>
<div id="data-container">
  <% if defined? data %>
    <%= render partial: "item", collection: data  %>
  <% else %>
    <%= render "loading" %>
  <% end %>
</div>

Since data is not defined in the initial controller action, we will only render a loading indicator. Let’s now deliver the results from our job:

# jobs/filter_huge_dataset_job.rb
class FilterHugeDatasetJob < ApplicationJob
  queue_as :default

  def perform(user, filters)
    data = search_huge_dataset(filters)
    notify_completed(user, data)
  end

  def search_huge_dataset(filters)
    # search your data
  end

  def notify_completed(user, data)
    Turbo::StreamsChannel.broadcast_replace_to(
      [user, :huge_datasets],
      target: "data-container",
      partial: "huge_datasets/index",
      locals: { user: user, data: data }
    )
  end
end

The important part here is the notify_completed method. It uses Turbo::StreamsChannel, broadcasting a replace event to the [user, :huge_datasets] notification stream that we subscribed to from our view.

That is everything we need to move complex operations from our controller to background jobs. The main advantage of moving tasks to the background is that background workers can be scaled independently of web servers. This frees up resources on the web server side considerably. For the user, such interfaces also feel much more responsive because we can respond quickly and deliver results incrementally.

Note: If you need help deciding between a background job worker, read Delayed Job vs. Sidekiq: Which Is Better?

Scaling a Database in Your Ruby on Rails Application

The last scalable resource that we will discuss in this post is the database. Databases form the core of most applications. As data and the number of servers accessing that data grows, databases start to feel the load.

The easiest way to scale a database is to add more processing power and memory to the database server. As opposed to scaling web servers, doing this with a database is usually a very slow operation, especially if you have high storage.

The second option to scale databases is to scale horizontally using multiple databases or by sharding your database. Check out Multiple Databases with Active Record for more details about this.

Instead, we'll focus on optimizing your database's performance by looking at PostgreSQL.

Find Time-Consuming Queries in PostgreSQL

First, we need to identify our most time-consuming queries. The way we can do that is to query the pg_stat_statements table that contains statistics about all SQL statements executed on the server. Let’s see how we can find the top 100 queries with the highest run times:

SELECT query, calls, (total_exec_time/calls)::integer as avg_time_ms
FROM pg_stat_statements
WHERE calls > 1000
ORDER BY avg_time_ms desc
LIMIT 100;

This will return the query, the number of calls, and the average run time of these queries. Try to find the ones you think could be faster and analyze why they were slow.

You can also run EXPLAIN or EXPLAIN ANALYZE on the query to see the query plan and actual execution details, respectively.

One of the most important things to look out for in the results is Seq Scan, which indicates that Postgres has to go through all the records sequentially to run the query. If this happens, try to bypass that sequential scan by adding an index to the columns that you've filtered.

Tables with the Most Sequential Scans

Another useful query that I like to run is to find the total number of sequential scans run against a table:

SELECT relname AS name, seq_scan as count
FROM pg_stat_user_tables
ORDER BY seq_scan DESC;

If you see a very large table (with a high number of rows) and a high count value from this result, then you have a problem. Try to check all queries against that table, find ones that can run sequential scans, and add indices to boil that down.

Index Usage

You can also find statistics about index usage by running this query:

SELECT relname,
   CASE idx_scan
     WHEN 0 THEN 'Insufficient data'
     ELSE (100 * idx_scan / (seq_scan + idx_scan))::text
   END percent_of_times_index_used,
   n_live_tup rows_in_table
 FROM
   pg_stat_user_tables
 ORDER BY
   n_live_tup DESC;

This returns the percentage of index usage for each table. A low number means that you are missing some indexes on that table.

Wrap Up

In this post, we explored several strategies to scale your Ruby on Rails applications, including caching and background workers. We also looked at optimizing your PostgreSQL database's performance.

Rails makes it really easy to add several layers of performance optimization to your application.

The most important consideration with scalability is to identify bottlenecks in an application before we can act on them. A good performance monitoring tool can help. If you need one, check out AppSignal for Ruby.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Build a Table Editor with Trix and Turbo Frames in Rails

Roy Tomeij — 2022-10-26T00:00:00+00:00

In this post, we will implement a basic ActionText table editor for your Rails application. We'll learn how:

ActionText and Trix handle attachments
To implement our own Attachable type, and leverage this to build a basic table editor
Turbo Frames can be used to edit the table
Turbo helps and gets in the way at the same time

This article draws inspiration from the excellent 'Adding Tables to ActionText With Stimulus.js' blog post from 2020. That was written before the advent of Turbo though, which we can expect to simplify matters quite a bit.

Let's get going!

ActionText Attachments in Rails 101

Note: This demonstration assumes some understanding of Trix and Turbo Frames. You might find our 'Get Started with Hotwire in Your Ruby on Rails App' post helpful in learning the basics of Hotwire and Turbo Frames.

You can follow along with the code demonstration with this GitHub repo.

As described in the ActionText documentation:

Action Text brings rich text content and editing to Rails. It includes the Trix editor that handles everything from formatting to links to quotes to lists to embedded images and galleries.

At a high level, attachments are part of ActionText's document model. They render custom templates for any resource resolvable by a Signed Global ID (SGID). In other words, ActionText stores a reference to a certain SGID as an <action-text-attachment> element:

<action-text-attachment sgid="BAh7CEkiCG…"></action-text-attachment>

Whenever ActionText encounters such an element, it calls the to_attachable_partial_path method on the respective resource. By default, this method delegates to to_partial_path.

So, as a preview, this is how our Table's representation in ActionText is going to look when rendered back to HTML:

<action-text-attachment sgid="...">
  <table>
    <tbody>
      <tr>
        <td>Cell 1</td>
        <td>Cell 2</td>
      </tr>
      <!-- more rows -->
    </tbody>
  </table>
</action-text-attachment>

To conform with the ActionText Attachment API, a class has to do only two things:

Implement to_sgid by including GlobalID::Identification. By default, all ActiveRecord::Base descendants already do this.
Include the ActionText::Attachable module.

The ActionText::Attachable module offers canonical ways to convert any model to and from an SGID via the attachable_sgid and from_attachable_sgid methods. We will make use of this later on.

It also provides convenience accessors for attachment metadata, such as file size and name, as well as content type.

Finally, it provides the default locations for the partials used to render an attachment in the editor and rich text views.

Adding a Table Model

We will capitalize on ActionText's Attachment API to implement our table solution. For this, we have to create a custom model capturing our tables' data and include Attachable. We'll use a simple JSON(B) column to hold a two-dimensional array for the table data.

To start our exploration, let's create a new Rails app with ActionText enabled:

$ rails new trix-tables-turbo-frames
$ bin/rails action_text:install
$ bin/rails db:migrate

Because I'm not feeling creative today, let's scaffold an Article model with a title and rich text content:

$ bin/rails g scaffold Article title:string content:rich_text
$ bin/rails g model ActionText::Table content:json
# or, if using postgres
$ bin/rails g model ActionText::Table content:jsonb

$ bin/rails db:migrate

Watch out, here's a surprising gotcha! The above install command created a CreateActionTextTables migration, so we need to rename it to CreateActionTextTablesTable. Additionally, we'll have it default to a 2x2 table using null: false, default: [["", ""], ["", ""]].

class CreateActionTextTablesTable < ActiveRecord::Migration[7.0]
  def change
    create_table :action_text_tables do |t|
      t.json :content, null: false, default: [["", ""], ["", ""]] # create a 2x2 table by default

      t.timestamps
    end
  end
end

Add a Table to a Rails ActionText Model

Before we continue with actually adding a table to rich text, we need to patch Trix's toolbar:

  // app/javascript/application.js
  import "@hotwired/turbo-rails";
  import "controllers";
- import "trix";
+ import Trix from "trix";
  import "@rails/actiontext";

+ const buttonHTML =
+   '<button type="button"
+      class="trix-button trix-button--icon trix-button--icon-table"
+      title="table" tabindex="-1"
+      data-action="trix-table#attachTable">table</button>';
+
+ const buttonGroupElement = document
+   .querySelector("trix-editor")
+   .toolbarElement.querySelector("[data-trix-button-group=file-tools]");
+
+ buttonGroupElement.insertAdjacentHTML("beforeend", buttonHTML);

Here, we manually append a button to Trix's toolbarElement. Wiring this up to a trix-table Stimulus controller (that we've yet to build) will insert a table into the document. Let's give this button a nice SVG as content in CSS and set up some table styles while we're at it:

/* app/assets/stylesheets/application.css */
/*
 *
 *= require_tree .
 *= require_self
 */

+  .trix-button--icon-table::before {
+    background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor' stroke-width='2'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' d='M3 10h18M3 14h18m-9-4v8m-7 0h14a2 2 0 002-2V8a2 2 0 00-2-2H5a2 2 0 00-2 2v8a2 2 0 002 2z' /%3E%3C/svg%3E");
+    top: 8%;
+    bottom: 4%;
+  }
+
+ table {
+   border: 1px solid black;
+   border-collapse: collapse;
+ }
+
+ td {
+   padding: 0.5rem!important;
+   border: 1px solid black;
+ }

As you can see, we have successfully added it to the "file-tools" group:

Now let's return to adding and manipulating tables with the help of Turbo. For this, we will first need a controller with a create action:

$ bin/rails g controller Tables create --no-helper --skip-routes

This action can be more or less borrowed from the 'On Rails' blog post we cited in the introduction. It constructs the JSON necessary to insert an attachment on the client side: including an SGID and content rendered from an editor partial, as we shall see later.

# app/controllers/tables_controller.rb

class TablesController < ApplicationController
  layout false

  def create
    @table = ActionText::Table.create

    render json: {
      sgid: @table.attachable_sgid,
      content: render_to_string(partial: "tables/editor", locals: {table: @table}, formats: [:html])
    }
  end
end

We add the relevant resourceful table routes to our configuration:

  # config/routes.rb

  Rails.application.routes.draw do
    resources :articles
+   resources :tables
  end

Now comes the moment to plunge into the deep end: we need to build our table model. First, let's include ActionText::Attachable and define the relevant partial paths:

  # app/models/action_text/table.rb

  class ActionText::Table < ApplicationRecord
+   include ActionText::Attachable
+
+   attribute :content_type, :string, default: "text/html"
+
+   def to_trix_content_attachment_partial_path
+     "tables/editor"
+   end
+
+   def to_partial_path
+     "tables/table"
+   end
  end

Note that we haven't defined how the table's content is stored yet. Because we declared it as a JSON(B) column in our database, we are free to choose any format. Deviating from the cited blog post a bit, let's go with a two-dimensional array. Thus, we can simply do a nested loop over the content like this:

<!-- app/views/tables/_table.html.erb -->
<table>
  <% table.content.each do |row| %>
    <tr>
      <% row.each do |column| %>
        <td>
          <%= column %>
        </td>
      <% end %>
    </tr>
  <% end %>
</table>

The above partial will render whenever it is requested by ActionView, for example. Next, we also have to devise an editor partial to be used inline in Trix:

<!-- app/views/tables/_editor.html.erb -->
<%= turbo_frame_tag "table_#{table.attachable_sgid}" do %>
  <table>
    <% table.content.each_with_index do |row, row_index| %>
      <tr>
        <% row.each_with_index do |column, column_index| %>
          <td>
            <div contenteditable><%= column %></div>
          </td>
        <% end %>
      </tr>
    <% end %>
  </table>
<% end %>

The only difference, as you have probably noticed, is that we now have wrapped it in a Turbo Frame, using the SGID as a DOM id. Furthermore, we provide row and column indexes to the separator blocks and prepare for inline editing by making the inner DIV contenteditable — we'll get to that later.

We will now connect our toolbar's table button to the server-side controller action we have just written. To do this, we first need to bring Rails' request.js library into the project. This library will help us administer post requests from the client, including proper CSRF-tokens, etc.:

$ bin/importmap pin @rails/request.js

Build a New Trix Table Stimulus Controller

Now that everything is set up, let's create a new trix-table Stimulus controller. In it, we will implement the attachTable action referenced by our toolbar button:

// app/javascript/controllers/trix_table_controller.js

import { Controller } from "@hotwired/stimulus";
import Trix from "trix";
import { post } from "@rails/request.js";

export default class extends Controller {
  static values = {
    url: String,
  };

  async attachTable(event) {
    const response = await post(this.urlValue);

    if (response.ok) {
      const tableAttachment = await response.json;
      this.insertTable(tableAttachment);
    } else {
      // error handling
    }
  }

  insertTable(tableAttachment) {
    this.attachment = new Trix.Attachment(tableAttachment);
    this.element
      .querySelector("trix-editor")
      .editor.insertAttachment(this.attachment);
    this.element.focus();
  }
}

It will POST to the table's create route, inserting the JSON response as a Trix attachment. This again borrows from the OnRails blog post, exchanging the deprecated rails-ujs calls for the newer request.js library.

Now we have to actually make use of this controller in our app by adding it to the form's markup:

  <!-- app/views/tables/_form.html.erb -->
- <%= form_with(model: article) do |form| %>
+ <%= form_with(model: article, data: {controller: "trix-table", trix_table_url_value: tables_path}) do |form| %>
    <% if article.errors.any? %>
      <div style="color: red">
        <h2><%= pluralize(article.errors.count, "error") %> prohibited this article from being saved:</h2>

        <ul>
          <% article.errors.each do |error| %>
            <li><%= error.full_message %></li>
          <% end %>
        </ul>
      </div>
    <% end %>

    <div>
      <%= form.label :title, style: "display: block" %>
      <%= form.text_field :title %>
    </div>

    <div>
      <%= form.label :content, style: "display: block" %>
      <%= form.rich_text_area :content %>
    </div>

    <div>
      <%= form.submit %>
    </div>
  <% end %>

The beauty of Stimulus.js is that only adding two data attributes to the form element achieves the desired result. We are now able to add tables to our article's content with a single button click:

Manipulating the Table via Turbo Frames

Now that we can create table attachments, let's shift our focus to manipulating the content. As it turns out, Turbo Frames are almost a natural fit here.

Add and Delete Table Rows and Columns

To add and delete table rows and columns, we create a mini-toolbar consisting of four buttons, one for each operation. Make use of the button_to helper and set the URL to the update route for the respective table. Let's add the respective operation we want to trigger as additional parameters:

  <!-- app/views/tables/_editor.html.erb -->
  <%= turbo_frame_tag "table_#{table.attachable_sgid}" do %>
+   <div style="display: flex">
+     <%= button_to "+ Row", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "addRow"} %>
+     <%= button_to "- Row", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "removeRow"} %>
+     <%= button_to "+ Column", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "addColumn"} %>
+     <%= button_to "- Column", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "removeColumn"} %>
+   </div>
    <table>
      <% table.content.each_with_index do |row, row_index| %>
        <tr>
          <% row.each_with_index do |column, column_index| %>
            <td>
              <div contenteditable><%= column %></div>
            </td>
          <% end %>
        </tr>
      <% end %>
    </table>
  <% end %>

In turn, we also need to add the respective controller action(s) to our TablesController. Observe that the update action delegates those actions to the model.

  # app/controllers/tables_controller.rb

  class TablesController < ApplicationController
+   before_action :set_table, only: %i[show edit update destroy]

    layout false

+   def edit
+   end

    def create
      @table = ActionText::Table.create

      render json: {
        sgid: @table.attachable_sgid,
        content: render_to_string(partial: "tables/editor", locals: {table: @table}, formats: [:html])
      }
    end

+   def update
+     if params["operation"] == "addRow"
+       @table.add_row
+     elsif params["operation"] == "removeRow"
+       @table.remove_row
+     elsif params["operation"] == "addColumn"
+       @table.add_column
+     elsif params["operation"] == "removeColumn"
+       @table.remove_column
+     else
+       flash.alert = "Unknown table operation: #{params["operation"]}"
+     end
+
+     if @table.save
+       redirect_to edit_table_path(id: @table.attachable_sgid)
+     else
+       render :edit
+     end
+   end
+
+   private
+
+   def set_table
+     @table = ActionText::Attachable.from_attachable_sgid params[:id]
+   end
  end

After changes to the table's structure are saved, we redirect to the table's edit view. It renders the same editor partial, which has the side-effect of referring to the same Turbo Frame. Thus Turbo can detect the matching frame and substitute one for the other.

<!-- app/views/tables/edit.html.erb -->
<%= render "tables/editor", table: @table %>

Now we have to implement the missing commands on the Table model.

  # app/models/action_text/table.rb

  class ActionText::Table < ApplicationRecord
    include ActionText::Attachable

    attribute :content_type, :string, default: "text/html"

    def to_trix_content_attachment_partial_path
      "tables/editor"
    end

    def to_partial_path
      "tables/table"
    end

+   def rows
+     content.size
+   end
+
+   def columns
+     content.map(&:size).max
+   end
+
+   def add_row(index = rows - 1)
+     content << Array.new(columns, "")
+   end
+
+   def remove_row(index = rows - 1)
+     content.delete_at(index)
+   end
+
+   def add_column(index = columns - 1)
+     content.each do |row|
+       row << ""
+     end
+   end
+
+   def remove_column(index = columns - 1)
+     content.each do |row|
+       row.delete_at(index)
+     end
+   end
  end

Notably, due to our simple data structure of a two-dimensional array, the add/removecolumn/row methods are mere proxies to modify the column and row count. Once that is in place, we can change our table's structure with button clicks:

Edit the Content of Table Cells

In addition to changing the number of columns and rows, we also want to edit the cells' content. To achieve this, we will again lean heavily on the cited blog post and create a Stimulus table editor controller.

// app/javascript/controllers/table_editor_controller.js

import { Controller } from "@hotwired/stimulus";
import { patch } from "@rails/request.js";

export default class extends Controller {
  static values = {
    url: String,
  };

  async updateCell(event) {
    const response = await patch(this.urlValue, {
      body: { value: event.target.textContent },
      query: {
        operation: "updateCell",
        row_index: event.target.dataset.rowIndex,
        column_index: event.target.dataset.columnIndex,
      },
      contentType: "application/json",
      responseKind: "json",
    });
  }
}

The updateCell method will issue a PATCH request whenever a cell is edited, passing the row and column index as parameters. Now, all we have to do is connect it to our DOM:

  <!-- app/views/tables/_editor.html.erb -->
- <%= turbo_frame_tag "table_#{table.attachable_sgid}" do %>
+ <%= turbo_frame_tag "table_#{table.attachable_sgid}",
+    data: {controller: "table-editor", table_editor_url_value: table_path(id: table.attachable_sgid)} do %>
    <div style="display: flex">
      <%= button_to "+ Row", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "addRow"} %>
      <%= button_to "- Row", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "removeRow"} %>
      <%= button_to "+ Column", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "addColumn"} %>
      <%= button_to "- Column", table_path(id: table.attachable_sgid), method: :patch, params: {operation: "removeColumn"} %>
    </div>
    <table>
      <% table.content.each_with_index do |row, row_index| %>
        <tr>
          <% row.each_with_index do |column, column_index| %>
            <td>
-             <div contenteditable><%= column %></div>
+             <div contenteditable
+                data-action="input->table-editor#updateCell"
+                data-row-index="<%= row_index %>"
+                data-column-index="<%= column_index %>">
+               <%= column %>
+             </div>
            </td>
          <% end %>
        </tr>
      <% end %>
    </table>
  <% end %>

The server-side TablesController, of course, now needs a way to handle this operation. Luckily, this is easily done in our simplified proof of concept by adding another branch to our condition. We also make sure that the update action can now handle JSON-type requests, even if it's merely returning an empty object here.

  # app/controllers/tables_controller.rb

  class TablesController < ApplicationController
    before_action :set_table, only: %i[show edit update destroy]

    layout false

    def edit
    end

    def create
      @table = ActionText::Table.create

      render json: {
        sgid: @table.attachable_sgid,
        content: render_to_string(partial: "tables/editor", locals: {table: @table}, formats: [:html])
      }
    end

    def update
      if params["operation"] == "addRow"
        @table.add_row
      elsif params["operation"] == "removeRow"
        @table.remove_row
      elsif params["operation"] == "addColumn"
        @table.add_column
      elsif params["operation"] == "removeColumn"
        @table.remove_column
+     elsif params["operation"] == "updateCell"
+       @table.content[params["row_index"].to_i][params["column_index"].to_i] = params["value"]
      end

      if @table.save
-       redirect_to edit_table_path(id: @table.attachable_sgid)
+       respond_to do |format|
+         format.html { redirect_to edit_table_path(id: @table.attachable_sgid) }
+         format.json {}
+       end
      else
        render :edit
      end
    end

    private

    def set_table
      @table = ActionText::Attachable.from_attachable_sgid params[:id]
    end
  end

Note that in a production app, I would advise you to choose a different strategy for sanitizing the operation than an if/elsif/else condition. I would probably reach for a Mediator or Proxy in this case.

The Limitations of Trix in Ruby

Up to this point, I assume this account has made perfect sense, but I have left out a critical detail. While we are persisting the underlying database model just fine, we are not syncing it to Trix's internal shadow representation. That's why the table snaps back to the previously stored representation when we focus out of it:

If we were to refresh the page now, the added content would appear, because Trix's document is freshly initialized.

I have pinned this problem down to where Trix syncs its internal document when the selection changes. It just unfurls it from the shadow element here.

I tried hooking into the turbo:submit event and preventing the sync just when blurring a table, but the solutions I came up with all seem very hairy and highly dependent on the internal API.

The most Turbo-esque way of dealing with this, I guess, would be to wrap the whole form in an eager-loaded Turbo Frame and tell it to reload whenever Trix's content changes.

Something like this should do the trick:

// app/javascript/controllers/trix_table_controller.js

// ...

connect() {
  this.element.addEventListener("turbo:submit-end", (e) => {
    this.element.closest("turbo-frame").reload();
  });
}

// ...

If you enclose your form in a Turbo Frame that you load from src:

<!-- app/views/articles/edit.html.erb -->
<h1>Editing article</h1>

<%= turbo_frame_tag dom_id(@article, :form), src: form_article_path(@article) %>

This approach only works with already persisted base records, though.

Final Words of Warning on Trix

The proof of concept we've built uses server-rendered HTML to do away with the added complexity of serializing tables to JSON and listening for JavaScript events. It is portable to any ActionText installation and could be easily extracted to a gem.

There are a couple of drawbacks, though, the most obvious one being the necessary re-syncing with Trix's document model. There might be situations where the proposed workaround is workable and others where it's a no-go. Until Trix gains a Turbo-compatible interface, there's no way around it.

The second catch is that it does not use Trix's undo functionality (but that is true of any Trix attachment). Likewise, it would be wise to wait for upstream changes instead of tweaking the internal API.

Wrap Up

In this post, we started by taking a quick look at the basics of ActionText Attachments. We then added a table to an ActionText model before tweaking it using Turbo Frames. Finally, we touched on some limitations of using Trix.

Given that Trix v2 is underway, featuring a translation from CoffeeScript to plain modern JavaScript, now would be a good time to address its Turbo compatibility. Currently, the scope of what such a wrapper might look like is beyond my capabilities, but it sure looks like a window of opportunity.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Improve Code in Your Ruby Application with RubyCritic

Roy Tomeij — 2022-10-19T00:00:00+00:00

RubyCritic provides visual reports highlighting code smells, code structure, ease of testing, and test coverage in your Ruby application.

It's in active development, with new code analysis tools often being introduced as new features. It's well worth keeping track of RubyCritic's releases.

This article will touch on some of RubyCritic's benefits, its dependencies, and how to read its code reports.

Let's get going!

Why Choose RubyCritic for Your Ruby on Rails Application?

You should consider using RubyCritic if you want a single place to review code improvements for your project. Including RubyCritic in your development process will certainly reduce the time a development team spends working on technical debts. Most technical debts will be mapped out at development time.

Some benefits that RubyCritic can provide to your project and development process include:

Unified information in one place
Visual reports
Easy installation
Zero configuration
Customization allowed
Being extensible — you can make your own open-source integration
A badge generator 🎉

To understand how RubyCritic works, let's look at the internal dependencies it uses to make reports.

Internal Dependencies in RubyCritic

When you add RubyCritic to your project, some dependencies will also be included.

Let's highlight the dependencies that make magic happen: the Reek, Flay, and Flog gems. These dependencies allow RubyCritic to show you valuable information about your code. Understanding how they work also makes RubyCritic easier to use.

Reek: Detect Code Smells in Ruby

Reek is a gem for detecting code smells in Ruby. A bad smell in code is not about identifying wrong code, it is more about analyzing if the code could be written better.

Reek's analysis identifies if something could be implemented in another way. It does not suggest how, as most code smells are associated with business logic and a developer's experience with a language.

For example, you could easily rewrite an if statement using metaprogramming techniques. The way to correct it, though, is up to a developer according to a project's context. In this case, no library will be able to indicate the best solution.

Reek detects an extensive list of smells. It examines and identifies possible smells in:

Classes
Attributes
Methods
Parameters
Modules
Iterators
The implementation of polymorphism

By finding smells, you can take steps to make your code more readable and maintainable.

Reek allows for custom configuration to:

Disable a detector by its type
Exclude directories from being scanned
Use filters to silence warnings

You can even define specific code to suppress in a scan, a very useful feature when code is not yet finalized or refactored, or even if it is legacy code.

Let's see a sample of how Reek works. In this code, the exception is just defined as e.

# app/controllers/erp/orders_controller.rb

def create
  ...

  rescue JSON::Schema::ValidationError => e
    render status: :unprocessable_entity, json: {
      type: "invalid-schema",
      title: "Your request does not match the expected schema.",
      detail: e.message
    }
  end
end

It is easy to imagine that e means an exception, but what if we have other exceptions? Identifying them correctly is the best way to maintain good code.

Reek will identify e as UncommunicativeVariableName and show a warning.

$ reek app/controllers/erp/orders_controller.rb
Inspecting 1 file(s):
S

app/controllers/erp/orders_controller.rb -- 1 warning:

  [91]:UncommunicativeVariableName: Erp::OrdersController#create has the variable name 'e' [https://github.com/troessner/reek/blob/v6.1.1/docs/Uncommunicative-Variable-Name.md]

Flay: Check for Ruby Code Duplication

Flay identifies structural Ruby code similarities, including:

Detecting code duplication within a project
Checking the difference at any code level
Generating a score to measure how good your code is (the lower your score, the better the code)

If Flay reports a similarity in your code, it's a high indicator that refactoring is needed. Don't ignore this! Duplicate code is a gateway to bugs. If you fix something in one place but forget about another, more bugs appear.

We can check how Flay works by running it in its own source code:

$ flay lib/flay.rb
Total score (lower is better) = 36

1) Similar code found in :iter (mass = 36)
  lib/flay.rb:80
  lib/flay.rb:105

Flay identifies similarities between these two:

# lib/flay.rb:80
opts.on("-m", "--mass MASS", Integer, "Sets mass threshold (default = #{options[:mass]})") do |m|
  options[:mass] = m.to_i
end

# lib/flay.rb:105
opts.on("-t", "--timeout TIME", Integer, "Set the timeout. (default = #{options[:timeout]})") do |t|
  options[:timeout] = t.to_i
end

Note that the code's spelling is not exactly the same, but its functionality is and can be refactored to avoid duplication. That's the magic of Flay!

Flog: Examine Your Code Complexity in Ruby

Flog checks how difficult your code is to test. It sets a complexity score for each line of code and sums up the score for each method and class.

The higher the score, the more your code needs to be refactored because it signifies that you have a highly complex implementation.

Let's see Flog in action! A small change can cause your score to variate.

def validate_expiration
  return if exp_month.blank? || exp_year.blank?

  ...
end

$ flog app/models/credit_card.rb

5.2: CreditCard#validate_expiration   app/models/credit_card.rb:12-15

Note that in the first part of the code, we have an or check that increases the score by 0.4 points.

def validate_expiration
  return if exp_month.blank?
  return if exp_year.blank?

  ...
end

$ flog app/models/credit_card.rb

4.8: CreditCard#validate_expiration app/models/credit_card.rb:12-15

Other RubyCritic Dependencies

RubyCritic also uses other runtime dependencies, such as:

byebug - this elevates debugging Ruby applications. It allows you to run a program line by line, add breakpoints, and evaluate and track values at runtime. If you still use puts for debugging, it's time you get to know Byebug's features and commands.
rubocop - a linter for Ruby code that helps you follow a style guide used by the Ruby community, or even apply your own code style. It's very useful to set standards in your team and avoid silly conflicts about spaces and tabs.
SimpleCov - a tool to check Ruby application code coverage. You can configure it to run alongside your tests. It provides metrics on code coverage so that you can identify what you need to pay attention to and where to invest your time to create better test cases.

Dive into RubyCritic's list of dependencies.

Using RubyCritic for Your Ruby on Rails App

RubyCritic has good documentation to help you get started without much configuration. Therefore, we will focus on utilizing its resources to help us analyze its reports.

RubyCritic provides 'Code', 'Smells', and 'Coverage' reports. We'll look at each of these features in turn.

The Overview in RubyCritic

The 'Overview' page shows a total score for your project on a donut chart, along with ratings (A being the best rating, F the worst). The 'Summary' section shows the details of each rating, including the number of files, churns (commit changes), and smells found.

In the 'Churn vs Complexity' section here, it is already possible to identify the class with the greatest complexity, which should probably be the first point of attention.

To better understand this graph, it is worth recapping code churn. Code that changes frequently can raise an alert that something is wrong — maybe in the logic or the business domain, for example. Either way, looking at 'Churn vs Complexity' can help you see where the pain points are across your project.

Code Report

The 'Code' report shows a score for each class, including indicators for churn, complexity, duplication, and smells.

You can sort this list by any column to view the highest ranking factors and address the most critical issues first.

In addition, this list has a filter that allows you to search by class name quickly.

Clicking on the class name will open a detailed page with the class code and metrics such as:

Code line
Quantity methods
Calculated churn
Complexity by method
Complexity score (total per class)
Amount of duplicates found
Number of smells

The line of code where an issue is found will be highlighted (based on information provided by the Reek gem).

If Flog identifies any issues, you'll see a score. You'll also see if a Flay report has detected any duplicate code.

Smells Report

The 'Smells' page displays the smell type, the exact location where a smell appears, and the fix status.

As mentioned earlier, the smells are detected by Reek, and sorting and filtering are also available on this page.

Clicking on a class name will open a page with your code details. You can also see the classes grouped by smell type (this is missing from the 'Code' page, which only displays the number of smells).

Coverage Report

Finally, you can see class classifications and the percentage of coverage for each class in the 'Coverage' report. In contrast to the lists on the 'Code' and 'Smells' reports, the list in 'Coverage' does not allow information to be sorted and filtered.

You can only see the percentage of code coverage — no additional information is available.

Integrating the SimpleCov report could add more value and usefulness to this page. But in any case, the 'Coverage' report can help if you need a simple report to examine your project's test coverage.

Wrap Up

In this post, we briefly looked at the benefits of RubyCritic for your Ruby application before diving into its internal dependencies: Reek, Flay, and Flog. We then ran through how to read and analyze RubyCritic's reports.

As a next step, figure out how to use RubyCritic in your pipeline.

Happy code refactoring!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Security Best Practices for Your Rails Application

Roy Tomeij — 2022-10-05T00:00:00+00:00

Alongside performance and usability, you should always focus on security when creating any web application. Keep in mind that hacking techniques are constantly evolving, just as fast as technology is. So you must know how to secure your users and their data.

This article will show you how to create a secure Rails application. The framework is known to be secure by default, but the default configuration is not enough to let you sleep well at night.

We will share some best coding practices and a few habits in the development process that can help you create secure code.

Let's dive in!

Security in Your App: Best Coding Practices

Modern web applications are often complex. They utilize multiple data sources and handle custom authorization rules as well as different methods of authentication. Knowing how to avoid SQL injections or ensuring that users can only read their data is not enough.

When you build a Rails application, you have to configure it properly, design the application securely, and write code that makes it bulletproof.

We'll look at:

Application configuration - The default configuration is exemplary, but we can make things even better with a few extra steps.
Business logic - Applications should be secure by design, not only by code. This principle is essential but is often ignored when an MVP has to be delivered quickly.
Code in controllers - These classes are the entry point to our application, so an extra dose of attention is always needed to design a reliable application.
Code in models - Many issues are related to a database, so it is essential to design and perform secure communication with the primary source of the data.
Code in views - The point where we expose data to the browser is also a popular target for hackers, so we have to ensure that we don't render anything that risks our users' data or privacy.

Application Configuration

It all starts here, in the configuration files. In most cases, unless you restart the application, the rules will remain the same. Rails' creators made an effort to create secure defaults, but we can still improve them with some extra steps.

Force SSL

You can force your Rails application to always use a secure connection with the HTTPS protocol. To do this, open the config/environments/production.rb file and set the following line:

config.force_ssl = true

This setting does a few things to the application:

Every time there is a request to the HTTP version of the application, Rails will redirect the request to the HTTPS protocol.
It sets a secure flag on cookies. Thanks to this, browsers won't send cookies with HTTP requests.
It tells the browser to remember your application as TLS-only (TLS is Transport Layer Security, an extension of the HTTPS protocol).

CORS

Cross-Origin Resource Sharing (CORS) is a security mechanism that defines which website can interact with your application's API. Of course, if you build a monolith app, you don't need to care about this protection.

If you build an API application, you can configure CORS by installing an additional gem called rack-cors.

Once you have done that, create a file called cors.rb in the initializers directory config/initializers. Define what endpoints a website can access (including request methods):

Rails.application.config.middleware.insert_before 0, Rack::Cors do
 allow do
   origins 'https://your-frontend.com'
   resource '/users',
     :headers => :any,
     :methods => [:post]
   resource '/articles',
     headers: :any,
     methods: [:get]
 end
end

In the above example, we allow the your-frontend.com website to call the /users endpoint (using only the POST method) and the /articles endpoint (using only the GET method).

Secure Environment Variables

You should never hardcode your API keys, passwords, or other sensitive credentials in the source code. You might accidentally make them public or give someone who's not authorized access to some sensitive application resources.

The Rails framework itself provides a way to store credentials securely. However, the implementation varies depending on the framework version:

Rails 4 - The feature is called 'secrets'. You store your sensitive information in a config/secrets.yml file that is not tracked in the git repository.
Rails 5 - The feature is called 'credentials'. Your sensitive information is stored encrypted in config/credentials.yml.enc — you can edit it with the config/master.key file. So while the YAML configuration file can be stored in the repository because it’s encrypted, you don’t track the master.key file.
Rails 6 - Also called 'credentials', you can store credentials per environment. Because of that, for every environment, you have the encrypted YAML file and key to decrypt it.

Alternatively, you can always set the values on the server level, so that they only load in the server environment. Locally, each developer sets them individually.

Business Logic

You should think about security not only when you write code, but also when you design the processes in your application.

Business logic is a set of rules that apply in the real world, and your goal is to map them in your code. Unfortunately, mapping them can cause weak points in your application and lead to security issues.

Strong Authentication and Authorization Rules

Let's first explain the difference between authentication and authorization, as these terms are sometimes misinterpreted:

Authentication - You validate a user's login and password against an application's database.
Authorization - You validate the role of a signed-in user and, based on that, render different information for different users. For example, a user with an admin role can access a list of users in an application, while in most cases, the typical user can't.

To improve the security level of your authentication, you can set high standards for your end-users:

Strong passwords - Set a strict password policy that doesn't allow for passwords that are too simple or weak. Of course, you can't control if your users share their passwords, but you can make their passwords hard to guess.
Two-factor authentication - Another layer of protection that will secure an account even if someone else knows the password.
Encrypted passwords - Never store passwords in your database as plain text. Then, even if your database is exposed, a hacker won't be able to get your users' passwords.

Authorization Best Practices

If your application is complex, you probably need different roles for users to manage their data. As business logic grows, it may be hard to control everything without making a mistake that leads to an information leak.

You can avoid issues by following some well-known good practices:

Keep authorization logic in one place - It's hard to modify business logic correctly if you have to change multiple areas in the code. Storing the rules in one file makes it easy.
Set clear rules - You won't be able to tell whether your application is secure if you can't validate the business logic against well-defined rules.
Set roles based on the group, not a single user - It is easier to control the authorization process if you have groups of permissions instead of defining rules per user.

Of course, good code reviews and well-written tests are also a must here to avoid introducing bugs to existing business logic.

Code in Controllers

Controllers are the first layer of MVC architecture and handle requests coming from users. Therefore, it is essential to filter incoming information correctly as it will be propagated on other application layers.

Filter Incoming Parameters

You should never pass a raw params value to your application. Thanks to the strong parameters feature, it is easy to control the data that we'd like to pass along.

Imagine that we have a User model we want to update:

class UsersController < ApplicationController
  def update
    current_user.update(params[:user])
  end
end

Someone can manipulate the params, and it would be possible to update other User model attributes (for example, the admin flag if you have one). To avoid such a situation, filter params that you pass to your model:

class UsersController < ApplicationController
  def update
    current_user.update(user_params)
  end

  private

  def user_params
    params.require(:user).permit(:first_name, :last_name)
  end
end

Use Scopes to Avoid Data Leaking

Usually, we don't want to show the user data that does not belong to them. We may inadvertently expose some records for URL address manipulation due to the wrong code design. Let's consider the following case:

class MessagesController
  before_action :authenticate_user!

  def show
    @message = Message.find(params[:id])
  end
end

Everything seems fine; the controller is protected from guests, and we assign the message object. However, a user just needs to change the id in the URL address to get a message that does not belong to them. This is a very dangerous situation.

We can avoid it by using scopes within the given context. In the mentioned example, the current user is the scope:

class MessagesController
  before_action :authenticate_user!

  def show
    @message = current_user.messages.find(params[:id])
  end
end

Avoid Insecure URL Redirects

Let's consider a straightforward example of a redirect based on the user input:

redirect_to params[:url]

We should never do that, as we expose our user to a redirect that can take them everywhere. The simplest solution to prevent this security issue is to avoid using redirects with user input. If you can't, you can always redirect to a path without the host:

path = URI.parse(params[:url]).path.presence || "/"
redirect_to path

Code in Models

Once Rails parses code in a controller, you will probably call models to receive the information from the database. Since models are responsible for communicating with your database as well as performing important computations, they are also often targeted by hackers.

Avoid SQL Injection

SQL injection is one of the most popular techniques used to access database information from the outside. The Rails framework tries to protect us from that threat, but we also need to write code that won't let this happen.

In general, we should avoid passing user input directly as a part of a query:

User.joins(params[:table]).order('id DESC')

If you need to, always validate the user input and assign a default value when the input is invalid:

valid_tables = %w[articles posts messages]
table = valid_tables.include?(params[:table]) ? params[:table] : "articles"
User.joins(table).order('id DESC')

Check out this extensive set of examples of what not to do when it comes to SQL injection.

Prevent Command Injection

Avoid using methods that allow a program to execute any code. Such methods include exec, system, `, and eval. You should never pass user input to those functions.

If your application allows users to execute code, you should run their code in separate Docker containers. In addition, you can remove dangerous methods before executing user input:

module Kernel
 remove_method :exec
 remove_method :system
 remove_method :`
 remove_method :eval
 remove_method :open
end

Binding.send :remove_method, :eval
RubyVM::InstructionSequence.send :remove_method, :eval

Avoid Unsafe Data Serialization

Insecure deserialization can lead to the execution of arbitrary code in your application. If you plan to serialize JSON:

data = { "key" => "value" }.to_json
# => "{\"key\":\"value\"}"

Instead of using load or restore methods, use parse:

# bad
JSON.load(data)
JSON.restore(data)

# good
JSON.parse(data)

If you plan to serialize YAML:

data = { "key" => "value" }.to_yaml
# => "---\nkey: value\n"

Instead of using load or restore methods from the Marshal module, use the safe_load method from Psych:

# bad
Marshal.load(data)
Marshal.restore(data)

# good
Psych.safe_load(data)

Code in Views

Whatever you render in views is visible to the end-user. When malicious code is rendered, it will affect your users directly by exposing them to untrusted websites and data sources.

Avoid CSS Injection

It would appear that CSS code is always secure; it only modifies the styles of a website. However, if you allow for user-defined styles in your application, there is always a risk of CSS code injection.

One of the most popular cases of user-defined CSS is a custom page background:

<body style="background: <%= profile.background_color %>;"></body>

A user with bad intentions can input a URL that an application will automatically load, and do damage to the current user viewing the content. To prevent such situations, provide a predefined set of values instead of allowing users to type any value.

Sanitize Rendered Output

In the modern versions of Rails, output is sanitized by default. Even if a user inputs HTML or JS code and the application renders it, the HTML or JS code will escape.

If you want to render HTML defined by users, always predefine which tags should render and which should escape:

<%= sanitize @comment.body, tags: %w(strong em a), attributes: %w(href) %>

In the above code, we allow users to render strong, em, and a HTML elements in their comments.

Don't Include Sensitive Data in Comments

This is primarily a reminder for junior developers unfamiliar with how web applications work on the front end. Don't ever put sensitive information in comments (especially in views, as it will be exposed to end-users):

<!--- password for the service is 1234 –>
<%= @some_service.result_of_search %>

Habits to Keep Your Rails Application Secure

To make your Rails app secure and reduce technical debt, establish some valuable development habits. Taking small preventative actions frequently is much less painful than refactoring more significant portions of your codebase.

Upgrade Rails and Other Libraries Often

An upgrade is often painful and time-consuming, unless you upgrade to minor versions of a library. Develop a habit of upgrading a library each time a new, stable version is published. Your application will then be in good shape (not only regarding security, but also performance).

If you use GitHub for day-to-day development, an extension like Depfu is useful as it performs frequent updates for you.

Perform Security Audits

I don't mean expensive audits by external companies. Often, it is enough to install a tool like Brakeman and scan code with every commit or pull request.

Also, it is a good idea to scan your Gemfile and find gems that need updating due to security issues discovered by the community. You can use bundler-audit to automate this process.

Have A Proper Logging Strategy

Logs, in many cases, are usually just thousands of lines of information you will never view. However, there might be a case where one of your users is attacked or experiences a suspicious action.

In such a situation, if you have detailed and easily searchable logs, you can collect information that will help you prevent similar problems in the future.

Wrapping Up: Keep Your Rails App Secure

In this post, we ran through some security best practices for your Rails app that reduce the risk of a data breach.

Making sure that your Rails application is secure might be challenging. Sticking to the framework's defaults is not enough; you have to know how to avoid creating security issues. For example, various injections and remote code executions have been well-known issues for the past few years, but still, you can't be 100% sure that your application won't be affected.

Don't forget the importance of frequent upgrades, security audits, and a culture of good logging. When combined, all of these things will help to make your Rails application more secure.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Debugging in Ruby with AppSignal

Roy Tomeij — 2022-09-21T00:00:00+00:00

An application monitoring tool (APM) is not just useful for seeing how your application performs through graphs and visuals. We can go deeper and use an APM to understand how your application behaves in a certain environment.

As developers, we should aim to be less reactive to errors and more predictive, avoiding crashes for end-users.

One way to accomplish this is by using monitoring tools to debug our application when an error occurs. AppSignal has advanced features that allow us to intelligently debug our logs and thus reflect this precautionary thinking in our applications.

This article will show you how to better log and debug your Ruby application using AppSignal.

Integrate Your Ruby App with AppSignal

To start debugging with AppSignal, you need to have an AppSignal account.

Integrate your Ruby application with AppSignal by following the steps in the article 'Ruby on Rails Application Monitoring with AppSignal'.

Log Options

Now that you have an application set up to send logs to AppSignal, let's look at the logging options for your Ruby application.

AppSignal offers various log levels for your application according to your project's needs. In this section, we will explore the difference between each log level.

At first glance, we might think these logs are related to the application logs shown in the 'Backtrace' section of AppSignal.

However, changes to log_level won't reflect the information displayed in the backtrace. It is important to understand that setting the log_level value relates to AppSignal's communication logs with your application. It has nothing to do with application logs.

This means that when we set the logging level in our application, we change the type of information that appears in the application logs about execution and integration with AppSignal.

In most cases, you can just set the log_level to error (unless you've had an issue with the AppSignal integration and need to send log information to AppSignal's Support team). The log_level must be defined in config/appsignal.yml, as in the example below:

# config/appsignal.yml

production:
  <<: *defaults
  log_level: error

Watch the application logs when we set error, warning, or info to log_level. We will see pretty much the same information in the logs.

Now, check out the logs when we set log_level as debug.

Finally, let's see what happens when we set log_level as trace.

Ignore Your Ruby App's Predicted Errors in AppSignal

You don't need to catch and keep all errors in logs. Some common errors can be expected in an application — we don't need to inflate our logs with 404 errors, for example. Now let's configure AppSignal to ignore errors that are irrelevant to trace.

Ignoring an error is useful when a production bug has already been identified, but the fix is still in development. We already know about the problem's existence, so there's no need to keep filling the error monitoring tool or be notified about it. We can ignore the error's appearance until the patch deployment goes into production. This approach can be applied to all known bugs that have already been cataloged in the issues list.

We can also ignore errors when we know that a service our application communicates with will be down for maintenance. This way, we don't flood the monitoring tool with errors regarding expected maintenance, alerting our monitoring team unnecessarily.

AppSignal's gem provides three ways to handle and ignore predicted errors.

`ignore_errors`

ignore_errors helps when you need to ignore all errors generated by a class error or when you don't know the exact error that could happen from there.

Maybe your application needs to do many math calculations, and AppSignal doesn't need to track all related errors. ignore_errors will reduce the errors list and only keep issues that need to be addressed.

`ignore_actions`

This is useful if you know exactly which method will generate an error you don't need to trace — for example, a method that will be re-executed until it succeeds. You can also use ignore_actions for queue processing when another monitoring tool handles logs specific to queue item failures.

`ignore_namespace`

ignore_namespace is great if you have an application separated by domain and want to group errors. Maybe you have a specific module to treat payments or to process an order in the background. Grouping it in a namespace for jobs makes sense. Then you can ignore the exceptions that occur inside this namespace.

Configure in Your Ruby App

Include ignore_errors, ignore_actions, and ignore_namespace in your config/appsignal.yml file, with a list of classes to ignore when this error occurs in your application.

# config/appsignal.yml

production:
  <<: *defaults
  ignore_errors:
    - Net::HTTPGatewayTimeout
    - NoMethodError
  ignore_actions:
    - HomeController#index
  ignore_namespace:
    - jobs
    - payment

Debugging Exceptions in Ruby with AppSignal

Now that we know how to ignore irrelevant errors, it's time to deal with the errors that matter. AppSignal provides functions to configure how an error is sent from your application to the AppSignal monitor. Let's learn about the listen_for_error feature and extract valuable information from exceptions.

Sometimes, we have a situation where an exception may or may not be thrown due to some parameters (if a job is not running or even if an external endpoint is unavailable). We still want to send detailed information to AppSignal, not just the exception itself. For this case, we'll create a custom exception and use it to send helpful information to the monitoring team.

The first thing we need to do is create a class for our custom exception. Make a new folder called exceptions inside your application directory. Then create a new class called CustomException inherited from StandardError. You can define any information as the content of this exception. By default, an exception already has msg as an attribute.

In this example, you need to include an exception name and the content field to report all details about the exception. In the exception constructor, all attributes are set and the method to send to AppSignal is called.

The block inside Appsignal.listen_for_error is where the magic happens! The raise will send the exception content to AppSignal in hash format to facilitate the separation of the different contents.

# app/exceptions/custom_exception.rb

class CustomException < StandardError
  attr_accessor :name, :content

  def initialize(name, msg, content)
    @name = name
    @msg = msg
    @content = content

    send_exception_to_appsignal
  end

  def send_exception_to_appsignal
    Appsignal.listen_for_error do
      exception_hash = { name: @name, msg: @msg, content: @content }
      raise "#{exception_hash}"
    end
  end
end

Now you can call this custom exception from anywhere, to use it in your application. For this example, we will include it in the controller inside the Home#index method. We need a block to handle the exception, to avoid sending it to AppSignal. The nil.object code will throw a NoMethodError exception, but this exception is handled, and only CustomException will be sent to AppSignal.

# app/controllers/home_controller.rb

class HomeController < ApplicationController
  def index
    begin
      nil.object
    rescue => e
      raise CustomException.new("Customized Exception", "A new exception occurred", e)
    end
  end
end

All done! After starting your application and accessing http://localhost:3000/home#index in your browser, you will see a page with a RuntimeError exception. In AppSignal, access the 'Errors > Issue list' to see the Customized Exception issue.

{:name=>"Customized Exception", :msg=>"A new exception occurred", :content=>#<NoMethodError: undefined method `object' for nil:NilClass>}

Wrap Up and Next Steps

AppSignal provides a well-documented gem to start truly monitoring your Ruby application. Why not test it? Send data from your local machine without deploying to see if the error configuration is how you want it.

Many other features in AppSignal make it easier to debug Ruby applications, like exception handling. Check AppSignal's Ruby documentation and choose the feature that makes the most sense in your project's context. Make your app easier to debug by decreasing the time your team needs to spend looking into the root cause of bugs in production.

Happy debugging!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

JIT Compilers for Ruby and Rails: An Overview

Roy Tomeij — 2022-09-07T00:00:00+00:00

A program is compiled at runtime using a different method from pre-execution compilation. This process is known as just-in-time compilation or dynamic translation.

In this post, we'll look at why JIT compilation can be a good choice for your Ruby on Rails app, before looking at some of the options available (YJIT, MJIT, and TenderJIT) and how to install them.

But first: how does JIT compilation work?

How a JIT Compiler Works

Just-in-time compilation is a method of running computer code that requires compilation while running a program.

This could entail translating the source code, but it's most frequently done by converting the bytecode to machine code, which is then run directly.

The code being executed is often continuously analyzed by a system using a JIT compiler. This identifies sections of code where the benefit of compilation or recompilation (in terms of speed) outweighs the cost.

Benefits of JIT Compilation for Ruby

JIT compilation combines some of the benefits (and shortcomings) of the two conventional methods for converting programs into machine code: interpretation and ahead-of-time compilation (AOT).

Roughly speaking, it combines the flexibility of interpretation with the speed of generated code, and the additional overhead of compiling and linking (not just interpreting).

JIT compilation is a type of dynamic compilation that enables adaptive optimization techniques, including dynamic recompilation and speed-ups tailored to certain microarchitectures. Due to a runtime system's ability to handle late-bound data types and impose security guarantees, dynamic programming languages like Ruby are particularly well-suited for interpretation and JIT compilation.

An optimizing compiler like GCC can more efficiently optimize instructions — a significant advantage of adopting a register-oriented architecture. Compilers operate on intermediate representation with register-based architecture.

Once your instructions reach an intermediate representation during compilation, GCC does additional passes to speed up the CPU's execution of your instructions.

JIT Compilers for Ruby: YJIT, MJIT, and TenderJIT

Now let's explore the different JIT compilers available for Ruby — YJIT, MJIT, and TenderJIT — and how you can set them up.

MJIT (Method-based Just-in-time Compiler) for Ruby

Vladimir Makarov implemented MJIT, and it was the first compiler methodology implemented in Ruby based on the C language. It works with Ruby 2.6, uses YARV instructions, and compiles instructions often used in binary code.

For programs that are not input/output-bound, MJIT enhances performance.

YJIT is better than this original C-based compiler in terms of performance. Ruby 3's JIT is the quickest JIT that MRI has ever had, made possible by the excellent work of MJIT.

How to Use MJIT

To use MJIT, you can enable the JIT in Ruby 2.6 and with the --jit option.

ruby --jit app.rb

If you skip this part, MJIT will show an error.

clang: error: cannot specify -o when generating multiple output files
MJIT warning: Making precompiled header failed on compilation. Stopping MJIT worker...
MJIT warning: failed to remove "/var/folders/3d/fk_588wd4g12syc56pjqybjc0000gn/T//_ruby_mjit_hp25992u0.h.gch": No such file or directory
Successful MJIT finish

A collection of JIT-specific settings included in Ruby 2.6 helps us understand how it functions. Run ruby --help to view these options.

In short, MJIT executes in a different thread and is asynchronous. It will begin just-in-time compilation following the first five runs of a calculation.

YJIT for Ruby on Rails

A recent JIT compiler called YJIT was released with Ruby 3.1. It promises a lot of improvements and better performance. Still a work-in-progress project designed by Shopify with experimental results, it must be used with caution, especially on larger applications.

With that in mind, YJIT enhances the performance of Ruby on Rails applications. The majority of real-world software benefits from the fast warm-up and performance enhancements provided by the YJIT basic block versioning JIT compiler.

A JIT compiler will be gradually built into CRuby as part of the YJIT project, eventually replacing the interpreter for most of the code execution.

Official benchmarks — see 'YJIT: Building a New JIT Compiler for CRuby' — show that YJIT improved performance over the default CRuby interpreter by:

20% on railsbench
39% on liquid template rendering
37% on activerecord

However:

Only about 79% of instructions in railsbench are executed by YJIT, and the rest run in the default interpreter.

Source: YJIT: Building a New JIT Compiler for CRuby

This means that a lot still needs to be done to improve YJIT's current results.

Even so, YJIT performs at least as well as the interpreter on every benchmark, even on the hardest ones, and reaches near-peak performance after just one iteration of every benchmark.

How to Use YJIT

Note: YJIT is currently limited to macOS and Linux on x86-64 platforms. Also, as mentioned, YJIT is not recommended for large applications (yet).

YJIT is disabled by default. If you want to enable it, first specify the --yjit command-line option.

You need to check if it is installed, so run ruby --enable-yjit -v. If warning: unknown argument for --enable: yjit'` shows up, you have to install it.

Then open irb and set RUBY_YJIT_ENABLE=1. You can exit and now, you're ready to use YJIT. The command ruby --enable-yjit -v must return something like ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [arm64-darwin21]

TenderJIT

With a design largely based off YJIT, TenderJIT is an experimental JIT compiler for Ruby. What's different about TenderJIT is that it's written in pure Ruby.

This is a demo project and the aim is to ship it as a gem. In the meantime, you can experiment with it, but bear in mind it's still a work in progress. Ruby 3.0.2 or later is required for TenderJIT.

How to Use TenderJIT

TenderJIT does not currently do method compilation automatically. To compile a method, you must manually configure TenderJIT.

Clone the repository and run the following commands:

$ bundle install
$ bundle exec rake test

You must set it manually on your code:

require "tenderjit"

def your_method
  ...
end

jit = TenderJIT.new
jit.compile(method(:your_method))

Each YARV instruction in the target method is read by TenderJIT, which then transforms it into machine code.

For more examples with TenderJIT, check one of these videos: A JIT compiler for Ruby with Aaron Patterson and Hacking on TenderJIT!

Wrapping Up

In this post, we've taken a quick look at three JIT compilers for Ruby — MJIT, YJIT, and TenderJIT — and how to set them up. Each of the options is experimental and comes with its own limitations.

However, YJIT is the most mature at the moment, and it has the biggest potential to grow and scale. It demonstrates better performance over the other Ruby JITs, was developed with Ruby 3.1.0, and is quickly becoming an important part of CRuby.

Check out this post if you want to build your own compiler for Ruby.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Monitor Ruby Application Performance with Magic Dashboards

Roy Tomeij — 2022-08-31T00:00:00+00:00

Application teams must understand what their customer experience is like. This is true not only from a general perspective (in terms of usability and responsiveness) but also on a day-to-day, minute-by-minute basis.

In particular, when you work with distributed systems, errors are inevitable. Site traffic fluctuates throughout the day, and any one of a system’s dependencies could also encounter an issue at any time.

In this article, we'll use magic dashboards to help monitor and resolve performance issues within a Ruby on Rails application.

But before we dive into magic dashboards, let's see what we should look out for when designing our apps.

Things to Consider When Building A Ruby App

As an application owner or team member, you'll want to know if there are any problems with your application before the customer does. This enables you to take corrective action immediately and hopefully avoid any disruption to your users.

There are a few key application questions you need to be able to answer at any point in time:

Is the page response time acceptable? According to Google, these days anything slower than two seconds will cause customers to leave your site and go elsewhere.
Are your users experiencing any errors? If so, what types of errors? What is the error rate?
Are any ongoing operational issues affecting your application? This could be with the network, storage, or security services. As we all know, cloud providers have outages that impact our application’s health as well.

Observability is key, yet it is often the last thing engineers think about. Besides, there is already time built into the schedule for operational readiness. It's called the weekend before the product launch!

Joking aside, you have to build your app first before something exists to be observed. This lends itself to delaying performance testing until close to the end of the development lifecycle. You don’t want to spend too much time performance testing when the product isn’t code-complete, because then you’ll have to do it all over again later.

The same is true for security testing. If you conduct it too early, a vulnerability could still be introduced after testing, but before a product goes into production.

So, we know monitoring is critical, but all of these concerns are fair points. Monitoring doesn’t get the attention it deserves until late in the process. That’s why AppSignal created magic dashboards.

Magic Dashboards in AppSignal

AppSignal understands the importance of metrics, dashboards, and your app's performance — but also that engineers have minimal time to work on these before launch.

Our magic dashboards are called "magic" because the metrics collection and associated dashboards are created automatically for you, simply when you connect your app and integrated components.

Magic Dashboards: An Example Ruby on Rails App

Let's find out how we can monitor and resolve performance issues within a Ruby on Rails app using magic dashboards.

Our application uses a simple machine learning (ML) model for cryptocurrency price prediction. An asynchronous job updates the model daily with the latest price data and improves accuracy over time. This Rails app has both web pages and a REST API, as shown in the architecture diagram below.

After we deploy our Rails app, magic dashboards are automatically created for the Rails Puma web server and the Sidekiq asynchronous jobs. Other supported integrations for magic dashboards include MongoDB and the Erlang VM.

You can use the Puma magic dashboard to evaluate performance based on threads, pool capacity, and puma workers. The Sidekiq magic dashboard monitors queue length, queue latency, job duration, job status, and memory usage.

Magic dashboards are detected and created based on the use of event-based metrics such as a Sidekiq job run, as well as minutely probes. The minutely probe feature allows you to register a Ruby block or class to send custom metrics to AppSignal. Magic dashboards provide out-of-the-box integration for supported components, but you can leverage this mechanism to send your custom metrics to AppSignal.

An Example of Custom Minutely Probes

Sidekiq integration is already built in, but imagine you want to monitor a proprietary background job mechanism. You can use the following class example. The probe class obtains a connection that is then used on each call to get the desired metrics.

# config/initializers/appsignal.rb or a file that's loaded on boot

# Creating a probe using a Ruby class
class BackgroundJobLibraryProbe
  def initialize
    # This is only called when the minutely probe gets initialized
    require "background_job_library"
    @connection = BackgroundJobLibrary.connection
  end

  def call
    stats = @connection.fetch_queue_stats
    Appsignal.set_gauge "background_job_library_queue_length", stats.queue_length
    Appsignal.set_gauge "background_job_library_processed_jobs", stats.processed_jobs
  end
end

# Registering a Class probe
Appsignal::Minutely.probes.register(
  :background_job_library_probe, BackgroundJobLibraryProbe
)

The call to Appsignal::Minutely.probes.register takes two parameters — a name for the probe and its implementation, which can be either a lambda or a class that implements the call method.

Configuration and System Requirements for AppSignal

Simply use AppSignal with your Rails application to get magic dashboards. If you haven’t already installed AppSignal, include the appsignal gem in your Gemfile and run a bundle install.

gem 'appsignal'

Then run the appsignal install command to configure your environment. This configuration can be stored in a config file or environment variables, and connects your application to your AppSignal account and dashboards.

bundle exec appsignal install appsignal-license-key

After you install and run your server, you will see a few emails similar to the following.

The system requirements for this example are:

AppSignal gem 2.9.0 or higher, which includes support for the minutely probe.
Puma integration - requires version 3.11.4 or higher.
Sidekiq integration - requires the Redis gem 3.3.5 or higher. For this integration, the AppSignal gem 2.9.5 or higher is recommended.

Performance Testing our Price Prediction API

The above example application uses a simple neural network implemented using the Ruby FANN gem to predict the next day's Bitcoin price. Percentage price changes from the last ten days are used as inputs to the model.

The REST API doesn't take any parameters, as it currently only predicts the price for tomorrow. The output is a simple JSON document, as shown below.

http://localhost:3000/crypto/predict_api

{
  "day":"2022-07-22",
  "price":23080.95619
}

A Sidekiq job gets the market price at the beginning of each day and updates the ML model. This job is scheduled to run every five minutes, enabling it to catch up to any outages or errors quickly.

You can find all of the code from this article on GitHub. Please note that nothing in this article or the software constitutes investment advice. Consult a financial advisor before making any investment decisions regarding cryptocurrency.

We'll use JMeter to simulate load on the REST API for performance testing. This allows us to generate traffic and evaluate performance using our magic dashboards. Our first test uses 5 concurrent clients, each making 100 requests.

The statistics show a fairly high range in response time, with an average of 342ms, but the P99 is 852ms. It seems that we can do better.

Improving Rails App Performance with the Puma Magic Dashboard

Our Puma magic dashboard includes a graph of the thread pool capacity, and we can see that it touches zero during our test run. This would explain why some requests take longer than others, so we'll increase the number of puma threads to 10.

Keep in mind, we did nothing to create the dashboard or this graph. AppSignal automatically created this for us.

After running the test with the increased puma thread count, the response times are much more consistent, and the dashboard confirms that pool capacity stays within acceptable levels.

Now it is time to turn up the dial. The number of JMeter concurrent clients is increased to 50, with a ramp-up time of 5 seconds. This test shows a poor response time and a number of API errors. The Puma magic dashboard again shows the available puma threads reaching zero.

Looking at the API code, we find that the model is being loaded from the database each time. This is not very efficient, so we change the Sidekiq job to not only grab the new daily price, but also run the ML model and save the prediction in the database.

We deploy this change.

Better API Performance with Sidekiq's Magic Dashboard

Now let's check our Sidekiq magic dashboard. Unfortunately, there is a bug in the code, but at least we can identify and fix it quickly.

With the Sidekiq PriceUpdateJob now working, the API is modified so that it only needs to retrieve the predicted price from the database. This improves the API performance, but we still see some API errors and lengthy response times.

Back to the Puma Magic Dashboard

A glance at the dashboard highlights that we have not yet configured Puma to use additional workers. A Puma worker is an OS-level process that can run several threads. The total thread count is calculated as the number of workers multiplied by the maximum threads. First, we use 2 workers, but the available threads are still exhausted. So we'll increase the amount to 4 workers.

Our API performance is now very good. The average response time is 419ms, and there are no API errors.

The magic dashboard confirms that there is still available thread capacity.

AppSignal's magic dashboards give us instant insights into the capacity of the Puma web server. The Sidekiq and Active Worker dashboards provide similar insights into our application's asynchronous jobs.

What We've Learned from Magic Dashboards

During our Rails application performance testing, the Puma magic dashboard helped us easily identify that the number of threads was insufficient for our desired throughput level. It shows the thread pool capacity, number of workers over time, and the total number of threads.

While the Sidekiq job did not have any performance issues, the magic dashboard helped us quickly identify that there was an error after deployment, which we were able to fix and redeploy rapidly. All of these monitoring capabilities were set up for us automatically by AppSignal.

AppSignal's Dashboard Features for Ruby and Rails Apps

Dashboards in AppSignal have namespaces, such as "web" application or "background" jobs, so you can create numerous monitoring views of your application. The built-in summary dashboard shows an overview of your application's health, including throughput, response time, and the latest errors.

Note that, as with any dashboard, you can edit what graphs and metrics are shown, as well as change the layout and the configuration of selected graphs.

Anomaly detection is a powerful feature. It allows you to define thresholds that send notifications when a metric value goes over or below a given value, such as free memory or the error rate.

Wrapping Up: Monitor Your Ruby App Today with AppSignal

Observability and performance are critical to the success of our applications. However, we often wait until the last minute to deal with these topics.

AppSignal's magic dashboards provide extensive out-of-the-box metrics, dashboards, and insights that are set up automatically. This allows you to rapidly tune your application’s performance and reach a state of operational readiness.

An Introduction to Ractors in Ruby

Roy Tomeij — 2022-08-24T00:00:00+00:00

In this post, we'll dive into ractors in Ruby, exploring how to build a ractor. You'll send and receive messages in ractors, and learn about shareable and unshareable objects.

But first, let's define the actor model and ractors, and consider when you should use ractors.

What is the Actor Model?

In computer science, the object-oriented model is very popular, and in the Ruby community, many people are used to the term 'everything is an object'.

Similarly, let me introduce you to the actor model, within which 'everything is an actor'. The actor model is a mathematical model of concurrent computation in which the universal primitive/fundamental agent of computation is an actor. An actor is capable of the following:

Receiving messages and responding to the sender
Sending messages to other actors
Determining how to respond to the next message received
Creating several other actors
Making local decisions
Performing actions (e.g., mutating data in a database)

Actors communicate via messages, process one message at a time, and maintain their own private state. However, they can modify this state via messages received, eliminating the need for a lock or mutex.

Received messages are processed one message at a time in the order of FIFO (first in, first out). The message sender is decoupled (isolated) from the sent communication, enabling asynchronous communication.

A few examples of the actor model implementation are akka, elixir, pulsar, celluloid, and ractors. A few examples of concurrency models include threads, processes, and futures.

What Are Ractors in Ruby?

Ractor is an actor-model abstraction that provides a parallel execution feature without thread-safety concerns.

Just like threads, ractors provide true parallelism. However, unlike threads, they do not share everything. Most objects are unshareable, and when they are made shareable, are protected by an interpreter or locking mechanism.

Ractors are also unable to access any objects through variables not defined within their scope. This means that we can be free of the possibility of race conditions.

In 2020, when Ruby 3.0.0 was released, these were the words of Matz:

It’s multi-core age today. Concurrency is very important. With Ractor, along with Async Fiber, Ruby will be a real concurrent language.

Ractors do not claim to have solved all thread-safety problems. In the Ractor documentation, the following is clearly stated:

There are several blocking operations (waiting send, waiting yield, and waiting take) so you can make a program which has dead-lock and live-lock issues.

Some kind of shareable objects can introduce transactions (STM, for example). However, misusing transactions will generate inconsistent state.

Without ractors, you need to trace all state mutations to debug thread-safety issues. However, the beauty of ractors is that we can concentrate our efforts on suspicious shared code.

When and Why Should I Use Ractors in Ruby?

When you create a ractor for the first time, you'll get a warning like this one:

<internal:ractor>:267: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.

However, that does not mean that you should avoid using ractors. Due to parallel execution, ractors can complete processes way faster than when processes are carried out synchronously.

In the Ruby 3.0.0 release notes, you'll find this benchmark example of the Tak function, where it is executed sequentially four times, and four times in parallel with ractors:

def tarai(x, y, z) =
  x <= y ? y : tarai(tarai(x-1, y, z),
                     tarai(y-1, z, x),
                     tarai(z-1, x, y))
require 'benchmark'
Benchmark.bm do |x|
  # sequential version
  x.report('seq'){ 4.times{ tarai(14, 7, 0) } }

  # parallel version with ractors
  x.report('par'){
    4.times.map do
      Ractor.new { tarai(14, 7, 0) }
    end.each(&:take)
  }
end

The results are as follows:

Benchmark result:
          user     system      total        real
seq  64.560736   0.001101  64.561837 ( 64.562194)
par  66.422010   0.015999  66.438009 ( 16.685797)

The Ruby 3.0.0 release notes state:

The result was measured on Ubuntu 20.04, Intel(R) Core(TM) i7-6700 (4 cores, 8 hardware threads). It shows that the parallel version is 3.87 times faster than the sequential version.

So if you need a faster process execution time that can run in parallel on machines with multiple cores, ractors are not a bad idea at all.

Modifying class/module objects on multi-ractor programs can introduce race conditions and should be avoided as much as possible. However, most objects are unshareable, so the need to implement locks to prevent race conditions becomes obsolete. If objects are shareable, they are protected by an interpreter or locking mechanism.

Creating Your First Ractor in Ruby

Creating a ractor is as easy as creating any class instance. Call Ractor.new with a block — Ractor.new { block }. This block is run in parallel with every other ractor.

It is important to note that every example shown from this point onwards was performed in Ruby 3.1.2.

r = Ractor.new { puts "This is my first ractor" }
# This is my first ractor

# create a ractor with a name
r = Ractor.new name: 'second_ractor' do
  puts "This is my second ractor"
end
# This is my second ractor

r.name
# => "second_ractor"

Arguments can also be passed to Ractor.new, and these arguments become parameters for the ractor block.

my_array = [4,5,6]
Ractor.new my_array do |arr|
  puts arr.each(&:to_s)
end
# 4
# 5
# 6

Recall how we talked about ractors being unable to access objects defined outside their scope? Let's see an example of that:

outer_scope_object = "I am an outer scope object"
Ractor.new do
  puts outer_scope_object
end
# <internal:ractor>:267:in `new': can not isolate a Proc because it accesses outer variables (outer_scope_object). (ArgumentError)

We get an error on the invocation of .new, related to a Proc not being isolated. This is because Proc#isolate is called at a ractor's creation to prevent sharing unshareable objects. However, objects can be passed to and from ractors via messages.

Sending and Receiving Messages in Ractors

Ractors send messages via an outgoing port and receive messages via an incoming port. The incoming port can hold an infinite number of messages and runs on the FIFO principle.

The .send method works the same way a mailman delivers a message in the mail. The mailman takes the message and drops it at the door (incoming port) of the ractor.

However, dropping a message at a person's door is not enough to get them to open it. .receive is then available for the ractor to open the door and receive whatever message has been dropped.

The ractor might want to do some computation with that message and return a response, so how do we get it? We ask the mailman to .take the response.

tripple_number_ractor = Ractor.new do
  puts "I will receive a message soon"
  msg = Ractor.receive
  puts "I will return a tripple of what I receive"
  msg * 3
end
# I will receive a message soon
tripple_number_ractor.send(15) # mailman takes message to the door
# I will return a tripple of what I receive
tripple_number_ractor.take # mailman takes the response
# => 45

As seen above, the return value of a ractor is also a sent message and can be received via .take. Since this is an outgoing message, it goes to the outgoing port.

Here's a simple example:

r = Ractor.new do
  5**2
end
r.take # => 25

Besides returning a message, a ractor can also send a message to its outgoing port via .yield.

r = Ractor.new do
  squared = 5**2
  Ractor.yield squared*2
  puts "I just sent a message out"
  squared*3
end
r.take
# => 50
r.take
# => 75

The first message sent to the outgoing port is squared*2, and the next message is squared*3. Therefore, when we call .take, we get 50 first. We have to call .take a second time to get 75 as two messages are sent to the outgoing port.

Let's put this all together in one example of customers sending their orders to a supermarket and receiving the fulfilled orders:

supermarket = Ractor.new do
  loop do
    order = Ractor.receive
    puts "The supermarket is preparing #{order}"
    Ractor.yield "This is #{order}"
  end
end

customers = 5.times.map{ |i|
  Ractor.new supermarket, i do |supermarket, i|
    supermarket.send("a pack of sugar for customer #{i}")
    fulfilled_order = supermarket.take
    puts "#{fulfilled_order} received by customer #{i}"
  end
}

The output is as follows:

The supermarket is preparing a pack of sugar for customer 3
The supermarket is preparing a pack of sugar for customer 2
This is a pack of sugar for customer 3 received by customer 3
The supermarket is preparing a pack of sugar for customer 1
This is a pack of sugar for customer 2 received by customer 2
The supermarket is preparing a pack of sugar for customer 0
This is a pack of sugar for customer 1 received by customer 1
This is a pack of sugar for customer 0 received by customer 0
The supermarket is preparing a pack of sugar for customer 4
This is a pack of sugar for customer 4 received by customer 4

Running it a second time yields:

The supermarket is preparing a pack of sugar for customer 0
This is a pack of sugar for customer 0 received by customer 0
The supermarket is preparing a pack of sugar for customer 4
This is a pack of sugar for customer 4 received by customer 4
The supermarket is preparing a pack of sugar for customer 1
This is a pack of sugar for customer 1 received by customer 1
The supermarket is preparing a pack of sugar for customer 3
The supermarket is preparing a pack of sugar for customer 2
This is a pack of sugar for customer 3 received by customer 3
This is a pack of sugar for customer 2 received by customer 2

The output can most definitely be in a different order every time we run this (because ractors run concurrently, as we have established).

A few things to note about sending and receiving messages:

Messages can also be sent using << msg, instead of .send(msg).
You can add a condition to a .receive using receive_if.
When .send is called on a ractor that is already terminated (not running), you get a Ractor::ClosedError.
A ractor's outgoing port closes after .take is called on it if it runs just once (not in a loop).

r = Ractor.new do
  Ractor.receive
end
# => #<Ractor:#61 (irb):120 running>
r << 5
# => #<Ractor:#61 (irb):120 terminated>
r.take
# => 5
r << 9
# <internal:ractor>:583:in `send': The incoming-port is already closed (Ractor::ClosedError)
r.take
# <internal:ractor>:694:in `take': The outgoing-port is already closed (Ractor::ClosedError)

Objects can be moved to a destination ractor via .send(obj, move: true) or .yield(obj, move: true). These objects become inaccessible at the previous destination, raising a Ractor::MovedError when you try to call any other methods on the moved objects.

r = Ractor.new do
  Ractor.receive
end
outer_object = "outer"
r.send(outer_object, move: true)
# => #<Ractor:#3 (irb):7 terminated>
outer_object + "moved"
# `method_missing': can not send any methods to a moved object (Ractor::MovedError)

Threads cannot be sent as messages using .send and .yield. Doing this results in a TypeError.

r = Ractor.new do
  Ractor.yield(Thread.new{})
end
# <internal:ractor>:627:in `yield': allocator undefined for Thread (TypeError)

Shareable and Unshareable Objects

Shareable objects are objects that can be sent to and from a ractor without compromising thread safety. An immutable object is a good example because once created, it cannot be changed — e.g., numbers and booleans.

You can check the shareability of an object via Ractor.shareable? and make an object shareable via Ractor.make_shareable.

Ractor.shareable?(5)
# => true
Ractor.shareable?(true)
# => true
Ractor.shareable?([4])
# => false
Ractor.shareable?('string')
# => false

As seen above, immutable objects are shareable and mutable ones aren't. In Ruby, we usually call the .freeze method on a string to make it immutable. This is the same method ractors apply to make an object shareable.

str = 'string'
Ractor.shareable?(str)
# => false
Ractor.shareable?(str.freeze)
# => true
arr = [4]
arr.frozen?
# => false
Ractor.make_shareable(arr)
# => [4]
arr.frozen?
# => true

Messages sent via ractors can either be shareable or unshareable. When shareable, the same object is passed around. However, when unshareable, ractors perform a full copy of the object by default and send the full copy instead.

SHAREABLE = 'share'.freeze
# => "share"
SHAREABLE.object_id
# => 350840
r = Ractor.new do
  loop do
    msg = Ractor.receive
    puts msg.object_id
  end
end
r.send(SHAREABLE)
# 350840
NON_SHAREABLE = 'can not share me'
NON_SHAREABLE.object_id
# => 572460
r.send(NON_SHAREABLE)
# 610420

As seen above, the shareable object is the same within and outside the ractor. However, the unshareable one isn't because the ractor has a different object, just identical to it.

Another method to send an exact object when it is unshareable is the previously discussed move: true. This moves an object to a destination without needing to perform a copy.

A few things to note about sharing objects in ractors:

Ractor objects are also shareable objects.
Constants that are shareable, but defined outside the scope of a ractor, can be accessed by a ractor. Recall our outer_scope_object example? Give it another try, defined as OUTER_SCOPE_OBJECT = "I am an outer scope object".freeze.
Class and module objects are shareable, but instance variables or constants defined within them are not if assigned to unshareable values.

class C
  CONST = 5
  @share_me = 'share me'.freeze
  @keep_me = 'unaccessible'
  def bark
   'barked'
  end
end

Ractor.new C do |c|
  puts c::CONST
  puts c.new.bark
  puts c.instance_variable_get(:@share_me)
  puts c.instance_variable_get(:@keep_me)
end
# 5
# barked
# share me
# (irb):161:in `instance_variable_get': can not get unshareable values from instance variables of classes/modules from non-main Ractors (Ractor::IsolationError)

An incoming port or outgoing port can be closed using Ractor#close_incoming and Ractor#close_outgoing, respectively.

Wrap Up and Further Reading on Ractors

In this article, we introduced the concept of ractors, including when and why to use them and how to get started. We also looked at how they communicate with one another, what objects are shareable and unshareable, and how to make objects shareable.

Ractors go deeper than this. Many other public methods can be called on ractors, like select to wait for the success of take, yield and receive, count, current, etc.

To expand your knowledge about ractors, check out the ractor documentation. This GitHub gist might also interest you if you'd like to experimentally compare ractors with threads.

Ractors are indeed experimental, but they certainly look like they have a bright future in Ruby's evolution.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

A Deep Dive into Memory Leaks in Ruby

Roy Tomeij — 2022-08-10T00:00:00+00:00

In the first part of this two-part series on memory leaks, we looked at how Ruby manages memory and how Garbage Collection (GC) works.

You might be able to afford powerful machines with more memory, and your app might restart often enough that your users don't notice, but memory usage matters.

Allocation and Garbage Collection aren't free. If you have a leak, you spend more and more time on Garbage Collection instead of doing what you built your app to do.

In this post, we'll look deeper into the tools you can use to discover and diagnose a memory leak.

Let's continue!

Finding Leaks in Ruby

Detecting a leak is simple enough. You can use GC, ObjectSpace, and the RSS graphs in your APM tool to watch your memory usage increase. But just knowing you have a leak is not enough to fix it. You need to know where it is coming from. Raw numbers can't tell you that.

Fortunately, the Ruby ecosystem has some great tools to attach context to those numbers. Two are memory-profiler and derailed_benchmarks.

`memory_profiler` in Ruby

The memory_profiler gem offers a very simple API and a detailed (albeit a little overwhelming) allocated and retained memory report — that includes the classes of objects that are allocated, their size, and where they were allocated. It's straightforward to add to our leaky program.

# leaky.rb
require "memory_profiler"

an_array = []

report = MemoryProfiler.report do
  11.times do

    1000.times { an_array << "A" + "B" + "C" }
    puts "Array is #{an_array.size} items long"
  end

  GC.start
end

report.pretty_print

Outputting a report that looks similar to this.

Total allocated: 440072 bytes (11001 objects)
Total retained:  440072 bytes (11001 objects)

allocated memory by gem
-----------------------------------
    440072  other

allocated memory by file
-----------------------------------
    440072  ./leaky.rb

allocated memory by location
-----------------------------------
    440000  ./leaky.rb:9
        72  ./leaky.rb:10

allocated memory by class
-----------------------------------
    440000  String
        72  Thread::Mutex

allocated objects by gem
-----------------------------------
     11001  other

allocated objects by file
-----------------------------------
     11001  ./leaky.rb

allocated objects by location
-----------------------------------
     11000  ./leaky.rb:9
         1  ./leaky.rb:10

allocated objects by class
-----------------------------------
     11000  String
         1  Thread::Mutex

retained memory by gem
-----------------------------------
    440072  other

retained memory by file
-----------------------------------
    440072  ./leaky.rb

retained memory by location
-----------------------------------
    440000  ./leaky.rb:9
        72  ./leaky.rb:10

retained memory by class
-----------------------------------
    440000  String
        72  Thread::Mutex

retained objects by gem
-----------------------------------
     11001  other

retained objects by file
-----------------------------------
     11001  ./leaky.rb

retained objects by location
-----------------------------------
     11000  ./leaky.rb:9
         1  ./leaky.rb:10

retained objects by class
-----------------------------------
     11000  String
         1  Thread::Mutex


Allocated String Report
-----------------------------------
     11000  "ABC"
     11000  ./leaky.rb:9


Retained String Report
-----------------------------------
     11000  "ABC"
     11000  ./leaky.rb:9

There is a lot of information here, but generally, the allocated objects by location and retained objects by location sections can be the most useful when looking for leaks. These are the file locations that allocate objects, ordered by the number of allocated objects.

allocated objects are all objects allocated (created) within the report block.
retained objects are objects that have not been garbage collected by the end of the report block. We forced a GC run before the end of the block so we could see the leaked objects more clearly.

Be careful about trusting the retained object counts. They depend heavily on what portion of the leaking code is within the report block.

For example, if we move the declaration of an_array into the report block, we might be fooled into thinking the code isn't leaky.

# leaky.rb
require "memory_profiler"

report = MemoryProfiler.report do
  an_array = []

  11.times do

    1000.times { an_array << "A" + "B" + "C" }
    puts "Array is #{an_array.size} items long"
  end

  GC.start
end

report.pretty_print

The top of the resulting report won't report many retained objects (just the report itself).

Total allocated: 529784 bytes (11002 objects)
Total retained:  72 bytes (1 objects)

`derailed_benchmarks` in Ruby

The derailed_benchmarks gem is a suite of very useful tools for all kinds of performance work, primarily aimed at Rails apps. For finding leaks, we want to look at perf:mem_over_time, perf:objects, and perf:heap_diff.

These tasks work by sending curl requests to a running app, so we can't add them to our little leaky program. Instead, we'll need to set up a small Rails app with an endpoint that leaks memory, then install the derailed_benchmarks on that app.

# Create a rails app with no database
rails new leaky --skip-active-record --minimal

## Add derailed benchmarks
cd leaky
bundle add derailed_benchmarks

# config/routes.rb
Rails.application.routes.draw do
  root "leaks#index"
end

# app/controllers/leaks_controller.rb
class LeaksController < ApplicationController
  def index
    1000.times { $an_array << "A" + "B" + "C" }

    render plain: "Array is #{$an_array.size} items long"
  end
end

# config/initializers/array.rb
$an_array = []

You should now be able to boot the app with bin/rails s. You'll be able to curl an endpoint that leaks on each request.

$ curl http://localhost:3000

Array is 1000 items long

$ curl http://localhost:3000

Array is 2000 items long

We can now use derailed_benchmarks to see our leak in action.

`perf:mem_over_time`

This will show us memory use over time (similarly to how we watched the memory growth of our leaky script with watch and ps).

Derailed will boot the app in production mode, repeatedly hit an endpoint (/ by default), and report the memory usage. If it never stops growing, we have a leak!

$ TEST_COUNT=10000 DERAILED_SKIP_ACTIVE_RECORD=true \
  bundle exec derailed exec perf:mem_over_time

Booting: production
Endpoint: "/"
PID: 4417
104.33984375
300.609375
455.578125
642.69140625
751.6953125

Note: Derailed will boot the Rails app in production mode to perform the tests. By default, it will also require rails/all first. Since we don't have a database in this app, we need to override this behavior with DERAILED_SKIP_ACTIVE_RECORD=true.

We can run this benchmark against different endpoints to see which one/s (if any) leak.

`perf:objects`

The perf:objects task uses memory_profiler under the hood so the produced report will look familiar.

$ TEST_COUNT=10 DERAILED_SKIP_ACTIVE_RECORD=true \
  bundle exec derailed exec perf:objects

Booting: production
Endpoint: "/"
Running 10 times
Total allocated: 2413560 bytes (55476 objects)
Total retained:  400000 bytes (10000 objects)

# The rest of the report...

This report can help narrow down where your leaked memory is being allocated. In our example, the report's last section — the Retained String Report — tells us exactly what our problem is.

Retained String Report
-----------------------------------
     10000  "ABC"
     10000  /Users/tonyrowan/playground/leaky/app/controllers/leaks_controller.rb:3

We've leaked 10,000 strings containing "ABC" from the LeaksController on line 3. In a non-trivial app, this report would be significantly larger and contain retained strings that you want to retain — query caches, etc. — but this and the other 'by location' sections should help you narrow down your leak.

`perf:heap_diff`

The perf:heap_diff benchmark can help if the report from perf:objects is too complex to see where your leak is coming from.

As the name suggests, perf:heap_diff produces three heap dumps and calculates the difference between them. It creates a report that includes the types of objects retained between dumps and the location that allocated them.

$ DERAILED_SKIP_ACTIVE_RECORD=true bundle exec derailed exec perf:heap_diff

Booting: production
Endpoint: "/"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-0.ndjson"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-1.ndjson"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-2.ndjson"

Diff
====
Retained STRING 999991 objects of size 39999640/40008500 (in bytes) at: /Users/tonyrowan/playground/leaky/app/controllers/leaks_controller.rb:3
Retained STRING 2 objects of size 148/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:265
Retained STRING 1 objects of size 88/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:266
Retained DATA 1 objects of size 72/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/3.1.0/objspace.rb:87
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/3.1.0/objspace.rb:88
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:259
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:260
Retained FILE 1 objects of size 8432/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:266

Run `$ heapy --help` for more options

You can also read Tracking a Ruby memory leak in 2021 to understand better what's going on.

The report points us exactly where we need to go for our leaky baby app. At the top of the diff, we see 999991 retained string objects allocated from the LeaksController on line 3.

Leaks in Real Ruby and Rails Apps

Hopefully, the examples we've used so far have never been put into real-life apps — I hope no one intends to leak memory!

In non-trivial apps, memory leaks can be much harder to track down. Retained objects are not always bad — a cache with garbage collected items would not be of much use.

There is something common between all leaks, though. Somewhere, a root-level object (a class/global, etc.) holds a reference to an object.

One common example is a cache without a limit or an eviction policy. By definition, this will leak memory since every object put into the cache will remain forever. Over time, this cache will occupy more and more of the memory of an app, with a smaller and smaller percentage of it actually in use.

Consider the following code that fetches a high score for a game. It's similar to something I've seen in the past. This is an expensive request, and we can easily bust the cache when it changes, so we want to cache it.

class Score < ApplicationModel
  def self.user_high_score(game, user)
    @scores = {} unless @scores

    if (score = @scores["#{game.id}:#{user.id}"])
      score
    else
      Score.where(game: game, user: user).order(:score).first.tap do |score|
        @scores["#{game.id}:#{user.id}"] = score
      end
    end
  end

  def self.save_score(game, user, raw_score)
    score = create!(game: game, user: user, score: raw_score)

    if raw_score > user_high_score(game, user).score
      @scores["#{game.id}:#{user.id}"] = score
    end
  end
end

The @scores hash is completely unchecked. It will grow to hold every single high score for every user — not ideal if you have a lot of either.

In a Rails app, we would probably want to use Rails.cache with a sensible expiry (a memory leak in Redis is still a memory leak!) instead.

In a non-Rails app, we want to limit the hash size, evicting the oldest or least recently used items. LruRedux is a nice implementation.

A more subtle version of this leak is a cache with a limit, but whose keys are of arbitrary size. If the keys themselves grow, so too will the cache. Usually, you won't hit this. But, if you're serializing objects as JSON and using that as a key, double-check that you're not serializing things that grow with usage as well — such as a list of a user's read messages.

Circular References

Circular references can be garbage collected. Garbage Collection in Ruby uses the "Mark and Sweep" algorithm. During their presentation introducing variable width allocation, Peter Zhu and Matt Valentine-House gave an excellent explanation of how this algorithm works.

Essentially, there are two phases: marking and sweeping.

In the marking phase, the garbage collector starts at root objects (classes, globals, etc.), marks them, and then looks at their referenced objects.

It then marks all of the referenced objects. Referenced objects that are already marked are not looked at again. This continues until there are no more objects to look at — i.e., all referenced objects have been marked.
The garbage collector then moves on to the sweeping phase. Any object not marked is cleaned up.

Therefore, objects with live references can still be cleaned up. As long as a root object does not eventually reference an object, it will be collected. In this way, clusters of objects with circular references can still be garbage collected.

Application Performance Monitoring: The Event Timeline and Allocated Objects Graph

As mentioned in the first part of this series, any production-level app should use some form of Application Performance Monitoring (APM).

Many options are available, including rolling your own (only recommended for larger teams). One key feature you should get from an APM is the ability to see the number of allocations an action (or background job) makes. Good APM tools will break this down, giving insight into where allocations come from — the controller, the view, etc.

This is often called something like an 'event timeline.' Bonus points if your APM allows you to write custom code that further breaks down the timeline.

Consider the following code for a Rails controller.

class LeaksController < ApplicationController
  before_action :leak

  def index
    @leaks = $leak.sample(100)
  end


  private

  def leak
    1000.times { $leak << Leak.new }
  end
end

When reported by an APM, the 'event timeline' might look something like the following screenshot from AppSignal.

This can be instrumented so we can see which part of the code makes the allocations in the timeline. In real apps, it is probably going to be less obvious from the code 😅

class LeaksController < ApplicationController
  before_action :leak

  def index
    Appsignal.instrument('leak.fetch_leaks') do
      @leaks = $leak.sample(100)
    end
  end


  private

  def leak
    return unless params[:leak]

    Appsignal.instrument('leak.create_leaks') do
      1000.times { $leak << Leak.new }
    end
  end
end

Here's an example of an instrumented event timeline, again from AppSignal:

Knowing where to instrument can often be difficult to grasp. There's no substitute for really understanding your application's code, but there are some signals that can serve as 'smells'.

If your APM surfaces GC runs or allocations over time, you can look for spikes to see if they match up with certain endpoints being hit or certain running background jobs. Here's another example from AppSignal's Ruby VM magic dashboard:

By looking at allocations in this way, we can narrow down our search when looking into memory problems. This makes it much easier to use tools like memory_profiler and derailed_benchmarks efficiently.

Read about the latest additions to AppSignal's Ruby gem, like allocation and GC stats tracking.

Wrapping Up

In this post, we dived into some tools that can help find and fix memory leaks, including memory_profiler, derailed_benchmarks, perf:mem_over_time, perf:objects, perf:heap_diff, the event timeline and allocated objects graph in AppSignal.

I hope you've found this post, alongside part one, useful in diagnosing and sorting out memory leaks in your Ruby app.

Connect a Ruby on Rails App with React in a Monolith

Roy Tomeij — 2022-08-03T00:00:00+00:00

More and more people are using Ruby on Rails to create a back-end API application for a front-end app.

But what if you want to create a rich and functional interface with JavaScript and use Rails for the back-end, without having them in separate repositories? You can create a monolithic Rails application.

This article will show you how to connect a Rails application with a front-end developed in React (without splitting the code into two separate applications). We'll also give some alternatives if React is not your framework of choice, but you like the idea of a monolithic app with a rich front-end.

But first, let’s start with an architecture overview to see why linking Rails and React is an option worth considering.

App Architecture: An Overview

A while back, when a developer started to build a web application, they didn’t have a list of possible architectures they could use. For the most part, web apps were monoliths without a very interactive user interface.

In modern software development, we have much more to choose from. We can:

Use the old monolith architecture
Go for a separated back-end and front-end
Use service-oriented architecture

Let’s take a closer look at the most common types of architecture.

Headless Architecture

In headless architecture, the head of an application is detached from its body. In other words, you create a back-end application using Ruby, Python, Node.js, or another programming language. This application manages a connection with a database and provides computing power.

The body is a front-end application created with a framework like React, Vue, or Angular.

CDN stands for content delivery network, a service designed to deliver assets faster for visitors worldwide. We can build the architecture in the cloud environment, or each setup piece can be a separate server.

Choose headless architecture when:

You aim for a large codebase. If your application is going to be very large, the separate back-end and front-end layers will make it more maintainable.
You expect different workloads for different system elements. With a more modular approach, you can scale modules separately.
You foresee that you'll change the technology in the future. It is easier to adopt new technology with a headless architecture, as the communication, in most cases, will be the same between the back-end and front-end, regardless of the stack.

Avoid a headless approach when:

You want to develop an MVP version of an app very quickly. You will likely start over again in the future and consider a different angle.
You have a very small team. Finding the right person to do the back-end, front-end, and DevOps stuff simultaneously might be hard.
You want to build a straightforward application. Unfortunately, the headless approach increases your app's complexity.

Monolith Architecture

One application handles the presentation layer (front) and computation layer (back) in a monolith architecture. Such an approach speeds up the application creation process and simplifies the server setup.

As you'll notice, in a monolith, we eliminate a part of the communication that's in the headless approach. Such architecture improves the performance and security of an application.

Choose this architecture when:

You develop an MVP version of your app. Frameworks like Ruby on Rails or Django make it easy and fast to build robust monolith applications.
You start with a small team. Back-end developers can easily handle the front-end in such an architecture, and the deployment process is straightforward.
Your application doesn’t have to depend on a very interactive front-end, and you don’t want to build a single-page application.

Avoid the monolith approach if:

You know that the front-end of the application will be huge. It is harder to maintain this part as it's connected directly with the back-end codebase.
You know the front-end will have a completely different workload than the back-end. As a result, you won’t scale certain elements of the monolith easily.
You may change the front-end or back-end tech stack in the future. Such a transition would be pretty complicated with this architecture.

Hybrid Architecture

An alternative approach to the monolith and headless architectures is a hybrid. You create a monolith application, but instead of using the front-end produced by the back-end framework, use the technology of headless applications.

Yet this comes with a few disadvantages which you should consider:

The codebase is shared. So as the application grows, it is harder to navigate through all the files and build a structure that is easy to understand.
Scaling only the back-end or front-end is very difficult as both parts connect in one application.
Being agile is not easy. With every change, you deploy the whole application, even if it’s just a change in the front-end style.

However, you also get some benefits that you wouldn’t get with a standard monolith architecture:

You can work separately on the back-end and front-end and update libraries independently. Changing the stack of the front-end is easier.
You can build a very interactive, single-page application based on the monolith with a reasonably simple deployment process.
You can be flexible. It's possible to use the front-end served by the JavaScript framework for some parts of your application.

With these pros and cons of hybrid architecture in mind, we will now go through the design process of a monolith application (created with Rails and a front-end React framework).

Designing a Monolith with React

There are three main things to keep in mind before you build a monolith application with a React front-end:

Front-end framework installation process - there are a few ways of adding React to a Rails application. However, each has some limitations, so choosing the right one for your project is essential.
Communication inside the application - The back-end needs to expose data to the front-end, and the front needs to present this data. This element of the system needs to be designed carefully to make information exchange as smooth and fast as possible. As always, your choice should depend on the type of application you would like to build.
Alternatives - You can still go for the hybrid approach but not use React. We will show you other solutions that you can quickly adapt to build a more interactive monolith.

Install React in Your Ruby on Rails App

There are three main ways to install React in your Rails application.

Install React Using Ruby Gems

We can extend the functionality of our app by installing external libraries called Ruby gems. If you are unfamiliar with JavaScript, this is the fastest way to add React to Rails.

The react-rails library is one of the most popular Ruby gems to integrate React with Rails. It provides generators for components, testing helpers, and view helpers to render JavaScript code inside the views.

If you accept another layer of logic for your front-end, using this Ruby gem is the best way to get started with React quickly. However, remember that besides the JS framework updates, you also have to depend on the gem updates. The more gems you use, the more problematic the Rails upgrade process can become.

Install React Using Import Maps

Import maps were first presented in Rails 7. The lib lets you build modern JS applications using libraries made for ES modules without transpiling or bundling.

Installing React with import maps is as easy as calling this command from the terminal:

$ ./bin/importmap pin react react-dom

You don’t even need to host source files on your server, as they are served from JavaScript’s CDN by default. By applying the --download flag, you can download the files to the vendor directory inside your application.

However, you cannot use JSX (the most popular rendering extension in the React community) with import maps. HTM library is an alternative solution.

Install React Using Package Manager

This option seems to be the most flexible and optimal way to add React to a Rails project. You don’t create another level of abstraction for the front-end as you fetch the library directly. Updating React is also simpler.

The most popular package managers in the Rails community are NPM and Yarn. You can add React libraries by invoking this command in the terminal:

yarn add react react-dom # or npm install react react-dom

With this installation method, you won’t get any Rails helpers to render JavaScript code. However, we'll show you how to install and run React components using this approach shortly.

Communication Between Your Ruby on Rails App and React

Choosing the right way to exchange information between your Rails application and React components is essential. Selecting the wrong method can harm your application's performance and make your code less maintainable.

There are two common, standardized ways of exposing information from the back-end — REST and GraphQL. Let's take a look at both options.

REST Communication

The REST API is a pretty simple way of exchanging information. We have different request types at our disposal to perform CRUD (create, retrieve, update, delete) operations. If you are familiar with the concept of resources in Rails, the REST API works the same way.

You call the /authors endpoint if you want to pull information about authors or a single author. You call the /posts endpoint to do the same, but for posts. If a single page in your application requires a lot of different data, you will perform multiple HTTP requests.

You should consider using REST in your application if:

Your front-end application doesn’t need to render data from different sources on a single page. On the other hand, this doesn’t have to mean that the application is very simple.
You don’t want to implement the cache mechanism on the application level. Using REST API, you can benefit from the HTTP cache provided by browsers.
You care about error reporting. Implementing an error report with REST is easy as you work on different HTTP responses. With GraphQL, it’s not, as your application always returns a 200 response status.

GraphQL Communication

GraphQL represents an entirely different approach to data fetching. Using this technology, you perform one request and get only the data you need. You can hit multiple endpoints at once and pull only one attribute per endpoint.

With GraphQL implemented on the back-end, you always call the same endpoint and modify the query parameter.

You should consider using GraphQL in your application if:

You are building a very interactive application requiring the front-end to pull a lot of different data at once.
You want to build your application very quickly. Building an API with REST is slower. With GraphQL, you get maximum flexibility, and all you need to care about in every step is the content of the query.
You are building an application where the API is used by multiple different applications like mobile, front-end, or services.

Front-end Alternatives to React for Your Ruby on Rails App

If you don’t want to use React but like the idea of having a front-end application inside the Rails monolith, there are some alternative solutions available.

Rails Turbo

Turbo allows you to write single-page applications without the need for any JavaScript code. You design the page from independent frames. They behave like components and can also be lazy-loaded if needed.

With Rails Turbo, you don’t spend time building the JSON responses from an API. You can simply reuse Rails views and render HTML — it’s automatically transferred and rendered in your components. This reduces the amount of code in your application.

You should consider using Rails Turbo if:

You don’t like writing JavaScript. It is possible to design rich and interactive web applications without the need for a single line of JavaScript.
You like the Rails framework. The Turbo library provides helper methods to quickly build interactive pages or transform existing ones into more dynamic forms.
You want to move fast. As mentioned before, you don’t have to care about the JSON responses from an API. You can simply serve HTML views and the Turbo library will automatically pull them.

Stimulus JS

Stimulus is a JavaScript framework created by the Rails team. It was designed to extend the HTML that you already have. The main concept of this framework is the controller — one controller per view.

Your Rails application will automatically load the JavaScript controller for the given view and map the HTML elements. You can respond to events or modify the view when you need to.

You should consider using Stimulus if:

You are already using Rails Turbo, but want more control over a page's granular elements.
You don’t want to build a single-page application but still want to make some page elements interactive or dynamic.

Other JavaScript Frameworks

Besides React, other interesting JavaScript frameworks on the market right now include Vue, Svelte, and Ember.js. So if you like writing JavaScript components, you can choose one of these.

Rails Turbo and/or Stimulus JS are the perfect choice if you're a Rails developer who's familiar with JavaScript but doesn’t want to write much of it. Both libraries heavily follow the principles of the Rails framework, so if you already work in this ecosystem, it will be easier for you to use these extensions.

Build a Monolith with React

It’s time to build a simple Rails application to manage a list of books we want to read. Let's start with generating the Rails skeleton.

Ruby on Rails Project Bootstrap

Ensure you have the latest version of Ruby on Rails installed locally on your operating system. Then generate the skeleton using the rails new command:

rails new booklist -d postgresql -j esbuild

The above command will generate a new Rails project in the booklist directory, with support for a PostgreSQL database. We can now enter the project’s directory, create the database, and run the server:

cd booklist/
./bin/rails db:create
rails s

Visit the localhost:3000 address in your browser to see the default Rails welcome screen.

Now generate a home controller for a single endpoint in the application so we can attach the React application to the layout. Run this command in the terminal:

./bin/rails g controller Home index

The last step is to update config/routes.rb file to set the root path:

Rails.application.routes.draw do
  root "home#index"
end

We can now focus on the front-end part with React.

React Installation

We will install the react, react-dom, and node-uuid libraries using yarn in our terminal:

yarn add react react-dom node-uuid

To properly handle JSX, we also have to update our build script. First, open the package.json file, and in the scripts section, ensure that the build command is the following:

esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --public-path=assets --loader:.js=jsx

You can now run ./bin/dev to kickstart both yarn build and the rails server or run them separately. For example, if you want processes in separate terminal tabs, run rails s in one and yarn build –watch in the second.

Render a React Application in a Rails Layout

We want to run the React application when a user enters the main root in our Rails application. To do this, update the app/views/layout.html.erb file, so that the body section looks like this:

<body>
  <div id="app"></div>
</body>

The React application will render the component with JavaScript logic inside the app div.

Let’s now create the entry point for the React application. Open the app/javascript/application.js file and write the following code:

import React from "react";
import { createRoot } from "react-dom/client";
import App from "./App";

const container = document.getElementById("app");
const root = createRoot(container);
root.render(<App />);

We have the container element, but we are missing the App component so the application will throw an error. Let’s create the App component then.

Writing a React Component

We won’t create a separate directory for components. Let's place our tiny application in one component for demonstration purposes.

Note: When writing a production app, always design the React app using multiple components and follow the best design principles.

Create a file App.js in the app/javascript directory and place the application skeleton there, with the header as a placeholder:

import React, { useState } from "react";

export default function App() {
  const [books, setBooks] = useState([]);

  return (
    <>
      <h1>Books {books.length}</h1>
    </>
  );
}

Rendering a Book List

Now it’s time to render a simple list of books. To do this, we have to update the render function:

return (
  <>
    <h1>Books {books.length}</h1>
    <div>
      <table>
        <thead>
          <tr>
            <th>Title</th>
            <th>Author</th>
          </tr>
        </thead>
        <tbody>
          {books &&
            books.map(({ id, title, author }, i) => (
              <tr key={id}>
                <td>{title}</td>
                <td>{author}</td>
              </tr>
            ))}
        </tbody>
      </table>
    </div>
  </>
);

Create a New Book

We don’t have anything we can render. Let’s add the form to create a new book so we can fill our list with some data that we can later edit and delete if needed.

However, before we do this, we have to update our component to store some more information:

import { default as UUID } from "node-uuid";

const [action, setAction] = useState("list");
const [formData, setFormData] = useState({ title: "", author: "" });

I’m not going to create a separate component for the form, so I have to use a conditional in the render function. Thanks to the action, I can tell when to render the list or form.

Our render function is the following:

return (
  <>
    <h1>Books {books.length}</h1>
    {action == "list" ? (
      <div>
        <button onClick={() => setAction("form")}>New book</button>
        <table>
          <thead>
            <tr>
              <th>Title</th>
              <th>Author</th>
              <th></th>
            </tr>
          </thead>
          <tbody>
            {books &&
              books.map(({ id, title, author }, i) => (
                <tr key={id}>
                  <td>{title}</td>
                  <td>{author}</td>
                  <td></td>
                </tr>
              ))}
          </tbody>
        </table>
      </div>
    ) : (
      <div>
        <form>
          <label>Title:</label>
          <input
            onChange={(e) =>
              setFormData({ ...formData, title: e.target.value })
            }
            name="title"
            value={formData.title}
          />
          <label>Author:</label>
          <input
            onChange={(e) =>
              setFormData({ ...formData, author: e.target.value })
            }
            name="author"
            value={formData.author}
          />
          <button onClick={(e) => saveBook(e)}>Submit</button>
          <button onClick={() => setAction("list")}>Back</button>
        </form>
      </div>
    )}
  </>
);

Each time you type something into the title or author input, formData will update with values from the form. The last missing piece is the saveBook function:

const saveBook = (e) => {
  e.preventDefault();

  setBooks([...books, { ...formData, id: UUID.v4() }]);
  setFormData({ title: "", author: "", id: "" });
  setAction("list");
};

The first step is to prevent a form submission — we don’t want to reload the page. Then we update the books collection and reset the form data. The last step is to render back the list view.

Update an Existing Book

Let's reuse the form we created in the previous step. We also have to store the id of the book we are currently editing:

const [currentBookId, setCurrentBookId] = useState(null);

The editBook function will set the id of the book we want to edit. Fill the form with values and render the form view:

const editBook = (id) => {
  const currentBook = books.find((book) => book.id == id);
  setCurrentBookId(id);
  setFormData({
    ...formData,
    title: currentBook.title,
    author: currentBook.author,
  });
  setAction("form");
};

Since we are going to use the saveBook function for a creation and update action, we have to modify it accordingly:

const saveBook = async (e) => {
  e.preventDefault();

  if (currentBookId) {
    bookIndex = books.findIndex((book) => book.id == currentBookId);
    updatedBooks = [...books];
    updatedBooks[bookIndex] = formData;
    setBooks(updatedBooks);
    setCurrentBookId(null);
  } else {
    setBooks([...books, { ...formData, id: UUID.v4() }]);
  }

  setFormData({ title: "", author: "", id: "" });
  setAction("list");
};

Delete a Single Book

Let’s start with building the simple code for destroying the book:

const deleteBook = (id) => {
  setBooks(books.filter((book) => book.id != id));
};

We get the array of books that aren't deleted and update the books collection. React will update our component automatically.

The Complete Component

We implemented all the actions needed to create, update, list, and destroy a book. View the code for the complete component in this Gist.

Recap

Congratulations, you created a React application inside a Rails monolith! Let’s summarize what we have learned and done during this article.

Different Types of Architecture for Web Apps

When you build a web application, it has a front and back part. So you need to decide how you want to connect these two sides. Here's a reminder of the most common types of architecture:

Headless architecture - The front-end application is separate from the back-end. Choose it if you expect to write a lot of code, want to change technologies in the future, or expect a different workload for the front-end and back-end. However, keep in mind that such architecture increases the complexity of an application, takes more time to build, and the development process might be complex for a small development team.
Monolith architecture - One application that handles both front-end and back-end. You can build one with popular frameworks like Ruby on Rails or Django. With this architecture, you can build fast with a small team. However, you can run into trouble in the future if you want to change the technology or scale only part of the application.
Hybrid architecture - A compromise between a monolith and headless architecture. You build one codebase, but the front-end is served by a different technology from the back-end. The codebase might be hard to navigate, and it will be challenging to scale only one piece of the system. However, it is more flexible than a monolith. If you don’t like monoliths, but don’t want to fully detach the front from the back-end, this solution is for you.

The Design Process to Build Hybrid Architecture

Don’t jump into coding straight away. Think about the specifications of your application. When building a Rails application that uses a React framework for the front-end, you have to consider the following things:

Installation process - You can install React using import maps if you don’t need to use JSX templates. Otherwise, you can select package managers like Yarn or NPM, or install React using a Ruby gem.
Communication between Rails and React - Both applications are inside the same codebase, but need a bridge to exchange information. Use REST API for simpler applications and GraphQL for more complex interfaces where a lot of different data is necessary for the view layer.

Alternatives to React for a Rails App

If you decide that you don’t want to use React, but you like the idea of hybrid architecture, you can consider the following alternative solutions for the front-end:

Rails Turbo - With this library, you can split the page into independent frames and update them when needed. Go for this option if you like the Rails framework, and don’t want to write a lot of JavaScript code.
Stimulus JS - A small library for the HTML you already have in your views. Choose it when you are happy with your Rails monolith, but you want to make some views more interactive.
Other JavaScript frameworks - If you want to write more JavaScript, you can use Vue.js, Svelte, or Ember.js, instead of React. You can easily incorporate these solutions into your Rails application.

Summing Up

In this post, we explored the three main types of architecture — headless, monolith, and hybrid — and looked at their pros and cons. We then created a React application inside a Rails monolith.

Now it should hopefully be easier for you to choose the right architecture and tools for your next project.

Until next time, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Track Down Memory Leaks in Ruby

Roy Tomeij — 2022-07-27T00:00:00+00:00

A memory leak is an unintentional, uncontrolled, and unending increase in memory usage. No matter how small, eventually, a leak will cause your process to run out of memory and crash. Even if you periodically restart your app to avoid this crash (no judgment, I've done that!), you still suffer the performance implications of a memory leak.

In this post, the first of a two-part series on memory leaks, we'll start by looking at how Ruby manages memory, how Garbage Collection (GC) works, and how to find a leak.

In the second part, we'll take a deeper dive into tracking down leaks.

Let's get started!

Ruby Memory Management

Ruby objects are stored on the heap, and each object fills one slot on the heap.

Prior to Ruby 3.1, all slots on the heap were the same size — 40 bytes, to be exact. Objects too large to fit in a slot were stored outside the heap. Each slot included a reference to where objects were moved.

In Ruby 3.1, variable width allocation for String objects was merged. Soon, variable width allocation will be the norm for all object types.

Variable width allocation aims to improve performance by improving cache locality — all the information of an object will be stored in one place rather than across two memory locations.

It should also simplify (some parts) of memory management. At the moment, there are two 'heaps':

The Ruby heap (or GC heap) that stores smaller Ruby objects.
The C heap (or malloc/transient heap) that stores larger objects.

Once variable width allocation is the norm, there should be no need for the latter heap.

The heap starts at a given size (10,000 slots by default) and objects are assigned to free slots as they are created. When Ruby tries to create an object and there are no free slots available, Garbage Collection (GC) occurs to make some free slots available.

If there are too few free slots after GC, the heap will be expanded (more on this a little later).

Here are the factors you can control, alongside their environment variables:

Initial size of the heap - RUBY_GC_HEAP_INIT_SLOTS
Number of free slots that should be available after GC occurs - RUBY_GC_HEAP_FREE_SLOTS
Amount the heap is expanded by - RUBY_GC_HEAP_GROWTH_FACTOR

Garbage Collection in Ruby

Garbage Collection in Ruby 'stops the world' — no other process occurs when GC occurs. Garbage Collection in Ruby (since 2.1) is also generational, meaning that the garbage collector has two modes:

Minor GC - inspects 'young' objects (objects created recently)
Major GC - inspects 'old' objects as well as 'young' objects (all the objects)

Note: An 'old' object has survived 3 GC runs, major or minor.

When the heap is full, minor GC is invoked first. If it can't free up enough slots to be below the limit, major GC will be invoked. Only then, if there are still not enough free slots, will the heap be expanded.

Major GC is more expensive than minor GC because it looks at more objects.

The theory behind why generational GC is more performant is that objects usually fall into two categories:

Objects that are allocated and then quickly go out of scope. In a Rails app, models fetched from the DB to render a page will go out of scope when the request ends.
Objects that are allocated and kept around for a long time. Classes and caches are likely to still be in use throughout the lifetime of an app.

Major GC will also run after minor GC if the number of old objects is above a certain threshold, even if there are sufficient free slots. This limit increases as the size of the heap grows and can be controlled by the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR environment variable.

When you have a leak, you create objects that can't be cleaned up — more and more old objects. This means that major (expensive) GC will run much more often than it should. Since nothing else runs when GC is running, this is time that you waste.

I've left some links at the end of this article for further reading on memory layout and the garbage collector in Ruby.

What Does A Memory Leak Look Like in Ruby?

You can see a memory leak using simple tools available on any Unix system. Take the following code as an example.

# leaky.rb

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts an_array.size
  sleep 1
end

To say this code 'leaks' is a little unfair — all it does is leak! — but it serves our purposes.

We can observe the leak quite simply from the command line by running this program in one terminal and watch-ing the memory increase over time with ps.

# In terminal one
$ ruby ./leaky.rb

# In terminal two
$ watch ps -p `pgrep -f "ruby ./leaky.rb"` -o pmem,pcpu,rss,args

The pgrep -f "ruby ./leaky.rb" finds the process ID for us, so that we can restrict the ps output to only the process we're interested in. As you may be able to guess, it's like grep for processes.

The watch tool allows us to poll the output of a given command and update it in place, giving us a live dashboard within our terminal.

You'll get output like this, which updates every couple of seconds.

Every 2.0s: ps -p 50866 -o pmem,pcpu,rss,args

%MEM  %CPU    RSS ARGS
 0.2   4.1 163408 /Users/tonyrowan/.asdf/installs/ruby/3.1.1/bin/ruby ./leaky.rb

You should see the %MEM and RSS increasing. They are:

%MEM - The amount of memory the process uses as a percentage of memory on the host machine.
RSS (resident set size) - The amount of RAM the process uses in bytes.

This basic OS-only information is enough to spot if you have a leak — if the memory keeps going up, it means you do!

Find Ruby Leaks with the Garbage Collector Module

We can also detect leaks within Ruby code itself with the GC module.

# leaky.rb

GC.disable # Only run GC when manually called

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts "Array is #{an_array.size} items long"

  GC.start # Run a major GC - use full_mark: false for minor GC
  puts "There are #{GC.stat(:heap_live_slots)} live objects"

  sleep 1
end

The GC.stat method will return a hash with a lot of useful information. Here, we're interested in :heap_live_slots, which is the number of slots on the heap that are in use. That's the opposite of :heap_free_slots. At the end of the loop, we force a major GC and print out the number of used slots, i.e., the number of objects that remain after GC.

When we run our little program, we see this increase ad infinitum. We have a leak! We could also have used GC.stat(:old_objects) to the same effect.

While the GC module can be used to see if we have a leak and (if you're smart with your puts statements) where the leak might be occurring, we can see the type of objects that might be leaking with the ObjectSpace module.

# leaky.rb

GC.disable # Only run GC when manually called

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts "Array is #{an_array.size} items long"

  GC.start # Run a major GC - use full_mark: false for minor GC
  pp ObjectSpace.count_objects

  sleep 1
end

The ObjectSpace.count_objects method returns a hash with the counts of live objects. T_STRING, for instance, is the number of strings live in memory. For our rather leaky program, this value increases with each loop, even after GC. We can see that we are leaking string objects.

Application Performance Monitoring in Production with AppSignal

While playing with ps and GC can be a sensible route for toy projects — they're also fun and informative to use! — I would not recommend them as your memory leak detection solution in production apps.

This is where you would use an Application Performance Monitoring (APM) tool. If you're a very large company, you can build these yourself. For smaller outfits, though, picking an APM off-the-shelf is the way to go. You do need to pay a monthly subscription, but the information they provide more than makes up for it.

For detecting memory leaks, you want to find server or process memory use (sometimes called RSS) graphs over time. Here's an example screenshot from AppSignal's 'process memory usage' dashboard of a healthy app shortly after being deployed:

And here's an unhealthy app after deployment:

AppSignal will even surface Ruby VM stats like GC and heap slots, which can give you an even clearer signal for a memory leak. If the number of live slots keeps growing, you have a leak!

Wrap Up and Further Reading

In this post, we took a quick tour of Ruby's memory management and garbage collector. We then diagnosed how to discover a memory leak using Unix tools and Ruby's GC module.

Next time, we'll see how to use memory_profiler and derailed_benchmarks to find and fix leaks.

In the meantime, you can read more about the tools we used:

Additional further reading:

Happy coding, and see you next time!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Deploy Your Ruby on Rails App Using Capistrano

Roy Tomeij — 2022-07-13T00:00:00+00:00

In this article, we will configure Capistrano in a Ruby on Rails application. We will then deploy the app to a cloud instance that runs Ubuntu as an operating system, independent of your hosting provider. You can use any cloud service, or even an on-premises server, to test or replicate the steps we'll take.

Once we've deployed the app, we'll look briefly at how you can monitor your app's deployments using AppSignal.

But first, you might ask: why should I use Capistrano in the first place?

Why Choose Capistrano for Your Ruby on Rails App?

You might be wondering if it still makes sense to continue using Capistrano for deployment these days. We now have many tools and services available for deployments and continuous integration.

But to answer that, we need to revisit the origin of Capistrano and its evolution to the present day. The first tag created in the Capistrano repository was in 2006. Its objective was to allow the execution of commands remotely via SSH in parallel on several machines, all using DSL. Since then, Capistrano has evolved, by adding several features to each version.

Currently at version 3 after 16 years, Capistrano continues to be actively developed and is now known as a framework for building automated deployment scripts.

So it's no longer just a tool to run commands on remote machines. In addition to Capistrano's features, many plugins and other gems specifically run with Capistrano.

Because of this and the fact that a large community still uses Capistrano, it remains a great tool for automated deployment.

Adding Capistrano to Your Ruby on Rails App

First, you'll need to configure a Ruby on Rails application to integrate it with Capistrano. You can follow Capistrano's documentation. However, it has more information than you'll need for the scope of this post. So for simplicity's sake, this article only includes the necessary steps and configuration that you will need.

To start the configuration, add the Capistrano gem in the development section of your Gemfile:

group :development do
  # Deployment
  gem "capistrano", "~> 3.10", require: false
  gem "capistrano-rails", "~> 1.3", require: false
end

You need the capistrano-rails gem to run a migration on your database and handle the assets that we will see in the next sections.

Now run bundle install to add Capistrano to your application:

bundle install

Then you can install Capistrano:

bundle exec cap install

This command will create a file called Capfile that imports and configures the required libraries. Furthermore, a file called deploy.rb (created inside the config folder) will contain all the configurations that should be included in a deployment with Capistrano.

Note: Capistrano creates a file to override the deploy.rb configuration for both the staging and production environments, but that will not be covered in this article.

With that, you're ready to start configuring and communicating between your Ruby on Rails application and the server.

Server Configuration with Capistrano for Your Rails App

In this step, we'll add instructions to the Capistrano configuration file. The instructions will communicate to your server on where your app will be deployed. This is where we'll define access information and the security configuration.

To ensure that your production environment runs the same version of Ruby as the version you use in development, use a Ruby version manager, such as RVM or rbenv (Capistrano supports both).

In this article, we'll use RVM. Make sure you have RVM installed on your development machine and production server. Also, check if your version of Ruby is the same in your environments and project.

Adding `capistrano-rvm` Gem

The Capistrano project provides a gem called capistrano-gem that lets you easily configure RVM. To include it in your project, add it to your Gemfile within your development group, and run bundle install again.

gem "capistrano-rvm"

Update your Capfile with the capistrano-rvm import:

# Capfile
require 'capistrano/rvm'

In the deploy file, add a configuration to set the Ruby version and the path to RVM (especially if you used a different path to install RVM).

If no version of Ruby is defined, Capistrano chooses the latest version that you have installed on the server where your application will be deployed.

# config/deploy.rb
set :rvm_ruby_version, "ruby-2.6.3"
set :default_env, { rvm_bin_path: "~/.rvm/bin" }

Capistrano::RVM has other settings for RVM, but the ones shown here are what you'll need to get started.

Set Up Your Server Configuration

It is good practice to define a user for the deployment. Don't forget to follow these steps on your server before you start a deployment:

Add the public key generated on your local machine to the ~/home/YOUR_USER/.ssh/authorized_keys file.
Create a key and include the public key in the code repository. This gives you access to clone your project within the production server. If you are using GitHub, you can follow this documentation.
Set permissions to write to the /var/www/ directory.

You can configure Capistrano for any environment you need. You just need to create a new configuration file inside config/deploy/<environment_name>.rb, then configure the IP or DNS, and username, to connect to your server.

# config/deploy/production.rb

server "11.22.333.444", user: "ubuntu", roles: %w{app db web}

To complete the deployment setup, include information about your application, such as the application name and code repository URL. Any information which is the same for all environments must be added to config/deploy.rb. Keep specific environment configurations in config/deploy/<environment_name>.rb.

# config/deploy.rb

set :application, "ruby_rails_app"
set :repo_url, "git@github.com:username/ruby_rails_app.git"

Set Up `capistrano-secrets-yml` Gem for Your Rails App

You should also set up the capistrano-secrets-yml gem.

Add the gem to your Gemfile and run the bundle to install:

# Gemfile
gem "capistrano-secrets-yml"

Import it into Capfile:

# Capfile
require "capistrano/secrets_yml"

Then create config/secret.yml in your application and include the secret key base as an environment variable. This variable will be created on your production server. Remember not to commit this file to your repository.

# config/secrets.yml
production:
  secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>

Inside your application folder in the terminal, run this command to generate the secret key base:

$ rails secret

Copy the generated key and use it to set your SECRET_KEY_BASE on the production server. Access your production server and export the SECRET_KEY_BASE variable in the final ~/.bashrc file, so it is always available:

# ~/.bashrc
export SECRET_KEY_BASE="YOUR_SECRET_KEY_BASE"

Run the source to make it instantly available in your environment:

$ source ~/.bashrc

Now, back on your machine, access the application folder in the terminal and run the Capistrano command below to create config/secrets.yml on the production server:

$ cap production setup

All done! Now it's time to configure Capistrano to run a migration.

Configure Capistrano to Perform a Database Migration

Now it's time to configure Capistrano to run database-related commands.

The capistrano-rails gem already included in your Gemfile can be configured to perform a migration on each deployment. We just need to import it into Capfile:

# Capfile
require "capistrano/rails/migrations"

But if you want to run the seeds on deployment, you need to create a new task to run after the migration:

# config/deploy.rb

namespace :deploy do
  desc "Run seed"
  task :seed do
    on roles(:all) do
      within current_path do
        execute :bundle, :exec, 'rails', 'db:seed', 'RAILS_ENV=production'
      end
    end
  end

  after :migrating, :seed
end

With that, your app is finally ready to be deployed!

Start Deployment of Your Ruby on Rails App

Let's see how to perform a deployment from our local machine by directly updating our server to the new version of our application.

When you execute a deployment command, Capistrano will connect to your server. From there, Capistrano will try to clone the code from the repository defined in the configuration file (config/deploy.rb). Then other tasks will be performed, following the deployment flow.

Capistrano's deploy:check command validates that the Git configuration is alright and that Capistrano has correct access to the directories that will be used in the deployment.

To use it, pass the environment. In this case, we are using production:

$ cap production deploy:check

If everything is ok with the configuration, you can start your deployment to production:

$ cap production deploy

In addition, you can check all available tasks in Capistrano with the command:

$ cap --tasks

Now, in our final step, let's integrate our Ruby on Rails application with AppSignal and start monitoring it.

How to Integrate AppSignal with Capistrano and Your Rails App

AppSignal is a monitoring tool, so you might wonder why it's included in this article. That is a valid question, as monitoring is not directly related to deployments.

That said, AppSignal is a valuable tool when it comes to monitoring your Ruby on Rails application and keeping track of deployments in each environment. So let's see how we can set up AppSignal for your app.

You will need an AppSignal account, which you can create on AppSignal's sign-up page. Pick Ruby as your language and follow the steps to install AppSignal in your application.

Once installed, a file will be created to configure AppSignal. What's important here is to define the revision — information that will be checked after deployment to see if a new version of an application has been deployed. In this case, we will use the Git log as revision information to ensure that each new code deployment will notify AppSignal.

# config/appsignal.yml

production:
  <<: *defaults
  active: true
  revision: "<%= `git log --pretty=format:'%h' -n 1` %>"

It is also good practice to export AppSignal environment variables to your production server (especially the API key, so as not to keep it in the AppSignal configuration file).

export APPSIGNAL_PUSH_API_KEY=XXX

And now we can rerun the deployment. If everything is set up correctly, you will see a message like this at the end of the deployment:

Notifying AppSignal of deploy with: revision: ee5e626fd31613ef068873f9cddb5f8c91538396, user: monteirobrena
AppSignal has been notified of this deploy!

You will now be able to see your application in AppSignal! 🎉

And information about any deployments:

Visit AppSignal's deployment documentation page to learn more about deployment integration and other features available for Ruby on Rails applications.

Wrap Up and Next Steps

In this post, we explored how to use Capistrano to deploy your Ruby on Rails app. We first added Capistrano to an app, then configured and deployed it, before finally setting up monitoring with AppSignal.

If you'd like to explore further, you can configure Capistrano to run from an integration connected to your repository's pipeline. Tools like Bitbucket, GitHub, and GitLab provide settings to manage this process and can be used in conjunction with Capistrano.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Get Started with Hotwire in Your Ruby on Rails App

Roy Tomeij — 2022-07-06T00:00:00+00:00

Hotwire is a hot topic at the moment for every Rails developer. If you work with Rails, there is a good chance you have already heard a lot about it.

In this post, we'll look at the main components of Hotwire and how to use it in your Rails app. But first: what is Hotwire and why should you use it?

What Is Hotwire?

Hotwire is not a single library, but a new approach to building web and mobile applications by sending HTML over the wire. It includes Turbo, Stimulus, and Strada (coming later this year). We will discuss each of these in detail in the next section.

Side note: While Hotwire is highly linked with Rails, it is completely language-agnostic, so it can work just as well with other applications. I have been using Stimulus in production on several non-Rails apps and some static websites. You can use Turbo without Rails as well.

But let us come back to the Rails world for now.

Why Use Hotwire in Your Rails App?

So when should you use Hotwire? The answer is anywhere you want to add interactivity to your application. For example, if you want:

Some content to be displayed/hidden conditionally based on a user's interaction (e.g., an address form where the list of states automatically changes based on the selected country).
To update some content in real-time (e.g., a feed like Twitter where new Tweets automatically get added to the page).
To lazy-load some parts of your pages (e.g., inside an accordion, you can load the titles and mark the details to be lazy-loaded to speed up load times).

Hotwire Components

As mentioned before, Hotwire is a collection of new (and some old) techniques for building web apps.

Let's discuss each of these in the next few sections.

Turbo

HTML drives Turbo at its core. Turbo provides several techniques to handle HTML data coming over the wire and display it on your application without performing a full page reload. It is composed of:

Turbo Drive

If you have used Turbolinks in the past, you will feel right at home with Turbo Drive. At its core, some JS code intercepts JavaScript events on your application, loads HTML asynchronously, and replaces parts of your HTML markup.
Turbo Frames

Turbo Frames decouple parts of your markup into different sections that can be loaded independently.

For example, if you have a blog application, the content of your post and the comments are two related but independent parts of the page. You can decouple them so navigation works independently or even load them asynchronously with turbo frames.
Turbo Streams

Turbo Streams offers utilities to easily bring in real-time data to your application. For example, let's say you are building a news feed like Twitter. You want to pull new tweets into a user's feed as soon as they are posted without reloading the page. Turbo Streams allow you to do this without writing a single line of JS.
Turbo Native

Turbo Native lets you build a native wrapper around your web application. Navigations and interactions will feel native without you having to redo all the screens natively.

You'll keep delivering the rest of the application through the web. That way, you can focus on the really interactive parts of your application and get them right.

Stimulus

Stimulus is a JavaScript framework for writing controllers that interact with your HTML.

Let's say we need to add some JavaScript attributes like data-controller, data-action, and data-target to elements on a page. We'll write a stimulus controller with access to elements that receives events based on those attributes. Here's an example:

<div data-controller="clipboard">
  PIN:
  <input data-clipboard-target="source" type="text" value="1234" readonly />
  <button data-action="clipboard#copy">Copy to Clipboard</button>
</div>

It is very easy to get an idea about what this does without even reading the associated Stimulus controller.

Here's a controller that goes with the HTML:

// src/controllers/clipboard_controller.js
import { Controller } from "@hotwired/stimulus";

export default class extends Controller {
  static targets = ["source"];

  copy() {
    navigator.clipboard.writeText(this.sourceTarget.value);
  }
}

That is at the core of Stimulus: keeping things simple and reusable.

Now, if you ever need a copy-to-the-clipboard button on another page, you can just re-use that controller. Add the data-* attributes on the markup to get everything working.

Strada

Unfortunately, we don't know much about Strada yet. But it will allow a web application to communicate (and possibly perform actions) with a native app using HTML bridge attributes.

How to Use Hotwire in Your Ruby on Rails Application

I don't want to spend too much time discussing Hotwire installation or a basic use case. The Hotwire team has already done an excellent job of it in their Hotwire screencast. For full instructions, see turbo-rails installation and Stimulus installation.

Let's jump straight into some common Hotwire use cases.

Endless Scroll

Using Turbo Frames, we can easily make a page with automatic pagination as the user scrolls. For this, we need to do two things:

Render each "page" inside its own frame by appending the page number to the frame id (e.g., turbo_frame_tag "posts_#{@posts.current_page}").
Use a lazy frame for the next page so that it doesn't load automatically unless it comes into view.

<%= turbo_frame_tag "posts_#{@posts.current_page}" do %>
  <%= render @posts %>
  <% unless @posts.last_page? %>
    <%= turbo_frame_tag "posts_#{@posts.next_page}", :src => path_to_next_page(@posts), :loading => "lazy" do %>
      <%= render "loading" %>
    <% end  %>
  <% end  %>
<% end %>

Note that this example uses methods from Kaminari, but you can adapt it to any other pagination method.

We don't need anything special in the controller. A standard index method works:

class PostsController < ApplicationController
  def index
    @posts = Post.page(params[:page]).per(params[:per_page])
  end
end

The trick here is that we use nested frames, with the frame for the next page nested inside the frame for the previous page. That way, when the first page loads, the frame for the next page is placed at the end. When the user scrolls to that frame, it is replaced with the content of the second page. The lazy frame for the third page renders at the end.

Dynamic Forms

You can easily implement dynamic forms with Hotwire without custom logic for toggling fields on the front end. This is a bit more involved than the endless scroll use case, as it includes the use of both Turbo Stream and Stimulus.

Let's start with our form first.

<!-- app/views/posts/new.html.erb -->
<div data-controller="refresh-form" data-refresh-form-url="<%= refresh_form_posts_url(:target => "new_post") %>">
  <%= render "form" %>
</div>

<!-- app/views/posts/_form.html.erb -->
<%= form_for(@post, :data => { :target => "refresh-form.form" }) do |f| %>
  <%= f.select :kind, options_for_select([["News", :news], ["Blog", :blog]], @post.kind), {}, data: { action: "change->refresh-form#refreshForm" } %>

  <%= f.select :category, options_for_select(categories_for_kind(@post.kind), @post.category) %>
<% end %>

The form is simple enough — we display a kind select with News and Blog options. We want to change the available categories' values based on the kind that is selected (assuming that categories_for_kind(@post.kind) returns the list of categories for the given kind).

If you look closer, you'll see that we've added some data attributes to the form. The data-target will link the form element to the RefreshFormController Stimulus Controller's form target. And the data-action with the value of change->refresh-form#refreshForm will call the refreshForm method on the linked Stimulus Controller every time the kind select is changed.

Let's look at our Stimulus Controller:

// app/javascript/controllers/refresh_form_controller.js
import { Controller } from "stimulus";
import { put } from "@rails/request.js";

export default class extends Controller {
  static targets = ["form"];

  refreshForm() {
    put(this.data.get("url"), {
      body: new FormData(this.formTarget),
      responseKind: "turbo-stream",
    });
  }
}

On all refreshForm calls, we just make a new PUT request to the controller's URL (set using the data-refresh-form-url on the same element with a data-controller="refresh-form"). The important part here is that the responseKind is set to turbo-stream. The @rails/request library understands this response and performs instructions based on the response stream.

Now all that's left is to return the correct stream from our refresh_form call for Turbo to understand and update our form.

class PostsController < ApplicationController
  def refresh_form
    @post = Post.new
    @post.attributes = post_params
    @post.valid?
    respond_to do |format|
      format.turbo_stream
    end
  end
end

Just update the attributes on the post and mark that you want to respond in a turbo_stream format (so that it looks up refresh_form.turbo_stream.erb).

<!-- app/views/posts/refresh_form.turbo_stream.erb -->
<%= turbo_stream.replace params[:target] do %>
  <%= render "form" %>
<% end %>

In this step, we are reusing our form partial, wrapping it inside a turbo_stream with a replace action.

And that's all you need to get a dynamic form working. I know this looks a bit advanced, but the refresh stimulus controller is a shared part you can now use for all your dynamic forms by adding the correct data-* attributes. So essentially, you now get server-side dynamic form refresh without writing any new JS for other forms. Pretty awesome, right?

Append Content to Pages Without Reloading

The next use case that Hotwire makes easy is streaming HTML over a WebSocket connection and updating a page with new content as it comes in. A good example of this is the GitHub comments section. You can implement this very easily using Turbo Streams.

There are two parts to this.

First, we embed a turbo stream listener on the listing page that opens a WebSocket connection to the server and listens for events.

<!-- app/views/comments/index.html.erb -->
<div id="comments">
  <%= turbo_stream_from @post, :comments %>

  <% @comments.each do |comment| %>
    <%= render comment %>
  <% end %>
</div>

Next, we update the model to broadcast new comments to the stream.

# app/models/coment.rb
class Comment < ApplicationRecord
  belongs_to :post

  after_create_commit :stream

  private

  def stream
    broadcast_prepend_later_to(post, :comments, target: :comments)
  end
end

You don't need anything else. Turbo will automatically render the app/views/comments/_comment.html.erb partial for each new comment and send it over a WebSocket connection. It will be picked up by Turbo's JS and prepended to the target with id comments.

Let's go one step ahead and add an indication to all newly added comments with a small Stimulus Controller.

First, modify the broadcast and comment partial to include the controller conditionally.

# app/models/coment.rb
# ...
def stream
  broadcast_prepend_later_to(post, :comments,
                             target: :comments,
                             locals: { highlight: true })
end

<!-- app/views/comments/_comment.html.erb -->
<div <%= %s(data-controller="highlight") if local_assigns[:highlight] %>
  <%= comment.body %>
</div>

This small Stimulus controller adds a special highlight class on connection for 3 seconds and then removes it.

export default class extends Controller {
  connect() {
    this.element.classList.add("highlight");
    this.timeout = setTimeout(
      () => this.element.classList.remove("highlight"),
      3000,
    );
  }

  disconnect() {
    clearTimeout(this.timeout);
  }
}

Note: You also need to update the CSS highlighting based on the presence of that class.

Once this controller is done, you can re-use it on anything that requires a highlight class. You could even modify it to get the duration and class name from data attributes if you need that flexibility.

That's the great thing about Hotwire — it takes you a long way, and you don't have to dip your hands in JS. When you do need to write some JS, Stimulus gives you the tools to build small generic controllers that can be re-used.

Wrap Up and Further Reading

The Rails community has been really excited with the introduction of Hotwire, and rightly so.

In this post, we looked at the key components of Hotwire and how to use Hotwire in your Rails app. We touched on how you can bring your application to life using Turbo and Stimulus.

The official Hotwire screencast introduction and the Turbo documentation are great places to see what Hotwire and Turbo can do for you.

For advanced usage, I suggest heading over to the turbo-rails GitHub repo. Sadly, the documentation is a bit sparse, but if you are not afraid to get your hands dirty, read the code and inline comments in:

Turbo::FramesHelper for Turbo Frames.
Turbo::Broadcastable for broadcasting to Turbo Streams from the code.
Turbo::Streams::TagBuilder for broadcasting to Turbo Streams as part of inline controller actions.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

State Machines in Ruby: An Introduction

Roy Tomeij — 2022-06-22T00:00:00+00:00

A state machine can hold all possible states of something and the allowed transitions between these states. For example, the state machine for a door would have only two states (open and closed) and only two transitions (opening and closing).

On the other hand, complex state machines can have several different states with hundreds of transitions between them. In fact, if you look around, you will notice finite state machines surrounding you — when you buy from a website, get a pack of crisps from the vending machine, or even just withdraw money from an ATM.

In this post, we'll look at how to set up a state machine in Ruby and use the state machines gem.

State Machines in Development

When do we need a state machine in development? The simple answer is whenever you want to model multiple rules for state transitions or perform side-effects on some transitions. The key here is to identify parts of your application that would benefit from a state machine. A good example that always works for me is an Order in the context of an e-commerce application.

For a simple application selling products online, an Order can be in one of several states:

created
processing
ready
shipped
delivered
void

You can see the allowed transitions in the following diagram.

Visualizing the order in the form of a state machine immediately allows us to clearly understand the full flow, from ordering the product to delivery. And for us developers, it enables clear, set steps on what can and cannot be done at particular points in that flow.

That being said, it is easy to jump down a rabbit hole by picking this up for a very wide problem and then end up with hundreds of states that are difficult to reason about and follow through. So always have a top-level idea and a state chart for your proposed state machine before implementing it.

Our First State Machine in Ruby

Let's try to implement our OrderStateMachine with Ruby.

class OrderStateMachine
  def initialize(order)
    @order = order
    @order.state = :created if @order.state.blank?
  end

  def mark_as_paid!
    raise "Invalid state #{@order.state}" unless @order.created?

    @order.state = :processing
  end

  def packed!
    raise "Invalid state #{@order.state}" unless @order.processing?

    @order.state = :ready
  end

  def shipped!
    raise "Invalid state #{@order.state}" unless @order.ready?

    @order.state = :shipped
  end

  # ...
end

That's a simple enough implementation. For each possible transition in the state machine, we define a new method that does some sanity checks and performs the transition.

The great thing about the above implementation is that everything is explicit: any new developer can very quickly understand the full extent of the state machine.

Let's see how we can add some side-effects to the transitions. We'll automatically ship orders that are packed and deliverable:

class OrderStateMachine
  # ...

  def packed!
    raise "Invalid state #{@order.state}" unless @order.processing?

    @order.state = :ready
    ship_order if @order.deliverable?
  end

  # ...

  private

  def ship_order
    DeliveryService.create_consigment!(@order)
    shipped!
  end
end

While a naïve implementation works great, it is overly verbose, especially when we have a lot of transitions and conditions.

A quick check on Ruby Toolbox brings up several gems for state machines. state_machines and aasm are the most popular ones, and both come with an ActiveRecord adapter if you want to use them with Rails. Both are thoroughly tested and production-ready, so do check them out if you need to implement a state machine.

Using the State Machines Gem in Ruby

For this post, I will describe how we can model the state machine for our Order using the state_machines gem.

class Order
  state_machine :state, initial: :created do
    event :confirm_payment do
      transition created: :processing
    end
    event :pack do
      transition processing: :ready
    end
    event :cancel do
      transition %i[created processing ready] => :void
    end
    event :return do
      transition delivered: :void
    end
    event :ship do
      transition ready: :shipped
    end
    event :fail_delivery do
      transition shipped: :processing
    end
  end
end

The above class defines our simple state machine and all of its possible transitions. On top of this, it also automatically exposes a lot of utility methods on an order:

order.state           # => "created"
order.state_name      # => :created
order.created?        # => true
order.can_pack?       # => false (since payment has not been confirmed yet)
order.confirm_payment # => true (and transitions to `processing`)
order.can_pack?       # => true

Execute Side-Effects with the State Machines Gem in Ruby

As is usual for any real-world system, many transitions in a state machine will come with side-effects. The state_machines gem makes it easy to define and execute them. Let's add two side-effects:

On all transitions to ready, create a delivery consignment if the order is deliverable.
On all fail_delivery events, send an email to the user notifying them of the event.

class Order
  attr_accessor :deliverable

  state_machine :state, initial: :created do
    # ...

    after_transition to: :ready, do: :create_delivery_consignment, if: :deliverable
    after_transition on: :fail_delivery, do: :notify_delivery_failure
  end

  private

  def create_delivery_consignment
    DeliveryService.create_consigment!(self)
    ship
  end

  def notify_delivery_failure
    UserMailer.delivery_failure_email(self).deliver_later
  end
end

We can perform transitions as usual in the order:

order.deliverable = true
order.pack            # => true
order.state           # => "shipped" (through side-effect)

I would suggest checking out the Github page for the state_machines gem to learn about all the possible options the gem offers.

Use the State Machines Gem with ActiveRecord in Ruby on Rails

As discussed before, both state_machines and aasm support ActiveRecord. When you use the state_machines gem with ActiveRecord, not a lot of things change with the implementation. You can continue using most of what we've already discussed in the previous sections of this post.

Here are some additional features that you get:

Side-effects happen inside a transaction. This means that if you do some database operations inside transition hooks and the transaction fails, the state change won't be committed. See use_transactions if you want to disable this behavior.
Like all other ActiveRecord callbacks, if you want to update the attributes of the current model from a transition, you must do this inside a before_transition callback. To update attributes inside after_transition, you need to save the model again. Check out the following example:

class Order < ActiveRecord::Base
  state_machine initial: :created do

    before_transition any => :processing do |order|
      # You can just update the attribute here and it will be saved.
      order.processing_start_at = Time.zone.now
    end

    after_transition any => :shipped do |order|
      # Call update! to update the order.
      # Just setting the attribute like above would not work here.
      order.update!(shipped_at: Time.zone.now)
    end
  end
end

The gem automatically provides scopes to filter models by their state. For example, you can do Order.with_state(:processing) to find all the processing orders or Order.without_state(:processing) to find all orders that are not under processing.
If a state change is attempted without a matching transition (for example, from processing to delivered), the record will fail to save, and an error will be added to the validation errors. To internationalize the generated error messages, just add some keys to provide translations for states and events inside the config files (for example, en.yml):

en:
  activerecord:
    state_machines:
      order:
        states:
          created: Created
          processing: Under Processing
          # ...
        events:
          return: Return Order
          # ...

Side note: Check out this post to troubleshoot ActiveRecord performance issues.

Wrap Up: Build State Machines in Ruby

In this post, we explored why we'd use a state machine in development, before building a simple state machine. Finally, we looked at using the state machines gem in Ruby and Ruby on Rails.

Now that you have a basic understanding of a state machine, I am sure you will recognize several scenarios around you that already use a state machine or will benefit greatly from one.

Until next time, I wish you luck building state machines!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Add Feature Flags in Ruby on Rails with Flipper

Roy Tomeij — 2022-06-08T00:00:00+00:00

Picture this scenario: you are a Rails developer and have spent the last couple of days developing that awesome feature that everyone is waiting for. It's big and complex, but it went through rigorous testing, so you are confident everything works as it should. There are deadlines to meet, so you deploy. Immediately, all hell breaks loose.

Your feature straight up breaks the entire app for some of your users. It's hard to say why. No bugs showed up during testing. You revert your change, but the damage has been done. Your customers aren't happy, and you will spend the foreseeable future doing damage control and investigating what went wrong.

The problem is that releasing complex changes is inherently risky, no matter how much testing you do. It would have been nice if you had rolled this feature out gradually, maybe only to a handful of users first. It would have been even better if you could have switched the feature off as soon as the first problem showed up.

This is precisely where feature flags come in. In this post, we'll dive into the ins and outs of feature flags in Rails, including how to set them up using the Flipper gem. But first: what are they?

What Are Feature Flags in Ruby on Rails?

Feature flags are a way to enable or disable a feature in your application while it is running. If a flag is enabled, the feature is used. Otherwise, it is not. Imagine something like this in pseudocode.

if feature_enabled?(:my_awesome_feature)
  # Run awesome feature code
else
  # Not so awesome code
end

The name 'feature flag' is somewhat misleading. Controlling application behavior at runtime is not limited to features, but applies to any function within your program. You can use feature flags to try out performance tweaks and run A/B tests, for example. A significant difference between a feature flag and any other old conditional is you can modify feature flags at runtime.

For illustration purposes, let's imagine a simple feature flagging mechanism using environment variables.

def feature_enabled(feature_name)
  ENV[feature_name.to_s.upcase]
end

By modifying the value of the environment variable, we can switch parts of our application on or off at will. Which part of an application? Anything you want!

You could hide parts of the UI if a feature is disabled. You could switch out parts of your code, which is very useful if you aren't confident in your latest refactor. One of my favorite uses of feature flags is to hide entire routes using routing constraints:

# lib/constraint/feature_constraint.rb
class FeatureConstraint
  def initialize(feature)
    @feature = feature
  end

  def matches?(_request)
    feature_enabled?(@feature)
  end
end

# config/routes.rb
Rails.application.routes.draw do
  get 'statistics', to: 'statistics#index', constraints: FeatureConstraint.new(:statistics)
end

We can already see that there are lots of applications for feature flags. Our naive implementation has one significant weakness, though. A feature is either on or off for every user.

What if we want the ability to enable features only for specific users or groups of users, or only for some of the time? Let's take a look at Flipper.

Using the Flipper Gem for Feature Flags

Flipper is a gem that makes feature flags and different ways to toggle them available in Rails. It is highly modular. Apart from the main gem, you'll also have to pick a storage adapter — but more on that later. Let's use the ActiveRecord adapter for now.

Add the following lines to your Gemfile and run bundle install:

gem 'flipper'
gem 'flipper-active_record'

When we use the ActiveRecord adapter, Flipper will store your feature flag in the database and thus requires setting up new database tables. Flipper comes with a handy generator that will create the necessary migrations.

rails g flipper:active_record
rails db:migrate

You can use Flipper in the same way as the naive implementation we used previously.

if Flipper.enabled?(:my_awesome_feature)
  # Run awesome feature code but with Flipper!
else
  # Not so awesome code
end

The big difference is that instead of modifying environment variables, you now get to toggle features using Flipper's user interface. Add the Flipper UI gem and install it.

gem 'flipper-ui'

Modify your routes file to mount Flipper UI.

Rails.application.routes.draw do
  mount Flipper::UI.app(Flipper) => '/flipper'
end

If you start your application and navigate to the Flipper UI, you will see an overview of all available features and their status. Go ahead and create a my_awesome_feature feature.

Feature Toggles in Flipper

You've probably noticed that you can do much more than simply enable or disable a feature in Flipper UI. Flipper allows us to enable features for a subset of users or even for a percentage of the time. You can configure your feature flags using one of five different mechanisms.

One of them is the boolean toggle, a global on-off switch. The other four are a lot more interesting than that: groups, actors, percentage of actors, and percentage of time.

Groups

To enable a feature only for a particular group of users, use the group toggle. If it doesn't exist, create the file config/initializers/flipper.rb and add the following code to register a new group — for example, paid users.

# config/initializers/flipper.rb
Flipper.register(:paid_users) do |actor, _context|
  actor.respond_to?(:paid_users?) && actor.paid?
end

To use this new toggle, modify the Flipper invocation to pass the user — or actor, as the documentation likes to call them — and the feature name.

if Flipper.enabled?(:my_awesome_feature, user)
  # Run awesome feature only if the user is paid
else
  # Not so awesome code
end

Flipper will use the block you defined in the initializer to decide if the feature is enabled for this user. In this case, it will invoke the paid? method. You can now use this group in Flipper UI. Click the 'Add a Group' button and select your group.

Your group doesn't have to be paid users. It could be members of your team or users who have signed up for a beta program. Using the group toggle is helpful if you want to release something to a subset of users. Imagine beta releases or internal releases.

Actors

Sometimes, you might want to release something to a single individual. In this case, the actor toggle is something to consider.

You don't have to change anything in your initializer to use this mechanism. However, you'll need to ensure that your actor model — which will be your user model most of the time — implements flipper_id. Including Flipper::Identifier takes care of that.

# models/user.rb
class User < ApplicationModel
  include Flipper::Identifier
end

User.create.flipper_id # User;1

You can then use this ID to enable a feature for individual users.

This feature toggle is handy if you want to preview functionality for particular individuals. Think about enabling specific test accounts for partners who need access to a feature before it's released to a broader audience.

Percentage of Actors

If you want to ramp up feature usage slowly, the percentage of actors toggle is for you. Imagine you've implemented some performance optimization, and you're not quite sure how it will behave in production. You would roll this out to a small percentage of users first and then slowly ramp up the usage while monitoring your application.

If things go sideways, you can always reset the percentage to zero.

Percentage of Time

Last but not least, you may enable a feature for a percentage of requests. You do not need to provide an actor when using this toggle.

if Flipper.enabled?(:my_awesome_feature)
  # Run awesome feature code only for a percentage of requests
else
  # Not so awesome code
end

I like to use this to compare the performance of two different implementations. Note that this feature toggle is random. The same user will experience different behavior on each request.

Advanced Configuration in Flipper

We've already seen that we can use the ActiveRecord adapter to store feature flag information in our database. But depending on your specific needs, you might want to use a different adapter.

For the most part, all you need to do when swapping out storage adapters is to install the respective gem and configure Flipper accordingly. For example, if you want to use the Redis adapter, you add the flipper-redis gem to your Gemfile and update the Flipper initializer.

gem 'flipper-redis'

# config/initializers/flipper.rb
Flipper.configure do |config|
  config.adapter { Flipper::Adapters::Redis.new(Redis.new) }
end

Apart from configuring storage adapters, there are a lot of other aspects of Flipper that you can tweak.

Adding feature flags incurs a slight performance penalty, so you usually want to add memoization and caching. You'll likely also want to restrict access to the Flipper UI to authorized users. These topics are out of scope for this post, but I recommend you check out the Flipper documentation if you want to dive deeper.

Caveats to Feature Flags in Rails

Feature flags are great. But as with so many things, there are tradeoffs to consider.

Adding feature flags and maintaining them adds organizational overhead. Releasing and rolling out hidden features using flags is not as simple as deploying 'regular' code — you'll have to remember to turn it on, of course! That usually isn't too hard, but cleaning up feature flags that you don't need anymore might be.

Every time you add a feature flag to your code, you add a conditional, and if you don't watch out, your code quickly becomes littered with them. Keeping track of when and why you created certain feature flags could be challenging. Sadly, Flipper UI itself does not offer much to manage feature flags.

You should also be aware that using feature flags will incur a minor performance penalty. There are ways to mitigate this, but it can have a noticeable impact if you misconfigure or misuse Flipper.

Wrap Up: Get Started with Feature Flags in Ruby on Rails

We looked at how feature flags can help you release with more confidence, learned how they function in principle, and saw how you can get started with feature flags using the Flipper gem.

There are many situations where feature flags are handy. Depending on your use case, you might reach for one of the five different toggles — boolean, group, actor, percentage of actors, or percentage of requests.

Although working with feature flags has its caveats and might take some getting used to, they are nevertheless a great tool to have at your disposal.

Until next time, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Polymorphism in Ruby on Rails

Roy Tomeij — 2022-05-25T00:00:00+00:00

If you have ever spent time building an Object-Oriented Program (OOP), you have likely used polymorphism in your application or, at the very least, heard the term.

It’s the kind of word you’d expect to see in a science or computer science textbook. You may have spent time researching polymorphism and even implemented it in your application without clearly understanding the concept.

This article will give you a greater understanding of polymorphism, specifically in Ruby on Rails. To accomplish this, we’ll dive into:

The use of polymorphism in the real world
Polymorphism in programming by way of OOP
How you can incorporate it into your Rails application to help maintain high-quality code

Let's get going!

Polymorphism in the Real World

There are several ways to define polymorphism in different contexts. A useful definition, regardless of context, is 'an object's ability to display more than one form'. In fact, if we break down the word itself, poly means ‘many’ and morph means ‘form’.

In the real world, a basic example of this definition could be a woman who is also a police officer, sister, someone’s child, someone’s mother, etc. Each role determines her behavior and contributes to the person she is.

Polymorphism and Genetics

Outside computer programming, polymorphism is a term commonly associated with biology and genetics. In this context, polymorphism is more specifically defined as genetic variations that result in several distinct forms or types of individuals within a species.

Think of jaguars. Jaguars can have multiple gene variations, which can affect their fur coloring. Most jaguars have tawny coloring with black circles. However, they can have lighter or darker circles due to an altered gene, and some can have black fur coloring.

Different pigmentation within the same species of birds is another example of polymorphism. Consider the Gouldian finch, which has obvious distinctions in its coloring between individuals.

Monomorphism vs. Polymorphism

If we turn our attention to monomorphism, we can further understand polymorphism. Sticking with biology, monomorphism can be defined as 'a species with just one form', that maintains that same form during the various phases of its development.

Penguins are monomorphic. It is difficult, even for experts, to distinguish between the sexes. Gene differences in a penguin are minimal. Therefore, the physical attributes of penguins are almost indistinguishable, especially in terms of their size and black and white coloring.

Behavioral cues are often the easiest way to discern between the sexes in monomorphic species.

Let's turn our attention to polymorphism in programming, specifically.

Polymorphism in OOP

If we consider our initial definition of polymorphism — the ability of an object to display more than one form — we can seamlessly relate it to OOP.

In OOP, we can use the same method to produce different results by passing in separate objects. We could use conditionals to achieve this. However, this can create chunky code and may veer us away from DRY principles. Polymorphism is essential in creating clean and logical OOP applications.

Let's look at two examples of how polymorphism can be implemented in an OOP language like Ruby: through inheritance and duck-typing.

Polymorphism and Inheritance in Ruby

Inheritance is where a child class inherits the properties of a parent class.

Below is an example of how we can implement polymorphism with inheritance:

class Instrument
  def instrument_example
    puts "Saxophone"
  end
end

class Stringed < Instrument
  def instrument_example
    puts "Guitar"
  end
end

class Percussion < Instrument
  def instrument_example
    puts "Drums"
  end
end

all_instruments = [Instrument.new, Stringed.new, Percussion.new]

all_instruments.each do |instrument|
  instrument.instrument_example
end

# Output

# Saxophone
# Guitar
# Drums

The above code has two child classes — Stringed and Percussion — inherited from the parent class Instrument. This example is polymorphic, as we are calling a method: instrument_example — and it outputs multiple forms: Saxophone, Guitar, and Drums.

This example of achieving polymorphism through inheritance is essentially overriding a method, but helps provide a clearer understanding of polymorphism in an OOP language.

Duck-Typing and Polymorphism in Ruby

A more practical example of polymorphism in OOP is through duck-typing, as referenced below.

class Guitar
  def brand
    'Gibson'
  end
end

  class Drums
    def brand
      'Pearl'
    end
  end

  class Bass
    def brand
      'Fender'
    end
  end

  class Keyboard
    def brand
      'Casio'
    end
  end

  all_instruments = [Guitar.new, Drums.new, Bass.new, Keyboard.new]

  all_instruments.each do |instrument|
    puts instrument.brand
  end
  # Output

  # Gibson
  # Pearl
  # Fender
  # Casio

Though each class method is named brand, we don't override the method (unlike in polymorphic inheritance). Instead of inheriting from a parent class, here we have four independent classes, each with its own method. Duck-typing is useful, as we can just iterate through the classes to get each method's output (as opposed to calling each method separately).

Again, duck-typing is polymorphic as we call a method — brand — and generate an output that takes multiple forms: Gibson, Pearl, Fender, and Casio. Of course, duck-typing and polymorphism aren’t essential in producing this outcome. However, it’s very useful to implement clean and logical code.

Polymorphism in Ruby on Rails

Polymorphism works well in Ruby on Rails as an Active Record association. If models essentially do the same thing, we can turn them into one single model to create a polymorphic relationship.

Sticking with the instrument theme, let's consider an application where users can post, comment, and review instruments. Examine the Entity-Relationship Diagram (ERD) below:

In this example, we have an ERD for an application where a user can post an instrument with its details. A user can also provide a comment about that posted instrument.

Other users can then provide a rating of the instrument and rate comments to determine their usefulness or validity. These Active Record associations work just fine and serve the purpose of our application.

What if we wanted to add other associations to our application? We would need to add and repeat duplicate associations.

For example, if we wanted to add a user_rating model to rate the trustworthiness of a user, we would need to create a separate table with its own associations. This would mean adding a new relationship between the user and user_rating models. The ERD would then look something like this:

Now we have three models essentially doing the same thing: rating an object, but in different contexts. These associations are ripe for a polymorphic association.

Let's take a look at the ERD with the rating models as polymorphic:

As we have a rating column, I named the model reviews as opposed to ratings to avoid confusion. Here, the non-review models still have associations with the other models, but the separate rating models have been merged into a single review model.

reviewable_type and reviewable_id now take on the same role as the separate rating models by representing which model the review is associated with.

The reviewable_type column stores the model class name (user, instrument_post, or comment), and the reviewable_id stores the corresponding ID of that model.

We can now use these two columns to link the rating integer with a specific user, post, or comment via Active Record queries and/or conditional statements. The foreign key user_id remains in the review model, as this allows us to track which user left a review.

Right now, the term ‘-able’ in our polymorphic model may seem strange, but its purpose will soon be made clear when we do some Rails magic.

The review model is considered polymorphic as we have one model or object that can represent and take on multiple forms: user, comment, and instrument post reviews.

Implementing Polymorphism in Ruby on Rails

Time to implement polymorphism in a Rails application! If we act as though we have already created our user, instrument_post, and comment models, we can get started on incorporating our polymorphic model: reviews.

Firstly, create a table and generate the model from the terminal, like so:

rails g model Review user:belongs_to reviewable:references{polymorphic}

This builds the migration file:

class CreateReviews < ActiveRecord::Migration[7.0]
  def change
    create_table :reviews do |t|
      t.belongs_to :user, null: false, foreign_key: true
      t.references :reviewable, polymorphic: true, null: false
      t.integer :rating

      t.timestamps
    end
  end
end

The schema.rb file updates after the migration is run. The polymorphic option transforms the reviewable column into the reviewable_type and reviewable_id columns:

  create_table "reviews", force: :cascade do |t|
    t.integer "user_id", null: false
    t.string "reviewable_type", null: false
    t.integer "reviewable_id", null: false
    t.integer "rating"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.index ["reviewable_type", "reviewable_id"], name: "index_reviews_on_reviewable"
    t.index ["user_id"], name: "index_reviews_on_user_id"
  end

# app/models/review.rb

class Review < ApplicationRecord
  belongs_to :user
  belongs_to :reviewable, polymorphic: true
end

Remember when we mentioned the term ‘-able’ above? It is a Rails naming convention for designating our polymorphic association, giving us the ability to make a user, instrument post, and comment 'reviewable'.

For this Rails magic to work, we need to ensure that our other models are associated correctly with our polymorphic model.

# app/models/user.rb

class User < ApplicationRecord
  has_many :instrument_posts
  has_many :comments
  # alias association for user who submitted the review
  has_many :submitted_reviews, class_name: "Review", foreign_key: :user_id
  # association for user, instrument_post and comment that has the review
  has_many :reviews, as: :reviewable
end

# app/models/instrument_post.rb

class InstrumentPost < ApplicationRecord
  belongs_to :user
  has_many :reviews, as: :reviewable
end

# app/models/comment.rb

class Comment < ApplicationRecord
  belongs_to :user
  has_many :reviews, as: :reviewable
end

The user, instrument_post, and comment models can now be reviewed and given ratings.

If we have already created at least two users, a comment, and an instrument post, we can then create and access the reviews through various ways with Active Record Queries, like so:

# A user (User 2) leaving a review for another user (User 1)
# The reviewable_id is the id of the user to which the review is given
# The user_id is the id of the user who created the user review
user_1 = User.first
user_1.reviews.create(user_id: 2, rating: 2)
Review.where(reviewable_type: "User").first # id: 1, user_id: 2, reviewable_type: "User", reviewable_id: 1, rating: 2
# or
user_1.reviews.first # id: 1, user_id: 2, reviewable_type: "User", reviewable_id: 1, rating: 2

# User 2 reviewing a instrument posted by User 1
# The reviewable_id is the id of the instrument_post
# The user_id is the id of the user who created the instrument_post review
post = InstrumentPost.first
post.reviews.create(user_id: 2, rating: 4)
post.reviews.first # id: 2, user_id: 2, reviewable_type: "InstrumentPost", reviewable_id: 1, rating: 4
post.reviews.first.reviewable_type # "InstrumentPost"

# User 2 reviewing a comment created by User 1
# The reviewable_id is the id of the comment
# The user_id is the id of the user who created the comment review
comment = Comment.first
comment.reviews.create(user_id: 2, rating: 5)
comment.reviews.first # id: 3, user_id: 2, reviewable_type: "Comment", reviewable_id: 1, rating: 5
comment.reviews.first.rating # 5

# We can then find the user that created the reviewed comment by associating the value of the reviewable_id to the comment id
Comment.where(id: 1) # id: 1, user_id: 1, content: "Comment created by user id 1"
# In this scenario the comment id and user_id just happen to be the same

user_1 = User.first
user_2 = User.last

# Reviews User 1 has given (none)
user_1.submitted_reviews # []

# Reviews User 2 has given
user_2.submitted_reviews
# id: 1, user_id: 2, reviewable_type: "User", reviewable_id: 1, rating: 2
# id: 2, user_id: 2, reviewable_type: "InstrumentPost", reviewable_id: 1, rating: 4
# id: 3, user_id: 2, reviewable_type: "Comment", reviewable_id: 1, rating: 5

# Counting the total amount of reviews User 2 has given
user_2.submitted_reviews.count # 3

There are many other ways to interact with the reviews model, depending on what data you need to render. It is important to share how the parent models can create a review. As long as the review is associated with a user and a reviewable model, Active Record automatically links the reviewable_id and reviewable_type with the associated model.

Without polymorphism in our Rails examples, there would be many more tables, unnecessary duplicate columns, belongs_to, and has_many associations in our models. Polymorphism has lessened the need to join tables, permitting easier and quicker Active Record queries and associations.

Wrap Up: Use Polymorphism for Clean and Logical Ruby Code

In this post, we explored polymorphism in two distinct environments: biology and Ruby programming. In both cases, polymorphism is the ability of an object to display more than one form.

We looked at how to implement polymorphism in Ruby through inheritance and duck-typing before diving into the uses of polymorphism in Ruby on Rails specifically.

Polymorphism can help you write clean and logical code. My goal is to help you add this essential OOP concept to your toolbelt for your current, future, and maybe even past applications.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Using Scientist to Refactor Critical Ruby on Rails Code

Roy Tomeij — 2022-05-18T00:00:00+00:00

Ask any software engineer to review key portions of production code, and inevitably, they will point out three things that need to be refactored. So why does so much bad, brittle, or misunderstood code remain running in production?

The answer is simple: engineers are afraid to touch it. Refactoring tasks get identified and added to the backlog, but rarely make it into the current sprint.

There are numerous reasons for this. The code may have been written by an engineer who left the team years ago, and no one completely understands it. In other cases, the capability is critical to the business. No one wants to be responsible for a potential outage or loss of revenue.

In this post, we'll examine how you can use Scientist to migrate, refactor, and change critical Ruby production code with confidence.

But first, you might ask — can't we use tests to dig up code issues?

This Is What Rails Testing Is For, Right?

Yes, and no. It is often difficult to gain complete confidence in code changes before deployment. The unit and system tests pass. It's good to go, right?

The reality is that there is no substitute for the real world, i.e. production. What if the data quality is bad or tests are missing? How can you know if the new software will perform well enough to handle production throughput?

Teams with public services sometimes find that they need to deal with “bugwards compatibility” issues. When a bug has existed in production for a while, clients may code in a way that depends on consistent incorrect behavior. Customers often use software in unexpected ways.

Observe Production Changes in Ruby and Rails with Scientist

If production is the best place to gain confidence in a change, then consider observing how code behaves there. This may sound scary at first, as the idea of “testing in production” contradicts classic software engineering practices.

However, the good news is that it’s easy and safe to do so in Ruby and Rails using the Scientist gem. Scientist's name is based on the scientific method of conducting experiments to verify a given hypothesis. In this case, our hypothesis is that the new code does the job.

The reason we can safely take this approach stems from the fact that experiments still use the result of the existing code. New code is only evaluated for observation and comparison purposes, both for accuracy and performance. We mitigate the test coverage concerns discussed earlier by evaluating performance using real-world data and parameters. Experiments typically evaluate a chosen sample rate of requests to minimize the impact on production. However, you can evaluate every request if desired.

Let's now take a quick look at how Scientist works in a branch by abstraction way.

The Branch by Abstraction Pattern in Ruby's Scientist

Scientist's approach begins with the Branch by Abstraction pattern described by Martin Fowler as making “a large-scale change to a software system in a gradual way.”

We introduce an abstraction layer to isolate the code being updated. This layer decides which implementation to use so that the experiment is transparent to the rest of the system. The technique is related to using a feature flag that determines the code path.

The Scientist gem, which originated from Github, implements this pattern using an experiment. The existing code is referred to as the control, and the new implementation is the candidate. Both code paths are run in randomized order, but only the control result is returned to the client.

Using Scientist to Refactor a Ruby Service

Consider a Ruby service that returns the largest prime factor for a given number. Assume that we've identified optimizations to prune the required set of candidates, speeding up the service.

However, service owners want to be sure no bugs were introduced. They also want to observe any performance improvements. Introduce the following code, modifying clients to call this method:

require 'scientist'

def largest_prime_factor(number)
  science "prime-factors" do |experiment|
    experiment.use { find_largest_prime_factor(number) }     # old way
    experiment.try { improved_largest_prime_factor(number) } # new way
  end  # returns the control value
end

At this point, only the use (control) expression is invoked. To make the experiment worthwhile, define a custom Experiment class to enable it (100% of the time below) and publish the results (in this case, just logging). Scientist generates fantastic data but it doesn’t do anything with it by default. That part is left up to you.

require 'scientist/experiment'
require 'pp'

class MyExperiment
  include Scientist::Experiment

  attr_accessor :name

  def initialize(name)
    @name = name
  end

  def enabled?
    true
  end

  def raised(operation, error)
    p "Operation '#{operation}' failed with error '#{error.inspect}'"
    super # will re-raise
  end

  def publish(result)
    pp result
  end
end

The results of the experiment will be logged, and we can make improvements over time based on the feedback. Once the new code meets the requirements and confidence is high, accomplish cutover to the new implementation by simply replacing the science code with a delegation to the new implementation.

LabTech Simplifies Scientist Experiments in Ruby on Rails

You can use the LabTech gem in your Rails application to easily configure Scientist and handle the results.

Applications that use AppSignal can use the Appsignal.instrument custom instrumentation helper to track how long Scientist events take to complete. Wrap it around the different experiment code blocks to see events appear in the performance timeline.

Now, going back to LabTech — the web page below simply accepts a number to factor.

Getting started is easy provided you have access to the console. First, add the LabTech gem to your Gemfile and run a bundle install.

gem 'lab_tech'

Tables store results and experiment configuration, so run a database migration.

rails lab_tech:install:migrations db:migrate

The abstraction layer is the same, except the LabTech module is used. The full code is available on GitHub.

def largest_prime_factor(number)
    LabTech.science "prime-factors" do |experiment|
      ...
    end
end

At this point, the experiment is disabled, so use the console to enable it in all cases or for a percentage of the time.

bin/rails console
LabTech.enable "prime-factors"
LabTech.enable "prime-factors", percent: 5

We can now run tests and the experiment will be evaluated. For a textual view of results, use either of the following commands from the Rails console.

LabTech.summarize_results "prime-factors"
LabTech.summarize_errors "prime-factors"

After a few successful runs and one manufactured error, here is an example of what the results summary looks like. There is an overview of successes and failures, as well as an ASCII chart showing performance differences.

--------------------------------------------------------------------------------
Experiment: prime-factors
--------------------------------------------------------------------------------
Earliest results: 2022-04-27T02:42:45Z
Latest result:    2022-05-01T17:27:39Z (5 days)

3 of 4 (75.00%) correct
1 of 4 (25.00%) mismatched

Median time delta: +0.000s  (90% of observations between +0.000s and +0.000s)

Speedups (by percentiles):
      0%  [                         ·         █               ]    +2.4x faster
      5%  [                         ·         █               ]    +2.4x faster
     10%  [                         ·         █               ]    +2.4x faster
     15%  [                         ·         █               ]    +2.4x faster
     20%  [                         ·         █               ]    +2.4x faster
     25%  [                         ·         █               ]    +2.4x faster
     30%  [                         ·         █               ]    +2.4x faster
     35%  [                         ·         █               ]    +2.4x faster
     40%  [                         ·         █               ]    +2.4x faster
     45%  [                         ·         █               ]    +2.4x faster
     50%  [ · · · · · · · · · · · · · · · · · █ · · · · · · · ]    +2.4x faster
     55%  [                         ·         █               ]    +2.4x faster
     60%  [                         ·         █               ]    +2.4x faster
     65%  [                         ·         █               ]    +2.4x faster
     70%  [                         ·                        █]    +6.9x faster
     75%  [                         ·                        █]    +6.9x faster
     80%  [                         ·                        █]    +6.9x faster
     85%  [                         ·                        █]    +6.9x faster
     90%  [                         ·                        █]    +6.9x faster
     95%  [                         ·                        █]    +6.9x faster
    100%  [                         ·                        █]    +6.9x faster
--------------------------------------------------------------------------------

The Blazer gem provides a nice way to analyze the results easily. It is simple to install and allows SQL queries to run against tables. The query here shows that the candidate implementation is significantly faster than the original.

In the example prime factoring service, the speedup in the improved implementation comes from a heuristic that eliminates some possible factors to consider. As we consider higher numbers and find a prime factor, we can stop searching after we get to our target number divided by that factor. The new code path only adds one statement to accomplish this.

We can also see the reduction in execution time using a Blazer query against the LabTech tables.

Use Cases and Limitations of Scientist

Optimal use cases for Scientist include searches, calculations, and code that has no side effects. Code that includes transactional updates or external integrations such as email does not fit cleanly into the model because the capability is run twice (both the old and new implementations).

This is not a trivial limitation, as it does eliminate several use cases. However, there are some workarounds if the experiment is critical to your success. Consider whether the side effects are relevant or if duplication is an issue. For example, it may not matter in some cases whether two emails are sent during the evaluation. Another option is to have the new code determine the result but not persist it. This would prevent any meaningful performance comparisons. However, it would allow you to verify accuracy.

Other limitations stem from Scientist’s focus on return values. In some cases, valid results may exhibit differences over time, whether they simply include timestamps in the response or certain factors vary. In many cases, we can write custom comparison logic in the experiment to verify accuracy beyond basic string comparisons.

Finally, a limitation of LabTech is that it has not been ported to Rails 7 as of the time of writing.

Best Practices for Effective Scientist Experiments in Rails

Consider these items when implementing your experiments:

In Rails projects, Scientist can either be configured in an initializer or a wrapper like the Rails LabTech gem. Most Rails applications already have a database, so LabTech leverages ActiveRecord to store results.
To avoid slowing down development and testing, enable your experiment only in staging and production environments.
To minimize any potential impact on production, only run the experiment on a percentage of requests. LabTech supports this out of the box as an optional parameter when you enable the experiment (it is initially disabled by default). Using pure Scientist, this logic is easy to code in the experiment’s enabled? method.
Some logic is resource-intensive, so a low sampling rate may be a good place to start. As you gain confidence with the results, ramp up the percentage of requests being evaluated.
You can add context attributes to get the most from your results. The experiment context can be set to a Symbol-keyed Hash of data that is then made available in published results, e.g.:

experiment.context :user => user

Wrap-Up: Observe and Monitor Your Ruby App with Scientist

In this post, we explored how to use the Scientist gem to change, migrate, and refactor Ruby code in production.

We examined Scientist's origins in the Branch by Abstraction pattern, then dived into refactoring. Next, we saw how LabTech could help with gathering results and your Scientist configuration.

We then touched on some limitations of Scientist before finally outlining a few best practices.

You must observe and monitor what is happening in your system. Integrate Scientist into your development process to make critical changes in your Ruby code with greater confidence.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Bootstrapping with Ruby on Rails Generators and Templates

Roy Tomeij — 2022-05-04T00:00:00+00:00

Rails' batteries-included approach is one of its greatest assets. No other framework makes it so effortless to get your application off the ground quickly, at least partially due to Rails' generators.

If you've used Rails for any amount of time, you have come across generators. Need to create a new application? Run rails new. Need to scaffold a bunch of new models and views? Run rails generate scaffold. There are dozens more available to help you get started rapidly or streamline your workflow.

But sometimes, using generators is just not enough. You might want to customize the behavior of these commands or even create your own. In this article, we'll take a closer look at generators - in particular, how to create your own custom Rails application using templates.

Let's get started!

What Are Rails Generators?

Not to be confused with generator functions (which you might be familiar with from Python or Javascript), Rails generators are custom Thor commands that focus on, well, generating things.

There are lots of examples. You'll likely be familiar with the model generator (rails generate model) for creating new ActiveRecord models or the migration generator (rails generate migration) for generating new migrations. There is also rails generate generator which — you guessed it — creates a new generator!

Generators can call each other — for example, rails scaffold will call numerous other generators — and provide methods to create or modify files, install gems, run specific rake tasks, and much more. Let's create a simple model spec generator to understand how this works.

Creating Your Own Generator in Ruby on Rails

Run the following:

rails generate generator model_spec

This will create several new files in /lib/generators/model_spec. We can modify model_spec_generator.rb in folder lib/generators/model_spec/ to create a model spec file in the correct directory:

class ModelSpecGenerator < Rails::Generators::NamedBase
  source_root File.expand_path('templates', __dir__)

  def create_model_spec
    template_file = File.join('spec/models', class_path, "#{file_name}_spec.rb")
    template 'model_spec.rb.erb', template_file
  end
end

The template command will look for a template file in the lib/generators/model_spec/templates directory and render it to the specified location — the spec/models directory. The command will replace ERB-style variables found in the template file.

By setting the source_root, we let our generator know where it can find referenced template files. Template model_spec.rb in folder lib/generators/model_spec/templates/ could look like this:

require 'rails_helper'

RSpec.describe <%= class_name %>, :model
  pending "add some examples to (or delete) #{__FILE__}"
end

Once you have created that file, you can run the generator to create a new spec file.

rails generate model_spec mymodel

Many gems ship with generators such as this one. In fact, we created a simplified version of the generator Rspec ships with. FactoryBot has a generator for factories. There are many more examples.

The generators in various gems are more sophisticated than the one we created. We could make our generator take arguments or even hook it into existing generators such as rails scaffold. Refer to the Rails generators documentation if you want to learn more.

So generators have the potential to simplify your workflow in an existing application. But can we also use generators to customize setting up a new application?

Enter templates!

Templates in Ruby on Rails

As the name suggests, templates are files for customizing your application setup. Don't confuse these with the template files that we previously discussed! Under the hood, they are just generators with a specific purpose, as evidenced by the template API. While not exactly identical to generators, they are very similar.

If you have an existing template file, you can use it like so:

rails new myapp -m mytemplate.rb

Rather than specifying a local file, you may also specify a URL. This is especially useful as it allows you to share application templates.

rails new myapp -m https://gist.github.com/appsignal/12345/raw/

You are not limited to using templates when running rails new either. If you've already set up an app, you can apply templates afterward by executing:

rails app:template LOCATION=http://example.com/template.rb

Templates can be extremely useful. Who doesn't want to automate adding the same couple of gems and making the same configuration changes every time they create a new app? Creating your own application template is great fun — even if it doesn't save a lot of time in the long run.

Creating Your Own Template in Rails

We now know about generators and how to use templates. Let's create a simple application template to automate some setup steps.

How you set up your Rails app is very much down to personal preference, but here's an example:

Install the dotenv gem.
Create a .env.development file for the development environment.
Adapt the database configuration file to use environment variables.
Optionally install and set up Rspec.

Let's create a local file — mytemplate.rb — and add dotenv using the gem command.

gem 'dotenv-rails', groups: [:development, :test]

As we've just added the dotenv gem, let's also create a .env.development file to contain our database configuration.

You can create a new file with specific content by using create_file. You won't find this in the template or generator documentation, as the method is supplied by Thor. You might also come across the alias file. Application templates are evaluated in the context of Rails::Generators::AppGenerator, and that's exactly where the file alias is defined.

create_file '.env.development', <<~TXT
  DATABASE_HOST=localhost
  DATABASE_USERNAME=#{app_name}
  DATABASE_PASSWORD=#{app_name}
TXT

The app_name variable contains the first argument of rails new. Using this variable, we can ensure our config file matches the generated application.

Next, let's use our environment variables to connect to the database. We could overwrite the entire config/database.yml using the create_file command, but let's modify it instead using inject_into_file.

inject_into_file 'config/database.yml', after: /database: #{app_name}_development\n/ do <<-RUBY
  host: <%= ENV['DATABASE_HOST'] %>
  username: <%= ENV['DATABASE_USERNAME'] %>
  password: <%= ENV['DATABASE_PASSWORD'] %>
  RUBY
end

We can use both strings or regex with the after argument to specify where to inject content.

Of course, using this kind of configuration only makes sense if a user isn't creating an application with SQLite. You can check for the presence of certain arguments by using the options variable. It's best you read the source code of Rails' app generator to see which options are available.

if options[:database] != 'sqlite3'
  # Set up env vars and db configuration
end

Last but not least, let's also allow users to install Rspec if they want to. There are various methods for taking user input and creating interactive templates. The yes? method asks a user for confirmation:

if yes?('Would you like to install Rspec?')
  gem 'rspec-rails', group: :test
  after_bundle { generate 'rspec:install' }
end

We already know the gem method, but generate and after_bundle are new.

As mentioned before, Rspec adds its own generators, and you can call these generators (or any other ones, for that matter) directly from your template. But there is a catch — gems specified with the gem method are only installed at the end of the template. Calling generate with a generator supplied by such a gem would fail — which is why you should register the command as a callback with after_bundle.

Note: Before we wrap up, a quick word about creating or modifying files. We used create_file and inject_into_file, but there are many other options. You may come across copy_file or template when reading different templates. I did not mention them here to keep things simple. If you want to create more advanced templates, you should know that these other methods for dealing with files exist.

The Result: The Final Template in Rails

The final template should look like this:

# Install dotenv
gem 'dotenv-rails', groups: [:development, :test]

if options[:database] != 'sqlite3'
  # Set up .env.development
  create_file '.env.development', <<~TXT
    DATABASE_HOST=localhost
    DATABASE_USERNAME=#{app_name}
    DATABASE_PASSWORD=#{app_name}
  TXT

  # Modify database.yml
  inject_into_file 'config/database.yml', after: /database: #{app_name}_development\n/ do <<-RUBY
  host: <%= ENV['DATABASE_HOST'] %>
  username: <%= ENV['DATABASE_USERNAME'] %>
  password: <%= ENV['DATABASE_PASSWORD'] %>
  RUBY
  end
end

# Optionally install Rspec
if yes?('Would you like to install Rspec?')
  gem 'rspec-rails', group: :test
  after_bundle { generate 'rspec:install' }
end

You can test this particular template by running:

rails new myapp -m mytemplate.rb

Or:

rails new myapp --database=postgresql mytemplate.rb

To take advantage of the custom database configuration.

As mentioned, this file is perfectly suited to share as a GitHub gist. Adapt it to suit your needs, upload it, and then share it with your colleagues, friends, and anyone else interested in your custom app template 😉.

Learn More About Rails Generators and Templates

Needless to say, we've just scratched the surface here.

Everyone has different preferences for writing and developing Rails apps, so there are numerous generators and application templates out there. You can learn a lot about doing specific customizations by reading about them. I recommend Chris Oliver's Jumpstart and RailsBytes, the latter of which is a community-curated collection of templates.

There is also Thoughtbot's Suspenders, which inspired me to dig deeper into Rails generators and templates. I even wrote my own application template — Schienenzeppelin — which, while not up-to-date, might still provide some inspiration.

Wrap Up: Get Started with Ruby on Rails Generators and Templates

In this post, we looked into the basics of Rails generators and how they are used. We created our own generators to simplify writing new model specs.

We then delved into templates and learned how you could create a simple template to customize your application setup. This can be a bit of work, but also quite rewarding. If writing your own templates is not for you, there are many existing ones online to choose from!

Happy templating!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

5 Tips to Design Ruby on Rails Transactions the Right Way

Roy Tomeij — 2022-03-30T00:00:00+00:00

Data integrity problems are among the most common database issues Rails developers face. Besides allowing for proper validation, correctly designed transaction blocks ensure that your data isn't partially created or updated.

However, transactions can also harm your application — or even take down your whole database — when not properly designed.

This article offers a set of good practices for working with transactions. The tips are pretty simple, but they will help make your transactions bulletproof, readable, and relatively safe.

Let's dive in!

1. Use Bang Methods in Rails When Possible

In Rails, method versions with ! can give you confidence that an error will be raised when something goes wrong.

For example, the #save method also exists in the save! version. You might want to use this version in the controller if you don't want to raise any errors:

def create
  user = User.new(user_params)

  if user.save
    # redirect
  else
    render :new
  end
end

The above approach won't work well in transactions. By using save, we can't roll back the process when the error is raised. That's why it is so important to use the ! version of the methods:

ActiveRecord::Base.transaction do
  user = User.create(user_attributes)
  user.memberships.create(membership_attributes)
end

In the above, the transaction succeeds even if the membership record isn't created, and we end up with a messed-up data structure in the database.

If you use the following version, the transaction reverts due to an ActiveRecord::RecordNotSaved error:

ActiveRecord::Base.transaction do
  user = User.create!(user_attributes)
  user.memberships.create!(membership_attributes)
end

2. Handle Errors in Rails Transactions Properly

When it comes to errors in transactions, there are a few rules that you should respect. By following these rules, you'll have readable and well-working code that will not create confusion among other developers or weird behavior that is hard to debug.

Do Not Rescue from `ActiveRecord::StatementInvalid`

ActiveRecord::StatementInvalid is a special error raised when something on the database level goes wrong. Never rescue from this error. You should always be explicitly notified when something goes wrong with the database query.

Avoid the following code:

def perform_action(...)
  User.transaction do
    # perform transaction
  end
rescue ActiveRecord::StatementInvalid
  # do something
end

Use the Rescue on the Right Level

If you use rescue on the following level, you'll catch the error:

User.transaction do
  user.perform_action!
  user.perform_another_action!
rescue SomeError
  # rescue
end

The transaction does not roll back because you caught the error. Let the error be raised and catch it outside the transaction block:

def some_method
  User.transaction do
    user.perform_action!
    user.perform_another_action!
  end
rescue SomeError
  # rescue
end

In the above approach, the transaction rolls back in case of an error, and you catch the error. This is the right approach to catch errors raised inside transactions without overwriting transaction behavior.

Do Not Catch Generic Errors

You should avoid catching generic errors like StandardError or ArgumentError. This is more like a general rule for readable and easily testable code, but it's worth mentioning.

Catching these errors can make debugging harder, as other places in the code may raise the errors. This could silence some serious issues in your app that are not necessarily related to the place you rescue them.

Use ActiveRecord's Default Rollback Error Wisely

ActiveRecord provides a particular error class that you can use inside a transaction to make a silent rollback. You roll back the transaction by raising the ActiveRecord::Rollback error, but the error isn't raised outside, as happens with other errors. Keep this behavior in mind and use it wisely.

3. Know When to Avoid Using Transactions in Rails

As with anything, you should not overuse transactions in your code. For example, a common mistake is to wrap only one query into your transaction. This does not make sense because, if the query doesn't succeed, there is no need to roll back anything.

Another common mistake is to wrap code unrelated to your database call into a transaction. You should avoid such an approach, as the transaction will hold the connection unless the code inside the block executes. Limit the code inside the block to call only your database, if possible.

4. Understand the Disadvantages of Transactions

Transactions help maintain data integrity inside a database, but you should also be aware of their disadvantages. For example, queries wrapped in a transaction block take more DB resources than single queries.

Another drawback of using transactions is that it leads to more complex code. Transactions can make your code less readable when used incorrectly.

5. Use the Transaction Block in the Right Context

You can use the transaction method when a class inherits from the ActiveRecord class. This does not mean that the version you use does not matter. Although it might not matter from a functional perspective, it matters in terms of ensuring your code is readable.

Three common versions use the transaction method:

ActiveRecord::Base.transaction
Model.transaction
Model.new.transaction

When you use many models and mix instance method invocations with classes inside a block, you should use ActiveRecord::Base.transaction:

ActiveRecord::Base.transaction do
  attributes = user.prepare_attributes(account)
  membership = Membership.create(attributes)
  LogService.log_creation(user, membership)
end

If you deal mostly with code that belongs to a given model, invoke the transaction method on a class:

User.transaction do
  user = User.create!(attributes)
  user.log_activity(‘creation’)
end

When you operate on a model instance, it makes sense to invoke the transaction method on the instance level:

user.transaction do
  user.make_transaction(attributes)
  user.log_activity(‘transaction’)
end

Of course, these rules are not official. They are just suggestions to make code more readable.

Next Steps: Review Transactions in Your Ruby on Rails Project

I hope you've found these tips for working with transactions in Ruby on Rails useful.

We've covered the importance of designing Rails transactions properly to improve data integrity and ensure that your processes perform without surprising side effects.

However, a proper error handling policy isn't only beneficial when using transactions — it will also improve your whole codebase. Keep that in mind the next time you expect your code to throw some errors.

Now is an excellent time to review transactions in your Ruby on Rails project design to avoid errors. Design for efficient and reliable communication with your database to make your application more stable.

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

The Perils of Parallel Testing in Ruby on Rails

Roy Tomeij — 2022-03-16T00:00:00+00:00

Have you ever heard someone complain about their tests being too fast? Me neither.

Fast tests mean fast feedback. Whether you run them locally or in a continuous integration pipeline, the earlier your tests finish, the earlier you can react to failures and improve your code. Besides the productivity gains, it is well known that slow tests make developers grumpy. Nobody likes their developers grumpy.

With all that said, creating a lightning-fast test suite isn't always as easy as you'd hope. Luckily, Rails 6 introduced an exciting feature called parallel testing. It is effortless to get started with and can speed up your tests by a lot. However, there are some pitfalls to watch out for.

What Is Parallel Testing?

What does parallel testing even mean?

When you run a test suite, your test runner will typically spawn a single process to execute your tests one after the other — or serially. A single test process uses a single CPU core. As you can probably imagine, this approach doesn't take full advantage of modern hardware, which often sports dozens of CPU cores.

You may have a fancy MacBook with 10 CPU cores, but sadly, that won't make your tests go any faster!

We can change this by distributing individual tests to multiple worker processes. Tests will no longer run after each other, but next to each other — in parallel. Running your test suite on two workers is twice as fast as running the same test suite on a single worker.

The more cores your machine has, the more worker processes are feasible, and thus, the faster your test suite will finish. Say your tests usually take eight minutes to run. Running those same tests using four worker processes will take the runtime down to around two minutes!

Imagine doing the same on a nice, fat 16 core machine, where we can spawn sixteen workers. Very nice indeed!

Configuring Parallel Testing in Rails

So how do we get there? Until recently, you could use third-party gems to parallelize your test suite, but starting from Rails 6, parallel tests come standard. It's as easy as adding parallelize to your tests:

class ActiveSupport::TestCase
  parallelize(workers: :number_of_processors)
end

Using this configuration, Rails will automatically spawn worker processes based on the number of processors in your machine. Rails will also create namespaced databases (e.g. database-test-0, database-test-1, etc.) to run your tests against.

That's all it takes to get started! Of course, there are some additional configuration options if you need them.

Sometimes, you may have to perform a specific setup or cleanup for parallel tests. Rails provides two hooks for you to use — parallelize_setup and parallelize_teardown. These are called before and after new worker processes spawn:

class ActiveSupport::TestCase
  parallelize_setup do |worker|
    # setup
  end

  parallelize_teardown do |worker|
    # cleanup
  end

You can also manually set the number of workers:

class ActiveSupport::TestCase
  parallelize(workers: 4) # Use 4 worker processes
end

Alternatively, use the PARALLEL_WORKERS environment variable to override an existing configuration:

PARALLEL_WORKERS=4 rails test

There is also the option to use threads instead of workers to parallelize your test suite.

class ActiveSupport::TestCase
  parallelize(workers: :number_of_processors, with: :threads)
end

with: :threads is the default option when using JRuby or TruffleRuby. Using threads, in theory, provides slightly better performance. Threads require less overhead than processes, after all. In practice, however, I never found using threads all too useful, and you should be fine just sticking to process-based parallelization for the most part.

Beware the Pitfalls

So all you need to do is add parallelize to your existing tests to experience incredible speedup? It's that easy!

If you are lucky, that really is true. It's more likely that you will hit some unexpected snags when first adding parallelization to your existing test suite. This certainly was the case for me!

Let's get one thing out of the way. If you use RSpec rather than Minitest, you are out of luck. RSpec does not support Rails 6 built-in parallel testing. There is an ongoing discussion about changing that, but there hasn't been any significant progress for a while. If you want parallel tests with RSpec, your best bet is still using third-party gems such as grosser/parallel_tests.

Another unexpected issue that you might face is that running a small number of tests in parallel ends up being slower then running them serially. Setting up parallel tests comes with a significant overhead — such as creating multiple databases — which can eliminate any gains you might get from parallelization.

You might be better off disabling parallel tests for a small number of tests. You can do so by using the PARALLEL_WORKERS environment variable:

PARALLEL_WORKERS=0 rails test test/controllers/my_controller_test.rb

Rails 7 addressed this by enabling parallel execution only when you execute many tests. So if you've already upgraded, you won't experience this problem. Per default, the parallelization threshold is set to 50, but you can override it:

config.active_support.test_parallelization_threshold = 123

The last problem I want to highlight is the one you are most likely to face, and it is also the most insidious and hardest to deal with. You may start seeing random failures when enabling parallel tests for your test suite. To understand how parallelization can cause this, let's look at a simple test case:

class FileTest
  def teardown
    File.delete('test.txt')
  end

  test 'create file' do
    file = File.write('test.txt', 'created')

    assert_path_exists('test.txt')
  end

  test 'delete file' do
    file = File.write('test.txt', 'deleted')

    File.delete('test.txt')

    assert_not(File.exist?('test.txt'))
  end
end

Besides being a bit useless, this test is perfectly fine. It will pass 100% of the time as long as it's run serially. Each test is isolated, and executing these tests in random order does not cause them to fail. That changes when you add multiple processes or threads to the mix.

When running tests in parallel, the execution order of individual statements in your tests can get changed up due to CPU scheduling. Looking at the example, you'll sometimes see execution orders such as this one:

# Worker 1 executes
file = File.write('test.txt', 'created')

# Worker 2 executes
file = File.write('test.txt', 'deleted')
File.delete('test.txt')
assert_not(File.exist?('test.txt'))

# Worker 1 executes
assert_path_exists('test.txt')

Since Worker 2 deleted the file before Worker 1's assertion was executed, the first test will fail — sometimes. To add insult to injury, a different execution order would sometimes make the second test fail and the first one pass.

This simple example illustrates an issue that extends not only to files but to any singleton resource that your tests access in a non-thread-safe way. Suppose you write to a Redis database or an Elasticsearch index. In that case, you'll likely experience similar complications. What's worse, it may take you a while to uncover all tests that cause random failures — and even more time to fix all of them.

There is no silver bullet to address flaky parallel tests. In general, you will need to ensure that multiple test processes do not share resources. For files, use Tempfiles. Use parallelize_setup to create namespaced resources (e.g. Redis databases). And so on.

Adding Parallel Testing to Existing Rails Tests

Let's say you struggle with random test failures due to parallelization and don't have the time to fix them. However, you still want to reap the benefits of parallel testing. You may prefer to enable it only for a subset of your tests.

Only tests that call parallelize will be parallelized after all, and by using concerns or parent classes, you can add parallel testing to your test suite one test class at a time. You could create a module like this:

module Parallelize
  def self.included(base)
   base.class_eval do
      parallelize(workers: :number_of_processors)

      # ...
    end
  end
end

Any test class that includes this module will now run in parallel:

class MyTest < ActiveSupport::TestCase
  include Parallelize
end

Alternatively, you could create a new test class like ParallelTest:

class ParallelTest < ActiveSupport::TestCase
  parallelize(workers: :number_of_processors)
end

Then, inherit from that for tests that should run in parallel — and leave out those that prove problematic.

Parallel Testing as a Bandaid

Parallel testing provides impressive speed gains for little effort. Don't be fooled, though: it is no substitute for other approaches to improve your test suites' speed, but rather an addition.

If you find your test suite is slow and can spare the effort, spend some time profiling it and addressing the root cause/s for the slowness. A slow test suite with parallel testing added to it will get faster, but never as fast as an already fast test suite that also runs in parallel!

Wrap Up

In this post, we looked at what parallel testing is, how you can set it up and how to configure it. If you need a way to make your tests go faster, parallel testing provides just that.

You might face some obstacles when adding parallel testing to your test suite. Don't be surprised when tests that ran stable for years suddenly start failing. You can work around them by parallelizing only a subset of your test suite.

No matter which approach you choose, parallel testing is a fantastic tool to speed up your tests!

Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Import Maps Under the Hood in Rails 7

Roy Tomeij — 2022-03-02T00:00:00+00:00

Import maps is the new feature in Rails 7 that allows us to say goodbye to Node.js and tools like Webpack. There's no need for bundling anymore. With this new mechanism, you can still manage your JavaScript libraries with a specific version. Instead of one big file, though, your application serves many small JavaScript files.

It’s essential you know how import maps work to benefit from the newest version of Rails (but don’t worry, you can still use tools like Webpack instead). However, it’s also worth discovering what is happening under the hood. This way, you can better understand the journey from installing a JavaScript library in your project to serving its content to your users.

This article will show you how to install, serve, and uninstall JavaScript libraries with import maps and what happens under the hood during each phase.

The Core of Import Maps

The feature itself is not complicated. However, before I show you what happens inside the library, I would like to introduce the JSPM tool. JSPM is a shortcut for JavaScript Package Management. Thanks to this tool, you can load any NPM package inside the browser without extra tooling, and it will be fully optimized.

Rails uses JSPM to serve JavaScript libraries in your application. You can either download the source files to the vendor directory or serve the code directly from the URL.

For example, to access the jQuery library, you can call https://api.jspm.io/generate?install=jquery URL in your browser, and you will receive the URL to the minified source code in the JSON response. Of course, the service provides some more options for requests, but this knowledge is enough to understand how Rails cooperates with JSPM in the import maps library.

Install Import Maps in Your Rails Project

The import maps feature is available in Rails as a Ruby gem. If you generate a new project with Rails 7, it’s included in the Gemfile by default. You can add it to existing projects by executing the following command:

bundle add importmap-rails

Once you have the gem installed, you have to generate the required files by using the following command:

./bin/rails importmap:install

What does this command do, exactly? First, it calls the install rake task included in the importmap namespace. The gem delivers the rake task. Inside the task, the standard rails rake task is executed:

system "#{RbConfig.ruby} ./bin/rails app:template LOCATION=#{File.expand_path("../install/install.rb",  __dir__)}"

The install.rb file does a few things in the following order:

Adds helper to the layout in the head section - if the application.html.erb layout exists in your project, it adds the javascript_importmap_tags line before the closing </head> tag. This helper includes JavaScript libraries pinned by import maps. If your project does not include the standard layout file, it will render the information about the helper so you can add it by yourself.
Creates app/javascript/application.js file - it adds the comment about import maps to let you know that there is a new place where you should define dependencies.
Creates vendor/javascript directory - this is where JavaScript libraries will be stored if they are not served directly via a URL.
Updates sprockets manifest, if it exists - sprockets needs to know about the JavaScript files placed inside the vendor/javascript directory.
Creates config/importmap.rb file - this will contain the list of libraries used by Rails (something like package.json file).
Copies bin/importmap file and sets the correct permission to execute it - this file is used to pin and unpin specific libraries.

As soon as this process finishes, you are ready to install JavaScript libraries using the ./bin/importmap file, and Rails is prepared to serve those files to your visitors.

Adding Libraries to Your Rails Project

When installing a library with import maps, you have two options: you can either use the code directly from the CDN URL or download the file and serve it directly from your server.

Let’s explore the CDN option first.

Using NPM Packages via JavaScript CDN’s

It all starts with the pin command and the library name:

./bin/importmap pin jquery

You pass two arguments to the program located inside the ./bin/importmap file. importmap is an executable file that loads a config/application.rb file from your project and the import maps commands file.

Import maps use the Thor gem to handle command line programs gently. The first argument, pin, is the command name. When you invoke it, the following things happen under the hood:

Request to JSPM API is executed

As I mentioned before, the gem uses JSPM to get the contents of the package into our application. Therefore, the very first step is the request URL formation. Without any extra parameters passed to the pin command, the request URL is https://api.jspm.io/generate with the following parameters:

install - ["jquery"] - the param is an array because you can install multiple packages simultaneously.
flattenScope - true - with this param set to true, the returned import map format will be more straightforward without scopes, if possible.
env - ["browser", "module", "production"] - this is the list of environment condition strings.
provider - "jspm" - besides JSPM, Skypack, JSdelivr, and Unpkg providers are available to use as well.

Try accessing https://api.jspm.io/generate?install=jquery&flattenScope=true in your browser, and you will get a simple JSON response with some simple attributes.

Response from JSPM is parsed

Parsing is a very straightforward phase. As you saw, the response is simple, and all we need is the library name and CDN URL. The Packager class is responsible for parsing the response in the import map library. Instead of returning the attributes, it returns a complete line that you can use directly in the config:

%(pin "#{package}", to: "#{url}")

If you are not yet familiar with %(), don’t worry, as it works almost the same as the following:

"pin \"#{package}\", to: \"#{url}\""

The only difference is that the %() notation escapes the double quotes automatically. The generated config line passes to another function that handles the config file update process.

Import map config is updated

The last step is to update the config file with the pin definition. Because the previous pin definition can be present, the gem first verifies the config file and searches for the existing library definition. It’s simple to do with the regex:

/^pin "#{package}".*$/

If the pin definition for the given package is present, the gem replaces the line with the new definition. If the previous definition is not present, the gem adds a new line at the end of the config/importmap.rb file.

Note: When you use this method, keep in mind that this option is free, but one person supported by the community manages the JSPM servers. This situation could change in the future, so I advise you to store libraries locally if you plan to use import maps with production-ready applications.

Downloading NPM Packages

If you don’t want to use source code from a CDN URL, you can download the library to the vendor directory inside your application. In this case, the process is very similar to the case with CDN URLs. What's different is that the code is downloaded into a .js file before the config line is returned for an update.

The download process

The download process consists of a few small steps:

The gem calls FileUtils.mkdir_p, which creates the vendor directory if it does not already exist.
The gem calls FileUtils.rm_rf to remove the previous package file if it exists.
The gem saves JS code located under the JSPM-provided URL into the .js file and places it into the vendor directory.

If the gem cannot download the package’s contents, you will be notified by a proper error raised in your command line.

Config line generation

If the package is named the same as the file with the package source, the config line is straightforward:

%(pin "#{package}" # #{version})

Otherwise, the import map needs to map the package name to the file directly:

%(pin "#{package}", to: "#{filename}" # #{version})

When the config line is formatted and returned, the config is updated with the package definition (as described in the CDN URL version above).

Using Pinned Packages in Your Rails App

The gem extensively uses the Rails engine mechanism to deliver features. The Importmap::Engine defines a bunch of initializers that perform the configuration.

Import Pinned Packages from config/importmap.rb

The configuration lines inside the config/importmap.rb file are replaced with references inside the application. First, the gem collects all definitions into a hash where the Struct object represents each definition to make it easier to access specific attributes.

You can call Rails.application.importmap.packages to access the hash with all definitions inside the application.

Including References to Packages inside Views

When pinned packages are imported, you can include them in your views to provide the code. The gem provides a javascript_importmap_tags helper, which you can simply render in your layout. It uses Rails.application.importmap to generate JSON output for all pinned libraries, and with the usage of asset_helper in Rails, it provides the correct paths to the libraries.

After adding the following line to your head section of the layout:

<%= javascript_importmap_tags %>

You will get the following output (assuming that you are only using jQuery):

<script type="importmap" data-turbo-track="reload">
  {
    "imports": {
      "application": "/assets/application-37f365cbecf1fa2810a8303f4b6571676fa1f9c56c248528bc14ddb857531b95.js",
      "jquery": "https://ga.jspm.io/npm:jquery@3.6.0/dist/jquery.js"
    }
  }
</script>

Handling Import Maps by Browser

From that moment on, the browser handles the rest. If you are not familiar with the importmap script type, let me explain it quickly.

Import map simply controls what is fetched when you use the following statement inside your JavaScript code:

import "jquery";

Removing Pinned Packages

To remove a pinned package, you have to execute the unpin command:

./bin/importmap unpin react

You don’t have to pass the --download parameter because the gem will automatically delete any files related to the package. The removal process consists of three steps:

Request to the JSPM API - the same request is executed when the package is added. The gem does this to get up-to-date package information.
Removal of the existing package files - the gem uses FileUtils.rm_rf to remove all related files even if they are placed in directories.
Remove the config lines related to the package - with the usage of File.readlines, the gem loads all lines from the config file, selects those that do not contain anything related to the removed package and saves them again in the config file. Simple as that.

You can also unpin multiple packages at once; just add their names after the unpin argument.

Wrap Up

We've reached the end of this short, yet valuable, journey. You now know that import maps is just a different way of loading JavaScript libraries in your web application. Instead of one big bundled file, you serve multiple smaller files that are easy to cache and control.

With the importmap-rails gem, it’s easy to adapt import maps in your project. The gem ships with a simple configuration file and command-line interface to install and remove packages using the JSPM.

Should you use import maps in your next Rails project? As always, it depends. The good thing is that you are not limited to import maps — you can always switch between it, Webpack, and similar, more complex tools.

Thanks for reading and happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Delayed Job vs. Sidekiq: Which Is Better?

Roy Tomeij — 2022-02-15T00:00:00+00:00

Most applications need background jobs for mailers, regular clean-ups, or any other time-consuming operation that doesn't require a user to be present.

Several gems support job queues and background processing in the Rails world — Delayed Job and Sidekiq being the two most popular ones.

In this post, we will take a detailed look at Delayed Job and Sidekiq, including how they fare against each other.

Let's go!

A Quick Introduction to Delayed Job

Delayed Job is a direct extraction from Shopify and uses a table to maintain all background jobs. It follows a very simple pattern. Any Ruby object that responds to a perform method can be enqueued in the jobs table.

In addition, if you don't need to maintain special job objects (although this is highly recommended for testability and clear separation of long-running operations), it also allows you to call .delay.method(params) on any Ruby object. It will process the method in the background.

The Delayed Job README does a great job of explaining all common usage patterns.

Many teams choose Delayed Job because it is simple and uses their already existing database. They don't need to spend/maintain other resources.

However, it will still take up space in your database table. If you have too many jobs queued at the same time, you might need more disk space to accommodate them all.

A Quick Introduction to Sidekiq

Sidekiq, on the other hand, uses Redis as its data store to maintain all job metadata. This comes with the obvious benefit of being much faster than the regular database systems Delayed Jobs uses. In addition to this, each Sidekiq process spawns multiple threads to process the jobs even faster.

For each background job in Sidekiq, we need a specialized class that includes the Sidekiq::Worker concern and responds to the perform method. To enqueue the job, we need to call perform_async(arg1, arg2) on the worker with the arguments.

The Sidekiq 'Getting Started' guide explains this and other usage patterns in good detail.

Using Active Job with Delayed Job or Sidekiq

Rails already provides a mature job framework for top-level declaration and handling of jobs. Both Delayed Job and Sidekiq support running jobs through ActiveJob's unified API. Just inherit from ApplicationJob and call perform_later on your job class to enqueue the job to the configured queuing backend.

The advantage of running jobs with Active Job is that your application code becomes framework agnostic, and switching from Delayed Job to Sidekiq (or vice versa) becomes pretty easy. The ActiveJob::TestHelper also makes testing enqueued jobs a breeze.

But the abstraction provided by Active Job also comes with a performance overhead, as job data has to be wrapped before it's pushed to the store. Sidekiq claims that ActiveJob is about 2-20x slower when pushing to Redis, with ~3x the processing overhead.

Delayed Jobs vs. Sidekiq

Now that we know the basics of Delayed Jobs and Sidekiq, let's dive deeper into their differences and what each brings to the table.

The Features

For basic applications, both Sidekiq and Delayed Job provide a good set of features out of the box. These include assigning job priorities, named queues, and auto-retry on failures.

Delayed Job also provides a way to configure max run time out of the box (Sidekiq does not).

Sidekiq, on the other hand, provides support for Middleware to update job metadata, skip queuing a job, or execute a job. Sidekiq supports more callbacks, though some hooks are available for Delayed Job apps. Instead of callbacks, you can use Delayed Job with Active Job (namely, the before_enqueue and around_perform callbacks inbuilt into Rails).

Web UI is another feature that comes out of the box with Sidekiq. This provides historical statistics about jobs and information about workers, currently enqueued and dead jobs. You can perform operations like deleting or running jobs immediately without going through the console.

Delayed Job does not have an inbuilt Web UI, but delayed_job_web gives access to a basic Web UI with similar features to Sidekiq's.

Sidekiq Wins at Performance

Performance-wise, Sidekiq beats Delayed Job quite convincingly. According to Sidekiq's open-source benchmark, it is approximately 30x faster than Delayed Job. There are two major reasons for this:

Redis is much faster at querying data than traditional databases like Postgres because it stores data in memory as opposed to the disk.
Delayed Job runs a single thread to process jobs, compared to Sidekiq, which uses multiple threads.

While all of this looks great on paper, the differences do not matter much unless you work on a big scale (something like 10k jobs per minute). The exact number also depends on the average run time of a job. The longer the run time, the less the performance overhead of Delayed Job matters.

If you're worried about the performance of Delayed Job, you can make some performance optimizations. The exact indexes to use will depend on the statistics of your job system. For example, if you use multiple queues and only one gets a major chunk of jobs, a simple index on the queue column (add_index :delayed_jobs, :queue) can significantly improve performance.

In AppSignal, Sidekiq magic dashboard gets automatically generated and it enables you to monitor queue length, queue latency, job duration, job statuses, memory usage, etc.

Deployment

Both Delayed Job and Sidekiq have a similar deployment strategy for workers. Using Heroku, you just need to add entries inside your Procfile to start the job processor and run the workers.

For Sidekiq:

worker: bundle exec sidekiq -t 25 -c ${SIDEKIQ_CONCURRENCY:-5} [-q name,priority [-q another_queue,another_priority]]

For Delayed Job:

worker: [QUEUE=x,y,z] bundle exec rails jobs:work

Memory

Here's where things start to get a bit more interesting. Sidekiq has a concurrency option to control how many threads it runs. Most of the Sidekiq vs. Delayed Job benchmarks mention Sidekiq's very high concurrency of up to 25 threads, which contributes to its super-fast performance.

But in a real setting, you have to limit the threads to something more conservative. The actual number depends on how heavy your application is and what kinds of jobs you perform. What I have seen in practice is that if you run a worker on 512MB memory (equivalent to standard-1x on Heroku), the number of threads is somewhere between 2 and 5 instead of 25.

'Taming Rails memory bloat' by Mike Perham, the creator of Sidekiq, discusses memory issues in more detail and is well worth a read. I won't jump into the full discussion, but he recommends that you set MALLOC_ARENA_MAX=2 on all workers that run Sidekiq.

Using jemalloc instead of regular malloc helps too. The exact way to do this depends on the platform you use, but it is pretty simple on Heroku. Just set heroku-buildpack-jemalloc as the first buildpack (ahead of the heroku/ruby buildpack).

Delayed Job Uses Simpler Resources

As we discussed, Delayed Job runs on your existing database instance. You might need to increase:

the available memory
disk space
max connections

depending on the job load or the number of workers you run. But the only resource you need is the job processor.

On the other hand, Sidekiq requires a Redis instance to handle jobs. If you also use Redis as a cache store, it is recommended that you use a separate instance configured as a "persistent store" for Sidekiq jobs.

Since Redis works best when everything fits in memory, if you have too many jobs (for example, if Sidekiq stops processing them for some time due to an issue in the app), it might take some downtime to clear everything up. This is especially troublesome if you have Redis on the same server as your app. They will start competing for memory, leading to swapping and eventually destroying your app's performance.

One important point to note about Redis is that it has to be configured with maxmemory-policy noeviction to avoid silent drops of Sidekiq's data. Otherwise, you will find yourself missing jobs that need to be performed, without any trace.

A Side-Note: Paid Upgrades in Sidekiq

If you need extra features, Sidekiq comes with Pro and Enterprise versions.

The most notable addition to Pro is Batch Jobs that can run in parallel, be monitored, and interact as a group, invoking a callback when all jobs are done. Pro also has improved reliability features to ensure that no jobs are dropped silently, even during network problems.

The Enterprise version comes with yet more features. If you are looking for something that a regular Sidekiq installation can't solve, explore the paid Sidekiq features.

In practice, the free version of Sidekiq still works great. But it is good to know that there are paid options you can upgrade to as needed, instead of switching to a different solution.

Community and Development Status: Sidekiq Has the Edge

There is a huge community behind both Sidekiq and Delayed Job. However, it is not always easy to find quick answers to your questions in StackOverflow or the official documentation.

On the development side, things are not looking very bright for Delayed Job. There was some minor work done on Delayed Job in December 2021 and January 2022, but it doesn't seem like it is getting any major developments going forward. It appears to be in maintenance-only mode, and there are a lot of open issues on Github.

In contrast, Sidekiq is still under active development, and its creator is working full time on it. There are very few open issues, and they get addressed regularly.

Wrap-up: Sidekiq or Delayed Job? It Depends on Your Needs

In this post, we covered two major job processing systems for Rails applications — Sidekiq and Delayed Job — taking a look at some of their pros and cons.

There are different use cases for each. It all depends on the budget and scale of your operation.

If performance and long-term maintainability are of importance, Sidekiq is a no-brainer. On the other hand, if running costs are a concern, Delayed Job can help you there.

Whether you choose Delayed Job or Sidekiq, good luck with your project and happy coding!

👋 If you liked this article, take a look at other Ruby (on Rails) performance articles in our Ruby performance monitoring checklist.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Test and Optimize Your Ruby on Rails Database Performance

Roy Tomeij — 2022-01-26T00:00:00+00:00

In this article, you will learn how to test database performance in Rails and solve some of the most common database performance issues.

When you develop a Rails application, ActiveRecord is the default tool that manages your database. ActiveRecord provides an easy and fast interface to query and insert data using commands like .where, .save, .create, and .update. Rails does the work of converting these commands to SQL queries, which is a good thing, but sometimes can cause performance issues. It is important that you understand some of the common issues and how to optimize performance.

A Quick Note on ActiveRecord in Ruby on Rails

Rails ActiveRecord is a layer in Model-View-Controller (MVC) that manages your database by representing it as a business object. The ActiveRecord pattern uses the ORM technique to connect objects of an application to the relational database table management system.

Now, let's get going!

3 Ways to Identify and Test Database Performance Issues in Rails

1. Run Explain on ActiveRecord Queries

An explain statement displays information about the execution plan of an SQL query - how a query will be executed, including how many rows will be scanned, what index will be used, and how tables are joined.

The execution plan helps us figure out what slows down query execution by looking at:

What index you should add to improve the performance of a query.
If tables are joined in an optimal order. You can use STRAIGHT_JOIN to force table order in a join statement for better performance.

Explain works on SELECT, DELETE, INSERT, REPLACE and UPDATE statements as well.

Using explain with ActiveRecord is very straightforward. Enter the following:

2.4.2 :004 > User.where(id: 1).explain

 => EXPLAIN for: SELECT "users".* FROM "users" WHERE "users"."id" = $1 [["id", 1]]
                                QUERY PLAN
--------------------------------------------------------------------------
 Index Scan using users_pkey on users  (cost=0.14..8.16 rows=1 width=792)
   Index Cond: (id = 1)
(2 rows)

2.4.2 :005 >

Adding .explain at the end of the command provides the query plan for the ActiveRecord commands. The above example is a very simple query that uses the id (primary key) to query the table. The output of the explain statement shows that it is using the pkey. From this, you can be sure that the statement is optimal and fast.

You can try adding an .explain command to your slow queries to uncover the order of execution, along with the index used. If the query plan shows Seq Scan, the index is not being used and requires a query change or a new index to be added.

Here's another example with join:

2.4.2 :062 > User.where(id: 1).joins(:collaborations).explain

 => EXPLAIN for: SELECT "users".* FROM "users" INNER JOIN "collaborations" ON "collaborations"."user_id" = "users"."uid" WHERE "users"."id" = $1 [["id", 1]]
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Hash Join  (cost=8.17..27.31 rows=4 width=792)
   Hash Cond: ((collaborations.user_id)::text = (users.uid)::text)
   ->  Seq Scan on collaborations  (cost=0.00..17.20 rows=720 width=32)
   ->  Hash  (cost=8.16..8.16 rows=1 width=792)
         ->  Index Scan using users_pkey on users  (cost=0.14..8.16 rows=1 width=792)
               Index Cond: (id = 1)
(6 rows)

Here, the user table is joined with the collaborations table. When you look into the query plan, the collaboration table is using seq scan and is executing first — whereas the user table is using the pkey index and is executing later. You can add an index on the user_id column in the collaborations table to optimize this query. Explain helps in breaking down the query, so you can figure out where optimization is needed.

2. Measure Key Database Metrics

Query time is not the only metric to measure to see if a query is performant — look at several other database metrics, including:

CPU usage
Memory usage
Disk queue for waiting IO
Network bandwidth for inbound and outbound traffic
Available disk space
Throughput

The query may slow down when these metrics go over a certain threshold. You must look at a data point across a time range to understand performance issues.

The data point that needs to be measured depends on lots of factors, like:

The database type:
- Relational
- In-memory
- No-SQL
- Data-Warehouse
How the server is hosted:
- On-premises
- On the cloud

There is no single way to monitor database metrics — it depends on different factors.

3. Measure Rails App Performance using AppSignal

It can get difficult to manage all your performance metrics without a central place that gives you visibility over all queries. Adding performance code to every code block can be cumbersome and unmanageable.

With tools like AppSignal, you can easily integrate performance measurement into your application. AppSignal supports Rails out of the box. Learn about the simple AppSignal installation process from the 'AppSignal for Ruby' documentation.

Some of the critical metrics to keep an eye on are:

Slow queries
Database performance based on throughput
N+1 queries
Database latency
Number of active connections

Here's how an AppSignal dashboard might look:

7 Ways to Optimize Ruby on Rails Database Performance

1. Eager Loading for N+1 Queries

N+1 queries are the most common database performance problem. Let us see an example of an N+1 query where you have two models — user and project:

class User < ActiveRecord::Base
  has_many :projects
end

class Project < ActiveRecord::Base
  belongs_to :user
end

Now, if you want to find the user and project names, run the following code:

users = User.where(country: "Germany")

users.each do |user|
  puts "#{user.name} | #{user.project.name}"
end

The code above will query the database with each loop and cause performance issues. The total number of queries executed will be the number of users + 1.

Eliminating this issue is very simple: eager load the association. Simply add .includes(:projects) at the end of the query:

users = User.where(country: "Germany").includes(:projects)

Now, executing the loop will not query the database, as the query above eager loads the projects:

users.each do |user|
  puts "#{user.name} | #{user.project.name}"
end

Rails 6.1 provides strict loading to ensure that the association is eager loaded before it is accessed. To enable strict loading, add the following line in the model:

class User < ApplicationRecord
  has_many :projects, strict_loading: true
end

Now, when you try to access the projects without eager loading, Rails will throw an ActiveRecord::StrictLoadingViolationError exception.

If you don't have Rails 6.1, you can use gems like Bullet.

2. Use a Database Index

Databases provide indexes to help retrieve data faster. Using the explain command that we covered earlier, you will be able to figure out if a query is using a proper index.

You can change a slow query to use an already existing index or an added index that helps improve performance.

There are four different types of indexes in MySQL:

Primary key - Index is automatically added to the primary key, which also ensures that it is unique
Unique - Unique key index ensures that the items added in an attribute are always unique
Index - Added to attributes other than the primary key
Full text - Helps to query against character-based data

An index is stored in a B-Tree or a Hash format.

Indexes can be added to a single field or created as a composition of multiple fields. A composite index is useful to optimize queries that include multiple fields. When only one index is used, it requires a large dataset scan.

For example, in the following query, there are two fields:

User.where(project: "abc", country: "Germany")

There could be a lot of users in the project "abc" and the country field will be scanned. This can be a slow process because of the large result dataset. In this case, you can add a composite index to both the project and country fields to improve performance.

You can add the index to Rails with the following ActiveRecord migration commands:

Single index:

class AddIndexOnProjectToUsers < ActiveRecord::Migration[6.0]
  def change
    add_index :users, :project
  end
end

Composite index:

class AddIndexOnProjectAndCountryToUsers < ActiveRecord::Migration[6.0]
  def change
    add_index :users, [:project, :country]
  end
end

3. Use Limits

The more records that are returned, the slower performance can get. It is better to do multiple queries than a single query that returns a large data set.

User.where(country: "Germany").limit(100)

To fetch the next 100 batches, you can use the offset:

User.where(country: "Germany").limit(100).offset(100)

This will significantly improve performance. One thing to keep in mind is that with a higher offset, a query gets slower. Add a limit to the offset.

4. Use `find_each` To Load a Large Number of Items

When iterating over records, batch them in Rails for better performance:

User.where(country: "Germany").each do |user|
  puts user
end

This will query all the records in the database once and cause memory and database performance issues.

Using find_each or find_in_batches will help improve performance by doing the same operation in a batch:

User.where(country: "Germany").find_each do |user|
  puts user
end

By default, find_each queries result in a batch of 1,000. You can change the batch size by defining it as an argument:

User.where(country: "Germany").find_each(:batch_size: 5000)

You can also use find_in_batches based on the operation you need to perform. The difference between find_in_batches and find_each is that find_in_batches yields the result as an array of models instead of individual records.

5. Select Your Required Field Using Pluck

The Pluck command directly converts a query's result to an array instead of an ActiveRecord object.

If a query returns large results, using Pluck will improve code performance. Pluck will only select a required field from a database:

User.pluck(:id)
# SELECT "users"."id" FROM "users"

The result is fetched from an index, not the main table, and is more effective with queries involving sorts.

6. Use Bulk Operations

Bulk Delete A delete operation looping over the ActiveRecord object will delete records one at a time:

users = User.where(country: "Germany")

users.each do |user|
  user.delete
end

# >
# DELETE FROM users WHERE id = 1;
# DELETE FROM users WHERE id = 5;

Deleting each record requires many queries to be made to the database. Instead, it's optimal to use a single bulk delete_all query:

users = User.where(country: "Germany")

users.delete_all

# >
# DELETE FROM users WHERE users.country = 'Germany';

Bulk Create Some people don't realize that similar to bulk delete, you can also perform a bulk insert with ActiveRecord. This can reduce the n number of queries to only one. The ActiveRecord::Base create method accepts an array of hashes as input:

users = [
  {name: "Milap", email: "milap@country.com", country: "Germany"},
  {name: "Aastha", email: "aastha@country.com", country: "Germany"}
]

User.create(users)

# INSERT INTO users (name, email)
# VALUES
#   ('Milap', 'milap@country.com', 'Germany'),
#   ('Aastha', 'aastha@country.com', 'Germany')

7. Use In-Memory Calculation if Needed

In some instances, an in-memory calculation is preferable to querying. Suppose we want to find countries in our database that do not have a record of users:

countries = [
  "Germany",
  "UK",
  "Norway",
  "Netherlands"
]

countries.each do |country|
  unless User.where(country: country).exists?
    puts country
  end
end
# SELECT 1 AS one FROM `users` WHERE users`.`country` = 'Germany' LIMIT 1
# SELECT 1 AS one FROM `users` WHERE users`.`country` = 'UK' LIMIT 1
# SELECT 1 AS one FROM `users` WHERE users`.`country` = 'Norway' LIMIT 1
# SELECT 1 AS one FROM `users` WHERE users`.`country` = 'Netherlands' LIMIT 1

The above query required N queries to get the result. Instead of this, we can write a single query to find the users in the given countries and do the other calculation in memory:

existing_countries = User.distinct.pluck(:countries)
puts countries - existing_countries

# SELECT DISTINCT `users`.`countries` FROM `users`

You can cache to reuse the request-response cycle and reduce database load in some cases. Rails provides three types of caching techniques: page, action, and fragment caching (fragment caching is offered by default).

Wrap Up: Optimize Your Ruby on Rails Performance with ActiveRecord and AppSignal

Okay, time to recap! In this post, we covered three ways to identify and test database performance issues in Rails by:

Running Explain on ActiveRecord queries
Measuring key database metrics
Measuring Rails app performance using AppSignal

And seven ways to optimize your database performance, including the use of:

Eager loading for N+1 queries
A database index
limits
find_each to load a large number of items
Pluck to select a required field
Bulk operations
In-memory calculation

Rails makes it super easy and fast to develop an application. ActiveRecord helps with the productivity, reusability, and maintainability of database code. Understanding how ActiveRecord queries translate to SQL queries and execute is important.

However, the most crucial thing you need to optimize your Rails database performance is visibility over performance data. Performance issues are common, but you can resolve them once you have this visibility. You need a proper monitoring tool that provides database metrics. We like AppSignal ;)

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Next Level Ruby on Rails Application Monitoring with AppSignal

Roy Tomeij — 2022-01-05T00:00:00+00:00

In the first of this two-part series, we covered how to set up AppSignal in a Ruby on Rails application for many great insights out of the box. AppSignal can automatically track errors, monitor performance, and report metrics about some dependencies.

But, in many cases, each of our applications behaves in different ways, so we'll want more than just generic monitoring.

In this post, we will run through adding custom instrumentation and monitoring to a Ruby on Rails application. This will give you deeper insights into how your application is behaving.

Prerequisites if you want to follow along with the code:

An account on www.appsignal.com
Docker installed and running (to use docker-compose)

To follow along with this post, you will need to set up AppSignal in the sample application with your own AppSignal account.

Custom Instrumentation and Monitoring

When you need more than what AppSignal instruments out of the box, the AppSignal gem allows you to add custom instrumentation to your Rails application.

Instrumenting Parts of the Code

Let's say you want to add a new feature to an application. When a user visits /posts to view all posts, they should be able to filter for posts where the title begins with a specific letter (or something a lot more complex 🪄).

This new search functionality has already been implemented in the Post model with the method Post.where_title_starts_with. Let's update the PostsController#index to use the new method if a specific query parameter is present:

# app/controllers/posts_controller.rb
  def index
    starts_with = params[:starts_with]
    @posts = if starts_with.present?
               Post.where_title_starts_with(starts_with)
             else
               Post.all
             end
  end

This is such a core part of your application that you'll want to know how it performs and when that performance changes. AppSignal provides a few ways to do this.

First, we will instrument the contents of the Post.where_title_starts_with method. If you want to receive insights about any code blocks, you can use instrumentation blocks to wrap the code blocks. Update the method like this:

# app/models/post.rb
def self.where_title_starts_with(letter)
  Appsignal.instrument('Post.where_title_starts_with', "Fetch posts that start with letter") do
    Analytics.track_post_title_search(letter.downcase)
    select('*, pg_sleep(0.01)').where("title ILIKE :letter", letter: "#{letter.downcase}%").load
  end
end

Secondly, we also want to instrument the Analytics.track_post_title_search method being called because app/services/analytics.rb is doing some heavy processing. In this case, we will use method instrumentation to instrument the entire method more accurately:

# app/services/analytics.rb
require 'appsignal/integrations/object'

class Analytics
  def self.track_post_title_search(letter, sleep = sleep(1))
    # Some heavy processing
    sleep 1
  end
  appsignal_instrument_class_method :track_post_title_search
end

Insights

A few minutes after saving the above to the application, take a look at whatever new information is available on your AppSignal dashboard (if you don't see the information, you may need to restart the docker containers again). You can verify that the new feature works by visiting the posts index page with a search param: http://localhost:3000/posts?starts_with=f

Depending on the number of posts created in the database, the /posts endpoint will have become a lot slower.

If you open up performance issues on AppSignal ('Performance' -> 'Issue list') and view the PostsController#index action, lower down on the page, you should be able to see an 'Event Timeline'. This gives you a breakdown of how much time is spent running specific code:

This timeline exists for all performance events, but here, we can also see the custom instrumentation events. It shows us that calling Post.where_title_starts_with took 8.84 seconds to run, with 2.01 seconds used up by the Analytics.track_post_title_search method, and the remaining time used up by an active record query. You can also click into individual events for further investigation and see more information about their performance — e.g. the sql.active_record event.

AppSignal's instrumentation helpers give you a more detailed breakdown of the application code, so it's easier to gain insights into particular pieces of code that you think could impact an application's performance. You can learn more about this in AppSignal's instrumentation guide.

Exception Handling

Besides monitoring the performance of your code, you also need to know when the application doesn't behave as expected and where things go wrong. We've already seen how AppSignal reports exceptions that have not been handled by our code. But there's always a bit more than what comes out of the box.

You can start by removing existing code that causes an intermittent error. We see where this error occurs in the backtrace when viewing the error on the dashboard. Inside of app/controllers/pages_controller.rb remove the if statement:

class PagesController < ApplicationController
  def home
    CreateRandomPostsJob.perform_later
  end
end

Now, in the overview dashboard, the application's error rate will drop significantly.

Currently, when a user tries to view a post that doesn't exist, the application crashes — e.g., http://localhost:3000/posts/doesnotexist. Instead, you might want to show them a message. Add a rescue to where this could happen inside the PostsController. Update the #set_post method:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
    .
    .
    .
    private
    def set_post
      @post = Post.find(params[:id])
    rescue ActiveRecord::RecordNotFound => e
      render json: { error: "Oops. That post isn't here" } , status: :not_found
    end
    .
    .
end

Because we are manually handling the exception, it won't automatically be reported to AppSignal. You can still track errors manually by using Appsignal.set_error.

The easiest way to track an error is to simply add it as the function's only argument like Appsignal.set_error(e). We also want to take advantage of the ability to add more context to the request. AppSignal allows you to tag events with your own arbitrary information using Appsignal.tag_request:

def set_post
  Appsignal.tag_request(user_id: 'user-from-params', post_id: params[:id])
  @post = Post.find(params[:id])
rescue ActiveRecord::RecordNotFound => e
  Appsignal.set_error(e)
  render json: { error: "Oops. That post isn't here" }, status: :not_found
end

Now visit http://localhost:3000/posts/doesnotexist to verify that you get back the JSON response as expected, instead of having the application crash.

Insights

After you try to view a post that does not exist, the added updates ensure that the errors are reported to AppSignal. On the AppSignal dashboard, in 'Errors -> Issue list', find and view the new reported error (ActiveRecord::RecordNotFound).

The error detail page gives us useful context about the error, which by default, includes information about the request such as the HTTP method, parameters, and session data. You can see that the custom tags are also included, which gives you the ability to filter for all of the errors with a matching tag.

Because we tagged the request, it adds this information to errors and other instrumented events. If you view an individual post a few times, e.g., http://localhost:3000/posts/1, you will notice that the tags are also included when you look at the performance measurement ('Performance' -> 'Issue list' -> 'PostsController#show'). You can read more about tagging transactions in the guides.

This ability to add custom metadata to the transaction opens up many opportunities to help diagnose issues in production. A great example of this is adding Kubernetes metadata to your errors.

Metrics

Now that there is some custom instrumentation and error monitoring in place, you might realize that sometimes, there are large spikes in posts searches. Whenever a user searches, Analytics#track_post_title_search is called, which does some calculations and makes an API call to a third-party service. This third party has rate limits on the API. We want to track how often it is called to keep an eye on how close the application is to its limits.

AppSignal allows you to track custom metrics throughout the application as you wish.

First, we will track how often we're calling our analytics service and with what data, using a counter and tags:

#app/services/analytics.rb
require 'appsignal/integrations/object'

class Analytics
  def self.track_post_title_search(letter, sleep = sleep(1))
    Appsignal.increment_counter("track_post_search", 1, { letter: letter })
    # Some heavy processing
    sleep 1
  end
  appsignal_instrument_class_method :track_post_title_search
end

Secondly, we will also track the number of posts being returned in the PostsController#index, because this is a core part of the application's behavior, and we know it keeps growing:

#app/controllers/posts_controller.rb
class PostsController < ApplicationController
    .
    .
  def index
    .
        .
    Appsignal.set_gauge("posts_index", @posts.size, starts_with: params[:starts_with])
  end
end

The fake traffic script still running on the application will generate some data, but to add more variety, let's also search for posts starting with f, l, and v.

Insights

To view the custom metrics, you will need to create a dashboard with custom graphs on AppSignal. This can be done through the UI, but we will just import one for this example. Under the 'Dashboard' section, click on 'Add dashboard' and import a dashboard with the following:

{
  "title": "Post Search",
  "description": "Sample dashboard about posts search activity",
  "visuals": [
    {
      "title": "Analytics",
      "line_label": "%name% %letter%",
      "display": "LINE",
      "format": "number",
      "draw_null_as_zero": true,
      "metrics": [
        {
          "name": "track_post_search",
          "fields": [
            {
              "field": "COUNTER"
            }
          ],
          "tags": [
            {
              "key": "letter",
              "value": "*"
            }
          ]
        }
      ],
      "type": "timeseries"
    },
    {
      "title": "Search",
      "line_label": "%name% %starts_with%",
      "display": "LINE",
      "format": "number",
      "draw_null_as_zero": true,
      "metrics": [
        {
          "name": "posts_index",
          "fields": [
            {
              "field": "GAUGE"
            }
          ],
          "tags": [
            {
              "key": "starts_with",
              "value": "*"
            }
          ]
        }
      ],
      "type": "timeseries"
    }
  ]
}

You should see data on your graphs within a few minutes. Hovering over the lines shows you a legend of the metrics collected within the timeframe you're viewing.

Notice that this shows different lines for each tag value. Currently, our fake traffic is only searching for the letter e, but because we manually searched for other letters, you will see a new line on the graph for each one to indicate another data point.

Thought that was enough? AppSignal has more custom instrumentation solutions to offer that we won't be covering here. One that's worth a quick mention is breadcrumbs. Breadcrumbs allow you to track a list of actions in your application, which will then be reported in an error. You'll have even more specific and ordered information about what led up to an error.

Read all about custom instrumentation in the guides.

Wrap-up: Custom Instrumentation and Monitoring for Ruby Apps with AppSignal

Part 1 of this series covered the basic setup and use of AppSignal for your Ruby applications.

In this part, we've taken an application that already has great out-of-the-box monitoring and made it even better using the AppSignal gem.

AppSignal's custom instrumentation, error tracking, and performance monitoring features give you the insights you need into how your applications are behaving. It gives your application a lot out of the box while letting you take control when needed.

It's time to let your code run free in the wild, as long as you keep an eye on how it's doing. Happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby on Rails Application Monitoring with AppSignal

Roy Tomeij — 2021-12-01T00:00:00+00:00

When running and maintaining an application in a production environment, we want to feel confident about the behavior of the application and know when it isn't working as expected. At the least, we want to track errors, monitor performance, and collect specific metrics throughout the application.

Because we're developers and love maintainable solutions (right?), we also don't want to end up in a jumble of tools, integrations, and dependencies that make it harder for us to keep track of everything.

In this post, we will add AppSignal to a Ruby on Rails application to help give clear insights into application behavior.

Prerequisites if you want to follow along with the code:

An account on www.appsignal.com
Docker installed and running (to use docker-compose). *only required if using the sample application in this post

Setting up AppSignal

Note: You can skip this section if you're adding AppSignal to your own Rails application. In that case, you can also ignore the instructions related to Docker or how to restart your Rails server throughout the post — you instead restart/redeploy your application as you usually would.

We will use a sample application to get us started and run it on our machine using Docker.

Run the following commands to clone the repository, install dependencies, and run the application:

$ git clone --branch appsignal-setup/start-docker --single-branch https://github.com/choncou/sample_rails_app appsignal-setup
$ cd appsignal-setup
$ yarn start:compose

When you run this for the first time, it may take a while to build the docker images and download all the dependencies. Once it is complete, it will start the Rails server, PostgreSQL database, and Redis. You should see requests in the logs and view the running application on http://localhost:3000/.

About Our Sample Application

The application we're working with is very minimal, so there are only a few things to note.

We have a Post model, PostsController, and all CRUD actions exposed via the /posts route.

We also have a home page rendered by the PagesController, which enqueues a background job CreateRandomPostsJob to generate random posts asynchronously.

Lastly, we're using Sidekiq for background job processing.

Our small blogging platform also already has some active users. There is a script (./bin/traffic) running in the background together with the server that regularly makes a few requests to imitate traffic on the applications.

That's it. We've just released a product, and everything works perfectly...

Or, at least that's what we think until we start seeing user reports about the application's performance and users running into errors.

So how do we figure out what's going on? When we're working in a development environment, it's easier to look into errors. But when our application is running in production, this becomes harder. There must be a better way to find errors than to search through those server logs.

AppSignal to the rescue!

Getting Started with AppSignal

Let's begin our monitoring journey by adding AppSignal to our application. You'll need to log into your account at www.appsignal.com.

If you're a new AppSignal user, you will see the page below to add an application (whereas existing users need to click on 'Add app').

Select "Install for Ruby", which will then give us a few simple steps to follow:

Add the gem to your Gemfile
```
# Gemfile
gem 'appsignal'
```

Install the gem by running bundle install inside of the docker container, which you can access with yarn compose:sh

$ yarn compose:sh
# We are now in a bash console within a docker container
$ bundle install
# Stay in this console for the next command

Install AppSignal

During the installation, you'll need to respond to two prompts:

Do you want to change how this is displayed in AppSignal? (y/n): n
How do you want to configure AppSignal? : Input 1 because we will use a config file instead of just environment variables

Use the API key shown on the AppSignal setup page:

$ bundle exec appsignal install <your-api-key>
...
...
#####################################
## AppSignal installation complete ##
#####################################

  Sending example data to AppSignal...
  Example data sent!
  It may take about a minute for the data to appear on https://appsignal.com/accounts

  Please return to your browser and follow the instructions.

After you run the install script, you'll notice when the AppSignal setup is complete in your browser. You can then 'Go to app' to view your AppSignal dashboard for your development environment:

The installation command creates a config/appsignal.yml which allows you to configure AppSignal settings for different environments. You can learn more in the AppSignal Ruby configuration docs.

In the config/appsignal.yml is your push API key. You would usually remove it from this file and use an environment variable instead. For this post, that won't be necessary.

We need to restart the application to ensure that our changes take effect because we have installed a new gem. The simplest way is to stop the docker containers with ctrl-c inside of the terminal window running the server and start docker-compose again with:

$ yarn start:compose

This can take a while because it rebuilds the docker image when the dependencies change.

AppSignal Main Dashboard

Now that we have the application running with AppSignal installed, we can view the dashboard. To view all the applications and environments you have running on AppSignal, you can go to appsignal.com/accounts:

Click on the application name, and you can explore what AppSignal monitors out-of-the-box.

Within a few minutes, some data will already be collected because of your traffic generation script.

From the main dashboard, you can get a good high-level overview of a few essential application metrics. We're now going to dig into some of the more specific areas within AppSignal.

Errors Dashboard in AppSignal

On the errors page, you find a list of all errors reported by your application. When we ran the install script, a test error was sent to AppSignal. You can see that we also have an error that seems to be occurring pretty regularly. Clicking on the error will open up a page with more details, which can be useful to debug the issue.

In this case, we can see from the backtrace section that on the homepage of the application (/) there is a validation error occurring on app/controllers/pages_controller.rb:7 home.

Other AppSignal Dashboards

Under 'Dashboard' in the sidebar, you can see that AppSignal has provided us with a few different dashboards already.

The 'Overview' dashboard is available for every application and provides:

Your application error rate
Throughput
Response times
Other recent activity that you can dig into for more details

You also have two magic dashboards — one for Active Job and another for Sidekiq. AppSignal has built-in integrations for many popular frameworks and gems that work automatically with your Rails application.

When viewing these dashboards, you can see information relevant to the activity of that integration. In the Active Job dashboard, you'll see graphs for the number of jobs in each queue, the recorded duration of each job, and more. Give it a look!

Performance of Your App

Inside of Performance → Issue list, you can view a list of actions that AppSignal measures. These measurements can provide some valuable insights into what parts of your application consume a lot of performance.

For example, by clicking into the issue for the action PostsController#index, we can dive deeper into the controller's performance. You can quickly see a breakdown of where the most time is spent, or the most object allocations happen.

Quick Wins and Hidden Gems from AppSignal: Using Puma

There are a few hidden gems (not the Ruby kind) that you can get from AppSignal without much effort. Let's see how easily we can get these up and running with an example.

Inside of config/puma.rb, add the following line:

# config/puma.rb
plugin :appsignal

Before we restart the application, add the following environment variable after line 12 in the docker-compose.yml:

# docker-compose.yml
APP_REVISION: "latest_version_tag"

Because we've updated the Puma server config and our docker-compose config, we will need to restart the docker containers with ctrl-c and start-up docker-compose again with yarn start:compose.

Tracking Your New Insights

After the server restarts, allow it to run for about a minute, then head back to the AppSignal dashboard and refresh the page. There are two main areas with new information now:

The first is the new automatically generated Puma metrics magic dashboard. Because of AppSignal's built-in integration with Puma, we now have information about the application server's activity. Specifying the plugin in the config file has activated a minutely probe that reports metrics about the Puma server.

Lastly, across the entire AppSignal dashboard, there are now deployment markers. You can set the APP_REVISION environment variable to any value that helps you determine the current version of your application, such as a git commit SHA.

You can also see a history of your deployments by viewing the 'Deploys' section in the AppSignal dashboard and filter insights (such as Errors/Performance issue lists) by deploy. View how performance or reliability has changed between deploys, making it a bit easier to dig into which changes could have introduced a new bug.

Next Up: Custom Instrumentation and Monitoring for Ruby Apps

This post has shown how you can set up and use AppSignal in your applications to help increase your code's visibility in the wild.

We've demonstrated how AppSignal automatically integrates into Rails and some of the application's dependencies, such as Sidekiq and Puma. AppSignal works with many more Ruby frameworks and gems out-of-the-box. You can find a complete list of AppSignal's Ruby integrations here.

Part 2 of this series will cover how to add custom instrumentation and monitoring to your Ruby on Rails application for deeper insights.

Until next time!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Practical Garbage Collection Tuning in Ruby

Roy Tomeij — 2021-11-17T00:00:00+00:00

We have discovered that the below post is based on an article by Nate Berkopec from 2017 called 'Understanding Ruby GC through GC.stat'. It appears that parts of this article were plagiarised, something we were unaware of until the original author mentioned it. We run all of our articles through a plagiarism tool before publishing, but it didn't pick this up. We give huge apologies to Nate and our readers for this inadvertent error.

It is vital that you understand how garbage collection works in Ruby to stay in complete control of your app's performance.

In this post, we will dive into how to implement and customize garbage collection in Ruby.

Let's get going!

The Ruby Garbage Collector Module

The Ruby Garbage Collector module is an interface to Ruby's mark and sweep garbage collection mechanism.

While it runs automatically in the background when needed, the GC module lets you call the GC manually whenever required and gain insights into how garbage collection cycles are running. The module provides some parameters which you can alter to moderate performance.

Some of the most commonly used methods of this module are:

start/garbage_collect: This method initiates the garbage collection cycle manually.
enable/disable: These methods enable or disable the automatic garbage collection cycles. They return a boolean value indicating if the operation was successful.
stat: This method provides a list of keys and values that describe the performance of the GC module. We will take a look at these metrics in detail in the next section.

Understanding Ruby Garbage Collector Parameters

To understand how Ruby’s GC works internally, let’s look at the GC module's metrics. Run the following command on a freshly booted irb:

puts GC.stat

You will notice that a bunch of numbers pop up on your screen, looking something like this:

{
    :count=>12,
    :heap_allocated_pages=>49,
    :heap_sorted_length=>49,
    :heap_allocatable_pages=>0,
    :heap_available_slots=>19975,
    :heap_live_slots=>19099,
    :heap_free_slots=>876,
    :heap_final_slots=>0,
    :heap_marked_slots=>16659,
    :heap_eden_pages=>49,
    :heap_tomb_pages=>0,
    :total_allocated_pages=>49,
    :total_freed_pages=>0,
    :total_allocated_objects=>66358,
    :total_freed_objects=>47259,
    :malloc_increase_bytes=>16216,
    :malloc_increase_bytes_limit=>16777216,
    :minor_gc_count=>10,
    :major_gc_count=>2,
    :remembered_wb_unprotected_objects=>191,
    :remembered_wb_unprotected_objects_limit=>312,
    :old_objects=>16024,
    :old_objects_limit=>23556,
    :oldmalloc_increase_bytes=>158824,
    :oldmalloc_increase_bytes_limit=>16777216
}

This holds all the information about how garbage collection has been happening in the runtime. Let's examine each of these numbers in detail.

Counts in Ruby Garbage Collector

We'll begin by describing these keys:

{
    :count=>12,
    #…
    :minor_gc_count=>10,
    :major_gc_count=>2,
}

These are GC counts, and they convey pretty straightforward information. minor_gc_count and major_gc_count are the counts of each type of garbage collection run.

There are two types of garbage collections in Ruby.

Minor GC refers to a garbage collection attempt that tries to garbage collect only those objects that are new, i.e., they have survived three or fewer garbage collection cycles.

On the other hand, major GC is a garbage collection attempt that tries to garbage collect all objects, even those that have survived more than three garbage collection cycles. count is the sum of minor_gc_count and major_gc_count.

Tracking the GC count can be helpful for a few reasons. You can figure out if a particular job or process always triggers GCs and the number of times it triggers them. It might not be 100% accurate in cases like multithreaded applications, but it is a good starting point to figure out where your memory is bleeding.

Heap Numbers: Slots and Pages

Next, let’s talk about these keys, also known as heap numbers:

{
    # page numbers
    :heap_allocated_pages=>49,
    :heap_sorted_length=>49,
    :heap_allocatable_pages=>0,

    # slots
    :heap_available_slots=>19975,
    :heap_live_slots=>19099,
    :heap_free_slots=>876,
    :heap_final_slots=>0,
    :heap_marked_slots=>16659,

    # Eden and Tomb pages
    :heap_eden_pages=>49,
    :heap_tomb_pages=>0,
}

The heap that we are talking about here is a C data structure. It contains references to all the currently live Ruby objects. A heap page is composed of memory slots, and each slot includes information on only one live Ruby object:

heap_allocated_pages is the number of currently allocated heap pages. These pages can be completely empty, completely filled, or partly filled.
heap_sorted_length is the actual size that the heap has occupied in memory and is different from heap_allocated_pages, as the length is the length of the heap pages put together, not their count. If you initially allocated 10 pages and then freed one from the middle of the set, your heap_allocated_pages would be 9, but the heap_sorted_length would still be 10.
Finally, heap_allocatable_pages is the count of heaps that Ruby currently owns that can be used when needed.

Now, coming to the slots:

heap_available_slots is the total number of available slots in the heap pages.
heap_live_slots is the number of live objects in the memory.
heap_free_slots are slots in the allocated heap pages that are empty.
heap_final_slots is the count of slots whose objects have finalizers attached to them. Finalizers are Procs that run when an object is freed, similarly to destructors in OOPS.
heap_marked_slots is the count of old objects (i.e., the objects that have been around for more than 3 GC cycles) and write-barrier unprotected objects.

Then we have tomb_pages and eden_pages.

tomb_pages is the count of pages that contain no live objects. These pages are eventually released back to the operating system by Ruby.

On the other hand, eden_pages is the count of those pages that contain at least one live object, so they can't be released back to the operating system.

Consider monitoring the metric heap_free_slots if you face memory bloat issues in your application.

A high number of free slots (more than 250,000) usually indicates that you have a handful of controller actions that allocate many objects at once and then free them. This can permanently bloat the size of your running Ruby process.

Cumulative Numbers

{
    :total_allocated_pages=>49,
    :total_freed_pages=>0,
    :total_allocated_objects=>66358,
    :total_freed_objects=>47259,
}

These numbers are cumulative or additive in nature for the entire life of the process. They are never reset by the GC and can't technically go down. All four of these numbers are self-explanatory.

Garbage Collection Thresholds

To understand these numbers, you first need to understand when GC is triggered:

{
    :malloc_increase_bytes=>16216,
    :malloc_increase_bytes_limit=>16777216,
    :remembered_wb_unprotected_objects=>191,
    :remembered_wb_unprotected_objects_limit=>312,
    :old_objects=>16024,
    :old_objects_limit=>23556,
    :oldmalloc_increase_bytes=>158824,
    :oldmalloc_increase_bytes_limit=>16777216
}

Contrary to a common assumption that GC runs happen at fixed intervals, GC runs are triggered when Ruby starts running out of memory space. Minor GC happens when Ruby runs out of free_slots.

If Ruby is still low on free_slots after a minor GC run — or the threshold of oldmalloc, malloc, old object count, or shady/write-barrier-unprotected count is exceeded — a major GC run is triggered. The above part of gc.stat shows the values of these thresholds.

malloc_increase_bytes refers to the amount of memory allocated outside of the heap we talked about so far. When the size of an object exceeds the standard size of a memory slot — say, 40 bytes — Ruby mallocs some space somewhere else just for that object. When the total extra allocated space exceeds malloc_increase_bytes_limit, a major GC is triggered.

oldmalloc_increase_bytes is a similar threshold for old objects. old_objects is a count of object slots marked as old. When the number exceeds the old_objects_limit, major GC is triggered.

remembered_wb_unprotected_objects is the total count of objects that are not protected by the write-barrier and are a part of the remembered set.

The write-barrier is an interface between Ruby runtime and its objects, which allows the interpreter to track references to and from the object as soon as they are created.

C-extensions can make new references to objects without using the write-barrier, in which case the objects are marked shady or write-barrier unprotected. The remembered set is simply a list of old objects with at least one reference to a new object.

Customize Ruby Garbage Collection Performance

Now that you understand how Ruby GC manages your application's memory, it is time to look at the options available to customize GC's behavior.

Here are environment variables that you can use to moderate the performance of Ruby GC and, in turn, improve the performance of your application:

RUBY_GC_HEAP_INIT_SLOTS
RUBY_GC_HEAP_FREE_SLOTS
RUBY_GC_HEAP_GROWTH_FACTOR
RUBY_GC_HEAP_GROWTH_MAX_SLOTS
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR
and other variables

Let's talk about the important parameters here, one by one:

RUBY_GC_HEAP_INIT_SLOTS: defines the initial number of slots on the Ruby heap and is set to 10000 by default. You might want to change this parameter if you're sure that your app will initially allocate most of its objects.
RUBY_GC_HEAP_FREE_SLOTS: controls the minimum number of free slots that must be available right after a GC cycle. Its default value is 4096. This value is only used once at runtime during the first heap growth.
RUBY_GC_HEAP_GROWTH_FACTOR: the factor by which available heaps to the Ruby interpreter grow. Its default value is 1.8. Changing this makes little sense, as Ruby is already aggressive at heap growth. It will not make much difference if you try decreasing it since heaps are allocated on-demand in modern interpreters.
RUBY_GC_HEAP_GROWTH_MAX_SLOTS: the maximum number of slots Ruby can add to the heap space at once. The default value is 0, which stands for no limit on the number. If your app needs to allocate millions of objects in its lifetime, you might want to put a cap on this parameter. However, it will have a pretty low effect on the GC time of your app.
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR forces the interpreter to carry out a major GC cycle when the total number of old objects in memory = more than this number x number of old objects in memory after the last GC cycle. You might want to increase this number if you think many of your objects will become unused after they enter the old generation. However, this is rarely needed.
RUBY_GC_MALLOC_LIMIT is the minimum limit of malloc calls for the new generation. Its default value is 16 MB. RUBY_GC_MALLOC_LIMIT_MAX is the maximum limit for the same malloc calls. Its default value is 32 MB. You might want to increase these two limits if your application uses higher than average memory. However, be careful not to raise these too much. Otherwise, they might lead to higher peak memory consumption. Always increase these limits incrementally — say, by 4 or 8 MB.
RUBY_GC_MALLOC_LIMIT_GROWTH_FACTOR is the growth factor of the malloc limit for the new generation. Its default value is 1.4. You should consider increasing this number if your app allocates memory in chunks rather than whole at once.
Similarly, RUBY_GC_OLDMALLOC_LIMIT and RUBY_GC_OLDMALLOC_LIMIT_MAX are the minimum and maximum malloc limits for the old generation. The default values of these parameters are 16 MB and 128 MB.
RUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR is the growth factor of this limit. Its default value is 1.2. You can consider changing these with the new generation limits for maximum effect.

Fine-tuning Garbage Collection in Ruby

We discussed some common and simple ways to customize the GC module to help you improve your application's overall performance. However, these tweaks might not work in all cases. You need to figure out the memory usage pattern of your app before deciding what to customize.

On the flip side, you can consider running an automated test that finds the best values of these parameters for you. Tools like TuneMyGC are pretty straightforward when figuring out the best set of values for your environment variables.

Definitely look at GC parameters if your application is behaving weirdly. A small change here can go a long way to bring down your app's memory consumption and prevent memory bloats.

I hope this article has given you a good idea of what to look out for when customizing your Ruby Garbage Collection module. For more of an introduction, check out the Introduction to Garbage Collection Part I and Part II.

Have fun coding!

Optimistic Locking in Rails REST APIs

Roy Tomeij — 2021-10-20T00:00:00+00:00

Imagine the following hypothetical scenario: in a rental property management system, Employee A starts editing contact info for Rental X, adding some extra phone numbers. Around the same time, Employee B notices a typo in the contact info for exactly that Rental X and performs an update. A couple of minutes later, Employee A updates Rental X's contact info with the new phone numbers, and ... the update fixing the typo is now gone!

That's definitely not great! And this is a pretty trivial scenario. Imagine a similar conflict happening in a financial system!

Could we avoid such scenarios in the future? Fortunately, the answer is yes! We need concurrency protection and locking — specifically, optimistic locking — to prevent such problems.

Let's explore optimistic locking in Rails REST APIs.

'Lost Updates' and Optimistic Locking vs. Pessimistic Locking

The scenario we've just gone through is a type of 'Lost Update'. When two concurrent transactions update the same column of the same row, the second one will override the changes from the first one, essentially as if the first transaction never happened.

Usually, this problem can be addressed by:

Setting a proper Transaction Isolation level, which handles the problem on the database level.
Pessimistic locking — preventing concurrent transactions from updating the same row. The second transaction waits for the first transaction to finish before it even reads the data. The great advantage here is that it is impossible to operate on stale data. The major disadvantage, though, is that it also blocks reading the data from a given row.
Optimistic locking — stops the modification of the given row if its state at the time of modification is different from when it was read.

Our problem is not about concurrent database transactions (more like business transactions) — so the first solution is not really applicable. This means we're left with pessimistic locking and optimistic locking.

Pessimistic locking would prevent the lost update from happening in our hypothetical scenario in the first place. However, it would also make life difficult for users if it blocked access to data for a very long time (imagine it reading and editing some fields for 30 minutes or more).

Optimistic locking would be far less restrictive, as it would allow multiple users to access data. However, if several users start editing the data concurrently, only one can perform the operation. The rest would see an error stating that they operated on stale data and need to retry. Not ideal, but with proper UX, this might not necessarily be that painful.

Let's see how we could implement optimistic locking in a hypothetical Rails REST API.

Optimistic Locking in REST APIs

Before we get to implementation in the actual Rails app, let's think about what optimistic locking could look like in the context of general REST APIs.

As described above, we need to track an object's original state when reading it to compare against its later state during the update. If the state doesn't change since the last read, the operation is allowed. If it has changed, though, it will fail.

What we need to figure out in the context of REST APIs is:

When reading the data of a given resource, how do we express the current state of an object and return it in a response for a consumer?
How should consumers propagate the original state of a resource to an API when performing the update?
What should the API return to the consumer if the state has changed and the update is not possible?

The great news is that all these questions can be answered and handled with HTTP semantics.

As far as tracking the state of a resource goes, we can take advantage of Entity Tags (or ETags). We can return the resource's fingerprint/checksum/version number in the dedicated ETag header to API consumers to send later with the PATCH request. We can use an If-Match header, making it pretty straightforward for the API server to check if the resource has changed or not. It is just a case of comparing the checksums/version number/whatever else you choose as the ETag.

The request will succeed if the current ETag and If-Match values are the same. If not, the API should respond with the rarely-used 412 Precondition Failed status, the most appropriate and expressive status that we can use for this purpose.

There is one other possible scenario. We can only compare the ETags if the API consumer provides the If-Match header. What if it doesn't? You could ignore concurrency protection and forget about optimistic locking, but that might not be ideal. One other solution would be to make it a requirement to provide the If-Match header and respond with 428 Precondition Required status if it's not.

Now that we have a solid overview of how optimistic locking could work in REST APIs, let's implement it in Rails.

Optimistic Locking in Rails

The great news is that Rails offers optimistic locking out-of-the-box — we can use the feature provided by ActiveRecord::Locking::Optimistic. When you add the lock_version column (or whatever else you want, although that requires additional declarations on the model level to define the locking column), ActiveRecord will increment it after each change and check if the currently assigned version is the expected one. If it's stale, the ActiveRecord::StaleObjectError exception will be raised on the update/destroy attempt.

The easiest way to handle optimistic locking in our API is to use the value from lock_version as an ETag. Let's do this as the first step in our hypothetical RentalsController:

class RentalsController
  after_action :assign_etag, only: [:show]

  def show
    @rental = Rental.find(params[:id])
    respond_with @rental
  end

  private

  def assign_etag
    response.headers["ETag"] = @rental.lock_version
  end
end

This is, of course, a very simplified version of the controller as we are only interested in whatever is required for optimistic locking, not authentication, authorization, or other concepts. This is enough to expose the proper ETag to the consumer. Let's now take care of the If-Match header that the consumers can provide:

class RentalsController
  after_action :assign_etag, only: [:show, :update]

  def show
    @rental = Rental.find(params[:id])
    respond_with @rental
  end

  def update
    @rental = Rental.find(params[:id])
    @rental.update(rental_params)
    respond_with @rental
  end

  private

  def assign_etag
    response.headers["ETag"] = @rental.lock_version
  end

  def rental_params
    params
      .require(:rental)
      .permit(:some, :permitted, :attributes).merge(lock_version: lock_version_from_if_match_header)
  end

  def lock_version_from_if_match_header
    request.headers["If-Match"].to_i
  end
end

And that's actually enough to have the minimal version of optimistic locking working! Although, clearly, we don't want to return 500 responses if there is any conflict. We will make If-Match required for any update too:

class RentalsController
  before_action :ensure_if_match_header_provided, only: [:update]
  after_action :assign_etag, only: [:show, :update]

  rescue_from ActiveRecord::StaleObjectError do
    head 412
  end

  def show
    @rental = Rental.find(params[:id])
    respond_with @rental
  end

  def update
    @rental = Rental.find(params[:id])
    @rental.update(rental_params)
    respond_with @rental
  end

  private

  def ensure_if_match_header_provided
     request.headers["If-Match"].present? or head 428 and return
  end

  def assign_etag
    response.headers["ETag"] = @rental.lock_version
  end

  def rental_params
    params
      .require(:rental)
      .permit(:some, :permitted, :attributes)
      .merge(lock_version: lock_version_from_if_match_header)
  end

  def lock_version_from_if_match_header
    request.headers["If-Match"].to_i
  end
end

And that's pretty much everything required to implement all the functionality that we discussed earlier. We could improve way more things — e.g., by providing some extra error messages besides just the response code — but that would be outside the scope of this article.

Wrap-up: The Importance of Optimistic Locking in Rails APIs

Concurrency protection is often overlooked when designing REST APIs, which can lead to severe consequences.

Nevertheless, implementing optimistic locking in Rails APIs is pretty straightforward — as demonstrated in this article — and will help avoid potentially critical issues.

Have fun coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Reduce Memory Bloat in Ruby

Roy Tomeij — 2021-09-21T00:00:00+00:00

The issue of memory bloat in Ruby applications is a topic of frequent discussion. In this post, we will look at how Ruby memory management can go wrong, and what you can do to prevent your Ruby application from blowing up.

First, we need to understand what bloat means in the context of an application’s memory.

Let's dive in!

What Is Memory Bloat in Ruby?

Memory bloat is when your application’s memory usage increases all of a sudden without an apparent explanation. This increase may or may not be quick, but in most cases, it becomes continuous. It is different from a memory leak, which results from multiple executions taking place over some time.

Memory bloat can be one of the worst things to happen to your Ruby application in production. However, it is avoidable if you take proper measures. For example, if you notice a spike in your application’s memory usage, it is best to check for signs of memory bloat before troubleshooting other issues.

Before we explore how to diagnose and fix memory bloat, let’s take a quick walkthrough of Ruby’s memory architecture.

Ruby's Memory Structure

Ruby's memory usage revolves around specific elements that manage the judicious use of available system resources. These elements include the Ruby language, host operating system, and the system kernel.

Apart from these, the garbage collection process also plays a vital role in determining how Ruby memory is managed and reused.

Ruby Heap Pages and Memory Slots

The Ruby language organizes objects into segments called heap pages. The entire heap space (available memory) is divided into used and empty sections. These heap pages are further split into equal-sized slots, which allow one unit-sized object each.

When allocating memory to a new object, Ruby first looks in the used heap space for free slots. If none are found, it allocates a new heap page from the empty section.

Memory slots are small memory locations, each nearly 40 bytes in size. The data that overflows out of these slots is stored in a different area outside the heap page and each slot stores a pointer to the external information.

A system’s memory allocator makes all allocations in the Ruby runtime environment, including heap pages and external data pointers.

Operating System Memory Allocation in Ruby

Memory allocation calls made by the Ruby language are handled and responded to by the host operating system’s memory allocator.

Usually, the memory allocator consists of a group of C functions, namely malloc, calloc, realloc, and free. Let’s quickly take a look at each:

Malloc: Malloc stands for memory allocation, and it is used to allocate free memory to objects. It takes in the size of the memory to be allocated and returns a pointer to the starting index of the allocated memory block.
Calloc: Calloc stands for contiguous allocation, and it allows the Ruby language to allocate consecutive blocks of memory. It is beneficial when allocating object arrays of known length.
Realloc: Realloc stands for re-allocation, and it allows the language to re-allocate memory with a new size.
Free: Free is used to clear out pre-allocated sets of memory locations. It takes in a pointer to the starting index of the memory block that has to be freed.

Garbage Collection in Ruby

The garbage collection process of a language runtime dramatically affects how well it utilizes its available memory.

Ruby happens to have pretty advanced garbage collection that uses all of the above-described API methods to optimize application memory consumption at all times.

An interesting fact about the garbage collection process in Ruby is that it halts the complete application! This ensures that no new object allocation happens during garbage collection. Because of this, your garbage collection routine should be infrequent and as quick as possible.

Two Common Causes of Memory Bloat in Ruby

This section will discuss two of the most significant reasons why memory bloats occur in Ruby: fragmentation and slow release.

Memory Fragmentation

Memory fragmentation is when object allocations in memory are scattered all over, reducing the number of contiguous chunks of free memory. Memory can't be allocated to objects without contiguous blocks, even if more than enough free memory is available on the disk. This problem can occur in any programming language or environment, and each language has its methods for solving the issue.

Fragmentation can occur at two different levels: the language’s level and the memory allocator’s level. Let’s take a look at both of these in detail.

Fragmentation at the Ruby Level

Fragmentation at the language level occurs due to the design of the garbage collection process. The garbage collection process marks a Ruby heap page slot as free, allowing reuse of that slot to allocate another object in the memory. If a complete Ruby heap page consists of only free slots, then that heap page can be released to the memory allocator for reuse.

But what if a tiny number of slots are not marked free on a heap? It will not be released back to the memory allocator. Now, think about the many slots in various heap pages simultaneously allocated and freed by garbage collection. It is improbable for entire heap pages to be released at once. Even though the garbage collection process frees memory, it can't be reused by the memory allocator since memory blocks partially occupy it.

Fragmentation at the Memory Allocator Level

The memory allocator itself faces a similar problem: it has to release OS heaps once they are all entirely free. But it is improbable that an entire OS heap can be freed at once considering the random nature of the garbage collection process.

The memory allocator also provisions OS heaps from system memory for an application’s use. It will simply move to provision new OS heaps, even if the existing heaps have enough free memory to satisfy the application’s memory requirements. This is the perfect recipe for a spike in the memory metrics of an application.

Slow Release

Another important cause of memory bloat in Ruby is a slow release of freed memory back to the system. In this situation, memory is freed much more slowly than the rate at which new memory blocks are allocated to objects. While this is not a conventional or a rookie issue to solve, it intensely affects memory bloat — even more so than fragmentation!

Upon investigating the memory allocator’s source, it turns out that the allocator is designed to release OS pages just at the end of OS heaps, and even then, only very occasionally. This is probably for performance reasons, but it can backfire and be counter-productive.

How to Fix Ruby Memory Bloat

Now that we know what causes Ruby’s memory to bloat, let’s take a look at how you can fix these issues and improve your app's performance through defragmentation and trimming.

Fix Ruby Memory Bloat with Defragmentation

Fragmentation happens due to the design of garbage collection, and there isn't much you can do to fix it. However, there are a few steps that you can follow to reduce the chances of ending up with a fragmented memory disk:

If you declare a reference to an object that uses a considerable amount of memory, make sure you free it manually when its job is done.
Try to declare all of your static object allocations in one big block. This will put all your permanent classes, objects, and other data on the same heap pages. Later, when you play around with dynamic allocations, you won’t have to worry about the static heap pages.
If possible, try to do large dynamic allocations at the beginning of your code. This will put them close to your bigger static allocation memory blocks and will keep the rest of your memory clean.
If you use a small and rarely cleared cache, it is better to group it with permanent static allocation in the beginning. You can even consider removing it altogether to improve your app’s memory management.
Use jemalloc instead of the standard glibc memory allocator. This small tweak can bring down your Ruby memory consumption by up to four times. The only caveat here is that it might not be compatible in all environments, so be sure to test your app thoroughly before rolling into production.

Trimming to Fix Ruby Memory Bloat

You need to override the garbage collection process and release memory more often to fix slow memory release. There is an API that can do this called malloc_trim. All you need to do is modify Ruby to call this function during the garbage collection process.

Here’s the modified Ruby 2.6 code that calls malloc_trim in gc.c function gc_start:

gc_prof_timer_start(objspace);
{
    gc_marks(objspace, do_full_mark);
    // BEGIN MODIFICATION
    if (do_full_mark)
    {
        malloc_trim(0);
    }
    // END MODIFICATION
}
gc_prof_timer_stop(objspace);

Note: This is not recommended in production applications as it can make your app unstable. However, it comes in handy when slow memory release causes major hits to your performance, and you are ready to try out any solution.

Wrap-up and Next Steps

Memory bloats are tricky to identify and even more challenging to fix.

This article looked at two significant reasons behind memory bloats in Ruby apps — fragmentation and slow release — and two fixes: defragmentation and trimming.

You must keep a constant eye on your app’s metrics to identify an impending bloat incident and fix it before it takes your app down.

I hope that I've helped you take some steps towards fixing memory bloat in your Ruby application.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Responsible Monkeypatching in Ruby

Roy Tomeij — 2021-08-24T00:00:00+00:00

When I first started writing Ruby code professionally back in 2011, one of the things that impressed me the most about the language was its flexibility. It felt as though with Ruby, everything was possible. Compared to the rigidity of languages like C# and Java, Ruby programs almost seemed like they were alive.

Consider how many incredible things you can do in a Ruby program. You can define and delete methods at will. You can call methods that don't exist. You can conjure entire nameless classes out of thin air. It's absolutely wild.

But that's not where the story ends. While you can apply these techniques inside your own code, Ruby also lets you apply them to anything loaded into the virtual machine. In other words, you can mess with other people's code as easily as you can your own.

What Are Monkeypatches?

Enter the monkeypatch.

In short, monkeypatches "monkey with" existing code. The existing code is often code you don't have direct access to, like code from a gem or from the Ruby standard library. Patches are usually designed to alter the original code's behavior to fix a bug, improve performance, etc.

The most unsophisticated monkeypatches reopen ruby classes and modify behavior by adding or overriding methods.

This reopening idea is core to Ruby's object model. Whereas in Java, classes can only be defined once, Ruby classes (and modules for that matter) can be defined multiple times. When we define a class a second, third, fourth time, etc, we say that we're reopening it. Any new methods we define are added to the existing class definition and can be called on instances of that class.

This short example illustrates the class reopening concept:

class Sounds
  def honk
    "Honk!"
  end
end

class Sounds
  def squeak
    "Squeak!"
  end
end

sounds = Sounds.new
sounds.honk    # => "Honk!"
sounds.squeak  # => "Squeak!"

Notice that both the #honk and #squeak methods are available on the Sounds class through the magic of reopening.

Essentially, monkeypatching is the act of reopening classes in 3rd-party code.

Is Monkeypatching Dangerous?

If the previous sentence scared you, that's probably a good thing. Monkeypatching, especially when done carelessly, can cause real chaos.

Consider for a moment what would happen if we were to redefine Array#<<:

class Array
  def <<(*args)
    # do nothing 😈
  end
end

With these four lines of code, every single array instance in the entire program is now broken.

What's more, the original implementation of #<< is gone. Aside from restarting the Ruby process, there's no way to get it back.

When Monkeypatching Goes Horribly Wrong

Back in 2011, I worked for a prominent social networking company. At the time, the codebase was a massive Rails monolith running on Ruby 1.8.7. Several hundred engineers contributed to the codebase on a daily basis, and the pace of development was very fast.

At one point, my team decided to monkeypatch String#% to make writing plurals easier for internationalization purposes. Here's an example of what our patch could do:

replacements = {
  horse_count: 3,
  horses: {
    one: "is 1 horse",
    other: "are %{horse_count} horses"
  }
}

# "there are 3 horses in the barn"
"there %{horse_count:horses} in the barn" % replacements

We wrote up the patch and eventually got it deployed into production... only to find that it didn't work. Our users were seeing strings with literal %{...} characters instead of nicely pluralized text. It didn't make sense. The patch had worked perfectly well in the development environment on my laptop. Why wasn't it working in production?

Initially, we thought we'd found a bug in Ruby itself, only later, to find that a production Rails console produced a different result than a Rails console in development. Since both consoles ran on the same Ruby version, we could rule out a bug in the Ruby standard library. Something else was going on.

After several days of head-scratching, a co-worker was able to track down a Rails initializer that added another implementation of String#% that none of us had seen before. To further complicate things, this earlier implementation also contained a bug, so the results we saw in the production console differed from Ruby's official documentation.

That's not the end of the story though. In tracking down the earlier monkeypatch, we also found no less than three others, all patching the same method. We looked at each other in horror. How did this ever work??

We eventually chalked the inconsistent behavior up to Rails' eager loading. In development, Rails lazy loads Ruby files, i.e., only loads them when they are required. In production, however, Rails loads all of the app's Ruby files at initialization. This can throw a big monkey wrench into monkeypatching.

Consequences of Reopening a Class

In this case, each of the monkeypatches reopened the String class and effectively replaced the existing version of the #% method with another one. There are several major pitfalls to this approach:

The last patch applied "wins", meaning, behavior is dependent on load order
There's no way to access the original implementation
Patches leave almost no audit trail, which makes them very difficult to find later

Not surprisingly, perhaps, we ran into all of these.

At first, we didn't even know there were other monkeypatches at play. Because of the bug in the winning method, it appeared the original implementation was broken. When we discovered the other competing patches, it was impossible to tell which won without adding copious puts statements.

Finally, even when we did discover which method won in development, a different one would win in production. It was also programmatically difficult to tell which patch had been applied last since Ruby 1.8 didn't have the wonderful Method#source_location method we now have.

I spent at least a week trying to figure out what was going on, time I essentially wasted chasing an entirely avoidable problem.

Eventually, we decided to introduce the LocalizedString wrapper class with an accompanying #% method. Our String monkeypatch then simply became:

class String
  def localize
    LocalizedString.new(self)
  end
end

When Monkeypatching Fails

In my experience, monkeypatches often fail for one of two reasons:

The patch itself is broken. In the codebase I mentioned above, not only were there several competing implementations of the same method, but the method that "won" didn't work.
Assumptions are invalid. The host code has been updated and the patch no longer applies as written.

Let's look at the second bullet point in more detail.

Even the Best-Laid Plans...

Monkeypatching often fails for the same reason you reached for it in the first place — because you don't have access to the original code. For precisely that reason, the original code can change out from under you.

Consider this example in a gem that your app depends on:

class Sale
  def initialize(amount, discount_pct, tax_rate = nil)
    @amount = amount
    @discount_pct = discount_pct
    @tax_rate = tax_rate
  end

  def total
    discounted_amount + sales_tax
  end

  private

  def discounted_amount
    @amount * (1 - @discount_pct)
  end

  def sales_tax
    if @tax_rate
      discounted_amount * @tax_rate
    else
      0
    end
  end
end

Wait, that's not right. Sales tax should be applied to the full amount, not the discounted amount. You submit a pull request to the project. While you're waiting for the maintainer to merge your PR, you add this monkeypatch to your app:

class Sale
  private

  def sales_tax
    if @tax_rate
      @amount * @tax_rate
    else
      0
    end
  end
end

It works perfectly. You check it in and forget about it.

Everything is fine for a long time. Then one day the finance team sends you an email asking why the company hasn't been collecting sales tax for a month.

Confused, you start digging into the issue and eventually notice one of your co-workers recently updated the gem that contains the Sale class. Here's the updated code:

class Sale
  def initialize(amount, discount_pct, sales_tax_rate = nil)
    @amount = amount
    @discount_pct = discount_pct
    @sales_tax_rate = sales_tax_rate
  end

  def total
    discounted_amount + sales_tax
  end

  private

  def discounted_amount
    @amount * (1 - @discount_pct)
  end

  def sales_tax
    if @sales_tax_rate
      discounted_amount * @sales_tax_rate
    else
      0
    end
  end
end

Looks like one of the project maintainers renamed the @tax_rate instance variable to @sales_tax_rate. The monkeypatch checks the value of the old @tax_rate variable, which is always nil. Nobody noticed because no errors were ever raised. The app chugged along as if nothing had happened.

Why Monkeypatch?

Given these examples, it might seem like monkeypatching just isn't worth the potential headaches. So why do we do it? In my opinion, there are three major use-cases:

To fix broken or incomplete 3rd-party code
To quickly test a change or multiple changes in development
To wrap existing functionality with instrumentation or annotation code

In some cases, the only viable way to address a bug or performance issue in 3rd-party code is to apply a monkeypatch.

But with great power comes great responsibility.

Monkeypatching Responsibly

I like to frame the monkeypatching conversation around responsibility instead of whether or not it's good or bad. Sure, monkeypatching can cause chaos when done poorly. However, if done with some care and diligence, there's no reason to avoid reaching for it when the situation warrants it.

Here's the list of rules I try to follow:

Wrap the patch in a module with an obvious name and use Module#prepend to apply it
Make sure you're patching the right thing
Limit the patch's surface area
Give yourself escape hatches
Over-communicate

For the remainder of this article, we're going to use these rules to write up a monkeypatch for Rails' DateTimeSelector so it optionally skips rendering discarded fields. This is a change I actually tried to make to Rails a few years ago. You can find the details here.

You don't have to know much about discarded fields to understand the monkeypatch, though. At the end of the day, all it does is replace a single method called build_hidden with one that effectively does nothing.

Let's get started!

Use `Module#prepend`

In the codebase I encountered in my previous role, all the implementations of String#% were applied by reopening the String class. Here's an augmented list of the drawbacks I mentioned earlier:

Errors appear to have originated from the host class or module instead of from the patch code
Any methods you define in the patch replace existing methods with the same name, meaning, there's no way of invoking the original implementation.
There's no way to know which patches were applied and therefore, which methods "won"
Patches leave almost no audit trail, which makes them very difficult to find later

Instead, it's much better to wrap your patch in a module and apply it using Module#prepend. Doing so leaves you free to call the original implementation, and a quick call to Module#ancestors will show the patch in the inheritance hierarchy so it's easier to find if things go wrong.

Finally, a simple prepend statement is easy to comment out if you need to disable the patch for some reason.

Here are the beginnings of a module for our Rails monkeypatch:

module RenderDiscardedMonkeypatch
end

ActionView::Helpers::DateTimeSelector.prepend(
  RenderDiscardedMonkeypatch
)

Patch the Right Thing

If you take one thing away from this article, let it be this: don't apply a monkeypatch unless you know you're patching the right code. In most cases, it should be possible to verify programmatically that your assumptions still hold (this is Ruby after all). Here's a checklist:

Make sure the class or module you're trying to patch exists
Make sure methods exist and have the right arity
If the code you're patching lives in a gem, check the gem's version
Bail out with a helpful error message if assumptions don't hold

Right off the bat, our patch code has made a pretty important assumption. It assumes a constant called ActionView::Helpers::DateTimeSelector exists and is a class or module.

Check Class/Module

Let's ensure that constant exists before trying to patch it:

module RenderDiscardedMonkeypatch
end

const = begin
  Kernel.const_get('ActionView::Helpers::DateTimeSelector')
rescue NameError
end

if const
  const.prepend(RenderDiscardedMonkeypatch)
end

Great, but now we've leaked a local variable (const) into the global scope. Let's fix that:

module RenderDiscardedMonkeypatch
  def self.apply_patch
    const = begin
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    if const
      const.prepend(self)
    end
  end
end

RenderDiscardedMonkeypatch.apply_patch

Check Methods

Next, let's introduce the patched build_hidden method. Let's also add a check to make sure it exists and accepts the right number of arguments (i.e. has the right arity). If those assumptions don't hold, something's probably wrong:

module RenderDiscardedMonkeypatch
  class << self
    def apply_patch
      const = find_const
      mtd = find_method(const)

      if const && mtd && mtd.arity == 2
        const.prepend(self)
      end
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
    end
  end

  def build_hidden(type, value)
    ''
  end
end

RenderDiscardedMonkeypatch.apply_patch

Check Gem Versions

Finally, let's check that we're using the right version of Rails. If Rails gets upgraded, we might need to update the patch too (or get rid of it entirely).

module RenderDiscardedMonkeypatch
  class << self
    def apply_patch
      const = find_const
      mtd = find_method(const)

      if const && mtd && mtd.arity == 2 && rails_version_ok?
        const.prepend(self)
      end
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
    end

    def rails_version_ok?
      Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR == 1
    end
  end

  def build_hidden(type, value)
    ''
  end
end

RenderDiscardedMonkeypatch.apply_patch

Bail Out Helpfully

If your verification code uncovers a discrepancy between expectations and reality, it's a good idea to raise an error or at least print a helpful warning message. The idea here is to alert you and your co-workers when something seems amiss.

Here's how we might modify our Rails patch:

module RenderDiscardedMonkeypatch
  class << self
    def apply_patch
      const = find_const
      mtd = find_method(const)

      unless const && mtd && mtd.arity == 2
        raise "Could not find class or method when patching "\
          "ActionView's date_select helper. Please investigate."
      end

      unless rails_version_ok?
        puts "WARNING: It looks like Rails has been upgraded since "\
          "ActionView's date_select helper was monkeypatched in "\
          "#{__FILE__}. Please reevaluate the patch."
      end

      const.prepend(self)
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
    end

    def rails_version_ok?
      Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR == 1
    end
  end

  def build_hidden(type, value)
    ''
  end
end

RenderDiscardedMonkeypatch.apply_patch

Limit Surface Area

While it may seem perfectly innocuous to define helper methods in a monkeypatch, remember that any methods defined via Module#prepend will override existing ones through the magic of inheritance. While it might seem as though a host class or module doesn't define a particular method, it's difficult to know for sure. For this reason, I try to only define methods I intend to patch.

Note that this also applies to methods defined in the object's singleton class, i.e. methods defined inside class << self.

Here's how to modify our Rails patch to only replace the one #build_hidden method:

module RenderDiscardedMonkeypatch
  class << self
    def apply_patch
      const = find_const
      mtd = find_method(const)

      unless const && mtd && mtd.arity == 2
        raise "Could not find class or method when patching"\
          "ActionView's date_select helper. Please investigate."
      end

      unless rails_version_ok?
        puts "WARNING: It looks like Rails has been upgraded since"\
          "ActionView's date_selet helper was monkeypatched in "\
          "#{__FILE__}. Please reevaluate the patch."
      end

      const.prepend(InstanceMethods)
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
    end

    def rails_version_ok?
      Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR == 1
    end
  end

  module InstanceMethods
    def build_hidden(type, value)
      ''
    end
  end
end

RenderDiscardedMonkeypatch.apply_patch

Give Yourself Escape Hatches

When possible, I like to make my monkeypatch's functionality opt-in. That's only really an option if you have control over where the patched code is invoked. In the case of our Rails patch, it's doable via the @options hash in DateTimeSelector:

module RenderDiscardedMonkeypatch
  class << self
    def apply_patch
      const = find_const
      mtd = find_method(const)

      unless const && mtd && mtd.arity == 2
        raise "Could not find class or method when patching"\
          "ActionView's date_select helper. Please investigate."
      end

      unless rails_version_ok?
        puts "WARNING: It looks like Rails has been upgraded since"\
          "ActionView's date_selet helper was monkeypatched in "\
          "#{__FILE__}. Please reevaluate the patch."
      end

      const.prepend(InstanceMethods)
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
    end

    def rails_version_ok?
      Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR == 1
    end
  end

  module InstanceMethods
    def build_hidden(type, value)
      if @options.fetch(:render_discarded, true)
        super
      else
        ''
      end
    end
  end
end

RenderDiscardedMonkeypatch.apply_patch

Nice! Now callers can opt-in by calling the date_select helper with the new option. No other codepaths are affected:

date_select(@user, :date_of_birth, {
  order: [:month, :day],
  render_discarded: false
})

Over-Communicate

The last piece of advice I have for you is perhaps the most important — communicating what your patch does and when it's time to re-examine it. Your goal with monkeypatches should always be to eventually remove the patch altogether. To that end, a responsible monkeypatch includes comments that:

Describe what the patch does
Explain why the patch is necessary
Outline the assumptions the patch makes
Specify a date in the future when your team should reconsider alternative solutions, like pulling in an updated gem
Include links to relevant pull requests, blog posts, StackOverflow answers, etc.

You might even print a warning or fail a test on a predetermined date to urge the team to reconfirm the patch's assumptions and consider whether or not it's still necessary.

Here's the final version of our Rails date_select patch, complete with comments and a date check:

# ActionView's date_select helper provides the option to "discard" certain
# fields. Discarded fields are (confusingly) still rendered to the page
# using hidden inputs, i.e. <input type="hidden" />. This patch adds an
# additional option to the date_select helper that allows the caller to
# skip rendering the chosen fields altogether. For example, to render all
# but the year field, you might have this in one of your views:
#
# date_select(:date_of_birth, order: [:month, :day])
#
# or, equivalently:
#
# date_select(:date_of_birth, discard_year: true)
#
# To avoid rendering the year field altogether, set :render_discarded to
# false:
#
# date_select(:date_of_birth, discard_year: true, render_discarded: false)
#
# This patch assumes the #build_hidden method exists on
# ActionView::Helpers::DateTimeSelector and accepts two arguments.
#
module RenderDiscardedMonkeypatch
  class << self
    EXPIRATION_DATE = Date.new(2021, 8, 15)

    def apply_patch
      if Date.today > EXPIRATION_DATE
        puts "WARNING: Please re-evaluate whether or not the ActionView "\
          "date_select patch present in #{__FILE__} is still necessary."
      end

      const = find_const
      mtd = find_method(const)

      # make sure the class we want to patch exists;
      # make sure the #build_hidden method exists and accepts exactly
      # two arguments
      unless const && mtd && mtd.arity == 2
        raise "Could not find class or method when patching "\
          "ActionView's date_select helper. Please investigate."
      end

      # if rails has been upgraded, make sure this patch is still
      # necessary
      unless rails_version_ok?
        puts "WARNING: It looks like Rails has been upgraded since "\
          "ActionView's date_select helper was monkeypatched in "\
          "#{__FILE__}. Please re-evaluate the patch."
      end

      # actually apply the patch
      const.prepend(InstanceMethods)
    end

    private

    def find_const
      Kernel.const_get('ActionView::Helpers::DateTimeSelector')
    rescue NameError
      # return nil if the constant doesn't exist
    end

    def find_method(const)
      return unless const
      const.instance_method(:build_hidden)
    rescue NameError
      # return nil if the method doesn't exist
    end

    def rails_version_ok?
      Rails::VERSION::MAJOR == 6 && Rails::VERSION::MINOR == 1
    end
  end

  module InstanceMethods
    # :render_discarded is an additional option you can pass to the
    # date_select helper in your views. Use it to avoid rendering
    # "discarded" fields, i.e. fields marked as discarded or simply
    # not included in date_select's :order array. For example,
    # specifying order: [:day, :month] will cause the helper to
    # "discard" the :year field. Discarding a field renders it as a
    # hidden input. Set :render_discarded to false to avoid rendering
    # it altogether.
    def build_hidden(type, value)
      if @options.fetch(:render_discarded, true)
        super
      else
        ''
      end
    end
  end
end

RenderDiscardedMonkeypatch.apply_patch

Conclusion

I totally get that some of the suggestions I've outlined above might seem like overkill. Our Rails patch contains way more defensive verification code than actual patch code!

Think of all that extra code as a sheath for your broadsword. It's a lot easier to avoid getting cut if it's enveloped in a layer of protection.

What really matters, though, is that I feel confident deploying responsible monkeypatches into production. Irresponsible ones are just time bombs waiting to cost you or your company time, money, and developer health.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby's Hidden Gems: Bullet

Roy Tomeij — 2021-08-11T00:00:00+00:00

A database is the heart of many applications, and having problems with it may result in serious performance issues.

ORMs such as ActiveRecord and Mongoid help us abstract implementation and deliver code faster, but sometimes, we forget to check what queries are running under the hood.

The bullet gem helps us identify some well-known database-related problems:

"N+1 Queries": when the application runs a query to load each item of a list
"Unused Eager Loading": when the application loads data, usually to avoid N+1 queries, but doesn't use it
"Missing Counter Cache": when the application needs to execute count queries to get the number of associated items

In this post, I'm going to show:

how to configure the bullet gem in a Ruby project,
examples of each problem mentioned before,
how bullet detects each,
how to fix each problem, and
how to integrate bullet with AppSignal.

I will use some examples from a project that I created for this post.

How to Configure Bullet in a Ruby Project

First, add the gem to Gemfile.

We can add it to all environments given, we can enable or disable it and use a different approach on each one:

gem 'bullet'

Next, it's necessary to configure it.

If you are in a Rails project, you can run the following command to generate the configuration code automatically:

bundle exec rails g bullet:install

If you are in a non Rails project, you can add it manually, for example, by adding the following code in spec_helper.rb after loading the application's code:

Bullet.enable        = true
Bullet.bullet_logger = true
Bullet.raise         = true

And adding the following code in the main file after loading the application's code:

Bullet.enable = true

I'm going to share more details on configurations in this post. If you want to see them all, go to bullet's README page.

Using bullet In Tests

With the previously suggested configuration, Bullet will detect bad queries executed in tests and raise exceptions for them.

Now, let's see some examples.

Detecting N+1 Queries

Given an index action as follows:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  def index
    @posts = Post.all
  end
end

And a view like this:

# app/views/posts/index.html.erb

<h1>Posts</h1>

<table>
  <thead>
    <tr>
      <th>Name</th>
      <th>Comments</th>
    </tr>
  </thead>

  <tbody>
    <% @posts.each do |post| %>
    <tr>
      <td><%= post.name %></td>
      <td><%= post.comments.map(&:name) %></td>
    </tr>
    <% end %>
  </tbody>
</table>

bullet will raise an error detecting an "N+1" when running an integrated test that executes code from the view and the controller, for example, using a request spec as follows:

# spec/requests/posts_request_spec.rb
require 'rails_helper'

RSpec.describe "Posts", type: :request do
  describe "GET /index" do
    it 'lists all posts' do
      post1 = Post.create!
      post2 = Post.create!

      get '/posts'

      expect(response.status).to eq(200)
    end
  end
end

In this case, it will raise this exception:

Failures:

  1) Posts GET /index lists all posts
     Failure/Error: get '/posts'

     Bullet::Notification::UnoptimizedQueryError:
       user: fabioperrella
       GET /posts
       USE eager loading detected
         Post => [:comments]
         Add to your query: .includes([:comments])
       Call stack
         /Users/fabioperrella/projects/bullet-test/app/views/posts/index.html.erb:17:in `map'
         ...
     # ./spec/requests/posts_controller_spec.rb:9:in `block (3 levels) in <top (required)>'

This happens because the view is executing one query to load each comment name in post.comments.map(&:name):

Processing by PostsController#index as HTML
  Post Load (0.4ms)  SELECT "posts".* FROM "posts"
  ↳ app/views/posts/index.html.erb:14
  Comment Load (0.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 1]]
  ↳ app/views/posts/index.html.erb:17:in `map'
  Comment Load (0.1ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 2]]

To fix it, we can simply follow the instruction in the error message and add .includes([:comments]) to the query:

-@posts = Post.all
+@posts = Post.all.includes([:comments])

This will instruct ActiveRecord to load all the comments with only 1 query.

Processing by PostsController#index as HTML
  Post Load (0.2ms)  SELECT "posts".* FROM "posts"
  ↳ app/views/posts/index.html.erb:14
  Comment Load (0.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN (?, ?)  [["post_id", 1], ["post_id", 2]]
  ↳ app/views/posts/index.html.erb:14

However, bullet will not raise an exception in a controller test like the following, because controller tests don't render views by default, so the N+1 query will not be triggered.

Note: controller tests are discouraged since Rails 5:

# spec/controllers/posts_controller_spec.rb
require 'rails_helper'

RSpec.describe PostsController do
  describe 'GET index' do
    it 'lists all posts' do
      post1 = Post.create!
      post2 = Post.create!

      get :index

      expect(response.status).to eq(200)
    end
  end
end

Another example of a test that Bullet will not detect an "N+1" is a view test because, in this case, it will not run the N+1 queries in the database:

# spec/views/posts/index.html.erb_spec.rb
require 'rails_helper'

describe "posts/index.html.erb" do
  it 'lists all posts' do
    post1 = Post.create!(name: 'post1')
    post2 = Post.create!(name: 'post2')

    assign(:posts, [post1, post2])

    render

    expect(rendered).to include('post1')
    expect(rendered).to include('post2')
  end
end

A Tip to Have More Chances to Detect an N+1 in Tests

I recommend creating at least 1 request spec for each controller action, just to test if it returns the correct HTTP status, then bullet will be watching the queries when rendering these views.

Detecting Unused Eager Loading

Given the following basic_index action:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  def basic_index
    @posts = Post.all.includes(:comments)
  end
end

And the following basic_index view:

# app/views/posts/basic_index.html.erb

<h1>Posts</h1>

<table>
  <thead>
    <tr>
      <th>Name</th>
    </tr>
  </thead>

  <tbody>
    <% @posts.each do |post| %>
    <tr>
      <td><%= post.name %></td>
    </tr>
    <% end %>
  </tbody>
</table>

When we run the following test:

# spec/requests/posts_request_spec.rb
require 'rails_helper'

RSpec.describe "Posts", type: :request do
  describe "GET /basic_index" do
    it 'lists all posts' do
      post1 = Post.create!
      post2 = Post.create!

      get '/posts/basic_index'

      expect(response.status).to eq(200)
    end
  end
end

Bullet will raise the following error:

  1) Posts GET /basic_index lists all posts
     Failure/Error: get '/posts/basic_index'

     Bullet::Notification::UnoptimizedQueryError:
       user: fabioperrella
       GET /posts/basic_index
       AVOID eager loading detected
         Post => [:comments]
         Remove from your query: .includes([:comments])
       Call stack
         /Users/fabioperrella/projects/bullet-test/spec/requests/posts_request_spec.rb:20:in `block (3 levels) in <top (required)>'

This happens because it's not necessary to load the list of comments for this view.

To fix the problem, we can simply follow the instruction in the error above and remove the query .includes([:comments]):

-@posts = Post.all.includes(:comments)
+@posts = Post.all

It's worth saying that it will not raise the same error if we run only a controller test, without render_views, as shown before.

Detecting Missing Counter Cache

Given a controller like this:

# app/controllers/posts_controller.rb
class PostsController < ApplicationController
  def index_with_counter
    @posts = Post.all
  end
end

And a view like this:

# app/views/posts/index_with_counter.html.erb

<h1>Posts</h1>

<table>
  <thead>
    <tr>
      <th>Name</th>
      <th>Number of comments</th>
    </tr>
  </thead>

  <tbody>
    <% @posts.each do |post| %>
    <tr>
      <td><%= post.name %></td>
      <td><%= post.comments.size %></td>
    </tr>
    <% end %>
  </tbody>
</table>

If we run the following request spec:

describe "GET /index_with_counter" do
  it 'lists all posts' do
    post1 = Post.create!
    post2 = Post.create!

    get '/posts/index_with_counter'

    expect(response.status).to eq(200)
  end
end

bullet will raise the following error:

1) Posts GET /index_with_counter lists all posts
  Failure/Error: get '/posts/index_with_counter'

  Bullet::Notification::UnoptimizedQueryError:
    user: fabioperrella
    GET /posts/index_with_counter
    Need Counter Cache
      Post => [:comments]
  # ./spec/requests/posts_request_spec.rb:31:in `block (3 levels) in <top (required)>'

This happens because this view is executing 1 query to count the number of comments in post.comments.size for each post.

Processing by PostsController#index_with_counter as HTML
  ↳ app/views/posts/index_with_counter.html.erb:14
  Post Load (0.4ms)  SELECT "posts".* FROM "posts"
  ↳ app/views/posts/index_with_counter.html.erb:14
   (0.4ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 1]]
  ↳ app/views/posts/index_with_counter.html.erb:17
   (0.1ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 2]]

To fix this, we can create a counter cache, which can be a bit complex, especially if there is data in the production database.

A counter cache is a column that we can add to a table, that ActiveRecord will update automatically when we insert and delete associated models. There are more details in this post. I suggest reading it to know how to create and sync the counter cache.

Using Bullet in Development

Sometimes, tests might not detect the problems previously mentioned, for example, if test coverage is low, so it's possible to enable bullet in other environments using different approaches.

In the development environment, we can enable the following configurations:

Bullet.alert         = true

Then, it will show alerts like this in the browser:

Bullet.add_footer    = true

It will add a footer on the page with the error:

It's also possible to enable errors to be logged in the browser's console:

Bullet.console    = true

It will add an error like this:

Using Bullet in Staging with Appsignal

In the staging environment, we don't want these error messages to be shown to end-users, but it would be great to know if the application starts to have one of the problems mentioned previously.

At the same time, bullet may degrade performance and increase memory consumption in the application, so it's better to enable it only temporarily in staging, but don't enable it in production.

Assuming the staging environment is using the same configuration file as the production environment, which is a good practice to reduce the difference between them, we can use an environment variable to enable or disable bullet as follows:

# config/environments/production.rb
config.after_initialize do
  Bullet.enabled   = ENV.fetch('BULLET_ENABLED', false)
  Bullet.appsignal = true
end

To receive notifications about issues Bullet has found in your staging environment, you can use AppSignal to report those notifications as errors. You'll need to have the appsignal gem installed and configured in your project. You can see more details in the Ruby gem docs.

Then, if a problem is detected by bullet, it will create an error incident like this:

This error is raised by the uniform_notifier gem which was extracted from bullet.

Unfortunately, the error message doesn't show enough information, but I sent in a Pull Request to improve this!

Conclusion

The bullet gem is a great tool that can help us detect problems that will degrade performance in applications.

Try to keep good test coverage, as previously mentioned, to have greater chances of detecting these problems before going to production.

As an extra tip, if you want to be even more protected against performance problems related to the database, take a look at the wt-activerecord-index-spy gem, which helps to detect queries that are not using proper indexes.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

An Introduction to Pattern Matching in Ruby

Roy Tomeij — 2021-07-28T00:00:00+00:00

Let's start with a brief discussion about pattern matching in Ruby, what it does, and how it can help improve code readability.

If you are anything like me a few years ago, you might confuse it with pattern matching in Regex. Even a quick Google search of 'pattern matching' with no other context brings you content that's pretty close to that definition.

Formally, pattern matching is the process of checking any data (be it a sequence of characters, a series of tokens, a tuple, or anything else) against other data.

In terms of programming, depending on the capabilities of the language, this could mean any of the following:

Matching against an expected data type
Matching against an expected hash structure (e.g. presence of specific keys)
Matching against an expected array length
Assigning the matches (or a part of them) to some variables

My first foray into pattern matching was through Elixir. Elixir has first class support for pattern matching, so much so that the = operator is, in fact, the match operator, rather than simple assignment.

This means that in Elixir, the following is actually valid code:

iex> x = 1
iex> 1 = x

With that in mind, let's look at the new pattern matching support for Ruby 2.7+ and how we can use it to make our code more readable, starting from today.

Ruby Pattern Matching with `case`/`in`

Ruby supports pattern matching with a special case/in expression. The syntax is:

case <expression>
in <pattern1>
  # ...
in <pattern2>
  # ...
else
  # ...
end

This is not to be confused with the case/when expression. when and in branches cannot be mixed in a single case.

If you do not provide an else expression, any failing match will raise a NoMatchingPatternError.

Pattern Matching Arrays in Ruby

Pattern matching can be used to match arrays to pre-required structures against data types, lengths or values.

For example, all of the following are matches (note that only the first in will be evaluated, as case stops looking after the first match):

case [1, 2, "Three"]
in [Integer, Integer, String]
  "matches"
in [1, 2, "Three"]
  "matches"
in [Integer, *]
  "matches" # because * is a spread operator that matches anything
in [a, *]
  "matches" # and the value of the variable a is now 1
end

This type of pattern matching clause is very useful when you want to produce multiple signals from a method call.

In the Elixir world, this is frequently used when performing operations that could have both an :ok result and an :error result, for example, inserted into a database.

Here is how we can use it for better readability:

def create
  case save(model_params)
  in [:ok, model]
    render :json => model
  in [:error, errors]
    render :json => errors
  end
end

# Somewhere in your code, e.g. inside a global helper or your model base class (with a different name).
def save(attrs)
  model = Model.new(attrs)
  model.save ? [:ok, model] : [:error, model.errors]
end

Pattern Matching Objects in Ruby

You can also match objects in Ruby to enforce a specific structure:

case {a: 1, b: 2}
in {a: Integer}
  "matches" # By default, all object matches are partial
in {a: Integer, **}
  "matches" # and is same as {a: Integer}
in {a: a}
  "matches" # and the value of variable a is now 1
in {a: Integer => a}
  "matches" # and the value of variable a is now 1
in {a: 1, b: b}
  "matches" # and the value of variable b is now 2
in {a: Integer, **nil}
  "does not match" # This will match only if the object has a and no other keys
end

This works great when imposing strong rules for matching against any params.

For example, if you are writing a fancy greeter, it could have the following (strongly opinionated) structure:

def greet(hash = {})
  case hash
  in {greeting: greeting, first_name: first_name, last_name: last_name}
    greet(greeting: greeting, name: "#{first_name} #{last_name}")
  in {greeting: greeting, name: name}
    puts "#{greeting}, #{name}"
  in {name: name}
    greet(greeting: "Hello", name: name)
  in {greeting: greeting}
    greet(greeting: greeting, name: "Anonymous")
  else
    greet(greeting: "Hello", name: "Anonymous")
  end
end

greet # Hello, Anonymous
greet(name: "John") # Hello, John
greet(first_name: "John", last_name: "Doe") # Hello, John Doe
greet(greeting: "Bonjour", first_name: "John", last_name: "Doe") # Bonjour, John Doe
greet(greeting: "Bonjour") # Bonjour, Anonymous

Variable Binding and Pinning in Ruby

As we have seen in some of the above examples, pattern matching is really useful in assigning part of the patterns to arbitrary variables. This is called variable binding, and there are several ways we can bind to a variable:

With a strong type match, e.g. in [Integer => a] or in {a: Integer => a}
Without the type specification, e.g. in [a, 1, 2] or in {a: a}.
Without the variable name, which defaults to using the key name, e.g. in {a:} will define a variable named a with the value at key a.
Bind rest, e.g. in [Integer, *rest] or in {a: Integer, **rest}.

How, then, can we match when we want to use an existing variable as a sub-pattern? This is when we can use variable pinning with the ^ (pin) operator:

a = 1
case {a: 1, b: 2}
in {a: ^a}
  "matches"
end

You can even use this when a variable is defined in a pattern itself, allowing you to write powerful patterns like this:

case order
in {billing_address: {city:}, shipping_address: {city: ^city}}
  puts "both billing and shipping are to the same city"
else
  raise "both billing and shipping must be to the same city"
end

One important quirk to mention with variable binding is that even if the pattern doesn't fully match, the variable will still have been bound. This can sometimes be useful.

But, in most cases, this could also be a cause of subtle bugs — so make sure that you don't rely on shadowed variable values that have been used inside a match. For example, in the following, you would expect the city to be "Amsterdam", but it would instead be "Berlin":

city = "Amsterdam"
order = {billing_address: {city: "Berlin"}, shipping_address: {city: "Zurich"}}
case order
in {billing_address: {city:}, shipping_address: {city: ^city}}
  puts "both billing and shipping are to the same city"
else
  puts "both billing and shipping must be to the same city"
end
puts city # Berlin instead of Amsterdam

Matching Ruby's Custom Classes

You can implement some special methods to make custom classes pattern matching aware in Ruby.

For example, to pattern match a user against his first_name and last_name, we can define deconstruct_keys on the class:

class User
  def deconstruct_keys(keys)
    {first_name: first_name, last_name: last_name}
  end
end

case user
in {first_name: "John"}
  puts "Hey, John"
end

The keys argument to deconstruct_keys contains the keys that have been requested in the pattern. This is a way for the receiver to provide only the required keys if computing all of them is expensive.

In the same way as deconstruct_keys, we could provide an implementation of deconstruct to allow objects to be pattern matched as an array. For example, let's say we have a Location class that has latitude and longitude. In addition to using deconstruct_keys to provide latitude and longitude keys, we could expose an array in the form of [latitude, longitude] as well:

class Location
  def deconstruct
    [latitude, longitude]
  end
end

case location
in [Float => latitude, Float => longitude]
  puts "#{latitude}, #{longitude}"
end

Using Guards for Complex Patterns

If we have complex patterns that cannot be represented with regular pattern match operators, we can also use an if (or unless) statement to provide a guard for the match:

case [1, 2]
in [a, b] if b == a * 2
  "matches"
else
  "no match"
end

Pattern Matching with `=>`/`in` Without `case`

If you are on Ruby 3+, you have access to even more pattern matching magic. Starting from Ruby 3, pattern matching can be done in a single line without a case statement:

[1, 2, "Three"] => [Integer => one, two, String => three]
puts one # 1
puts two # 2
puts three # Three

# Same as above
[1, 2, "Three"] in [Integer => one, two, String => three]

Given that the above syntax does not have an else clause, it is most useful when the data structure is known beforehand.

As an example, this pattern could fit well inside a base controller that allows only admin users:

class AdminController < AuthenticatedController
  before_action :verify_admin

  private

  def verify_admin
    Current.user => {role: :admin}
  rescue NoMatchingPatternError
    raise NotAllowedError
  end
end

Pattern Matching in Ruby: Watch This Space

At first, pattern matching can feel a bit strange to grasp. To some, it might feel like glorified object/array deconstruction.

But if the popularity of Elixir is any indication, pattern matching is a great tool to have in your arsenal. Having first-hand experience using it on Elixir, I can confirm that it is hard to live without once you get used to it.

If you are on Ruby 2.7, pattern matching (with case/in) is still experimental. With Ruby 3, case/in has moved to stable while the newly introduced single-line pattern matching expressions are experimental. Warnings can be turned off with Warning[:experimental] = false in code or -W:no-experimental command-line key.

Even though pattern matching in Ruby is still in its early stages, I hope you've found this introduction useful and that you're as excited as I am about future developments to come!

P.S. If you’d like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

General Ruby on Rails Problems and Takeaways

Roy Tomeij — 2021-07-07T00:00:00+00:00

Welcome to the last part of my Ruby on Rails Patterns and Anti-Patterns series. It's been quite a ride writing and researching all of these topics. In this blog post, we'll go over the most common problems I've encountered when building and shipping Ruby on Rails applications through the years.

The ideas I'll go through here apply to almost anywhere in the code. So consider them as general ideas, not something related to the Model-View-Controller pattern. If you are interested in patterns and anti-patterns related to the Rails MVC, you can check out the Model, View, and Controller blog posts.

So let's jump into general problems and takeaways.

Selfish Objects and the Law of Demeter

The Law of Demeter is a heuristic that got its name when a group of people worked on the Demeter Project. The idea is that your objects are fine as long as they call one method at a time and don't chain multiple method calls. What this means in practice is the following:

# Bad
song.label.address

# Good
song.label_address

So now, the song object no longer needs to know where the address comes from — the address is the responsibility of the label object. You are encouraged to chain only one method call and make your objects 'selfish' so that they don't share their full information directly but through helper methods.

Luckily, in Rails, you don't have to write a helper method per se — you can use the delegate helper:

def Label < ApplicationModel
  belongs_to :song

  delegate :address, to: :song
end

You can go ahead and play around with the options that delegate accepts in delegete's docs. But the idea and execution are pretty simple. By applying the Law of Demeter, you reduce structural coupling. Together with the powerful delegate, you do it in fewer lines and with great options included.

Another idea that's very similar to the Law of Demeter is the Single-responsibility Principle (or SRP for short). It states that a module, class, or function should be responsible for a single part of a system. Or, presented in another way:

Gather together the things that change for the same reasons. Separate those things that change for different reasons.

Folks can often have a different understanding of SRP, but the idea is to keep your building blocks responsible for a single thing. It might be challenging to achieve SRP as your Rails app expands, but be aware of it when refactoring.

When adding features and increasing the LOC, I've found that folks often reach out for a quick solution. So let's go through grabbing the quick fix.

I Know a Guy (Do You Need That Ruby Gem?)

Back in the day when Rails was a hot topic, there was a boom in open-source collaboration, with new Ruby gems popping up on every corner (like it is nowadays with all the emerging JavaScript libraries, but on a much smaller scale):

👆 Information from Module Counts.

Anyway, a common approach was to find an existing gem to solve your problem.

There's nothing wrong with that, but I'd like to share some bits of advice before you decide to install a gem.

First, ask yourself these questions:

What portion of the gem's features are you going to use?
Is there a similar gem out there that is 'simpler' or more up-to-date?
Can you implement the feature you need easily and with confidence?

Evaluate whether it's worth doing the implementation if you don't plan to use the whole array of gem features. Or, if the gem's implementation is too complex and you believe you can do it more simply, opt for a custom solution.

Another factor that I consider is how active the gem's repository is — are there any active maintainers? When was the last time a release happened?

You should also watch out for the gem's dependencies. You don't want to get locked into a specific version of a dependency, so always check the Gemfile.spec file. Consult the RubyGems way of specifying gem versions.

While we're on the topic of gems, there is a related idea that I've encountered: the 'Not Invented Here' (or NIH) phenomenon that applies to the Rails/Ruby world. Let's see what it's about in the next section.

Not Invented Here (Maybe You Need That Ruby Gem after All?)

In a couple of occurrences in my career, I had a chance to experience people (me included) fall for 'Not Invented Here' syndrome. The idea is similar to 'reinventing the wheel'. Sometimes, teams and organizations do not trust libraries (gems) that they can't control. Lack of trust might be a trigger for them to reinvent a gem that is already out there.

Sometimes, experiencing NIH can be a good thing. Making an in-house solution can be great, especially if you improve it over the other solutions out there. If you decide to open-source the solution, that can be even better (take a look at Ruby on Rails or React). But if you want to reinvent the wheel for the sake of it, don't do it. The wheel itself is pretty great already.

This topic is quite tricky, and if you ever get caught in such a situation, ask yourself these questions:

Are we confident that we can make a better solution than existing ones?
If the existing open-source solution differs from what we need, can we make an open-source contribution and improve it?
Furthermore, can we become the maintainers of the open-source solution and possibly improve lots of developers' lives?

But sometimes, you just have to go your own way and create a library yourself. Maybe your organization doesn't like licensing an open-source library, so you are forced to build your own. But whatever you do, I'd say avoid reinventing the wheel.

Lifeguard on Duty (Over-rescuing Exceptions)

People tend to rescue more exceptions than they originally aimed for.

This topic is a bit more related to the code than the previous ones. It might be common sense to some, but it can be seen in the code from time to time. For example:

begin
  song.upload_lyrics
rescue
  puts 'Lyrics upload failed'
end

If we don't specify the exception we want to rescue, we will catch some exceptions that we didn't plan to.

In this case, the problem might be that the song object is nil. When that exception gets reported to the error tracker, you might think that something is off with the upload process, whereas actually, you might be experiencing something totally different.

So, to be safe, when rescuing exceptions, make sure you get a list of all the exceptions that might occur. If you can't obtain every exception for some reason, it's better to under-rescue than to over-rescue. Rescue the exceptions that you know and handle the others at a later stage.

You Ask Too Much (Too Many SQL Queries)

In this section, we are going to go through another web development, relation-database problem.

You bomb the webserver with too many SQL queries in one request. How does that problem arise? Well, it can happen if you try to fetch multiple records from multiple tables in one request. But what most often happens is the infamous N+1 query problem.

Imagine the following models:

class Song < ApplicationRecord
  belongs_to :artist
end

class Artist < ApplicationRecord
  has_many :songs
end

If we want to show a couple of songs in a genre and their artists:

songs = Song.where(genre: genre).limit(10)

songs.each do |song|
  puts "#{song.title} by #{song.artist.name}"
end

This piece of code will trigger one SQL query to get ten songs. After that, one extra SQL query will be performed to fetch the artist for each song. That's eleven (11) queries total.

Imagine the scenario if we load more songs — we'll put the database under a heavier load trying to get all the artists.

Alternatively, use includes from Rails:

songs = Song.includes(:artists).where(genre: genre).limit(10)

songs.each do |song|
  puts "#{song.title} by #{song.artist.name}"
end

After the includes, we now only get two SQL queries, no matter how many songs we decide to show. How neat.

One way you can diagnose too many SQL queries is in development. If you see a group of similar SQL queries fetching data from the same table, then something fishy is going on there. That's why I strongly encourage you to turn on SQL logging for your development environment. Also, Rails supports verbose query logs that show where a query is called from in the code.

If looking at logs is not your thing, or you want something more serious, try out AppSignal's performance measuring and N+1 query detection. There, you will get an excellent indicator of whether your issue comes from an N+1 query. Here's how it looks below:

Sum Up

Thanks for reading this blog post series. I'm glad you joined me for this interesting ride, where we went from introducing patterns and anti-patterns in Rails to exploring what they are inside the Rails MVC pattern, before this final blog post on general problems.

I hope you learned a lot, or at least revised and established what you already know. Do not stress about memorizing all of it. You can always consult the series if you are having trouble in any area.

You will surely encounter both patterns and anti-patterns because this world (and software engineering especially) is not ideal. That shouldn't worry you either.

Mastering patterns and anti-patterns will make you a great software engineer. But what makes you even better is knowing when to break those patterns and molds, because there is no perfect solution.

Thanks again for joining and reading. See you in the next one — and cheers!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Performance, Stress, and Load Tests in Rails

Roy Tomeij — 2021-06-09T00:00:00+00:00

Tests are an integral part of most well-working Rails applications where maintenance isn't a nightmare and new features are consistently added, or existing ones are improved. Unfortunately, for many applications, a production environment is where they are put under heavy workload or significant traffic for the first time. This is understandable as such tests are costly.

Thankfully, Rails has good support not only for unit, end-to-end, and integration tests but also for tests related to performance and loading. I’ll cover all of them in the article and show some practical examples that will help you understand how to efficiently use tools that test the performance level of your application.

The article is divided into two sections:

Theoretical — I'll show you why testing is necessary, the kinds of tests we can perform and the metrics that are essential when performing tests on an application
Practical — we'll get our hands dirty and write tests for an actual application to get the output

After reading the two sections, you'll have a deeper understanding of the different types of tests and how to perform them on your Rails application. Sounds interesting? Then let's get started with a pinch of theory about tests.

Testing in Theory

Testing should always be an inherent part of the development of any type of application. If you are still not convinced about that or haven’t written any tests yet, here are some arguments for testing that will help you:

Introduce changes without worrying about breaking something — this is the major reason why tests are necessary. Imagine working on a huge app where you have to click through the whole app to make sure nothing is broken each time you introduce some change, even a small one. With tests, you just execute one command and the verification process is automatic and fast.
Easy refactoring process — I mentioned above that tests are essential when adding new features or making changes. With testing in place, you are also more comfortable with improving your existing code.
Tests are a form of documentation — well-written tests can be a form of documentation for various sets of features in the application. They not only describe what the feature is but also how it should be working.
Opportunity to rethink the implementation — when you write a test, you have a chance to think again if the way you want to implement the code is correct and reasonable. Also, you simply check if your code is working the way you expect it.

I hope the above arguments convinced you to use tests during the development of any app. While knowing why to test the code is essential, it’s also crucial to learn about different types of tests.

Different Types of Tests

There are three primary types of tests that you can write to ensure that your Rails application’s performance is correct and the infrastructure is working well under the heavy workload. Those types are the following:

Load testing — this type of test answers the following question: how many simultaneous users can the system handle for a given period. Imagine that you launch a top-rated product on your website and thousands of users want to make the order at the same time. Without proper loading tests, you risk a crash during the most critical time.
Stress testing — with this type of test, you don’t focus on verifying the number of users the system can handle simultaneously, but on how the system will behave when the limit of the users is hit.
Performance testing — I would say that this type of test is a parent of stress and load testing. The primary purpose of such tests is to get a specific set of metrics on which base we can take some action to improve the application’s code. I will talk about those metrics in a while.

That being said, we are now prepared to move to the last step of the theory part: learning what metrics are essential when doing performance testing on a Rails application. Without that knowledge, we won’t correctly interpret the test output and decide if we should change the code or not.

Important Metrics

The type of metrics you can receive can be different depending on the tool you use for testing, but generally, we can group them into a set of metrics that are pretty common:

Response time — the time between the request being made and the response getting rendered in the browser. This metric shows us how long the user needs to wait before receiving the information he requested. It’s sometimes called process time.
Memory usage — the amount of memory consumed for the given request. This is a piece of essential information as it points you to the place where you can improve the code so the system can respond faster and use fewer resources.
Objects allocation — a high memory allocation causes high memory usage and long response times. This metric can lead you to the exact place in code where many objects are allocated, so you can immediately inspect that.

You can have more metrics when testing, but those three are the most important and will be valid for any application that you test. We can now get our hands dirty and write real tests.

Practice

We aren't able to write tests without having something to test. That’s why the first step in the practice part is to write a simple Rails application that we can write the tests for.

Sample Rails Application

I will use Ruby 3.0.1 and Rails 6.1.3.1 but feel free to use any version you are comfortable with. If you have Ruby and Rails installed, the next step is to create the application’s skeleton:

rails new simpleapp -d=postgresql

For the article's purpose, I'll create an app where a list of users is presented along with their pet’s names. Such a structure will allow us to easily create the N+1 queries that will offer more fun when doing performance tests and checking the impact on speed and other metrics that changes will have.

Before we generate the models, let’s create the database first:

cd simpleapp/
bin/rails db:create

Now, we can generate the models:

rails g model User name:string
rails g model Animal name:string user:references
bin/rails db:migrate

Just one small update to the User model to reflect the relationship with the Animal model:

class User < ApplicationRecord
  has_many :animals
end

We can now add some seeds in db/seeds.rb file:

people = {
  'Tim' => ['Pinky', 'Rick'],
  'Martha' => ['Rudolph'],
  'Mark' => ['Niki', 'Miki', 'Bella'],
  'Tina' => ['Tom', 'Luna']
}

people.each_pair do |name, pets|
  user = User.create(name: name)
  pets.each do |pet_name|
    user.animals.create(name: pet_name)
  end
end

and load the data into the database:

bin/rails db:seed

I'll create one controller with the users’ assignment, and then in view, I'll list all users with their pets’ names. I’m intentionally using code that is causing performance problems so you can measure the improvements later.

touch app/controllers/home_controller.rb
mkdir app/views/home
touch app/views/home/index.html.erb

The controller is simple:

class HomeController < ApplicationController
  def index
    @users = User.all
  end
end

and the view also:

<h1>List</h1>

<ul>
  <% @users.each do |user| %>
    <li><%= user.name %> (<%= user.animals.count %>)
      <ul>
        <% user.animals.each do |animal| %>
          <li><%= animal.name %></li>
        <% end %>
      </ul>
    </li>
  <% end %>
</ul>

The last step is to update the config/routes.rb file to let Rails know what we would like to see when visiting the main URL:

Rails.application.routes.draw do
  root to: 'home#index'
end

Load Tests With JMeter

JMeter is an open-source software created by the Apache software foundation, designed to load test functional behavior. Since it’s a program created with Java, you can install it on any operating system. You can download the files here: https://jmeter.apache.org/download_jmeter.cgi

If you are working on a macOS system, you can easily install JMeter with Homebrew:

brew install jmeter

After installation, you can run the program with the following command:

jmeter

Configuring the test

The configuration process consists of the following steps:

Adding the thread group — specifying the number of users and how long each will visit your website
Configuring HTTP request — specifying the endpoint that JMeter should hit
Setting the metrics we are interested in

Let’s walk step-by-step through a simple test configuration to simulate a single user request to the main page of the simple app we created before.

Add thread group

Select the Add -> Threads (Users) -> Thread Group from the menu that expands after you right-click on the “Test Plan”:

Specify the number of users and additional attributes:

Configure HTTP request

Right-click on the thread we created in the previous step and select Add -> Sampler -> HTTP Request:

Configure the protocol, server name, port, and the path of the request:

Specify the result view

Right-click on the HTTP request and select Add -> Listener -> View Results Tree:

Running the Test

The test is now configured, and we can trigger it. To do this, simply click on the green play button:

As you can see, the application passed the test, but it was just a single request, so the result was obvious. You can now experiment with the number of users and other configuration options to see how the application will behave. From my tests, the simple app started to crash when around 200 users started accessing it simultaneously.

Next Steps

After performing the load test, you'll know the pain points of your application. Understanding the user limit, you can now perform the stress test to see how the application will behave.

Performance Tests With Ruby-prof

The performance test feature was built-in in Rails until version 3, and then it was extracted to the separate gem https://github.com/rails/rails-perftest. Since I had some problems using it with the latest version of Rails, I decided not to include it in this article. Instead, I will use the ruby-prof library that works very well.

As usual, the first step is to add a gem to our application:

bundle add ruby-prof

The second and the last step of the configuration process is to update the config/application.rb and use the middleware for the gem so the library can automatically inspect our requests and produce reports based on them:

module Simpleapp
  class Application < Rails::Application
    config.middleware.use Rack::RubyProf, :path => './tmp/profile'
  end
end

You can now access the app, and each time you perform a request, the gem will generate a new report. It looks like this:

You can find it under the configured path, which is tmp/profile in our case. The second report is also generated, and it shows the call stack, which is also a pretty helpful metric when debugging performance issues in a Rails application.

It’s important to remember that setting the cache_classes and cache_template_loading settings to true will slow down the application and overwhelm the application metrics as Rails will try to load the required files.

Summary

Testing is an essential part of every development process. Checking if the code behaves as we want it to is as crucial as verifying if our solutions have good performance. Skipping tests leads to serious problems that impact the app’s performance and your users’ trust. Hopefully, testing is not that hard.

In this article, we covered the following important aspects of testing:

the reason you should test your code
the different types of performance tests
the way you can test the performance of your Rails app

I hope that you are now more convinced why it is important to write tests since you know why and how.

If you're interested in monitoring your app’s performance not just locally but also in the production or staging environments, you should also check out AppSignal.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Three Ways To Avoid Duplicate Sidekiq Jobs

Roy Tomeij — 2021-05-12T00:00:00+00:00

This post was updated on 4 August 2023 to reference the latest pricing for Sidekiq Enterprise. Some links were also updated.

Chances are, if you are writing Ruby code, you are using Sidekiq to handle background processing. If you are coming from ActiveJob or some other background, stay tuned, some of the tips covered can be applied there as well.

Folks utilize (Sidekiq) background jobs for different cases. Some crunch numbers, some dispatch welcome emails to users, and some schedule data syncing. Whatever your case may be, you might eventually run into a requirement to avoid duplicate jobs. By duplicate jobs, I envision two jobs that do the exact same thing. Let's dive in on that a bit.

Why De-Duplicate Jobs?

Imagine a scenario where your job looks like the following:

class BookSalesWorker
  include Sidekiq::Worker

  def perform(book_id)
    crunch_some_numbers(book_id)

    upload_to_s3
  end

  ...
end

The BookSalesWorker always does the same thing — queries the DB for a book based on the book_id and fetches the latest sales data to calculate some numbers. Then, it uploads them to a storage service. Keep in mind that every time a book is sold on your website, you will have this job enqueued.

Now, what if you got 100 sales at once? You'd have 100 of these jobs doing the exact same thing. Maybe you are fine with that. You don't care about S3 writes that much, and your queues aren't as congested, so you can handle the load. But, "does it scale?"™️

Well, definitely not. If you start receiving more sales for more books, your queue will quickly pile up with unnecessary work. If you have 100 jobs that do the same thing for a single book, and you have 10 books selling in parallel, you are now 1000 jobs deep in your queue, where in reality, you could just have 10 jobs for each book.

Now, let's go through a couple of options on how you can prevent duplicate jobs from piling up your queues.

1. DIY Way

If you are not a fan of external dependencies and complex logic, you can go ahead and add some custom solutions to your codebase. I created a sample repo to try out our examples first-hand. There will be a link in each approach to the example.

1.1 One Flag Approach

You can add one flag that decides whether to enqueue a job or not. One might add a sales_enqueued_at in their Book table and maintain that one. For example:

module BookSalesService
  def schedule_with_one_flag(book)
    # Check if the job was enqueued more than 10 minutes ago
    if book.sales_enqueued_at < 10.minutes.ago
      book.update(sales_enqueued_at: Time.current)

      BookSalesWorker.perform_async(book.id)
    end
  end
end

That means that no new jobs will be enqueued until 10 minutes have passed from the time the last job got enqueued. After 10 minutes have passed, we then update the sales_enqueued_at and enqueue a new job.

Another thing you can do is set one flag that is a boolean, e.g. crunching_sales. You set crunching_sales to true before the first job is enqueued. Then, you set it to false once the job is complete. All other jobs that try to get scheduled will be rejected until crunching_sales is false.

You can try this approach in the example repo I created.

1.2 Two Flags Approach

If "locking" a job from being enqueued for 10 minutes sounds too scary, but you are still fine with extra flags in your code, then the next suggestion might interest you.

You can add another flag to the existing sales_enqueued_at — the sales_calculated_at. Then our code will look something like this:

module BookSalesService
  def schedule_with_two_flags(book)
    # Check if sales are being calculated right now
    if book.sales_enqueued_at <= book.sales_calculated_at
      book.update(sales_enqueued_at: Time.current)

      BookSalesWorker.perform_async(book.id)
    end
  end
end

class BookSalesWorker
  include Sidekiq::Worker

  def perform(book_id)
    crunch_some_numbers(book_id)

    upload_to_s3

    # New adition
    book.update(sales_calculated_at: Time.current)
  end

  ...
end

To try it out, check out the instructions in the example repo.

Now we control a portion of the time between when a job is enqueued and finished. In that portion of time, no job can be enqueued. While the job is running, the sales_enqueued_at will be larger than sales_calculated_at. When the job finishes running, the sales_calculated_at will be larger (more recent) than the sales_enqueued_at and a new job will get enqueued.

Using two flags might be interesting, so you could show the last time those sales numbers got updated in the UI. Then the users that read them can have an idea of how recent the data is. A win-win situation.

Flag Sum Up

It might be tempting to create solutions like these in times of need, but to me, they look a bit clumsy, and they add some overhead. I would recommend using this if your use case is simple, but as soon as it proves complex or not enough, I'd urge you to try out other options.

A huge con with the flag approach is that you will lose all the jobs that tried to enqueue during those 10 minutes. A huge pro is that you are not bringing in dependencies, and it will alleviate the job number in queues pretty quickly.

1.3 Traversing The Queue

Another approach you can take is to create a custom locking mechanism to prevent the same jobs from enqueueing. We will check the Sidekiq queue we are interested in and see if the job (worker) is already there. The code will look something like this:

module BookSalesService
  def schedule_unique_across_queue(book)
    queue = Sidekiq::Queue.new('default')

    queue.each do |job|
      return if job.klass == BookSalesWorker.to_s &&
        job.args == [book.id]
    end

    BookSalesWorker.perform_async(book.id)
  end
end

class BookSalesWorker
  include Sidekiq::Worker

  def perform(book_id)
    crunch_some_numbers(book_id)

    upload_to_s3
  end

  ...
end

In the example above we are checking whether the 'default' queue has a job with the class name of BookSalesWorker. We are also checking if the job arguments match the book ID. If the BookSalesWorker job with the same Book ID is in the queue, we will return early and not schedule another one.

Note that some of them might get scheduled if you schedule jobs too fast because the queue is empty. The exact thing happened to me when testing it locally with:

100.times { BookSalesService.schedule_unique_across_queue(book) }

You can try it out in the example repo.

The good thing about this approach is that you can traverse all queues to search for an existing job if you need it. The con is that you can still have duplicate jobs if your queue is empty and you schedule a lot of them at once. Also, you are potentially traversing through all the jobs in the queue before scheduling one, so that might be costly depending on the size of your queue.

2. Upgrading to Sidekiq Enterprise

If you or your organization has some money lying around, you can upgrade to the Enterprise version of Sidekiq. It starts at $229 per month, and it has a cool feature that helps you avoid duplicate jobs. Unfortunately, I don't have Sidekiq Enterprise, but I believe their documentation is sufficient. You can easily have unique (non-duplicated) jobs with the following code:

class BookSalesWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.minutes

  def perform(book_id)
    crunch_some_numbers(book_id)

    upload_to_s3
  end

  ...
end

And that's it. You have a similar job implementation to what we described in the 'One Flag Approach' section. The job will be unique for 10 minutes, meaning that no other job with the same arguments can be scheduled in that time period.

Pretty cool one-liner, huh? Well, if you have Enterprise Sidekiq and you just found out about this feature, I am truly glad I helped. Most of us are not going to use it, so let's jump into the next solution.

3. sidekiq-unique-jobs To The Rescue

Yes, I know we are about to mention a gem. And yes, it has some Lua files in it which might put some people off. But bear with me, it's a really sweet deal you are getting with it. The sidekiq-unique-job gem comes with a lot of locking and other configuration options — probably more than you need.

To get started quickly, put sidekiq-unique-jobs gem into your Gemfile, do bundle and configure your worker as shown:

class UniqueBookSalesWorker
  include Sidekiq::Worker

  sidekiq_options lock: :until_executed,
                  on_conflict: :reject

  def perform(book_id)
    book = Book.find(book_id)

    logger.info "I am a Sidekiq Book Sales worker - I started"
    sleep 2
    logger.info "I am a Sidekiq Book Sales worker - I finished"

    book.update(sales_calculated_at: Time.current)
    book.update(crunching_sales: false)
  end
end

There are a lot of options, but I decided to simplify and use this one:

sidekiq_options lock: :until_executed, on_conflict: :reject

The lock: :until_executed will lock the first UniqueBookSalesWorker job until it is executed. With on_conflict: :reject, we are saying that we want all other jobs that try to get executed to be rejected into the dead queue. What we achieved here is similar to what we did in our DIY examples in the topics above.

A slight improvement over those DIY examples is that we have a kind of log of what happened. To get a sense of how it looks, let's try the following:

5.times { UniqueBookSalesWorker.perform_async(Book.last.id) }

Only one job will fully execute, and the other four jobs will be sent off to the dead queue, where you can retry them. This approach differs from our examples where duplicate jobs were just ignored.

There are a lot of options to choose from when it comes to locking and conflict resolving. I suggest you consult the gem's documentation for your specific use case.

Great Insights

The great thing about this gem is that you can view the locks and the history of what went down in your queues. All you need to do is add the following lines in your config/routes.rb:

# config/routes.rb
require 'sidekiq_unique_jobs/web'

Rails.application.routes.draw do
  mount Sidekiq::Web, at: '/sidekiq'
end

It will include the original Sidekiq client, but it will also give you two more pages — one for job locks and the other for the changelog. This is how it looks:

Notice how we have two new pages, "Locks" and "Changelogs". Pretty cool feature.

You can try all of this in the example project where the gem is installed and ready to go.

Why Lua?

First of all, I am not the author of the gem, so I'm just assuming things here. The first time I saw the gem, I wondered: why use Lua inside a Ruby gem? It might appear odd at first, but Redis supports running Lua scripts. I guess the author of the gem had this in mind and wanted to do more nimble logic in Lua.

If you look at the Lua files in the gem's repo, they aren't that complicated. All of the Lua scripts are called later from the Ruby code in the SidekiqUniqueJobs::Script::Caller here. Please take a look at the source code, it is interesting to read and figure out how things work.

Alternative Gem

If you use ActiveJob extensively, you can try the active-job-uniqueness gem right here. The idea is similar, but instead of custom Lua scripts, it uses Redlock to lock items in Redis.

To have a unique job using this gem, you can imagine a job like this one:

class BookSalesJob < ActiveJob::Base
  unique :until_executed

  def perform
    ...
  end
end

The syntax is less verbose but very similar to the sidekiq-unique-jobs gem. It might solve your case if you highly rely on ActiveJob.

Final Thoughts

I hope you gained some knowledge in how to deal with duplicate jobs in your app. I definitely had fun researching and playing around with different solutions. If you didn't end up finding what you were looking for, I hope that some of the examples inspired you to create something of your own.

Here's the example project with all the code snippets.

I will see you in the next one, cheers.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Linting Ruby Code

Roy Tomeij — 2021-04-28T00:00:00+00:00

Linting is the process of statically analyzing code in search of potential problems.

What constitutes a problem, in this case, can vary across programming languages, or even across projects within the same language. I would put these problems under a few different categories:

Programmatic
Security
Stylistic
Performance

Let's take a look at a few examples of each.

Stylistic Problems

There's no objectively right way of styling code, as it's all about the reader's preference. The key is consistency though. Common points of debate include:

double-quotes vs single-quotes
tabs vs space
maximum line length
indentation of multiline calls, as shown below:

# always single-line
foo(:a, :b, :c)

# aligned with first argument
foo(:a,
    :b,
    :c
)

# aligned with function name
foo(
  :a,
  :b
)

These are entirely subjective, but it's usually beneficial to agree on a standard for each particular project, to keep the entire codebase consistent.

Programmatic Problems

I include here such problems as:

Extremely long method bodies, which leads to readability & maintainability issues
Cyclomatic Complexity, a metric commonly used to measure code complexity
Assignments within conditions. Most likely, if you type if x = true, you actually meant if x == true. and even if you did mean an assignment, it's still a less-intuitive approach

Security Problems

Some functions or practices have potential security problems that developers may not be aware of.

For instance, in Ruby, Kernel#open is a flexible function that allows opening files or external URLs. But it also allows arbitrary filesystem access, with weird calls such as open("| ls"). Therefore, it is sensible to warn developers about it so that they can either use a safer approach (File#open, IO.popen, URI.parse#open), or explicitly decide to keep the behavior at their own risk.

Performance Problem

There are many details about Ruby's inner workings that make some options more performant than others, depending on the context.

A linter warning us about them helps us learn along the way while optimizing some details of our program.

For example, Ruby 2.5 introduced String#delete_suffix which deletes a substring from the end of a string. These two lines are equivalent, but the latter one is more performant since it doesn't rely on a generic string regex match:

str = 'string_with_suffix'

# bad
str.gsub(/suffix\z/, '')

# good
str.delete_suffix('suffix')

Auto Fixing

An important aspect of linters is their ability to automatically fix some or all of the found issues. Styling aspects such as line length are easily automated, so it makes sense to remove that burden from the developer. Other issues might be subjective or require human intervention, such as refactoring a large method. In these cases, no automation is possible.

Convention or Configuration

There is often heavy debate within a community or project on which rules make sense.

The traditional solution is to allow each team to solve the debate within its members by allowing them to configure linting rules to their own taste. However, in recent years, there has been a push across several languages to standardize to a single convention. While this isn't enforced everywhere, the general idea is to entirely remove the mental overhead developers have when styling code. Instead of discussing which line length works best, everyone just uses the community-agreed rules.

In Ruby, this more or less translates to the two existing linters: RuboCop, which allows full-configuration and StandardRB, which takes the opposite approach and defines a common standard.

RuboCop

Employes the usual approach of providing a documented set of rules, each of which looks for a particular problem. Developers are able to disable or tweak certain rules within their own projects:

# configure max allowed line length
Layout/LineLength:
  Max: 80

# disable cyclomatic complexity
Metrics/CyclomaticComplexity:
  Enabled: false

It already includes sane defaults, so configuration is only needed for the specific rules you want to change.

Running bundle exec rubocop will make RuboCop analyze the entire codebase and list all the issues it found:

# test.rb
def badName
  if something
    return "inner result"
  end

  "outer result"
end

$ bundle exec rubocop
Inspecting 1 file
C

Offenses:

test.rb:1:1: C: [Correctable] Style/FrozenStringLiteralComment: Missing frozen string literal comment.
def badName
^
test.rb:1:5: C: Naming/MethodName: Use snake_case for method names.
def badName
    ^^^^^^^
test.rb:2:3: C: [Correctable] Style/IfUnlessModifier: Favor modifier if usage when having a single-line body. Another good alternative is the usage of control flow &&/||.
  if something
  ^^
test.rb:3:12: C: [Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols.
    return "inner result"
           ^^^^^^^^^^^^^^
test.rb:6:3: C: [Correctable] Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols.
  "outer result"
  ^^^^^^^^^^^^^^

1 file inspected, 5 offenses detected, 4 offenses auto-correctable

You can then run bundle exec rubocop --auto-correct, and a large majority of your issues will be fixed according to your configurations.

Setting up bundle exec rubocop as part of your CI pipeline will ensure no code gets through without first fulfilling the linting rules.

StandardRB

A more recent project, which actually uses RuboCop under the hood. The main goal of StandardRB is not to build an entirely separate linter, but to reach a standard that everyone can just use instead of arguing about.

The lightning talk where it was first announced is pretty clear about the motivations: If people spend less time arguing about syntactic details and just follow the decisions of a community-wide agreement, we can all spend more time doing what really matters: building great products and libraries.

Since RuboCop is used underneath, the output you get is actually in the same format. The only difference is that you're not allowed to customize any of the rules.

StandardRB recently reached 1.0.0, which means most of the discussion about which rules to use has already happened on their issues page. If you care or disagree about a particular rule, chances are you'll see a related discussion about it in there.

Ultimately though, you can be confident it was discussed in some depth. It's impossible for an entire community to agree 100% on all points. The philosophy of this approach is that people are flexible and can disagree and commit to a decision.

Final Thoughts

After spending more time than I'm proud of nitpicking linting rules in past projects, I definitely see the value in StandardRB's approach and I recommend using it whenever possible.

Keeping all the benefits of consistency while removing the overhead of discussions, along with the automated fixes for most rules helps us deliver better software more efficiently, and focus on what really matters.

Other languages are adopting similar low-configurability code formatters. mix formatter in Elixir, and rustfmt in Rust both allow for some configurability, but the community is surprisingly onboard with keeping within the standard.

With that said, RuboCop is still a perfectly valid option if you, ironically, disagree with this sentiment.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby on Rails Controller Patterns and Anti-patterns

Roy Tomeij — 2021-04-14T00:00:00+00:00

Welcome back to the fourth installment of the Ruby on Rails Patterns and Anti-Patterns series.

Previously, we covered patterns and anti-patterns in general as well as in relation to Rails Models and Views. In this post, we are going to analyze the final part of the MVC (Model-View-Controller) design pattern — the Controller. Let's dive in and go through the patterns and anti-patterns related to Rails Controllers.

At The Front Lines

Since Ruby on Rails is a web framework, HTTP requests are a vital part of it. All sorts of Clients reach out to Rails backends via requests and this is where controllers shine. Controllers are at the front lines of receiving and handling requests. That makes them a fundamental part of the Ruby on Rails framework. Of course, there is code that comes before controllers, but controller code is something most of us can control.

Once you define routes at the config/routes.rb, you can hit the server on the set route, and the corresponding controller will take care of the rest. Reading the previous sentence might give an impression that everything is as simple as that. But, often, a lot of the weight falls on the controller's shoulders. There is the concern of authentication and authorization, then there are problems of how to fetch the needed data, as well as where and how to perform business logic.

All of these concerns and responsibilities that can occur inside the controller can lead to some anti-patterns. One of the most 'famous' ones is the anti-pattern of a "fat" controller.

Fat (Obese) Controllers

The problem with putting too much logic in the controller is that you are starting to violate the Single Responsibility Principle (SRP). This means that we are doing too much work inside the controller. Often, this leads to a lot of code and responsibilities piling up there. Here, 'fat' refers to the extensive code contained in the controller files, as well as the logic the controller supports. It is often considered an anti-pattern.

There are a lot of opinions on what a controller should do. A common ground of the responsibilities a controller should have include the following:

Authentication and authorization — checking whether the entity (oftentimes, a user) behind the request is who it says it is and whether it is allowed to access the resource or perform the action. Often, authentication is saved in the session or the cookie, but the controller should still check whether authentication data is still valid.
Data fetching — it should call the logic for finding the right data based on the parameters that came with the request. In the perfect world, it should be a call to one method that does all the work. The controller should not do the heavy work, it should delegate it further.
Template rendering — finally, it should return the right response by rendering the result with the proper format (HTML, JSON, etc.). Or, it should redirect to some other path or URL.

Following these ideas can save you from having too much going on inside the controller actions and controller in general. Keeping it simple at the controller level will allow you to delegate work to other areas of your application. Delegating responsibilities and testing them one by one will ensure that you are developing your app to be robust.

Sure, you can follow the above principles, but you must be eager for some examples. Let's dive in and see what patterns we can use to relieve controllers of some weight.

Query Objects

One of the problems that happen inside controller actions is too much querying of data. If you followed our blog post on Rails Model anti-patterns and patterns, we went through a similar problem where models had too much querying logic. But, this time we'll use a pattern called Query Object. A Query Object is a technique that isolates your complex queries into a single object.

In most cases, Query Object is a Plain Old Ruby Object that is initialized with an ActiveRecord relation. A typical Query Object might look like this:

# app/queries/all_songs_query.rb

class AllSongsQuery
  def initialize(songs = Song.all)
    @songs = songs
  end

  def call(params, songs = Song.all)
    songs.where(published: true)
         .where(artist_id: params[:artist_id])
         .order(:title)
  end
end

It is made to be used inside the controller like so:

class SongsController < ApplicationController
  def index
    @songs = AllSongsQuery.new.call(all_songs_params)
  end

  private

  def all_songs_params
    params.slice(:artist_id)
  end
end

You can also try out another approach of the query object:

# app/queries/all_songs_query.rb

class AllSongsQuery
  attr_reader :songs

  def initialize(songs = Song.all)
    @songs = songs
  end

  def call(params = {})
    scope = published(songs)
    scope = by_artist_id(scope, params[:artist_id])
    scope = order_by_title(scope)
  end

  private

  def published(scope)
    scope.where(published: true)
  end

  def by_artist_id(scope, artist_id)
    artist_id ? scope.where(artist_id: artist_id) : scope
  end

  def order_by_title(scope)
    scope.order(:title)
  end
end

The latter approach makes the query object more robust by making params optional. Also, notice that we can now call AllSongsQuery.new.call. If you're not a big fan of this, you can resort to class methods. If you write your query class with class methods, it will no longer be an 'object', but this is a matter of personal taste. For illustration purposes, let's see how we can makeAllSongsQuery simpler to call in the wild.

# app/queries/all_songs_query.rb

class AllSongsQuery
  class << self
    def call(params = {}, songs = Song.all)
      scope = published(songs)
      scope = by_artist_id(scope, params[:artist_id])
      scope = order_by_title(scope)
    end

    private

    def published(scope)
      scope.where(published: true)
    end

    def by_artist_id(scope, artist_id)
      artist_id ? scope.where(artist_id: artist_id) : scope
    end

    def order_by_title(scope)
      scope.order(:title)
    end
  end
end

Now, we can call AllSongsQuery.call and we're done. We can pass in params with artist_id. Also, we can pass the initial scope if we need to change it for some reason. If you really want to avoid calling new over a query class, try out this 'trick':

# app/queries/application_query.rb

class ApplicationQuery
  def self.call(*params)
    new(*params).call
  end
end

You can create the ApplicationQuery and then inherit from it in other query classes:

# app/queries/all_songs_query.rb
class AllSongsQuery < ApplicationQuery
  ...
end

You still kept the AllSongsQuery.call, but you made it more elegant.

What's great about query objects is that you can test them in isolation and ensure that they are doing what they should do. Furthermore, you can extend these query classes and test them without worrying too much about the logic in the controller. One thing to note is that you should handle your request parameters elsewhere, and not rely on the query object to do so. What do you think, are you going to give query object a try?

Ready To Serve

OK, so we've handled ways to delegate the gathering and fetching of data into Query Objects. What do we do with the pilled-up logic between data gathering and the step where we render it? Good that you asked, because one of the solutions is to use what are called Services. A service is oftentimes regarded as a PORO (Plain Old Ruby Object) that performs a single (business) action. We will go ahead and explore this idea a bit below.

Imagine we have two services. One creates a receipt, the other sends a receipt to the user like so:

# app/services/create_receipt_service.rb
class CreateReceiptService
  def self.call(total, user_id)
    Receipt.create!(total: total, user_id: user_id)
  end
end

# app/services/send_receipt_service.rb
class SendReceiptService
  def self.call(receipt)
    UserMailer.send_receipt(receipt).deliver_later
  end
end

Then, in our controller we would call the SendReceiptService like this:

# app/controllers/receipts_controller.rb

class ReceiptsController < ApplicationController
  def create
    receipt = CreateReceiptService.call(total: receipt_params[:total],
                                        user_id: receipt_params[:user_id])

    SendReceiptService.call(receipt)
  end
end

Now you have two services doing all the work, and the controller just calls them. You can test these separately, but the problem is, there's no clear connection between the services. Yes, in theory, all of them perform a single business action. But, if we consider the abstraction level from the stakeholders' perspective — their view of the action of creating a receipt involves sending an email of it. Whose level of abstraction is 'right'™️?

To make this thought experiment a bit more complex, let's add a requirement that the total sum on the receipt has to be calculated or fetched from somewhere during the creation of the receipt. What do we do then? Write another service to handle the summation of the total sum? The answer might be to follow the Single Responsibility Principle (SRP) and abstract things away from each other.

# app/services/create_receipt_service.rb
class CreateReceiptService
  ...
end

# app/services/send_receipt_service.rb
class SendReceiptService
  ...
end

# app/services/calculate_receipt_total_service.rb
class CalculateReceiptTotalService
  ...
end

# app/controllers/receipts_controller.rb
class ReceiptsController < ApplicationController
  def create
    total = CalculateReceiptTotalService.call(user_id: receipts_controller[:user_id])

    receipt = CreateReceiptService.call(total: total,
                                        user_id: receipt_params[:user_id])

    SendReceiptService.call(receipt)
  end
end

By following SRP, we make sure that our services can be composed together into larger abstractions, like the ReceiptCreation process. By creating this 'process' class, we can group all the actions needed to complete the process. What do you think about this idea? It might sound like too much abstraction at first, but it might prove beneficial if you are calling these actions all over the place. If this sounds good to you, check out the Trailblazer's Operation.

To sum up, the new CalculateReceiptTotalService service can deal with all the number crunching. Our CreateReceiptService is responsible for writing a receipt to the database. The SendReceiptService is there to dispatch emails to users about their receipts. Having these small and focused classes can make combining them in other use cases easier, thus resulting in an easier to maintain and easier to test codebase.

The Service Backstory

In the Ruby world, the approach of using service classes is also known as actions, operations, and similar. What these all boil down to is the Command pattern. The idea behind the Command pattern is that an object (or in our example, a class) is encapsulating all the information needed to perform a business action or trigger an event. The information that the caller of the command should know is:

name of the command
method name to call on the command object/class
values to be passed for the method parameters

So, in our case, the caller of a command is a controller. The approach is very similar, just that the naming in Ruby is 'Service'.

Split Up The Work

If your controllers are calling some 3rd party services and they are blocking your rendering, maybe it's time to extract these calls and render them separately with another controller action. An example of this can be when you try to render a book's information and fetch its rating from some other service that you can't really influence (like Goodreads).

# app/controllers/books_controller.rb

class BooksController < ApplicationController
  def show
    @book = Book.find(params[:id])

    @rating = GoodreadsRatingService.new(book).call
  end
end

If Goodreads is down or something similar, your users are going to have to wait for the request to Goodreads servers to timeout. Or, if something is slow on their servers, the page will load slowly. You can extract the calling of the 3rd party service into another action like so:

# app/controllers/books_controller.rb

class BooksController < ApplicationController
  ...

  def show
    @book = Book.find(params[:id])
  end

  def rating
    @rating = GoodreadsRatingService.new(@book).call

    render partial: 'book_rating'
  end

  ...
end

Then, you will have to call the rating path from your views, but hey, your show action doesn't have a blocker anymore. Also, you need the 'book_rating' partial. To do this more easily, you can use the render_async gem. You just need to put the following statement where you render your book's rating:

<%= render_async book_rating_path %>

Extract HTML for rendering the rating into the book_rating partial, and put:

<%= content_for :render_async %>

Inside your layout file, the gem will call book_rating_path with an AJAX request once your page loads, and when the rating is fetched, it will show it on the page. One big gain in this is that your users get to see the book page faster by loading ratings separately.

Or, if you want, you can use Turbo Frames from Basecamp. The idea is the same, but you just use the <turbo-frame> element in your markup like so:

<turbo-frame id="rating_1" src="/books/1/rating"> </turbo-frame>

Whatever option you choose, the idea is to split the heavy or flaky work from your main controller action and show the page to the user as soon as possible.

Final Thoughts

If you like the idea of keeping controllers thin and picture them as just 'callers' of other methods, then I believe this post brought some insight on how to keep them that way. The few patterns and anti-patterns that we mentioned here are, of course, not an exhaustive list. If you have an idea on what is better or what you prefer, please reach out on Twitter and we can discuss.

Definitely stay tuned on this series, we are going to do at least one more blog post where we sum up common Rails problems and takeaways from the series.

Until next time, cheers!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Troubleshooting ActiveRecord Performance

Roy Tomeij — 2021-02-24T00:00:00+00:00

ActiveRecord is Ruby on Rails’ most magical feature. We don’t usually need to worry about its inner workings, but when we do, here’s how AppSignal can help us know what’s going on under the hood.

What Is ActiveRecord?

To talk about ActiveRecord, we need to first think of frameworks, specifically about MVC frameworks. MVC stands for Model-View-Controller, and it’s a popular software design pattern for graphical and web applications.

MVC frameworks are composed of:

Model: handles business logic and data persistence.
View: drives the presentation layer and draws the user interface.
Controller: ties everything together.

ActiveRecord is the model component in the Ruby in Rails framework. It introduces an abstraction layer between code and data, so we don’t have to write SQL code ourselves. Each model is mapped to one table and provides various methods to perform CRUD operations (Create, Read, Update and Delete).

Monitoring ActiveRecord With AppSignal

Abstractions feel magical — they help us ignore details we don’t need to know and focus on the task at hand. But when things don’t work as expected, the complexity they add can make it harder to determine the root cause. AppSignal can give us a detailed breakdown of what’s really happening in Rails.

The Response Time Graph

Let’s get a first look into the problem. Troubleshooting performance issues is an iterative process. Once you have your Rails application reporting to AppSignal, go to Incidents > Performance.

The Response Time graph will show the response time percentiles for each namespace. ActiveRecord events are automatically assigned to the namespace the request or background job the queries are executed in.

Next, look at the Event Group graph. It shows how much time is consumed by category. Check how much relative time is spent by active_record. Check the usage in all your namespaces.

The graph will immediately tell us where we should focus our code-optimizing efforts.

While you’re in the performance graph dashboard, check the response time and the throughput to ensure there isn’t any higher-than-usual activity on your application.

The Slow Queries Dashboard

Now that we’ve identified that the problem is data-bound let’s see if we can zoom in to determine the root cause.

Open the Improve > Slow Queries dashboard. This page shows the list of SQL queries ranked by impact on the overall time. All ActiveRecord-originated queries are shown as a sql.active_record events.

Try clicking on the topmost query to see its details. The dashboard shows the average duration and the query text.

Scrolling below will show you the query’s response time in the last few hours and the originating action.

You might find that some of the actions have an associated incident. This means that AppSignal created a performance incident while the query was running, but it doensn’t necessarily mean that ActiveRecord is the cause of it.

Performance Measurements Dashboard

Performance measurement incidents are opened when AppSignal records a new endpoint or background job.

Incidents are located on Performance > Issue List dashboard.

The incident page shows the elapsed time and number of allocations for each of the MVC components. ActiveRecord problems will exhibit long durations in the active_record category.

The event timeline shows how the event progressed over time.

Finding ActiveRecord Issues

In this section, we’ll see how AppSignal can help us identify some common ActiveError issues.

Selecting the Relevant Columns

Database wisdom says that we should always retrieve the columns needed for the job. For example, instead of SELECT * FROM people we should SELECT first_name, surname, birthdate FROM people. That’s all well and good, but how do we do it on Rails?

By default, ActiveRecord retrieves all columns.

Person.all.each {
      # process data
}

Luckily, we have the select method to pick and choose the required columns:

Person.select(:name, :address, :birthdate).each {
      # process data
}

It may sound like I’m being obsessive about little details. But on wide tables, selecting all columns is just wasteful. You’ll notice that when this happens, ActiveRecord allocates big chunks of memory:

N+1 Problems

The N+1 problem happens when the application gets a set of records from the database and loops through it. This causes the application to execute N+1 queries, where N is the number of rows initially obtained. As you might imagine, this pattern scales poorly as the table grows. It’s such a damaging problem that AppSignal specifically warns you about it:

N+1 problems usually appear with associated models. Imagine we have a Person model:

class Person < ApplicationRecord
    has_many :addresses
end

Each person can have many addresses:

class Address < ApplicationRecord
    belongs_to :person
end

The most straightforward way of retrieving the data leads to the N+1 problem:

class RelatedTablesController < ApplicationController
    def index
        Person.all.each do |person|
            person.addresses.each do |address|
            address.address
            end
        end
    end
end

You can see in AppSignal that the application is running a SELECT per person:

The fix for this particular case is simple: use includes, which tells ActiveRecord to optimize the query for the needed columns:

class RelatedTablesController < ApplicationController
    def index
        Person.all.includes(:addresses).each do |person|
            person.addresses.each do |address|
            address.address
            end
        end
    end
end

Now we have two queries instead of N+1:

Processing by RelatedTablesController#index as HTML
   (0.2ms)  SELECT sqlite_version(*)
  ↳ app/controllers/related_tables_controller.rb:12:in `index'
  Person Load (334.6ms)  SELECT "people".* FROM "people"
  ↳ app/controllers/related_tables_controller.rb:12:in `index'

  Address Load (144.4ms)  SELECT "addresses".* FROM "addresses" WHERE "addresses"."person_id" IN (1, 2, 3, . . .)

Execute the Least Necessary Queries Per Table

This is sometimes confused with the N+1 problem, but it’s a bit different. When we query a table, we should retrieve all the data we think we’ll need to minimize read operations. There is, however, a lot of innocent-looking code that triggers redundant queries. For example, look how count always results in a SELECT COUNT(*) query in the following view:

<ul>
    <% @people.each do |person| %>
        <li><%= person.name %></li>
    <% end %>
</ul>

<h2>Number of Persons: <%= @people.count %></h2>

Now ActiveRecord does two queries:

Rendering duplicated_table_query/index.html.erb within layouts/application
  (69.1ms)  SELECT COUNT(*) FROM "people" WHERE "people"."name" = ?  [["name", "John Waters"]]
↳ app/views/duplicated_table_query/index.html.erb:3
Person Load (14.6ms)  SELECT "people".* FROM "people" WHERE "people"."name" = ?  [["name", "John Waters"]]
↳ app/views/duplicated_table_query/index.html.erb:6

In AppSignal, the symptom you’ll notice is that there are two active_record events on the same table:

The reality is that we don’t need two queries; we already have all the data we need in memory. In this case, the solution is to swap count with size:

<ul>
<% @people.each do |person| %>
<li><%= person.name %></li>
<% end %>
</ul>

<h2>Number of Persons: <%= @people.size %></h2>

Now we have a single SELECT, as it should be:

Rendering duplicated_table_query/index.html.erb within layouts/application
Person Load (63.2ms)  SELECT "people".* FROM "people" WHERE "people"."name" = ?  [["name", "Abdul Strosin"]]
↳ app/views/duplicated_table_query/index.html.erb:5

Another solution is to use preload to cache the data in memory.

Computing Aggregated Data in Rails

Aggregation is used to compute a value based on a set of data. Databases are great at working with big datasets. This is what they do and what we use them for. On the other hand, using Rails to aggregate doesn't scale since it requires getting all the records from the database, holding them in memory, and then calculating using high-level code.

We’re doing aggregation in Rails whenever we resort to Ruby functions like max, min, or sum over ActiveRecord elements or other enumerables.

class AggregatedColumnsController < ApplicationController
    def index
        @mean = Number.pluck(:number).sum()
    end
end

Fortunately, ActiveRecord models include specific methods that map to aggregation functions in the database. For example, the following query maps to SELECT SUM(number) FROM ..., which is much faster and cheaper to run than the previous example:

# controller

class AggregatedColumnsController < ApplicationController
    def index
        @mean = Number.sum(:number)
    end
end

Processing by AggregatedColumnsController#index as */*
  (2.4ms)  SELECT SUM("numbers"."number") FROM "numbers"

If you need more complex or combined aggregation functions, you may need to include a bit of raw SQL code:

sql = "SELECT AVG(number), STDDEV(number), VAR(number) FROM ..."
@results = ActiveRecord::Base.connection.execute(sql)

Managing Big Transactions

SQL transactions ensure consistent and atomic updates. When we use a transaction to make a change, either every row is updated successfully, or the whole thing is rolled back. In any case, the database always remains in a consistent state.

We can bundle a batch of changes in a single transaction with ActiveRecord::Base.transaction.

class BigTransactionController < ApplicationController
    def index
        ActiveRecord::Base.transaction do
            (1..1000).each do
                Person.create(name: 'Les Claypool')
            end
        end
    end
end

There are a lot of legitimate cases for using large transactions. These, however, run into the risk of slowing the database down. Besides, transactions exceeding certain thresholds will result in the database rejecting them.

The first sign of transactions being too big is spending a lot of time on commit transaction events:

Barring configuration issues on the database itself, the solution is to break down the transaction into smaller chunks.

Monitoring The Database

Sometimes even though we search the code, we can’t find anything wrong with it. Then, there may be a problem in the database itself. Database engines are complex, and many things can go wrong: low memory, default settings, missing indexes, inconveniently-scheduled backup jobs. The picture can’t be complete unless we get information about the machine running the database.

If we’re running our own databases, we can install the standalone agent to capture host metrics. To learn more about how to use the standalone agent, read: Monitoring Any System with StatsD and AppSignal’s Standalone Agent.

The following signs might show something is going on with the database. Go to the Inspect > Host Metrics dashboard to see the resource usage in your server:

High memory usage: databases need a lot of memory — more than most other systems — to run correctly. As the dataset grows, memory requirements usually scale along. We’ll need to either add more memory or split the dataset among different machines from time to time.
Swap usage: ideally, database machines should not need swap memory at all. Swapping kills database performance. Its presence means there’s a more in-depth configuration or memory problem.
High I/O usage: peaks in disk activity can be due to maintenance tasks such as re-indexing or backing up the database. Doing these tasks during peak hours will definitely hit performance.

👋 If you like this article, there is a lot more we wrote about Ruby (on Rails) performance, check out our Ruby performance monitoring checklist.

Conclusion

Diagnosing performance issues is never an easy task. Today, we’ve learned how to use Appsignal to quickly pinpoint the source of the problem.

Let’s keep learning about AppSignal and Ruby on Rails:

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Using Webpacker in Your Ruby on Rails Application — a Deep Dive

Roy Tomeij — 2021-02-17T00:00:00+00:00

At the beginning of the internet age, websites were much simpler and not very interactive. With the advancement of technology, devices, and programming languages, they became more complex and consisted of several files, including assets like images and CSS stylesheets.

The more interactive your website is, the more JavaScript code you have to use. To use such code, you have to include all HTML code files using the script tag. Such an approach is error-prone because you have to remember to include every single file and keep the correct order, otherwise, your code may not work. With the Webpack tool, these problems go away. Webpacker is a bridge between Webpack and a Rails application.

This article takes a deep dive into Webpacker and offers a detailed explanation that will enable you to understand how this tool works under the hood. To make the content as valuable as possible, I decided to divide it into the following sections:

High-level overview — before we do the deep dive, I will explain the high-level overview of both Webpack and Webpacker. You can treat it as the preparation for more advanced topics.
Structure explanation — since we'll explore multiple files, it is good to know why the particular structure was used and how it responds to the general purpose.
Anatomy explanation — this part covers the investigation of the most important files of the Webpack setup. Knowing how they work is essential for the effective and hassle-free development process.
Environment specific setup — the last part explains how Webpacker works in development and production environments.

After reading this article, you'll know what Webpack and Webpacker are and why we need them in our Rails application. The deep dive into Webpacker’s internals will help you understand how the tool connects with Webpack and how it communicates with the Rails application.

Before Diving In

Before going out into deep water, we first need to prepare. A high-level overview is a great way to start working with any technology as it helps us understand the purpose of a given tool and the problems it solves without going deep into its internals.

Webpack

I already mentioned that Webpack helps us organize our JavaScript file code to avoid errors and improve a website’s performance. It organizes the code into bundles.

A bundle is a file where multiple modules are intelligently placed, respecting the dependency graph that is first created. Thanks to this process, we can be sure that our code will work as expected and before we invoke a given library’s code, it will have been loaded.

Webpack isn't the only tool that is used by default in the newest version of Rails. When the project is generated, the configuration files for Babel and PostCSS are created as well. PostCSS is a tool for transforming CSS with JavaScript, and Babel is a JavaScript compiler that enables us to write modern JavaScript without worrying about browser support.

Webpacker

Webpacker is a tool that integrates Webpack with a Rails application. It makes it easy to configure and develop JavaScript-like applications and optimize them for the production environment.

The source code is available as a gem, and the library comes with a development server that makes the development of the app very fast as you don’t have to stop and start the server to see the changes introduced in the JavaScript files.

The Deep Immersion

Now that you know why Webpack was created and what role Wepacker plays in a Rails application, we can focus on Webpacker’s internals to see how it organizes files, speeds up the development process, and optimizes files for the production environment.

Webpacker is available out-of-the-box in the newest version of Rails. Before creating a new project, ensure that the Node version is greater or equal to 10.17.0 as the webpacker:install command will be automatically invoked with rails new command.

Webpacker File Structure

After the install command is executed, the following files are created:

config/webpacker.yml — the main configuration file that contains the default configuration and configs for specific environments
config/webpacker/ — the directory where the JavaScript configuration files for particular environments are created
bin/webpack — an executable file that invokes webpack to create bundles
bin/webpack-dev-server — an executable file that starts the development server, which reloads webpack every time you make a change inside the JavaScript files included in the bundle

The Anatomy of the Configuration File

The main configuration file for Webpacker is located under the config directory, and it’s named webpacker.yml. Unlike Webpacker, configuration for Webpack is stored separately for each environment under the config/webpack directory.

Webpacker

By default, the configuration file contains many entries — default ones, and sections for each environment where you can overwrite specific settings. Let’s take a look at the most important settings:

source_path — the primary source of javascript files in your application. It’s set to app/javascript by default, and usually, there's no need to change this value.
source_entry_path — the name of the directory under the source_path where you keep the pack files, aka entry points. By default, this setting is set to the packs’ directory.
public_root_path — path to the directory in your application that is accessible from a browser. In a typical Rails application, it’s a public directory
public_output_path — when Webpacker compiles your files, it will put all compiled files in this directory under the public_root_path. By default, the directory is named packs, but you can call it whatever you want.
webpack_compile_output — if the flag is set to true, then output messages are displayed when files are compiled. It’s useful when compilation fails, as you'll be aware of that and can take action.

It is worth mentioning that the development section also contains configuration for the dev server that is used to compile files in the development environment without the need for restarting the server.

If you would like to access the configuration from the rails console or code level, you can call Webpacker.manifest.config, which will return the configuration class instance.

Webpack

Each environment has its configuration file, but in every file, the main environment file is imported: config/webpack/environment.js. In the environment configuration file, there is a place for loading custom plugins that will modify the default behavior of Webpack. We can also add custom rules for compiling our files.

The Anatomy of the Pack File

Packs are located under the app/javascript/packs directory. Each pack file is treated by Webpacker as an entry point when the compilation process starts.

The Default Pack File

When you generate a new Rails project, a default pack file named application.js is created with the following content:

import Rails from "@rails/ujs";
import Turbolinks from "turbolinks";
import * as ActiveStorage from "@rails/activestorage";
import "channels";

Rails.start();
Turbolinks.start();
ActiveStorage.start();

As you can see, the first step is to import the given library and then call its initialization method when it’s needed. When you call import, the system searches for a given node module or a local library. With the standard approach, you don’t have to explicitly initialize the library as it’s initialized when you include it in the page source with the script tag.

Good Practices for a Pack File

While you can put regular JavaScript code inside the pack file and execute it, it is not recommended to keep JavaScript code inside the pack file. The best approach is to keep the files clean and only import and initialize libraries here and keep other logic outside the packs directory.

Including Pack Files in the Application

Pack files are not automatically included in the website’s source. Just like in the asset pipeline case, we have to use a special tag inside our views:

<%= javascript_packs_with_chunks_tag 'application' %>

How does this method work? Under the hood, it calls the Webpacker configuration that holds the data you put in the config/webpacker.yml file. It looks for an application.js file inside the entry points directory, which, in our case, is app/javascript/packs. You can verify it by calling the following code:

Webpacker.manifest.config.source_entry_path

When entries are found, Webpacker calls javascript_include_tag method, a helper from the ActionView library available by default in Rails. The method accepts one or more sources and returns script tags that you can apply straight to your view.

The Development Server

Webpacker development server’s entry point is the executable file placed inside the bin folder and named webpack-dev-server. The process of running the server consists of five steps:

Setting environment — the environment name for Node and Rails is set. When the environment is not specified, the program assumes that the server will be executed in the development environment.
Loading configuration — the file config/webpacker.yml is loaded with settings for the environment set in the previous step.
Validating command options — the program checks if we passed appropriate options to the server command. For example, it will throw an error when the --https option is given, but we didn’t specify it in the config/webpacker.yml file.
Verifying port’s availability — the program checks if the given port is available to use. If another program is already using it, it will throw an error letting you know that you have to update the webpack server configuration inside the config/webpacker.yml file.
Executing webpack serve command — when the node modules directory exists, the program executes the command within the directory, otherwise, it executes it with yarn. The program passes the configuration option to the command that points to one of the files placed inside the config/webpack/ directory.

The server is now running, and it compiles your code on the fly, so you don’t have to restart the server to see the changes and check if webpack was able to create the bundle with the new code.

The development server’s code is quite simple. It only loads the config from the configuration file in YAML format and passes the proper config in JavaScript format directly to the webpack serve command. You can also run it by hand, but you don’t have to take care of proper command arguments with the built-in command.

Compiling Files for the Production Environment

Suppose you want to deploy the application that's using webpacker. In that case, you can simply invoke the assets:precompile task as Webpacker automatically hooks up the task webpacker:compile to it.

How does the webpacker:compile task work under the hood? Let’s dig into it and see. It does the following things:

It wraps the main call into two blocks ensuring that the NODE environment is set and Webpacker logs are output to the standard out.
It calls Webpack and then parses its logs to determine if the action was successful or not.

You can now visit the public/packs/js directory to see the compiled files. They should be deployed to the server.

Summary

We have just learned how Webpacker works under the hood. You can treat it as a bridge between the Webpack library and a Rails application that allows you to configure Webpack with Ruby and easily use javascript code inside your application.

To summarize the whole article, let’s recall the main elements of Webpacker again:

Configuration — it’s placed inside the config/webpacker.yml file and allows you to point Webpacker to the directory where you keep your javascript files and packs
Pack — it’s a single entry point from which Webpacker begins compilation. Each library imported and initialized in that file will be automatically compiled.
Development server — a simple script that allows you to compile files on the fly, so you don’t have to restart the server each time you modify JavaScript files
Compile task — a simple rake task that invokes Webpack and compiles files under the public directory so you can deploy them to the production server and make them available for end-users
View helper — helper that allows you to include compiled files inside the views

For many developers without much experience, Wepacker seems to be a little magical, but in fact, it’s a simple library that helps us use Webpack in a Ruby way.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby on Rails View Patterns and Anti-patterns

Roy Tomeij — 2021-02-10T00:00:00+00:00

Welcome back to the third installment of the Ruby on Rails Patterns and Anti-Patterns series. In the previous posts, we covered patterns and anti-patterns in general as well as in relation to Rails Models. In this post, we are going to go over some patterns and anti-patterns associated with Rails views.

Rails views can sometimes work perfectly and be fast, and at other times, they can have all sorts of issues. If you want to increase confidence over how you handle your views or you just want to learn more on the topic, then this blog post is for you. Let's dive right in.

As you probably know, the Rails framework follows convention over configuration. And since Rails is big on the Model-View-Controller (MVC) pattern, the motto naturally applies to the View code as well. This includes your markup (ERB or Slim files), JavaScript and CSS files. At first glance, you might think that the View layer is pretty straightforward and easy, but keep in mind that these days, there is a mix of technologies living in the View layer.

We use JavaScript, HTML, and CSS in the view. These three can lead to confusion and disorganization of code — leading to implementation that doesn't make much sense in the long run. Luckily, today we are going to go through some common problems and solutions with the Rails View layer.

Powerlifting Views

This is a mistake that doesn't happen that often, but when it does, it's an eyesore. Sometimes, people tend to put the domain logic or querying directly inside the View. This makes the View layer do the heavy-lifting or powerlifting. What is interesting is that Rails actually allows this to easily happen. There is no 'safety net' when it comes to this, you are allowed to do whatever you want in the View layer.

By definition, the View layer of the MVC pattern should contain presentation logic. It shouldn't be bothered with domain logic or with querying data. In Rails, you get ERB files (Embedded Ruby) that allow you to write Ruby code that will then get evaluated into HTML. If we consider an example of a website that lists songs on the index page, then the view logic would be in the app/views/songs/index.html.erb.

To illustrate what "powerlifting" means and what not do to, let's take a look at the following example:

# app/views/songs/index.html.erb

<div class="songs">
  <% Song.where(published: true).order(:title) do |song| %>
    <section id="song_<%= song.id %>">
      <span><%= song.title %></span>

      <span><%= song.description %></span>

      <a href="<%= song.download_url %>">Download</a>
    </section>
  <% end %>
</div>

A huge anti-pattern here is the fetching of songs right in the markup. The responsibility of fetching the data should be delegated to the controller or a service that is being called from the controller. I sometimes see people prepare some data in the controller and later fetch more data in the views. This is bad design and it makes your website slower because you are stressing your database with queries more often.

What you should do instead is to expose a @songs instance variable from the controller action and call that in the markup, like so:

class SongsController < ApplicationController
  ...

  def index
    @songs = Song.all.where(published: true).order(:title)
  end

  ...
end

# app/views/songs/index.html.erb

<div class="songs">
  <% @songs.each do |song| %>
    <section id="song_<%= song.id %>">
      <span><%= song.title %></span>

      <span><%= song.description %></span>

      <a href="<%= song.download_url %>">Download</a>
    </section>
  <% end %>
</div>

These examples are far from perfect. If you want to keep your controller code more readable and avoid SQL Pasta, I urge you to check out the previous blog post. Also, leaving out the logic in the View layer increases the chances that other people will try to build their solutions off of it.

Make Use of What Rails Gives You

We will keep it short here. Ruby on Rails as a framework comes with a lot of neat helpers, especially inside the view. These nifty little helpers allow you to build your View layer quickly and effortlessly. As a beginner user of Rails, you might be tempted to write the full HTML inside your ERb files like so:

# app/views/songs/new.html.erb

<form action="/songs" method="post">
  <div class="field">
    <label for="song_title">Title</label>
    <input type="text" name="song[title]" id="song_title">
  </div>

  <div class="field">
    <label for="song_description">Description</label>
    <textarea name="song[description]" id="song_description"></textarea>
  </div>

  <div class="field">
    <label for="song_download_url">Download URL</label>
    <textarea name="song[download_url]" id="song_download_url"></textarea>
  </div>

  <input type="submit" name="commit" value="Create Song">
</form>

With this HTML, you should get a nice form for a new song as seen in the screenshot below:

But, with Rails, you don't need and you shouldn't write plain HTML like that since Rails has your back right there. You can use the form_with view helper that will generate the HTML for you. form_with was introduced in Rails 5.1 and it is there to replace form_tag and form_for that might be familiar to some folk. Let's see how form_with can relieve us from writing extra code:

<%= form_with(model: song, local: true) do |form| %>
  <div class="field">
    <%= form.label :title %>
    <%= form.text_field :title %>
  </div>

  <div class="field">
    <%= form.label :description %>
    <%= form.text_area :description %>
  </div>

  <div class="field">
    <%= form.label :download_url do %>
      Download URL
    <% end %>
    <%= form.text_area :download_url %>
  </div>

  <%= form.submit %>
<% end %>

Besides generating HTML for us, form_with also generates an authenticity token that prevents CSRF attacks. So in almost all cases, you are better off using designated helpers since they might play well with the Rails framework. If you tried to submit a plain HTML form, it will fail because there was no valid authenticity token submitted with the request.

Besides form_with, label, text_area, and submit helpers, there are a bunch more of these view helpers that come out-of-the-box with Rails. They are there to make your lives easier and you should get to know them better. One of the "all-stars" is definitely link_to:

<%= link_to "Songs", songs_path %>

Which will generate the following HTML:

<a href="/songs">Songs</a>

I won't go into much detail on each helper, since this post will be too long and going through all of them is not part of today's topic. I suggest you go through Rails Action View helpers guide and pick what you need for your website.

Reusing and Organizing View Code

Let's imagine the perfect web application. In the perfect use-case, there are no if-else statements, just pure code that takes data from the controller and puts it between HTML tags. That kind of application exists maybe in hackathons and dreams, but real-world applications have a bunch of branches and conditions when rendering views.

What should you do when the logic for showing parts of a page gets too complex? Where do you go from there? A general answer would be to perhaps reach for a modern JavaScript library or framework and build something complex. But, since this post is about Rails Views, let's look at the options we have inside them.

After-Market (Custom) Helpers

Let's say you want to show a call-to-action (CTA) button below a song. But, there is a catch — a Song can either have a download URL or, for whatever reason, it can be missing. We might be tempted to code something similar to the following:

app/views/songs/show.html.erb

...

<div class="song-cta">
  <% if @song.download_url %>
    <%= link_to "Download", download_url %>
  <% else %>
    <%= link_to "Subscribe to artists updates",
                artist_updates_path(@song.artist) %>
  <% end %>
</div>

...

If we look at the example above as an isolated presentational logic, it doesn't look too bad, right? But, if there are more of these conditional renders, then the code becomes less readable. It also increases the chances of something, somewhere not getting rendered properly, especially if there are more conditions.

One way to fight these is to extract them to a separate helper. Luckily, Rails provides us a way to easily write custom helpers. In the app/helpers we can create a SongsHelper, like so:

module SongsHelper
  def song_cta_link
    content_tag(:div, class: 'song-cta') do
      if @song.download_url
        link_to "Download", @song.download_url
      else
        link_to "Subscribe to artists updates",
                artist_updates_path(@song.artist)
      end
    end
  end
end

If we open up the show page of a song, we will still get the same results. However, we can make this example a bit better. In the example above, we used an instance variable @song. This might not be available if we decide to use this helper at a place where @song is nil. So to cut off an external dependency in the form of an instance variable, we can pass in an argument to the helper like so:

module SongsHelper
  def song_cta_link(song)
    content_tag(:div, class: 'song-cta') do
      if song.download_url
        link_to "Download", song.download_url
      else
        link_to "Subscribe to artists updates",
                artist_updates_path(song.artist)
      end
    end
  end
end

Then, in the view, we can call the helper like below:

app/views/songs/show.html.erb

...

<%= song_cta_link(@song) %>

...

With that, we should get the same results in the view as we did before. The good thing about using helpers is that you can write tests for them ensuring that no regression happens regarding them in the future. A con is that they are globally defined and you have to ensure that helper names are unique across your app.

If you are not a big fan of writing Rails custom helpers, you can always opt-in for a View Model pattern with the Draper gem. Or you can roll your own View Model pattern here, it shouldn't be that complicated. If you are just starting out with your web app, I suggest starting slowly by writing custom helpers and if that brings pain, turn to other solutions.

DRY up Your Views

What I really liked when I started with Rails was the ability to easily DRY up your markup that it was almost unbelievable to me. Rails gives you the ability to create partials — reusable code pieces that you can include anywhere. For example, if you are rendering songs in multiple places, and you have the same code across multiple files, it makes sense to create a song partial.

Let's say you show your song as shown below:

# app/views/songs/show.html.erb

<p id="notice"><%= notice %></p>

<p>
  <strong>Title:</strong>
  <%= @song.title %>
</p>

<p>
  <strong>Description:</strong>
  <%= @song.description %>
</p>

<%= song_cta_link %>

<%= link_to 'Edit', edit_song_path(@song) %> |
<%= link_to 'Back', songs_path %>

But, you also want to show it on another page with the same markup. Then you can create a new file with an underscore prefix like app/views/songs/_song.html.erb.

# app/views/songs/_song.html.erb

<p>
  <strong>Title:</strong>
  <%= @song.title %>
</p>

<p>
  <strong>Description:</strong>
  <%= @song.description %>
</p>

<%= song_cta_link(@song) %>

And then wherever you want to include the song partial, you just do the following:

...

<%= render "song" %>

...

Rails will do an auto-lookup of whether the _song partial exists and it will render it. Similar to an example with custom helpers, it is best if we get rid of the instance variable @song in our partial.


# app/views/songs/_song.html.erb
<p>
  <strong>Title:</strong>
  <%= song.title %>
</p>

<p>
  <strong>Description:</strong>
  <%= song.description %>
</p>

<%= song_cta_link(song) %>

Then, we will need to pass in the song variable to the partial, making it more reusable and suitable to being included in other places.

...

<%= render "song", song: @song %>

...

Final Thoughts

That's all folks for this post. To summarize, we went through a few patterns and anti-patterns that you can come across in the Rails View realm. Here are a few takeaways:

Avoid complex logic in the UI (do not make the View do lots of powerlifting)
Learn what Rails gives you out-of-the-box in terms of View helpers.
Structure and reuse your code with custom helpers and partials
Do not depend on instance variables too much.

In the next post, we will cover Rails Controller patterns and anti-patterns where things can get pretty messy. Stay tuned for that.

Until the next one, cheers!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

RBS: A New Ruby 3 Typing Language in Action

Roy Tomeij — 2021-01-27T00:00:00+00:00

The long-awaited version 3.0.0 of Ruby has finally been released. Along with many great improvements, such as a 3x faster performance boost compared to the previous version, concurrency-parallel experimental features, etc., the Ruby team also introduced a new syntax language for dynamic typing in Ruby: RBS.

That was something the team had been discussing for years, based on the success of community-developed tools for static type checking such as Sorbet.

Sorbet is a powerful type checker backed by Stripe. It checks your code by both annotating and/or defining RBI files. RBI files, in turn, work as interfaces between static and dynamic components providing a "description" of them (constants, ancestors, metaprogramming code, and more).

So, if Sorbet mostly deals with static checking and RBS was made to address dynamic typing, what's the difference between them? How will they both coexist? When should I use one instead of the other?

Those are fairly common questions about the major role of RBS. That's why we decided to write this piece. To clarify, in practice, why you should consider adopting it based on what it's capable of. Let's dive right in!

Starting with the Basics

Let's start with a clear understanding of the difference between static typing _and dynamic typing_. Although it's basic, it is a key concept to grasp in order to understand the role of RBS.

Let's take a code snippet from a statically typed language as a reference:

➜
String str = "";
str = 2.4;

It's no news that such a language cares for the types of its objects and variables. That being said, code like the one above will throw an error.

Ruby, like many other languages such as JavaScript, Python, and Objective-C, doesn't give that much attention to which types you're targeting for your objects.

The same code in Ruby will just successfully run, as seen below:

➜  irb
str = ""
str = 2.4
puts str # prints 2.4

This is possible because Ruby's interpreter knows how to dynamically switch from one type to another.

However, there's a limit to what the interpreter allows. For example, take the following code change:

➜  irb
val = "6.0"
result = val + 2.0
puts result

This will, in turn, produce the following error:

Error: no implicit conversion of Float into String

Running the same code with JavaScript, for instance, would run just fine.

Moral of the story: Ruby does infer types dynamically, indeed; but, unlike other major dynamic languages, it won't accept everything. Be aware of that!

And that's where type checkers (whether static or dynamic) become useful.

RBS vs Sorbet

Right, I've got your point about the dynamic vs static thing. But, what about Sorbet? Will it get deprecated?

Not at all. The primary (and perhaps the most important) difference between RBS and Sorbet is that the former is just a language, while the latter is a complete type checker itself.

The Ruby team asserts RBS' main goal as to describe the structure of your code. It won't perform the type checking, but rather, define the structure that type checkers (like Sorbet or any other) can use to, well, type check. The code structure is stored within a new file extension — .rbs.

To check it out, let's take the following Ruby class as an example:

class Super
    def initialize(val)
      @val = val
    end


    def val?
      @val
    end
end

class Test < Super
  def initialize(val, flag)
    super(val)
    @flag = flag
  end

  def flag?
    @flag
  end
end

It represents a simple inheritance in Ruby. The interesting thing to note here is that we can't guess the types of each attribute used in the class, except for flag.

Since flag comes initialized with a default value, both the developer and a type checker can infer the type to prevent further misuse.

The following will be a proper representation of the above class in RBS format:

class Super
  attr_reader val : untyped

  def initialize : (val: untyped) -> void
end

class Test < Super
  attr_reader flag : bool

  def initialize : (val: untyped, ?flag: bool) -> void
  def flag? : () -> bool
end

Take some time to digest this. It's a declaration language, so only signatures may appear in an RBS file. Simple, isn’t it?

Whether it was autogenerated by a CLI tool (more on that later) or by you, it's safer to annotate a type as untyped when it can't be guessed.

If you're sure about the type of val, for example, your RBS mapping could be switched to the following:

class Super
  attr_reader val : Integer

  def initialize : (val: Integer) -> void
end

It's also important to note that both the Ruby and Sorbet teams were working (and still are) towards the creation and improvement of RBS. It was the Sorbet team's experience with type checking, for years, that helped the Ruby team to fine-tune a lot of stuff with this project.

The interoperability between RBS and RBI files is still under development. The goal is for Sorbet and any other checker tools to have an official and centralized basis to follow.

The RBS CLI Tool

One important consideration the Ruby team had when developing RBS was to ship a CLI tool that could help developers to try it out and learn how to use it. It's called rbs and comes by default with Ruby 3. If you still haven't upgraded your Ruby version, you can add its gem directly to your project, as well:

➜  gem install rbs

The command rbs help will show the command usage along with the available commands.

List of available commands

Most of these commands focus on parsing and analyzing the Ruby code structures. For example, the command ancestors sweeps the hierarchical structure of a given class to check for its ancestors:

➜  rbs ancestors ::String
::String
::Comparable
::Object
::Kernel
::BasicObject

The command methods displays all the method structures of a given class:

➜  rbs methods ::String
! (public)
!= (public)
!~ (public)
...
Array (private)
Complex (private)
Float (private)
...
autoload? (private)
b (public)
between? (public)
...

Want to see a specific method structure? Go for method:

➜  rbs method ::String split
::String#split
  defined_in: ::String
  implementation: ::String
  accessibility: public
  types:
      (?::Regexp | ::string pattern, ?::int limit) -> ::Array[::String]
    | (?::Regexp | ::string pattern, ?::int limit) { (::String) -> void } -> self

For those starting with RBS today, the command prototype can help a lot with scaffolding types for classes that already exist. The command generates prototypes of RBS files.

Let's take the previous Test < Super inheritance example and save the code into a file called appsignal.rb. Then, run the following command:

➜  rbs prototype rb appsignal.rb

Since the command allows for rb, rbi, and runtime generators, you need to provide the specific type of file you're scaffolding right after the prototype command, followed by the file pathname.

The following is the result of the execution:

class Super
  def initialize: (untyped val) -> untyped

  def val?: () -> untyped
end

class Test < Super
  def initialize: (untyped val, ?flag: bool flag) -> untyped

  def flag?: () -> untyped
end

Pretty similar to our first RBS version. As mentioned earlier, the tool marks as untyped any type that couldn't be guessed.

It also counts for method returns. Notice the return type of the flag definition. As a developer, you're probably sure that the method always returns a boolean, but due to Ruby’s dynamic nature, the tool is unable to 100% say that it is so.

And that's when another Ruby 3 child comes to the rescue: the TypeProf.

The TypeProf Tool

TypeProf is a type analysis tool for Ruby that was created on top of some syntax tree interpretation.

Despite still being experimental, it has proved to be very powerful when it comes to understanding what your code is trying to do.

If you don't have Ruby 3 yet, simply add the gem to your project:

➜  gem install typeprof

Now, let's run the same appsignal.rb file against it:

➜  typeprof appsignal.rb

This is the output:

# Classes
class Super
  @val: untyped
  def initialize: (untyped) -> untyped
  def val?: -> untyped
end

class Test < Super
  @val: untyped
  @flag: true

  def initialize: (untyped, ?flag: true) -> true
  def flag?: -> true
end

Note how the flag is mapped now. This is only possible because, unlike what the RBS prototype does, the TypeProf scans the method's body trying to understand what actions are being performed over that specific variable. Since it couldn't identify any direct change to this variable, TypeProf safely mapped the method return as a boolean.

Consider, for example, that TypeProf will have access to other classes that instantiate and use the Test class. With that in hand, it can go even deeper into your code and fine-tune its predictions. Say that the following code snippet is added at the end of the appsignal.rb file:

testSub = Test.new("My value", "My value" == "")
testSup = Super.new("My value")

And that you changed the initialize method signature to the following:

def initialize(val, flag)

When you re-run the command, this should be the output:

# Classes
class Super
  @val: String

  def initialize: (String) -> String
  def val?: -> String
end

class Test < Super
  @val: String
  @flag: bool

  def initialize: (String val, bool flag) -> bool
  def flag?: -> bool
end

Super cool!

TypeProf can't deal with inherited attributes very well. That's why we're instantiating a new Super object. Otherwise, it wouldn't get that val is a String.

The major pro of TypeProf is its safety. Whenever it's unable to figure something out for sure, then untyped will be returned.

Partial RBS Specification

One important warning from the official docs states that, although TypeProf is very powerful, you should be aware of its limitations regarding what it can and cannot generate in terms of RBS code.

For example, a common practice among Ruby developers is method overloading in which you invoke different behavior of a method depending on its arguments.

Consider that a new method spell is added to the Super class, which returns an Integer or a String based on the parameter type:

def spell(val)
  if val.is_a?(String)
    ""
  else
    0
  end
end

RBS embraces this practice by allowing you to deal with overloading through the union type (a value that represents multiple possible types) syntax:

def spell: (String) -> String | (Integer) -> Integer

TypeProf can't infer this just by analyzing the method's body. To help it out, you can manually add such a definition to your RBS file and TypeProf will always check there first for instructions.

For this, you must add the RBS file path at the end of the command:

typeprof appsignal.rb appsignal.rbs

See below the new output:

class Super
  ...
  def spell: (untyped val) -> (Integer | String)
end

Plus, we can also verify the real types during runtime via Kernel#p to test if the overloading is working by adding the next two lines to the end of the appsignal.rb file:

p testSup.spell(42)
p testSup.spell("str")

This should be the output:

# Revealed types
#  appsignal.rb:11 #=> Integer
#  appsignal.rb:12 #=> String

...

Make sure to refer to the official docs for more information, especially the section concerning TypeProf limitations.

Duck Typing

You've heard of that before. If a Ruby object does everything a duck does, then it is a duck.

As we've seen, Ruby doesn't care about what your objects are meant to be. Types can change dynamically as well as object references.

Although helpful, duck typing can be tricky. Let's see an example.

Suppose that, from now on, the val attribute you've declared for the Super class, which is a String, must always be convertible to an integer.

Rather than trusting that developers will always guarantee the conversion (perhaps, throwing an error otherwise), you can create an interface stating that:

interface _IntegerConvertible
   def to_int: () -> Integer
end

Interface types provide one or more methods that are detached from concrete classes and modules. This way, when you want a certain type to be passed on to the Super instantiation, you can simply do the following:

class Super
  attr_reader val : _IntegerConvertible

  def initialize : (val: _IntegerConvertible) -> void
end

The concrete class or module that implements this interface will have to make sure the proper validation is done.

Metaprogramming

Perhaps one of the most dynamic features of Ruby is the capability of creating code that creates code by itself during runtime. That's metaprogramming.

Because of the uncertain nature of things, the RBS CLI tool isn't able to generate RBS out of metaprogramming code.

Let's take the following snippet as an example:

class Test
    define_method :multiply do |*args|
        args.inject(1, :*)
    end
end

p Test.new.multiply(2, 3, 5)

This class defines a method called multiply at runtime and instructs it to inject the arguments and multiply each one with the previous result.

Once you run the RBS prototype command, this should be the output:

class Test
end

Depending on the complexity of your metaprogramming code, TypeProf will still try its best to extract something from it. But it's not always guaranteed.

Remember, you can always add your own type mappings to the RBS file and TypeProf will obey them in advance. That's also valid for metaprogramming.

It's also important to keep updated with the latest repository changes since the team is constantly releasing new features, which may well include updates on metaprogramming.

That being said, if your codebase includes some type of metaprogramming, be careful with these tools. Don't use them blindly!

Wrapping Up

There are plenty more details about what we've discussed so far, as well as edge use cases for both RBS and TypeProf that you should be aware of.

So, make sure to refer to the official docs for more on that.

RBS is still so fresh but has already caused a huge impact on Rubyists that are used to type checking their codebases with other tools.

What about you? Have you tried it out? What are your thoughts on RBS?

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Ruby Magic by AppSignal

Data Sovereignty: How to Keep All of Your Services in Europe (AppSignal + Hatchbox)

Why Data Sovereignty Matters More Than Ever

Traditional Trade-offs (and Why They Suck)

The Third Way: Own Your Cloud Without the Complexity

Practical Setup: A European-First Rails Stack

Benefits and Drawbacks

Conclusion

Setting Up Server Monitoring for a Rails App on Hatchbox

What AppSignal Captures at the Host Level

Reading the Host Dashboard

Memory

CPU and Load

Disk Usage

Connecting Host Metrics to App Problems

Alerts Worth Setting Up

Wrapping Up

Monitoring Sidekiq Job Performance with AppSignal

The Problem

What You Get Out of the Box

Three Signals That Matter

Adding AppSignal to the Ruby/Sidekiq Project

Job Duration

Queue Latency

Error Rate vs. Retry Count

Get Alerted of Failures Before Users Do

Slow Actions

Error Rate Increase

Queue Time

Wrap-Up

An Introduction to Ruby Parsing with Prism

Interpreters 101

Why Is Prism Useful for Ruby Parsing?

Your First Transpiler

Emojify the Ruby Tokens

Fix Indentation and Spacing

Onto Parsing with Prism for Ruby

Make Fixes with a Method

Mutation: The Source of All Evil

Wrapping Up

AppSignal’s Top 5 Ruby Posts in 2025

Top 5 Ruby Blog Posts in 2025 💎

An Introduction to Solid Queue for Ruby on Rails

A Deep Dive into Solid Queue for Ruby on Rails

Advanced Queries in ActiveRecord for Ruby on Rails

An Introduction to Game Development with DragonRuby

How to Read Code from the Showcase Ruby on Rails Engine

Season's Greetings ❆ ⛄

Create a Markdown Editor in Ruby on Rails

Markdown in Ruby on Rails

Markdown and Markdown Flavors

Creating a Markdown Editor with Ruby on Rails

Preview with Turbo Streams

Improving the User Experience

Adding Image Uploads to our Markdown Editor in Ruby on Rails

Wrap Up

Completing, Integrating, and Publishing Our Game with DragonRuby

Scene Management

Collision Detection

Storing and Managing a Score

Playing Sounds

Integration with an HTTP Server to Store and Retrieve High Scores

Publishing

Wrapping Up

Rendering Samples with Showcase for Ruby on Rails

Samples in our Ruby App

Sample Methods in Showcase for Ruby on Rails

About block.arity

About the consume Method

Calling extract_source

A More Complex Method

Back To extract_source

The Rouge Gem in Showcase for Ruby on Rails

Sample Instance Variables Render the View

Wrapping Up

An Introduction to Game Development with DragonRuby

Initial Concerns

First Steps: Game Development with DragonRuby

Getting Started with DragonRuby Development

The Anatomy of a DragonRuby Game

About `block.arity`

About the `consume` Method

Calling `extract_source`

Back To `extract_source`

About `view_context`

About the `Showcase::Preview` Initializer