appsignal

The Citadel Architecture at AppSignal

Thijs Cadier

Thijs Cadier on

The Citadel Architecture at AppSignal

DHH just coined the term "Citadel," which finally gives us an excellent way to reference how we approach tech at AppSignal. We said, "Hey, this is us! Our thing has a name now".

To explain how AppSignal uses the Citadel pattern, we'll share a bit on how our system works. AppSignal is a monitoring product that has a user-facing application and an API that the monitoring agent sends data to. This data is then processed and turned into graphs and insights.

Monolith

The application our customers interact with is a monolithic Rails app, with parts of the front-end written in React. The backend is entirely written in Ruby and talks to a few databases (we split out data from different customers to separate clusters for scaling reasons). This Rails app also handles a bunch of other tasks such as sending out alerts to external services.

When we started this Rails app processed incoming data from our monitoring agent as well, we foresaw that data ingestion would turn out to be a bottleneck. So we used a Sinatra app running on a subdomain that ingested data and created Sidekiq jobs that were processed by the Rails app.

Growing Pains

This architecture worked well for years. As our business grew, it became clear that the specific task of processing incoming data from the agents was going to need special treatment. When you're monitoring billions and billions of requests, you run into hard limits. The main limiting factor wasn't so much that Ruby is slow (we all know that it is not 😉), but that the way we had architected things caused too much locking in our databases.

An Outpost

We looked at several possibilities and then decided Kafka was the best fit for our situation. We had some experience with Rust and thought that its speed and reliability would be a very good fit for this system. We rewrote our data ingestion and processing system in Rust, using Kafka as a combination of queue and storage system.

We only moved the incoming data processing part of the Rails app to this outpost service. The rest of the systems works well in the form of a monolithic app. We understand it deeply, and we like keeping things simple. The monolith still handles most of the logic and interacts with Kafka heavily. Our wish to keep our monolith led to us writing a Kafka gem, so the main app can communicate with the outpost easily.

If you want to learn more about how Kafka works at AppSignal check out the Railsconf talk I gave about this.

Life at the Citadel

This brings us to the current situation where we are very happy in our citadel. As DHH said:

A single Majestic Monolith captures the majority mass of the app with a few auxiliary outpost apps for highly specialized and divergent needs.

In our case, we have a single outpost service for our highly specialized needs. If there were a RailsConf this year, we would have given DHH some extra stroopwafels as appreciation for giving it a name. 🍪

Thijs Cadier

Thijs Cadier

Thijs is a co-founder of AppSignal who sometimes goes missing for months on end to work on our infrastructure. Makes sure our billions of requests are handled correctly. Holds the award for best drummer in the company.

All articles by Thijs Cadier

Become our next author!

Find out more

AppSignal monitors your apps

AppSignal provides insights for Ruby, Rails, Elixir, Phoenix, Node.js, Express and many other frameworks and libraries. We are located in beautiful Amsterdam. We love stroopwafels. If you do too, let us know. We might send you some!

Discover AppSignal
AppSignal monitors your apps