Ask any software engineer to review key portions of production code, and inevitably, they will point out three things that need to be refactored. So why does so much bad, brittle, or misunderstood code remain running in production?
The answer is simple: engineers are afraid to touch it. Refactoring tasks get identified and added to the backlog, but rarely make it into the current sprint.
There are numerous reasons for this. The code may have been written by an engineer who left the team years ago, and no one completely understands it. In other cases, the capability is critical to the business. No one wants to be responsible for a potential outage or loss of revenue.
In this post, we'll examine how you can use Scientist to migrate, refactor, and change critical Ruby production code with confidence.
But first, you might ask — can't we use tests to dig up code issues?
This Is What Rails Testing Is For, Right?
Yes, and no. It is often difficult to gain complete confidence in code changes before deployment. The unit and system tests pass. It's good to go, right?
The reality is that there is no substitute for the real world, i.e. production. What if the data quality is bad or tests are missing? How can you know if the new software will perform well enough to handle production throughput?
Teams with public services sometimes find that they need to deal with “bugwards compatibility” issues. When a bug has existed in production for a while, clients may code in a way that depends on consistent incorrect behavior. Customers often use software in unexpected ways.
Observe Production Changes in Ruby and Rails with Scientist
If production is the best place to gain confidence in a change, then consider observing how code behaves there. This may sound scary at first, as the idea of “testing in production” contradicts classic software engineering practices.
However, the good news is that it’s easy and safe to do so in Ruby and Rails using the Scientist gem. Scientist's name is based on the scientific method of conducting experiments to verify a given hypothesis. In this case, our hypothesis is that the new code does the job.
The reason we can safely take this approach stems from the fact that experiments still use the result of the existing code. New code is only evaluated for observation and comparison purposes, both for accuracy and performance. We mitigate the test coverage concerns discussed earlier by evaluating performance using real-world data and parameters. Experiments typically evaluate a chosen sample rate of requests to minimize the impact on production. However, you can evaluate every request if desired.
Let's now take a quick look at how Scientist works in a branch by abstraction way.
The Branch by Abstraction Pattern in Ruby's Scientist
Scientist's approach begins with the Branch by Abstraction pattern described by Martin Fowler as making “a large-scale change to a software system in a gradual way.”
We introduce an abstraction layer to isolate the code being updated. This layer decides which implementation to use so that the experiment is transparent to the rest of the system. The technique is related to using a feature flag that determines the code path.
The Scientist gem, which originated from Github, implements this pattern using an experiment. The existing code is referred to as the control, and the new implementation is the candidate. Both code paths are run in randomized order, but only the control result is returned to the client.
Using Scientist to Refactor a Ruby Service
Consider a Ruby service that returns the largest prime factor for a given number. Assume that we've identified optimizations to prune the required set of candidates, speeding up the service.
However, service owners want to be sure no bugs were introduced. They also want to observe any performance improvements. Introduce the following code, modifying clients to call this method:
require 'scientist' def largest_prime_factor(number) science "prime-factors" do |experiment| experiment.use { find_largest_prime_factor(number) } # old way experiment.try { improved_largest_prime_factor(number) } # new way end # returns the control value end
At this point, only the use (control) expression is invoked. To make the experiment worthwhile, define a custom Experiment class to enable it (100% of the time below) and publish the results (in this case, just logging). Scientist generates fantastic data but it doesn’t do anything with it by default. That part is left up to you.
require 'scientist/experiment' require 'pp' class MyExperiment include Scientist::Experiment attr_accessor :name def initialize(name) @name = name end def enabled? true end def raised(operation, error) p "Operation '#{operation}' failed with error '#{error.inspect}'" super # will re-raise end def publish(result) pp result end end
The results of the experiment will be logged, and we can make improvements over time based on the feedback. Once the new code meets the requirements and confidence is high, accomplish cutover to the new implementation by simply replacing the science code with a delegation to the new implementation.
LabTech Simplifies Scientist Experiments in Ruby on Rails
You can use the LabTech gem in your Rails application to easily configure Scientist and handle the results.
Applications that use AppSignal can use the Appsignal.instrument custom instrumentation helper to track how long Scientist events take to complete. Wrap it around the different experiment code blocks to see events appear in the performance timeline.
Now, going back to LabTech — the web page below simply accepts a number to factor.
Getting started is easy provided you have access to the console. First, add the LabTech gem to your Gemfile and run a bundle install.
gem 'lab_tech'
Tables store results and experiment configuration, so run a database migration.
rails lab_tech:install:migrations db:migrate
The abstraction layer is the same, except the LabTech module is used. The full code is available on GitHub.
def largest_prime_factor(number) LabTech.science "prime-factors" do |experiment| ... end end
At this point, the experiment is disabled, so use the console to enable it in all cases or for a percentage of the time.
bin/rails console LabTech.enable "prime-factors" LabTech.enable "prime-factors", percent: 5
We can now run tests and the experiment will be evaluated. For a textual view of results, use either of the following commands from the Rails console.
LabTech.summarize_results "prime-factors" LabTech.summarize_errors "prime-factors"
After a few successful runs and one manufactured error, here is an example of what the results summary looks like. There is an overview of successes and failures, as well as an ASCII chart showing performance differences.
-------------------------------------------------------------------------------- Experiment: prime-factors -------------------------------------------------------------------------------- Earliest results: 2022-04-27T02:42:45Z Latest result: 2022-05-01T17:27:39Z (5 days) 3 of 4 (75.00%) correct 1 of 4 (25.00%) mismatched Median time delta: +0.000s (90% of observations between +0.000s and +0.000s) Speedups (by percentiles): 0% [ · █ ] +2.4x faster 5% [ · █ ] +2.4x faster 10% [ · █ ] +2.4x faster 15% [ · █ ] +2.4x faster 20% [ · █ ] +2.4x faster 25% [ · █ ] +2.4x faster 30% [ · █ ] +2.4x faster 35% [ · █ ] +2.4x faster 40% [ · █ ] +2.4x faster 45% [ · █ ] +2.4x faster 50% [ · · · · · · · · · · · · · · · · · █ · · · · · · · ] +2.4x faster 55% [ · █ ] +2.4x faster 60% [ · █ ] +2.4x faster 65% [ · █ ] +2.4x faster 70% [ · █] +6.9x faster 75% [ · █] +6.9x faster 80% [ · █] +6.9x faster 85% [ · █] +6.9x faster 90% [ · █] +6.9x faster 95% [ · █] +6.9x faster 100% [ · █] +6.9x faster --------------------------------------------------------------------------------
The Blazer gem provides a nice way to analyze the results easily. It is simple to install and allows SQL queries to run against tables. The query here shows that the candidate implementation is significantly faster than the original.
In the example prime factoring service, the speedup in the improved implementation comes from a heuristic that eliminates some possible factors to consider. As we consider higher numbers and find a prime factor, we can stop searching after we get to our target number divided by that factor. The new code path only adds one statement to accomplish this.
We can also see the reduction in execution time using a Blazer query against the LabTech tables.
Use Cases and Limitations of Scientist
Optimal use cases for Scientist include searches, calculations, and code that has no side effects. Code that includes transactional updates or external integrations such as email does not fit cleanly into the model because the capability is run twice (both the old and new implementations).
This is not a trivial limitation, as it does eliminate several use cases. However, there are some workarounds if the experiment is critical to your success. Consider whether the side effects are relevant or if duplication is an issue. For example, it may not matter in some cases whether two emails are sent during the evaluation. Another option is to have the new code determine the result but not persist it. This would prevent any meaningful performance comparisons. However, it would allow you to verify accuracy.
Other limitations stem from Scientist’s focus on return values. In some cases, valid results may exhibit differences over time, whether they simply include timestamps in the response or certain factors vary. In many cases, we can write custom comparison logic in the experiment to verify accuracy beyond basic string comparisons.
Finally, a limitation of LabTech is that it has not been ported to Rails 7 as of the time of writing.
Best Practices for Effective Scientist Experiments in Rails
Consider these items when implementing your experiments:
- In Rails projects, Scientist can either be configured in an initializer or a wrapper like the Rails LabTech gem. Most Rails applications already have a database, so LabTech leverages ActiveRecord to store results.
- To avoid slowing down development and testing, enable your experiment only in staging and production environments.
- To minimize any potential impact on production, only run the experiment on a percentage of requests. LabTech supports this out of the box as an optional parameter when you enable the experiment (it is initially disabled by default). Using pure Scientist, this logic is easy to code in the experiment’s
enabled?method. - Some logic is resource-intensive, so a low sampling rate may be a good place to start. As you gain confidence with the results, ramp up the percentage of requests being evaluated.
- You can add context attributes to get the most from your results. The experiment context can be set to a Symbol-keyed Hash of data that is then made available in published results, e.g.:
experiment.context :user => user
Wrap-Up: Observe and Monitor Your Ruby App with Scientist
In this post, we explored how to use the Scientist gem to change, migrate, and refactor Ruby code in production.
We examined Scientist's origins in the Branch by Abstraction pattern, then dived into refactoring. Next, we saw how LabTech could help with gathering results and your Scientist configuration.
We then touched on some limitations of Scientist before finally outlining a few best practices.
You must observe and monitor what is happening in your system. Integrate Scientist into your development process to make critical changes in your Ruby code with greater confidence.
Happy coding!
P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!
