Visit AppSignal.com

Ruby Magic

Building a Ruby HTTP server, part 2: Running a Rails app

Jeff Kreeftmeijer on

In an earlier edition of Ruby Magic we implemented a 30-line HTTP server in Ruby. Without having to write a lot of code, we were able to handle HTTP GET requests and serve a simple Rack application. This time, we’ll take our home made server a bit further. When we’re done, we’ll have a web server that can serve Rails’ famous fifteen minute blog that allows you to create, update and delete posts.

Where we left off

Last time, we implemented just enough of a server to have it serve Rack::Lobster as an example application.

  1. Our implementation opened a TCP server and waited for a request to come in.
  2. When that happened, the request-line (GET /?flip=left HTTP/1.1\r\n) was parsed to get the request method (GET), the path (/), and the query parameters (flip=left).
  3. The request method, the path and the query string were passed to the Rack app, which returned a triplet with a status, some response headers and the response body.
  4. Using those, we were able to build an HTTP response to send back to the browser, before closing the connection to wait for a new request to come in.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# http_server.rb
require 'socket'
require 'rack'
require 'rack/lobster'

app = Rack::Lobster.new
server = TCPServer.new 5678

#1
while session = server.accept
  request = session.gets
  puts request

  #2
  method, full_path = request.split(' ')
  path, query = full_path.split('?')

  #3
  status, headers, body = app.call({
    'REQUEST_METHOD' => method,
    'PATH_INFO' => path,
    'QUERY_STRING' => query
  })

  #4
  session.print "HTTP/1.1 #{status}\r\n"
  headers.each do |key, value|
    session.print "#{key}: #{value}\r\n"
  end
  session.print "\r\n"
  body.each do |part|
    session.print part
  end
  session.close
end

We’ll be continuing with the code we wrote last time. If you want to follow along, here’s the code we ended up with.

Rack and Rails

Ruby frameworks like Rails and Sinatra are built on top of the Rack interface. Just like the instance of Rack::Lobster we’re using to test our server right now, Rails’ Rails.application is a Rack application object. In theory, this would mean that our server should already be able to serve a Rails application.

To test that, I’ve prepared a simple Rails application. Let’s clone that into the same directory as our server.

1
2
3
4
5
6
7
8
9
10
11
12
$ ls
http_server.rb
$ git clone https://github.com/jeffkreeftmeijer/wups.git blog
Cloning into 'blog'...
remote: Counting objects: 162, done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 162 (delta 32), reused 162 (delta 32), pack-reused 0
Receiving objects: 100% (162/162), 29.09 KiB | 0 bytes/s, done.
Resolving deltas: 100% (32/32), done.
Checking connectivity... done.
$ ls
blog           http_server.rb

Then, in our server, require the Rails application’s environment file instead of rack and rack/lobster, and put the Rails.application in the app variable instead of Rack::Lobster.new.

1
2
3
4
5
6
7
# http_server.rb
require 'socket'
require_relative 'blog/config/environment'

app = Rails.application
server = TCPServer.new 5678
# ...

Starting the server (ruby http_server.rb) and opening http://localhost:5678 shows us we’re not quite there yet. The server doesn’t crash, but we’re greeted with an internal server error in the browser.

500 Internal Server Error. If you are the administrator of this website, then please read this web application's log file and/or the web server's log file to find out what went wrong.

Checking our server’s logs, we can see that we’re missing something called rack.input. It turns out that we’ve been lazy while implementing our server last time, so there’s more work to do before we can get this Rails application to work.

1
2
3
4
5
$ ruby http_server.rb
GET / HTTP/1.1
Error during failsafe response: Missing rack.input
  ...
  http_server.rb:15:in `<main>'

The Rack environment

Back when we implemented our server, we glossed over the Rack environment and ignored most of the variables that are required to properly serve Rack applications. We ended up only implementing the REQUEST_METHOD, PATH_INFO, and QUERY_STRING variables, as those were sufficient for our simple Rack app.

As we’ve already seen from the exception when we tried to start our new application, Rails needs rack.input, which is used as an input stream for raw HTTP POST data. Besides that, there are some more variables we need to pass, like the server’s port number, and the request cookie data.

Luckily, Rack provides Rack::Lint to help make sure all variables in the Rack environment are present and valid. We can use it to test our server by wrapping our Rails app in it by calling Rack::Lint.new and passing the Rails.application.

1
2
3
4
5
6
7
# http_server.rb
require 'socket'
require_relative 'blog/config/environment'

app = Rack::Lint.new(Rails.application)
server = TCPServer.new 5678
# ...

Rack::Lint will throw an exception when a variable in the environment is missing or invalid. Right now, starting our server again and opening http://localhost:5678 will crash the server and Rack::Lint will notify us of the first error: the SERVER_NAME variable wasn’t set.

1
2
3
4
5
~/Appsignal/http-server (master) $ ruby http_server.rb
GET / HTTP/1.1
/Users/jeff/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/rack-2.0.1/lib/rack/lint.rb:20:in `assert': env missing required key SERVER_NAME (Rack::Lint::LintError)
        ...
        from http_server.rb:15:in `<main>'

By fixing each error that is thrown at us, we can keep adding variables until Rack::Lint stops crashing our server. Let’s go over each of the variables Rack::Lint requires.

After adding all missing variables, Rack::Lint will notify us of one more problem in our environment.

1
2
3
4
5
$ ruby http_server.rb
GET / HTTP/1.1
/Users/jeff/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/rack-2.0.1/lib/rack/lint.rb:20:in `assert': env variable QUERY_STRING has non-string value nil (Rack::Lint::LintError)
        ...
        from http_server.rb:18:in `<main>'

When there’s no query string in the request, we’ll now pass nil as the QUERY_STRING, which is not allowed. In that case, Rack expects an empty string instead. After implementing the missing variables and updating the query string, this is what our environment looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# http_server.rb
# ...
  method, full_path = request.split(' ')
  path, query = full_path.split('?')

  input = StringIO.new
  input.set_encoding 'ASCII-8BIT'

  status, headers, body = app.call({
    'REQUEST_METHOD' => method,
    'PATH_INFO' => path,
    'QUERY_STRING' => query || '',
    'SERVER_NAME' => 'localhost',
    'SERVER_PORT' => '5678',
    'rack.version' => [1,3],
    'rack.input' => input,
    'rack.errors' => $stderr,
    'rack.multithread' => false,
    'rack.multiprocess' => false,
    'rack.run_once' => false,
    'rack.url_scheme' => 'http'
  })

  session.print "HTTP/1.1 #{status}\r\n"
# ...

Restarting our server and reloading http://localhost:5678, we’re greeted with another error. This time, however, it’s Rails’ “something went wrong”-page, confirming that we’re one step further.

We're sorry, but something went wrong. If you are the application owner check the logs for more information.

Looking through the Rails logs in blog/log/development.log, we’ll find the exception that’s causing the error:

1
2
3
4
5
Started GET "/" for  at 2017-01-23 08:00:26 +0100

IPAddr::InvalidAddressError: invalid address
        ...
        from http_server.rb:18:in `<main>'

Besides conforming to the Rack environment specification, Rails needs the REMOTE_ADDR variable to be set to hold the client’s IP address. Since we’re only running our server locally right now, we can hardcode it to “127.0.0.1”. Let’s add that to our environment.

1
2
3
4
5
6
7
8
9
10
# http_server.rb
# ...
  status, headers, body = app.call({
      # ...
      'SERVER_PORT' => '5678',
      'REMOTE_ADDR' => '127.0.0.1',
      'rack.version' => [1,3],
      # ...
  })
# ...

Restarting the server one more time and visiting http://localhost:5678 again, we’ll be greeted with Rails’ “You’re on Rails!”-page, meaning we’re now running an actual Rails application on our home made server!

Yay! You're on Rails!

Parsing HTTP POST bodies

This application is more than just that index page. Visiting http://localhost:5678/posts will display an empty list of posts. If we try to create a new post by filling in the new post form and pressing “Create post”, we’re once again greeted by an exception. This time, it’s ActionController::InvalidAuthenticityToken.

The authenticity token is sent along when posting a form and is used to check if the request came from a trusted source. Our server is completely ignoring POST data right now, so the token isn’t sent, and the request can’t be verified.

Back when we first implemented our HTTP server, we used session.gets to get the first line (called the Request-Line), and parsed the HTTP method and path from that. Besides parsing the Request-Line, we ignored the rest of the request.

To be able to extract the POST data, we’ll first need to understand how an HTTP request is structured. Looking at an example, we can see that the structure resembles an HTTP response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
POST /posts HTTP/1.1\r\n
Host: localhost:5678\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-us\r\n
Content-Type: application/x-www-form-urlencoded\r\n
Origin: http://localhost:5678\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14\r\n
Cookie: _wups_session=LzE0Z2hSZFNseG5TR3dEVEwzNE52U0lFa0pmVGlQZGtZR3AveWlyMEFvUHRPeXlQUzQ4L0xlKzNLVWtqYld2cjdiWkpmclZIaEhJd1R6eDhaZThFbVBlN2p6QWpJdllHL2F4Z3VseUZ6NU1BRTU5Y1crM2lLRVY0UzdSZkpwYkt2SGFLZUQrYVFvaFE0VjZmZlIrNk5BPT0tLUpLTHQvRHQ0T3FycWV0ZFZhVHZWZkE9PQ%3D%3D--4ef4508c936004db748da10be58731049fa190ee\r\n
Connection: keep-alive\r\n
Upgrade-Insecure-Requests: 1\r\n
Referer: http://localhost:5678/posts/new\r\n
Content-Length: 369\r\n
\r\n
utf8=%E2%9C%93&authenticity_token=3fu7e8v70K0h9o%2FGNiXxaXSVg3nZ%2FuoL60nlhssUEHpQRz%2BM4ZIHjQduQMexvXrNoC2pjmhNPI4xNNA0Qkh5Lg%3D%3D&post%5Btitle%5D=My+first+post&post%5Bcreated_at%281i%29%5D=2017&post%5Bcreated_at%282i%29%5D=1&post%5Bcreated_at%283i%29%5D=23&post%5Bcreated_at%284i%29%5D=18&post%5Bcreated_at%285i%29%5D=47&post%5Bbody%5D=It+works%21&commit=Create+Post

Much like a response, an HTTP request consists of:

After using session.gets to take the first line of the request (the Request-Line), we’re left with some header lines and a body. To get the header lines, we need to retrieve lines from the session until we find a newline (\r\n).

For each header line, we’ll split on the first colon. Everything before the colon is the key, and everything after is the value. We #strip the value to remove the newline from the end.

To know how many bytes we need to read from the request to get the body, we use the “Content-Length” header, which the browser automatically includes when sending a request.

1
2
3
4
5
6
7
8
9
10
# http_server.rb
# ...
  headers = {}
  while (line = session.gets) != "\r\n"
    key, value = line.split(':', 2)
    headers[key] = value.strip
  end

  body = session.read(headers["Content-Length"].to_i)
# ...

Now, instead of sending an empty object, we’ll send a StringIO instance with the body we received via the request. Also, since we’re now parsing the cookies from the request’s header, we can add them to the Rack environment in the HTTP_COOKIE variable to pass the request authenticity check.

1
2
3
4
5
6
7
8
9
10
11
12
# http_server.rb
# ...
  status, headers, body = app.call({
    # ...
    'REMOTE_ADDR' => '127.0.0.1',
    'HTTP_COOKIE' => headers['Cookie'],
    'rack.version' => [1,3],
    'rack.input' => StringIO.new(body),
    'rack.errors' => $stderr,
    # ...
  })
# ...

There we go. If we restart the server and try to submit the form again, you’ll see that we successfully created the first post on our blog!

Post was successfully created.

We seriously upgraded our web server this time. Instead of just accepting GET requests from a Rack app, we’re now serving a complete Rails app that handles POST requests. And we still haven’t written more than fifty lines of code in total!

If you want to play around with our new and improved server, here’s the code. Let us know at @AppSignal if you want to know more, or have a specific question.

Go back

Subscribe to

Ruby Magic

Magicians never share their secrets. But we do. Sign up for our Ruby Magic email series and receive deep insights about garbage collection, memory allocation, concurrency and much more.