When Local is Remote in Rails, and Other Tales of Eccentric Error Enforcement
(*Note*: This applies to Rails 1.1.6)
Yesterday I was adding custom error pages that can be rendered like any other page in our application, and came across some very odd (and imho broken) logic deep inside Rails. In today's "What's Really Reird with Rails", I'll show when a local connection is not a local connection and hope you can't always believe the documentation.
To set the stage, I first need to explain the theoretical difference between a local request and a remote (or public) request. In Rails, a local request gets special treatment on error. When your request is local, you will see handy dandy debug messages full of stack trace goodness. The idea here is that if you're a local request, you must be a developer and you would want to see just what caused the error.
In contrast, a public request is considered not coming from a developer, and a user friendly error page will be displayed instead of the debug stack traces.
Just what is considered a local request? Stay tuned, for the answer may surprise you.
Let's talk about the error pages themselves for a moment. In a local request, the debug error message is generated from a template found inside the Rails code itself. Specifically, the templates can be found in `vendor/rails/actionpack/lib/action_controller/templates/rescues`. When the controller encounters an error, and it considers the request a local request, it will reach into that directory for the template to render.
For those following along at home, this logic is in rescue_action_locally in `vendor/rails/actionpack/lib/action_controller/rescue.rb`.
However, when the request is considered public (not local), Rails by default will render the file `public/404.html` (when encountering a RoutingError or UnknownAction) or the message "<h1>Application error (Rails)</h1>" on any other error. These are static files, and Rails does not attempt to render them through the normal layout process or process them in any way. The assumption here is that if you have encountered an error, you should do the most simple thing and show a static file.
You can see this for yourself at `rescue_action_public` in `vendor/rails/actionpack/lib/action_controller/rescue.rb`.
Let's review before we get to the weirdness.
1. Rails makes a distinction between local request and public requests.
1. On error, two different things happen depending on if the request is local or public.
1. With a local request, Rails internal templates are rendered optimized for developers and displaying exception details.
1. With a public request, Rails renders either public/404.html or a static error message.
Now to the Business Problem:
The default logic for displaying errors to public requests is weak at best. Display rich error pages in production mode, rendered through the layout system so that error pages look like regular DSES pages.
OK, no problem. If you've been following along at home, you'll remember that there are two error handlers in rescue.rb: `rescue_action_locally` and `rescue_action_public`. It's the later method that we want to override in order to display rich error messages.
Why not override both? We still want to have local requests to display those handy debug error messages, so we want to leave rescue_action_locally alone.
So we've go ahead and implemented our own rescue_action_public in our application's application.rb. We restart the application, generate an error, but we continue to get the debug page instead of our pretty 404 page. What they diddly??!?!
Here's where it starts to get interesting, and the concept of a local request gets a bit fuzzy.
As I mentioned above, rails makes a distinction between local and public requests. If you are testing on your own machine, you are without a doubt generating a *local* request. Not many people will dispute that. But what if you want to see your public error pages even if you are testing on your local machine? Turns out you can do this, but you have to override more methods in your application.rb. More on this soon.
Some more background on how Rails tests if a connection is local or public:
There is a configuration option called `config.action_controller.consider_all_requests_local` which should be set to false in production mode. When set to false, it forces Rails to consider the actual IP address of the remote user when determining if a request is local or public. In development mode, this configuration option is set to true, which always forces all requests to be considered local (no matter what IP address the connection actually comes from).
Our production mode, camber_production, does have this config option set to false. I thought all I have to do to test my new error pages is use another other machine to connect to my instance and generate the error. Because they weren't local (that is, 127.0.0.1) they should see the public error messages. So I bounced my server into production mode and walked over to Wenyi's computer. When I generated the same error, I saw the developer local debug error message!?!!??!!!
So how could a connection from an external machine be treated as local, even when running in production mode with `config.action_controller.consider_all_requests_local` set to false? To find out, we need to dive deeper into Rails.
We've now reached the true mystery of this adventure: When is a Remote Request a Local Request?
To explain, let's begin by looking at rescue_action in rescue.rb. This method handles all exceptions from controllers and determines if `rescue_action_locally` or `rescue_action_public` should be called. Here's what it looks like:
def rescue_action(exception)
log_error(exception) if logger
erase_results if performed?
if consider_all_requests_local || local_request?
rescue_action_locally(exception)
else
rescue_action_in_public(exception)
end
end
Notice the check `consider_all_requests_local || local_request?`. We know we've set `consider_all_requests_local` to false, so we turn our attention to `local_request?` This method attempts to determine if the request is a local one. For some reason, this is returning true even when I use Wenyi's computer (which is clearly not local). So what's going on? Here's the logic:
def local_request? #:doc:
[@request.remote_addr, @request.remote_ip] == ["127.0.0.1"] * 2
end
Hmmm, OK. It delegates to `@request.remote_addr` and `@request.remote_ip`. Those can't possibly be equal to 127.0.0.1 when connecting from Wenyi's computer, so how come this method returns true?
We have to move to `@request.remote_ip`, which is found in `vendor/rails/actionpack/lib/action_controller/request.rb`. Here's the logic:
def remote_ip
return @env['HTTP_CLIENT_IP'] if @env.include? 'HTTP_CLIENT_IP'
if @env.include? 'HTTP_X_FORWARDED_FOR' then
remote_ips = @env['HTTP_X_FORWARDED_FOR'].split(',').reject do |ip|
ip =~ /^unknown$|^(10|172\.(1[6-9]|2[0-9]|30|31)|192\.168)\./i
end
return remote_ips.first.strip unless remote_ips.empty?
end
@env['REMOTE_ADDR']
end
So what's going on here? Well, correctly, this method attempts to look for the HTTP Header `HTTP_X_FORWARDED_FOR`, which is set by a forwarding proxy such as our Apache 2.2 proxy balancer. This header is set to the IP address of the original requesting computer (in this case, Wenyi's computer). If we didn't have this header, then all requests would have a remote IP address of the proxy server and that would muck things up. So, OK, +1 to Rails for checking the right header.
But look at what it does with that header. It rejects values that are equal to the known internal IP address ranges for internal, non-public addresses. Specifically, that's the 10.0.0.0, 172.16.0.0, and 192.168.0.0 networks. If the IP address of the remote connection is first hitting a forwarding proxy (as it is on my machine) and the remote connection is coming from one of those IP ranges (which it is here at Camber Hawaii) then the address is dropped!!! So it falls back to `@env['REMOTE_ADDR']` which, (wait for it......) because we are running behind a proxy on the same machine, will always be 127.0.0.1.
And because 127.0.0.1 is considered local (and rightly so), even when I run in production mode, and even when I use Wenyi's computer, my requests are considered local.
Holy Hack Batman.
As a kicker, let's look at the docs for the remote_ip method:
# Determine originating IP address. REMOTE_ADDR is the standard
# but will fail if the user is behind a proxy. HTTP_CLIENT_IP and/or
# HTTP_X_FORWARDED_FOR are set by proxies so check for these before
# falling back to REMOTE_ADDR. HTTP_X_FORWARDED_FOR may be a comma-
# delimited list in the case of multiple chained proxies; the first is
# the originating IP.
There's NOTHING about ripping out IPs that match non-public IP address ranges.
So, how do we force showing the public clean error page? We jump back to rescue.rb and we have to override `local_request?` in application.rb to always return `false`. The nice thing about this is if you are in development mode you'll still see the helpful debug error messages. However, once in production mode, you'll always see the pretty error message.
I definitely think that removing those IPs from `remote_ip` method is wrong.
I hope you've enjoyed this mini-tour of some Rails internals. Back to development!
Yesterday I was adding custom error pages that can be rendered like any other page in our application, and came across some very odd (and imho broken) logic deep inside Rails. In today's "What's Really Reird with Rails", I'll show when a local connection is not a local connection and hope you can't always believe the documentation.
To set the stage, I first need to explain the theoretical difference between a local request and a remote (or public) request. In Rails, a local request gets special treatment on error. When your request is local, you will see handy dandy debug messages full of stack trace goodness. The idea here is that if you're a local request, you must be a developer and you would want to see just what caused the error.
In contrast, a public request is considered not coming from a developer, and a user friendly error page will be displayed instead of the debug stack traces.
Just what is considered a local request? Stay tuned, for the answer may surprise you.
Let's talk about the error pages themselves for a moment. In a local request, the debug error message is generated from a template found inside the Rails code itself. Specifically, the templates can be found in `vendor/rails/actionpack/lib/action_controller/templates/rescues`. When the controller encounters an error, and it considers the request a local request, it will reach into that directory for the template to render.
For those following along at home, this logic is in rescue_action_locally in `vendor/rails/actionpack/lib/action_controller/rescue.rb`.
However, when the request is considered public (not local), Rails by default will render the file `public/404.html` (when encountering a RoutingError or UnknownAction) or the message "<h1>Application error (Rails)</h1>" on any other error. These are static files, and Rails does not attempt to render them through the normal layout process or process them in any way. The assumption here is that if you have encountered an error, you should do the most simple thing and show a static file.
You can see this for yourself at `rescue_action_public` in `vendor/rails/actionpack/lib/action_controller/rescue.rb`.
Let's review before we get to the weirdness.
1. Rails makes a distinction between local request and public requests.
1. On error, two different things happen depending on if the request is local or public.
1. With a local request, Rails internal templates are rendered optimized for developers and displaying exception details.
1. With a public request, Rails renders either public/404.html or a static error message.
Now to the Business Problem:
The default logic for displaying errors to public requests is weak at best. Display rich error pages in production mode, rendered through the layout system so that error pages look like regular DSES pages.
OK, no problem. If you've been following along at home, you'll remember that there are two error handlers in rescue.rb: `rescue_action_locally` and `rescue_action_public`. It's the later method that we want to override in order to display rich error messages.
Why not override both? We still want to have local requests to display those handy debug error messages, so we want to leave rescue_action_locally alone.
So we've go ahead and implemented our own rescue_action_public in our application's application.rb. We restart the application, generate an error, but we continue to get the debug page instead of our pretty 404 page. What they diddly??!?!
Here's where it starts to get interesting, and the concept of a local request gets a bit fuzzy.
As I mentioned above, rails makes a distinction between local and public requests. If you are testing on your own machine, you are without a doubt generating a *local* request. Not many people will dispute that. But what if you want to see your public error pages even if you are testing on your local machine? Turns out you can do this, but you have to override more methods in your application.rb. More on this soon.
Some more background on how Rails tests if a connection is local or public:
There is a configuration option called `config.action_controller.consider_all_requests_local` which should be set to false in production mode. When set to false, it forces Rails to consider the actual IP address of the remote user when determining if a request is local or public. In development mode, this configuration option is set to true, which always forces all requests to be considered local (no matter what IP address the connection actually comes from).
Our production mode, camber_production, does have this config option set to false. I thought all I have to do to test my new error pages is use another other machine to connect to my instance and generate the error. Because they weren't local (that is, 127.0.0.1) they should see the public error messages. So I bounced my server into production mode and walked over to Wenyi's computer. When I generated the same error, I saw the developer local debug error message!?!!??!!!
So how could a connection from an external machine be treated as local, even when running in production mode with `config.action_controller.consider_all_requests_local` set to false? To find out, we need to dive deeper into Rails.
We've now reached the true mystery of this adventure: When is a Remote Request a Local Request?
To explain, let's begin by looking at rescue_action in rescue.rb. This method handles all exceptions from controllers and determines if `rescue_action_locally` or `rescue_action_public` should be called. Here's what it looks like:
def rescue_action(exception)
log_error(exception) if logger
erase_results if performed?
if consider_all_requests_local || local_request?
rescue_action_locally(exception)
else
rescue_action_in_public(exception)
end
end
Notice the check `consider_all_requests_local || local_request?`. We know we've set `consider_all_requests_local` to false, so we turn our attention to `local_request?` This method attempts to determine if the request is a local one. For some reason, this is returning true even when I use Wenyi's computer (which is clearly not local). So what's going on? Here's the logic:
def local_request? #:doc:
[@request.remote_addr, @request.remote_ip] == ["127.0.0.1"] * 2
end
Hmmm, OK. It delegates to `@request.remote_addr` and `@request.remote_ip`. Those can't possibly be equal to 127.0.0.1 when connecting from Wenyi's computer, so how come this method returns true?
We have to move to `@request.remote_ip`, which is found in `vendor/rails/actionpack/lib/action_controller/request.rb`. Here's the logic:
def remote_ip
return @env['HTTP_CLIENT_IP'] if @env.include? 'HTTP_CLIENT_IP'
if @env.include? 'HTTP_X_FORWARDED_FOR' then
remote_ips = @env['HTTP_X_FORWARDED_FOR'].split(',').reject do |ip|
ip =~ /^unknown$|^(10|172\.(1[6-9]|2[0-9]|30|31)|192\.168)\./i
end
return remote_ips.first.strip unless remote_ips.empty?
end
@env['REMOTE_ADDR']
end
So what's going on here? Well, correctly, this method attempts to look for the HTTP Header `HTTP_X_FORWARDED_FOR`, which is set by a forwarding proxy such as our Apache 2.2 proxy balancer. This header is set to the IP address of the original requesting computer (in this case, Wenyi's computer). If we didn't have this header, then all requests would have a remote IP address of the proxy server and that would muck things up. So, OK, +1 to Rails for checking the right header.
But look at what it does with that header. It rejects values that are equal to the known internal IP address ranges for internal, non-public addresses. Specifically, that's the 10.0.0.0, 172.16.0.0, and 192.168.0.0 networks. If the IP address of the remote connection is first hitting a forwarding proxy (as it is on my machine) and the remote connection is coming from one of those IP ranges (which it is here at Camber Hawaii) then the address is dropped!!! So it falls back to `@env['REMOTE_ADDR']` which, (wait for it......) because we are running behind a proxy on the same machine, will always be 127.0.0.1.
And because 127.0.0.1 is considered local (and rightly so), even when I run in production mode, and even when I use Wenyi's computer, my requests are considered local.
Holy Hack Batman.
As a kicker, let's look at the docs for the remote_ip method:
# Determine originating IP address. REMOTE_ADDR is the standard
# but will fail if the user is behind a proxy. HTTP_CLIENT_IP and/or
# HTTP_X_FORWARDED_FOR are set by proxies so check for these before
# falling back to REMOTE_ADDR. HTTP_X_FORWARDED_FOR may be a comma-
# delimited list in the case of multiple chained proxies; the first is
# the originating IP.
There's NOTHING about ripping out IPs that match non-public IP address ranges.
So, how do we force showing the public clean error page? We jump back to rescue.rb and we have to override `local_request?` in application.rb to always return `false`. The nice thing about this is if you are in development mode you'll still see the helpful debug error messages. However, once in production mode, you'll always see the pretty error message.
I definitely think that removing those IPs from `remote_ip` method is wrong.
I hope you've enjoyed this mini-tour of some Rails internals. Back to development!