Heisenbug: Ruby on Rails Corrupted/Broken POST data on a big form, it's all Rack's fault

We have been chasing a bug for quite a long time that made our lives miserable, and our client, very unhappy: When submitting a large form, intermittently the post data is broken (basically only the the first parts of the submitted form data get to Rails). The bug is caused by an incorrect implementation of parsing multipart/form-data in Rack 1.0 and earlier. This is the commit that fixes the issue : you can also look at the added test case to try and reproduce the error.

We are without doubt not the first ones to fall on this, but as finding the bug was very, very hard. I am posting this as a public service hoping someone down the line will have an easier time finding the cause… the solution is easy. Either upgrade Rack to at least 1.1 or patch lib/rack/utils.rb. Also updating Ruby on Rails to > 2.3.6 will solve this as it has a dependency on Rack 1.1

The symptoms can vary, and what makes this bug very very hard to find is that it needs a lot of parameters to manifest itself. You might get validation errors on you models, or missing required parameters at the controller level: or just some unexpected behavior (Not tested, but might result in corrupted uploads). Basically what happened is that when the body data of a post is longer then 16384 and a multipart boundary is found at precisely 16384 bytes… the parser will return with therefor only the first parts parsed. So first, you need to have a huge form, now imagine the form represents something in your database, and you change the data: you need your new data to make the boundary arrive at precisely the slicing point. Considering that each browser may have a different size of boundary, the same change to the same data will not reproduce the error between two different browsers. Also because this executes inside the Application Controller at Racks level, you will find that the app's logs are going to be of very little service. We had to go wireshark on this bug, and follow the post data through all the levels to find where it broke. And we got lucky: we found offending POST data while testing… So basically you might find yourself with an application where from time to time, in what seems to be a completely random way, you will find fields missing from the POST data in production… but reproducing this on test servers might prove very very hard.

Note: this bug is not Rails specific and will happen with any framework using Rack. So just to help people find this: Broken form data on Sinatra, ruby corrupted post, big form fails intermittently with rack. Uploading Files randomly fails Ruby on Rails. Missing Post Data Ruby. Ruby multipart bug. Rails multipart problem.

blog comments powered by Disqus