Apache, fastcgi, proxy_fcgi, and empty POST bodies with chunked transfer

I've been working on building a reproducible configuration for Drupal Photo Gallery, a project born out of this year's Acquia Build Hackathon.

We originally built the site on an Acquia Cloud CD environment, and this environment uses a pretty traditional LAMP stack. We didn't encounter any difficulty using AWS Lambda to post image data back to the Drupal site via Drupal's RESTful Web Services API.

The POST request is built in Node.js using:

var reqPost = http.request(options_post, function(res) {
    res.on('data', function (chunk) {
        body += chunk;
    });
});
reqPost.write(JSON.stringify(jsonObject));
reqPost.end();

The way this kind of HTTP request works is using Transfer-Encoding: chunked. Chunked encoding is used in many cases when you need to send potentially large request bodies, and don't necessarily know the full size beforehand (e.g. you can't provide a Content-Length in the request).

Apache handles non-chunked transfers without issue, but in many cases, it seems Apache can't correctly pass on the request body if using Transfer-Encoding: chunked. Specifically, using PHP-FPM as the backend, I could not get the request body using either:

I was using the test PHP script and cURL command listed in this comment to test—basically, make a POST or PUT request with chunked encoding, and see what PHP gets via file_get_contents('php://input').

To verify the server itself (pre-Apache) was receiving the request body, I used sudo ngrep "POST" tcp and port 80 to monitor incoming POST requests... and it indicated the data was definitely reaching the server (ruling out any AWS, DigitalOcean, or on-server firewall or proxy eating the data):

$ sudo ngrep "POST" tcp and port 80
interface: eth0 (165.227.96.0/255.255.240.0)
filter: (ip or ip6) and ( tcp and port 80 )
match: POST
####
T 54.237.220.121:40842 -> 165.227.102.121:80 [AP]
  POST /test.php HTTP/1.1..Content-Type: application/json..Host: gallery.jeffgeerling.com..Connection:
  close..Transfer-Encoding: chunked....2d..{"Bucket":"test-bucket","Name":"Test Object"}..
#######

So then I decided maybe something in Apache was eating the data. I had been using proxy_fcgi to route requests to PHP using CGI, as it seems to be the most modern and efficient approach when running Apache 2.4 and later, but I realized maybe that mod was breaking things.

So I switched to the older (but often-used) Apache fastcgi mod and configured it to route to PHP... and instead of having the POST body completely disappear, it gave a warning (so at least that was more helpful...):

<title>411 Length Required</title>
</head><body>
<h1>Length Required</h1>
<p>A request of the requested method POST requires a valid Content-length.<br />
</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at gallery.jeffgeerling.com Port 80</address>

So finally, I decided to switch to the old but reliable mod_php, which runs PHP inside Apache's process forks. And what do you know? The chunked request body arrived intact, and my problem was solved!

I'm guessing I could've also just switched to Nginx, which doesn't seem to suffer from the same CGI bug when proxying requests to PHP-FPM, but in this particular case, I wanted to make sure I was coming closer to replicating the earliest test environment we were using on Acquia Cloud.

During the course of my research on this issue, I ran into a few bug reports that seem slightly circular in nature:

The PHP bug comments are correct—it's not an issue with PHP, since PHP-FPM seems to handle things correctly with other web frontends (Nginx, maybe others?). But it's not encouraging seeing the Apache issue has been open since 2012, and even has a patch, but has zero reviews and only one comment in the past 5 years ?.

So the takeaway? If you need to handle chunked content transfer, don't rely on Apache with mod_proxy_fcgi or mod_fastcgi!

For more background and debugging on the issue, check out the original GitHub issue after which this blog post was written: Requests from Lambda logging "POST body is: null".

Further reading

apache
transfer
chunked
http
fastcgi
proxy
php
post
drupal
open source
drupal planet

Comments

Hey Jeff, thanks for the detailed notes with sample codes. We hit this bug at work and your post saved us a lot of time figuring out what was wrong.

BTW The referenced Apache bug is marked as fixed as of March 2019.