My curiosity grow stronger, and made me check out nginx. The fact that I have an Apache already running with its slowly-grown configuration made this a real fun task (in the nerd sense), as introducing nginx as an additional component into the system does not look as an obvious improvement.
After a few days, it seems it is. Apache is still used (mainly because of zend optimizer, see later), but the box looks more healthy now. There are still real pikes in the load, but the CPUs are at least twice less loaded (the machine runs at half to 1/10 of the load average than before). One of the reasons is obviously that now Apache does not need to care about the static files with all its infrastructure built around running PHP stuff. A bit less apparent reason is that even when serving dynamic pages, Apache does not need to linger around, waiting for the clients to fetch the last bytes of the data. All the data produced by the dynamic pages are quickly passed to nginx, which will care about all those pesky clients with their TCP bugs with its few processes in an event loop optimized for Linux 2.6 features.
But the feature that made me stop in awe was the impactless software upgrade (see also here). When you send an nginx master process the USR2 signal, it will fork, and start another copy of the executable (mind you, which may be a new version of nginx). The appropriate socket handles are passed to the new process, so it can start serving immediately. In fact, you can keep two versions of nginx serving your site(s), though I can't see how this is really an advantage :) The usual process is that you then send a WINCH signal to the old nginx (change the window over to the new process, eh?), then only the new nginx is doing the job, but the old one is still hanging around just in case. If you realize running a new nginx was a mistake, just HUP the old one, and it will be brought to live (you still need to get rid of the new one, of course).
You can do a few more things to the processes. There's usually a "master" process, which has its pid written to the usual locations, and one or more "worker" processes (as you configured), running as the specified unprivileged users. Usually, you only need to signal the master processes, unless some of the workers smell weird (didn't happen yet). Masters respond to WINCH by (gracefully) killing their worker child processes. You can also HUP them to reread the configuration, and spawning new workers. This is also why it's not a good idea to change config and try to upgrade at the same time -- it never is if you think about it... QUIT is the signal to send if you want a master to stop its workers serving and quitting if all of them is finished -- this is also the final step when you are happy with the upgrade.
During the whole process, you can see the master process in the usual pid file, say /var/run/nginx.pid, and the pid of the planned master process in PIDFILE.oldbin (/var/run/nginx.pid.oldbin), so it's very easy to script for it. Now that I think of it, I'd refrain from trying to run three masters because of this :)
This impactless upgrade also needs to exec the old pathname, so if you just run "nginx" without the full path, it won't work. But you aren't out of luck either. Just find where it wants to exec the new binary, chances are it isn't in the PATH, but the docroot of the first site, and put a simple script there which will exec /usr/sbin/nginx, and your just as good as if you started it that way. [hackety-hack]
I also happen to like the configuration syntax, just about comfortably C like. Not too many $s peppered around, and nothing resembling XML (<IfModule> anyone?). Still there's a few roadblock to make a system multi-tiered, leaving the one Apache way. You need to think about the IP addresses used for each server -- I moved Apache to 127.0.0.1:80, and it still thinks it's serving the main site. Because now every client to Apache is localhost, you might want to fake the real IP address to some scripts, for example with mod_rpaf. When you want to use the IP addresses for cookie access control, this is the easiest way, and the only one if you don't have access to the PHP source...
So, why did not I get rid of Apache already? Well, php is the reason. The situation isn't really grave, nginx apparently has a promising FastCGI interface (thought less featureful than when proxying simple HTTP), and php can run as a FastCGI server, so it can usually work, except if you must run Zend Optimized scripts. Apparently, php either runs as a FastCGI server (-b PORT option), or loads the Zend Optimizer modules (-z MODULE), which looks kinda silly, but may even be a feature as far as I know... So Apache will hum for a while in my server, but at a lower pitch.
I have plenty of config examples now :)