The threats we need to protect against are:
By exploiting a remote vulnerability in httpd, an attacker could gain control of the apache account. A similar attack might take place against any other service running on the machine, possibly resulting in immediate root access if the other service runs as root.
Such an attack would likely disrupt the functioning of the entire service while the vulnerability responsible for the attack is uncovered and the machine is reinstalled. User data could be lost or corrupted and might need to be restored from backup, although the Apache account should not have access to most user content.
Mitigation: run as few services on the machine besides httpd as possible. Be alert to publically known httpd vulnerabilities and patch them as quickly as possible.
Web applications are generally implemented in PHP or another scripting language and are not usually subject to buffer overflow attacks. Nevertheless, they have traditionally contained a rich variety of vulnerabilities, generally of one of the following forms:
Moreover, these vulnerabilities are the subject of widespread and frequent exploits. Mitigation of this risk will come in two forms:
This is a complicated topic and is best considered in pieces. This is also an area where we are in conflict with the world of web hosting at large. Most commercial web services do not insulate users from one another, and most web application installation instructions only concern themselves with instructing users how to break down what barriers exist, not how to work within them.
Public static content (e.g HTML files) is not an issue as long as it is only writable by users other than the owner. However, some number of users will naively set their static content to be group or other-writable. It may be necessary to protect such users against themselves.
Private static content (anything protected by a .htaccess file) must be readable by the apache user if httpd is going to be able to serve it normally. This means making those files world-readable, which is an issue if it allows other users to read the content without going through httpd access controls. If all users are placed within the same group, the files can be protected against other users by making them not group-readable--assuming users have no ability to read the content from the apache account using dynamic content features (see below). Again, it may be necessary to protect users against themselves.
Recommendations:
SIPB's scripts.mit.edu service has, for various reasons, opted to make static content read by the uid of the content owner, rather than by the apache accounts. This decision comes at a hefty penalty in performance and complexity, and should not be necessary in our environment as long as we adhere to the above principles.
It may be possible to use SELinux to more robustly isolate user accounts from one another. More research is required.
Dynamic content features in a web hosting environment include CGI or FastCGI scripts, scripting modules (mod_php, mod_perl, etc.), and server-side includes. Most of these mechanisms are not limited in power; they allow dynamic content to take arbitrary actions including executing shell commands. mod_php can be configured to be limited in power, but not in a robust manner, and generally only at the expense of compatibility with pre-existing web applications.
It is traditional in web hosting services to execute dynamic content as the apache account, possibly within the httpd worker process itself. This practice is grossly insecure; although the apache account may be unable to modify any system files, all user dynamic content becomes undifferentiated in terms of what actions it can take. If my dynamic content can read my database password from a file in my home directory, your dynamic content can do the same. The commercial web hosting provider I evaluated provided the option of executing dynamic content as your own uid (using cgiwrap) so that you could protect files from other users, but none of the cookie-cutter installation scripts they provided made use of this feature.
<incomplete>