My Take on Georgi Guninski's qmail Security Advisories

On his web site, Georgi Guninski has published two bugs he found in qmail, a product about the use of which I have my own reservations, apparently by researching the source code and/or experimenting with it:

In The qmail security guarantee, D. J. Bernstein denies that Guninski found (as of May 2005) an exploitable bug in qmail-smtpd. I don't disagree with that specific denial, but Bernstein has not yet addressed the issue with qmail-popup, and I'm not sure I can agree with the basis Bernstein has claimed for the denial.

Further, using the language of his own web page, I believe qmail is "guilty" of violating Bernstein's own rule #7, "Write bug-free code". The particular bug I'm looking at, as a result of Guninski's work, is a classic example of just how easy it is to write buggy code in a bug-inviting language such as C.

The bugs Guninski has found appear, to me, to be genuine bugs. In fact, I published a simple fix to the latter problem (with invalid array access) shortly after he published his discovery, and this fix was incorporated into a widely-used patchset for qmail.

As of 2005-05-31, following a recent update by Guninski of his own web page on the former problem, I have personally researched at least one of the more-recent bugs (of the former type).

It seems that there is indeed at least one exploitable bug in qmail-popup, a "trusted" component within the qmail architecture, in that it runs as "root".

Further, based on the location of this one bug, I cannot as yet rule out the potential that other exploitable forms of this bug exist in other qmail components that are "trusted".

Important: Guninski's own page on the "dereferencing" bug, as well as most online discussion, depicts 64-bit systems (or any systems where sizeof(int) is less than sizeof(char *)) as the only systems of concern. My own understanding of one of the bugs Guninski has found strongly suggests that even ordinary 32-bit CPUs, such as those based on AMD's or Intel's 32-bit processors, could theoretically be affected, though probably only while running an operating system employing a segmented-addressing model (and, therefore, on which pointers are indeed larger in size than unsigned int variables).

Advice: Pending a patch for this bug, either disable incoming POP3 service, or ensure that it runs with an operating-system-imposed limit on allocatable (heap) memory of less — ideally, much less, for reasons outlined by Bernstein himself — than 1GB. (Though I haven't tried this myself, test it out using one of Guninski's own test programs, qpopup2.pl. Apparently, some systems, such as Apple's OS X, do not effectively impose such limits?)

Note that, upon reviewing my own installation for qmail, though I did not recall doing this, I'd already imposed a suitable limit on each qmail-popup process run by my tcpserver process listening on port 110. That's probably because "best practices" for installing qmail have long specified imposing such limits. But since my server is a plain-vanilla Intel Pentium II running GNU/Linux, a 32-bit operating system, it probably couldn't be exploited via this or any of the other Guninski bugs.

Therefore, Bernstein's basis for denying Guninski's claim regarding qmail-smtpd — "Nobody gives gigabytes of memory to each qmail-smtpd process" — probably will be applied by him to this concern regarding qmail-popup. I hesitate to assume this, because a "root" exploit is something quite different from a bug in an ordinary, untrusted program. I don't believe it's acceptable to rely on operating-system-imposed limits on memory utilization to prevent outright coding errors from potentially granting external users "root" access, especially since Bernstein himself discusses the necessity of imposing such limits in the context of limiting DOS and dDOS attacks. Bernstein does not, to my knowledge, specify that the qmail code assumes a maximum useable VM space of (substantially?) less than the number of bytes uniquely addressable via the unsigned int (or int) type provided by the C compiler used to compile qmail, in order to be sure that overflows do not allow an untrusted network client to inject and possibly execute (using the privileges of the server process) arbitrary code.

In particular, I'm not sure I'd consider the failure of an operating system to assuredly terminate execution of a running program once it tried to use more than the limit on virtual memory set by its calling program to be an actual bug in that operating system, assuming the system would otherwise keep running. (For example, a hypothetical system might have 128TB of virtual memory and 1TB of RAM, and thus be hardly discomfited by a few POP3 clients trying to DOS it by transmitting only a few GB. The C library's memory allocation mechanism might be insufficiently precise to allow the OS to efficiently detect exactly when it overflows the virtual-memory limitation imposed on qmail-popup in such a scenario. It could potentially request VM from the OS in 1GB increments!)

Any delay (in terms of qmail-popup continuing to run after reaching the imposed limit on virtual memory) could result in an attack, using the vector Guninski has exposed, being successful before the OS notices that the limit has been reached and terminates the process.

Regardless, I'm honoring Guninski's apparent choice to be reticent about the precise nature of the bug(s) he's found, because I don't want to be the one who exposes exactly how to exploit any qmail bugs until and unless I have (or someone else has) published what I believe would be a highly reliable patch to qmail to eliminate the bugs.

Prior to Guninski's web page on the "dereferencing" problem being recently updated, I had concerns with his reporting of these bugs as "exploits".

In particular, bugs in qmail-smtpd and qmail-pop3 are, by design, confined to being non-exploitable insofar as they run with no particularly interesting privileges.

That is, qmail-smtpd and qmail-pop3 code are designated as "untrusted" programs within the qmail architecture. They don't "own" or communicate via any external network sockets; they don't have write privileges to most of the file system; they can't even read qmail's own internal queue, much less muck with it; and they certainly aren't run as "root" when installed as designed, so even if they could be convinced, via such bugs, to attempt to gain the privileges of another user, the operating system would presumably refuse to allow that.

(Though qmail-pop3 presumably has some degree of privilege in order to do its work, in a typical qmail installation, it can only obtain that as a result of a valid username and password having been transmitted. Thus, only someone who can log in as a given user can "exploit" a bug in qmail-pop3, and can in fact only play with his own files — his own Maildir — or do whatever else he could do by just logging in via telnet or similar.)

But qmail-popup is another story, since it runs as root, and at least one of the bugs is in low-level "library" code used throughout qmail, so it might be exploitable within the context of other "trusted" components of qmail.

(It is not yet clear to me just how an attacker might exploit the bug I'm looking at. It appears that it is at least possible to place an arbitrary string of bytes into memory, outside the bounds of a pointer to a chunk of heap memory.)

My tentative conclusion is that Guninski probably deserves the $500 promised via qmail's "security guarantee", though not based solely on the claim in that message he posted.

But, since it's not my $500, that's easy for me to say, and it's not yet clear to me that this bug can in fact be exploited "to take over another account", though I personally wouldn't make qmail-popup, at least, accessible to the Internet at large without first fixing all the bugs Guninski has found (which would be fairly straightforward for me, given my experience working on the internals of qmail and similar software).

My reasoning here is that Bernstein's guarantee sensibly rules out exploits that result from inherent flaws in the compiler, libraries, or operating system, but not from choices made by those components with regard to how memory is laid out or used. So if those choices, combined with a bug in a trusted component such as qmail-popup, could even theoretically allow an attacker to gain "root" access, as appears to be the case here, then the flaw is, indeed, in qmail itself.

Regardless of whether qmail will have been demonstrated to have an actual security flaw in any reasonable, deployed configuration, it is a testament to Bernstein's design principles (if not his personal C coding skills) that only after many years of being widely deployed, without any new official version being released, was any such flaw discovered.

The lesson I take away from this is mostly to add, to Bernstein's own list of rules, "Don't code in bug-inviting languages such as C". C can be a fine "write-only" language — that is, useful as a language for code output by a processor/compiler for a higher-level language — but, even after ripping out many of the evident vulnerabilities such as the libraries, as Bernstein has done, it remains bug-inviting.

Other "solutions" widely promoted online include "(re)code qmail in full ISO C, using prototypes, size_t, and so on" and "fix/improve the qmail license". Based on the nature of the bug in question, I don't believe those "solutions" would have helped as much as not having coded in C in the first place, especially if on-the-fly rather than buffer-oriented parsing had been employed throughout the design and implementation of qmail.

Back to my qmail page.