Categorizing OpenBSD Bugs
Microsoft recently gave a talk in which they claimed that 70% of their security-related bugs were memory-related: buffer overflows, use-after-free, and so on [1]. Mozilla have also reported similar figures: they claim that 74% of the security-related bugs in Firefox's CSS component would have been prevented if it had been written in Rust [2].
I thought it would be interesting to see if something similar were true of OpenBSD bugs. I went through two years of OpenBSD errata for the most recent four releases (6.1, 6.2, 6.3 and 6.4) and categorized each bug.
The 70% statistic from Microsoft's presentation refers to bugs that were fixed by security patch releases, so I think OpenBSD's published errata provide a fair comparison. However, they are different in two important respects:
Microsoft and Mozilla spoke only about security-related bugs, whereas OpenBSD's errata cover all serious bugs. OpenBSD categorize each patch as a "Security Fix", a "Reliability Fix" or an "Interoperability Fix". I calculated two sets of statistics: one including all OpenBSD errata and one including only those patches designated as a "Security Fix".
Microsoft and Mozilla spoke only about their own software, whereas an OpenBSD release includes a lot of third-party software. For example, a number of memory-related security flaws were found in Perl, which is not developed by OpenBSD volunteers. Including such bugs in an analysis of OpenBSD bugs seems a little unfair. Nevertheless, they are part of the base system so I have included them.
In my own categorizations, I have included the category "Logic". This is meant in the sense of "Business Logic" or "Application Logic". This is the sort of bug that would be present in pseudo-code.
Finally, note that I am not an OpenBSD developer. I am not familiar with this code: my categorizations are based solely on the content of the patches. It is quite possible that I have mischaracterized some of them: if you spot a mistake, please let me know!
Patch | OpenBSD Category | My Category |
---|---|---|
6.1 001 | Interoperability | Logic |
6.1 002 | Security | Logic |
6.1 003 | Security | Logic |
6.1 004 | Reliability | Logic |
6.1 005 | Reliability | Memory leak |
6.1 006 | Reliability | Null pointer dereference |
6.1 007 | Security | Heap buffer overflow |
6.1 008 | Security | Logic |
6.1 009 | Reliability | Memory leak |
6.1 010 | Security | Inter-process race condition |
6.1 011 | Security | Integer overflow and buffer overflow |
6.1 012 | Reliability | Logic |
6.1 013 | Reliability | Logic |
6.1 014 | Reliability | Logic |
6.1 015 | Reliability | Use after free |
6.1 016 | Reliability | Logic |
6.1 017 | Security | Kernel buffer overflow |
6.1 018 | Security | Logic |
6.1 019 | Security | Logic |
6.1 020 | Security | Buffer overflow |
6.1 021 | Security | Uninitialized value used |
6.1 022 | Security | Uninitialized value used |
6.1 023 | Reliability | Integer overflow and buffer overflow |
6.1 024 | Security | Race condition |
6.1 025 | Security | Kernel out-of-bounds read; reuse of unzeroed buffer |
6.1 026 | Security | Logic |
6.1 027 | Security | Logic |
6.1 028 | Security | Out-of-bounds read and heap overflow |
6.1 029 | Reliability | Logic |
6.1 030 | Reliability | Logic |
6.1 031 | Security | Logic |
6.2 001 | Reliability | Logic |
6.2 002 | Security | Logic |
6.2 003 | Reliability | Kernel buffer overflow |
6.2 004 | Reliability | Logic |
6.2 005 | Reliability | Kernel buffer overflow |
6.2 006 | Reliability | Kernel buffer overflow |
6.2 007 | Security | Double free |
6.2 008 | Security | Logic |
6.2 009 | Security | Meltdown mitigation |
6.2 010 | Reliability | Kernel buffer overflow |
6.3 001 | Security | Heap buffer overflow |
6.3 002 | Reliability | Logic |
6.3 003 | Reliability | Logic |
6.3 004 | Security | Memory leak and use after free |
6.3 005 | Reliability | File descriptor leak |
6.3 006 | Reliability | Kernel buffer overflow |
6.3 007 | Reliability | Logic |
6.3 008 | Reliability | Kernel buffer overflow |
6.3 009 | Security | Timing side-channel attack |
6.3 010 | Security | Intel FPU speculation attack |
6.3 011 | Security | Logic |
6.3 012 | Reliability | Kernel buffer overflow |
6.3 013 | Reliability | Logic |
6.3 014 | Security | AMD Spectre mitigation |
6.3 015 | Security | Logic |
6.3 016 | Reliability | Logic |
6.3 017 | Security | Logic |
6.3 018 | Security | Intel L1TF bug mitigation |
6.3 019 | Security | Logic |
6.3 022 | Security | Timing side-channel attack |
6.4 001 | Security | Logic |
6.4 002 | Reliability | Logic |
6.4 003 | Security | Intel Hyper-Threading side-channel attack |
6.4 004 | Reliability | Logic |
6.4 005 | Security | Integer overflow and heap buffer overflow |
6.4 006 | Reliability | Logic |
6.4 007 | Reliability | Logic |
6.4 008 | Reliability | Integer overflow |
6.4 009 | Reliability | Logic |
6.4 010 | Security | Kernel buffer overflow |
6.4 011 | Security | Logic |
6.4 012 | Reliability | Buffer overflow |
6.4 013 | Security | Memory leak |
6.4 014 | Security | Logic |
6.4 015 | Security | Logic |
6.4 016 | Security | Logic |
For all the errata, that produces the following statistics:
Category | No. | Percentage |
---|---|---|
Total | 77 | 100% |
Memory | 25 | 32% |
Logic | 39 | 51% |
And for only those that OpenBSD categorized as security fixes:
Category | No. | Percentage |
---|---|---|
Total | 42 | 100% |
Memory | 12 | 29% |
Logic | 19 | 45% |
So roughly a third of all serious bugs in OpenBSD are caused by unsafe memory use. This is a significant fraction... but less than half of the fraction found by Microsoft and Mozilla.
This analysis isn't really large, detailed or rigorous enough to draw any definite conclusions. However, it suggests that OpenBSD's development process may be better at mitigating the dangers of using memory-unsafe languages than either Microsoft's or Mozilla's. Or, at the very least, it suggests that we should be wary of assuming that one organization's bug analysis applies equally well to other organizations.
It also shows that while memory safety is a significant cause of security-related bugs in OpenBSD, the majority of the bugs are not caused by unsafe memory use. Most engineering decisions are trade-offs: when considering any new technology intended to improve memory safety, the OpenBSD developers should first make sure that it does not make other, more significant, categories of bug more likely.