Written on: 2019-04-22

Categorizing OpenBSD Bugs

Microsoft recently gave a talk in which they claimed that 70% of their security-related bugs were memory-related: buffer overflows, use-after-free, and so on [1]. Mozilla have also reported similar figures: they claim that 74% of the security-related bugs in Firefox's CSS component would have been prevented if it had been written in Rust [2].

I thought it would be interesting to see if something similar were true of OpenBSD bugs. I went through two years of OpenBSD errata for the most recent four releases (6.1, 6.2, 6.3 and 6.4) and categorized each bug.

The 70% statistic from Microsoft's presentation refers to bugs that were fixed by security patch releases, so I think OpenBSD's published errata provide a fair comparison. However, they are different in two important respects:

In my own categorizations, I have included the category "Logic". This is meant in the sense of "Business Logic" or "Application Logic". This is the sort of bug that would be present in pseudo-code.

Finally, note that I am not an OpenBSD developer. I am not familiar with this code: my categorizations are based solely on the content of the patches. It is quite possible that I have mischaracterized some of them: if you spot a mistake, please let me know!

Patch OpenBSD Category My Category
6.1 001 Interoperability Logic
6.1 002 Security Logic
6.1 003 Security Logic
6.1 004 Reliability Logic
6.1 005 Reliability Memory leak
6.1 006 Reliability Null pointer dereference
6.1 007 Security Heap buffer overflow
6.1 008 Security Logic
6.1 009 Reliability Memory leak
6.1 010 Security Inter-process race condition
6.1 011 Security Integer overflow and buffer overflow
6.1 012 Reliability Logic
6.1 013 Reliability Logic
6.1 014 Reliability Logic
6.1 015 Reliability Use after free
6.1 016 Reliability Logic
6.1 017 Security Kernel buffer overflow
6.1 018 Security Logic
6.1 019 Security Logic
6.1 020 Security Buffer overflow
6.1 021 Security Uninitialized value used
6.1 022 Security Uninitialized value used
6.1 023 Reliability Integer overflow and buffer overflow
6.1 024 Security Race condition
6.1 025 Security Kernel out-of-bounds read; reuse of unzeroed buffer
6.1 026 Security Logic
6.1 027 Security Logic
6.1 028 Security Out-of-bounds read and heap overflow
6.1 029 Reliability Logic
6.1 030 Reliability Logic
6.1 031 Security Logic
6.2 001 Reliability Logic
6.2 002 Security Logic
6.2 003 Reliability Kernel buffer overflow
6.2 004 Reliability Logic
6.2 005 Reliability Kernel buffer overflow
6.2 006 Reliability Kernel buffer overflow
6.2 007 Security Double free
6.2 008 Security Logic
6.2 009 Security Meltdown mitigation
6.2 010 Reliability Kernel buffer overflow
6.3 001 Security Heap buffer overflow
6.3 002 Reliability Logic
6.3 003 Reliability Logic
6.3 004 Security Memory leak and use after free
6.3 005 Reliability File descriptor leak
6.3 006 Reliability Kernel buffer overflow
6.3 007 Reliability Logic
6.3 008 Reliability Kernel buffer overflow
6.3 009 Security Timing side-channel attack
6.3 010 Security Intel FPU speculation attack
6.3 011 Security Logic
6.3 012 Reliability Kernel buffer overflow
6.3 013 Reliability Logic
6.3 014 Security AMD Spectre mitigation
6.3 015 Security Logic
6.3 016 Reliability Logic
6.3 017 Security Logic
6.3 018 Security Intel L1TF bug mitigation
6.3 019 Security Logic
6.3 022 Security Timing side-channel attack
6.4 001 Security Logic
6.4 002 Reliability Logic
6.4 003 Security Intel Hyper-Threading side-channel attack
6.4 004 Reliability Logic
6.4 005 Security Integer overflow and heap buffer overflow
6.4 006 Reliability Logic
6.4 007 Reliability Logic
6.4 008 Reliability Integer overflow
6.4 009 Reliability Logic
6.4 010 Security Kernel buffer overflow
6.4 011 Security Logic
6.4 012 Reliability Buffer overflow
6.4 013 Security Memory leak
6.4 014 Security Logic
6.4 015 Security Logic
6.4 016 Security Logic

For all the errata, that produces the following statistics:

Category No. Percentage
Total 77 100%
Memory 25 32%
Logic 39 51%

And for only those that OpenBSD categorized as security fixes:

Category No. Percentage
Total 42 100%
Memory 12 29%
Logic 19 45%

So roughly a third of all serious bugs in OpenBSD are caused by unsafe memory use. This is a significant fraction... but less than half of the fraction found by Microsoft and Mozilla.

This analysis isn't really large, detailed or rigorous enough to draw any definite conclusions. However, it suggests that OpenBSD's development process may be better at mitigating the dangers of using memory-unsafe languages than either Microsoft's or Mozilla's. Or, at the very least, it suggests that we should be wary of assuming that one organization's bug analysis applies equally well to other organizations.

It also shows that while memory safety is a significant cause of security-related bugs in OpenBSD, the majority of the bugs are not caused by unsafe memory use. Most engineering decisions are trade-offs: when considering any new technology intended to improve memory safety, the OpenBSD developers should first make sure that it does not make other, more significant, categories of bug more likely.