Given the fact that security bugs are critical, one of the basic pursuits in every new software release should be to mitigate such bugs. In essence, security bugs should decrease as a project evolves. In a previous post I described how I measured the occurrence of security bugs through time and observed that security bugs actually increase as projects evolve. However, the corresponding experiment involved only four projects.

To examine more thoroughly the aforementioned hypothesis, together with four other colleagues (Vasilios Karakoidas, Georgios Gousios, Panos Louridas and Diomidis Spinellis), we performed a similar experiment that involved 17,505 projects coming from the Maven central repository. To achieve this, we ran FindBugs on all the project versions (115,214 JARs) of all the Java projects that exist in the repository to identify all bugs contained in it. In another post I presented how bugs coming from different categories (performance, style etc.) were distributed in this repository. In this post I further elaborate on the evolution of security bugs through time.

The relation between bugs and time can be traced from the number of bugs per category in each project version. Hence, we calculated the Spearman correlations between the defects count and the ordinal version number across all projects to see if bigger versions relate to higher or lower defect counts. The results are shown in Table 1. Although the tendency is for bugs (coming from all categories) to increase, this tendency is extremely slight.

Table 1: Correlations between version and defects count.

The zero tendency applies to all versions of all projects together. The situation might be different in individual projects. We therefore performed Spearman correlations between bug counts and version ordinals in all projects we examined. These paint a different picture from the above table, shown in Figure 1. Looking closer, that spike at point zero is explained by the large number of projects in which no correlation could be established (note that the scale is logarithmic). Still, we can see that there were projects where a correlation could be established, either positive or negative. The Security and the Malicious Code categories are particularly bimodal. Overall, Table 1 and Figure 1 suggest that we cannot say that across projects security bugs increase or decrease significantly across time. In individual projects, however, defect counts can have a strong upwards or downwards tendency. Thus we can observe projects that have their own idiosyncrasies, quality features, and coding practices.

Figure 1: Histograms of correlations between bug counts and version ordinals per project. In brackets the total population size and the number of no correlation instances.

Ozment and Schechter examined the code base of the OpenBSD operating system to determine whether the software vulnerabilities it contains are increasing over time. In particular, they measured the rate at which new code has been introduced and the rate at which defects have been reported over a 7.5-year period and 15 releases. Even though the authors present statistical evidence of a decrease in the rate at which vulnerabilities are being reported, defects seem to appear persistent even for a period of 2.6 years. Massacci et al. observed the evolution of software defects by examining six major versions of Firefox. To achieve this they created a database schema that contained information coming from the “Mozilla Firefox-related Security Advisories” (MFSA) list, Bugzilla entries, and others. Their findings indicate that security bugs are persistent over time. Edwards et al. have analyzed four large-scale projects, and their results demonstrate that the number of exploitable bugs does not always improve with each new release. They also found that the rate of discovery of exploitable bugs begins to drop three to five years after the initial release.

Interestingly, none of the above findings (including ours) clearly indicates that security bugs decrease as projects evolve. Hence, such bugs and the attacks stemming from them, appears to be an issue that will continue to draw the attention of researchers and practitioners alike.