Tuesday, 3 January 2012

The Business v Security Bugs – Risk Management of Software Security Vulnerabilities by ISVs

The title might sound like a court case, and in reality security bugs like all defects, are an annoyance that the business has to deal with on an increasingly regular basis that may lead to it feeling like one. We say annoyance as security bugs are exactly that. Without their existence life would be simpler. When feature complete, functional bugs may exist, but the business can ship product safe in the knowledge that if the bugs are stumbled upon they won’t likely be splashed all over the BBC, The Register, H-Online or CERTs front page and their key customers won’t be phoning for a chat.

We think it’s safe to say that software security vulnerabilities (and likely hardware too) are going to be a fact of life for at least our careers (and we’re being optimistic). This statement is based on the fact they’ve been around for over forty years already and there are no signs of the volume or diversity slowing down. This leads us to an on-going decision battle that all businesses who write software have to wrestle with when shipment time nears - ‘Should we stop the release for a fix?’ In reality the question when asked explodes into ‘Should we stop the release to do root cause analysis, development, testing, regression testing, packaging and then release?’.

The modern reality is that this needs to be a risk management decision. On one side you have the business and on the other the engineering facts about the vulnerability or vulnerabilities. The alternate perspective is that software development organizations, inside every sector, have wrestled with this problem since the first programs were written. Bugs, both new and old are found in every release, at every stage of the development cycle; which leads us to the question: what does a ‘stop release’ bug actually mean? Much has been written about end user organization software vulnerability/patch risk management, but little in the context of software security from an independent software vendor release risk management perspective.

Microsoft has done some good work here with their SDL bug bar concept. Even as going so far as to provide close integration with their tooling such as Visual Studio 2010 Team Foundation Server. However the Microsoft SDL concepts are absolute.

“A bug bar is a quality gate that applies to the entire software development project and defines the severity thresholds of security vulnerabilities—for example, no known vulnerabilities in the application with a “critical” or “important” rating at time of release. The bug bar, once set, should never be relaxed.”
“... none of these factors should play any role in determining whether to fix security bugs—that is, bugs that could potentially lead to security vulnerabilities in the product. Classification of security bugs must be objective and consistent. It doesn’t make any difference to an attacker that you found a vulnerability only a week before your code-complete milestone; he’ll exploit it just the same”
We agree that vulnerability classification should be objective and consistent. However, alone this classification will rarely be used to decide delay product shipment in organizations where security is seen as a cost. Software vulnerabilities are a reality that organizations who develop software have to deal with, and in our experience, it is rarely an purely objective decision unless the business both embraces security and is exceptionally flexible with cash to burn.

Ask The Questions

If your product or system has vulnerabilities that are sufficiently severe enough in terms of impact or importance there are a number of questions that require answers before a release delay decision can be made. We've raised some example questions below for both the business, engineering and security teams; the answers of which can then be fed into a risk decision process or model.

Of The Business
  1. What are the direct costs of delaying the release? Financial cost to the business if the planned release date is missed.
  2. Are there any overriding internal factors? Internal factors aside from fiscal costs which mean that the shipment can’t be delayed, such as dependant products.
  3. Are there overriding external factors to consider? External non-cost factors, which mean the shipment can't be delayed, like external commitments.
  4. What are the direct costs if an emergency patch is required? Financial costs associated with producing, testing, releasing and communicating an out   of band / emergency patch.
  5. Are there any indirect costs if an emergency patch is required? Various indirect costs which include company, brand and product reputation, customer good will and other less tangible costs.
  6. Will clients deploy an emergency / critical patch? Once clients have deployed a system a critical security patch out of the patch cycle potentially introduces a greater risk of outage and therefore a failure to meet SLAs. Customers typically want a regular predicatble patch cycle they can build into their SLAs. If emergency critical patches will be precluded from deployment in the majority of customer deployments then there is a counter argument for developing / releasing one.
Of The Engineering and Security Teams
  1. Is this an internal or external application? The position of the application, such as: is it purely internal, Internet facing, in active support or being sold to end customers.
  2. Who is the discoverer? Whether the source of the exposure is internal or external to the business and the way in which it was raised.
  3. What is the technical risk of the vulnerability? Microsoft’s STRIDE (although their DREAD model will likely need answering too), CVSS or equivalent.
  4. What is the likelihood of discovery? A purely subjective view that could well be wrong, but used as an indication.
  5. How complex is the fix: Requires root cause analysis to have been completed followed by a technical estimate from the engineering team.
  6. What sort of testing effort is required? Once the impacted code has been identified (through root cause analysis) the scope and scale of the quality testing required can be understood and effort estimates created.
  7. When is the next patch opportunity? Next scheduled patch opportunity or release window. This then becomes the period of exposure for the business; which when combined with typical patch application the customer exposure period can be derived.
  8. Are there any mitigating factors? Anything else not captured previously that could be used to define a business case for either supporting or delaying the release.
Without being in the position to answer most of these questions it is unlikely that the business will be able to make an informed risk decision; and will likely push back or just ship regardless. Of course, we can rely on gut instinct and experience (or shouting); but if we want something repeatable that can stand up to scrutiny (and the inevitable worst case fallout) then you’ll likely want to consider adopting something similar. A list of questions to inform the decision process, prior to presenting the case to the judge and jury (aka the business) on the risk being carried with a release.

At which point we should probably be posting an elegant equation that is a panacea to software vulnerability risk that gives a yes or no answer to the 'should we ship question?'. But we're going to point out the uncomfortable truth and say it's just not that simple. There will be horse trading, there will be discussion, there will be bartering. So be prepared and don't be a risk automaton.

Now we're pragmatists, so we do recognize we've painted a utopia with the above list of questions. In our experience it’s extremely hard to answer a number of the questions posed. For example there is a dearth of research on the tangible impact to a brand resulting from security issues. We found a single paper published in 2011, by Carnegie Melon University on the subject, titled Impact of Software Vulnerability Announcements on the Market Value of Software Vendors – an Empirical Investigation.  In the paper the authors conclude:
“Our results show that vulnerability disclosure leads to a significant loss of market value for software vendors. This indicates that the stock markets react negatively to the news of a vulnerability disclosure, because the discovery of a vulnerability could suggest a loss in future cash flow for the software vendors.”
This conclusion comes with many caveats, so we encourage you to read the paper rather than take our salacious, out of context quote, as read. For example it doesn't deal with internal applications, nor touch upon positive security vulnerabilities. By positive security vulnerabilities we mean those vulnerabilities that allow users to exercise a product in a way the original manufacturer did not intend. Yet through the bugs existence, subsequent exploitation and press coverage, leads to more sales of the product in question or improved brand recognition. Additionally, if you've never experienced a public security vulnerability you'll be unlikely able to answer how much it costs to respond.

Finally, it’s important to keep in mind; that even with the most comprehensive risk analysis process the reality is the final veto lies with the business. If the appetite exists to assume the risk no matter how grave, then we as security professionals must provide support whilst documenting, planning and mitigating as much as possible given the constraints placed upon us.

In conclusion, we don't feel bug bars and similar on their own are enough given the reality in most software development organizations. The decision of whether a product should or should not ship given the presence of serious security issues is a complex one. However, making a business aware risk decision process, which captures input from all stake holders can remove barriers and reduce confrontation, whilst providing support to the goal of security teams, assessors and accreditors - and more secure products in the long run.

If you’ve found this post interesting you may also be interested in our blog post on ‘Breaking the Inevitable Niche/Vertical TechnologySecurity Vulnerability Lifecycle’.

We'd also like to tip our hat to Phil Huggins who gave this post a once over for us prior to publication. He was good enough to suggest the SLA business consideration.

No comments:

Post a Comment