The White House’s 2023 National Cybersecurity Strategy looks good on paper, with clear big picture milestones to achieve. But “neither the strategy nor the implementation plan included outcome-oriented performance measures… to gauge success” a watchdog has warned.
President Biden’s administration set up the Office of the National Cyber Director (ONCD) to implement the strategy – which has been broken into 69 initiatives, the responsibility for which sits with 18 federal agencies
The ONCD says the percentage of these being completed on time “will serve as an overall performance measure.” But the Government Accountability Office (GAO), a congressional watchdog that investigates federal spending and performance, does not think this is good enough.
Cybersecurity outcomes: Define...
In a report this week it argued that the ONCD should be thinking about outcome-oriented performance measures (a challenge for CISOs as well for governments) and criticised the fact that the ONCD “did not identify how much it will cost to implement the initiatives…” (The ONCD objected to this, saying that legislative rules restrict agencies from disclosing future year budget plans outside of the current budget cycle.)
“While certain initiatives may not warrant a specific cost estimate, other activities supporting some of the key initiatives with potentially significant costs justify the development of a cost estimate. Such cost estimates are essential to effectively managing programs. Without [them] uncertainty can emerge about investing in programs,” GAO said.
Ramy Houssaini, an experienced cyber risk leader and Chair of the Cyber Poverty Line Institute agreed. He told The Stack: “The National Cybersecurity Strategy establishes a robust framework for reshaping incentives and fostering collaboration among various stakeholders. However, the effectiveness of any strategy hinges on its execution…
“Emphasising the measurement and demonstration of impact is crucial for strengthening cyber capabilities, as it helps to uncover blind spots and prevent unexpected challenges. It might be hard to do but it is absolutely necessary," Houssaini told The Stack.
So what does GAO have in mind for the ONCD?
That's not outcome-based!
One example: “Initiative 2.1.2 called for strengthening the National Cyber Investigative Joint Task Force’s capacity to coordinate takedown and disruption campaigns with greater speed, scale, and frequency. ONCD may be able to measure the number of government disruption campaigns that occurred and the speed at which the joint task force is able to coordinate the takedown of these disruption campaigns,” it suggested.
“Initiative 1.2.5,” meanwhile, it noted, “tasked [CISA] with establishing and codifying a sector risk management agency support office capability to serve as the single point of contact for all sector risk management agencies. A cost estimate for this initiative would provide the Cybersecurity and Infrastructure Security Agency with information to support a request in its budget submission for funding this capability.”
The NCD told GAO that it was at least in part, mistaken: “The example measures [it] identified earlier in the report are not outcome-based but output-based measures of success," it told the watchdog pointedly: “Developing outcome-oriented measures remains an open research problem” it added – something many in this domain would agree with.
As Kevin Jackson, CEO at Level 6 Cybersecurity previously put it: “What is missing from standards and frameworks are outcomes. Very little attention is paid to what has worked well in the real world because such data are only available in narrow information silos. Imagine if, instead of relying on vendor inputs, consultant opinions and trend-of-the-day approaches, each cybersecurity decision maker could leverage analytic results about what works best in a given cybersecurity domain for similar organizations. This would enable practitioners to pursue the strategies that work, including the impacts of scarce cybersecurity resources and extreme security management complexity, then constantly adapt those strategies to further pursue increasingly positive cybersecurity outcomes.”
Imagine if, indeed…
It's a viewpoint many have been advocating. As Gartner puts it: "Traditional cybersecurity delivery metrics such as number of incidents closed, or number of attacks faced have lost their relevance as they do not reflect the true impact of cybersecurity investments. Cybersecurity outcome-driven metrics (ODMs), on the other hand, link security and risk operational metrics to the business outcomes they support."
Gartner Distinguished VP Paul Proctor put it plainly in late 2023: "For example, incident remediation is an ODM that measures the outcomes of your investment in response capabilities... [each of the 16] can be explained to a business executive with no technical background. Second, they represent high value, common security control investments. Everyone needs these." Even Presidents...
