"It is impossible for ideas to compete in the marketplace if no forum for
  their presentation is provided or available."           Thomas Mann, 1896

---

METRIC OF NETWORK INTEGRITY

By Kevin Soo Hoo
Contributed by Sygate Technologies, Inc.

 

"To measure is to know." - Maxwell

"When you can measure what you are speaking about and express
  it in numbers, you know something about it." - Lord Kelvin

Challenge of Metrics

The recent spate of network worms has focused attention, once again, upon information security, particularly network security. After more than a quarter century of corporate computing, business managers and technologists alike can do little more to articulate the state of their network security than give qualitative descriptions of their "gut feel" about it. Why is this the case? Certainly, the lack of concrete metrics may confer a certain degree of job security for some, but they, alone, cannot account fully for the absence. A more plausible root cause may rest in the fast pace of innovation in network technology and the inherent difficulties associated with measuring security itself.

Other types of security have metrics. Those metrics, however, are usually process-based ones that were developed for well-established avenues of attack. Whether they are force counts for national security or guards, locks, and doors for physical perimeter security, the threats and adversary capabilities are static enough to have
known countermeasures, thus enabling metrics of security. However, even in these areas, the metrics are still indirect measures of security; that is, they are correlated with good security, but not direct measures of security itself. And as a result, they can be fooled. The number of soldiers in an army was once an extremely accurate measure of its strength, but mechanization and technological advances have changed that metric. Even today, force comparisons must be modified for technology in the balance of power calculations of strength.

Network security suffers from the twin drawbacks of rapid innovation and poorly understood adversaries. The fast pace of technological change creates opportunities for new modes of attack and network failure. While network defenders are busy developing countermeasures for known exploits, their adversaries are equally busy concocting new ones. And, as the network becomes ever more complex, the supply of weaknesses to exploit appears inexhaustible. Similarly, the supply of network attackers is highly varied and also seemingly unending.

The often-cited laundry list of malefactors, for example, terrorists, business competitors, national spies, hackers, and thieves, does little to reveal the true motivations, capabilities, and interests of those actually attacking networks today.

Good Metrics

Given the challenges of measuring network security, the slow progress of metrics development is hardly surprising. Strategies for characterizing or validating security generally fall into one of the two categories below.

1. Examine the results of security assessments, including red-teaming exercises, penetration testing, vulnerability scanning, and other means of probing defenses for weaknesses in security.

2. Examine the building blocks, processes, and controls associated with security efforts to infer the prevalence of vulnerabilities. Activities include auditing business processes and procedures for security policy compliance, assessing the quality of security in infrastructure components, and reviewing system development and administration processes for security best practices.

These measurement strategies, however, do not necessarily imply good metrics. In fact, many of them are biased, subjective, and not repeatable. If measurements are instantaneous snap-shots of a particular parameter, then metrics are more complete pictures, typically comprised of several measurements, baselines, and other supporting information that provide context for interpreting the measurements. Good metrics are goal oriented and exhibit, according to George Jelen of the International Systems Security Engineering Association, SMART characteristics: Specific, Measurable, Attainable, Repeatable, and Time dependent.
http://csrc.nist.gov/csspab/june13-15/jelen.pdf

Network Integrity

Today, measurements of network security are often conducted by separate organizations that independently define, collect, and analyze technical metrics in isolation. These metrics include the numbers of vulnerabilities found in network scans, known incidents reported, estimated losses from security events, security bug discovery rate in a new software application, intrusion detection system alerts, number of virus infected e-mails intercepted, and others. But, are these good metrics? Looking back at the criteria set forth, this assortment of measurable attributes tends to be subjective in nature, highly dependent upon the measurement taker, and generally lacking a coherent story or goal.

The concept of network integrity is a useful means for limiting the scope of network security metrics to a tractable subset of what can be measured. Adapting the NIST definition of system integrity, network integrity is a state in which the network "performs its intended function in an unimpaired manner, free from deliberate or inadvertent unauthorized manipulation of the system."

This simplification places network integrity as part of network reliability and divorces it from questions of network information asset value, loss, and opportunity cost. The problem can be further constrained by looking only at electronic (i.e. non-physical) threats. The diagram below illustrates how network threats precipitate tactical security objectives, which in turn support network integrity and reliability.  http://csrc.nist.gov/publications/nistpubs/800-12/

The nearly ubiquitous nature of the network in enterprises today means that it affects practically everyone in the organization, from the receptionist at the front door to the chief executive officer in the boardroom. Depending upon their role in interacting with the network, various stakeholders will be concerned about different aspects of network integrity. Executive officers, being responsible for the overall performance of the enterprise, are concerned with the ability of the network to support operations. Because they have the authority to allocate resources, both personnel and financial, to deal with problems of network integrity, they would be interested in answers to the following questions.

1. How does the organization's network integrity compare to that of its peers?

2. How does network integrity this year compare to last year?

3. Am I spending the right amount of money on network integrity?

4. What am I getting for the money I am spending and about to spend?

5. What are the costs and consequences of not acting to improve network integrity?

Information technology operations personnel, responsible for server and desktop maintenance, are generally interested in a more granular view of the network integrity to maintain network services. Whereas executives look for support for resource allocation decisions, network operations people seek help to prevent, detect, and respond to network integrity compromises. Thus, questions of concern include:

1. What computers, applications, or services are compromising network integrity and where are they?

2. How is the compromise taking place? Is it spreading/getting worse? How and where?

3. How serious is the impact of the compromise?

4. What technical measure can be taken to isolate and remediate the problem machines?

Finally, the network security team is typically responsible for an organization's security policies and programs. Although they may not have direct operational responsibility, they are quite interested in how security policies, procedures, and programs are ensuring or failing to ensure network integrity.

1. Were the computers responsible for compromising the network policy compliant?

2. What changes should be made to security policies and procedures?

3. If policies are not working, what behavior changes should policy modifications be aiming to achieve?

4. What technologies could help prevent future compromises?

5. What was the impact of the compromise?

Metrics of Network Integrity

Executive Officers

• Network Service Level - Percentage of time that network services are available for a given period of time as well as part of a time series to give historical context.

• Business Requirements Met - Percentage of business needs placed upon the network that are being met.

• Number of Compromises - Number of incidents during a given period in which network integrity was compromised.

• Organizational Impact of Compromises - For each incident, the number of hours, time of day, and people affected by the degradation or disruption of network service.

• Costs and Benefits of Improvements - The direct and indirect costs and benefits of steps that can be taken to improve network integrity.

• Peer performance - Network service level benchmarks from peer organizations.

Information Technology Operations

• Compliant Devices - Percentage of network devices that are security policy compliant.

• Managed Devices - Counts of systems and devices under active management

• Total Devices and Users - Total numbers of devices and users on the network.

• Network Performance - Mean time for packet delivery in the network.

• Network Utilization - Bandwidth utilization at key gateways in the network.

• Network Wait Times - Length of time to perform defined network functions, such as download a specific file, retrieve e-mail, open a web page, etc.

• Impact of Compromise - Users affected (service degraded, disrupted, or otherwise compromised), number of devices participating in compromise, decrease in network performance, increase in network utilization, and increases in wait times during a network compromise.

Security Team

• Vulnerability Counts - Numbers of vulnerabilities found on the network, broken out by those on policy-compliant devices vs. those found on devices that are not.

• Detailed Compliance Reports - Numbers of users and devices compliant with each element of the security policy.

• Incident Forensics - The numbers of incidents attributable to policy failures vs. policy compliance failures.
  Impact of Compromise - Users affected (service degraded, disrupted, or otherwise compromised); data lost, modified, or destroyed; number of devices participating in compromise; decrease in network performance; increase in network utilization; and increases in wait times during a network compromise.

• Remediation Time - Time between compromise discovery and completion of system remediation.

Note: Security policies are the primary vehicle by which security expectations and requirements are communicated throughout the enterprise. The best practices that constitute good security policy are too numerous to enumerate here. A good metrics regime provides necessary feedback to an organization's policy maintenance and enforcement process to keep theoretical security expectations consistent with realistic security practice. Measuring the quality of security policy is a separate and important exercise, but it is not the subject of this paper.

Instrumentation

Attainability and repeatability are the remaining attributes of good metrics to be addressed. Automation of the measurement process with some form of instrumentation technology is one scalable strategy for ensuring them. Limiting the role of human intervention in the process serves to minimize both subjective judgment and measurement errors. Impartiality and consistency are necessary conditions for credible measurement comparisons across either time (time-series) or organizations (benchmarks).

Conclusion

The need for metrics is real and growing more acute. In this paper, we have put forward a list of good metrics of network integrity that help inform security decisions across the enterprise. A number of industry consortia and trade groups are putting forward candidates for general IT security metrics. TechNet, the Global Security Consortium, and others are advocating new approaches to metrics. The hope is that formal benchmarks will soon emerge to give both private and public entities visibility into the state of information security at local, industry, and national levels. The Department of Homeland Security in a recent Cyber Security Summit, in Santa Clara, CA, put the information technology industry on notice that solutions to the apparent cyber insecurity and lack of standard metrics must be found soon, or the government will intervene. What shape that intervention might take is uncertain, but the threat alone is motivation enough to begin a concrete discussion about metrics.

Visit the Authors Web Site