Optimal Experimental Planning, Resilience, and Simulation Methods Applied to Cybersecurity Experimentation

Abstract

Many business situations can be called “games” because outcomes depend on multiple decision makers with differing objectives. Yet, in many cases the payoffs for all combinations of player options are not available, but the ability to experiment offline is available. For example, war gaming exercises, test marketing, cyber range activities, and many types of simulations can all be viewed as offline gaming-related experimentation. We address the decision problem of planning and analyzing offline experimentation for games with an initial procedure seeking to minimize the errors in payoff estimates. Then, we provide a sequential algorithm with reduced selections from option combinations that are irrelevant to evaluating candidate Nash, correlated, cumulative prospect theory or other equilibria. We also provide an efficient formula to estimate the chance that a given Nash equilibria exists, provide convergence guarantees relating to general equilibria, and provide a stopping criterion called the estimated expected value of perfect offline information (EEVPOI). The EEVPOI is based on bounded gains in expected utility from further offline experimentation. An example of using a simulation model to illustrate all the proposed methods is provided based on a cyber security Capture the Flag (CTF) game. The example demonstrates that the proposed methods enable substantial reductions in both the number of test runs (half) compared with a full factorial and the computational time for the stopping criterion. In the second project, we propose a novel framework for resilience engineering which generalizes robust engineering utilizing defender-attacker-defender (DAD) model as an application for cyber-resilience. There are two types of settings: preparation, in design stage, and recovery which can be set as response to observe an adversarial event. We propose the bathtub model to study the throughput before, during, and after the adversarial event and assess at the steady state comparing to the baseline (equilibrium). In our experiments, both signal and noise factors are considered in the model and the interaction between them are studied. Conducting the real experiments is expensive and risky to the security for the network, e.g. DDoS. Hence, simulated and pen-testing data that been used to estimate the means and variances using regression. Then, we propose two-stage stochastic optimization models to determine the optimal options for both of preparation and recovery settings. The alternatives are minimizing the expected value of loss, minimize the worst-case loss, minimize the maximum regret, and conditional value at risk (CVaR) which minimize the loss in (1-α) worst cases. We demonstrate the solutions to the robust methods with an example of DDoS attack and discuss the insights their solutions provide into mitigation selection. Our study provides valuable tools and insights for decision-making process with various risk formulations to be able to assess cybersecurity risks under uncertainty. In the third project, we propose a discrete event simulation to forecast the inspection costs for cyber maintenance under various policy options. There are three types of scan options, network, internal, and both (network and internal), which relate to the type of cybersecurity vulnerability involved. We also consider the possibility of sampling only a portion of the hosts for scan and decreasing the interval between scan periods from four to two weeks. The results indicate that a more nuanced approach of small samples collected every two weeks of scan results using a mixture of scan options can result in significantly reduced average monthly costs by over 40%.