On Evaluating Cyber Defense by Humans and Reinforcement Learning Agents

Abstract

A significant factor that drives the investment in/integration of a cyber defense technique is its proven effectiveness, in terms of performance, usability, and security, against an intended adversary. Measuring the effectiveness of security defenses is complex, mainly because evaluation is a never-ending process, a natural outcome of evolving technologies and adversarial capabilities. To facilitate cyber defense evaluation, it is, thus, imperative to examine novel methods through which researchers could gain a deeper understanding of adversarial behavior from both technical and cognitive perspectives.

To this purpose, the first part of this dissertation investigated the applicability of using two popular paid crowdsourcing platforms to run hacking experiments (i.e., Capture-the-Flag challenges). Paid crowdsourcing platforms can potentially facilitate studies in the Oppositional Human Factors (OHF), a field that considers attackers’ cognitive biases when examining the effectiveness of defensive techniques. Such platforms are unique in that they offer schemes eliciting real-world adversarial behaviors (e.g., maliciousness) while minimizing the constraints imposed by typical recruitment methods. Findings showed that the platforms vary significantly in data quality and workers’ technical skills. As a result, I examined the possibility of using Prolific to conduct a randomized study to analyze attackers’ cognitive biases and limitations when they encounter deception under an incentive-compatible scheme (i.e., monetary) and a design different from prior studies. Findings showed consistent behavioral patterns between the populations under consideration but revealed some shortcomings of online cybersecurity experimentation.

As online experimentation with qualified human participants remains challenging, the third part of this research examined another alternative in which datasets from experimental psychology studies designed to analyze deficits in decision-making processes (e.g., gambling tasks) are used to guide the design of human-like cyber agents. More specifically, I investigated the role memory systems and settings play in predicting the behavior of healthy and unhealthy populations. The motivation behind this work came from the realization that healthy individuals and individuals with pathological tendencies—the latter are more inclined to commit cyber crimes [5]—exhibit different risk attitudes that could be partially attributed to how memories are processed and used to guide future decisions. The results demonstrated differences in the role memory systems and settings play in predicting actions taken by the populations under consideration. Having examined the role of memory in predicting real-world behavior, I augmented reinforcement learning agents with similar settings, analyzed the agents’ behavior in simulated network environments, and observed interactions between the type of memory systems used to guide the agents’ learning process, the type of experiences used in the learning process, the underlying learning strategy, and the observation spaces. The results highlighted the importance of analyzing the aforementioned factors when designing predictive models for adversarial decision-making.