Workshop on the Future of Cyber Deception Presented by the Army Research Office

This workshop brought together researchers from cognitive science, artificial intelligence, and cybersecurity to discuss the direction of cyber deception research. One purpose of the workshop was to center human behavior and decision making in the discussion on where to take research into cyber deception. From my perspective, the future of cyber deception will involve in some way human decision making, whether as human in the loop systems or human on the loop systems, or as a reviewer and analyzer of the behavior of automated systems. This is an important reality to account for when thinking of the future of cyber deception, as oftentimes automated system design does not take into account the realities of human decision making.

While it may seem like humans have become less and less involved in areas relevant to cyber deception throughout the years, they still remain an important factor in key areas. One area that our lab is currently researching is in phishing, where human or automated systems attempt to leverage deception into compromising security in a variety of ways. In these instances, it is crucial to understand how humans evaluate the danger of potential instances of phishing, such as emails, as well as how new automated systems are being designed to thwart these defenses.

The remainder of this post is my personal notes of the presenters of this workshop. As always, these notes are my own thoughts and not necessarily claims made by the presenters.

Cognition and Cyber Deception

Christian Lebiere

• Overview of Cognitive Architectures applied to predicting human decision making in cyber security tasks.
• It is possible to determine the cost and benefits of cyber deception from an Instance-Based Learning model by querying it to determine the associated value.
• Cognitive salience for explainable AI. Determine which features an individual are more or less sensitive to based on their previous behavior.

Drew Cranford

• Personalized Cognitive Models of Phishing for Training and Defense.
• Train users to better classify spam/ham emails.
• IBL model takes in the raw information of the email in the form of the sender, subject, and body. Uses a NLP method UMBC semantic similarity, to determine similarity.
• Model of an individual using model tracing, what emails can we give to a participant to maximize their learning at a given time, use a database of emails to determine which will be maximally educational.
• Optimize organization training based on optimally selecting which members of the organization receive training.
• Attacker modeling for personalized cyber-inception.

## Yinuo Du

• The Cyborg experimental environment provides a simulated environment of a network that can integrate red and blue agents with different actions that impact the network.
• Blue agents monitor, remove, and restore. Goals are to prevent the red agent from accessing the higher network resources. Penalty for disrupting the normal user access to the network.
• Study different types of attack strategies to see how they work against different types of defending agents, such as random, IBL based decision making, and decisions made from human participants in an experiment.
• IBL attackers learn a better strategy than a static optimization of performance, compared to how humans learn in this environment.

Palvi Aggarwal

• Deploy different human attackers against defenses defined from machine learning approaches like SSG.
• Additionally simulate attacker strategies using IBL.
• Use these attacker simulations to improve defense strategies.
• Attacking agents are susceptible to cognitive biases, the question is how to leverage these biases to improve defense.
• Manipulating deception in two directions, making honeypots appear to be real and making real systems appear to be honeypots.
• Humans exhibit patterns of cognitive biases that can be exploited in different environments

Prashanth Rajivan

• Representing the cyber context within emails.
• Instances in cognitive modeling are typically defined in a hand-crafted approach, however measuring things like emails or network traffic makes it difficult to define instances.
• Different attackers can observe a network differently which can impact their behavior.
• Learned representation is a fixed size n-dimensional embedding outputted by the large language model.
• Perception BERT model can produce similarity judgements for IBL models that most accurately predict human learning and decision making in phishing email contexts.
• Transfer from one learning data set to another using the same model.

Discussion Presented by Coty Gonzalez

• Can model attackers personality behavior though it can be complicated to try and infer how personality impacts attack strategies.
• Many factors can impact attack strategies, personality can be complicated and other aspects can impact strategy besides personality.
• Also we can’t get personality tests for attackers so it may not be that useful for predicting different attacker performance anyway.
• Some methods integrate longer processes of interacting with attackers to try and delay the attack and continue the interaction with them, this motivates the use of techniques that are more cognizant of longer term decision processes.
• Cultural effects of decision strategies in cyber contexts have not been integrated into cognitive architectures yet.
• Can attackers break the defense by knowing the defense strategy? Christian: these cognitive models are adaptive and thus if the attack strategy changes the defender can adapt to the exploitation of the defender.
• Methods in cognitive architecture can model and predict the behavior and decision making of other agents and then optimize their behavior in relation to the decisions made by those agents.
• Differences in representation strategies between attackers and defenders can be important for predicting the behavior of opponents in complex cyber security contexts.
• This could use our approach of theory of mind transfer of learning by gaining experience as the opponent and learning the representation strategy.

Game Theory, Systems and Algorithms for Cyber Deception

Chris Kiekintveld

• Attack graphs represent how attackers choose actions that move through a network or an environment that is under attack.
• Zero day vulnerabilities can be defended against by taking existing attack graphs and adjusting them slightly to determine how best to defend against novel attacks that are similar to existing ones.
• Targeted models of bounded rationality. Compare how bounded rationality impacts decision making in cybersecurity environments at different levels of abstraction.
• Attacker reconnaissance modeling with Bayesian approaches, determine which features of the environment are important for forming attacker beliefs.

Munindar Singh

• Goal moving forward is to develop an understanding of deception that can be applied at different levels, individual, group, or nation state attackers or defenders.
• Future research in deceptions should identify mission level metrics for the cost and benefit of deception, as well as measures of the complexity of deception.
• Deception should be ‘coherent’ across different levels of abstraction to avoid the deception from being identified, this would require more of a planning method that can integrate and control deception across these levels.

Michael Reiter

Leveraging ML Evasion Attacks to Deceive Adversaries.

• Techniques in ML Evasion attacks apply seemingly random noise onto images which can change the classification of the images, one defense against this is to limit how they can be perturbed, but this doesn’t apply to cyber deception.
• One way of thinking of adversaries in reconnaissance is as an ML classifier, so we could try to change the features of raw binaries to fool the attacker into giving the wrong classifier.
• Virus total feed is a source of real malware, and these are modified in a way that changes their features while retaining the functionality of the malware the same.
• Training systems to identify modified malware with adversarial training is challenging, and needs a lot of examples which are expensive to generate.

Deception as a defense against credential abuse

• Leverage reused passwords and apply credential stuffing to gain access.
• Honeywords can be used to generate decoy passwords that can then be detected. However the system needs to know which is the right password.
• Don’t always need the password index checker, instead mark a limited number of marked passwords. This can allow for breach detection once the user retries their password after a breach.
• Another issue is when using passwords across different sites, if attackers get 2 sites they can tell which is real. To fix this we can detect remotely stuffed honeywords, but this can lead to security vulnerabilities if we just sent them all the passwords.

Tiffany Bao

• Real world applications in cyber deception for security, seems to be a gap between the research in deception and real world applications.
• The desires of security engineers and researchers investigating cyber deception are different, some want ease of use and others want more complex control over the environment.
• To achieve both goals, developed the CDGym, based on the sdn network as well as plugins that define cyber deception strategies and network topology plugins.
• Looking at the real world applications highlights the importance of practicality in using strategies for defense that work, example would be a defense strategy that recommends hiding the OS of an asset, but this might be unrealistic and impractical.
• Existing models for defense deception don’t take into account the practicality. Models of attacker behavior assume that the entire attack graph is known but they might not.

Discussion Lead by Chris Kiekintveld

• Perturbations made to the malware were determined using an adversarial learning approach using existing models, and then tested on online applications that tested malware.
• Would it be possible to optimize which passwords are selected to receive the marking to appear like it might be the genuine password, not the honeypot password.
  • How are the honeypot passwords generated in a way that the ‘real’ password doesn’t stand out, if it was the only one with an English word or date or other genuine password features?
• Why do deception researchers have this impractical view of deception, is it just that they want to consider the worst-case scenario?
• How to quantify the metric of revealing that you are being deceptive. Significant difference in how attackers behave against deceptive agents, tend to be much more aggressive against deceptive networks, more scans, etc. Metrics of success are time duration, frustration, mistakes, etc. of the attacker.
• Assumptions of attacker behavior are, knowing the value of a machine, knowing the attack graph, defenders knowing the attacker incentive and cost. Many of these aren’t known exactly but can be predicted based on past behavior.
• Question of whether or not it is possible to do deception when there are tight constraints on what is allowed of the changing network configuration, limitations on how it impacts the bandwidth of the system, etc.

New Directions on Systems and Algorithms for Cyber Deception

Raj Boppana

Cyber Deception Using GPT

• Attacker provides the command and the LLM provides the attacker input as the prompt and some system information. LLM outputs are sometimes not modified.
• Only use necessary history to reduce response time and extend the contextual length.
  • Could be an option for our study.
• Also sanitize the LLM model outputs.
• Levenshtein distance is used to measure the difference between the natural language that is produced by the different versions of the model which use shorter/longer prompts.
• Ideally prompts can reduce the length while keeping performance the same and producing similar responses.
• Spawning LLM honeypots dynamically and at scale on deployed systems would require this type of reduced input space to lower latency.

Kyriakos Vamvoudakis

• Complex adaptive systems research,
• Poisoning data attacks, actuation attacks allows an attacker to adjust the dynamics of a learning environment to force the defender to learn something that is deleterious instead of the true optimal.
• Goal is to consider an environment where there is a group of attackers, and they want to stay below the threshold of detection, which can be done using collective intelligence.
• Individual attackers are assumed to be boundedly rational , with different levels of intelligence, forcing the more intelligent agents to coordinate and account for the behavior of less intelligent agents.

Brendan Dolan-Gavitt

• Increase the number of bugs but do so in a way that makes un-exploitable bugs, there is a large amount of time and resources (skilled exploit developers) to determine if a bug is exploitable.
• Real bugs in that they can crash the program, server side programs that relaunch.
• Goals are for there to be many bugs, guarantee non-exploitability and that it can’t be determined if it is an exploit first.
• Overflow something but have it been unused, use pointless variables to complicate the control flow, overwrite returns so that it always returns null.
• Issues are that it slows performance, doesn’t work on open-source, could be distinguished by attackers to make sure that the vulnerability is exploitable.

Quanyan Zhu

• Role of Information in Cyber Deception
• Open loop vs. feedback information multi-agent environments are differentiated by the information that is known about other players’ state or behavior.
• Initially the benefit of information is initially small but increases as the number of players in the game increases.
• More information is not always better for an equilibrium.
• Multi-agent environments like RPS can improve performance by always playing the same thing and not observing anything compared to a model that tries to predict the decision making of the opponent and act accordingly to that.
  • Under what situations is this true?
• Overt and Covert deception. Overt deception has a bigger impact on the behavior of others compared to covert deception.
  • Overt use of something like chat gpt could be beneficial for people assessing the papers.

Ritu Chadha

• Deception goals for cybersecurity. View isolation, prevent sharing network information. Proactive view change to create moving target defense. Large search space that makes it hard to scan addresses. Active deception to manipulate the knowledge of the environment.
• Techniques: Software defined network control switches where every attached device is on top of a sdn. OpenFlow and HoneyD.

  Discussion Chair: by Michael Reiter

• The goals of cyber deception, specifically on whether or not we want to be imperceptible depend on the context. Sometimes we want to make attacks as unlikely as possible, and so making deception perceptible may not be a big issue. Other times we want to waste attacker time and resources and trap them in a useless task.
• Another relevant question to ask when deciding what the goals for cyber deception are is the length of time that the system will be up.
• One method of honeypots would verbally assault the attackers, which resulted in them revealing more information about who they might be (country of origin, etc.)

New Directions on Artificial Intelligence and Cognition for Cyber Deception

Jin-Hee Cho

• Hypergame theory for tactical networks.
• unmanned aerial vehicle denial of service attacks.
• Protect against DoS attacks, identify best deception strategies. Deep reinforcement learning.
• Honey drones used to keep the mission drones in a fleet of drones safe. Goal of the RL model is to determine which strategies are optimal for the mission drones and honeypot drones.
• Hyper-game theories parameterize the uncertainty of an environment in calculating the utility. HGT also has the concept of a sub-game, where separate actions are taken place, reduced searching space.
• Differences in the information of the attackers and the defenders regarding the true configuration of the game state.
• Method for DRL specifically designed for the hyper-game theoretic task. Determines probabilities distributions of agent’s actions and then use that as an input to another model which is trained in a transfer learning paradigm to improve performance improvement speed.

Fei Fang

• Learning an attacker’s preferences for features of computers in a deception game. Then based on that model we can configure the machines to deceive the attackers. Use mathematical programming to optimize for this deception.
• Attack graph security games have been solved with neural architecture search methods.
• This method stopped working when the game became more complex by introducing dynamic changes to the environment, using reinforcement learning improved the performance in the dynamic setting.
• Potential new research areas include sample efficient MARL for cyber deception, which could be applied for HoneyIoT devices. Need to be adaptive to the defender/attacker behavior, find what the equilibrium strategy is as fast as possible.
• Cyber deception in evolving environments. Many times we need to re-train defenders from scratch. Would be more efficient to instead adapt to changes in the environment.
  • Attention mechanisms allow for variable state spaces.
  • Human experts can provide insight into the relevant changes in the environment.
  • LLMs + RL for cyber security education.

Aron Laszka

• Issue in applying AI for optimizing the deployment of cyber deception. Easy to quantify the costs associated with deception, but the benefits are difficult to quantify.
• Requires an accurate model of both the defender and the adversary, typically research has focused more so on modeling deception and not reconnaissance.
• The process of observing the system, by the adversary, is a difficult process and can be difficult to form a clear idea of the structure based on experience, which makes it difficult to quantify the benefit of cyber deception.
• Guarantees of benefits even if the attacker knows that there is deception taking place.
• Human studies to determine how they behave in these systems, and if it is any different from their behavior in non deceptive environments.

Tyler Malloy

• My presentation.

Kimberly Ferguson-Walter

• Tulurosa cyber deception experiment.
• The presence of decoys significantly changes participant behavior regardless of whether or not the decoys were known to be present by the participants.
• ReScind program.

Discussion Chair: Christian Lebiere

• I was part of this discussion