Centering the Human in AI Decision Making

IHMC, in collaboration with Carnegie Mellon University, has developed a new AI, but it’s not a Large Language Model; it doesn’t need a data center; and you can’t talk to it.

Yet.

In the DARPA-funded project MIMIR (Metacognitive Intelligence Maneuvers for Influence and Resolution), Dr. Ian Perera, with co-Principal Investigators Dr. Matt Johnson and Dr. Mark Orr, leads a team of researchers in building an AI system that rapidly learns to make decisions and, perhaps more importantly, learns how we as humans make them. Leaving data-hungry methods behind, MIMIR advances the IHMC mission of centering the human in the AI decision-making process, while paving the way to systems that learn to explore the world without very much input from us at all.

If you want to find an answer to a question, solve a common problem, reorganize a document, or write a program, you probably will still use a Large Language Model, or LLM. But the MIMIR team is targeting different kinds of problems – those where the data is sparse, complex, and the best strategy is dynamic and responsive to changing environments or conditions. Think cybersecurity, disaster response, negotiations – domains where human expertise is immensely valued but difficult to write down as a simple procedure.

The key is that human element. “I’m actually less interested in making better AI than I am in making humans better through AI,” Dr. Perera explains. “While LLMs are trained to produce acceptable answers to the average person, I wanted to explore how we could leverage human strengths, while making our limitations and biases something to improve, not ignore or reinforce.”

In some ways, MIMIR can be seen as a pushback against the ways LLMs seem to be making us say more while thinking less, reinforcing preconceptions and biases, all while sitting behind a black box that not even their engineers can understand. “When you are looking at a MIMIR decision, you can drill down into every piece that went into that decision – it retrieved this example from a similar situation in its memory; it made this inference of missing data based on a particular reasoning method; it assumed this type of behavior or goal based on context; and it used this model of decision bias when making a prediction.”

To achieve this alignment with human decision-making, MIMIR builds on decades of work in cognitive modeling by team members Dr. Christian Lebiere, Dr. Drew Cranford, Dr. Mark Orr, Dr. Brent Venable, and Connor Tate. MIMIR isn’t trained like LLMs and prior data-heavy approaches in AI, but instead, learns more like a human – trying options, seeing what works, and storing its experiences in memory that it can retrieve when a similar past situation might provide valuable insight into the problem at hand. “Our cognitive modeling approaches have a quantifiable advantage in data efficiency,” Dr. Lebiere explains, “but beyond that, we also have an opportunity to model human decision-making biases and then account for, defend against, or leverage that knowledge.”

Human trials have shown that these models predict known human cognitive biases including recency bias, risk aversion, and confirmation bias. By modeling these behaviors, MIMIR can span the gap of decision-making from optimal to human-like.

“In the cybersecurity domain, you often have limited resources to defend important assets,” explains Dr. Cranford. “If you assume every attacker is behaving optimally, you could quickly become overwhelmed. But if you know how those attackers value objectives and risk, you can find ways to make some assets seem like a less attractive target, while strongly defending the others.”

MIMIR rapidly learns to build archetypes of decision-makers – best guesses of their behavior, motivations, and biases. These initial hypotheses provide a first pass allowing the system to make a split-second decision. It may be wrong, but with the cognitive modeling approach, it’s more likely to be wrong in the way a human might be – a key element of building trust when using AI in critical applications like medicine or defense.

“We’ve found that if your AI is going to make mistakes, it’s much better for its mistakes to resemble those of a human,” Dr. Perera explains. “When the AI is open about its weaknesses rather than giving a potentially hallucinated but confident response, a human can make better use of the AI and we actually get better results than when using even a more performant AI.”

Individualized knowledge of human behavior has impacts outside of the adversarial case. If you’ve ever been sent a report or news article written by AI, you’ve probably felt like you’ve just been made to waste your time reading through text when the real meat of the question was buried in a reference. The LLM doesn’t know what you find important, or what questions you would want to ask. That’s where one component of MIMIR, Multialternative Decision Field Theory (MDFT), comes in.

“With MDFT, we can analyze not only the decisions you make, but also where you’re looking at, or where your mouse lingers, and figure out what is important to you even before you say anything,” Connor Tate explains.

MDFT learns a model of preferences and attention, and with that, can predict your next actions or identify when you are deviating from its prediction. “We aren’t just using it to predict actions, but also when physiological or psychological stressors are changing how you are processing information in real-time,” adds Tate.

A key part of MIMIR’s ability to rapidly adapt to new situations centers around work by Dr. Matthew Johnson and Michael Vignati on Joint Activity Graphs (JAGs). JAGs represent actions and activities, but more importantly, how those activities are interdependent. In the domains where MIMIR operates, these linkages are vitally important for guiding the system towards having a coherent strategy. Dr. Johnson has successfully implemented JAGs in other situations like search-and-rescue and firefighting before they were used in MIMIR.

“In real-world, complex decision-making, you have different procedures and processes that interact, and you often don’t see the desired outcome, metric, or reward until the very end of a long effort. JAGs allow the cognitive model to explore a strategic space that obeys the constraints between activities while having a vision towards the long-term goals it wants to achieve.”

The close integration with human knowledge elements, like those encoded in JAGs, is both a strength and a weakness of MIMIR. Dr. Perera explains, “You aren’t going to download MIMIR or access it from a website, because that encoding of domain-specific knowledge takes some time and expertise. But to have that expertise explicitly present and verifiable in the system is, I think, a real strength.”

Even with these caveats, there is a future where MIMIR is engaged with a broader range of problem solving and insight generation, not just working of specific applications behind the scenes. Dr. Perera already has ideas for extending MIMIR to act like a scientist or program manager, finding promising research avenues and gaps in existing knowledge.

“If you think about the frontier of scientific or mathematical research, you have this relatively understudied combination of known processes mixed in with conceptual leaps like analogical reasoning or perspective shifting. MIMIR is positioned to be able to take in that kind of flexible decision-making and exploratory activity to do the sustained work that a standard LLM just can’t.”

This is not to say that LLMs can’t make MIMIR more extensible and perhaps broadly available. On the contrary, Dr, Perera explains that the representations MIMIR uses make it an ideal testbed for integrating with LLMs to explore new domains. “I can envision a system where an LLM acts as a means to operationalize knowledge, whereas MIMIR acts as the prefrontal cortex to actually make the decisions. Because LLMs are trained on large amounts of data, they often have a bias toward what’s already been done or what the user thinks should be done. MIMIR gives us an opportunity to look toward the future and the unknown.”

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited). DISTAR Case 43101. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA). The views, opinions, and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of DARPA or the U.S. Government.