But how does the brain know when it's time to abandon an old strategy and try something new?

A new study published in Nature Communications sheds light on that question. Neuroscientists at the Okinawa Institute of Science and Technology (OIST) have identified a key brain mechanism that helps animals adapt when circumstances suddenly change. The findings could improve our understanding of conditions that make it difficult to break habits, including addiction, obsessive compulsive disorder (OCD), and Parkinson's disease.

"The brain mechanisms behind changing behaviors have remained elusive, because adapting to a given scenario is very neurologically complex. It requires interconnected activity across multiple areas of the brain," explains co-author Professor Jeffery Wickens, head of the Neurobiology Research Unit at OIST.

"Previous work has indicated that cholinergic interneurons, brain cells that release a neurotransmitter called acetylcholine, are involved in enabling behavioral flexibility. Here, we were able to use advanced imaging techniques to see neurotransmitter release in real time and delve into the fundamental mechanisms behind behavioral flexibility."

How the Brain Responds to Unexpected Disappointment

To investigate, researchers trained mice to navigate a virtual maze. The animals learned which route would lead to a reward and gradually developed a reliable strategy for reaching it.

The scientists then changed the rules.

After the reward pathway was switched, the mice unexpectedly failed to receive the reward they anticipated. Using two-photon microscopy, the researchers monitored activity inside the animals' brains as they reacted to this surprising outcome.

"Neurally, we saw a significant increase in acetylcholine release in certain areas of the brain. And behaviorally, we saw more mice displaying what's known as 'lose-shift' behavior, changing their choices in the maze after non-reward," says Dr. Gideon Sarpong, first author on the study.

"The greater the increase in acetylcholine the more likely the mice were to change their future choices. Our results demonstrated the importance of acetylcholine in breaking habits and enabling new choices to be made."

Acetylcholine Helps Break Old Habits

To test whether acetylcholine was truly responsible for this behavioral flexibility, the team reduced the animals' ability to produce the neurotransmitter.

The effect was clear. Mice showed far less lose-shift behavior, making them less likely to adjust their decisions after an unexpected outcome. The results confirmed that acetylcholine plays an essential role in helping the brain adapt when circumstances change.

Interestingly, not every group of cholinergic interneurons reacted in the same way. While most released more acetylcholine, some small clusters of cells showed little change or even a decrease in activity.

According to the researchers, this may help preserve information about previously successful behaviors.

"This indicates that the mice may not necessarily forget the previous pathway to reward, but retain this information in case the situation changes again," says Dr. Sarpong.

Implications for Addiction, OCD, and Parkinson's Disease

The researchers emphasize that behavioral flexibility involves a much larger network than a single neurotransmitter or brain cell type. Multiple brain regions and chemical signaling systems work together to help animals and humans adapt to changing situations.

Still, the new findings provide an important piece of the puzzle.

"But it's an important piece of the puzzle, as the activity of the striatum, where these cholinergic interneurons are held, is a central component of this system," emphasizes Prof. Wickens.

Beyond advancing basic neuroscience, the research may eventually contribute to better treatments for neurological and psychiatric disorders.

"Acetylcholine levels are often altered in treatments for neuropsychiatric disorders like Parkinson's disease or schizophrenia, so understanding the function of this neurotransmitter is essential in treating many neuropsychiatric disorders," says Prof. Wickens.

"In particular, with conditions such as addiction and obsessive-compulsive disorder we see a difficulty in breaking habits and shifting behavior. So, understanding the mechanics of behavioral flexibility may one day help us develop better treatments."