That cup of coffee you had in the morning to jolt yourself into a semi-functioning state might be powering your brain for way longer than you intended, even through the night and into your sleep, several hours after you took that first sip.

According to a review detailed in Study Finds, scientists say caffeine might be altering your brain activity long after the jitteriness wears off, impacting your quality of sleep in ways you probably don’t even notice. The review, published in the journal Nutrients, analyzed 32 studies spread across over 40 years of research on how caffeine affects the brain during sleep.

In short, the study of several studies found that while you might technically be able to fall asleep, the caffeine is still doing its work, still coursing through parts of your brain, ruining your sleep as you sleep.

Researchers focused on brainwave activity recorded through EEG scans, which track the electrical patterns the brain produces overnight. On the rare occasion you get that kind of deep, restorative sleep that we all yearn for, your brain generates slow, heavy waves that are linked to recovery and memory processing. Across multiple studies, the researchers found that caffeine reduced those slow-wave patterns and replaced them with faster activity associated with being awake.

The Time of Day You Drink Your Coffee Might Be Irrelevant

This means that caffeine isn’t just making it harder for you to fall asleep, but when you eventually do knock out, the sleep is less effective. The review also found that caffeine can interfere with the brain’s ability to recover after sleep deprivation. So if one cup of coffee so messed with your sleep that you wound up sleep-deprived, the brain will no longer be able to compensate for it by increasing deep sleep activity the following night. Caffeine blunts the recovery response, making exhaustion harder to fully recover from even after you finally get a full night’s rest.

Of course, there is a simple logic to all of this. The higher amounts of caffeine you cram into your system, the stronger the disruptions, especially when caffeine is consumed later in the day. But some of the studies in the review found that even a morning coffee left enough residual activity in the body to alter nighttime brain waves several hours later.

Other factors play into this, too. Age, genetics, and metabolism all have a say in how much caffeine affects sleep. Some people are just way more sensitive to it than others.