This is a response to Minas Karamanis’ excellent post on AI. You can find it here.

I’ve been working on my physics analysis for about a year. Part of my PhD is looking at a particular particle (the eta meson) and trying to make a better measurement on the probability it decays in a particular way (into two muons), among other information about it. The process for doing a typical analysis can be condensed to: get data, extract decay info, characterize background, apply fit, handle uncertainties, systematics, and efficiencies, report measurement. My analysis is a standard flavor physics one, relatively straightforward and well-defined, yet even this is hard for me. I constantly make mistakes, waste days (or weeks or months) figuring out how to make formal requests or add a missing line of code. It’s frustrating, it’s hard, and it’s all-consuming. All my time is spent in the what and how; barely any time gets spent on the why because I just don’t have the time. I try to maintain a good work-life balance, get proper sleep, and stay active, but my funding comes from a project outside my analysis, so working on two projects regularly takes me past the idealized forty hours per week. If I also wanted to understand the underlying physics behind what I do, physics would have to be everything. It might need to be everything to even try to be one of the greats. Most of the time I honestly forget why I’m even doing this. Don’t even ask me what the Schrödinger equation is, at this point. Luckily, my undergraduate advisor warned me of the nature of the beast before I started, that a PhD begins as a sprint then becomes a marathon.

Recently I started work on doing my fits. I’m still actively working on this, but the idea is that you want to try and approximate the number of signal versus background in a data sample so that you can count the amount of your decay that occurs, take a ratio of two different signal counts, and find the probability your decay occurs (with some level of uncertainty). I don’t know the first thing about performing a fit with a data sample via code, so as I began work I thought: “Well… doesn’t hurt to throw it into Claude.” In my mind, it’s essentially zero effort and, when it inevitably fails, at least I’ll have a starting point. Plus, writing a prompt forces me to create a well-defined list of tasks for myself.

It worked. I spent some time typing it out, hit enter… and within minutes I had a working script that correctly generated both a gaussian fit and double crystal ball fit (with pull plots, parameter values and uncertainty, estimated signal versus background counts). Sure, I initially had to make a few changes to correctly load the data from ROOT, but it worked. I couldn’t believe it. For all its purported issues, AI created a complete working script using Python RooFit. While I didn’t understand the code, I’ll admit the temptation to just pass it off to my advisor like it was nothing and move on was high. None of this is really physics. Using AI saved time and got me closer to my PhD. That’s good, right?

And that’s the moment you fail yourself. It might even be when the world fails you. Minas articulated what I long lacked the words to express but always felt. When I first entered grad school, I said I was here “to learn how to learn, to learn how to think, and to learn how to ask the right question and investigate it”. That’s a good reason, I think, but Minas went even further to point out that what matters is the process, not the results: “…the development and application of methods, the training of minds, the creation of people who know how to think about hard problems. If you hand that process to a machine…You’ve removed the only part of it that anyone actually needed.” Admittedly, this may sound idealistic. After all, funding agencies are probably going to give funding to the groups that produce the most research output. Publish-or-perish is real. So on one hand, we need to understand what we’re doing, but we also need to keep up with everyone else who may be twice as productive using AI tools, who are less concerned with such questions. Will this come at the cost of a generation of young minds?

Progress, indeed.

An additional point: Minas’ argument works very well in physics. I do believe the results are secondary to the cultivation of researchers, but this is not necessarily the case everywhere in academia. I have graduate student friends who live in a very different world, where results are everything, progress is mandatory, and student learning, student health, and eventual graduation are not the concern of the principal investigator. Are these people bad? I don’t think so. The results they produce are practically useful. It saves companies money, contributes to national defense, or saves lives. Moreover, these PIs are simply following incentives. The behavior encouraged is the behavior to expect. Minas again puts it well: “The more papers you produce during your PhD, the better your chances of landing a competitive postdoc, which improves your chances of a good fellowship, which improves your chances of a tenure-track position, each step compounding the last (so many levels, almost like a pyramid).” His answer is that the logic of offloading thinking to AI works until it doesn’t, until eventually the career ladder demands something greater than what AI can ever hope to provide. Like everything else, I agree with this and plan to apply it in my own life, but I also wonder at humanity’s aptitude for long-term delayed gratification like he does, to day after day deny the impulse of turning to a chatbot to make one’s life easier when they’re very overworked and very stressed, all in the name of some eventual promised payoff.

I wish I could think of a policy that would fix this, but the truth is that a lot of this comes down to the group or individual. I’m fortunate to have an advisor that cares about cultivating minds and being an advisor his students can look up to. I regularly find myself and others naturally working harder out of admiration and respect for my advisor. They want to be a better researcher. Look, I use AI every day for all kinds of things. I experiment with its use cases, wrestle with how to use it responsibly, and sometimes feel a twinge of guilt when I ask it to write some code. This might not be as positive or palatable of a take as others might have, but this has been my experience so far. Generative AI is no doubt a transformative tool¹. There’s no going back. All we can do now is work to understand our relationship with this new technology. Unfortunately, society, culture, and most tragically government and policy are always slower than technology. I still don’t think we as a society have even adapted to the internet, but that’s a whole different topic.

Going back to this fit analysis code, AI created a working fit analysis script in Python… or did it? This is the second issue with using AI in research. I’ve used AI to quickly write something I’ve done before or could easily do but can’t be bothered to do, i.e., boilerplate code. I think that’s fine. We look up physics concepts (on Wikipedia, or even use the AI search answer sometimes) rather than go to the library, and no one seems troubled by this. When it created the fit code for me, I felt in my gut that a line was being crossed. I told my advisor, and we had a good conversation about AI use in research. For any grad students reading this, if you have a good relationship with your advisor you should also have a conversation. The agreement we came to was: I will tell him when I use AI for something, he will tell me if something is important for me to do myself.

I think some rough guidelines and takeaways are emerging for me. This list will change, but I think a good starting point for responsible AI use is as follows:

1. If you can’t do it yourself, don’t have AI do it for you.
2. Try to get chatbots to ask you leading questions rather than perform tasks. Ask for sources. Check those sources.
3. Do not trust the output. Think of AI as 1,000 really enthusiastic interns.
4. AI is not a catch-all for all problems. Focus on what LLMs are good at: auto-complete, auto-format, file conversion, etc.

Applying this to the recurring fit code example, I couldn’t have written that code on my own, so I am now going back and (1) rewriting the code in iminuit (I avoid RooFit like the plague), (2) creating a full LaTeX write-up on fitting, and (3) volunteering to give a seminar on fitting in high-energy physics. That’s probably more than necessary, but I’ve found that the amount of time it’s taking to understand the code the AI wrote is about equal to how much time I would’ve had to spend figuring out how to code it myself. Goes to show that what we need to understand something hasn’t changed, even if the tools do. It reminds me of the learning pyramid, which roughly goes like: teaching others > practice by doing > discussion group > demonstration > audio-visual > reading > lecture. Despite the gulf of time, the two-thousand year old proverb still holds: docendo discimus, or “by teaching, we learn”.

I don’t consider this a single story nicely wrapped up in a single post. It’s an evolving discussion. No doubt some of what I said will prove wrong. That’s hard for me to accept, that I’m putting something out into the world that will almost certainly be partially wrong, if not totally wrong. Some people will agree with me, some disagree, some strongly object. I don’t have a comment section or forum on this website (yet), but I’d be happy to email back and forth one-on-one or in a larger email chain. You can reach me at [email protected].