JetBrains Research
Research is crucial for progress and innovation, which is why at JetBrains we are passionate about both scientific and market research
The main message
Advances in AI are already single-handedly changing how we interact with technologies. This moment might become the first interaction revolution in roughly 60 years, since the mouse-and-windowing paradigm crystallized in the 1970s and 1980s. For us at JetBrains, the most interesting part of this interaction is related to code. LLMs have introduced the conversational format to software engineering. Something that might appear no less consequential than AI several years from now is extended reality (XR) hardware, and how it is (slowly) becoming good enough to be taken seriously as a working medium. XR is relevant for AI not merely as a more immersive output device; it is, even more importantly, an unprecedentedly rich input surface: using gaze, hands, voice, head pose, body posture, spatial context, and even physiological signals. Altogether, with all these new types of data, it enables a more efficient, focused, and personalized multimodal human-AI experience. Crucially, all these inputs are available without a dedicated lab.
When the input richness of XR is paired with modern AI, the result is a qualitatively new interaction substrate, one we’re just starting to figure out how to use. Our Human-AI eXperience (HAX) team at JetBrains Research is now studying what this means specifically for tech creators. By tech creators, we mean software engineers, of course, but also hardware engineers, UI/UX designers, and data scientists, as the question is broader than any one role.
What we asked
To anchor our exploration of what XR can bring to HAX empirically, we conducted semi-structured expert interviews. Thirteen senior researchers and practitioners from leading academic institutions and industry labs (including groups at Cambridge, Aarhus, Stuttgart, and Meta) participated. We analyzed these interviews using the classic Braun and Clarke’s (2006) six-phase approach. The interviews continued until we reached thematic saturation: when three consecutive interviews introduced no new parent themes.
We were interested in understanding:
1.What are the needs and recurring pain points of tech creators that multimodal AI-XR tools could plausibly address?
2.Which multimodal interaction techniques are most promising for addressing those needs, and what might the resulting tooling look like?
3.What are the main constraints – technical, cognitive, organizational – for building such tools today?
What we found, in brief
We’ve identified more than 150 topics of interest from the answers to our interview questions. All these topics were then clustered (first independently by two of the authors and then updated until a consensus was reached) into five overarching themes.
The initial codes and thematic clusters of AI-XR for multimodal Human-AI Experience
156 consolidated codes across 5 themes
New interaction paradigms for spatial computing Future of interactions Natural interaction Multimodal context and input No preferences in multimodality New inputs and UI placement Matching input with the task Explicit and implicit interactions AI-based hand and gesture tracking AI-based voice control AI-based body language AI-based haptics and tactile feedback Gaze control No controllers for XR Embodied interaction Personalization of inputs Spatial reasoning Wearables for XR AI-XR communication and telepresence Switching perspectives for communication Human communication-inspired interaction Better UI Visual overlaying Mobile/on-the-go interaction Flexible interaction patterns Feeling depth in XR Audio is underused Vocalization will not be enough Lack of freedom for interaction Gorilla arm / interaction fatigue Midas touch problem “No interface” / zero-UI paradigm Constraints can aid efficiency 2D metaphors are poor fits for 3D
AI as a context-aware co-creator and mediator Agentic AI AI for understanding context/user Inherent personalization of AI Personalized XR through AI AI lacking personal/user context AI for content creation Communicating/decoding intent AI for disambiguation Immediate interactive context/content Interactive environment Directing attention Attention as a scarce resource Digital twinning Semantic graphs from visual scenes Environmental sensing and physical-world understanding Multimodal data collection New data for learning in XR AI for tracking body and behavior Humanlike behavior of AI in 3D Control of environment AI-XR guidance and explanation XR for navigation Always-on/ambient AI-XR Continuous AI-XR experience Proactive/predictive AI-XR Ubiquitous/situated computing in XR Situated/physical-world computing and augmentation Diminished reality AI-XR augmenting people Merging physical spaces Rule-based systems and predetermined patterns limit AI Incomplete AI models of digital/physical context
Feasibility, adoption, and ecosystem readiness Technological and hardware barriers Technological availability Computational requirements Insufficient bandwidth Cost barriers New technical challenges New infrastructure for AI Interaction precision and resolution limits Calibration vs. quality trade-off (eye tracking) Responsiveness of input and system Trade-offs of AI-XR XR proving its value XR not improving AI Barriers in personal performance and user experience Ergonomics Intrusive wearables Aesthetics of XR Social acceptability XR for public settings XR UI for long sessions Frustrating experience Initial friction of adoption Easy step-in/step-out from XR workflows Fast restart Workflow transition/switching friction AI-assisted workflow transitions 3D metaphors 3D is not always better than 2D Text is bad outside desktop Text-centered ecosystem XR UI for tech is the same as general XR UI Lack of specificity of AI for technical work SE in XR is niche SE is more demanding for interaction Lack of human connection Interoperability and shared XR standards Content creation intertwined with coding
XR as a situated, persistent workspace Extended workspace for coding and engineering Mobile workstations XR as primary computing/display platform Personalized organization of the XR workspace XR/AI-XR visualization for data and complex systems XR for robotics/teleoperation Teaching and understanding 3D XR for teaching and training AI-XR prototyping and 3D prototyping AI-XR sketching, drawing, and design AI for general productivity AI for deep research AI as counterfactual / devil’s advocate AI-XR collaboration XR for human-AI interaction (HAX) AI-XR multi-tasking XR avatars and digital self-representation AI-XR video creation/editing End-to-end creation in XR Iterative creation in XR AI-XR for creating variations Tools for 3D engineering Tools for technology creation Tech creation when no more applications Writing code is not the core concern anymore VR for 3D printing XR for game development/gaming 4D videography Immersive tourism Deep immersion
Privacy concerns / convenience trade-off Security in XR (next-level phishing) Ethical concerns Governmental regulations AI regulations do not work in advance Rights, authorship, and ownership AI transparency and black-box concerns XR for explaining AI and transparency Alignment Preserving human-in-the-loop Technology overreliance Overuse of AI features (e.g., beautify) Potential exploitation through XR Commercialization of AR/XR environments Societal changes Social concerns Changing emotional landscape 24/7 technology use Constantly-on devices Digital copies, twinning, and identity risks
AI-XR for creation and professional work. This is the cluster most directly related to our main question: What is the future of technology creation? Here the main questions center on the productive use: an XR environment as a place where knowledge workers, designers, and engineers actually build things. The emerging interaction insight is that the boundary between authoring, coding, and designing is dissolving in XR, and AI is accelerating that dissolution. One unexpected topic that stood out here and genuinely surprised us was framing the XR for robotics as a potential sandbox for training the next generation of machine learning models. The idea is the following: if you put simulated robots into a highly controlled virtual environment with rich structured multimodal data, and then create an interaction loop within this virtual environment, you can then use that loop to generate the kind of physical interaction data that today’s language and vision models mostly lack.
AI’s impact on coding is, at this point, well-documented and widely discussed. Together with other researchers, our group is trying to figure out these changes through surveys, longitudinal studies, and analyses of developers’ evolving needs. However, we believe another big change is coming. Consider the output we already see: Consumer-grade XR hardware is in the middle of a rapid generational shift.
The Apple Vision Pro is currently the most visible example of this push toward high-resolution spatial computing (although it’s now on pause, it’s still a big thing for the industry). However, other devices like PICO or the ultra-lightweight BigScreen Beyond are proving that form factors and devices are diversifying rapidly. Google is also pushing boundaries in this space, notably with their work on vibe-coding XR using XR Blocks and Gemini. While the ecosystem around these gadgets is still sparse, it is growing very quickly. None of these are perfect, and none are a complete replacement for a workstation. The trajectory is unmistakable, however: headsets are getting lighter, displays are sharper, tracking is more reliable, and ecosystems are thicker.
In parallel (and most easily overlooked), the input side is exploding. Eye tracking is becoming a default rather than an accessory. Hand tracking has reached the precision needed for controller-free interaction to be a viable tool for many tasks. Beyond that, a generation of new sensors is moving from research lab prototypes toward future integration: ear-EEG that captures brain activity from the auditory canal, electromyography-based silent speech interfaces, fine-grained physiological monitoring (e.g. heart rate variability, pupillometry, galvanic skin response), and high-resolution facial and posture tracking. The image below shows examples of new sensor types from Tang et al (2026).
Take capable headsets and an unprecedentedly broad input bandwidth together, add modern multimodal AI, and you arrive at something genuinely new. This is the territory our project is trying to map.
To calibrate our research against the state of the field, a researcher from our team attended IEEE VR 2026 in Daegu, Korea, one of the leading conferences covering virtual, augmented, and mixed reality. The picture that emerged was useful both for what it confirmed and for what it revealed to be complicated.
The picture confirmed that XR is no longer primarily a graphics or hardware community. The program — paper tracks across three days and four parallel sessions — covered haptic feedback and rendering, multimodal perception, locomotion and redirection, collaboration and social XR, presence and embodiment, training and education, ethical and psychological dimensions of virtual identity, immersive analytics, input devices, tracking and sensing, and software architectures. Moreover, 18 workshops spanned highly specialized, interdisciplinary topics, such as:
• Healthcare and medicine — XR Health, XR4OR for operating rooms, and XR-MED
• Human cognition and physiology — GEMINI on gaze and eye movement, XRMemory on spatial memory, and WISP on immersive sickness prevention
• Collaborative and social XR — SIC-XR and NESXR26 for networking in the Metaverse
• Industrial applications — XRIOS for occupational support
• Education — KELVAR for K-12 embodied learning
The sheer scope of the sessions underscores how far the field has moved from being primarily a graphics or hardware problem: this is now fundamentally a human-centered research community. The strongest theme across the technical program was that the physical interface needs to be re-imagined. The “Minority Report” fantasy of constant mid-air gesturing is, in practice, fatiguing, imprecise, and visually expensive.
Several 3DUI papers explicitly pushed in the opposite direction. BlanchTouch detects firm touch by measuring the physiological color change in the fingertip under pressure. FanType tackled text entry, the most stubborn problem for any knowledge-work scenario, with a thumb-centered fan layout and intention-aware disambiguation tailored to actual hand geometry. SurfaceXR anchors interactions to physical surfaces — desks, walls — treating the real environment as part of the interface rather than a limitation. The following images illustrate the concepts outlined in these papers.
For coding specifically, this matters immediately. Code is still text; specifically, dense, high-precision text that is read, written, and edited over long sessions. Better headset resolution helps, but the more binding constraint is whether the interaction model sustains that kind of work without fatigue. The most credible near-term architecture, on the evidence of this year’s program, is hybrid: a real keyboard on a real desk, with spatial overlays where they earn their place.
Another theme worth flagging is large-scale shared infrastructure. VERA (from UCF’s SREAL group) is a representative example. It is a community research infrastructure pairing hardware, software, and a participant pool to enable controlled human-subject XR studies that run online rather than in dedicated labs. Lab-based XR research is slow, narrow, and difficult to replicate; infrastructure like VERA is the kind of connective tissue the field needs to graduate from demos to evidence.
AI as a contextual layer. The core idea across this second cluster is based on the fact that XR generates enormous amounts of contextual data (gaze, location, body, task state), and that AI is what makes that data actionable in real time. According to our experts, the most promising direction today is personalized multimodality that decodes implicit intent. Combining gaze, gestures, voice, and context might be used infer what a person is about to do, not just execute what they explicitly say. Another important topic of discussion here was that the hard bottleneck is shifting from technology to cognition. Building the contextual layer is already becoming technically possible, but the hard part is shaping it to meet human needs. Hardware will keep improving, but human attention is finite. Several experts argued that the cognitive ceiling will soon become more limiting than the hardware ceiling, and the AI-based context may become as important for creating diminished reality (selectively removing information) as for augmented reality.
New interaction paradigms. This third cluster is the most populated and is related to AI as a contextual layer. With the rich context available and new AI-based systems and tools becoming more widely adopted, traditional user experience might no longer be sufficient, demanding reinvention of interaction scenarios. The tension here is between the expressiveness of new inputs (i.e. you can gesture, look, speak, or move with varying amounts of information with different inputs) and the reliability needed for new paradigms (i.e. the system misunderstands, fatigues you, or fires unintentionally). The 2D desktop paradigm cannot be just ported to 3D; something new has to be invented. Repeating windows, mice, and folders in 3D is the failure mode. The field is still searching for its “desktop metaphor” equivalent for spatial computing.
Software engineering in XR is not uncharted territory, though it remains thin. Several projects have leveraged immersive 3D environments to support spatial comprehension of code structure: Primitive converts source into navigable 3D structures for distributed teams; ExplorViz and CodeCity XR use the “software city” metaphor to let developers physically navigate architecture. Educational tooling has gone further than production tooling: BlocklyXR and Cubely integrate tangible manipulation into programming exercises; Hack.VR and Imikode move coding instruction into game-like immersive environments. Evaluations (Kao et al., 2020; Sunday et al., 2022; Hedlund et al., 2023) point to high engagement and motivation, although formal learning-gain evidence remains preliminary.
Adjacent to this is a small body of work at the XR–machine-learning intersection — visualizing neural networks in VR for comprehension (Meissler et al., 2019), or providing immersive interfaces for non-experts to train ML models (Hilton et al., 2021). Across the literature, the pattern is consistent: end-to-end production coding in XR is not yet mature (or was not real a couple of years ago, things seem to have started changing), but specific workflows, i.e. comprehension, onboarding, collaborative review, and education, already offer demonstrable value.
We think it is worth noting on the demand side that in the JetBrains 2024 Developer Ecosystem Survey, 49% of respondents said they “would love to try” a VR headset for coding. They cited visualization of complex data (42%) and multi-screen virtual workspaces (40%) as the most attractive reasons. The interest is real; the question is what to build.
Our hypothesis going in was that the interesting frontier is not “XR for coding” or “AI for coding” considered separately, but their intersection. Specifically, we thought that the interesting frontier is actually the part of that intersection that takes the cognitive nature of software engineering seriously.
The reasons for this are the following. Code is unusual: highly abstract, yet productive of very tangible results. Developers constantly move between local detail and global structure, holding execution paths, dependencies, naming conventions, architectural decisions, and unfinished hypotheses in working memory. We barely understand how that works on a flat screen, let alone immersively.
The promise of AI in this picture is not as a backend service but as a spatially aware co-inhabitant of the workspace. Mar Gonzalez-Franco‘s IEEE VR keynote made the point sharply: reality is becoming blended, and the formats that machines can understand and generate are changing faster than our interfaces are.
The next step is to embed world-understanding directly into the interface. Recent papers showed early evidence of this direction. For example, Shape-Shifting Splats and LIVE-GS hook LLMs directly into Gaussian Splatting pipelines to generate not just static 3D images but the physical and interactive behavior of scenes; the GenAI workshop showcased reconstructions of 360-degree memories from text or old 2D footage; and Anthony Steed’s Ubiq-Genie demonstrates generative prototyping of social XR scenes on the fly (see image below).
A persistent thread across these projects is referential interaction, or how humans and AI converge on what they are talking about in shared space. In study after study, gaze is emerging as the most efficient channel for directing intelligent systems — more accurate than pointing and less cognitively expensive than verbal commands.
This connects directly to a piece of work we found especially provocative: Niknam et al.’s Oopses and Ohs, which used 18 months of longitudinal eye-tracking data to predict human errors and cognitive slips. The natural question for our research is whether equivalent models can be built for developers. For example, whether gaze patterns preceding mistakes in abstract reasoning are detectable, or whether an interface might intervene before the bug is written.
For our work, the challenge is more abstract, as code is highly abstract. If generative AI can reconstruct 3D memories or build virtual objects, can it generate spatial “affordances” for code? By referencing concepts like Google’s XR-Objects, we need to figure out what an algorithm looks like in space, to translate abstract logic into paradigms developers can naturally manipulate.
Adoption frictions. This fourth cluster captures both technical (e.g. hardware, latency, precision of existing solutions) and social/cultural debt (e.g. XR is still weird to wear, awkward in public, not proven valuable enough to justify switching). Any new viable multimodal interaction paradigm has to address the frictions people experience to succeed.
Ethics, control, and the human position. The fifth cluster is related to the human consequences of more widespread use of AI-XR systems. Because XR and AI rely on sensing bodies, environments, behavior, and attention, they raise concerns about privacy, agency, social norms, regulation, authorship, and well-being. The essence is that the future of any new future AI-XR interaction paradigm depends not only on technical possibility, but also on whether these systems remain acceptable, safe, and humane.
Sketches of what could be built
Several ideas for potential multimodal prototypes were inspired by interviews we conducted. We sketch them in the following.
Multimodal input
Because AI can now process multimodal inputs through XR advancements, we can leverage biometric data like gaze and spatial context to fundamentally reshape how developers review code. For example, we can envision the following scenarios:
- Gaze-Triggered Socratic Code Reviews: As a developer tries to comprehend a complex code block, the multimodal HAX system will monitor cognitive load using predictive data (such as pupil diameter, gaze dynamics, or blink models) so that it can automatically initiate a guided, Socratic dialogue when the developer gets stuck, rather than waiting for a typed prompt.
- Asynchronous Immersive Walkthroughs: As a code reviewer is processing a pull request, AI can generate a narrated 3D walkthrough of the change so that the developer can experience the diff as an active, first-person replay. This significantly lowers the cognitive cost of review—an essential evolution as developer workflows shift from writing code to reviewing larger volumes of it.
Multimodal output
With the possibility of AI expressing outputs across multiple modalities via XR, developers are no longer restricted to flat monitors. Instead, they can interact physically and spatially with their code, immersing themselves in the HAX ecosystem. The potential prototypes can look like:
- Blended Coding Environments: As a developer is trying to grasp how a massive new pull request affects the broader system architecture, the AI-XR system can combine a physical keyboard on a tracked desk for tactile speed with a 3D dependency graph projected directly above it. Hand tracking can allow the developer to pull functions apart in space to inspect their connections, offering an entirely new, futuristic mode of interaction.
- Programmable Affordances: As a technology creator is working on a new system, AI can translate abstract algorithms into XR objects with distinct physical properties. An expensive function might manifest as a “heavy” block, while tightly coupled modules appear as a “tangled” shape. As developers manipulate these objects spatially, the underlying code rewrites itself accordingly, providing a deeply embodied development experience.
Behind all of this lies an open question, and it is the one we keep returning to as builders of integrated development environments: Is it time to rethink how code is represented and interacted with? Coding is changing today due to new intelligent and agentic possibilities that AI is already offering. But there are some core things: code is highly abstract yet produces concrete consequences; it bridges ideas and reality in a way few other artifacts do. Whether spatial computing offers better metaphors for that bridge (or whether text remains, after everything, the right interface) is genuinely impossible to predict right now. We are not sure about the answer, but we do think that the point of asking it is not to answer it immediately. Rather, the point is to ensure the question’s salience while the next generation of tools is being built to be prepared to study these new modes of interaction and respond to the upcoming change.
Subscribe to JetBrains Research blog updates