惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

G
Google Developers Blog
Apple Machine Learning Research
Apple Machine Learning Research
小众软件
小众软件
Recent Announcements
Recent Announcements
阮一峰的网络日志
阮一峰的网络日志
IT之家
IT之家
A
About on SuperTechFans
量子位
Engineering at Meta
Engineering at Meta
B
Blog
The Cloudflare Blog
博客园 - 【当耐特】
Hugging Face - Blog
Hugging Face - Blog
Y
Y Combinator Blog
J
Java Code Geeks
D
DataBreaches.Net
aimingoo的专栏
aimingoo的专栏
T
Tailwind CSS Blog
H
Help Net Security
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
V
V2EX
Stack Overflow Blog
Stack Overflow Blog
C
Check Point Blog
酷 壳 – CoolShell
酷 壳 – CoolShell

cs.HC updates on arXiv.org

From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds What LLMs Must Forget to Teach Effectively: A DIY Approach to Premodern Japanese Language Pedagogy Quantitative Movement Testing: Measuring Patient Movements from a Single Smartphone Video Vision-Language Models Suppress Female Representations Under Ambiguous Input The New Social Image: How AI Competency and AI Proactivity Influence Self- and Peer-Perceptions in the Workplace TUX: Measuring Human--AI Tacit Understanding LLUMI: Improving LLM Writing Assistance for Mental Health Support with Online Community Feedback VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions MetaRanker: Human-in-the-loop Active Ranking for Metalens Image Quality Analyzing Persona Effects in Generated Explanations from Multimodal LLM Agents in Urban Perception First head-to-head comparison of agentic AI applied to the analysis of simulated data of the Einstein Telescope Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering The Timing Dependencies of Trust: Speed, Accuracy, and cBCI Neuro-Decoupling in Human-AI Teams Bayesian Distributional Models of Executive Functioning Visual Matters: Connecting Aesthetic Appeal and Production Quality of Photos, Infographics and Data Visualizations to Credibility of Social Media Posts Data-driven Head Motion Generation through Natural Gaze-Head Coordination Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why Perceptually Lossless Tactile Texture Synthesis with Compact Spectral Envelope Models MambaGaze: Bidirectional Mamba with Explicit Missing Data Modeling for Cognitive Load Assessment from Eye-Gaze Tracking Data CogAdapt: Transferring Clinical ECG Foundation Models to Wearable Cognitive Load Assessment via Lead Adaptation Augmented Analytics and Decision Quality: The Role of Trust among Non-Technical BI Users Faster Completion, Less Learning: Generative AI Reduced Study Time on Math Problems and the Knowledge They Build PaintCopilot: Modeling Painting as Autonomous Artistic Continuation Personality Engineering with AI Agents: A New Methodology for Negotiation Research PULSE: Agentic Investigation with Passive Sensing for Proactive Intervention in Cancer Survivorship Access Timing as Scaffolding: A Reinforcement Learning Approach to GenAI in Education MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysis
Grounded GUI Understanding for Vision-Based Spatial Intel...
Shuqing Li, Binchang Li, Yepang Liu, Cuiyun Gao, Jianping Zhang, · 2024-09-17 · via cs.HC updates on arXiv.org

In recent years, spatial computing a.k.a. Extended Reality (XR) has emerged as a transformative technology, offering users immersive and interactive experiences across diversified virtual environments. Users can interact with XR apps through interactable GUI elements (IGEs) on the stereoscopic three-dimensional (3D) graphical user interface (GUI). The accurate recognition of these IGEs is instrumental, serving as the foundation of many software engineering tasks, including automated testing and effective GUI search. The most recent IGE detection approaches for 2D mobile apps typically train a supervised object detection model based on a large-scale manually-labeled GUI dataset, usually with a pre-defined set of clickable GUI element categories like buttons and spinners. Such approaches can hardly be applied to IGE detection in XR apps, due to a multitude of challenges including complexities posed by open-vocabulary and heterogeneous IGE categories, intricacies of context-sensitive interactability, and the necessities of precise spatial perception and visual-semantic alignment for accurate IGE detection results. Thus, it is necessary to embark on the IGE research tailored to XR apps. In this paper, we propose the first zero-shot cOntext-sensitive inteRactable GUI ElemeNT dEtection framework for virtual Reality apps, named Orienter. By imitating human behaviors, Orienter observes and understands the semantic contexts of XR app scenes first, before performing the detection. The detection process is iterated within a feedback-directed validation and reflection loop. Specifically, Orienter contains three components, including (1) Semantic context comprehension, (2) Reflection-directed IGE candidate detection, and (3) Context-sensitive interactability classification. Extensive experiments demonstrate that Orienter is more effective than the state-of-the-art GUI element detection approaches.