PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
Abstract
PreView is an auxiliary preprocessor that enhances GUI grounding by providing localized zoomed views and visual-textual target cues to frozen grounding models, improving accuracy without retraining.
GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.
Community
Hi everyone, author here! ๐
Can we improve GUI grounding by better preparing what the grounding model sees?
We introduce PreView, an auxiliary preprocessor that combines task-relevant cropped and enlarged views with visual and textual cues about candidate targets. The idea is simple: zooming helps reveal small controls, while candidate-specific cues help disambiguate which control the instruction refers to.
A trained lightweight auxiliary model prepares these inputs, while the grounding model stays frozen. We evaluate PreView across five benchmarks and three grounding models.
Happy to discuss the approach and hear your feedback!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding (2026)
- ConClick: Element-Wise Contrastive Learning Improves GUI Grounding (2026)
- RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection (2026)
- GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning (2026)
- AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding (2026)
- Recurrent Latent Visual Search for GUI Grounding (2026)
- Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.33645 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper