Skip to content

New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens - #63

Merged
Luodian merged 4 commits into
EvolvingLMMs-Lab:mainfrom
hunterheiden:hsh/new_task/screenspot
Apr 26, 2024
Merged

Luodian merged 4 commits into
EvolvingLMMs-Lab:mainfrom
hunterheiden:hsh/new_task/screenspot

Conversation

@hunterheiden

Copy link
Copy Markdown
Contributor

This PR adds the ScreenSpot benchmark from the SeeClick paper. This benchmark was designed for grounding (i.e., referring expression comprehension (REC)), but can also be inverted to yield an instruction generation (i.e., referring expression generation (REG)) benchmark. The data domain is screens: mobile, desktop, and web screenshots.

This PR simply adds ScreenSpot, mirroring RefCOCO style datasets for REC/REG.

Attached results.json shows that LLaVA-v1.5-7B achieves:

  • 9.7% center accuracy (for grounding)
  • 0.97 CIDEr (for instruction generation)

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants