CHING-YI TSAI

HapticSeer
A Multi-Channel, Black-Box, Platform-Agnostic Approach to Detecting Video Game Events for Real-Time Haptic Feedback
ACM CHI'21 HONORABLE MENTION | 15 APRIL 2021

A key obstacle for haptic devices is the closed-source nature of most commercial games, which hinders the creation of custom devices for high-quality gaming experiences. To address this issue, we developed HapticSeer, the first software framework designed to monitor multi-channel signals like video, audio, and controller I/O in real-time for game event detection. My role in this project involved extracting game UI using computer vision techniques. HapticSeer can successfully identify a variety of game events across the top 20 commercial games with a latency of just 50 ms, leveraging signals from multiple channels. This framework is open-sourced and was recognized with an Honorable Mention Award when presented as a paper at CHI 2021.