Hands-Free Display Focus System with Mixed Reality

University of Pittsburgh researchers have developed a novel mixed reality (MR) system to improve efficiency and focus during split-attention information-mediated tasks. This approach allows users to display and control information from computer displays using an MR head mounted display (i.e., MR headset) while also focusing on tasks requiring fine motor skills in the physical world. Information containing instructions or other complex information can be displayed as a hologram through a virtual interface to the user and controlled using hand gestures. Using an MR headset, complex tasks can be completed with an unobstructed view, while simultaneously viewing instructions. The use of MR may improve efficiency in split-attention tasks and would have various applications in healthcare, engineering and the military.

MR headsets can be used to select and view information from a computer or other digital source, while simultaneously viewing and completing complex tasks in the physical world. This technology could improve workflows and task performance by removing the need to split focus between a computer and a physical task. Additionally, users would no longer need to be tethered to a computer to regularly access material. 

Description

MR allows users to interact with virtual information in the physical world and is growing in its commercial availability and utility. An area that may benefit substantially from MR is complex information-mediated tasks where, traditionally, information may be on a fixed screen and workers need to split attention between the physical task and a screen fixed in another location. This split in attention limits their ability to focus on tasks and required information simultaneously, negatively impacting on task performance. This MR approach could allow users to simultaneously display segments of relevant information from a computer (e.g., images) in their field of vision while simultaneously focusing on the physical task, reducing split attention and cognitive load on workers while improving workflows, efficiency and task performance.

Applications

• Medical education and surgical procedure
• Military functionality including bomb disposal

Advantages

Many complex tasks require operators to split focus between the physical task and visual displays (e.g., car mechanic viewing diagnostic data while repairing an engine or surgeons viewing diagnostic images during treatment). Switching focus between a physical task and visual information, often at a fixed workstation or shared resource, can limit workflows (e.g., user may physically have to move to see screens, or risk contaminating keyboards with soiled hands when manipulating a view).

This novel MR approach can allow users to simultaneously view information streamed from a computer as a hologram over the physical environment through an MR headset. Using pinch and pull hand gestures, the information to be viewed in the headset can be selected and manipulated. This MR approach will allow users to view information ‘hands free’ and untethered from a fixed workstation while performing complex tasks, improving workflow through reduced task switching.

Invention Readiness

A MR system has been developed using a commercially available MR headset. A computer screen is captured and streamed to a headset using WebRTC. Users can select portions of the screen to display as a faint hologram using hand gestures. Users spoke favorably of the ‘hands free’ nature of MR, increased task attention and potential benefits in work environments.

IP Status

Patent Pending

Related Publication(s)

Hurt, G. J., Khan, T., Kann, M., Andrews, E., & Biehl, J. (2024). EndovasculAR: Utility of Mixed Reality to Segment Large Displays in Surgical Settings. 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 1059–1060. https://doi.org/10.1109/vrw62533.2024.00326