Interacting with Multi-Robot Systems via Mixed Reality

Florian Kennel-Maushart, Roi Poranne, Stelian Coros

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Mobile robots are becoming safer and more affordable, and their presence in the workspace is increasing. However, many tasks that involve reasoning, long-term planning or human preferences are still hard to automate. While some solutions in specialised areas slowly emerge, an alternative to full autonomy can be to actively leverage intuition and experience of human operators. To do this, suitable interfaces and modes of interaction have to be explored. Inspired by Real-Time Strategy games, we implement a Mixed Reality interface that can be used with either a Microsoft HoloLens 2 headset or a tablet. The interface allows users to interact with multiple mobile robots simultaneously. We conduct a user study to compare the headset and tablet versions of the interface in different scenarios inspired by a real-world construction setting. We show that, while performance and preference of interface are dependent on the task and the complexity of the required interaction, users are able to solve non-trivial tasks on both platforms using our system.

Original languageEnglish
Title of host publicationProceedings - ICRA 2023
Subtitle of host publicationIEEE International Conference on Robotics and Automation
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9798350323658
StatePublished - 2023
Event2023 IEEE International Conference on Robotics and Automation, ICRA 2023 - London, United Kingdom
Duration: 29 May 20232 Jun 2023

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)1050-4729


Conference2023 IEEE International Conference on Robotics and Automation, ICRA 2023
Country/TerritoryUnited Kingdom

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Electrical and Electronic Engineering
  • Artificial Intelligence


Dive into the research topics of 'Interacting with Multi-Robot Systems via Mixed Reality'. Together they form a unique fingerprint.

Cite this