Literature Review of Mixed Reality Research

In the global context, while mixed reality has been an emerging concept for years, recent technological and scientific advancements have now made it poised to revolutionize industries and daily life by offering enhanced functionalities and improved services. Besides reviewing the highly cited papers in the last 20 years among over a thousand research papers on mixed reality, this systematic review provides the state-of-the-art applications and utilities of the mixed reality by primarily scrutinizing the associated papers in 2022 and 2023. Focusing on the potentials that this technology have in providing digitally supported simulations and other utilities in the era of large language models, highlighting the potential and limitations of the innovative solutions and also bringing focus to emerging research directions, such as telemedicine, remote control and optimization of direct volume rendering. The paper's associated repository is publicly accessible at https://aizierjiang.github.io/mr.


Introduction
The last few years have seen the rapid development of Metaverse concept, particularly with the emergence of Large Language Models (LLMs) like ChatGPT.These large language models hold the potential to significantly enhance human-computer interaction, propelling it to new levels of sophistication and capability.As an important stream of future research in human-computer interaction (HCI), mixed reality (MR) demonstrates its full potential in leveraging LLMs within the modern era.Moreover, concurrent with hardware enhancements, various innovative and cutting-edge MR devices with remarkable advancements such as Microsoft HoloLens series, Magic Leap series, Meta Quest series, Lenovo ThinkReality VRX, Varjo XR, Google Glass and Apple Vision Pro, etc., have emerged.These are all evident facts depicting an extensive potential in MR application and research.However, based on the review towards recent papers published in the last 2 decades including the year 2023, we found that the research regarding MR is declining since 2020.The annual publication rate (Figure 1) for MR research peaked in 2020, coinciding with the introduction of the Metaverse concept during the COVID-19 pandemic, followed by a subsequent decline in 2023, nearly aligning with the publication rate observed in 2016.Does this imply that MR research is experiencing a slowdown, reaching a technological bottleneck?If so, what potential applications could expedite academic research?Based on such questions, this paper discusses highly cited MR research papers and recently published scholarly works focusing on MR. (2020) provides a comprehensive overview of MR devices.Previous surveys and reviews extensively featured advancements in hardware and software within the MR domain, emphasizing the prevalent industry challenges.However, there remains a dearth of studies that articulate how MR, in turn, expedites the advancement of research pertaining to its underlying technologies and theories.Rather than reiterating the recognized definitions of MR and listing features of prevalent MR instruments, this paper directly focus on the highly cited and recently published academic papers on MR, aiming to uncover potential benefits of MR for future research.

Search Strategy
Utilizing the software Publish or Perish, an efficient search tool for researchers, our initial step involved keyword-based searches spanning from 2003 to 2023.We initiated our study from the year 2003 to track the longitudinal development trends within the past two decades.It was intriguing to observe that despite the absence of much of the current hardware ten years prior to our starting point, research in this domain has continued to persist.Utilizing specific title keywords in Google Scholar, we retrieved pertinent papers, enabling a comprehensive overview of the field's evolution over the course of twenty years.Sorting these papers by their citation count facilitated our comparative analysis.Through sorting by their citation count, we were surprised to find that the highly cited papers predominantly originated from diverse fields such as education, medicine, business, and management.Upon closer examination of each collected paper, they focus more on conceptual designs and definitions rather than delving into the technical aspects of MR.To refine our study's scope and prevent redundancy, we manually curated highly cited papers within computer graphics, computer vision, communication and networks, and other areas like education and medicine, excluding duplicate work that had already been covered by other surveys and literature reviews.
Among the selected 100 highly cited papers from 2017, a period marked by the escalating research interest in MR, the majority focus on discussing the application of MR in various fields rather than delving into the indepth research underlying its advanced technologies like spatial computing, graphics and rendering, computer vision, pattern recognition, and other pertinent components, indicating that MR is actually just a concept behind the customer-oriented applications rather than a specific technology or a set of theories that can be deeply researched by the scientists and scholars (Figure 2).Research in computer science often focus on a specific topic at least in the stage of Technologies and high level of research cannot be conducted without certain abstraction into the stage of Theories.It is most common that the industry proposes the concepts and designs of the customer-oriented applications from the products, which later be treated as a demand to the academics, and the researchers dig deeper into the technologies behind, and scientists dig much deeper into explore their theories and make optimizations.From Customer-oriented Applications, which are directly accessible for end-users, to the core principles and Theories primarily investigated by scientists in relevant fields, the entire spectrum plays a crucial role in shaping a comprehensive high-tech product.

Literature Review
Since 2017, the predominant choice in academia of MR has been the utilization of HoloLens as the primary device to materialize research concepts and ideas, establishing it as one of the renowned MR devices widely recognized within both academic and industrial spheres.The majority of the investigations conducting experiments and creating applications in recent years originate from the gaming industry and medical science, exceeding involvement in other domains.It is evident that significant opportunities are present for immersive simulations within gaming content as well as clinical and medical scenarios.
Significant benefits have been discerned in utilizing the HoloLens for medical applications, spanning from educational initiatives in anatomy and diagnostics to acute and critical patient care.These advantages encompass visualizing organs before surgical procedures, facilitating the education of dental students, and enhancing pathology education, among other applications.Researchers have employed various methodologies within the field, notably utilizing standard neuro-navigation magnetic resonance imaging (MRI) images for intra-operative planning, a process accomplished by employing the HoloLens 2 system to transform these images into a 3D model.This innovative integration facilitated surgeons in overlaying the 3D rendered image onto the patient's anatomy, effectively addressing the limitations inherent in traditional neuronavigation systems (Jain et al., 2022).Additionally, other researchers [2] have successfully utilized the HoloLens 2 device to visualize data from computerized tomography scans and Digital Imaging and Communications in Medicine (DICOM) files (Cetinsaya et al., 2022).To address the challenges associated with rendering extensive volumes on mobile devices, particularly limited memory capacity and device constraints, an optimization algorithm called Empty Space Skipping was integrated into the HoloLens platform.This algorithm enhances the rendering of large models by generating rays for each voxel, originating from the camera's center and passing through an imaginary image plane positioned between the camera and the volume used for rendering.
Apart from medical imaging, significant applications of MR devices are observed in the fields of mental science and anatomy.The scholars in [3] introduced an innovative educational instrument designed to manage phobia-related apprehension.By employing MR software, the patient can be exposed to a range of distressing scenarios associated with their phobia and observe the ensuing effects on their emotional state.This facilitates a more accurate evaluation of the phobia's severity and subsequently enhances coaching strategies for the patient to manage their fear.In a systematic review, researchers explored the contemporary applications of MR in medical and healthcare scenarios.Moreover, our investigation unveiled the comprehensive review conducted by Palumbo (2022), which underscores the capacity of MR technology to support clinical care digitally.This review meticulously evaluated studies showcasing the feasibility and applicability of MR in medical and healthcare environments.Palumbo's review serves to highlight both the potential and limitations inherent in contemporary MR-driven innovative solutions, while also shedding light on burgeoning research domains such as telemedicine, remote control, and motor rehabilitation.
The studies discussed above predominantly delve into the use of MR technology for simulation objectives.Conversely, Wersényi (2022) offers a concise examination, specifically concentrating on medical applications using smart glasses.The study evaluates the HoloLens glasses in terms of latency and data rates, employing both WiFi and the 5G campus network as part of the assessment.Given that 5G ensures improved network conditions for seamless and immersive communication among 3D content, [4] introduces a prototype for real-time point cloud streaming (Figure 3).The study includes live-action demonstrations of point cloud streaming using MR devices.Their presentation on network delay illustrated the prototype system's potential capacity to support uninterrupted live streaming without any interruptions point cloud playback.Remote collaboration's increasing significance in demonstrating practicality necessitates high-quality rendering, especially in remote environments.In the study presented in [5], the examination of latency issues within remote collaboration is emphasized.It introduces a technique for remote collaboration employing holographic avatars to enable instantaneous movement transfer.This approach facilitates immersive interactions with three-dimensional visualized data among numerous remote users.Another recent study by Long et, al. (2022) introduced an advanced system incorporating optimized remote rendering capabilities to present the 3D city model of New York City via HoloLens (Figure 4).The findings suggest that remote rendering exhibits superior performance compared to local rendering for the model, resulting in a substantial enhancement in the average Quality of Experience (QoE) by a minimum of 21%.
Among scholarly pursuits, besides endeavors in medical science and surgical disciplines, a plethora of investigations and practical applications thrive within the domains of game development and real-time rendering.The work by Jung et al. ( 2022) delves deeper into the intricacies of rendering optimizations within the realm of MR from an algorithmic standpoint.Direct volume rendering stands as a conventional method for the threedimensional visualization of volumetric scientific data.The researchers in the aforementioned paper scrutinized a critical necessity: the rendering latency capability for MR head-mounted displays.To address this, they proposed a benchmark application encompassing 5 volumes and 30 variations in rendering parameters, outlined in Figure 5.The research evaluates three rendering parameters: sampling rate, rendering distance, and mipmapping, resulting in the generation of 30 variations.Using HoloLens, rendering latency is evaluated across 100 frames during random rotations of the volume.Volumes with lower resolutions exhibit superior average rendering latency.Mipmapping notably improves average rendering latency on HoloLens, categorizing more direct volume renderings as optimal, practical, or minimal.Higher sampling rates correlates positively with rendering latency, while rendering distance had a negative correlation.Overall, the study suggests that enabling mipmapping on HoloLens enables interactive direct volume rendering with standard resolution volumes.
Ultimately, the primary constraint observed in MR devices, as highlighted in [6], is their limitation in outdoor settings due to their restricted operational distances, which typically extend up to 3.5 meters.
Besides the traditional aspects of MR, some attempts on integrating generative AI to MR have appeared in recent studies.In line with the Metaverse trend, Xu et al. (2023) have introduced a straightforward framework for AI-driven simulations in autonomous driving within vehicular MR environments.While the framework might appear to merely align with the trend of concepts like the Metaverse and may not significantly contribute to MR development, it does outline the communication challenges within MR. [7] surveyed on the remote assistance and training applications in MR, while [8] introduces using replicas to improve remote collaboration in MR.The highly cited paper in 2023 about MR is also a survey clarifying the differences between virtual reality, augmented reality and MR.

Gaps and Controversies
The exploration of MR research underscores a notable disparity in approach among researchers.While those in the computer science domain primarily concentrate on the technological aspects underpinning MR applications, counterparts in other fields actively employ MR as an emerging paradigm to elevate their areas of inquiry.A discernible gap exists between researchers specializing in computer graphics, computer vision, networks, and the domain of MR itself.The majority of researchers in these fields have not directed their investigations towards MR applications or sought solutions within the realm of MR, consequently impeding the advancement of MR technology to a certain extent.Additionally, it is unrealistic to expect researchers outside the computer science domain to address the complexities inherent in MR-related technologies.
Regrettably, despite our exhaustive investigation, a noticeable gap persists in the integration of LLMs to MR by the end of 2023.While LLMs have undergone rapid development and are widely adopted across various research domains, the full amalgamation of LLMs with MR research remains elusive.The diminutive size of these devices poses challenges for running LLMs; however, opportunities exist to optimize the network efficiency of MR devices for the seamless integration of cutting-edge applications like AI consultancy, AI assistant and AI 3D virtual friend.Should LLM-based applications be successfully integrated into MR systems, it becomes evident that the Metaverse is within imminent reach.

Conclusion
Our analysis involves an intricate examination of both recent works over the last two years and the overarching evolution of MR over a span of two decades.Globally, MR has displayed a gradual emergence, yet recent strides in technology and science have propelled it towards transformative potential across various industries and everyday life.This comprehensive review thoroughly investigates the landscape of MR by methodically assessing highly cited papers from the past two decades, encompassing a broad spectrum of research publications on this subject.With a primary focus on scrutinizing papers released in 2022 and 2023, this study explores the contemporary applications and functionalities of MR.As of present, the integration of LLMs into MR remains absent.However, we anticipate a promising future trajectory by leveraging LLMs to enhance MR capabilities and augment its effectiveness.MR presents immense promise in transforming learning methodologies but lacks comprehensive understanding of its impact on educational technology and social culture.Despite its gradual improvments, MR's integration into education remains underexplored, limiting empirical evidence on its effective use for educational enhancement.MR holds promise in educational technology, offering immersive learning and customized content delivery.However, there's limited scholarly exploration regarding its pedagogical impact, learning outcomes, and socio-cultural implications in education.The influence of MR on social culture, including interactions and knowledge sharing, demands in-depth investigation for understanding societal implications and ethical considerations.Extensive research is crucial to grasp MR's transformative potential, bridge research gaps, and meet evolving needs.Its integration may reshape education and social norms, underscoring the need for further scholarly exploration.The search strategy and the metrics we used when making this literature review is open source and publicly accessible through the project page https://aizierjiang.github.io/mr.

Fig. 1 .
Fig. 1.Annual publication trend of MR-related research from 2003 to 2023 Speicher et al. in 2019 introduced the definitions of MR and explores the scope of MR applications.[1] enriches the study on the utility of MR from different aspects in our life.The survey conducted by Verhey et al. (2020) provides a comprehensive overview of MR devices.Previous surveys and reviews extensively featured advancements in hardware and software within the MR domain, emphasizing the prevalent industry challenges.However, there remains a dearth of studies that articulate how MR, in turn, expedites the advancement of research pertaining to its underlying technologies and theories.Rather than reiterating the recognized definitions of MR and listing features of prevalent MR instruments, this paper directly focus on the highly cited and recently published academic papers on MR, aiming to uncover potential benefits of MR for future research.

Fig. 2 .
Fig. 2. The hierarchical model of a specific high-tech product.

Fig. 3 .
Fig. 3.The structural layout of the point cloud streaming system's architecture.