Microsoft Azure for Mixed Reality

What is Mixed Reality and Azure?

The mixed reality is often defined as the cross between Virtual Reality and Augmented Reality. But what will interest us the most is the Augmented Reality part, the goal being to understand the real world in order to increase virtual information that allow the user to act and control his devices. Via this chain, several synchronizations can take place between the real environment and its virtual representation: this is the concept of Digital twins , two twins , real and virtual, which are one for the user. The business data displayed can be 3D(CAD files, BIM model …) and 2D (PDF, images, videos …). To enable this, the HoloLens takes its place on the field while Azure takes on an omniscient role.

Azure is a Cloud-Service-Provider , Microsoft has built gigantic data centers around the world and makes them available through a service platform. Microsoft also offers Azure Data Stack which allows you to install Azure on its own infrastructure. We can distinguish Azure services by two groups :

1- IAAS services (Infrastructure as a Service) allow a company to rent optimized servers to easily create a large IT infrastructure without a dedicated in-house team. These services are based on technologies that allow automatic management of applications within a datacenter such as containers and orchestrators . These services invisible to the end user allow to optimize its IT infrastructure costs. These will be adjusted in real time based on Azure service consumption. We talk more often about migrating applications to the cloud than creating them for new use cases.

2- PAAS (Platform as a service) services are on the contrary innovative and set Azure apart from its competitors. They implicitly use IAAS services because they are additional software bricks (in the form of a micro-service) for more precise use cases .

You can find below some examples of these services for Mixed Reality :

Azure Spatial Anchors

The first problematic of Mixed Reality is to synchronize the position of the virtual world with the real world (Spatial Alignment), both in real time (collaborative applications) and one session to another . This can be done with trackers (QR Code, object recognition) or with anchors (invisible to the user and automatic). An application can use the anchors provided by the device or use the service Azure Spatial Anchors (ASA), the two solutions have different advantages and disadvantages. ASA is cloud-based, the devices must have an internet connection to access servers allowing centralize anchors between multiple devices. Each type of device (HoloLens 1, HoloLens 2, Android, iOS) handles native anchors differently, they are therefore not compatible with each other. ASA, on the contrary, uses a identical system for all devices, which is based on a hash of the points of interest recognized by the camera. This has the advantage of being multi-platform and anonymous, at the cost of reduced accuracy on HoloLens.

Azure Remote Rendering

One of the limitations of the HoloLens is its graphics capabilities, due to the requirements of the two transparent screens and the size and weight constraints of the components. Microsoft is currently developing Azure Remote Rendering in order to render 3D on a cloud server. The display capacities then become similar to a high-end PC which allows display complex CAD or some more realistic light effects . You can view an example of its use in the project Apollo 11. The Hololens sends its position in relation to the holograms in real time and receives the images to be displayed on the screen, this requires a good internet connection and is done all the same at the cost of a slight loss of stability of the holograms.

Azure Cognitive Services

Microsoft offers a wide range of services for artificial intelligence , using machine learning and grouped within Azure Cognitive Services (ACS) which we can separate into two categories:

1- Conversational

• Speech-To-Text : voice recognition, to write notes without a keyboard (very useful on Hololens)
• Speaker recognition : identify the person speaking
• LUIS : intention recognition, interpret a natural sentence and extract the relevant data, to transform the user request into a command in an application
• Chatbot : generate natural and precise responses to the user
• Real-time translation : support a large number of languages to make an application multilingual easily
• TTS : speech synthesis, generate in real time a natural voice (with very good results) from a text

All these services are independent but can be combined in order to create a complete virtual assistant with which it is possible to chat naturally. It is possible to create a custom dictionary and there add business terms. The execution time of these services is short enough for a real-time use .

2- The vision

• Face and object recognition: This service analyzes the video stream from the camera to see if an object or a person is present. There is no 3D subject tracking yet.
• Form recognition : When a form is placed in front of the camera, the information is extracted taking into account the formatting. This allows you to quickly save a document and send it.

The advantage of these services is that Microsoft manages machine learning models on its own , there is no need to have dedicated skills to use them.

Azure IoT

The Internet of Objects is a key sector of the cloud because a large number of devices must be able to send data to a centralized server. HoloLens (or any other mobile device) has a dual role: it is both a IoT device in the field (which can send data related to its sensors or its use) and a viewer data sent by connected objects. It is very useful in the field where we can associate ASA to consult the data on the devices in question as well for analytics than for real time. Azure offers a series of services for IoT including IoT Hub which centralizes the registration of devices and the redirection of data to storage servers that can be consulted with PowerBI or PowerApp .

Azure pushes some of the capabilities of the Hololens further, but above all makes possible to create a complete ecosystem of technologies for industry. It is important to be aware of the possibilities and constraints of combining these two areas in order to succeed in your digital transformation!