In an article published in the Harvard Business Manager (February 2018), Prof. Michael E. Porter, Harvard Business School and James E. Heppelmann, President and CEO of PTC, explain that AR is a completely new way to reach people, not simply another communication channel and that every company needs augmented reality
The adoption of augmented reality (AR) is only just starting to take off. But the full range of possibilities offered by these technologies means they have the potential to bring about a sea change in both our private and working environments over the next few years and decades. AR will change how we learn, how we make decisions and how we interact with our physical environment. It will bridge the gap between the digital and physical worlds, between man and machine — and will become even more important when combined with smart, networked systems.
AR will close the gap between our limited mental and absorption capacity and the ever-increasing quantities of data and knowledge we face in this world of virtual, digital products. We also currently face the limitation that, while our world is three-dimensional, we use two-dimensional screens and paper print-outs when making decisions based on data from the virtual world.
What is AR?
Broadly speaking, AR is a new way of providing information, and the authors believe that it will have far-reaching effects on how data is structured, managed and made available via the internet. Currently, users need to transform data provided on a two-dimensional medium into three-dimensional reality. If you have ever tried to follow a manual when setting up or repairing a slightly more complicated device, you will be familiar with the mental process involved, and how easy it is to get it wrong.
These difficulties are linked to the five senses and the way in which we process information. An estimated 80 to 90 per cent of the information we absorb reaches us visually. The cognitive load grows with every task and takes up a chunk of our available mental capacity. As we find it easier to absorb and process information visually, reading a text and then processing the information involves a greater cognitive load than hearing the information and/or seeing it presented in a visual format. There is some truth in the saying ‘a picture is worth a thousand words’.
In AR applications, the solution is to project digital data on top of the physical world, for example as an image, a 3D model or an animation, merging the real world with the digital. This approach is revolutionary because it always takes the context of the environment into consideration, so the right information is always displayed at the right time and for the right object or physical environment. This then reduces the mental effort required to overcome the cognitive distance, the gap between how the information is presented and the context to which it relates. Take the simple example of getting directions from a smartphone GPS app, remembering them, and then acting on them at the appropriate moment. This presents a much greater cognitive load than if the information were projected directly onto the windscreen in the driver’s field of vision, as has been done for some time now in vehicles with integrated heads-up displays.
What can AR do?
The mantra that experience is what counts might have been written for augmented (and virtual) reality, as real-life experience of these technologies is essential. The two authors attempt to substantiate the fields of application and core functionality with examples that are as concrete as possible. They define ‘three plus one’ core functional areas where AR adds value: visualisation; instruction, training and coaching; and interacting with and controlling products – plus applications that combine AR with the complementary (but distinct) technology of virtual reality.
AR applications based on the principle of visualisation provide a sort of x-ray vision, revealing internal features that are usually hidden. Medical technology manufacturer AccuVein, for example, uses the heat signature of a patient’s veins to generate an image, which is then superimposed on their skin. When a blood sample is taken, this increases the likelihood of finding a vein threefold. Bosch Rexroth visualises the internal workings of its CytroPac hydraulic power unit. The 3D image that appears when the user looks at a certain point on the outer body displays various configurations for the interaction between the sub-systems and the cooling options available for the internal pump.
In terms of instruction and training, AR will deliver considerable savings. As we have seen, following written work plans and assembly instructions is often difficult and time-consuming. An onsite AR application can provide real-time, step-by-step explanations of the work process, ideally using smart glasses so that the operator has both hands free. This turns conventional manuals into interactive 3D holograms. In a pilot project, Boeing reduced the time required to assemble an aircraft wing by 35 per cent by using AR technology, and the number of employees who completed the task correctly first time rose by 90 per cent.
Providing remote support will be a key application of AR. AR devices will forward the image seen by the engineer on site to an expert at headquarters, who can then talk the engineer through the operation or display instructions in the engineer’s field of vision. Lee Company, which sells and maintains building systems, saves $500 in labour and travel expenses per technician each month and calculates a return of $20 on every dollar invested in AR.
While we currently control and interact with products or machines using buttons, handles or, more recently, even integrated touchscreens, AR will bring this to a new level. For example, factory staff wearing smart glasses can use gestures and voice commands to control virtual control panels and can check and adjust machine parameters while walking through the workshop without physically touching the machines themselves. Although this functionality promises huge potential, it is still in its infancy; especially when it comes to commercial products, with many of the necessary technologies still in the research laboratory. It is worth noting, however, that progress is being made in voice control in challenging environments, as well as in the detection of gestures and eye movements.
Virtual reality (VR) refers to the reproduction of physical reality using computer-generated images. Its uses include training; especially where machines and working environments are in dangerous or remote areas, or where the scenarios involve future or past situations. As the fourth core function of AR, VR provides the virtual background reality for the three other core functions. Audi is currently testing the VR holodeck, an accessible, virtual environment where the 3D image of a car can be used to evaluate designs during the transition from development to production. The US Department of Homeland Security is combining AR instructions with VR simulations to train disaster relief workers in responding to dangerous situations, like explosions. In a similar vein, the energy company BP uses VR to simulate drilling conditions as a backdrop for AR training procedures. Both companies are using these technologies to reduce costs and risks.
Two ways that AR can add economic value
In principle, AR can be used in two different areas of application. Firstly, it can add value to a product and secondly, it can help to further streamline processes in all areas of the company, from product development and manufacturing to sales and service. In products, for example, using AR to display instructions or safety information can represent a unique selling point. Heads-up displays have been available in cars for some time now and have also been used in aeroplanes for several years. If these displays are too expensive, an app for smartphones or smart glasses can be provided; this can be used to create a personalised virtual display to set up and operate the product.
Although 3D design models have been used in product development for many years, they are still represented using two-dimensional technology, hindering the creation of an integrated design process. With AR technology, full-size 3D models can be projected as holograms in the physical environment. For example, the 3D hologram of a construction machine can be ‘installed’ in its future working environment, and developers can walk around and climb into the full-size model to test lines of sight or ergonomics. And, while Volkswagen previously used 2D drawings to laboriously check the quality of prototypes, the company now uses an AR application to merge the prototype with the 3D model, making the process ten times faster.
In manufacturing, operators are given exactly the right information at the right time for the numerous manufacturing steps at their workstations. They are also given monitoring or diagnostic data for the systems, allowing them to perform proactive maintenance. In manufacturing and assembly environments, AR is also very effective for training purposes. In logistics, removing items from shelves using a paper list makes up 65 per cent of warehouse costs. At DHL, AR applications have improved the order picking process by 25 per cent by guiding workers along the shortest path through the warehouse. These kinds of solutions can also highlight the relevant storage bin and specific storage location in colour.
AR also offers a wide range of possible applications in marketing and sales. Showrooms and product demonstrations will offer fascinating, personalised customer experiences, and products can be viewed in their actual target environments, which will also support online retail. IKEA offers product illustrations and apps allowing customers to visualise furniture or decorative items in their own home. Aftersales service can also benefit greatly from the step-by-step repair assistance and remote support made possible by AR technology. In some circumstances, customers may even be able to get involved in maintenance work and perform many of the tasks themselves. This will deliver considerable savings when it comes to service.
Deploying AR
As we have seen, the full range of technical possibilities offered by AR is already being exploited for many applications. Although most of these are still at the pilot testing or PoC (proof of concept) stage, each AR installation requires a carefully thought-out implementation plan. This clearly identifies the strategic benefits of the application and sets out the technical and organisational solutions and skills required in detail. Porter and Heppelmann pose several questions relating to the deployment of AR.
The first question is: “Where does AR knowledge come from?” Specialists in AR development are scarce. The required skill set includes user experience or user interface design (UX/UI design), handling of 3D data and models and their implementation in AR applications, and, most importantly, content creation. Each project involves long-term decisions on whether the company should develop its in-house AR skills or whether the project’s requirements can be met by outsourcing the work or engaging an external partner. There are now also several specialist AR service providers on the start-up scene in Germany.
Not all AR applications are as complex; apps that visualise products (static 3D models) in different environments (such as IKEA) are relatively easy to create. Dynamic 3D content, such as instructions or training in manufacturing applications, presents a greater challenge. The most complex apps involve interactive solutions using technologies that have not yet reached maturity, such as gesture or speech recognition.
Regardless of the application, how the content is created has top priority. To obtain detailed digital images of products, CAD models can be adapted from product development or technologies such as 3D scanners need to be used. Advanced AR applications must also connect to business systems or other external data sources such as sensors so real-time data can be incorporated into the content. Last but not least, the decision must be made on whether a standalone software app should be developed or whether a content publishing model with content from the cloud would be more appropriate.
Most AR applications currently run as apps on mobile devices such as smartphones or tablets. Although the use of wearables such as head-mounted displays (HMDs) and smart glasses is still only just starting to take off, these devices are rapidly gaining ground in production environments. The idea is that, instead of carrying screens in their pockets, users will wear smart glasses as second nature, like sunglasses. In any case, the race is on to produce the best smart glasses, given that the company that dominates this market will also have a degree of control over AR technology. However, initial experience with the technology indicates that there will not be one set of universal glasses for all applications. For example, a Microsoft HoloLens in its present form will not be suited to harsh production environments. Devices like the smart glasses from the German manufacturer Trivisio would perform better here and can also be tailored to individual customers’ needs.
To summarise, Porter and Heppelmann believe that AR is an innovation of historic significance, and it is hard to disagree with this assessment.
