Here at EGGS we truly believe that having a thorough understanding of emerging technologies is a necessary prerequisite for being great facilitators for innovation, something that makes our Creative Technologists invaluable resources. We got our hands on the HoloLens development kit and have spent a few months testing where things are headed, and honestly, I'm stoked.
While the HoloLens may not initially become a huge success in consumer markets, I'm certain we'll see some big professional contenders for early adoption. Read on to see why!
Out of all the head-mounted displays that I've tried in the past couple of decades, the HoloLens was the best in its class.Brian Blau, Research Director of Consumer Technology and Markets at Gartner
Enter the HoloLens
Microsoft released a pre-production version of their HoloLens to developers during March 2016. What's the deal and how is this different from traditional AR? The HoloLens promises an immersion so believable that they chose to coin the new term Mixed Reality. They achieved this by overcoming two main challenges:
It detects and maps out the real world to a decent level of accuracy
It projects virtual objects into real 3D space
Why is this important you ask? Because if the HoloLens can understand the real world, then it can begin to create a believable intersection between itself, and the virtual world. In essence, the ability to create holograms!
For scanning and interpreting the real world - technically called Spatial Mapping - the HoloLens has an impressive array of sensors - some of which are direct descendants from the Kinect:
Two environment sensing cameras on either side
Inertial Measurement Unit, including an accelerometer, gyroscope and magnetometer
Depth camera with a 120° × 120° angle of view
2.4 megapixel photographic video camera
Four microphone array
Ambient light sensor
For projecting virtual objects, the HoloLens places a set of transparent lenses in front of the user’s eyes. These lenses use optical waveguides to colour blue, green and red across three different layers - each with diffractive features, with a light engine mounted above the lenses which projects light down into them. This combination helps to trick your eyes into perceiving virtual objects at virtual distances.
Why the HoloLens works
So you may be thinking…
This all sounds a bit complex, what does this really mean?
The beautiful thing is that Microsoft have done what we at EGGS truly believe in - taking something inherently complex and making it simple. From the user's perspective, holograms simply seem to exist in the real world as though they were there in reality.
Virtual objects co-exist with the real world - real objects will interact with and occlude virtual ones, solidifying the illusion.
In terms of interaction - and as far as holograms go - the sky is the limit; whilst standard interactions are simple: In the centre of the screen is the mouse pointer, and as you move your head around, it will latch on to real and virtual objects in the 3D space, and let you interact with them by either tapping the air in front of you or pinching the air with your fingers.
There is also an advanced natural speech recognition engine in place which lets you interact with everything using only your voice.
The HoloLens comes with software that lets you easily explore the capabilities and possibilities that this technology represents; From placing pre-made holograms around the office, to battling robot bugs that crawl out of holes in the walls and exploring the galaxy in your living room, one quickly realizes that we’re in for some interesting times ahead.
Let's Nerd Out
Our Creative Technologists have truly enjoyed working with the HoloLens. From the developers’ perspective, here are a few reflections on the technical side of things:
Since the HoloLens runs a special edition of Windows, you must develop on a Windows machine.
If you want to create a simple 2D Universal Windows Platform application, you can use Visual Studio and XAML, and have something up and running quite easily with the option of choosing between C#, CPP and JS as your programming language.
Anything 3D and you'll want to download a special HoloLens edition of Unity 3D for Windows. For many - like myself - this is a familiar playground, and anyone having worked on a 3D game for mobile platforms before is likely to already have the skill set required. From Unity you will be able to export to a Windows 10 UWP project; which lets you build, run and debug in the HoloLens emulator, or on the HoloLens device itself.
What do you get for free?
What you get for free out of the box with the Unity SDK is Spatial Mapping and Microsoft's system of anchoring virtual objects in real 3D space. And this is no mean feat! Without it holograms wouldn't work, and having to write this oneself based on raw sensor data is not feasible. Thanks to this your virtual objects can easily interact with, and co-exist in, the real world.
Of course, you also have access to the natural speech recognition engine which makes it easy to implement voice commands in your applications. It is relatively trivial to create objects with which you can interact with your voice.
One might optimistically think that you get other things such as object recognition or path finding as part of the SDK. However, for such functionality you'll have to roll your own.
Although somewhat limited by the hardware, developers have at their disposal all the tools and features that Unity provides meaning anyone with prior experience with Unity, or even similar engines, is all set to start creating amazing holographic experiences.
The HoloLens is in no uncertain terms a fantastic piece of engineering which does a fantastic job of creating a Mixed Reality world. However, there are limitations
The price point: As a development kit the HoloLens is a bit on the pricey side at $3000, which may or may not reflect the price when hitting the market for consumers.
The size of the thing: While it is impressive what they have managed to stuff into this thing with today’s technology - and how well it works - it is still a bit on the bulky, and heavy side.
The viewing angle: The viewing angle, whilst OK, is not great. This leads to holograms and content getting cut off, which is both annoying and breaks the realism. Apparently, MS chose a sharp screen with good brightness and resolution over a large viewing angle in this iteration.
Mostly indoor: Since the HoloLens has no GPS and no 4G capabilities, we're still limited to indoor use where there is a Wi-Fi connection.
Performance: While the performance in general is quite good, there is from time to time lagging and shearing of the display. The nature of the lenses can cause virtual content to visually split into their RGB components.
3D vs 2D: The HoloLens cannot simultaneously display 2D and 3D content. So any UWP 2D part of your application must be hidden if you want to show 3D through Direct3D, and vice versa.
Finally, both VR and AR technologies are maturing to the point where they're sufficient to adding real value, and solving real problems. We're still in the early stages, yet everyone seems to agree that AR/MR particularly, will become more and more part of our lives as the technology advances further.
“AR is going to take a while, because there are some really hard technology challenges there. But it will happen. It will happen in a big way, and we will wonder when it does, how we ever lived without it. Like we wonder how we lived without our phone today.Tim Cook, Apple
Microsoft have reached out to and joined forces with quite a few partners to explore the future of AR, no doubt to capitalize on their head start into AR.
NASA is researching how they can leverage AR in some of their projects - promising a new level of interaction for their Mars explorations.
Back on earth, Case Western Reserve University is delving into the future of teaching and understanding anatomy and the human body.
The way forward
If the world of personal computing is anything to go by, it is likely that as the advances continue in hardware, we will see smaller and more powerful AR spectacles emerging. These will provide us with similar experiences to the HoloLens, only vastly improved and even more believable.
As the software side of things mature, we will see applications and interactions that innovate and make hands free operations of virtual content - which is augmented into our lives - more prevalent.
Will AR be the death of the smart phone? Will we soon wear all our computer needs as contact lenses? It's not as far-fetched as it might seem. Until then, drop by at EGGS in Oslo if you want to experience the HoloLens in person.
Back to the future
We've known about Virtual Reality since the 80s. When VR games finally hit the arcades in the 90s, it quickly became apparent how distant we still were from everything movies and games had fed into our imaginations. The beginning of the millennium brought the term Augmented Reality to the masses; promising to seamlessly combine the virtual world with the real world. But who wants to use their phone or tablet as a window to see rudimentary 3D objects? Microsoft and Google are now promising to take AR to new heights with the HoloLens, and MagicLeap respectively. Are we finally on the cusp of realizing our 30-year-old dreams?
The world of VR has recently made some big strides since the early days, and we now see several big corporations investing heavily in VR - Facebook and Google to name but a few.
A few years ago, EGGS started experimenting with the Oculus Rift, and more recently the HTC Vive. These technologies do an excellent job of creating believable, immersive virtual environments. Sensors such as gyroscopes and IR based head tracking allows these headsets to place us into the virtual world and give us a sense of immersion that hasn't been possible before.
Stuck in a 100% virtual world
With the incursion of Google Cardboard and Samsung Gear - and more recently Google Daydream - VR has become even more accessible by leveraging the technologies already present in most modern smart phones; thus avoiding the high cost of purchasing a dedicated VR headset.
However, the user is still stuck in an entirely virtual world, with no ability to move around physically. HTC have attempted to give some limited ability for physical movement with laser position sensors covering a 4.6 x 4.6m space, but this is both cumbersome to set up and still restricts your movements to a limited space. Also, many people will find that these VR headsets induce nausea and headaches, whilst interaction inside the virtual world is limited to custom controllers.
Augmented Reality: Smartphones, Google Glass and Pokémon Go
Augmented Reality is no new concept. In fact, it was introduced in 1901 by L. Frank Baum, who first mentioned the idea of "an electronic display/spectacle that overlays data onto real life".
Nothing of note happened for consumers until the early 2000’s when AR arrived on smart phones and tablets. Augmented Reality could now leverage the technologies in smart phones, such as GPS, magnetometers (compasses), gyroscopes and cameras; to augment the real world using the screen as a window.
To take AR to the next level, Google introduced Google Glass; an optical head-mounted display in the shape of a pair of eyeglasses. The aim was to augment reality and allow for natural language voice commands. Unfortunately, the limited functionality resulted in heavy criticism, and in 2015 Google announced that it would stop producing the Google Glass prototype.
Wow, you must be as enthusiastic about exciting new tech and how they will change our lives as I am.
Don't hesitate to get in touch to discuss the HoloLens, technology or innovation!