Features of the Apple Vision Pro

Apple states that its Vision Pro is the first spatial computing device in the world. This device is designed to bring the digital and physical worlds together in an intuitive manner. This is not genuine augmented reality; the physical world that’s observed through the Vision Pro is presented via a pair of almost 4k screens. These are adjusted automatically to the placement of your eyes for comfortable visual feedback. There are 12 cameras installed in the headset, the majority of which track the physical world and then deliver the images to the screens. But what key features can you expect from this innovative device? 

Sensitive Eye Tracking 

The Apple Vision Pro facilitates interaction with the interface via vision, touch, and voice. The eyes are critical to navigating the interface, and eye movements act as the mouse pointer. The sensitive eye tracking allows the user to point at the immersive digital landscape with a simple glance. This is made possible with a ring of LEDs and infrared cameras that project invisible light onto the surface of the eye. With even minute eye movements, the Vision Pro can determine where the user is looking without any head movements. To ensure that the eye tracking is accurate, there is an initial calibration process when the headset is used for the first time. The fine tuning of the eye tracking goes beyond the on board hardware, and certain nuances are required to make this feature possible. The built-in tiny cameras track and even anticipate the slight movements in the eye pupils. This is used to accurately predict where the user will click next, and this was made possible with AI. Apple used AI to study emotional and physiological responses to develop its eye tracking features. This data is related to eye tracking only using a background process, and it cannot be seen by third party websites and apps. Whether the Vision Pro can be used to observe and analyze data to train future data sets based on real-world usage is unclear at this time. 

Rotating Crown To Tailor immersiveness  

The Vision Pro has a 3D interface displayed on tiny OLED displays for each eye. These displays are sharp; they are almost 4k and present impressive visual quality. The virtual object that’s simulated in the software is augmented with effects layered over the physical world. This gives the virtual objects a realistic appearance that feels almost like a natural part of the real world surroundings. At first, this may seem overwhelming, but the level of immersion can be adjusted with a rotating crown that will be familiar to Apple Watch users. This crown can be twisted to tune the view further in or out of the virtual setting. This is useful to observe the setting in a focused or more passive manner as needed. To place this in the context of a real world scenario, someone working can dial down the immersion to interact clearly with a colleague as required. Alternatively, a user at home might dial up the immersion to enjoy a movie or TV show. The rotating dial can be used to fully surround the user with a virtual space known as “Environments”. These Environments are imagery taken from real world locations that have been dynamically captured volumetrically with a high resolution 3D camera. This is designed to fully immerse the user in the setting and make them feel like they are physically present. This can be extremely soothing for relaxation to play a game or watch a video or it can be something more exciting to experience. 

Intuitive Hand Gesture Interface Control 

The Apple Vision Pro interface can be controlled with hand actions, such as scrolls and clicks to zoom in and out. This negates the requirement for a separate bulky controller that would need to be strapped to the wrist. Tapping the index finger on the thumb can manipulate objects and flicking a finger scrolls. This headset supports scaling gestures to open windows and zoom media for a more natural interactive experience. To make this possible, Apple uses an array of sensors that transmit and record the visual data. The sensors have main front-facing, side-facing, and down-facing cameras for the left and right sides of the unit. The cameras impart a sense of the hand location, and additional sensors add 3D depth and real-time perception. This is possible even in dim light with infrared illuminators. TrueDepth cameras and LiDAR scanners. 

Translucent Screen Appearance  

Apple wanted to create an immersive experience without isolating users from other people around them. Our eyes are important in conversations, and allowing the eyes to be shown through the headset when the alternative reality is not in use is important. A translucent screen shows the eyes, but this is really a digital representation of the eyes shown on the external OLED display. This function is known as “Eyesight,” and it’s engaged automatically when people approach the user. The Vision Pro cameras in the eyepieces draw upon a real-time feed of how the user's eyes move as they talk to people. So, the user can continue to wear the headset and look at other people and their surroundings using the live camera feed. The external display sends a signal to other people, and the eyes look hazy if the user is using augmented reality. This can only occur at a longer distance because Eyesight is engaged when people come closer to engage in conversations with the user. 

Realistic Avatars 

To engage with Facetime using the Vision Pro, Apple has introduced realistic avatars that place participants on tiles that float around the user. Each virtual tile can be interacted with as normal and scaled as required. Facetime uses the spatial audio of the Vision Pro headset to deliver audio from participants in the same direction and distance as the tile. Users can also share their screens from apps and web browsers that they are using. Another exciting aspect of Facetime is that users are displayed without a phone or other device obscuring their faces. The authentic representation is created with the TrueDepth cameras in a similar way to how Memojis are created with iPads and iPhones. Neural networks create a person that looks real. This will closely match eye movements, hand gestures, and even facial expressions, which are recorded with inline sensors. An iPhone, iPad, or Mac user will see a detailed 2D avatar, but another Vision Pro user will get the full 3D realistic personal experience.