Augmented reality

_{^{4 min read (727 words)}}

Augmented reality is a term that defines a direct or indirect view of a physical real-world environment, which has been enhanced in real-time by computer-generated virtual information on it [1]. Combining real and virtual objects, it registers them in 3D space. According to Milgram’s reality-virtuality continuum [2], AR is closer to the real world, in contrast to AV, which is related more to the virtual environment.

Fig. 1. Milgram’s reality-virtuality continuum. Adapted from [1]

The objective of augmented reality is to ease the user’s activities with the help of virtual information, which could be brought not only to his surroundings but any indirect view of the real world, e.g., during a live video stream. It essentially improves real-world perception and can provide means of interaction with it the user. Referring to Milgram [2], while VR technology aims to immerse users into a synthetic environment, isolating them from seeing the real world, AR augments understanding of reality by overlaying virtual objects, thereby preserving the sense of presence in the real world. Although head-mounted displays are more familiar, the term is not limited to particular display technology. In fact, researchers suggest [3] AR can potentially cover all senses, such as smell, touch, and hearing, not bound to sight only. The technology can simplify daily routines, enhance training, and aid complex, hazardous procedures by visualization of digital content.

In principle, computer vision techniques are applied to render 3D virtual objects from the same viewpoint from which the real scene images are being taken using tracking cameras. Two-stage procedure (tracking and reconstruction) is required to perform image registration. One approach for tracking could be feature-based: exploring the link between 2D images and their corresponding 3D world frame coordinates [4]. After the connection is placed from a 2D image to the 3D world frame, it could be possible to estimate the pose of the camera by minimizing of the distance between the observed and actual 2D feature coordinates. The next reconstruction step utilizes the information acquired from the previous stage to restore a real-world coordinate system.

Fig. 2. Point constraints for the camera pose problem adapted from [5]

As illustrated in the Figure 2 above, two principal coordinate systems are represented: the 2D image coordinate system and \(W\), the world coordinate system. Let \(p_i\left(x_i, y_i, z_i\right)^T\), where \(i=1,…,n\) with \(n\geq3\) be a set of 3D non-collinear reference in the world frame coordinate and \(q_i\left(x_i^{\prime}, y_i^{\prime}, z_i^{\prime}\right)^T\) be respective camera-space coordinates, then the relation of \(p_i\) to \(q_i\) is as follows:

where \(R=\left(\begin{array}{c} r_1^T \\ r_2^T \\ r_3^T \end{array}\right)\) and \(T=\left(\begin{array}{l} t_x \\ t_y \\ t_z \end{array}\right)\) represents a rotation matrix and a translation matrix correspondingly. Assumption here is that camera is calibrated, and perspective projection model is used, e.g., when a point has coordinates \((x,y,z)^T\) in the camera frame, its projection onto the image plane is \((x/z,y/z,1)^T\).

Let’s define the image point \(h_i(u_i,v_i,1)^T\) be the projection of \(p_i\) on the normalized image plane. Using camera pinhole, the collinearity equation which sets the relationship between \(h_i\) and \(p_i\) is:

Medical applications of AR have been extensively investigated. Review papers suggest research efforts related to computer-assisted surgery, three-dimensional imaging, and laparoscopy dominate among others [5, 6]. For instance, major achievement has been reported to augment video images recorded by an endoscopic camera presented on a monitor viewing the patient’s operating site. However, in the work, direct and intuitive 3D visual cues are constrained due to the additional displaying environment. Researchers used Google Glass to offer surgeons image-guided margins for cancer resection [7]. Although the surgical navigation system demonstrated potential, the study was limited to superficial tissue, moreover, optimization and validation works were required for the software. The groups utilized AR to aid physicians to inspect patients’ hands and wrists for arthritis [8]. By augmenting 3D real-time MR data on top of the patient’s hand view, the setup supported diagnosis, performing morphological and functional analyses. Unfortunately, display and tracking challenges still exist in current HMDs, delaying medical applications. These occur from accurate placing depth perception to 3D models. Furthermore, the doctor’s ability to see his instruments through the projected images must be maintained. Another category of issues could appear from retraining medical specialists to learn AR systems. Respective simplistic technology should be developed to match the protocols the doctors are used to [1, 9]. Nevertheless, augmented reality is an emerging technology to affect patient treatment in the future. With more capable hardware and reduced size, it will become more gradual for AR to integrate into existing workflows as well as create new opportunities for doctors, patients, and healthcare professionals.

How to cite

Askaruly, S. (2022). Augmented reality. Tuttelikz blog: tuttelikz.github.io/blog/2022/03/augmented

References

[1] Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E. and Ivkovic, M., 2011. Augmented reality technologies, systems and applications. Multimedia tools and applications, 51(1), pp.341-377.
[2] Milgram, P. and Kishino, F., 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), pp.1321-1329.
[3] Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S. and MacIntyre, B., 2001. Recent advances in augmented reality. IEEE computer graphics and applications, 21(6), pp.34-47.
[4] Lee, B. and Chun, J., 2010, April. Interactive manipulation of augmented objects in marker-less ar using vision-based hand interaction. In 2010 Seventh International Conference on Information Technology: New Generations (pp. 398-403). IEEE.
[5] Eckert, M., Volmerg, J.S. and Friedrich, C.M., 2019. Augmented reality in medicine: systematic and bibliographic review. JMIR mHealth and uHealth, 7(4), p.e10967.
[6] Ha, H.G. and Hong, J., 2016. Augmented reality in medicine. Hanyang Medical Reviews, 36(4), pp.242-247.
[7] Shao, P., Ding, H., Wang, J., Liu, P., Ling, Q., Chen, J., Xu, J., Zhang, S. and Xu, R., 2014. Designing a wearable navigation system for image-guided cancer resection surgery. Annals of biomedical engineering, 42(11), pp.2228-2237.
[8] Gallo, L., Minutolo, A. and De Pietro, G., 2010. A user interface for VR-ready 3D medical imaging by off-the-shelf input devices. Computers in Biology and Medicine, 40(3), pp.350-358.
[9] Akinbiyi, T., Reiley, C.E., Saha, S., Burschka, D., Hasser, C.J., Yuh, D.D. and Okamura, A.M., 2006, September. Dynamic augmented reality for sensory substitution in robot-assisted surgical systems. In 2006 International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 567-570). IEEE.