Stereotactic surgery is a surgical intervention, which utilizes 3D coordinate system to locate operative targets inside the body. Its main goal is to minimize the invasiveness of procedures such as ablation, biopsy, lesion, injection, stimulation, implantation, and so on. The first stereotactic surgical instrument was designed at University College London Hospital in 1908, by Sir Victor Horsley, a physician and neurosurgeon, and Robert H. Clarke, an engineer. The Horsley-Clarke apparatus implemented a Cartesian system through a metallic stereotactic frame that could be mounted on the body of the subject. Although this design was only tested in animal experiments, it provided the necessary means to permit unique landmarks fixed to the subject to be visualized on an image (X- ray), enabling the spatial matching between “patient-space” and “image-space”. In 1947, the group of Spiegel further developed the technique of stereotaxy, using the concept of a three-dimensional Cartesian coordinate system specialized for the human brain. Later, many had also given their contributions to the design and testing of classic stereotactic frames, which all borrowed mechanical devices to allow the position of the operative targets to be described in terms of a frame-based coordinate system. As a result of modern tracking technology and development of various medical imaging methods, the current image-guided surgery, namely the frameless stereotactic surgery, no longer requires the aid of mechanical frames.Besides the exciting advancement of research in computer technology, medical imaging methods have always been a close companion of the discipline of surgery. As early as 1896, within three months of the discovery of x-ray, one of the earliest applications of radiography in guiding a surgery was performed by a McGill University Professor of Physics, John Cox, who successfully removed a bullet from the leg of a victim based on the X-ray radiograph that had been made of the limb. However, X-ray was not ideal for soft tissue imaging. Later, two imaging techniques, pneumo- encephalography, and ventriculography pushed the field of stereotactic surgery further after the initial Horsley-Clarke apparatus. All bearing their advantages and drawbacks, medical imaging techniques such as X-ray, ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI) have majorly involved with today’s pre-surgical planning and real-time guidance of surgical instruments benefiting both the patients and surgeons. While possibilities of new imaging methods are still under investigation, through the study of computer vision, multiple imaging modalities can be manipulated and combined together to extend the image dimensionality, mitigate their spatial and temporal resolution limitations, and create the most suitable view of visualization for the surgeons under different circumstances (e.g. neurosurgery).