This demo used You Only Look Once (YOLO) AI computer vision to identify the objects and position in the video feed. We then used the 3D mesh generated from the HoloLens 2 to cast from the headset location to identify the real world locations of the identified parts.

To improve the training data we 3D modeled the relevant parts and created synthetic data to mix in with the real photos that were manually labeled. The synthetic data automatically generated bounding boxes to help reduce manual labelling.