Implementation of a 3D object localization and recognition system using stereo cameras and neural networks in real time

Sustainable management of horticultural production requires an eff ective multilevel analysis of the environment, which is impossible without integrating data from various sensors, such as video cameras, laser lidars, thermal imagers, etc. Object detection is a key stage in image processing; however, conventional computer vision may fail to meet the necessary level of accuracy and performance. For example, such an analysis can be complicated by distorted or lost data due to optical barriers and diff using coatings of objects. These shortcomings can be overcome using stereoscopic vision to obtain accurate data on the scene depth and distance to objects, as well as to create a 3D model of the environment. In this article, we propose a system for 3D localization and recognition of objects that combines the capabilities of the RealSense D455 stereo camera and the YOLO v8 neural network model. The system ensures accurate and effi cient image processing in real time. The use of stereoscopic vision made it possible to achieve high accuracy in recognizing objects and constructing a 3D image of the surrounding area. A distinctive feature of the conducted study was the use of the authors’ own dataset, which was compiled by more than 4000 images collected experimentally, including those with visual noises. The dataset allowed the system to be adapted to the specifi c conditions of actual usage, increasing its accuracy and resilience to visual interference. The eff ectiveness evaluation showed the high rates of completeness (up to 95 %) of the system when processing up to 10 frames per second, which makes its suitable for real-time tasks. The stereo camera was calibrated to accurately estimate distances to objects, which further increased the model reliability. The results obtained confi rm the potential of the proposed approach for automation and analysis tasks in robotic systems

1. Hort D. O., Kutyrev A. I., Smirnov I. G., Voronkov I. V. Development of an Automated Management System for Agricultural Technologies in Horticulture, Agricultural machinery and technologies. 2021;15(2):61-68. DOI: 10.22314/2073-7599-2021-15-2-61-68. (in Russ.).

2. Guff anti D., Brunete A., Hernando M., Rueda J., Navarro C. E. The Accuracy of the Microsoft Kinect V2 Sensor for Human Gait Analysis, A Diff erent Approach for Comparison with the Ground Truth, Sensors. 2020;20(16):4405. DOI: 10.3390/s20164405.

3. Andriyanov N., Khasanshin I., Utkin D., Gataullin T., Ignar S., Shumaev V., Soloviev V. Intelligent System for Estimation of the Spatial Position of Apples Based on YOLOv3 and Real Sense Depth Camera D415, Symmetry. 2022;14:148. DOI: 10.3390/sym14010148.

4. Intel® RealSense™ LiDAR Camera L515, URL: https://www.intelrealsense.com/lidar-camera-l515/. Ссылка активна на 29.06.2024.

5. Intel® RealSense™ Depth Camera D455, URL: https:// www.intelrealsense.com/depth-camera-d455/. Ссылка активна на 01.06.2024.

6. Калибровка камеры, URL: https://ru.wikipedia.org/wiki/%D0%9A%D0%B0%D0%BB%D0%B8%D0%B1%D1%80%D0%BE%D0%B2%D0%BA%D0%B0_%D0%BA%D0%B0%D0%BC%D0%B5%D1%80%D1%8B, дата обращения: 10.03.2025