Classification and Analysis of Interactable Objects in an Agricultural Environment

Abstract

In the field of agriculture, autonomous systems have developed rapidly in recent years. Gaining an understanding of the surrounding environment is an important element for autonomous systems to operate safely. This thesis aims to address this need by creating an egocentric dataset with depth and gaze data. The dataset contains over three hours of agricultural tasks performed by both experts and non-experts, including picking, cutting, and planting. An object affordance model was trained on this dataset to predict interactable areas within a given image. These areas were categorized into three different classes: pickable, cuttable, and plantable. The code used in this project can be found on: https://github.com/SigurdKvaal97/ Classification-and-Analysis-of-Movable-Objects-in-an-Agricultural-Environment/ tree/master Gaze and depth data were introduced to the model to analyze their effects on performance. This was done by creating four different implementations of the model: the baseline model without depth or gaze, the model with the addition of gaze points, the model with the addition of depth data, and the model with the addition of gaze over time. These implementations were then compared using the Kullback-Leibler Divergence (KLD), the Similarity Metric (SIM), and the Area Under the Curve - Jackknife (AUC-J) scores to assess model performance. Introducing gaze did not result in any improvements, while the inclusion of depth data managed to achieve an improvement in model performance.