Abstract

K-Nearest Neighbors (KNN) is a widely used nonparametric supervised learning method that can be used for both classification and regression tasks. The method assigns predictions to new observations based on the responses of the K training observations that are closest to them in the feature space. Despite its conceptual simplicity, KNN remains widely used in many application domains because of its flexibility and its ability to capture complex decision boundaries without requiring strong parametric assumptions about the underlying data-generating process.

KNN belongs to the class of lazy learning or instance-based algorithms. Rather than estimating an explicit model during training, it stores the training data and performs most computations at prediction time, when the nearest neighbors of a new observation must be identified. Consequently, the effectiveness of KNN depends critically on how proximity between observations is defined. The choice of the distance metric directly determines which observations are considered neighbors and therefore plays a central role in determining prediction accuracy and stability.

Standard distance measures used in KNN may perform poorly for mixed-type data. This limitation has motivated the development of similarity and distance measures specifically designed for heterogeneous datasets. A well-known solution is Gower’s similarity coefficient, which combines variable-specific dissimilarities through normalization and averaging. However, this measure is typically unsupervised, meaning that does not incorporate information about the response variable. Moreover, Gower's distance introduces a bias towards the categorical variables.

To address these limitations, several studies have proposed supervised distance measures that incorporate information from the response variable. These approaches fall within the broader framework of distance metric learning. By adapting the geometry of the feature space to reflect predictive relationships, such methods can substantially improve the performance of KNN. Unlike general metric learning approaches, which typically assume numerical features, the proposed distance is specifically designed for heterogeneous datasets containing both numerical and categorical variables.

Specifically, we introduce a supervised distance that combines variable-type-specific dissimilarities with response-based weighting, allowing KNN to better exploit heterogeneous predictors. The proposed metric accommodates heterogeneous variables while incorporating information about their association with the response variable. In particular, the distance function is constructed so that observations with similar responses tend to be closer in the feature space, whereas observations with different responses are placed farther apart.

The proposed approach combines two elements. First, it allows variable-type-specific treatments for numerical and categorical attributes, ensuring that each variable contributes appropriately to the overall distance. Second, it integrates association-based information derived from the response variable, allowing the metric to emphasize features that are more relevant for prediction.

By integrating response-driven information into the construction of the distance metric, the proposed method improves the discriminative power of the KNN algorithm. Observations with similar responses are more likely to be identified as neighbors, while those with different responses are more clearly separated in the feature space. The resulting framework provides a practical and interpretable solution for applying KNN in heterogeneous datasets, where traditional distance measures may be inadequate.