Optimal Design of the Adaptive Normalized Matched Filter Detector

This article addresses improvements on the design of the adaptive normalized matched filter (ANMF) for radar detection. It is well-acknowledged that the estimation of the noise-clutter covariance matrix is a fundamental step in adaptive radar detection. In this paper, we consider regularized estimation methods which force by construction the eigenvalues of the scatter estimates to be greater than a positive regularization parameter rho. This makes them more suitable for high dimensional problems with a limited number of secondary data samples than traditional sample covariance estimates. While an increase of rho seems to improve the conditioning of the estimate, it might however cause it to significantly deviate from the true covariance matrix. The setting of the optimal regularization parameter is a difficult question for which no convincing answers have thus far been provided. This constitutes the major motivation behind our work. More specifically, we consider the design of the ANMF detector for two kinds of regularized estimators, namely the regularized sample covariance matrix (RSCM), appropriate when the clutter follows a Gaussian distribution and the regularized Tyler estimator (RTE) for non-Gaussian spherically invariant distributed clutters. Based on recent random matrix theory results studying the asymptotic fluctuations of the statistics of the ANMF detector when the number of samples and their dimension grow together to infinity, we propose a design for the regularization parameter that maximizes the detection probability under constant false alarm rates. Simulation results which support the efficiency of the proposed method are provided in order to illustrate the gain of the proposed optimal design over conventional settings of the regularization parameter.