This paper is to present the performance analysis of recursive least square algorithm with error-saturation in mixture noise. The algorithm is referred to as nonlinear RLS (NRLS). Generalized clipping function is considered for the error-saturation nonlinearity. An improved mean square behavior of NRLS is carried out. It is shown that the theoretical analysis and the simulation result are close to each other. By the analysis, we can relate the convergence and the mean square error in terms of the slope and clipping level of the nonlinear function. Based on the normalized mse, an instrumental variable is derived for yielding a variable clipping function to provide fast convergence and small mean square error. Copyright © 2002 by The Institute of Electrical and Electronics Engineers, Inc. All rights reserved.