An alternative approach to developing two-sample hypothesis testing procedure using Bayesian method and Kullback-Leibler divergency measure considered. The main purpose of this paper is to show that minimizing the expected posterior Kullback-Leibler divergency measure., the Bayesian method leads to the standard tests obtained using the classical methods. The method is applied to develop tests for testing the equality of (i) location parameters of two independent normal distributions: univariate and multivariate cases, and exponential distributions; (ii) slope parameters of two simple regression models. Finally, tests for multiple comparison for both normal and exponential populations have been developed.