The parameters of support vector machines (SVMs) such as the penalty parameter and the kernel parameters have a great impact on the classification accuracy and the complexity of the SVM model. Therefore, the model selection in SVM involves the tuning of these parameters. However, these parameters are usually tuned and used as a black box, without understanding the mathematical background or internal details. In this paper, the behavior of the SVM classification model is analyzed when these parameters take different values with balanced and imbalanced data. This analysis including visualization, mathematical and geometrical interpretations and illustrative numerical examples with the aim of providing the basics of the Gaussian and linear kernel functions with SVM. From this analysis, we proposed a novel search algorithm. In this algorithm, we search for the optimal SVM parameters into two one-dimensional spaces instead of searching into one two-dimensional space. This reduces the computational time significantly. Moreover, in our algorithm, from the analysis of the data, the range of kernel function can be expected. This also reduces the search space and hence reduces the required computational time. Different experiments were conducted to evaluate our search algorithm using different balanced and imbalanced datasets. The results demonstrated how the proposed strategy is fast and effective than other searching strategies.