Multiple input multiple output techniques are considered attractive for future wireless communication systems, due to the continuing demand for high data rates, spectral efficiency, suppress interference ability and robustness of transmission. MIMO-OFDM is very helpful to transmit high data rate in wireless transmission and provides good maximum system capacity by getting the advantages of both MIMO and OFDM. The main problem in this system is that increase in number of transmit and receive antennas lead to hardware complexity. To tackle this issue, an effective optimal transmit antenna subset selection method is proposed in paper with the aid of Adaptive Mutation Genetic Algorithm (AGA). Here, the selection of transmit antenna subsets are done by the adaptive mutation of Genetic Algorithm in MIMO-OFDM system. For all the mutation points, the fitness function are evaluated and from that value, best fitness based mutation points are chosen. After the selection of best mutation points, the mutation process is carried out, accordingly. The implementation of proposed work is done in the working platform MATLAB and the performance are evaluated with various selection of transmit antenna subsets. Moreover, the comparison results between the existing GA with mutation and the proposed GA with adaptive mutation are discussed. Hence, using the proposed work, the selection of transmit antenna with the maximum capacity is made and which leads to the reduced hardware complexity and undisturbed data rate in the MIMO-OFDM system