Understanding the evolution of binary traits, which affects the birth and survival of species and also the rate of molecular evolution, remains challenging. A typical example is the evolution of mating systems in plant species. In this work, we present a probabilistic modeling framework for binary trait, random species trees, in which the number of species and their traits are represented by a two-type, continuous time Markov branching process. We develop our model by considering the impact of mating systems on dN/dS, the ratio of nonsynonymous to synonymous substitutions. A methodology is introduced which enables us to match model parameters with parameter estimates from phylogenetic tree data. The properties obtained from the model are applied to outcrossing and selfing species trees in the Geraniaceae and Solanaceae family. This allows us to investigate not only the branching tree rates, but also the mutation rates and the intensity of selection.