We propose a model to analyze citation growth and influences of fitness (competitiveness) factors in an evolving citation network. Applying the proposed method to modeling citations to papers and scholars in the InfoVis 2004 data, a benchmark collection about a 31-year history of information visualization, leads to findings consistent with citation distributions in general and observations of the domain in particular. Fitness variables based on prior impacts and the time factor have significant influences on citation outcomes. We find considerably large effect sizes from the fitness modeling, which suggest inevitable bias in citation analysis due to these factors. While raw citation scores offer little insight into the growth of InfoVis, normalization of the scores by influences of time and prior fitness offers a reasonable depiction of the field's development. The analysis demonstrates the proposed model's ability to produce results consistent with observed data and to support meaningful comparison of citation scores over time.