The partition algebra is an associative algebra with a basis of set-partition diagrams and multiplication given by diagram concatenation. It contains as subalgebras a large class of diagram algebras including the Brauer, planar partition, rook monoid, rook-Brauer, Temperley-Lieb, Motzkin, planar rook monoid, and symmetric group algebras. We give a construction of the irreducible modules of these algebras in two isomorphic ways: first, as the span of symmetric diagrams on which the algebra acts by conjugation twisted with an irreducible symmetric group representation and, second, on a basis indexed by set-partition tableaux such that diagrams in the algebra act combinatorially on tableaux. The first representation is analogous to the Gelfand model and the second is a generalization of Young's natural representation of the symmetric group on standard tableaux. The methods of this paper work uniformly for the partition algebra and its diagram subalgebras. As an application, we express the characters of each of these algebras as nonnegative integer combinations of symmetric group characters whose coefficients count fixed points under conjugation.