Manipulation models for electoral systems are a core research theme in social choice theory; they include bribery (unweighted, weighted, swap, shift, ...), control (by adding or deleting voters or candidates), lobbying in referenda and others. We develop a unifying framework for manipulation models with few types of people, one of the most commonly studied scenarios. A critical insight of our framework is to separate the descriptive complexity of the voting rule R from the number of types of people. This allows us to finally settle the computational complexity of R-Swap Bribery, one of the most fundamental manipulation problems. In particular, we prove that R-Swap Bribery is fixed-parameter tractable when R is Dodgson's rule and Young's rule, when parameterized by the number of candidates. This way, we resolve a long-standing open question from 2007 which was explicitly asked by Faliszewski et al. [JAIR 40, 2011]. Our algorithms reveal that the true hardness of bribery problems often stems from the complexity of the voting rules. On one hand, we give a fixed-parameter algorithm parameterized by number of types of people for complex voting rules. Thus, we reveal that R-Swap Bribery with Dodgson's rule is much harder than with Condorcet's rule, which can be expressed by a conjunction of linear inequalities, while Dodson's rule requires quantifier alternation and a bounded number of disjunctions of linear systems. On the other hand, we give an algorithm for quantifier-free voting rules which is parameterized only by the number of conjunctions of the voting rule and runs in time polynomial in the number of types of people. This way, our framework explains why Shift Bribery is polynomial-time solvable for the plurality voting rule, making explicit that the rule is simple in that it can be expressed with a single linear inequality, and that the number of voter types is polynomial.