Let $G=(V,E)$ be a vertex-colored graph, where $C$ is the set of colors used to color $V$. The Graph Motif (or GM) problem takes as input $G$, a multiset $M$ of colors built from $C$, and asks whether there is a subset $S\subseteq V$ such that (i) $G[S]$ is connected and (ii) the multiset of colors obtained from $S$ equals $M$. The Colorful Graph Motif (or CGM) problem is the special case of GM in which $M$ is a set, and the List-Colored Graph Motif (or LGM) problem is the extension of GM in which each vertex $v$ of $V$ may choose its color from a list $\mathcal{L}(v)\subseteq C$ of colors. We study the three problems GM, CGM, and LGM, parameterized by the dual parameter $\ell:=|V|-|M|$. For general graphs, we show that, assuming the strong exponential time hypothesis, CGM has no $(2-ε)^\ell\cdot |V|^{\mathcal{O}(1)}$-time algorithm, which implies that a previous algorithm, running in $\mathcal{O}(2^\ell\cdot |E|)$ time is optimal [Betzler et al., IEEE/ACM TCBB 2011]. We also prove that LGM is W[1]-hard with respect to $\ell$ even if we restrict ourselves to lists of at most two colors. If we constrain the input graph to be a tree, then we show that GM can be solved in $\mathcal{O}(3^\ell\cdot |V|)$ time but admits no polynomial-size problem kernel, while CGM can be solved in $\mathcal{O}(\sqrt{2}^{\ell} + |V|)$ time and admits a polynomial-size problem kernel.