The stacked contact process is a three-state spin system that describes the co-evolution of a population of hosts together with their symbionts. In a nutshell, the hosts evolve according to a contact process while the symbionts evolve according to a contact process on the dynamic subset of the lattice occupied by the host population, indicating that the symbiont can only live within a host. This paper is concerned with a generalization of this system in which the symbionts may affect the fitness of the hosts by either decreasing (pathogen) or increasing (mutualist) their birth rate. Standard coupling arguments are first used to compare the process with other interacting particle systems and deduce the long-term behavior of the host-symbiont system in several parameter regions. The mean-field approximation of the process is also studied in detail and compared to the spatial model. Our main result focuses on the case where unassociated hosts have a supercritical birth rate whereas hosts associated to a pathogen have a subcritical birth rate. In this case, the mean-field model predicts coexistence of the hosts and their pathogens provided the infection rate is large enough. For the spatial model, however, only the hosts survive on the one-dimensional integer lattice.