In areas infested with Aedes aegypti mosquitoes it may be possible to control dengue, and some other vector-borne diseases, by introducing Wolbachia-infected mosquitoes into the wildtype population. Thus far, empirical and theoretical studies of Wolbachia release have tended to focus on the dynamics at the community scale. However, Ae. aegypti mosquitoes typically dwell in and around the same houses as the people they bite and it can be insightful to explore what happens at the household scale where small population sizes lead to inherently stochastic dynamics. Here we use a continuous-time Markov framework to develop a stochastic household model for small populations of wildtype and Wolbachia-infected mosquitoes. We investigate the transient and long term dynamics of the system, in particular examining the impact of stochasticity on the Wolbachia invasion threshold and bistability between the wildtype-only and Wolbachia-only steady states previously observed in deterministic models. We focus on the influence of key parameters which determine the fitness cost of Wolbachia infection and the probability of Wolbachia vertical transmission. Using Markov and matrix population theory, we derive salient characteristics of the system including the probability of successful Wolbachia invasion, the expected time until invasion and the probability that a Wolbachia-infected population reverts to a wildtype population. These attributes can inform strategies for the release of Wolbachia-infected mosquitoes. In addition, we find that releasing the minimum number of Wolbachia-infected mosquitoes required to displace a resident wildtype population according to the deterministic model, only results in that outcome about 20% of the time in the stochastic model; a significantly larger release is required to reach a steady state composed entirely of Wolbachia-infected mosquitoes 90% of the time.