Moderate Density Parity Check (MDPC) codes are defined here as codes which have a parity-check matrix whose row weight is $O(\sqrt{n})$ where $n$ is the length $n$ of the code. They can be decoded like LDPC codes but they decode much less errors than LDPC codes: the number of errors they can decode in this case is of order $Θ(\sqrt{n})$. Despite this fact they have been proved very useful in cryptography for devising key exchange mechanisms. They have also been proposed in McEliece type cryptosystems. However in this case, the parameters that have been proposed in \cite{MTSB13} were broken in \cite{GJS16}. This attack exploits the fact that the decoding failure probability is non-negligible. We show here that this attack can be thwarted by choosing the parameters in a more conservative way. We first show that such codes can decode with a simple bit-flipping decoder any pattern of $O\left(\frac{\sqrt{n} \log \log n}{\log n}\right)$ errors. This avoids the previous attack at the cost of significantly increasing the key size of the scheme. We then show that under a very reasonable assumption the decoding failure probability decays almost exponentially with the codelength with just two iterations of bit-flipping. With an additional assumption it has even been proved that it decays exponentially with an unbounded number of iterations and we show that in this case the increase of the key size which is required for resisting to the attack of \cite{GJS16} is only moderate.