This work presents some novel techniques to enhance an encryption scheme motivated by classical McEliece cryptosystem. Contributions include: (1) using masking matrices to hide sensitive data, (2) allowing both legitimate parties to incorporate randomness in the public key without sharing any additional public information, (3) using concatenation of a repetition code for error correction, permitting key recovery with a negligible decoding complexity, (4) making attacks more difficult by increasing the complexity in verifying a given key candidate has resulted in the actual key, (5) introducing memory in the error sequence such that: (i) error vector is composed of a random number of erroneous bits, (ii) errors can be all corrected when used in conjunction with concatenation of a repetition code of length 3. Proposed techniques allow generating significantly larger keys, at the same time, with a much lower complexity, as compared to known post-quantum key generation techniques relying on randomization.