In this article, we have proposed a generalized Lucas matrix (recursive matrix of higher order) having relation with generalized Fibonacci sequences and established many special properties in addition to that usual matrix algebra. Further, we have proposed a modified public key cryptography using these matrices as keys in Affine cipher and key agreement for encryption-decryption with the combination of terms of generalized Lucas sequences under residue operations. In this scheme, instead of exchanging the whole key matrix, only a pair of numbers(parameters) need to be exchanged, which reduces the time complexity as well as space complexity of the key transmission and has a large key-space.