In this paper, we have proposed a modified cryptographic scheme based on the application of recursive matrices as key in ECC and ElGamal. For encryption, we consider mapping analogous to affine Hill cipher in which a plaintext matrix has been constructed by points corresponding to letters on elliptic curves. In the formation of key-space, the generalized Fibonacci matrices have been taken into account, which is the sequence of matrices. The beauty of considering Fibonacci matrices is their construction where we need only two parameters(integers) in place of $n^2$ elements. The use of a recursive matrix makes a large keyspace for our proposed scheme and increases its efficiency. Thus, it reduces time as well space complexity, and its security \& strength is based on EC-DLP which is a hard problem in number theory.