At the end of Silicon roadmap, keeping the leakage power in tolerable limit and bridging the bandwidth gap between processor and memory have become some of the biggest challenges. Several promising Non-Volatile Memories (NVMs) such as, Spin-Transfer Torque RAM (STTRAM), Magnetic RAM (MRAM), Phase Change Memory (PCM), Resistive RAM (RRAM) and Ferroelectric RAM (FeRAM) are being investigated to address the above issues since they offer high density and consumes zero leakage power. On one hand, the desirable properties of emerging NVMs make them suitable candidates for several applications including replacement of conventional memories. On the other hand, their unique characteristics such as, high and asymmetric read/write current and persistence bring new threats to data security and privacy. Some of these memories are already deployed in full systems and as discrete chips and are believed to become ubiquitous in future computing devices. Therefore, it is of utmost important to investigate their security and privacy issues. Note that these NVMs can be considered for cache, main memory or storage application. They are also suitable to implement in-memory computation which increases system throughput and eliminates Von-Neumann Bottleneck. Compute-capable NVMs impose new security and privacy challenges that are fundamentally different than their storage counterpart. This work identifies NVM vulnerabilities, attack vectors originating from device level all the way to circuits and systems considering both storage and compute applications. We also summarize the circuit/system level countermeasures to make the NVMs robust against security and privacy issues.