The continued evolution and diversity of malware constitutes a major threat in modern systems. It is well proven that security defenses currently available are ineffective to mitigate the skills and imagination of cyber-criminals necessitating the development of novel solutions. Deep learning algorithms and artificial intelligence (AI) are rapidly evolving with remarkable results in many application areas. Following the advances of AI and recognizing the need for efficient malware detection methods, this paper presents a new approach for malware detection based on binary visualization and self-organizing incremental neural networks. The proposed method's performance in detecting malicious payloads in various file types was investigated and the experimental results showed that a detection accuracy of 91.7% and 94.1% was achieved for ransomware in .pdf and .doc files respectively. With respect to other formats of malicious code and other file types, including binaries, the proposed method behaved well with an incremental detection rate that allows efficiently detecting unknown malware at real-time.