A system and method for detecting malware using hierarchical clustering analysis. Unknown files classified by clustering and in view of known malicious and known safe files. Machine learning models and detection rules are used to enhance classification accuracy.