Microsoft Malware Detection

Malware (a portmanteau for malicious software) is any software intentionally designed to cause damage to a computer, server, client, or computer network (by contrast, software that causes unintentional harm due to some deficiency is typically described as a software bug). A wide variety of types of malware exist, including computer viruses, worms, Trojan horses, ransomware, spyware, adware, and scareware. Programs are also considered malware if they secretly act against the interests of the computer user. For example, at one point Sony music Compact discs silently installed a rootkit on purchasers’ computers with the intention of preventing illicit copying, but which also reported on users’ listening habits, and unintentionally created extra security vulnerabilities.

Microsoft has been very active in building anti-malware products over the years and it runs it’s anti-malware utilities over 150 million computers around the world. This generates tens of millions of daily data points to be analyzed as potential malware. In order to be effective in analyzing and classifying such large amounts of data, we need to be able to group them into groups and identify their respective families. This dataset provided by Microsoft contains about 9 classes of malware.

Total train dataset consist of 200GB data out of which 50Gb of data is .bytes files and 150GB of data is .asm files. Lots of Data for a single-box/computer. There are total 10,868 .bytes files and 10,868 asm files total 21,736 files.

Types of Malware : Ramnit, Lollipop, Kelihos_ver3, Vundo, Simda, Tracur, Kelihos_ver1, Obfuscator.ACY, Gatak

Problem Statement : In the past few years, the malware industry has grown very rapidly that, the syndicates invest heavily in technologies to evade traditional protection, forcing the anti-malware groups/communities to build more robust softwares to detect and terminate these attacks. The major part of protecting a computer system from a malware attack is to identify whether a given piece of file/software is a malware.

Source : https://www.kaggle.com/c/malware-classification/data

Real-world/Business Objectives and Constraints :

  1. Minimize multi-class error.
  2. Multi-class probability estimates.
  3. Malware detection should not take hours and block the user’s computer. It should fininsh in a few seconds or a minute.

To learn more please visit : Here