Data de-duplication and event processing for security applications on an embedded processor

Network security schemes generally deploy sensors and other network devices which generate huge volumes of data, overwhelming the underlying decision making algorithms. An example is corporate networks employing intrusion detection systems where there is a deluge of alert data, confounding the computations involved in sensor information fusion and alert correlation. One way to obtain fast and real-time responses is to preprocess such data to manageable sizes. In this paper, we show that data de-duplication using computationally efficient fingerprinting algorithms can provide real-time results. We present an algorithm which utilizes Rabin Fingerprinting/hashing scheme for the purpose of data de-duplication. We have implemented this algorithm on Intel Atom, which is a powerful, energy efficient embedded processor. Our study is intended to show that the relatively low performing embedded processors are capable of providing the needed computational support if they were to handle security functions in the field. When compared to the algorithmic performance on a high end system, viz. Intel Core 2 Duo processor, the positive results obtained make a case for using the Atom processor in networked applications employing mobile devices.