FAST AND SECURE INTEGRITY VERIFICATION FOR HIGH-SPEED DATA TRANSFERS

Abstract

The amount of data generated by scientific and commercial applications is growing at an ever-increasing pace. This data is often moved between geographically distributed sites for various purposes such as collaboration and archival, leading to significant increase in data transfer rates. Surge in data transfer rates when combined with proliferation of scientific applications that cannot tolerate data corruption paved the way for the development of end-to-end integrity verification technique to protect data transfers against silent data corruption. Integrity verification minimizes the likelihood of silent data corruption by comparing checksum of files at source and destination servers using secure hash algorithms, such as MD5 and SHA256. However, it imposes significant performance penalty due to overhead of checksum computation. In this dissertation, we propose Fast Integrity VERification (FIVER) algorithm which overlaps checksum computation and data transfer operations to reduce the overhead of integrity verification. Experimental results show that FIVER is able to bring down the overhead of end-to-end integrity verification from 60% by the state-of-the-art solutions to below 10% by concurrently executing transfer and checksum operations.