Smart-to-Compress: A Predictive and Game-Theoretic Framework for Data Reduction Decisions

With the rapid growth of data, redundancy among different users in cloud environments has become increasingly prominent. Detecting and removing these redundant parts can effectively improve storage efficiency. But these processes may dramatically degrade the system performance, especially when dealing with similar data. Although deduplication and delta compression are common data reduction techniques, their high overhead can outweigh the benefits. As a result, users often cannot determine in advance whether compression is worthwhile for their datasets. Some approaches have attempted to solve this, but each has important limitations. Danny Harnik et al. proposed a sampling-based deduplication estimation method using linear programming, which efficiently estimates redundancy from exact duplicates. However, it fails to capture redundancy arising from similar data, thus underestimating the full compression potential. To address this limitation, we propose Smart-to-Compress, a predictive compression decision framework. We introduce the Super Feature Frequency Histogram (SFH) to capture redundancy among similar data. Combined with the Duplication Frequency Histogram (DFH), our method estimates the overall Data Reduction Ratio (DRR) without scanning the entire dataset. Furthermore, we design a game-theoretic decision model to weigh compression benefits against predicted costs, providing users with guidance on whether compression should be applied. Experiments on real-world datasets show that our method accurately predicts compression value, reduces unnecessary overhead, and offers reliable decision-making support for users.