Research Cache Replacement Strategy in Memory Optimization of Spark

Caili Zhao, Yong Liu, Ximei Du, Xuezhen Zhu

In Spark, using LRU to implement RDD cache replacement. Its metrics do not take the data characteristics of Spark into account, resulting in memory not being effectively utilized, affecting task execution efficiency. This paper optimizes the LWR (Least Weight Replacement) algorithm, and a new replacement algorithm is proposed. Considering the parallel computing, the dependency integrity impact factor is added to the weight calculation to make the RDD partition weight value more accurate, so as to improve the accuracy of the cache replacement object selection, and the relevant factor values are dynamically adjusted according to the task execution, so that the cache replacement can adapt to the changes in the task execution process. The source of the experimental data set for this article is the Stanford Network Analysis Project. According to comparison experiments, this methods can effectively improve task execution efficiency.

https://doi.org/10.31871/IJNTR.5.9.10