基于强化学习的Lustre文件系统的性能调优

Translated title of the contribution: Performance Optimization of Lustre File System Based on Reinforcement Learning

Wentao Zhang, Lu Wang, Yaodong Cheng

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

Computing of high energy physics is a typical data-intensive application. The throughput and response time of distributed storage system are key performance indicators, and they are often the targets of performance optimization. There are a large number of parameters that can be adjusted in a distributed storage system. The setting of these parameters has great influence on the performance of the system. At present, these parameters are either set with static values or automatically tuned by some heuristic rules defined by experienced administrators. Neither of the method is optimistic taking into account the diversity of data access patterns and hardware capabilities, and the difficulty of finding heuristic rules for hundreds of interacted parameters based on human experience. In fact, if the tuning engine is regarded as an agent and the storage system is regarded as the environment, the parameter adjustment problem of the storage system can be treated as a typical sequential decision problem. Therefore, based on data access characteristics of high energy physics calculation, we propose an automated parameter tuning method using the reinforcement learning. Experiments show that in the same test environment, using the default parameters of the Lustre file system as a baseline, this method can increase the throughput by about 30%.

Translated title of the contributionPerformance Optimization of Lustre File System Based on Reinforcement Learning
Original languageChinese (Traditional)
Pages (from-to)1578-1586
Number of pages9
JournalJisuanji Yanjiu yu Fazhan/Computer Research and Development
Volume56
Issue number7
DOIs
StatePublished - 1 Jul 2019
Externally publishedYes

Fingerprint

Dive into the research topics of 'Performance Optimization of Lustre File System Based on Reinforcement Learning'. Together they form a unique fingerprint.

Cite this