PROJECT TITLE :

MarVeLScaler: A Multi-View Learning-Based Auto-Scaling System for MapReduce

ABSTRACT:

Tenants are clamoring for automatic tools that can auto-estimate the amount of resources that MapReduce jobs will require so that cloud computing can transition from its current pay-per-request model to a genuine pay-per-use model. These types of tools require accurately quantifying the relationship between the amount of work to be done, the resources available, and the amount of time needed to complete the task. There have been many different models of prediction put forth. However, none of these models takes the performance variance of virtual machines (VMs) during the execution of a job into consideration. This causes an underestimate of the required resources and causes the deadline for the job to be missed. We propose a multi-view deep learning model as a solution to this issue. This model will capture real-time performance variance and will automatically scale out the cloud cluster whenever it is required to do so. We put together a prototype called MarVeLScaler, which consists of two helpful modules called Scale Estimator and Scale Controller. A specific workload and a due date are input into the Scale Estimator, which then generates a preliminary estimate of the required cluster size for a MapReduce job. Scale Controller makes adjustments to the scale of the cluster during the runtime based on the real-time running status of the cluster in order to ensure that the job is completed on time. We conduct an analysis on how well MarVeLScaler performs in Alibaba Cloud, which is based on Hadoop. Experiments have shown that MarVeLScaler is capable of providing an accuracy of prediction that is 98.4 percent accurate when determining the initial cluster size. Additionally, it can save 30.8 percent of the expense while still guaranteeing similar performance in comparison to the methods that are considered to be state-of-the-art.