Paper Title :Dynamic Task Management of Big Data In Hadoop Yarn: A Synergistic Approach Using Deep Learning and Federated Learning

Author :Reshu Grover, Manisha Agarwal

Article Citation :Reshu Grover ,Manisha Agarwal , (2024 ) " Dynamic Task Management of Big Data In Hadoop Yarn: A Synergistic Approach Using Deep Learning and Federated Learning " , International Journal of Advances in Computer Science and Cloud Computing (IJACSCC) , pp. 18-26, Volume-12,Issue-2

Abstract : Hadoop-based big data YARN (Yet Another Resource Negotiator) is a framework for managing and analyzing massive amounts of data across distributed clusters. Many existing approaches developed various DL approaches, but they contain some limitations.Therefore, the proposed paper presents the conceptual model for enhancing the efficiency and effectiveness of big data processing in the distributed setting. Initially, the job submission and parameterization are where the users present the Hadoop YARNwith the required number of CPU cores, memory, and time for the job. An ensemble deep learning model is employed for the prediction of resources, time, and energy. Then the tasks are divided as per the predicted load and clustered by using a new hybrid Walrus Optimize (WO) and Secretary Bird Optimization (SBO) Algorithm. After clustering, load balancing, and resource allocation, new methods like Consistent Hashing with Weighted Round Robin are employed for the task allocation to minimize the waiting time. After that, dynamic load balancing with reinforcement learning is incorporated where the jobs are dynamically assigned and rebalanced using Deep Q-Networks (DQN). Then, federated learning enhances the predictive capacity of the models and at the same time, the data is safeguarded. Finally, realtime monitoring of the resources and identification of the anomalies in which the actions of predictive maintenance are taken based on the performance indicators. Keywords - YARN (Yet Another Resource Negotiator), Deep Q-Networks (DQN), Hadoop Distributed File System (HDFS), Big Data Analytics (BDA), Deep Reinforcement Learning (DRL)

Type : Research paper

Published : Volume-12,Issue-2

	|		PDF	|	Viewed - 12	|	Published on 2025-03-13