| Description |
1 online resource (1 volume) : illustrations |
| Series |
Community experience distilled |
|
Community experience distilled.
|
| Contents |
Cover; Copyright; Credits; About the Authors; Acknowledgments; About the Reviewers; www.PacktPub.com; Table of Contents; Preface; Chapter 1: Starting with YARN Basics; Introduction to MapReduce v1; Shortcomings of MapReducev1; An overview of YARN components; ResourceManager; NodeManager; ApplicationMaster; Container; The YARN architecture; How YARN satisfies big data needs; Projects powered by YARN; Summary; Chapter 2: Setting up a Hadoop-YARN Cluster; Starting with the basics; Supported platforms; Hardware requirements; Software requirements; Basic Linux commands / utilities; Sudo |
|
Nano editorSource; Jps; Netstat; Man; Preparing a node for a Hadoop-YARN cluster; Install Java; Create a Hadoop dedicated user and group; Disable firewall or open Hadoop ports; Configure domain name resolution; Install SSH and configure passwordless SSH from the master to all slaves; The Hadoop-YARN single node installation; Prerequisites; Installation steps; Step 1 -- Download and extract the Hadoop bundle; Step 2 -- Configure the environment variables; Step 3 -- Configure the Hadoop configuration files; Step 4: Format NameNode; Step 5: Start Hadoop daemons; An overview of web user interfaces |
|
Run a sample applicationThe Hadoop-YARN multi-node installation; Prerequisites; Installation steps; Step 1: Configure the master node as a single-node Hadoop-YARN installation; Step 2: Copy the Hadoop folder to all the slave nodes; Step 3: Configure environment variables on slave nodes; Step 4: Format NameNode; Step 5: Start Hadoop daemons; An overview of the Hortonworks and Cloudera installations; Summary; Chapter 3: Administering a Hadoop-YARN Cluster; Using the Hadoop-YARN commands; The user commands; Jar; Application; Node; Logs; Classpath; Version; Administration commands |
|
ResourceManager / NodeManager / ProxyServerRMAdmin; DaemonLog; Configuring the Hadoop-YARN services; The ResourceManager service; The NodeManager service; The Timeline server; The web application proxy server; Ports summary; Managing the Hadoop-YARN services; Managing service logs; Managing pid files; Monitoring the YARN services; JMX monitoring; The ResourceManager JMX beans; The NodeManager JMX beans; Ganglia monitoring; Ganglia daemons; Integrating Ganglia with Hadoop; Understanding ResourceManager's High Availability; Architecture; Failover mechanisms |
|
Configuring ResourceManager High AvailabilityDefine nodes; The RM state store mechanism; The failover proxy provider; Automatic failover; High Availability admin commands; Monitoring NodeManager's health; The health checker script; Summary; Chapter 4: Executing Applications Using YARN; Understanding application execution flow; Phase 1 -- application initialization and submission; Phase 2 -- allocate memory and start ApplicationMaster; Phase 3 -- ApplicationMaster registration and resource allocation; Phase 4 -- launch and monitor containers; Phase 5 -- application progress report |
|
Phase 6 -- application completion |
| Summary |
Moving beyond MapReduce - learn resource management and big data processing using YARN About This Book Deep dive into YARN components, schedulers, life cycle management and security architecture Create your own Hadoop-YARN applications and integrate big data technologies with YARN Step-by-step guide to provision, manage, and monitor Hadoop-YARN clusters with ease Who This Book Is For This book is intended for those who want to understand what YARN is and how to efficiently use it for the resource management of large clusters. For cluster administrators, this book gives a detailed explanation of provisioning and managing YARN clusters. If you are a Java developer or an open source contributor, this book will help you to drill down the YARN architecture, write your own YARN applications and understand the application execution phases. This book will also help big data engineers explore YARN integration with real-time analytics technologies such as Spark and Storm. What You Will Learn Explore YARN features and offerings Manage big data clusters efficiently using the YARN framework Create single as well as multi-node Hadoop-YARN clusters on Linux machines Understand YARN components and their administration Gain insights into application execution flow over a YARN cluster Write your own distributed application and execute it over YARN cluster Work with schedulers and queues for efficient scheduling of applications Integrate big data projects like Spark and Storm with YARN In Detail Today enterprises generate huge volumes of data. In order to provide effective services and to make smarter and more intelligent decisions from these huge volumes of data, enterprises use big-data analytics. In recent years, Hadoop has been used for massive data storage and efficient distributed processing of data. The Yet Another Resource Negotiator (YARN) framework solves the design problems related to resource management faced by the Hadoop 1.x framework by providing a more scalable, efficient, flexible, and highly available resource management framework for distributed data processing. This book starts with an overview of the YARN features and explains how YARN provides a business solution for growing big data needs. You will learn to provision and manage single, as well as multi-node, Hadoop-YARN clusters in the easiest way. You will walk through the YARN administration, life cycle management, application execution, REST APIs, schedulers, security framework and so o.. |
| Notes |
Includes index |
|
English |
|
Online resource; title from cover (Safari, viewed September 20, 2015) |
| SUBJECT |
Apache Hadoop. http://id.loc.gov/authorities/names/n2013024279
|
|
Apache Hadoop fast |
| Subject |
Electronic data processing -- Distributed processing.
|
|
COMPUTERS -- Mathematical & Statistical Software.
|
|
Electronic data processing -- Distributed processing
|
| Form |
Electronic book
|
| Author |
Mehrotra, Shrey, author
|
| ISBN |
9781784394585 |
|
1784394580 |
|
