Introduction

==

This project's main goal is to define a simpler (developer friendly) YARN API, allowing end user to concentrate on the functional aspects of an application while delegating infrastructure and boilerplate code dealing with YARN internals to the framework.

At the moment the framework is still evolving and primarily the result of personal investigation and understanding of YARN, so updates may be frequent. For now I just felt like sharing what's been learned to this point.

At the end, the main goal is to make YARN simpler to use and more developer friendly. This means not only the API but ability for rapid development where you can write and test/deploy your applications right from your IDE.

For example:

Below is the simplest possible way to create and start YARN application using just a few defined abstractions_

// Create YARN application
YarnConfiguration yarnConfiguration = new YarnConfiguration();
YarnApplication<Void> yarnApplication = YarnAssembly.forApplicationContainer("ping -c 4 yahoo.com").
		withApplicationMaster(yarnConfiguration).
			build("CommandBasedYarnApplicationDemo");

// Launch YARN application
yarnApplication.launch();

This is it! The above application will create an instance of a YARN's Application Master and a single Application Container which in this case will simply execute the Unix ping -c 4 yahoo.com command printing its output to the containers log.

==

Assembly DSL

Application Master and Application Container

Demos

==

Assembly DSL

You can see that you begin by assembling your Application Container specification. You do so by calling YarnAssembly.forApplicationContainer(..) method. That is the only method you can invoke on YarnAssembly. Once invoked, you can invoke any of the available methods to override default setting for Application Container (e.g., memory, virtualCores etc). Invocation of any available method will exclude such method from subsequence invocation. For example; YarnAssembly.forApplicationContainer("ls -all").containerCount(2).containerCount(2) will not compile, thus precluding you from making assembly mistakes. It is accomplished through context-aware type inference during the assembly. In other words this "hold-your-hand" approach essentially implies that if a method available during the compilation then its legal to be invoked, otherwise it wouldn't be visible.

Once you've provided specification for your Application Container you can now define specification for the Application Master.

==

Application Master and Application Container

YAYA formalizes and exposes for configuration and assembly two core abstractions provided by YARN:

1. Application Master

2. Application Container

The above application will use the default Application Master and Application Container settings. For example; the containerCount will be set to 1, the memory for Application Master will be set to 512Mb and 256Mb for Application Container. For more on these default values please refer to YarnAssembly javadoc. While defaults are there they may not always cary the values required by your application. Through a convenient and intuitive DSL you can set/override these values. For example:

// Create YARN application
YarnConfiguration yarnConfiguration = new YarnConfiguration();
YarnApplication<Void> yarnApplication = YarnAssembly.forApplicationContainer("ping -c 4 yahoo.com").
                memory(1024).
                withApplicationMaster(yarnConfiguration).
                     memory(2048).
                     build("CommandBasedYarnApplicationDemo");

// Launch YARN application
yarnApplication.launch();

The above application assembly is identical to the previous one in many ways. However you can see that it explicitly sets memory to 1024Mb and 2048Mb on Application Container and Application Master respectively. Please refer to YarnAssembly for more settings.

Once all of the parameters were set on the Application Master specification you build("application_name") method which will return YarnApplication.

Once you have an instance of YarnApplication you can no longer update specification of either Application Container or Application Master. You are now ready to launch.

==

YarnApplication

YarnApplication represents a fully assembled YARN application. It encapsulates all of the required specification information for both Application Master and Application Container allowing YAYA to use this information and launch your application on YARN.

To launch your application simply invoke its YarnApplication.launch() method. You will notice that based on your YarnAssembly the launch() method may return void or an array of ContainerDelegates. For more information on this return type and what it means read the Application Container styles section of the Core Features.

==

DEMOS

There are several demos available in YAYA Demos project. They are all simple and self-contained which means you can run them with little to no preparation. Since the framework comes with its own Mini Cluster, all demos that require cluster are already setup to run in it since the appropriate yarn-site.xml is already in the classpath.

NOTE: The Mini Cluster is already setup with In-JVM Container Executor. Please read more here to learn how to change back to the default executor. For testing in real cluster sample configuration files are provided here. More demos will come, but few worth mentioning:

Interactable Yarn Application

This demo showcases long-running reusable java-based containers you can interact with by exchanging messages. More information about it is available in javadoc

Java-based Application

This demo showcases java-based Application Containers through implementing and configuring ApplicationContainer strategy. More information about it is available in javadoc

Command-based Application

This demo showcases Application Containers implemented as command-based (unix, perl, etc.) containers. More information about it is available in javadoc

===

Please send question and updates via pull requests and/or raising issues on this project.

Cheers!