Cluster Management using Apache Mesos, Marathon

Static Partitioning :

Static Partitioning for different applications.
  • Memory Utilization : We are not fully utilizing our existing servers.
  • Higher cost

Problems before Mesos ?

  • How to manage NFR (Non Functional Requirenment)— out of the Box.

Types of Resource utilization :

  • Intra machine resource sharing : Share a single machine’s resources between multiple apps (multi-tenancy).
  • Intra-datacenter resource sharing : Share multiple machine’s resources between multiple apps.

What is a Cluster Manager?

  • provides a level-of-indirection between Hardware resources (machines) and Applications/Jobs
  • Piece of software doing the mediation
  • Humans will not manually schedule Apps/Jobs on your cluster nodes.

What is Mesos?

 Mesos the Cluster Manager :

Dynamic Partitioning Using Mesos for all applications.

Components of Mesos :

  • Modern general purpose Cluster manager:

Mesos — Design philosophy :

  • Kernel — does the resource allocation and sharing
  • Frameworks — task scheduling, execution & Fault tolerance

Mesos is like an OS/Datacenter kernel :

  • User space scheduling
  • Level of Abstraction
  • Builds and runs distributed systems using Resources
  • Build & run a PAAS on top of Mesos kernel
  • Run Mesos on top of Physical machines or EC2 or OpenStack [IAAS]

Solution Architecture : (The most important part of this page)

Solution Architecture for Mesos using Marathon as its scheduler.
  1. Mesos slaves reports resources availability (offers) to master.
  2. Mesos master send these offers to marathon framework.
  3. Framework scheduler replies to the master with information about tasks to run on slave using required CPUs and RAMs for different tasks.
  4. The Mesos master sends the tasks details to load balancer, which distributes the task to different slaves.
  5. Finally the task is used to send to slaves which allocate resources to framework, which launches to tasks.
  6. Resource will be free after task completion.

Mesos — functionality for every Distributed System :

  • Failure detection
  • Package distribution
  • Resource isolation
  • Task distribution
  • Task starting
  • Task monitoring
  • Task killing
  • Task cleanup

Master — The kernel :

  • Allocating resources to different frameworks
  • Flexibility — accommodate diverse frameworks
  • Scalability- scheduler can scale as number of machines & apps increases
  • Fairness — in allocating resources to users/frameworks
  • Fine grained resource sharing using resource offers
  • Manages the Task lifecycle for frameworks
  • An offer represents some resources available on slave.

Slave :

  • Have resources
  • Responsible for executing tasks — assigned by Frameworks
  • Isolation for each Task
  • Each Task should get the exact resources — not more or less
  • Master manages resources on Slaves
  • Resources of Slaves are consumed by Tasks
  • Slave resources are managed by Master and allocated to Frameworks
  • Slave will send offers to Mesos Master along with Key-Value pairs [ attributes ] of the Slave
  • Frameworks uses the Slave attributes in Task management

Resources — offers :

  • Resource attributes — cpus, mem, disk, ports
  • Mem and disk are in MB

Frameworks (Marathon):

  • Distributed apps which run on Mesos cluster are called Frameworks
  • 2 components : Scheduler & Executor
  • Runs number of Tasks
  • Tasks consume resources
  • Tasks Lifecycle & management — functions
  • Schedulers- coordinating the execution
  • Executors — control the Task execution, run multiple tasks
  • Provides API to communicate with Scheduler and Executor

Scheduler :

  • What Computation should run?
  • Where should Computation run?
  • Talks to Master
  • Master responsible for allocation of Resources

Executer :

  • Start the task on the scheduled slave.

Mesos — value addition :

  • Provides a Data center kernel
  • High level abstraction to develop apps that treats Distributed Infra, just like a Single large computer
  • Devs can only focus on App logic & not worry about Infrastructure
  • Helps in resource allocation, deployment, monitoring and isolation
  • Devs need to know what Resources are needed and not how to get resources

Apache Mesos Installation :

--

--

--

DevOps Consultant | Cloud Engineer | Security | CI/CD | HA | AWS | Docker | Kubernetes | Aerospike | Cassandra | Rabbitmq | Consul | MongoDB

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

PRV For Your Thoughts: Flex your PrivacyCards!

A Software Degree I never Used (But will change that)

Alvin’s Dew Drop Daily — Issue #57

Permutation using Python

Flutter Custom Paint Tutorial | Build a Radial Progress

CakePHP 4 Tutorial Part 3: Working with Elements and Detault Layout

java.lang.Object.finalize() is finally deprecated

Why use Meteor Up(MUP) to deploy your app?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Kailash Verma

Kailash Verma

DevOps Consultant | Cloud Engineer | Security | CI/CD | HA | AWS | Docker | Kubernetes | Aerospike | Cassandra | Rabbitmq | Consul | MongoDB

More from Medium

Git Rebase — Behind the scenes

Writing Custom Operators and Hooks for MWAA

Deep Dive into Google’s AlloyDB Architecture for PostgreSQL

Parsing CVS file from Google Storage using Cloud Function, DataProc and java application using…