Ambari – Identify Nodes with Slave Down components

Apache Ambari Project If you are using Apache Ambari to manage your hadoop cluster you know that it provides a neat interface to administer your cluster. It however can have some issues when you get the size of your cluster growing. Specially if you are performing troubleshooting a really large size of cluster. To troubleshoot the nodes where services are failing to startup for example can be a problem on this size of cluster.

With all the filters available in Ambari you can identify the hosts or nodes where the slave components are down. But this can get complicated when you have multiple slave components down in a cluster running thousands of nodes.

A simple filter in this case does not really help. In fact, it does makes matter worse at times. So how do you identify the nodes where slave components are down. Ambari API is the answer!

Ambari – Identify Nodes with Slave Down components

I would like to walk through this article with an example. Let’s consider the example of a cluster running 1200 nodes. As a part of maintenance we have all the services down. While bringing the services back up I am reported around 15 datanodes down. I am yet to get the node managers up. So the “Slave Down” filter won’t really help in this case.

What helps me here is the following Ambari API REST call:

GET /api/v1/clusters/c1/services/HDFS/components/DATANODE

The only catch here is that I will provide the fields I want to see. The following is what I did to get the list of Nodes where the slave components is down (Datanode in this case).

curl -s -u admin:admin -H GET http://localhost:8080/api/v1/clusters/CLUSTER_NAME/services/HDFS/components/DATANODE?fields=host_components/HostRoles/state | grep -B1 INSTALLED | grep host_name | cut -d\” -f4

And there you have it! A nice list of nodes where the DataNode is down.

Note: Please make sure you use the parameters which suit your Ambari installation.

* admin:admin – with the username and password you’ve set for Ambari. I am using the default username and password in my examples.
* CLUSTER_NAME – with the cluster name you have set for your cluster in Ambari.

You can simply modify the above REST call and use it for your requirement to a different service component.

Ambari – Identify Nodes with Slave Down components

No Comments

Post a Comment

Time limit is exhausted. Please reload CAPTCHA.