This article is contributed. See the original author and article here.

This article explain Migration of HBase 1.1 (HDI 3.6) Accelerated Write Cluster with Default Ambari Meta DB to HBase 2.1 (HDI 4.0) Accelerate write Cluster with custom Ambari Meta DB. In normal cluster creation, as described in other articles such as Set up clusters in HDInsight, Ambari is deployed in an S0 Azure SQL Database that is managed by HDInsight and is not accessible to users.


Also Starting July 1st 2021 , Microsoft will offer only Basic support plan for certain HDInsight 3.6 cluster types. This plan will be available till April 3rd 2022. So it is recommended to migrate to HDInsight4.0 at the earliest.


 


Understanding the Use Case:


HDInsight allows you to take control of your data and metadata with external data stores. This feature is available for Apache Hive metastore, Apache Oozie metastore, and Apache Ambari database. Here we will focus on Apache Ambari database. Ambari is used to monitor HDInsight clusters, make configuration changes and store cluster management information as well as job history. HDInsight provides a default SQL Database for each cluster which is good for test work load. For Production usage it is recommended to use Custom SQL Database to handle the load of cluster according to the business growth requirements. It is also possible to start with a basic database and upgrade later.
In this example We will create a Custom Meta DB and configure it to HDI4.0 HBase cluster and migrate the Data from HDI3.6 to HDI4.0 followed by validation.


somnathghosh_1-1621280486941.png


Below are the steps for Migration.


Source and Destination Cluster setup


Step 1 : Create a source HBase HDI 3.6 with Default meta DB


 HDInsight Cluster Setup


Step 2: Create a Destination HBase HDI 4.0 clusters with a custom Ambari DB


   Step 2.1: From Azure Portal Create an External SQL Database.


   HDInsight Custom Ambari DB Setup


   Step 2.2: Choose the right DTU based on the Nodes.


somnathghosh_2-1621280486947.png


 


somnathghosh_3-1621280486958.png


    Step 2.3: Choose the above Database while Creating HDInsight Cluster as Ambari Meta DB.


somnathghosh_4-1621280486966.png


Once the cluster is ready follow the below steps to Migrate:


 


Steps to be followed on Source Cluster HDInsight 3.6


Step 1: Login to Source Cluster and Create Sample Table using HBase perf.


somnathghosh_5-1621280486969.png


Step 2: Flush the Table Data


somnathghosh_6-1621280486971.png


Step 3: Stop the HBase from Ambari.


somnathghosh_7-1621280486982.png


Step 4: Backup WAL folder


somnathghosh_8-1621280486983.png


 


Steps to be followed on Destination Cluster HDInsight 4.0


Step 1: Stop the HBase from Ambari


somnathghosh_9-1621280487002.png


Step 2: Under Services > HDFS > Configs > Advanced > Advanced core-site, change the fs.defaultFS HDFS setting to point to the source cluster’s container name, for example cluster1testhbase-2021-05-12t07-23-50-453z


somnathghosh_10-1621280487019.png


Step 3: Under Services > HBASE > Configs > Advanced > Advanced hbase-site change the hbase.rootdir path to point to the container of the source cluster.


somnathghosh_11-1621280487034.png


Step 4: Clean the Zookeeper data on the destination cluster by running the following commands in any of the Zookeeper nodes or worker nodes:


somnathghosh_12-1621280487034.png


 


somnathghosh_13-1621280487035.png


Step 5: Restart all the component required restart from Ambari.


Step 6: Clean the WAL FS data for the destination cluster, and copy the WAL directory from the source cluster into the destination cluster’s HDFS. Copy the directory by running the following commands in any of the Zookeeper nodes or worker nodes:


somnathghosh_14-1621280487036.png


somnathghosh_15-1621280487040.png


Step 7: Copy apps folder from destination container to source container


somnathghosh_16-1621280487042.png


Step 8: Restart all the component required restart from Ambari.


Step 9: Validation


Validation of the table and count of record in source cluster


somnathghosh_17-1621280487044.png


Count:


somnathghosh_18-1621280487045.png


 


Validation of the table and count of record in destination cluster


somnathghosh_19-1621280487046.png


Count:


somnathghosh_20-1621280487047.png


 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.