CentOS分布式环境安装HBase,4下HBase集群安装

环境:CentOS
6.4,Hadoop
2.6.0,ZooKeeper 3.4.6,HBase 1.0.1.1 

环境:CentOS 6.4,Hadoop 2.6.0,ZooKeeper 3.4.6,HBase 1.0.1.1 

一、安装zookeeper

集群角色规划:

集群角色规划:

在安装好分布式Hadoop的环境下,继续安装zookeeper-3.4.5。

HostName
HBase Role
ZooKeeper
Hadoop Role
HDP1
Master YES Slave
HDP2
Backup Master & RegionServer
YES
Master
HDP3
RegionServer
YES
Slave
HDP4
RegionServer
YES
Slave
HostName
HBase Role
ZooKeeper
Hadoop Role
HDP1
Master YES Slave
HDP2
Backup Master & RegionServer
YES
Master
HDP3
RegionServer
YES
Slave
HDP4
RegionServer
YES
Slave
  1. $ wget
  2. $ tar -xf zookeeper-3.4.5.tar.gz
  3. $ cp zookeeper-3.4.5/conf/zoo_sample.cfg
    zookeeper-3.4.5/conf/zoo.cfg

    $ wget
    $ tar -xf zookeeper-3.4.5.tar.gz
    $ cp zookeeper-3.4.5/conf/zoo_sample.cfg zookeeper-3.4.5/conf/zoo.cfg

 

 

修改zookeeper配置文件:

1. 任意Master(HDP1)节点解压HBase

1. 任意Master(HDP1)节点解压HBase

  1. $ vim zookeeper-3.4.5/conf/zoo.cfg

    $ vim zookeeper-3.4.5/conf/zoo.cfg

  sudo tar zxvf hbase-1.0.1.1-bin.tar.gz -C /usr/local/

  sudo tar zxvf hbase-1.0.1.1-bin.tar.gz -C /usr/local/

做如下修改:

  sudo mv hbase-1.0.1.1 hbase

  sudo mv hbase-1.0.1.1 hbase

  1. tickTime=2000
  2. dataDir=/home/ruifeng.shan/zookeeper
  3. clientPort=2181
  4. initLimit=5
  5. syncLimit=2
  6. server.1=master:2888:3888
  7. server.2=node:2888:3888
  8. server.3=slave:2888:3888

    tickTime=2000
    dataDir=/home/ruifeng.shan/zookeeper
    clientPort=2181
    initLimit=5
    syncLimit=2
    server.1=master:2888:3888
    server.2=node:2888:3888
    server.3=slave:2888:3888

  sudo chown -R hdp:hdp hbase

  sudo chown -R hdp:hdp hbase

在你指定的dataDir目录下新建一个文件myid,在里面添加你指定的server编号。例如,你为hostname名为master的指定了如下配置

 

 

server.1=master:2888:3888 ,所以它的编号应该是1.

2. 修改hbase-env.sh 

2. 修改hbase-env.sh 

将zookeeper-3.4.5分发到其他机器。然后修改对应的myid文件内容为之前给其指定的编号。如node为2,slave为3.

   vim conf/hbase-env.sh,出掉JAVA_HOME的注释,并修改为本地的JAVA_HOME

   vim conf/hbase-env.sh,出掉JAVA_HOME的注释,并修改为本地的JAVA_HOME

分别启动zookeeper服务。在每台机器上切换到zookeeper-3.4.5目录,执行

CA88新登入,   export JAVA_HOME=/usr/lib/jdk7

   export JAVA_HOME=/usr/lib/jdk7

$bin/zkServer.sh start

 
 修改HBASE_MANAGES_ZK为false,默认为true,表示HBase托管zookeeper实例,如果是HBase独占的zookeeper,可以设定为true。

 
 修改HBASE_MANAGES_ZK为false,默认为true,表示HBase托管zookeeper实例,如果是HBase独占的zookeeper,可以设定为true。

就可以启动zookeeper服务。

    export HBASE_MANAGES_ZK=false

    export HBASE_MANAGES_ZK=false

二、安装hbase-0.96.0

 

 

为了和hadoop-2.2.0匹配,选择的hbase版本为0.96.0

3. 修改hbase-site.xml

3. 修改hbase-site.xml

  1. $ wget
  2. $ tar -xf hbase-0.96.0-hadoop2-bin.tar.gz
  3. $ cd hbase-0.96.0

    $ wget
    $ tar -xf hbase-0.96.0-hadoop2-bin.tar.gz
    $ cd hbase-0.96.0

   
以下只配置几个必需要修改的项,其它的项使用默认值。hbase-site.xml的默认配置可以从帮助文档(Default
Configuration)获得。

   
以下只配置几个必需要修改的项,其它的项使用默认值。hbase-site.xml的默认配置可以从帮助文档(Default
Configuration)获得。

修改 conf 下的配置:

 <configuration>

 <configuration>

  1. vim conf/hbase-env.sh

    vim conf/hbase-env.sh

  <property>

  <property>

修改 JAVA_HOME:

    <name>hbase.rootdir</name>

    <name>hbase.rootdir</name>

  1. # The java implementation to use. Java 1.6 required.
  2. export JAVA_HOME=/usr/lib/jvm/java/jdk1.6.0_37

    # The java implementation to use. Java 1.6 required.
    export JAVA_HOME=/usr/lib/jvm/java/jdk1.6.0_37

    <value>hdfs://HDP2:9000/hbase</value>

    <value>hdfs://HDP2:9000/hbase</value>

修改HBase 使用zookeeper的方式:

  </property>

  </property>

  1. # Tell HBase whether it should manage it’s own instance of
    Zookeeper or not.
  2. export HBASE_MANAGES_ZK=false

    # Tell HBase whether it should manage it’s own instance of Zookeeper or not.
    export HBASE_MANAGES_ZK=false

  <property>

  <property>

这个参数默认是 true。意思是是否让hbase 单独管理 zookeeper

     <name>hbase.cluster.distributed</name>

     <name>hbase.cluster.distributed</name>

如果安装的机器上已经安装了zookeeper,并且这个zookeeper是多个平台共用的,那么这个参数应该设置为
false

     <value>true</value>

     <value>true</value>

如果集群上没有安装zookeeper,并且只有hbase
自己使用zookeeper,那么这个参数应该设置为true。

  </property>

  </property>

修改其conf目录下的hbase-site.xml文件。内容如下:

  <property>

  <property>

  1. <configuration>
  2. <property>
  3. <name>hbase.rootdir</name>
  4. <value>hdfs://master:8089/hbase</value>
  5. </property>
    1. <property>
  6. <name>hbase.master</name>
  7. <value>hdfs://master:60000</value>
  8. </property>
    1. <property>
  9. <name>hbase.cluster.distributed</name>
  10. <value>true</value>
  11. </property>
    1. <property>
  12. <name>hbase.zookeeper.property.clientPort</name>
  13. <value>2181</value>
  14. </property>
    1. <property>
  15. <name>hbase.zookeeper.quorum</name>
  16. <value>master,node,slave</value>
  17. </property>
    1. <property>
  18. <name>hbase.zookeeper.property.dataDir</name>
  19. <value>/home/aaron/zookeeper</value>
  20. </property>
    1. <property>
  21. <name>hbase.client.scanner.caching</name>
  22. <value>200</value>
  23. </property>
    1. <property>
  24. <name>hbase.balancer.period</name>
  25. <value>300000</value>
  26. </property>
    1. <property>
  27. <name>hbase.client.write.buffer</name>
  28. <value>10485760</value>
  29. </property>
    1. <property>
  30. <name>hbase.hregion.majorcompaction</name>
  31. <value>7200000</value>
  32. </property>
    1. <property>
  33. <name>hbase.hregion.max.filesize</name>
  34. <value>67108864</value>
  35. <description>
  36. Maximum HStoreFile size. If any one of a column families’
    HStoreFiles has
  37. grown to exceed this value, the hosting HRegion is split in
    two.</description>
  38. </property>
    1. <property>
  39. <name>hbase.hregion.memstore.flush.size</name>
  40. <value>1048576</value>
  41. <description>
  42. Memstore will be flushed to disk if size of the memstore
  43. exceeds this number of bytes. Value is checked by a thread that runs
  44. every hbase.server.thread.wakefrequency.</description>
  45. </property>
    1. <property>
  46. <name>hbase.server.thread.wakefrequency</name>
  47. <value>30000</value>
  48. <description>Time
    to sleep in between searches for work (in milliseconds).
  49. Used as sleep interval by service threads such as log roller.</description>
  50. </property>
    1. </configuration>

    hbase.rootdir
    hdfs://master:8089/hbase
    hbase.master
    hdfs://master:60000
    hbase.cluster.distributed
    true
    hbase.zookeeper.property.clientPort
    2181
    hbase.zookeeper.quorum
    master,node,slave
    hbase.zookeeper.property.dataDir
    /home/aaron/zookeeper
    hbase.client.scanner.caching
    200
    hbase.balancer.period
    300000
    hbase.client.write.buffer
    10485760
    hbase.hregion.majorcompaction
    7200000
    hbase.hregion.max.filesize
    67108864

    Maximum HStoreFile size. If any one of a column families’ HStoreFiles has
    grown to exceed this value, the hosting HRegion is split in two.
    hbase.hregion.memstore.flush.size
    1048576

    Memstore will be flushed to disk if size of the memstore
    exceeds this number of bytes. Value is checked by a thread that runs
    every hbase.server.thread.wakefrequency.
    hbase.server.thread.wakefrequency
    30000
    Time to sleep in between searches for work (in milliseconds).
    Used as sleep interval by service threads such as log roller.

     <name>hbase.master.port</name>

     <name>hbase.master.port</name>

其中hbase.rootdir要保持与hadoop的core-site.xml文件中的fs.default.name中的值一致。

     <value>16000</value>

     <value>16000</value>

hbase的master 和
regionserver的时钟要保持一致,否则启动regionserver的时候会报如下错误:

  </property>

  </property>

org.apache.hadoop.hbase.ClockOutOfSyncException:
org.apache.hadoop.hbase.ClockOutOfSyncException: Server
hadoop02,60020,1374576225508 has been rejected; Reported time is too far
out of sync with master. Time difference of 62062623ms > max allowed
of 30000ms