Hadoop2.6集群情状搭建(HDFS
HA+YAWranglerN卡塔 尔(英语:State of Qatar)原本4G内部存储器也能随意三次。

布置逻辑架构:

有备无患职业:

图片 1

1、台式机4G内部存款和储蓄器 ,操作系统WIN7 (土憋的安插卡塔尔

HDFS HA陈设物理架构

2、工具VMware Workstation

图片 2

3、虚拟机:CentOS6.4共四台

注意:

虚构机设置:

每台机械:内部存储器512M,硬盘40G,网络适配器:NAT格局

 图片 3

挑选高级,新生成虚机Mac地址(克隆设想机,Mac地址不会变动,每趟最终手动重新生成卡塔尔

 图片 4

YARN多机部署,6集群环境搭建。编写制定设想机网络:

 图片 5

点击NAT设置,查看虚机网关IP,并记住它,该IP在虚机的互联网设置中拾壹分关键。

NAT设置暗中认可IP会自动生成,不过大家的集群中IP须要手动设置。

 图片 6

本机Win7 :VMnet8 互联网设置

 图片 7

JournalNode使用财富少之又少,固然在实际的坐褥条件中,也是把JournalNode和DataNode安顿在同生龙活虎台机器上;

实验意况:

Ip

hostname

role

192.168.249.130

SY-0130

ActiveNameNode

192.168.249.131 

SY-0131

StandByNameNode

192.168.249.132

SY-0132

DataNode1

192.168.249.133

SY-0133

DataNode2

 

临盆条件中,提出主备NameNode各单唯风度翩翩台机器。

Linux互联网设置:

1、新建客户如:hadoop。不建议利用root客户搭建集群(root权限过大卡塔尔

2、使得hadoop客户拿到sudo权限。

 

[root@SY-0130 ~]# vi /etc/sudoers

## Allow root to run any commands anywhere

root    ALL=(ALL)       ALL

hadoop  ALL=(ALL)       ALL

 

3、查看当前虚机当前互连网接受网卡设备

[root@SY-0130 hadoop]# ifconfig
eth2      Link encap:Ethernet  HWaddr 00:50:56:35:8E:E8  
          inet addr:192.168.249.130  Bcast:192.168.249.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe35:8ee8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:877059 errors:0 dropped:0 overruns:0 frame:0
          TX packets:597769 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:865720294 (825.6 MiB)  TX bytes:324530557 (309.4 MiB)
          Interrupt:19 Base address:0x2024 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1354 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:196675 (192.0 KiB)  TX bytes:196675 (192.0 KiB)

 

[root@SY-0130 ~]# cat /proc/net/dev,当前网卡,我的为eth2

[root@SY-0130 ~]# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:  196675    1354    0    0    0     0          0         0   196675    1354    0    0    0     0       0          0
  eth2:865576893  875205    0    0    0     0          0         0 324425517  596433    0    0    0     0       0          0

 

4、查看当前网卡对应的Mac地址

 

[root@SY-0130 ~]# cat /etc/udev/rules.d/70-persistent-net.rules 
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x1022:0x2000 (vmxnet)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:b5:fd:bb", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x1022:0x2000 (vmxnet)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:35:8e:e8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

5、Network Configuration

  [root@SY-0130 ~]# setup 

 图片 8

图片 9

 

慎选eth2,进行安装,改善为如今网卡设备eth2,並且打开IP、网管、DNS设置。

 图片 10

DNS Server 与Win7的互连网中的DNS
Server朝气蓬勃致,那样设想机也能够连接Internet网了,方便下载安装软件。

别的还应该有将防火墙关闭。

6、修改hostname

[root@SY-0130 ~]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=SY-0130

7、修改hosts

[hadoop@SY-0130 ~]$ sudo vi /etc/hosts

#丰裕如下内容

192.168.249.130 SY-0130
192.168.249.131 SY-0131
192.168.249.132 SY-0132
192.168.249.133 SY-0133

8、重启虚机.  reboot 

CentOS安装和布局Hadoop2.2.0 
http://www.linuxidc.com/Linux/2014-01/94685.htm

Ubuntu
13.04上搭建Hadoop环境
http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1本子集群配置
http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu上搭建Hadoop遇到(单机情势+伪布满方式卡塔尔国
http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu下Hadoop蒙受的配置
http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建Hadoop境况图像和文字化教育程详整
http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建Hadoop情状(在Winodws遭受下用编造机虚构两个Ubuntu系统开展搭建卡塔尔
http://www.linuxidc.com/Linux/2011-12/48894.htm

越来越多详细情况见请继续阅读下生龙活虎页的精彩内容
http://www.linuxidc.com/Linux/2015-01/111461p2.htm 

图片 11

YA福特ExplorerN陈设架构:

图片 12

个体实验蒙受布署图:

图片 13

Ubuntu12 32bit

apache Hadoop 2.2.0

jdk1.7

===============================================

Ubuntu 13.04上搭建Hadoop环境
http://www.linuxidc.com/Linux/2013-06/86106.htm

Ubuntu 12.10 +Hadoop 1.2.1本子集群配置
http://www.linuxidc.com/Linux/2013-09/90600.htm

Ubuntu上搭建Hadoop情况(单机情势+伪布满方式卡塔 尔(阿拉伯语:قطر‎
http://www.linuxidc.com/Linux/2013-01/77681.htm

Ubuntu下Hadoop际遇的布局
http://www.linuxidc.com/Linux/2012-11/74539.htm

单机版搭建Hadoop景况图像和文字化教育程安详严整
http://www.linuxidc.com/Linux/2012-02/53927.htm

搭建Hadoop境遇(在Winodws蒙受下用编造机设想五个Ubuntu系统开展搭建卡塔尔国
http://www.linuxidc.com/Linux/2011-12/48894.htm

===============================================

准备干活

1.在4台机械都布置hosts;

2.安插NameNode节点可避防密码登陆到任何具备节点,只须求单向免密登入就可以,无需双向;

免密码登陆仅仅在起步、结束集群时接收。

3.安装jdk

4.创立特地的账号,不要用root账号铺排、管理hadoop




部署hadoop

先是步:把hadoop安装包解压到每贰个节点(可以解压到多少个节点,然后成功后续第2步的安插后,再scp拷贝到别的节点卡塔尔国的稳固目录下(各节点目录统生龙活虎卡塔尔国,比如/home/yarn/Hadoop/hadoop-2.2.0

第二步:修正配置文件(只需在叁个节点上布署,配置好后再用scp分发到别的节点卡塔尔国

布署文件路线:etc/hadoop/

hadoop-env.sh

修改JDK路线,在文书中搜寻以下行,将JAVA_HOME设置为JDK安装路线就能够:

# The java implementation to use.

export
JAVA_HOME=/usr/lib/jvm/java-6-sun


core-site.xml

钦点Active NameNode的host名/ip和端口号,端口号能够依赖自个儿的须求校订:

<configuration>

<property>

 
<name>fs.defaultFS</name>

 
<value>hdfs://SY-0217:8020</value>

</property>

</configuration>

小心:以上配置的SY-0217是向来host,只适用于手动切换主备NameNode的场景,如若急需经过ZooKeeper来自动切换,则要求配置逻辑名称,后边会详述。


mapred-site.xml

 

<configuration>

<!– MR YARN Application properties –>

<property>

 
<name>mapreduce.framework.name</name>

  <value>yarn</value>

  <description>The runtime framework
for executing MapReduce jobs.

  Can be one of local, classic or
yarn.

  </description>

</property>

 

<!–

jobhistory properties

jobhistory
server,能够透过它查看已经运营完的应用程序的音信。

–>

<property>

 
<name>mapreduce.jobhistory.address</name>

 
<value>SY-0355:10020</value>

  <description>MapReduce JobHistory
Server IPC host:port</description>

</property>

 

<property>

 
<name>mapreduce.jobhistory.webapp.address</name>

 
<value>SY-0355:19888</value>

  <description>MapReduce JobHistory
Server Web UI host:port</description>

</property>

</configuration>


hdfs-site.xml

足够主要的布局文件!

<configuration>

 

<property>

 
<name>dfs.nameservices</name>

 
<value>hadoop-test</value>

  <description>

   
钦命命名空间名称,可随意起名

    Comma-separated list of
nameservices.

  </description>

</property>

 

<property>

 
<name>dfs.ha.namenodes.hadoop-test</name>

 
<value>nn1,nn2</value>

  <description>

   
在命名空间下钦赐NameNode逻辑名

    The prefix for a given nameservice,
contains a comma-separated

    list of namenodes for a given
nameservice (eg EXAMPLENAMESERVICE).

  </description>

</property>

 

<property>

 
<name>dfs.namenode.rpc-address.hadoop-test.nn1</name>

 
<value>SY-0217:8020</value>

  <description>

   
为“命名空间名.NameNode逻辑名”配置rpc地址

    RPC address for nomenode1 of
hadoop-test

  </description>

</property>

 

<property>

 
<name>dfs.namenode.rpc-address.hadoop-test.nn2</name>

 
<value>SY-0355:8020</value>

  <description>

    为“命名空间名.NameNode逻辑名”配置rpc地址

    RPC address for nomenode2 of
hadoop-test

  </description>

</property>

 

<property>

 
<name>dfs.namenode.http-address.hadoop-test.nn1</name>

 
<value>SY-0217:50070</value>

  <description>

    为“命名空间名.NameNode逻辑名”配置http地址

    The address and the base port where
the dfs namenode1 web ui will listen on.

  </description>

</property>

 

<property>

 
<name>dfs.namenode.http-address.hadoop-test.nn2</name>

 
<value>SY-0355:50070</value>

  <description>

    为“命名空间名.NameNode逻辑名”配置http地址

    The address and the base port where
the dfs namenode2 web ui will listen on.

  </description>

</property>

 

<property>

 
<name>dfs.namenode.name.dir</name>

 
<value>file:///home/dongxicheng/hadoop/hdfs/name</value>

  <description>

   
配置NameNode元数据贮存的路线;

   
假使机器上有多块硬盘的话,推荐配置多少个门路,用逗号分隔。

Determines where on the local filesystem
the DFS name node

      should store the name
table(fsimage).  If this is a comma-delimited list

      of directories then the name table
is replicated in all of the

      directories, for redundancy.
</description>

</property>

 

<property>

 
<name>dfs.datanode.data.dir</name>

 
<value>file:///home/dongxicheng/hadoop/hdfs/data</value>

  <description>

     
  配置DataNode数据贮存的门路;

   
如果机器上有多块硬盘的话,推荐配置多少个渠道,用逗号分隔。

Determines where on the local filesystem
an DFS data node

  should store its blocks.  If this is a
comma-delimited

  list of directories, then data will be
stored in all named

  directories, typically on different
devices.

相关文章