Hadoop环境搭建1

未知0072019-03-14 15:35:27

说明:

    目的:

        最终实现Hadoop集群    

    镜像:

            ubuntu-14.04.5-server-amd64.iso     ---->服务器版(没有图形界面)

            CentOS7.2            ----->格式qcow2

    hadoop版本:hadoop-2.7.1.tar.gz

    jdk版本:jdk-7u71-linux-x64.tar.gz

    (Hadoop3.x需要使用jdk8.+)

    环境部署说明:

            使用virtualbox搭建Ubuntu

            openstack上启动3台云主机,centos系统

    网络说明:

            交换机(内网1,,未连接外网)

            172.16.1.1           

                172.16.1.100    笔记本(网线)

                172.16.1.110    虚拟机Ubuntu

            服务器controller节点

            192.168.100.1    

            服务器compute节点

            192.168.200.1

                192.168.200.101    opensstack搭建的云主机1(浮动IP)

            路由器(内网2,,已连通外网)

            192.168.1.1

                192.168.1.108    笔记本(无线)

                192.168.1.110    虚拟机Ubuntu

    步骤说明:

        一、Ubuntu搭建后配置

        二、CentOS配置

        三、在Ubuntu上面搭建伪分布式Hadoop集群

        四、实现完全分布式

        五、参考资料     

    


一、Ubuntu搭建后配置(搭建过程略过、双网卡配置)

注意事项

        两个网卡使用的均为桥接(网卡1:桥接的笔记本无线网卡,网卡2:有线网卡)、如果使用NAT模式:虚拟机只能与笔记本进行通信,无法与局域网内其他机器通信

        添加双网卡时,虚拟机需要关机

        

        最好安装英文版,中文版可能会出现乱码

1、安装Ubuntu之后进入系统,默认并不是root用户需要进行设置root用户密码(root默认密码为空)

Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-31-generic x86_64)

  System load:  0.06              Processes:           111

  Usage of /:   5.1% of 18.32GB   Users logged in:     1

  Memory usage: 7%                IP address for eth0: 192.168.1.110

  Swap usage:   0%                IP address for eth1: 172.16.1.110

ubuntu@ubuntu:~$ sudo passwd            #设置root用户密码

[sudo] password for ubuntu:

Enter new UNIX password:

Retype new UNIX password:

passwd: password updated successfully

ubuntu@ubuntu:~$ su        #进入root用户

Password:

root@ubuntu:/home/ubuntu#

2、配置网络(/etc/network/interfaces)

    配置IP

root@ubuntu:~# cd /etc/network

root@ubuntu:/etc/network# cp interfaces interfaces.bak

root@ubuntu:/etc/network# vi interfaces

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

auto eth0

iface eth0 inet static

address 192.168.1.110

gateway 192.168.1.1

# The primary network interface

auto eth1

iface eth1 inet static

address 172.16.1.110

gateway 172.16.1.1

3、配置DNS

root@ubuntu:/etc/network# vi /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 114.114.114.114

nameserver 8.8.8.8

4、重启网卡

root@ubuntu:~# ifdown -a

root@ubuntu:~# ifup -a

root@ubuntu:~# /etc/init.d/networking restart

root@ubuntu:~# service network restart

5、本地源配置

需要将ubuntu-14.04.5-server-amd64.iso挂载  或者  上传到Ubuntu

(1)创建挂载目录

root@ubuntu:~# mkdir /mnt/cdrom

(2)挂载到mnt/cdrom目录

root@ubuntu:~# mount /dev/cdrom /mnt/cdrom/            

mount: block device /dev/sr0 is write-protected, mounting read-only

mount -o loop ubuntu.iso /opt/cdrom

(3)实现永久挂载(开机自动挂载)

root@ubuntu:~# cat /etc/issue

root@ubuntu:~# lsb_release -a

root@ubuntu:~# mount

/dev/sr0 on /mnt/cdrom type iso9660 (ro)

root@ubuntu:~# vi /etc/fstab

/dev/sr0 /mnt/cdrom iso9660 loop 0 0

~

(4)配置源文件

root@ubuntu:~# cd /etc/apt/

root@ubuntu:/etc/apt# cp sources.list sources.list.bak

删除全部重新配置(trusty 包的版本类型,cat /opt/cdrom/dists可以查看到,,或者查看Linux版本也可以查看到)(「main」,官方维护的自由软件。「restricted」,设备专有驱动。

root@ubuntu:/etc/apt# vi sources.list

deb file:///mnt/cdrom trusty main

deb file:///mnt/cdrom trusty restricted

(5)更新源索引

root@ubuntu:~# apt-get update

(6)安装SSH(远程连接使用)

root@ubuntu:~# apt-get install openssh-server

安装之后默认为开启状态

root@ubuntu:~# netstat -ant |grep 22

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     

tcp        0      0 192.168.1.110:22        192.168.1.108:28429     ESTABLISHED

tcp6       0      0 :::22   

如果安装之后还是远程连接不上,则需要查看防火墙设置

root@ubuntu:~# ufw disable

6、添加路由(实现双网卡正常使用)(这一步需要根据实际环境进行配置)

root@ubuntu:~# ifconfig

eth0      Link encap:Ethernet  HWaddr 08:00:27:61:6e:44  

          inet addr:192.168.1.110  Bcast:192.168.1.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 08:00:27:28:c3:5e  

          inet addr:172.16.1.110  Bcast:172.16.255.255  Mask:255.255.0.0

lo        Link encap:Local Loopback  

          inet addr:127.0.0.1  Mask:255.0.0.

route add -net 192.168.100.0 netmask 255.255.255.0 gw 172.16.1.1 dev eth1

route add -net 192.168.200.0 netmask 255.255.255.0 gw 172.16.1.1 dev eth1

二、CeontOS配置

[root@controller ~]# source /etc/keystone/admin-openrc.sh

在controller节点上传镜像

glance image-create --name "CeontOS7.2" --disk-format qcow2 --container-format bare  --progress < /opt/iaas/images/CentOS_7.2_x86_64_XD.qcow2

[root@controller ~]# glance image-list

+--------------------------------------+------------+

| ID                                   | Name       |

+--------------------------------------+------------+

| 0e8e07a6-96ad-45c0-9b24-b55e7b6651eb | CeontOS7.2 |

+--------------------------------------+------------+

在http://192.168.100.10/dashboard创建云主机

配置网络yum源

[root@ceontos ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=enp7s0

ONBOOT=yes

IPADDR=192.168.200.10

PREFIX=24     #或NETMASK=255.255.255.0

GATEWAY=192.168.200.1

[root@ceontos ~]# systemctl restart network

[root@ceontos ~]# rm -rf /etc/yum.repos.d/*

[root@ceontos ~]# scp 192.168.100.20:/etc/yum.repos.d/* /etc/yum.repos.d/ftp.repo

[root@ceontos ~]# yum clean all

[root@ceontos ~]# yum makecache

三、在Ubuntu上面搭建伪分布式Hadoop集群

1、创建Hadoop用户组

root@ubuntu:~# addgroup hadoop

2、创建Hadoop用户

root@ubuntu:~# adduser -ingroup hadoop hadoop

3、为Hadoop用户添加权限

root@ubuntu:~# vi /etc/sudoers

# User privilege specification

root    ALL=(ALL:ALL) ALL

hadoop  ALL=(ALL:ALL) ALL

4、安装ssh

ubuntu@ubuntu:~$ su hadoop

hadoop@ubuntu:~$ sudo apt-get install openssh-server

hadoop@ubuntu:~$ ps -e |grep ssh

3711 ?        00:00:00 sshd

3744 ?        00:00:00 sshd

3834 ?        00:00:00 sshd

3877 ?        00:00:00 sshd

3967 ?        00:00:00 sshd

hadoop@ubuntu:~$ netstat -ant|grep 22

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     

tcp        0      0 192.168.1.110:22        192.168.1.108:43653     ESTABLISHED

tcp        0      0 192.168.1.110:22        192.168.1.108:46625     ESTABLISHED

tcp6       0      0 :::22                   :::*                    LISTEN     

hadoop@ubuntu:~$

5、配置ssh免密钥

hadoop@ubuntu:~$ ssh-keygen -t rsa

hadoop@ubuntu:~/.ssh$ cat id_rsa.pub >>authorized_keys

hadoop@ubuntu:~$ ssh localhost

6、安装jdk

hadoop@ubuntu:~$ sudo mkdir /usr/jdk64

hadoop@ubuntu:~$ sudo tar -zxvf jdk-7u71-linux-x64.tar.gz -C /usr/jdk64/

hadoop@ubuntu:/usr/jdk64/jdk1.7.0_71$ pwd

/usr/jdk64/jdk1.7.0_71

hadoop@ubuntu:~$ vi .profile

export JAVA_HOME=/usr/jdk64/jdk1.7.0_71

export PATH=$JAVA_HOME/bin:$PATH

hadoop@ubuntu:~$ source .profile

hadoop@ubuntu:~$ java -version

java version "1.7.0_71"

Java(TM) SE Runtime Environment (build 1.7.0_71-b14)

Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

7、安装Hadoop并配置Hadoop环境变量

hadoop@ubuntu:~$ tar -zxvf hadoop-2.7.1.tar.gz

hadoop@ubuntu:~$ vi .profile

export HADOOP_HOME=/home/hadoop/hadoop-2.7.1

export PATH=$PATH:$HADOOP_HOME/bin

export HADOOP_HOME_WARN_SUPPRESS=1

hadoop@ubuntu:~$ source .profile

hadoop@ubuntu:~$ hadoop version

Hadoop 2.7.1

Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a

Compiled by jenkins on 2015-06-29T06:04Z

Compiled with protoc 2.5.0

From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a

This command was run using /home/hadoop/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar

hadoop@ubuntu:~$

配置Hadoop的Java环境

hadoop@ubuntu:~$ vi hadoop-2.7.1/etc/hadoop/hadoop-env.sh

#export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/usr/jdk64/jdk1.7.0_71

配置Hadoop API和缓存路径

hadoop@ubuntu:~$ vi hadoop-2.7.1/etc/hadoop/core-site.xml

<configuration>

        <property>

                <name>fs.defaultFS</name>

                <value>hdfs://localhost:8020</value>

        </property>

        <property>

                <name>hadoop.tmp.dir</name>

                <value>/home/hadoop/hadoop_tmp</value>

        </property>

</configuration>

格式化namenode仅第一次执行一次

hadoop@ubuntu:~$ hdfs namenode -forma

启动HDFS

hadoop@ubuntu:~$ hadoop-2.7.1/sbin/./start-dfs.sh

验证

hadoop@ubuntu:~$ jps

5316 Jps

4857 NameNode

5206 SecondaryNameNode

4990 DataNode

或浏览器访问http://192.168.1.110:50070进行验证

配置hdfs副本系数

hadoop@ubuntu:~$ vi hadoop-2.7.1/etc/hadoop/hdfs-site.xml

        <property>

                <name>dfs.replication</name>

                <value>1</value>

        </property>

配置YARN,告诉MapReduce跑在YARN上

hadoop@ubuntu:~$ cp hadoop-2.7.1/etc/hadoop/mapred-site.xml.template hadoop-2.7.1/etc/hadoop/mapred-site.xml

hadoop@ubuntu:~$ vi hadoop-2.7.1/etc/hadoop/mapred-site.xml

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

告诉框架yarn的nodemanager用的是哪一个services

hadoop@ubuntu:~$ vi hadoop-2.7.1/etc/hadoop/yarn-site.xml

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

启动YARN

hadoop@ubuntu:~$ hadoop-2.7.1/sbin/./start-yarn.sh

验证

hadoop@ubuntu:~$ jps

4857 NameNode

5797 Jps

5510 NodeManager

5374 ResourceManager

5206 SecondaryNameNode

4990 DataNode

或访问http://192.168.1.110:8088/cluster

测试

hadoop@ubuntu:~$ hadoop fs -put jdk-7u71-linux-x64.tar.gz /

hadoop@ubuntu:~$ hadoop fs -ls /

Found 3 items

-rw-r--r--   1 hadoop supergroup  142216602 2018-01-28 23:10 /jdk-7u71-linux-x64.tar.gz

drwx------   - hadoop supergroup          0 2018-01-28 23:04 /tmp

drwxr-xr-x   - hadoop supergroup          0 2018-01-28 23:04 /user

hadoop@ubuntu:~$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 1 1            计算圆周率

统计单词,源文件在本地(源文件不能为隐藏文件)

hadoop@ubuntu:~$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount file:///home/hadoop/demo /output

hadoop@ubuntu:~$ hadoop fs -cat /output/part-r-00000 |head -10

(bytes)=242360320       1

(ms)=110        1

=1      1

BAD_ID=0        1

Bytes   2

CONNECTION=0    1

Combine 2

Counters        3

Errors  1

FILE:   5

统计单词实例,源文件在HDFS文件系统

hadoop@ubuntu:~$ hadoop fs -mkdir /input

hadoop@ubuntu:~$ hadoop fs -put demo /input

hadoop@ubuntu:~$ hadoop fs -ls /input/

hadoop@ubuntu:~$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /input/demo /output1

hadoop@ubuntu:~$ hadoop fs -ls /output1

Found 2 items

-rw-r--r--   1 hadoop supergroup          0 2018-01-28 23:22 /output1/_SUCCESS

-rw-r--r--   1 hadoop supergroup        664 2018-01-28 23:22 /output1/part-r-00000

hadoop@ubuntu:~$ hadoop jar hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount hdfs://localhost:8020/input/demo /output2

hadoop@ubuntu:~$ hadoop fs -ls /output2

Found 2 items

-rw-r--r--   1 hadoop supergroup          0 2018-01-28 23:23 /output2/_SUCCESS

-rw-r--r--   1 hadoop supergroup        664 2018-01-28 23:23 /output2/part-r-00000

hadoop@ubuntu:~$ hadoop fs -cat /output2/part-r-00000

hadoop@ubuntu:~$ cat demo |sed 's/[,.:;/!?]/ /g'|awk '{for(i=0;i<NF;i++)array[$i]++;}END{for(i in array) print i,array[i]}'            #shell命令单词统计

hadoop@ubuntu:~$ hadoop fs -get /output2/part-r-00000

hadoop@ubuntu:~$ vimdiff part-r-00000 shell_w        #对比

四、实现完全分布式

需要注意:各节点需要在相同用户下搭建

                  

先将基础环境配置好

然后在Ubuntu下进行配置

配置之后复制到其他节点。

---

配置网络

配置源

安装ssh

修改主机名

配置hosts

关闭防火墙

新建用户

进入新用户ssh免密钥

进入新用户下载Hadoop、jdk包

解压

配置环境变量

配置Hadoop配置文件

主节点配置完成后将环境变量和hadoop、jdk分发到所有从节点

在主节点格式化namenode

在主节点启动

1、修改主机名

ubuntu@ubuntu:~$ hostnamectl set-hostname cMaster

ubuntu@cmaster:~$ hostnamectl

   Static hostname: cmaster

   Pretty hostname: cMaster

[root@cs1 ~]# hostnamectl set-hostname cSlave1

[root@cmaster ~]# hostnamectl set-hostname cSlave2

[root@cslave0 ~]# hostnamectl set-hostname cSlave3

[root@cslave1 ~]# hostnamectl set-hostname cSlave4

2、配置hosts

root@cmaster:~# vi /etc/hosts

127.0.0.1       localhost

127.0.1.1       cMaster

192.168.1.110 cMaster

172.16.1.110 cMaster

192.168.200.101 cSlave1

10.0.0.101 cSlave1

192.168.200.102 cSlave2

10.0.0.102 cSlave2

192.168.200.103 cSlave3

10.0.0.103 cSlave3

192.168.200.104 cSlave4

10.0.0.104 cSlave4

root@cmaster:~# scp /etc/hosts 192.168.200.101:/etc/hosts

root@cmaster:~# scp /etc/hosts 192.168.200.102:/etc/hosts

root@cmaster:~# scp /etc/hosts 192.168.200.103:/etc/hosts

root@cmaster:~# scp /etc/hosts 192.168.200.104:/etc/hosts

测试

[root@cslave1 ~]# ping cMaster

PING cMaster (127.0.1.1) 56(84) bytes of data.

64 bytes from cMaster (127.0.1.1): icmp_seq=1 ttl=64 time=0.036 ms

hadoop@cmaster:~$ ping cSlave1

PING cSlave1 (192.168.200.101) 56(84) bytes of data.

64 bytes from cSlave1 (192.168.200.101): icmp_seq=1 ttl=62 time=1.70 ms

......

......

3、Ubuntu新建用户组、用户

root@cmaster:~# addgroup hadoop

root@cmaster:~# adduser -ingroup hadoop hadoop

root@cmaster:~# vi /etc/sudoers                #需要强制保存并退出:wq!

# User privilege specification

root    ALL=(ALL:ALL) ALL

hadoop  ALL=(ALL:ALL) ALL

4、ssh免密钥配置

hadoop@cmaster:~$ ssh-keygen -t rsa

[root@cslave1 ~]# ssh-keygen -t rsa

[root@cslave2 ~]# ssh-keygen -t rsa

[root@cslave3 ~]# ssh-keygen -t rsa

[root@cslave4 ~]# ssh-keygen -t rsa

hadoop@cmaster:~/.ssh$ cat id_rsa.pub >>authorized_keys

将其他从节点的公钥复制到主节点authorized_keys中

再将主节点的authotized_keys分发给从节点

hadoop@cmaster:~/.ssh$ vi authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2Zz133bYMsMhyJ/SYgPcwcp/LswrsIBZS9LZ6C4BsFYKvgbtblOAG9+Hzc1vWZKl5bF8Ra637ijr/gGiTaOw4vkvto+7sjd0gCAS+jP/P0J/Gr3CQWklsL0HdQ33hYDiOExAA9QEtai9Pi8mc4iaiU1vf3Wb/B89L5ZuvRKwWJa3X6gq74Ob0Aet5wq+N8APadYMwzjqc+ZvxTLmsVbOx4MDe1KsAiNm9eL9/6rMbGLXl7pBrxf82WLx3TlQj1t8U5U4EBKOMzqwfdNtmR2ieinVK7vsAfIcDL1AKbQ0chhpuj5LPbn2+Ao79x2KBYUSsnbcEUdcM09fH0nFJAGCR hadoop@cmaster

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMcQdNJo4zYqx8Aa3Wt/szzb5Fqk+QY/G7LPyRedt9CFMIxUUoy2lqUD2rlymXoMpy7Vx2bT7Y/Pg12Zk+z5PLaL5CapJENoRX/BO45HMvV3zOKqI3XYrOjN0vKuChxlv4mxdeCtSBaVEhB8qGrWIrt41oFjSDetrFMYgf4u3m7yI5zUa/8fGiAYHlZ/o+nsuCHozdvtn8XcI7Rn9y+u0lLY5CI1ByfVyRbMTGTUOLXfnoZUVm0ljnQ4z9nxOqHP/DKQnEIg7njEAeTuSy1IJGAoZVPNUwIQQLJhGzSCj74UIQT3LG99/PLotUJPBVOVvkIwcctAGCbUUnZSPAyRPN root@cslave1

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF7WZ+ifcfiqzgowRVYaUQLsaFetJrNqC9lX9J62tVcLZ3Cic3E/9Y1hbdlATL/ZDjE1AdwdZkQpK0JUNszFFVjX1gkN/v9br1Q+fWtCyuiO04QCs7X3OUu2hUU40mRoVc4YnZGXQv3G8Cl6v98abUkFjXJEpwGQ3/LeBPITkYkb3YtMeOfEDyLM5gd6vrLu0PMRSETBqgYH+yBXj8nJNYic1JDkLGFmNYLvkYe2+axjMgi95his7Y82RhyVvtpSm8jnEisV7esgxrBzET4ngt7hqtT4AJWtikUYP+EgICWuGR1pPIQc+yKk+CejumX5tiOukzY3M98bIZYIEaebtV root@cslave2

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFSNFduLgEFmdZ1/Dy5r/TDeZWGWflE3/136IlSijd0FRudR+JXXUJBirr/4maBPl1TtQ3WlrIFLsQB1AuWEIrSFdZrfVEq4Alq7nPPBqzz4EA0SYmi8PWj8MWGVJL+B8qzwiqNLc82OrbkPMenhANwc+ond+iKIyouGiAvAp4zmXpQwZ0BHUkAMhai9wXWRm2c7ZzlA8wTVEqIfE7GOkp2g3YWEK8MxnebKy183u9IZEWNjFUkAGu4hkaAB+3idJ5zjUm3yatFTSb9SrQ8aarCosY4OPTK1tCrBWMB6UQVidg9f+oYfc9CkFm6y7qeS8satkz98HtB1J2enDCE/fP root@cslave3

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJGrXvHXddy+YkfhOdTIEtm2qvPylqeOeD/fqpowPC0WozgsijPRdUNz8owZKdyJyDqt00Y/JV019xleLowPreKKkhLEuvKyekCHc9CTRv8FV+MalfJRt0OSOClHqdIkgX8nUjk4QyUJLQRG04b9d2FYGSJWh2IrQRFOOGDRczHrqQzz9uX1d+Zz31v25a4uvsfM3LCJJpDAp9EvBkeONDFaMhP6h90ROOCFcHDPm68WZllMi4KJYcVqdliJL7Bq8+XUl7bqEt11TyNqCmVrlPRlp7ixbaujqHwflbt41XX9RyG4SimDe0ZeHkyu8y7L8ut7AP7lXDIfM19yWkE1h3 root@cslave

hadoop@cmaster:~/.ssh$ sudo scp authorized_keys cSlave1:/root/.ssh

hadoop@cmaster:~/.ssh$ sudo scp authorized_keys cSlave2:/root/.ssh

hadoop@cmaster:~/.ssh$ sudo scp authorized_keys cSlave3:/root/.ssh

hadoop@cmaster:~/.ssh$ sudo scp authorized_keys cSlave4:/root/.ssh

ssh-copy-id -i ~/.ssh/id_rsa.pub 目标机器IP或主机名

测试

[root@cslave1 ~]# ssh hadoop@172.16.1.110

5、下载jdk、Hadoop

hadoop@cmaster:~$ wget ftp://192.168.100.10/jdk-7u71-linux-x64.tar.gz

hadoop@cmaster:~$ wget ftp://192.168.100.10/hadoop-2.7.1.tar.gz

6、配置JDK

hadoop@cmaster:~$ mkdir app

hadoop@cmaster:~$ tar -zxvf jdk-7u71-linux-x64.tar.gz -C app/

hadoop@cmaster:~/app/jdk1.7.0_71$ pwd

/home/hadoop/app/jdk1.7.0_71

hadoop@cmaster:~$ sudo vi /etc/profile

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_71

export PATH=$JAVA_HOME/bin:$PATH

hadoop@cmaster:~$ source /etc/profile

hadoop@cmaster:~$ java -version

java version "1.7.0_71"

Java(TM) SE Runtime Environment (build 1.7.0_71-b14)

Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

7、配置Hadoop

hadoop@cmaster:~$ tar -zxvf hadoop-2.7.1.tar.gz -C app/

/home/hadoop/app/hadoop-2.7.1

hadoop@cmaster:~$ sudo vi /etc/profile

export HADOOP_HOME=/home/hadoop/app/hadoop-2.7.1

export PATH=$PATH:$HADOOP_HOME/bin

export HADOOP_HOME_WARN_SUPPRESS=1

hadoop@cmaster:~$ source /etc/profile

hadoop@cmaster:~$ hadoop version

Hadoop 2.7.1

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/hadoop-env.sh

# The java implementation to use.

#export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_71

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/core-site.xml

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://Master:9000</value>

<description>HDFS的URI,文件系统://namenode标识:端口号</description>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/usr/hadoop/tmp</value>

<description>namenode上本地的hadoop临时文件夹</description>

</property>

</configuration>

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/hdfs-site.xml

<configuration>

<!—hdfs-site.xml-->

<property>

<name>dfs.name.dir</name>

<value>/usr/hadoop/hdfs/name</value>

<description>namenode上存储hdfs名字空间元数据 </description>

</property>

<property>

<name>dfs.data.dir</name>

<value>/usr/hadoop/hdfs/data</value>

<description>datanode上数据块的物理存储位置</description>

</property>

<property>

<name>dfs.replication</name>

<value>5</value>

<description>副本个数,配置默认是3,应小于datanode机器数量</description>

</property>

</configuration>

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/yarn-site.xml

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>Master:8088</value>

</property>

</configuration>

hadoop@cmaster:~$ cp app/hadoop-2.7.1/etc/hadoop/mapred-site.xml.template app/hadoop-2.7.1/etc/hadoop/mapred-site.xml

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

hadoop@cmaster:~$ vi app/hadoop-2.7.1/etc/hadoop/slaves

localhost

cSlave1

cSlave2

cSlave3

cSlave4

分发

hadoop@cmaster:~$ sudo scp -r app/ cSlave1:/root

......

[root@cslave1 ~]# vi /etc/profile

# /etc/profile

export JAVA_HOME=/root/app/jdk1.7.0_71

export PATH=$JAVA_HOME/bin:$PATH

export HADOOP_HOME=/root/app/hadoop-2.7.1

export PATH=$PATH:$HADOOP_HOME/bin

export HADOOP_HOME_WARN_SUPPRESS=1

[root@cslave1 ~]# source /etc/profile

[root@cslave1 ~]# java -version

java version "1.7.0_71"

Java(TM) SE Runtime Environment (build 1.7.0_71-b14)

Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

hadoop@cmaster:~/app/hadoop-2.7.1/sbin$ hdfs namenode -format

./start-all.sh

http://172.16.1.110:8088/cluster

http://172.16.1.110:50070/explorer.html#/

五、参考资料 

HDFS

YARN

创建用户组

root@ubuntu:~# groupadd hadoop-user

创建Hadoop用户,所属组hadoop-user

root@ubuntu:~# useradd -g hadoop-user Hadoop

查看Hadoop所属组

root@ubuntu:~# groups Hadoop

Hadoop : hadoop-user

为Hadoop用户创建密码

root@ubuntu:~# passwd Hadoop

打开/etc/passwd文件

root:x:0:0:root:/root:/bin/bash

ubuntu:x:1000:1000:ubuntu,,,:/home/ubuntu:/bin/bash

Hadoop:x:1001:1001::/home/Hadoop:

用户对应的最后一个字段就指定了用户登陆时使用的shell类型,如果不指定,那么就使用默认的/bin/sh

sudo usermod -s shell类型  用户名  

root@ubuntu:~# usermod -s /bin/bash Hadoop

所有权转移到用户user,所属组hadoop-user

root@ubuntu:/home# chown Hadoop:hadoop-user Hadoop/

root@ubuntu:/home# chmod 007 Hadoop/

root@ubuntu:/home# groupadd hadoop-user

root@ubuntu:/home# useradd -s /bin/bash -g hadoop-user -G hadoop-user,root -d /home/Hadoop -m Hadoop

root@ubuntu:/home# ll

total 16

drwxr-xr-x  4 root   root        4096 Jan 28 22:26 ./

drwxr-xr-x 22 root   root        4096 Jan 28 08:54 ../

drwxr-xr-x  2 Hadoop hadoop-user 4096 Jan 28 22:26 Hadoop/

drwxr-xr-x  3 ubuntu ubuntu      4096 Jan 28 09:21 ubuntu/

route -n查看网关

------END


Copyright © 古田计算器虚拟社区@2017