Ambari是什么

Ambari 是 Apache 软件基金会的一个顶级项目。

Apache Ambari项目用于配置、管理和监视Apache Hadoop集群的软件，简化Hadoop管理。Ambari提供了一个直观、易于使用的Hadoop管理web UI。

但是这里的 Hadoop 是广义，指的是 Hadoop 整个生态圈（例如 Hive，Hbase，Sqoop，Zookeeper 等），而并不仅是特指 Hadoop。

用一句话来说，Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个工具。

Ambari组成

Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。简单来说，用户通过 Ambari Server 通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI（图形用户界面），方便用户了解到集群的各种状态，并进行相应的维护。

HDP是什么

Hortonworks Data Platform (HDP)是一个用于分布式存储和处理大型多源数据集的开源框架。

安装步骤

一、集群规划

主机名	IP地址	功能
hadoop101	192.168.10.101	Yum源
hadoop102	192.168.10.102	Ambri Server + MySQL
hadoop103	192.168.10.103	Compute Node
hadoop104	192.168.10.104	Compute Node

二、下载安装包

HDP：

http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos6-rpm.tar.gz

ambari：

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari-2.6.1.0-centos6.tar.gz

HDP-UTILS：

http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6/HDP-UTILS-1.1.0.21-centos6.tar.gz

三、新建虚拟机，安装CentOS

安装步骤省略

注意：磁盘空间不低于50G

四、关闭防火墙

service iptables stop
chkconfig iptables off

五、关闭SELINUX

vim /etc/selinux/config
修改其中的 SELINUX=enforcing 为 SELINUX=disabled

六、为提升hadoop性能，hadoop环境上要求关闭Transparent Hugepages（THP，页面内存透明化）

1、查看是否启用

[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never

[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

2、禁用　编辑rc.local启动文件

[root@localhost ~]# vim /etc/rc.d/rc.local
增加下列内容：
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

3、重启

[root@localhost ~]# reboot

七、配置aliYum源

https://developer.aliyun.com/mirror/centos?spm=a2c6h.13651102.0.0.53322f708nr7q9

1、备份

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

2、下载新的CentOS-Base.repo 到/etc/yum.repos.d/

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo

3、之后运行`yum makecache`生成缓存

八、配置时间同步

yum -y install ntp
service ntpd start
chkconfig ntpd on

九、安装jdk

#解压
tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr
ln -s jdk1.8.0_144 java
# 配置环境变量
vim /etc/profile
export JAVA_HOME=/usr/java
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile

十、配置IP映射

vim /etc/hosts
192.168.10.101 hadoop101
192.168.10.102 hadoop102
192.168.10.103 hadoop103
192.168.10.104 hadoop104

十一、克隆虚拟机

步骤略

十二、安装MySQL【hadoop102】

1、检查是否安装了MySQL
[root@hadoop201 ~]# rpm -qa | grep -i mysql
mysql-libs-5.1.71-1.el6.x86_64
2、卸载
[root@hadoop201 ~]# rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
3、安装
[root@hadoop201 ~]# rpm -ivh /opt/MySQL-server-5.5.28-1.linux2.6.x86_64.rpm
4、启动MySQL
[root@hadoop201 ~]# service mysql start
chkconfig mysql on
5、安装客户端
[root@hadoop201 ~]# rpm -ivh  /opt/MySQL-client-5.5.28-1.linux2.6.x86_64.rpm
6、执行配置脚本（注意：尽量不要使用全数字密码，避免出问题）
我的密码是 hadoop102
[root@hadoop201 ~]# /usr/bin/mysql_secure_installation
7、授权无主机登录
[root@hadoop102 ~]# mysql -uroot -p hadoop102
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'hadoop102';
mysql> FLUSH PRIVILEGES;

十三、安装httpd(hadoop101节点安装)

1、安装httpd

yum -y install httpd

2、在/var/www/html下创建ambari和hdp目录

cd /var/www/html/
mkdir ambari
mkdir hdp
mkdir -p hdp/HDP-UTILS

3、上传资源包

上传ambari-2.6.1.0-centos6.tar.gz、HDP-2.6.4.0-centos6-rpm.tar.gz、HDP-UTILS-1.1.0.21-centos6.tar.gz

4、解压到指定目录

tar -zxvf /opt/ambari-2.6.1.0-centos6.tar.gz -C /var/www/html/ambari/
tar -zxvf /opt/HDP-2.6.4.0-centos6-rpm.tar.gz -C /var/www/html/hdp/
tar -zxvf /opt/HDP-UTILS-1.1.0.21-centos6.tar.gz -C /var/www/html/hdp/HDP-UTILS/

5、启动httpd

service httpd restart
chkconfig httpd on

6、web访问

现在可以通过访问http://192.168.10.101/ambari/查看是否能成功访问

十四、配置Ambari+HDP本地yum源

1、安装本地源制作相关工具

yum install -y yum-utils createrepo yum-plugin-priorities

2、编辑ambari.repo

[root@hadoop101 ~]# cd  /var/www/html/ambari
[root@hadoop101 ~]# vim  ambari/centos6/2.6.1.0-143/ambari.repo
#VERSION_NUMBER=2.6.1.0-143
[ambari-2.6.1.0]
name=ambari Version - ambari-2.6.1.0
baseurl=http://192.168.10.101/ambari/ambari/centos6/2.6.1.0-143/
gpgcheck=1
gpgkey=http://192.168.10.101/ambari/ambari/centos6/2.6.1.0-143/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

3、复制ambari.repo

cp ambari/centos6/2.6.1.0-143/ambari.repo  /etc/yum.repos.d/

4、编辑hdp.repo

[root@hadoop101 ~]# cd /var/www/html/hdp
[root@hadoop101 ~]# vim HDP/centos6/2.6.4.0-91/hdp.repo
#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://192.168.10.101/hdp/HDP/centos6/2.6.4.0-91/
gpgcheck=1
gpgkey=http://192.168.10.101/hdp/HDP/centos6/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1


[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.10.101/hdp/HDP-UTILS
gpgcheck=1
gpgkey=http://192.168.10.101/hdp/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

5、复制hdp.repo

cp HDP/centos6/2.6.4.0-91/hdp.repo /etc/yum.repos.d/

6、配置免密登录

ssh-keygen
ssh-copy-id root@hadoop101
ssh-copy-id root@hadoop102
ssh-copy-id root@hadoop103
ssh-copy-id root@hadoop104

7、将创建好的文件拷贝到子节点

[root@hadoop101 etc]# scp -r yum.repos.d/ hadoop102:$PWD
[root@hadoop101 etc]# scp -r yum.repos.d/ hadoop103:$PWD
[root@hadoop101 etc]# scp -r yum.repos.d/ hadoop104:$PWD

8、生成本地源

createrepo -v /var/www/html/hdp/HDP/centos6/
createrepo -v /var/www/html/hdp/HDP-UTILS/

十五、安装Ambari Server【hadoop102】

1、安装和配置ambari-server

yum -y install ambari-server

#在mysql上创建 database ambari;
mysql -uroot -phadoop102
create database ambari;
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
#在mysql上创建 database hive;
create database hive;


#mysql驱动放入到：
mkdir /usr/share/java
cp /opt/mysql-connector-java-5.1.39.jar /var/lib/ambari-server/resources/
cp /opt/mysql-connector-java-5.1.39.jar /usr/share/java/
cp /opt/mysql-connector-java-5.1.39.jar /usr/lib/ambari-server/

2、python中英文编码问题

#为了解决后期安装服务时python中英文编码问题，需要vim /usr/lib/python2.6/site-packages/resource_management/core/logger.py文件（master.one节点）
#在import sys下添加如下代码：
reload(sys)
sys.setdefaultencoding('utf-8')

3、配置ambari-server

ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 192.160.10.102
Port (3306): 3306
Database name (ambari): 
Username (ambari): root
Enter Database Password (bigdata): hadoop102
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.6.1.0.143.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

4、启动ambari

ambari-server start

注意：出现Server not yet listening on http port 8080 after 50 seconds. Exiting异常执行以下语句：

echo 'server.startup.web.timeout=120' >> /etc/ambari-server/conf/ambari.properties

十六、安装agent

在所有节点上安装ambari-agent

yum install -y ambari-agent
service ambari-agent start
chkconfig ambari-agent on

成功启动后在浏览器输入Ambari地址：

http://192.168.10.102:8080

十七、官方文档

https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.4/bk_release-notes/bk_release-notes.pdf

https://docs.cloudera.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/bk_ambari-installation.pdf

更多技术文章：请点击 木子李博客

来源：CSDN

作者：木子李G

链接：https://blog.csdn.net/ligang_blog/article/details/104042392

标签

ambari

yum

var

rpm

mysql

etc

Hadoop

大数据平台的利器--Ambari+HDP