- 内容概述
1.安装及配置Phoenix
2.Phoenix的基本操作
3.使用Phoenix bulkload数据到HBase
4.使用Phoenix从HBase中导出数据到HDFS
- 测试环境
1.CDH5.11.2
2.RedHat7.2
3.Phoenix4.7.0
- 前置条件
1.CDH集群正常
2.HBase服务已经安装并正常运行
3.测试csv数据已准备
4.Redhat7中的httpd服务已安装并使用正常
2.在CDH集群中安装Phoenix
1.到Cloudera官网下载Phoenix的Parcel,注意选择与操作系统匹配的版本,因为本次测试使用的是Redhat7,所以选择后缀名为el7的文件。下载地址为:
http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/
具体需要下载的三个文件地址为:
http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/CLABS_PHOENIX-4.7.0-1.clabs_phoenix1.3.0.p0.000-el7.parcel http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/CLABS_PHOENIX-4.7.0-1.clabs_phoenix1.3.0.p0.000-el7.parcel.sha1 http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/manifest.json
2.将下载好的文件发布到httpd服务,可以用浏览器打开页面进行测试。
[ec2-user@ip-172-31-22-86 phoenix]$ pwd /var/www/html/phoenix [ec2-user@ip-172-31-22-86 phoenix]$ ll total 192852 -rw-r--r-- 1 root root 41 Jun 24 2016 CLABS_PHOENIX-4.7.0-1.clabs_phoenix1.3.0.p0.000-el7.parcel.sha1 -rw-r--r-- 1 root root 197466534 Jun 24 2016 CLABS_PHOENIX-4.7.0-1.clabs_phoenix1.3.0.p0.000-el7.parcel -rw-r--r-- 1 root root 4687 Jun 24 2016 manifest.json [ec2-user@ip-172-31-22-86 phoenix]$
3.从Cloudera Manager点击“Parcel”进入Parcel管理页面
点击“配置”,输入Phoenix的Parcel包http地址。
点击“保存更改“回到Parcel管理页面,发现CM已发现Phoenix的Parcel。
点击“下载”->“分配”->“激活”
4.回到CM主页,发现HBase服务需要部署客户端配置以及重启
重启HBase服务
安装完成。
3.如何在CDH集群中使用Phoenix
3.1Phoenix的基本操作
进入Phoenix的脚本命令目录
[ec2-user@ip-172-31-22-86 bin]$ cd /opt/cloudera/parcels/CLABS_PHOENIX/bin [ec2-user@ip-172-31-22-86 bin]$ ll total 16 -rwxr-xr-x 1 root root 672 Jun 24 2016 phoenix-performance.py -rwxr-xr-x 1 root root 665 Jun 24 2016 phoenix-psql.py -rwxr-xr-x 1 root root 668 Jun 24 2016 phoenix-sqlline.py -rwxr-xr-x 1 root root 674 Jun 24 2016 phoenix-utils.py
使用Phoenix登录HBase
[ec2-user@ip-172-31-22-86 bin]$ ./phoenix-sqlline.py Zookeeper not specified. Usage: sqlline.py <zookeeper> <optional_sql_file> Example: 1. sqlline.py localhost:2181:/hbase 2. sqlline.py localhost:2181:/hbase ../examples/stock_symbol.sql
需要指定Zookeeper
[ec2-user@ip-172-31-22-86 bin]$ ./phoenix-sqlline.py ip-172-31-21-45:2181:/hbase ... sqlline version 1.1.8 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> !tables +------------+--------------+-------------+---------------+----------+------------+--------------------+ | TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS | TYPE_NAME | SELF_REFERENCING_C | +------------+--------------+-------------+---------------+----------+------------+--------------------+ | | SYSTEM | CATALOG | SYSTEM TABLE | | | | | | SYSTEM | FUNCTION | SYSTEM TABLE | | | | | | SYSTEM | SEQUENCE | SYSTEM TABLE | | | | | | SYSTEM | STATS | SYSTEM TABLE | | | | | | | ITEM | TABLE | | | | +------------+--------------+-------------+---------------+----------+------------+--------------------+ 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
创建一张测试表
注意:建表必须指定主键。
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> create table hbase_test . . . . . . . . . . . . . . . . . . . . . .> ( . . . . . . . . . . . . . . . . . . . . . .> s1 varchar not null primary key, . . . . . . . . . . . . . . . . . . . . . .> s2 varchar, . . . . . . . . . . . . . . . . . . . . . .> s3 varchar, . . . . . . . . . . . . . . . . . . . . . .> s4 varchar . . . . . . . . . . . . . . . . . . . . . .> ); No rows affected (1.504 seconds)
在hbase shell中进行检查
插入一行数据。注意:Phoenix中没有insert语法,用upsert代替。参考:http://phoenix.apache.org/language/index.html
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test values('1','testname','testname1','testname2'); 1 row affected (0.088 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test; +-----+-----------+------------+------------+ | S1 | S2 | S3 | S4 | +-----+-----------+------------+------------+ | 1 | testname | testname1 | testname2 | +-----+-----------+------------+------------+ 1 row selected (0.049 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
在hbase shell中进行检查
删除这行数据,delete测试
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> delete from hbase_test where s1='1'; 1 row affected (0.018 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test; +-----+-----+-----+-----+ | S1 | S2 | S3 | S4 | +-----+-----+-----+-----+ +-----+-----+-----+-----+ No rows selected (0.045 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
在hbase shell中进行检查
更新数据测试,注意Phoenix中没有update语法,用upsert代替。插入多条数据需要执行多条upsert语句,没办法将所有的数据都写到一个“values”后面。
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test values('1','testname','testname1','testname2'); 1 row affected (0.017 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test values('2','testname','testname1','testname2'); 1 row affected (0.007 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test values('3','testname','testname1','testname2'); 1 row affected (0.008 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test; +-----+-----------+------------+------------+ | S1 | S2 | S3 | S4 | +-----+-----------+------------+------------+ | 1 | testname | testname1 | testname2 | | 2 | testname | testname1 | testname2 | | 3 | testname | testname1 | testname2 | +-----+-----------+------------+------------+ 3 rows selected (0.067 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test values('1','fayson','testname1','testname2'); 1 row affected (0.009 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test; +-----+-----------+------------+------------+ | S1 | S2 | S3 | S4 | +-----+-----------+------------+------------+ | 1 | fayson | testname1 | testname2 | | 2 | testname | testname1 | testname2 | | 3 | testname | testname1 | testname2 | +-----+-----------+------------+------------+ 3 rows selected (0.037 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
在hbase shell中进行检查
批量更新测试,创建另外一张表hbase_test1,表结构与hbase_test一样,并插入五条,有两条是hbase_test中没有的(主键为4,5),有一条与hbase_test中的数据不一样(主键为1),有两条是完全一样(主键为2,3)。
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> create table hbase_test1 . . . . . . . . . . . . . . . . . . . . . .> ( . . . . . . . . . . . . . . . . . . . . . .> s1 varchar not null primary key, . . . . . . . . . . . . . . . . . . . . . .> s2 varchar, . . . . . . . . . . . . . . . . . . . . . .> s3 varchar, . . . . . . . . . . . . . . . . . . . . . .> s4 varchar . . . . . . . . . . . . . . . . . . . . . .> ); No rows affected (1.268 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test1 values('1','fayson','testname1','testname2'); 1 row affected (0.031 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test1 values('2','testname','testname1','testname2'); 1 row affected (0.006 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test1 values('3','testname','testname1','testname2'); 1 row affected (0.005 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test1 values('4','testname','testname1','testname2'); 1 row affected (0.005 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test1 values('5','testname','testname1','testname2'); 1 row affected (0.007 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test1; +-----+-----------+------------+------------+ | S1 | S2 | S3 | S4 | +-----+-----------+------------+------------+ | 1 | fayson | testname1 | testname2 | | 2 | testname | testname1 | testname2 | | 3 | testname | testname1 | testname2 | | 4 | testname | testname1 | testname2 | | 5 | testname | testname1 | testname2 | +-----+-----------+------------+------------+ 5 rows selected (0.038 seconds)
批量更新,我们用hbase_test1中的数据去更新hbase_test。
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> upsert into hbase_test select * from hbase_test1; 5 rows affected (0.03 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from hbase_test; +-----+-----------+------------+------------+ | S1 | S2 | S3 | S4 | +-----+-----------+------------+------------+ | 1 | fayson | testname1 | testname2 | | 2 | testname | testname1 | testname2 | | 3 | testname | testname1 | testname2 | | 4 | testname | testname1 | testname2 | | 5 | testname | testname1 | testname2 | +-----+-----------+------------+------------+ 5 rows selected (0.039 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
批量更新发现对于已有的数据,如果值不一样,会覆盖,对于相同的数据会保持不变,对于没有的数据会直接作为新的数据插入。
3.2使用Phoenix bulkload数据到HBase
准备需要批量导入的测试数据,这里使用TPC_DS的item表数据。
[ec2-user@ip-172-31-22-86 ~]$ ll item.dat -rw-r--r-- 1 root root 28855325 Oct 3 10:23 item.dat [ec2-user@ip-172-31-22-86 ~]$ head -1 item.dat 1|AAAAAAAABAAAAAAA|1997-10-27||Powers will not get influences. Electoral ports should show low, annual chains. Now young visitors may pose now however final pages. Bitterly right children suit increasing, leading el|27.02|23.23|5003002|exportischolar #2|3|pop|5|Music|52|ableanti|N/A|3663peru009490160959|spring|Tsp|Unknown|6|ought|
因为Phoenix的bulkload只能导入csv,所以我们先把该数据的分隔符修改为逗号,并且后缀名改为.csv
[ec2-user@ip-172-31-22-86 ~]$ sed -i 's/|/,/g' item.dat [ec2-user@ip-172-31-22-86 ~]$ mv item.dat item.csv [ec2-user@ip-172-31-22-86 ~]$ ll item.csv -rw-r--r-- 1 ec2-user ec2-user 28855325 Oct 3 10:26 item.csv [ec2-user@ip-172-31-22-86 ~]$ head -1 item.csv 1,AAAAAAAABAAAAAAA,1997-10-27,,Powers will not get influences. Electoral ports should show low, annual chains. Now young visitors may pose now however final pages. Bitterly right children suit increasing, leading el,27.02,23.23,5003002,exportischolar #2,3,pop,5,Music,52,ableanti,N/A,3663peru009490160959,spring,Tsp,Unknown,6,ought,
上传该文件到HDFS
[ec2-user@ip-172-31-22-86 ~]$ hadoop fs -mkdir /fayson [ec2-user@ip-172-31-22-86 ~]$ hadoop fs -put item.csv /fayson [ec2-user@ip-172-31-22-86 ~]$ hadoop fs -ls /fayson Found 1 items -rw-r--r-- 3 ec2-user supergroup 28855325 2017-10-03 10:28 /fayson/item.csv [ec2-user@ip-172-31-22-86 ~]$
通过Phoenix创建item表,注意为了方便阅读,只创建了4个字段
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> create table item . . . . . . . . . . . . . . . . . . . . . .> ( . . . . . . . . . . . . . . . . . . . . . .> i_item_sk varchar not null primary key, . . . . . . . . . . . . . . . . . . . . . .> i_item_id varchar, . . . . . . . . . . . . . . . . . . . . . .> i_rec_start_varchar varchar, . . . . . . . . . . . . . . . . . . . . . .> i_rec_end_date varchar . . . . . . . . . . . . . . . . . . . . . .> ); No rows affected (1.268 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
执行bulkload命令导入数据
[ec2-user@ip-172-31-22-86 ~]$ HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol-1.2.0-cdh5.12.1.jar:/opt/cloudera/parcels/CDH/lib/hbase/conf hadoop jar /opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-4.7.0-clabs-phoenix1.3.0-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool -t item -i /fayson/item.csv 17/10/03 10:32:24 INFO util.QueryUtil: Creating connection with the jdbc url: jdbc:phoenix:ip-172-31-21-45.ap-southeast-1.compute.internal,ip-172-31-22-86.ap-southeast-1.compute.internal,ip-172-31-26-102.ap-southeast-1.compute.internal:2181:/hbase; ... 17/10/03 10:32:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-45.ap-southeast-1.compute.internal:2181,ip-172-31-22-86.ap-southeast-1.compute.internal:2181,ip-172-31-26-102.ap-southeast-1.compute.internal:2181 sessionTimeout=60000 watcher=hconnection-0x7a9c0c6b0x0, quorum=ip-172-31-21-45.ap-southeast-1.compute.internal:2181,ip-172-31-22-86.ap-southeast-1.compute.internal:2181,ip-172-31-26-102.ap-southeast-1.compute.internal:2181, baseZNode=/hbase 17/10/03 10:32:24 INFO zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-45.ap-southeast-1.compute.internal/172.31.21.45:2181. Will not attempt to authenticate using SASL (unknown error) ... 17/10/03 10:32:30 INFO mapreduce.Job: Running job: job_1507035313248_0001 17/10/03 10:32:38 INFO mapreduce.Job: Job job_1507035313248_0001 running in uber mode : false 17/10/03 10:32:38 INFO mapreduce.Job: map 0% reduce 0% 17/10/03 10:32:52 INFO mapreduce.Job: map 100% reduce 0% 17/10/03 10:33:01 INFO mapreduce.Job: map 100% reduce 100% 17/10/03 10:33:01 INFO mapreduce.Job: Job job_1507035313248_0001 completed successfully 17/10/03 10:33:01 INFO mapreduce.Job: Counters: 50 ... 17/10/03 10:33:01 INFO mapreduce.AbstractBulkLoadTool: Loading HFiles from /tmp/fef0045b-8a31-4d95-985a-bee08edf2cf9 ...
在Phoenix中查询该表
0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase> select * from item limit 10; +------------+-------------------+----------------------+-----------------+ | I_ITEM_SK | I_ITEM_ID | I_REC_START_VARCHAR | I_REC_END_DATE | +------------+-------------------+----------------------+-----------------+ | 1 | AAAAAAAABAAAAAAA | 1997-10-27 | | | 10 | AAAAAAAAKAAAAAAA | 1997-10-27 | 1999-10-27 | | 100 | AAAAAAAAEGAAAAAA | 1997-10-27 | 1999-10-27 | | 1000 | AAAAAAAAIODAAAAA | 1997-10-27 | 1999-10-27 | | 10000 | AAAAAAAAABHCAAAA | 1997-10-27 | 1999-10-27 | | 100000 | AAAAAAAAAKGIBAAA | 1997-10-27 | 1999-10-27 | | 100001 | AAAAAAAAAKGIBAAA | 1999-10-28 | 2001-10-26 | | 100002 | AAAAAAAAAKGIBAAA | 2001-10-27 | | | 100003 | AAAAAAAADKGIBAAA | 1997-10-27 | | | 100004 | AAAAAAAAEKGIBAAA | 1997-10-27 | 2000-10-26 | +------------+-------------------+----------------------+-----------------+ 10 rows selected (0.054 seconds) 0: jdbc:phoenix:ip-172-31-21-45:2181:/hbase>
在hbase shell中查询该表
hbase(main):002:0> scan 'ITEM', LIMIT => 10 ROW COLUMN+CELL 1 column=0:I_ITEM_ID, timestamp=1507041176470, value=AAAAAAAABAAAAAAA 1 column=0:I_REC_START_VARCHAR, timestamp=1507041176470, value=1997-10-27 1 column=0:_0, timestamp=1507041176470, value= 10 column=0:I_ITEM_ID, timestamp=1507041176470, value=AAAAAAAAKAAAAAAA 10 column=0:I_REC_END_DATE, timestamp=1507041176470, value=1999-10-27 10 column=0:I_REC_START_VARCHAR, timestamp=1507041176470, value=1997-10-27 10 column=0:_0, timestamp=1507041176470, value= ... 100004 column=0:I_REC_START_VARCHAR, timestamp=1507041176470, value=1997-10-27 100004 column=0:_0, timestamp=1507041176470, value= 10 row(s) in 0.2360 seconds
入库条数检查
条数相等,全部入库成功。
3.3使用Phoenix从HBase中导出数据到HDFS
Phoenix还提供了使用MapReduce导出数据到HDFS的功能,以pig的脚本执行。首先准备pig脚本。
[ec2-user@ip-172-31-22-86 ~]$ cat export.pig REGISTER /opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-4.7.0-clabs-phoenix1.3.0-client.jar; rows = load 'hbase://query/SELECT * FROM ITEM' USING org.apache.phoenix.pig.PhoenixHBaseLoader('ip-172-31-21-45:2181'); STORE rows INTO 'fayson1' USING PigStorage(',');
[ec2-user@ip-172-31-22-86 ~]$
执行该脚本
[ec2-user@ip-172-31-22-86 ~]$ pig -x mapreduce export.pig ... Counters: Total records written : 102000 Total bytes written : 4068465 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 Job DAG: job_1507035313248_0002 2017-10-03 10:45:38,905 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
导出成功后检查HDFS中的数据
[ec2-user@ip-172-31-22-86 ~]$ hadoop fs -ls /user/ec2-user/fayson1 Found 2 items -rw-r--r-- 3 ec2-user supergroup 0 2017-10-03 10:45 /user/ec2-user/fayson1/_SUCCESS -rw-r--r-- 3 ec2-user supergroup 4068465 2017-10-03 10:45 /user/ec2-user/fayson1/part-m-00000 [ec2-user@ip-172-31-22-86 ~]$ hadoop fs -cat /user/ec2-user/fayson1/part-m-00000 | head -2 1,AAAAAAAABAAAAAAA,1997-10-27, 10,AAAAAAAAKAAAAAAA,1997-10-27,1999-10-27 cat: Unable to write to output stream. [ec2-user@ip-172-31-22-86 ~]$
检查条数为10200与原始数据一致,全部导出成功。
4.总结
- 使用Cloudera提供的Phoenix Parcel,可以很方便的安装Phoenix。
- 使用Phoenix可以对HBase进行建表,删除,更新等操作,都是以大家熟悉的SQL方式操作。
- Phoenix提供了批量导入/导出数据的方式。批量导入只支持csv格式,分隔符为逗号。
- Phoenix中的SQL操作,可以马上同步到HBase,通过hbase shell检查都成功。
- 目前Cloudera官方提供的Phoenix版本较旧,为4.7.0,社区最新版本为4.11.0
- Phoenix提供的SQL语法较为简陋,没有insert/update,一律用upsert代替。
- 使用upsert插入数据时,只能一条一条插入,没法将全部字段值写到一个“values”后面。
1、下载
https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.8-HBase-1.2-cdh5.8
2、编译(编译时间较长,耐心等待)
mvn clean package -DskipTests
3、解压
将编译好的phoenix-4.8.0-cdh5.8.0.tar.gz解压出来
[root@cmbigdata1 phoenix]# tar -zxvf phoenix-4.8.0-cdh5.8.0.tar.gz [root@cmbigdata1 phoenix]# cd phoenix-4.8.0-cdh5.8.0 [root@cmbigdata1 phoenix-4.8.0-cdh5.8.0]# ll total 166152 drwxr-xr-x 2 root root 4096 Apr 18 16:41 bin -rw-r--r-- 1 root root 1930 Aug 8 2016 build.txt drwxr-xr-x 3 root root 4096 Aug 8 2016 dev drwxr-xr-x 2 root root 4096 Aug 8 2016 docs drwxr-xr-x 3 root root 4096 Aug 8 2016 examples drwxr-xr-x 2 root root 4096 Apr 18 16:40 lib -rw-r--r-- 1 root root 113247548 Apr 18 14:43 phoenix-4.8.0-cdh5.8.0-client.jar -rw-r--r-- 1 root root 6619716 Apr 18 14:30 phoenix-4.8.0-cdh5.8.0-queryserver.jar -rw-r--r-- 1 root root 22498517 Apr 18 14:43 phoenix-4.8.0-cdh5.8.0-server.jar -rw-r--r-- 1 root root 27739579 Apr 18 14:29 phoenix-4.8.0-cdh5.8.0-thin-client.jar
4、将phoenix-4.8.0-cdh5.8.0-server.jar拷贝到每一个RegionServer下
[root@cmbigdata2~]# find / -name 'phoenix-4.8.0-cdh5.8.0-server.jar' /soft/bigdata/clouderamanager/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hbase/lib/phoenix-4.8.0-cdh5.8.0-server.jar
cmbigdata2和cmbigdata3和cmbigdata4一样。
5、增加hbase-site.xml 配置
<property> <name>hbase.table.sanity.checks</name> <value>false</value> </property>
CDH修改方法:
在集群管理页面点击Hbase,进入Hbase管理界面
点击配置:
选择高级:
增加如下配置:
6、重启Hbase
这个很简单,不多说,会玩cloudermanager的人都知道。
7、登录phoenix
进入phoenix-4.8.0-cdh5.8.0/bin目录执行。
一,Phoenix的介绍
1,Phoenix, (“凤凰”),它相当于一个Java中间件,提供jdbc连接,操作hbase数据表。
2,Apache Phoenix是构建在HBase之上的关系型数据库层,作为内嵌的客户端JDBC驱动用以对HBase中的数据进行低延迟访问。
二,Phoenix的下载
1,官网上下载的Phoenix的都会在文件名上标注需要搭配的hbase版本号,注意要一致。
2,要注意在官网上http://apache.fayea.com/phoenix/ 下载,如果自己电脑上的安装的hbase版本是cdh的话,则这两者会冲突,使用sqlline.py连接hbase时候会报类似以下错误:
出错原因:phoenix官方版本pom文件里的hbase依赖并不是使用cdh版本的。
解决的方法: 所以,为了能够使得phoenix与cdh对应,我们需要从phoenix官网下载对应版本(4.6.0)的phoenix源码,修改pom文件依赖以及部分源码,并重新编译,得到适配于cdh5.4 hbase1.0.0 的phoenix。
三,解决的步骤
1,下载cdh版本的Phoenix,注意它需要搭配的hbase版本是hbase1.2版本。
https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.8-HBase-1.2-cdh5.8
2,然后把该文件夹(phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8)拷贝解压到如下路径:
D:\Software\Phoenix\phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8
3,利用maven对该文件夹(phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8)进行重新编译,具体操作如下:
(1),首先电脑 要安装maven包,安装过程网上自己百度一下,不再介绍了
(2), 然后在window终端里,进入该文件夹路径下(phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8):
D:\Software\Phoenix\phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8>
(3),然后输入如下命令:
D:\Software\Phoenix\phoenix-for-cloudera-4.8-HBase-1.2-cdh5.8> mvn clean package -DskipTests -Dcdh.flume.version=1.6.0
(4), 最后如果显示:
则说明编译成功。
(5) 将编译打包好后的\Software\Phoenix\phoenix-for-cloudera-4.8-Hbase-1.2-cdh5.8\phoenix-assembly\target\phoenix-4.8.0-cdh5.8.0.tar.gz进行解压phoenix-4.8.0-cdh5.8.0,解压后的文件可以放在当前路径上 。
4,接下来把编译后的整个文件夹(phoenix-for-cloudera-4.8-Hbase-1.2-cdh5.8)上传到集群上。
5, 将phoenix-4.8.0-cdh5.8.0中的phoenix-4.8.0-cdh5.8.0-server.jar拷贝到每一个RegionServer下/opt/cloudera/parcels/CDH/lib/hbase/lib
6,最后一步重启hbase集群。
7,进入集群中phoenix文件夹下的bin子文件夹下输入如下命令来开启phoenix了:
./sqlline.py dsbbzx1,dsbbzx4,dsbbzx5:2181
出现如下结果:
则说明Phoenix在集群上安装成功了,接下来就可以使用Phoenix了。
-------------------
Apache Phoenix
Phoenix supports thick
and thin
connection types:
- Thick client is faster, but must connect directly to ZooKeeper and HBase RegionServers.
- Thin client has fewer dependencies and connects through a Phoenix Query Server instance.
Use the appropriate default.driver
, default.url
, and the dependency artifact for your connection type.
Thick client connection
Properties
Name | Value |
---|---|
default.driver | org.apache.phoenix.jdbc.PhoenixDriver |
default.url | jdbc:phoenix:localhost:2181:/hbase-unsecure |
default.user | phoenix_user |
default.password | phoenix_password |
Dependencies
Artifact | Excludes |
---|---|
org.apache.phoenix:phoenix-core:4.4.0-HBase-1.0 |
Maven Repository: org.apache.phoenix:phoenix-core
Thin client connection
Properties
Name | Value |
---|---|
default.driver | org.apache.phoenix.queryserver.client.Driver |
default.url | jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF |
default.user | phoenix_user |
default.password | phoenix_password |
Dependencies
Before Adding one of the below dependencies, check the Phoenix version first.
Artifact | Excludes | Description |
---|---|---|
org.apache.phoenix:phoenix-server-client:4.7.0-HBase-1.1 | For Phoenix 4.7 | |
org.apache.phoenix:phoenix-queryserver-client:4.8.0-HBase-1.2 | For Phoenix 4.8+ |
Maven Repository: org.apache.phoenix:phoenix-queryserver-client
详见:http://zeppelin.apache.org/docs/0.7.1/interpreter/jdbc.html#apache-phoenix
来源:oschina
链接:https://my.oschina.net/u/1998220/blog/3016196