首页 > 其他分享> > |NO.Z.00069|——————————|BigDataEnd|——|Hadoop&PB级数仓.V03|---------------------------------------|PB数仓.v

|NO.Z.00069|——————————|BigDataEnd|——|Hadoop&PB级数仓.V03|---------------------------------------|PB数仓.v

2022-04-11 19:34:57 作者：互联网

[BigDataHadoop：Hadoop&PB级数仓.V03] [BigDataHadoop.PB级企业电商离线数仓][|章节三|Hadoop|元数据管理工具Atlas：Hive血缘关系导入|]

一、Hive血缘关系导入

### --- 配置HIVE_HOME环境变量；
~~~     将atlas服务下atlas-application.properties配置文件拷贝到hive服务目录下

### --- 创建软连接：
~~~     将 $ATLAS_HOME/conf/atlasapplication.properties 拷贝到 $HIVE_HOME/conf 目录下
[root@hadoop02 ~]# ln -s $ATLAS_HOME/conf/atlas-application.properties \
$HIVE_HOME/conf/atlas-application.properties
~~~     # 查看创建的软连接
[root@hadoop02 ~]# ll $HIVE_HOME/conf/atlas-application.properties
/opt/yanqi/servers/hive-2.3.7/conf/atlas-application.properties -> /opt/yanqi/servers/atlas-1.2.0/conf/atlas-application.properties

### --- 拷贝jar包
~~~     若是在执行的时候发现少很多包
~~~     可以将$ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/目录下的jar包拷贝到$ATLAS_HOME/hook/hive/atlas-hive-plugin-impl/目录下

### --- 查看ATLAS_HOME下的三个jackson包
~~~     $ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/ 目录下的3个jar，
~~~     拷贝到 $ATLAS_HOME/hook/hive/atlas-hive-plugin-impl/ 目录下


[root@hadoop02 ~]# ls $ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/
jackson-jaxrs-base-2.9.9.jar           
jackson-jaxrs-json-provider-2.9.9.jar 
jackson-module-jaxb-annotations-2.9.9.jar

### --- 创建软连接

ln -s $ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-base-2.9.9.jar \
$ATLAS_HOME/hook/hive/atlas-hive-plugin-impl/jackson-jaxrs-base-2.9.9.jar

ln -s $ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-json-provider-2.9.9.jar \
$ATLAS_HOME/hook/hive/atlas-hive-plugin-impl/jackson-jaxrs-json-provider-2.9.9.jar

ln -s $ATLAS_HOME/server/webapp/atlas/WEB-INF/lib/jackson-module-jaxb-annotations-2.9.9.jar \
$ATLAS_HOME/hook/hive/atlas-hive-plugin-impl/jackson-module-jaxb-annotations-2.9.9.jar

### --- 查看创建的软连接

[root@hadoop02 ~]# cd /opt/yanqi/servers/atlas-1.2.0/hook/hive/atlas-hive-plugin-impl
 
[root@hadoop02 atlas-hive-plugin-impl]# ll jackson-jaxrs-base-2.9.9.jar \
jackson-jaxrs-json-provider-2.9.9.jar \
jackson-module-jaxb-annotations-2.9.9.jar
~~~ 输出参数
jackson-jaxrs-base-2.9.9.jar -> /opt/yanqi/servers/atlas-1.2.0/server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-base-2.9.9.jar
jackson-jaxrs-json-provider-2.9.9.jar -> /opt/yanqi/servers/atlas-1.2.0/server/webapp/atlas/WEB-INF/lib/jackson-jaxrs-json-provider-2.9.9.jar
jackson-module-jaxb-annotations-2.9.9.jar -> /opt/yanqi/servers/atlas-1.2.0/server/webapp/atlas/WEB-INF/lib/jackson-module-jaxb-annotations-2.9.9.jar

二、修改hive配置

### --- 修改Hive的配置：hive-site.xml增加 hook

[root@hadoop02 ~]# vim $HIVE_HOME/conf/hive-site.xml
<!-- Atlas-hive配置 -->
<property>
        <name>hive.exec.post.hooks</name>
        <value>org.apache.atlas.hive.hook.HiveHook</value>
</property>

### --- $HIVE_HOME/conf/hive-env.sh中添加HIVE_AUX_JARS_PATH变量
~~~     $HIVE_HOME/conf/hive-env.sh中添加HIVE_AUX_JARS_PATH变量

[root@hadoop02 ~]# cd $HIVE_HOME/conf/
[root@hadoop02 conf]# cp hive-env.sh.template hive-env.sh
[root@hadoop01 conf]# vim hive-env.sh
~~~ 最后一行加入如下参数
export HIVE_AUX_JARS_PATH=/opt/yanqi/servers/atlas-1.2.0/hook/hive

三、向Atlas导入hive数据

### --- 启动hive

~~~     # 在Hadoop01启动hdfs和yarn服务
[root@hadoop01 ~]# start-dfs.sh
[root@hadoop01 ~]# start-yarn.sh
~~~     # 在Hadoop01和Hadoop05启动hive.server
[root@hadoop01 ~]# nohup hive --service metastore &
[root@hadoop01 ~]# lsof -i:9083
~~~     # 在Hadoop02进入hive.client端
[root@hadoop02 ~]# hive

### --- 批量导入hive数据
~~~     hive批量导入数据
~~~     Hive能正常启动；在执行的过程中需要用户名/口令：admin/admin

[root@hadoop02 ~]# cd /opt/yanqi/servers/atlas-1.2.0/bin/
[root@hadoop02 bin]# ./import-hive.sh
~~~输入参数
Enter username for atlas :- admin
Enter password for atlas :- 
~~~输入成功参数   
Hive Meta Data imported successfully!!!      # 成功导出可以看见最后的提示信息：

四、在浏览器中可以看见：Search 中的选项有变化

### --- Hive hook 可捕获以下操作：

create database
create table/view, create table as select
load, import, export
DMLs (insert)
alter database
alter table
alter view

===============================END===============================

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart ——W.S.Landor

来自为知笔记(Wiz)

标签：---------------------------------------,jackson,v03,jar,hive,PB,2.9,atlas,HOME
来源： https://www.cnblogs.com/yanqivip/p/16131533.html