首页 > 其他分享> > |NO.Z.00047|——————————|^^ 案例 ^^|——|Hadoop&PB级数仓.V05|---------------------------------------|PB数仓.v0

|NO.Z.00047|——————————|^^ 案例 ^^|——|Hadoop&PB级数仓.V05|---------------------------------------|PB数仓.v0

2022-04-10 15:02:44 作者：互联网

[BigDataHadoop：Hadoop&PB级数仓.V05] [BigDataHadoop.PB级企业电商离线数仓][|章节一|Hadoop|核心交易分析：ODS层建表&数据加载|——案例|]

一、ODS层建表与数据加载

### --- ODS建表：

~~~     ODS层的表结构与源数据基本类似（列名及数据类型）；
~~~     ODS层的表名遵循统一的规范；
~~~     所有的表都是分区表；字段之间的分隔符为, ；为表的数据数据文件指定了位置；

二、ODS层建表与数据加载

### --- ODS层建表：订单表

DROP TABLE IF EXISTS `ods.ods_trade_orders`;

CREATE EXTERNAL TABLE `ods.ods_trade_orders`(
`orderid` int,
`orderno` string,
`userid` bigint,
`status` tinyint,
`productmoney` decimal(10, 0),
`totalmoney` decimal(10, 0),
`paymethod` tinyint,
`ispay` tinyint,
`areaid` int,
`tradesrc` tinyint,
`tradetype` int,
`isrefund` tinyint,
`dataflag` tinyint,
`createtime` string,
`paytime` string,
`modifiedtime` string)
COMMENT '订单表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/orders/';

### --- ODS层建表：订单明细表

DROP TABLE IF EXISTS `ods.ods_trade_order_product`;

CREATE EXTERNAL TABLE `ods.ods_trade_order_product`(
`id` string,
`orderid` decimal(10,2),
`productid` string,
`productnum` string,
`productprice` string,
    `money` string,
`extra` string,
`createtime` string)
COMMENT '订单明细表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/order_product/';

### --- ODS层建表：产品信息表

DROP TABLE IF EXISTS `ods.ods_trade_product_info`;

CREATE EXTERNAL TABLE `ods.ods_trade_product_info`(
`productid` bigint,
`productname` string,
`shopid` string,
`price` decimal(10,0),
`issale` tinyint,
`status` tinyint,
`categoryid` string,
`createtime` string,
`modifytime` string)
COMMENT '产品信息表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/product_info/';

### --- ODS层建表：产品分类表

DROP TABLE IF EXISTS `ods.ods_trade_product_category`;

CREATE EXTERNAL TABLE `ods.ods_trade_product_category`(
`catid` int,
`parentid` int,
`catname` string,
`isshow` tinyint,
`sortnum` int,
`isdel` tinyint,
`createtime` string,
`level` tinyint)
COMMENT '产品分类表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/product_category';

### --- ODS层建表：商家店铺表

DROP TABLE IF EXISTS `ods.ods_trade_shops`;

CREATE EXTERNAL TABLE `ods.ods_trade_shops`(
`shopid` int,
`userid` int,
`areaid` int,
`shopname` string,
`shoplevel` tinyint,
`status` tinyint,
`createtime` string,
`modifytime` string)
COMMENT '商家店铺表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/shops';

### --- ODS层建表：商家地域组织表

DROP TABLE IF EXISTS `ods.ods_trade_shop_admin_org`;

CREATE EXTERNAL TABLE `ods.ods_trade_shop_admin_org`(
`id` int,
`parentid` int,
`orgname` string,
`orglevel` tinyint,
`isdelete` tinyint,
`createtime` string,
`updatetime` string,
`isshow` tinyint,
`orgType` tinyint)
COMMENT '商家地域组织表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/shop_org/';

### --- ODS层建表：支付方式表

DROP TABLE IF EXISTS `ods.ods_trade_payments`;

CREATE EXTERNAL TABLE `ods.ods_trade_payments`(
`id` string,
`paymethod` string,
`payname` string,
`description` string,
`payorder` int,
`online` tinyint)
COMMENT '支付方式表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by ','
location '/user/data/trade.db/payments/';

二、ODS层数据加载

### --- ODS层数据加载概述

~~~     DataX仅仅是将数据导入到了 HDFS ，数据并没有与Hive表建立关联。
~~~     脚本的任务：数据迁移、数据加载到ODS层；
~~~     # 对于增量加载数据而言：初始数据加载；该任务仅执行一次，不在脚本中。

### --- 创建ODS层数据加载脚本
~~~     特点：工作量大，繁琐，容易出错；与数据采集工作在一起；

[root@hadoop02 ~]# vim /data/yanqidw/script/trade/ods_load_trade.sh

#!/bin/bash

source /etc/profile

if [ -n "$1" ] ;then
    do_date=$1
else
    do_date=`date -d "-1 day" +%F`
fi

# 创建目录
hdfs dfs -mkdir -p /user/data/trade.db/product_category/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/shops/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/shop_org/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/payments/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/orders/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/order_product/dt=$do_date
hdfs dfs -mkdir -p /user/data/trade.db/product_info/dt=$do_date

# 数据迁移
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/product_category.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/shops.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/shop_org.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/payments.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/orders.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/order_product.json
python $DATAX_HOME/bin/datax.py -p "-Ddo_date=$do_date" /data/yanqidw/json/product_info.json

# 加载 ODS 层数据
sql="
alter table ods.ods_trade_orders add partition(dt='$do_date');
alter table ods.ods_trade_order_product add partition(dt='$do_date');
alter table ods.ods_trade_product_info add partition(dt='$do_date');
alter table ods.ods_trade_product_category add partition(dt='$do_date');
alter table ods.ods_trade_shops add partition(dt='$do_date');
alter table ods.ods_trade_shop_admin_org add partition(dt='$do_date');
alter table ods.ods_trade_payments add partition(dt='$do_date');
"

hive -e "$sql"

三、执行脚本：ODS加载数据

### --- 清理hdfs的环境
~~~     # 清理环境：实验环境可清理，生产环境慎重

hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/
hdfs dfs -rm -r -f  /user/data/trade.db/

### --- 执行脚本：ODS加载数据

[root@hadoop02 ~]# sh /data/yanqidw/script/trade/ods_load_trade.sh 2020-07-12

===============================END===============================

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart ——W.S.Landor

来自为知笔记(Wiz)

标签：V05,---------------------------------------,string,do,ods,trade,PB,date,data
来源： https://www.cnblogs.com/yanqivip/p/16125892.html