第38期：MySQL 时间类分区具体实现
作者：互联网
适用分区或者说分表最多的场景依然是针对时间字段做拆分， 这节我们详细讲讲如何更好的基于时间字段来拆分。分别按照年、月、日几个维度的实现方法以及一些细节注意事项。
第一，以年为维度做拆分
日期字段拆分粒度的选择跟业务检索请求密切相关。比如保留10年数据，每次查询基于某个具体年份做为过滤条件，那按照年拆分肯定最好。例如下面SQL：
select * from ytt_pt1 where log_date >='2018-01-01' and log_date < '2019-01-01';
那我们来看下按照年单独拆分的实际例子：表ytt_pt1 ，包含1000W条记录，以年为粒度建立分区表。
mysql> create table ytt_pt1(id bigint, log_date date);
Query OK, 0 rows affected (0.18 sec)
mysql> insert into ytt_pt1 select id,log_date from ytt_p1 limit 10000000;
Query OK, 10000000 rows affected (3 min 49.53 sec)
Records: 10000000 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE ytt_pt1 PARTITION BY RANGE (year(log_date))
-> (
-> PARTITION p0001 VALUES LESS THAN (2012),
-> PARTITION p0002 VALUES LESS THAN (2013),
-> PARTITION p0003 VALUES LESS THAN (2014),
-> PARTITION p0004 VALUES LESS THAN (2015),
-> PARTITION p0005 VALUES LESS THAN (2016),
-> PARTITION p0006 VALUES LESS THAN (2017),
-> PARTITION p0007 VALUES LESS THAN (2018),
-> PARTITION p0008 VALUES LESS THAN (2019),
-> PARTITION p0009 VALUES LESS THAN (2020),
-> PARTITION p0010 VALUES LESS THAN (2021),
-> PARTITION p_max VALUES LESS THAN (maxvalue)
-> );
Query OK, 10000000 rows affected (2 min 33.31 sec)
Records: 10000000 Duplicates: 0 Warnings: 0
看下按年为粒度的查询效果：以下SQL 直接走分区p0008，查询时间0.91秒， 这个时间不算短，后期可以增加过滤条件来减少查询时间。
mysql> select count(*) from ytt_pt1 where log_date >='2018-01-01' and log_date < '2019-01-01';
+----------+
| count(*) |
+----------+
| 1000204 |
+----------+
1 row in set (0.91 sec)
mysql> explain select count(*) from ytt_pt1 where log_date >='2018-01-01' and log_date < '2019-01-01'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ytt_pt1
partitions: p0008
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 998002
filtered: 11.11
Extra: Using where
1 row in set, 1 warning (0.00 sec)
需要注意：查询只能基于字段来直接过滤，如果基于字段表达式来过滤，MySQL 不确定走哪个分区，会扫描所有分区，处理方法和单表查询一样。比如语句：
select count(*) from ytt_pt1 where year(log_date) = '2018' ;
看下执行情况: MySQL扫描所有分区，查询执行时间9秒多。
mysql> select count(*) from ytt_pt1 where year(log_date) = '2018' ;
+----------+
| count(*) |
+----------+
| 1000204 |
+----------+
1 row in set (9.19 sec)
mysql> explain select count(*) from ytt_pt1 where year(log_date) = '2018' \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ytt_pt1
partitions: p0001,p0002,p0003,p0004,p0005,p0006,p0007,p0008,p0009,p0010,p_max
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 9982648
filtered: 100.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
如果非坚持这种写法，可以给优化器一个提示，具体到指定分区去检索数据，或者是基于字段表达式建一个虚拟列：
mysql> select count(*) from ytt_pt1 partition(p0008) where year(log_date) = '2018' ;
+----------+
| count(*) |
+----------+
| 1000204 |
+----------+
1 row in set (0.84 sec)
如果查询按照月作为维度过滤比较频繁，那肯定是按照月来拆最好；比如需要检索2020年当月的某些记录来做后续数据处理，大致SQL如下：
select * from ytt_pt1_按月拆分表 where log_date in ('2020-01-01'，'2020-01-02'，...)
第二，以月为维度做拆分
按照月来拆分，有以下两种写法：
第一种： 直接按照月来拆12个分区: 下面表 ytt_pt1_month1 分区类型为 LIST ，基于函数 month 直接计算。
mysql> show create table ytt_pt1_month1\G
*************************** 1. row ***************************
Table: ytt_pt1_month1
Create Table: CREATE TABLE `ytt_pt1_month1` (
`id` bigint DEFAULT NULL,
`log_date` date DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
/*!50100 PARTITION BY LIST (month(`log_date`))
(PARTITION p0001 VALUES IN (1) ENGINE = InnoDB,
PARTITION p0002 VALUES IN (2) ENGINE = InnoDB,
PARTITION p0003 VALUES IN (3) ENGINE = InnoDB,
PARTITION p0004 VALUES IN (4) ENGINE = InnoDB,
PARTITION p0005 VALUES IN (5) ENGINE = InnoDB,
PARTITION p0006 VALUES IN (6) ENGINE = InnoDB,
PARTITION p0007 VALUES IN (7) ENGINE = InnoDB,
PARTITION p0008 VALUES IN (8) ENGINE = InnoDB,
PARTITION p0009 VALUES IN (9) ENGINE = InnoDB,
PARTITION p0010 VALUES IN (10) ENGINE = InnoDB,
PARTITION p0011 VALUES IN (11) ENGINE = InnoDB,
PARTITION p0012 VALUES IN (12) ENGINE = InnoDB) */
1 row in set (0.00 sec)
比如要查询2020年前半个月的记录：查询限定在分区 p0001 里，但是时间不太理想，得0.66秒。
mysql> select count(*) from ytt_pt1_month1 where log_date in ('2020-01-01','2020-01-02','2020-01-03','2020-01-04','2020-01-05','2020-01-06','2020-01-07','2020-01-08','2020-01-09','2020-01-10','2020-01-11','2020-01-12','2020-01-13','2020-01-14','2020-01-15');
+----------+
| count(*) |
+----------+
| 41540 |
+----------+
1 row in set (0.66 sec)
mysql> explain select count(*) from ytt_pt1_month1 where log_date in ('2020-01-01','2020-01-02','2020-01-03','2020-01-04','2020-01-05','2020-01-06','2020-01-07','2020-01-08','2020-01-09','2020-01-10','2020-01-11','2020-01-12','2020-01-13','2020-01-14','2020-01-15')\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ytt_pt1_month1
partitions: p0001
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 848224
filtered: 50.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
第二种： 对于每年的数据，单独划分12个分区，也就是按照年月联合维度来分区，一共有 144 个分区，每个分区对应具体某一年某一月数据。
添加这个分区稍微麻烦些，克隆表 ytt_pt1_month1 为 ytt_pt1_month2 ，这里写个存储过程来添加分区信息：
DELIMITER $$
USE `ytt`$$
DROP PROCEDURE IF EXISTS `sp_add_partition_ytt_pt1_month2`$$
CREATE DEFINER=`root`@`%` PROCEDURE `sp_add_partition_ytt_pt1_month2`()
BEGIN
DECLARE i,j INT UNSIGNED DEFAULT 1;
DECLARE v_tmp_date DATE;
SET @stmt = '';
SET @stmt_begin = 'ALTER TABLE ytt_pt1_month2 PARTITION BY RANGE COLUMNS (log_date)(';
SET i = 2010;
WHILE i <= 2020 DO
SET j = 1;
WHILE j <= 12 DO
SET v_tmp_date = CONCAT(i,'-01-01');
SET @stmt = CONCAT(@stmt,'PARTITION p',i,'_',LPAD(j,2,"0"),' VALUES LESS THAN (''',DATE_ADD(v_tmp_date,INTERVAL j MONTH),'''),');
SET j = j + 1;
END WHILE;
SET i = i + 1;
END WHILE;
SET @stmt_end = 'PARTITION p_max VALUES LESS THAN (maxvalue))';
SET @stmt = CONCAT(@stmt_begin,@stmt,@stmt_end);
PREPARE s1 FROM @stmt;
EXECUTE s1;
DROP PREPARE s1;
SET @stmt = NULL;
SET @stmt_begin = NULL;
SET @stmt_end = NULL;
END$$
DELIMITER ;
mysql> call sp_add_partition_ytt_pt1_month2;
Query OK, 0 rows affected (2 min 20.48 sec)
结果类似这样：
PARTITION p2010_01 VALUES LESS THAN ('2010-02-01') ENGINE = InnoDB,
...
PARTITION p2010_12 VALUES LESS THAN ('2011-01-01') ENGINE = InnoDB,
PARTITION p2011_01 VALUES LESS THAN ('2011-02-01') ENGINE = InnoDB,
...
PARTITION p2011_12 VALUES LESS THAN ('2012-01-01') ENGINE = InnoDB,
...
PARTITION p2020_12 VALUES LESS THAN ('2021-01-01') ENGINE = InnoDB,
PARTITION p_max VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB)
加好分区后，来观察下刚才那个获取2020年前半个月记录的查询：
mysql> select count(*) from ytt_pt1_month2 where log_date in ('2020-01-01','2020-01-02','2020-01-03','2020-01-04','2020-01-05','2020-01-06','2020-01-07','2020-01-08','2020-01-09','2020-01-10','2020-01-11','2020-01-12','2020-01-13','2020-01-14','2020-01-15');
+----------+
| count(*) |
+----------+
| 41540 |
+----------+
1 row in set (0.06 sec)
mysql> explain select count(*) from ytt_pt1_month2 where log_date in ('2020-01-01','2020-01-02','2020-01-03','2020-01-04','2020-01-05','2020-01-06','2020-01-07','2020-01-08','2020-01-09','2020-01-10','2020-01-11','2020-01-12','2020-01-13','2020-01-14','2020-01-15')\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: ytt_pt1_month2
partitions: p2020_01
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 85498
filtered: 50.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
查询时间为0.06秒，比第一种拆分方法要快10倍。
那是不是说按照年月联合来分区一定比按照单月来分区更加优化？也不一定，如果查询语句过滤条件包含分区的固定月，比如每年的12月份都要查，这时候直接按月来分区肯定要优于按照年月来分区。
第三，以天为维度做拆分
按照天为维度过滤比较频繁的查询，以天来拆分最好。比如只想查询2020年1月1日当天的数据，大致 SQL 如下：
select * from ytt_pt1 where log_date = '2020-01-01'
类似按照年月联合维度，写个脚本或者存储过程来添加分区，这里唯一要注意的一点就是 MySQL 分区表数量有限制，最大为8192个，所以如果按照天来分区，存放10年数据，分区数量为3650个，也在限制之内。
修改下之前的存储过程，代码如下：
DELIMITER $$
USE `ytt`$$
DROP PROCEDURE IF EXISTS `sp_add_partition_ytt_pt1_day`$$
CREATE DEFINER=`root`@`%` PROCEDURE `sp_add_partition_ytt_pt1_day`(
IN f_year_start YEAR,
IN f_year_end YEAR
)
BEGIN
DECLARE v_days INT UNSIGNED DEFAULT 365;
DECLARE v_year DATE DEFAULT '2010-01-01';
DECLARE v_partition_name VARCHAR(64) DEFAULT '';
DECLARE v_log_date DATE;
DECLARE i,j INT UNSIGNED DEFAULT 1;
SET @stmt = '';
SET @stmt_begin = 'ALTER TABLE ytt_pt1_day PARTITION BY RANGE COLUMNS (log_date)(';
SET i = f_year_start;
WHILE i <= f_year_end DO
SET v_year = CONCAT(i,'-01-01');
SET v_days = DATEDIFF(DATE_ADD(v_year,INTERVAL 1 YEAR),v_year);
SET j = 1;
WHILE j <= v_days DO
SET v_log_date = DATE_ADD(v_year,INTERVAL j DAY);
SET v_partition_name = CONCAT('p',i,'_',LPAD(j,3,'0'));
SET @stmt = CONCAT(@stmt,'PARTITION ',v_partition_name,' VALUES LESS THAN (''',v_log_date,'''),');
SET j = j + 1;
END WHILE;
SET i = i + 1;
END WHILE;
SET @stmt_end = 'PARTITION p_max VALUES LESS THAN (maxvalue))';
SET @stmt = CONCAT(@stmt_begin,@stmt,@stmt_end);
PREPARE s1 FROM @stmt;
EXECUTE s1;
DROP PREPARE s1;
SELECT NULL,NULL,NULL INTO @stmt,@stmt_begin,@stmt_end;
END$$
DELIMITER ;
mysql> CALL sp_add_partition_ytt_pt1_day('2010','2020');
Query OK, 1 row affected (14 min 13.69 sec)
接下来，以天来查询时间一定是最短的，只有0.01秒。
mysql> select count(*) from ytt_pt1_day where log_date = '2020-01-01';
+----------+
| count(*) |
+----------+
| 2675 |
+----------+
1 row in set (0.01 sec)
此时这样的查询要是基于年或者月性能肯定不是最优。
mysql> select count(*) from ytt_pt1 where log_date = '2020-01-01';
+----------+
| count(*) |
+----------+
| 2675 |
+----------+
1 row in set (0.68 sec)
mysql> select count(*) from ytt_pt1_month1 where log_date = '2020-01-01';
+----------+
| count(*) |
+----------+
| 2675 |
+----------+
1 row in set (0.87 sec)
mysql> select count(*) from ytt_pt1_month2 where log_date = '2020-01-01';
+----------+
| count(*) |
+----------+
| 2675 |
+----------+
1 row in set (0.09 sec)
可以看到，此类查询基于其他方法分区时间明显比按天来的长。
总结下：
本篇主要讲述日常工作中最常用的数据拆分方式：按时间来拆分。时间字段又常按照年，月，日三种不同的维度来具体实现，并且详细讲述了具体的实现方式以及适用场景。
关于 MySQL 的技术内容，你们还有什么想知道的吗？赶紧留言告诉小编吧！
标签：38,分区,PARTITION,01,2020,MySQL,date,ytt,pt1 来源： https://blog.csdn.net/ActionTech/article/details/122176806