标签:cell 行键 RowFilter filterList rf 过滤器 new Hbase null
数据总数:746条数据
因为后面需要进行算法合成,而且spark目前对这种算法支持并不好,因此采用代码编写,所以在查询hbase的过程中采用的是java直接查询,
但是为了加快查询速度,我尽可能的使用了过滤器
1:初期Hbase的rowkey组合:时间+"_"+订单id
查询思路:
1:能快速检索,减少GC,采用过滤器
2:支持时间段查询
根据上面两点,我采用时间过滤,比如:startTime=201904010000 endTime=201904180000|;【注意这个符号:“|” 】然后根据行键过滤器
CompareFilter.CompareOp.GREATER_OR_EQUAL和
CompareFilter.CompareOp.LESS_OR_EQUAL进行大小对比
使用代码在查询的时候,添加了行键过滤器
FilterList filterList=new FilterList(); //time+id if(startTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.GREATER_OR_EQUAL, new BinaryComparator(Bytes.toBytes(startTime))); filterList.addFilter(rf); } if(endTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.LESS_OR_EQUAL, new BinaryComparator(Bytes.toBytes(endTime))); filterList.addFilter(rf); } scan.setFilter(filterList);
完整代码:
/** * 行键过滤器 * */ public static List<Map<String , String>> rowFilter(String tableName , String startTime , String endTime){ Connection connection = null; Scan scan = new Scan(); scan.setCacheBlocks(false); ResultScanner rs = null; Table table = null; List<Map<String , String>> list = new ArrayList<Map<String , String>>(); try{ connection = ConnectionFactory.createConnection(config); table = connection.getTable(TableName.valueOf(tableName)); FilterList filterList=new FilterList(); //time+id if(startTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.GREATER_OR_EQUAL, new BinaryComparator(Bytes.toBytes(startTime))); filterList.addFilter(rf); } if(endTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.LESS_OR_EQUAL, new BinaryComparator(Bytes.toBytes(endTime))); filterList.addFilter(rf); } scan.setFilter(filterList); rs = table.getScanner(scan); for (Result r : rs) { Map<String , String> map = new HashMap<String , String>(); for (Cell cell : r.listCells()) { map.put(Bytes.toString(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength()) , Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength())); } list.add(map); } }catch (Exception e){ e.printStackTrace(); }finally { if (null != rs) { rs.close(); } try { if (null != table) { table.close(); } if (null != connection && !connection.isClosed()) { System.out.println("scan Result is closed"); connection.close(); } } catch (IOException e) { e.printStackTrace(); } } return list; }初期完整代码
那么这种方案查询后返回的结果是:361条数据! 实际Hbase测试表中有746条数据,那么可以肯定,是行键过滤器出错了(后面再研究为啥出错)
改善:
更改rowkey结构,采用:订单id+"_"+time来实现
然后过滤器代码改善:
FilterList filterList=new FilterList(); //id+time if(startTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.GREATER_OR_EQUAL, new RegexStringComparator(".*_"+startTime)); filterList.addFilter(rf); } if(endTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.LESS_OR_EQUAL, new RegexStringComparator(".*_"+endTime)); filterList.addFilter(rf); } scan.setFilter(filterList);
上面其实就是采用正则表达式进行后缀识别,这样我就可以根据后缀进行时间过滤
完整代码:
/** * 行键过滤器 * */ public static List<Map<String , String>> rowEndFilter(String tableName , String startTime , String endTime){ Connection connection = null; Scan scan = new Scan(); scan.setCacheBlocks(false); ResultScanner rs = null; Table table = null; List<Map<String , String>> list = new ArrayList<Map<String , String>>(); try{ connection = ConnectionFactory.createConnection(config); table = connection.getTable(TableName.valueOf(tableName)); FilterList filterList=new FilterList(); //id+time if(startTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.GREATER_OR_EQUAL, new RegexStringComparator(".*_"+startTime)); filterList.addFilter(rf); } if(endTime != null){ RowFilter rf = new RowFilter(CompareFilter.CompareOp.LESS_OR_EQUAL, new RegexStringComparator(".*_"+endTime)); filterList.addFilter(rf); } scan.setFilter(filterList); rs = table.getScanner(scan); for (Result r : rs) { Map<String , String> map = new HashMap<String , String>(); for (Cell cell : r.listCells()) { map.put(Bytes.toString(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength()) , Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength())); } list.add(map); } }catch (Exception e){ e.printStackTrace(); }finally { if (null != rs) { rs.close(); } try { if (null != table) { table.close(); } if (null != connection && !connection.isClosed()) { System.out.println("scan Result is closed"); connection.close(); } } catch (IOException e) { e.printStackTrace(); } } return list; }View Code
上面就会查询出完整数据。
标签:cell,行键,RowFilter,filterList,rf,过滤器,new,Hbase,null
来源: https://www.cnblogs.com/niutao/p/10733272.html
本站声明:
1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。