hbase读取快照数据-lzo压缩遇到的问题
作者：互联网
1.读取hbase快照数据时报UnsatisfiedLinkError: no gplcompression in java.library.path错：
2019-09-04 17:36:07,441 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] GPLNativeCodeLoader:35 | Could not load native gpl library java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) at java.lang.Runtime.loadLibrary0(Runtime.java:870) at java.lang.System.loadLibrary(System.java:1122) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:127) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:328) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-09-04 17:36:07,460 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] LzoCodec:70 | Cannot load native-lzo without native-hadoop
在使用java 命令执行时可加上 -Djava.library.path=/opt/hadoop/lib/native，就没有此报错
java -Djava.library.path=/opt/hadoop/lib/native -cp lib/*:conf/*:stream-0.0.1.jar com.hbasereader.HbaseReader
2.解决完上述异常后，又出现native-lzo library not available异常
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) ... 21 more at com.suning.data.integration.engine.stream.core.exception.DiException.asDiException(DiException.java:51) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:131) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73) at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452) at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60) at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98) at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152) at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95) ... 12 more Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330) at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97) ... 21 more
错误是从这报出来的：
经过一周的排查终于找到原因了，真是坑爹，因为引用了hadoop-gpl-compression-xxx.jar，而hadoop-gpl-compression是比较老的版本，
将hadoop-gpl-compression注释掉，使用hadoop-lzo的jar包，问题解决
目前在使用Twitter维护的版本，在GitHub上，下载源码 git clone https://github.com/twitter/hadoop-lzo.git ，使用maven编译打包后，将hadoop-lzo-xxx.jar加入到项目中，完美解决上述native-lzo library not available 问题。
至于如何使用maven编译hadoop-lzo，可以参照我的一篇博客： Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法
注意：hadoop-gpl-compression已经过时，请使用hadoop-lzo
hadoop-gpl-compression已经过时，请使用hadoop-lzo
hadoop-gpl-compression已经过时，请使用hadoop-lzo
重要的事情说三遍！！！
标签：HbaseReader,java,hadoop,lzo,org,apache,快照,hbase 来源： https://www.cnblogs.com/qixing/p/11514823.html