系统相关
首页 > 系统相关> > 在centos7.2上Flink1.9.1源码编译(hadoop2.6.0-cdh5.15.1)

在centos7.2上Flink1.9.1源码编译(hadoop2.6.0-cdh5.15.1)

作者:互联网

编译步骤参照官网:https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html
0.按照官网:In addition you need Maven 3 and a JDK (Java Development Kit). Flink requires at least Java 8 to build.配置好前置条件。
1.github获取flink 最新代码

git clone https://github.com/apache/flink

2.切换到flink 1.9分支

git checkout release-1.9

3.查看自己的maven版本

[hadoop@hadoop002 ~]$ mvn -v
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /home/hadoop/app/apache-maven-3.6.1
Java version: 1.8.0_161, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_161/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
[hadoop@hadoop002 ~]$ 

如果版本是 3.1.x and 3.2.x  则直接 mvn clean install -DskipTests
如果版本是 3.3.x,则采用如下步骤
mvn clean install -DskipTests  -Dfast
cd flink-dist
mvn clean install
4.如要在其他第三方厂家使用flink 打包,例如要在cdh ,Hortonworks 平台上使用,我们需要指定hadoop 版本进行打包。那么要先对flink 的前置依赖 flink-shaded-hadoop-2 进行打包,在进行编译。
新建一个目录,从git上获取flink-shaded 的源码

git clone https://github.com/apache/flink-shaded.git

根据自己缺少的版本切换对应的代码分支,这里我缺少的是7.0版本的flink-shaded-hadoop-2

git checkout  release-7.0

修改flink-shaded项目中的pom.xml 这里修改是为了加入cdh 等中央仓库,否则编译对应版本可能找不到cdh 相关的包。
在pom.xml文件中的<profiles></profiles>添加

<profile>
	<id>vendor-repos</id>
	<activation>
		<property>
			<name>vendor-repos</name>
		</property>
	</activation>
	<!-- Add vendor maven repositories -->
	<repositories>
		<!-- Cloudera -->
		<repository>
			<id>cloudera-releases</id>
			<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
		<!-- Hortonworks -->
		<repository>
			<id>HDPReleases</id>
			<name>HDP Releases</name>
			<url>https://repo.hortonworks.com/content/repositories/releases/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<repository>
			<id>HortonworksJettyHadoop</id>
			<name>HDP Jetty</name>
			<url>https://repo.hortonworks.com/content/repositories/jetty-hadoop</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<!-- MapR -->
		<repository>
			<id>mapr-releases</id>
			<url>https://repository.mapr.com/maven/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
	</repositories>
</profile>

然后选择自己需要的CDH版本进行编译  flink-shaded-hadoop-2

mvn  clean install -DskipTests -Drat.skip=true -Pvendor-repos  -Dhadoop.version=2.6.0-cdh5.15.1
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] flink-shaded 7.0 ................................... SUCCESS [  1.619 s]
[INFO] flink-shaded-force-shading 7.0 ..................... SUCCESS [  0.446 s]
[INFO] flink-shaded-asm-6 6.2.1-7.0 ....................... SUCCESS [  0.563 s]
[INFO] flink-shaded-guava-18 18.0-7.0 ..................... SUCCESS [  1.176 s]
[INFO] flink-shaded-netty-4 4.1.32.Final-7.0 .............. SUCCESS [  2.720 s]
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-7.0 SUCCESS [  0.625 s]
[INFO] flink-shaded-jackson-parent 2.9.8-7.0 .............. SUCCESS [  0.014 s]
[INFO] flink-shaded-jackson-2 2.9.8-7.0 ................... SUCCESS [  1.097 s]
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-7.0 . SUCCESS [  0.770 s]
[INFO] flink-shaded-hadoop-2 2.6.0-cdh5.15.1-7.0 .......... SUCCESS [ 10.969 s]
[INFO] flink-shaded-hadoop-2-uber 2.6.0-cdh5.15.1-7.0 ..... SUCCESS [ 21.086 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  41.375 s
[INFO] Finished at: 2020-02-02T06:45:24+08:00
[INFO] ------------------------------------------------------------------------

flink-shaded-hadoop-2编译成功之后,在切换到flink的源码目录,因之前已经在根目录编译过了,切换到flink-dist,进行flink的编译:

mvn clean install -DskipTests -Pvendor-repos -Drat.skip=true -Pinclude-hadoop -Dhadoop.version=2.6.0-cdh5.15.1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.090 s
[INFO] Finished at: 2020-02-02T06:45:56+08:00
[INFO] ------------------------------------------------------------------------

在..flink/flink-dist/target/flink-1.9-SNAPSHOT-bin/flink-1.9-SNAPSHOT中,找到自己需要的flink bin包

[hadoop@hadoop002 flink-1.9-SNAPSHOT-bin]$ ll
total 0
drwxrwxr-x 9 hadoop hadoop 117 Feb  2 06:45 flink-1.9-SNAPSHOT

 

muyingmiao 发布了134 篇原创文章 · 获赞 4 · 访问量 1万+ 私信 关注

标签:INFO,SUCCESS,flink,Flink1.9,hadoop,7.0,源码,centos7.2,shaded
来源: https://blog.csdn.net/muyingmiao/article/details/104141399