Flink SQL实战演练之脚手架开发

flink sql目前开发的方式包括纯sql的开发，可以运行在zeppelin上，也可以采用代码进行开发，打包成jar运行，代码开发的话每次都要新建工程，会有一些重复性的工作要做，本章讲述通过脚手架的方式快速构建flink sql基本工程。

  • 创建工程，作为蓝本
mvn archetype:generate
-DarchetypeGroupId=org.apache.flink
-DarchetypeArtifactId=flink-quickstart-java
-DarchetypeVersion=1.14.2
-DgroupId=com.dpf.flink
-DartifactId=flink-sql
-Dversion=0.1
-Dpackage=com.dpf.flink
  • 添加基础代码
package com.dpf.flink;

import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.table.catalog.hive.HiveCatalog;

public class BatchSql {

    public static String hiveCatalog = "hive";
    public static String defaultDatabase = "default";
    public static String hiveConfDir = "/Users/dengpengfei/bigdata/apache-hive-3.1.2-bin/conf";

    public static void main(String[] args) {
        EnvironmentSettings settings = EnvironmentSettings.inBatchMode();
        TableEnvironment tableEnv = TableEnvironment.create(settings);

        //添加hive catalog
        HiveCatalog hive = new HiveCatalog(hiveCatalog, defaultDatabase, hiveConfDir);
        tableEnv.registerCatalog(hiveCatalog, hive);

        tableEnv.useCatalog(hiveCatalog);

        tableEnv.executeSql("select * from hive_table").print();
    }
}
  • 在项目pom中加入archetype插件
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-archetype-plugin</artifactId>
    <version>3.2.0</version>
</plugin>

  • 在maven中，双击archetype:create-from-project运行archetype插件


    脚手架生成

    *注：配置高版本的maven，idea中默认的可能出问题，运行结果如下：

[INFO] --------------------------[ maven-archetype ]---------------------------
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ flink-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Using 'null' encoding to copy filtered properties files.
[INFO] Copying 7 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ flink-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Using 'null' encoding to copy filtered properties files.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-archetype-plugin:3.2.0:jar (default-jar) @ flink-archetype ---
[INFO] Building archetype jar: /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/flink-archetype-1.0-SNAPSHOT.jar
[INFO] Building jar: /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/flink-archetype-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.389 s
[INFO] Finished at: 2021-12-28T20:54:10+08:00
[INFO] ------------------------------------------------------------------------
[INFO] Archetype project created in /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.704 s
[INFO] Finished at: 2021-12-28T20:54:10+08:00
[INFO] ------------------------------------------------------------------------
  • 执行安装命令
david.dong@localhost archetype % cd /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype
david.dong@localhost archetype % mvn clean install
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------------< com.dpf:flink-archetype >-----------------------
[INFO] Building flink-archetype 1.0-SNAPSHOT
[INFO] --------------------------[ maven-archetype ]---------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink-archetype ---
[INFO] Deleting /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ flink-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Using 'null' encoding to copy filtered properties files.
[INFO] Copying 7 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ flink-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Using 'null' encoding to copy filtered properties files.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-archetype-plugin:3.2.0:jar (default-jar) @ flink-archetype ---
[INFO] Building archetype jar: /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/flink-archetype-1.0-SNAPSHOT.jar
[INFO] Building jar: /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/flink-archetype-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-archetype-plugin:3.2.0:integration-test (default-integration-test) @ flink-archetype ---
[INFO] Processing Archetype IT project: basic
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: flink-archetype:1.0-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: archetype.it
[INFO] Parameter: artifactId, Value: basic
[INFO] Parameter: version, Value: 0.1-SNAPSHOT
[INFO] Parameter: package, Value: it.pkg
[INFO] Parameter: packageInPathFormat, Value: it/pkg
[INFO] Parameter: version, Value: 0.1-SNAPSHOT
[INFO] Parameter: package, Value: it.pkg
[INFO] Parameter: groupId, Value: archetype.it
[INFO] Parameter: artifactId, Value: basic
[INFO] Project created from Archetype in dir: /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/test-classes/projects/basic/project/basic
[INFO] 
[INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ flink-archetype ---
[INFO] Installing /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/target/flink-archetype-1.0-SNAPSHOT.jar to /Users/david.dong/.m2/repository/com/dpf/flink-archetype/1.0-SNAPSHOT/flink-archetype-1.0-SNAPSHOT.jar
[INFO] Installing /Users/david.dong/workspace/archetype/flink/target/generated-sources/archetype/pom.xml to /Users/david.dong/.m2/repository/com/dpf/flink-archetype/1.0-SNAPSHOT/flink-archetype-1.0-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-archetype-plugin:3.2.0:update-local-catalog (default-update-local-catalog) @ flink-archetype ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.417 s
[INFO] Finished at: 2021-12-28T21:07:43+08:00
[INFO] ------------------------------------------------------------------------
  • 安装成功之后，就可以在本地仓库的根目录(如：/Users/david.dong/.m2/repository)中看到一个文件：archetype-catalog.xml，内容如下：
<?xml version="1.0" encoding="UTF-8"?>
<archetype-catalog xsi:schemaLocation="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd"
    xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <archetypes>
    <archetype>
      <groupId>com.dpf</groupId>
      <artifactId>flink-archetype</artifactId>
      <version>1.0-SNAPSHOT</version>
      <description>flink-archetype</description>
    </archetype>
  </archetypes>
</archetype-catalog>

  • 在idea中引入脚手架 点击菜单File -> new -> Project，选择maven，勾选Create from archetype，点击 Add Archetype按钮，填写groupId和artifactId、version信息（就是archetype-catalog.xml文件中的信息），点击OK，便添加完成，如下图：


    idea添加脚手架

  • 创建项目时使用脚手架


    使用脚手架

结

工欲善其事，必先利其器，将基础代码构建成脚手架很有必要。

©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

推荐阅读更多精彩内容