11-flink如何正确分流

11-flink如何正确分流

分流方式

  1. filter分流
  2. split分流 (只能一次分流，分流后的流不能继续分流)
  3. side output分流 (推荐使用)

场景

flink-分流场景.png

输入数据：

{"key":"001","type":"1","data":"data1"}
{"key":"001","type":"11","data":"data11"}
{"key":"001","type":"12","data":"data12"}
{"key":"002","type":"2","data":"data2"}
{"key":"002","type":"21","data":"data21"}
{"key":"002","type":"22","data":"data22"}

Operator中处理：转为jsonObject处理,并在jsonObject对象中添加流标志；

pom中需要引入fastjson (fastjson使用教程);

<dependency>
  <groupId>com.alibaba</groupId>
  <artifactId>fastjson</artifactId>
  <version>1.2.60</version>
</dependency>

输出结果为：

11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}

发送数据客户端

nc程序发送(nc程序实现)

Fliter分流

代码实现：

public class FliterStreamTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception {

        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        DataStream<JSONObject> stream1 = jsonObjectStream
                .filter(e -> e.get(TYPE).equals("1") || e.get(TYPE).equals("11") || e.get(TYPE).equals("12"));
        DataStream<JSONObject> stream11 = stream1.filter(e -> e.get(TYPE).equals("11"));
        DataStream<JSONObject> stream12 = stream1.filter(e -> e.get(TYPE).equals("12"));

        DataStream<JSONObject> stream2 = jsonObjectStream
                .filter(e -> e.get(TYPE).equals("2") || e.get(TYPE).equals("21") || e.get(TYPE).equals("22"));
        DataStream<JSONObject> stream21 = stream2.filter(e -> e.get(TYPE).equals("21"));
        DataStream<JSONObject> stream22 = stream2.filter(e -> e.get(TYPE).equals("22"));

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }
}

输出结果(与正确结果一直)：

11> {"data":"data1","stream":"stream1","type":"1","key":"001"}
12> {"data":"data11","stream":"stream1","type":"11","key":"001"}
12> {"data":"data11","stream":"stream11","type":"11","key":"001"}
1> {"data":"data12","stream":"stream1","type":"12","key":"001"}
1> {"data":"data12","stream":"stream12","type":"12","key":"001"}
2> {"data":"data2","stream":"stream2","type":"2","key":"002"}
3> {"data":"data21","stream":"stream2","type":"21","key":"002"}
3> {"data":"data21","stream":"stream21","type":"21","key":"002"}
4> {"data":"data22","stream":"stream2","type":"22","key":"002"}
4> {"data":"data22","stream":"stream22","type":"22","key":"002"}

Split分流

代码实现：

public class SplitStreamTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception {

        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        //flink split流已被标记为过期
        SplitStream<JSONObject> splitStream = jsonObjectStream.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("1") || type.equals("11") || type.equals("12")) {
                    tags.add("stream1");
                } else if (type.equals("2") || type.equals("21") || type.equals("22")) {
                    tags.add("stream2");
                }
                return tags;
            }
        });

        DataStream<JSONObject> stream1 = splitStream.select("stream1");

        DataStream<JSONObject> stream2 = splitStream.select("stream2");

        SplitStream<JSONObject> splitStream1 = stream1.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("11")) {
                    tags.add("stream11");
                } else if (type.equals("12")) {
                    tags.add("stream12");
                }
                return tags;
            }
        });

        SplitStream<JSONObject> splitStream2 = stream1.split(new OutputSelector<JSONObject>() {
            @Override
            public Iterable<String> select(JSONObject jsonObject) {
                List<String> tags = new ArrayList<>();
                String type = jsonObject.get("type").toString();
                if (type.equals("21")) {
                    tags.add("stream21");
                } else if (type.equals("22")) {
                    tags.add("stream22");
                }
                return tags;
            }
        });

        DataStream<JSONObject> stream11 = splitStream1.select("stream11");
        DataStream<JSONObject> stream12 = splitStream1.select("stream12");
        DataStream<JSONObject> stream21 = splitStream2.select("stream21");
        DataStream<JSONObject> stream22 = splitStream2.select("stream22");

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }

}

输出结果：

Flink 1.9以前可以输出，但是第二次split将会失效；Flink 1.9 如果二次split，直接报错并提示使用side-outputs

Exception in thread "main" java.lang.IllegalStateException: Consecutive multiple splits are not supported. Splits are deprecated. Please use side-outputs.

Side-outputs分流(推荐使用)

步骤：

  1. 定义 OutputTag
  2. process 算子中收集数据
    1. ProcessFunction
    2. KeyedProcessFunction
    3. CoProcessFunction
    4. ProcessWindowFunction
    5. ProcessAllWindowFunction

代码实现：

public class SideOutputTest {

    public static final String TYPE = "type";

    public static void main(String[] args) throws Exception{
        //获取执行环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        final ParameterTool params = ParameterTool.fromArgs(args);
        String hostName = params.get("hostname", "localhost");
        int port = params.getInt("port", 9000);

        DataStream<String> sourceStream = env.socketTextStream(hostName, port, "\n");

        SingleOutputStreamOperator<JSONObject> jsonObjectStream = sourceStream.map(s -> JSONObject.parseObject(s));

        //定义OutputTag
        OutputTag<JSONObject> outputTag1 = new OutputTag<JSONObject>("stream1") {};
        OutputTag<JSONObject> outputTag2 = new OutputTag<JSONObject>("stream2") {};

        SingleOutputStreamOperator<JSONObject> outputStream = jsonObjectStream.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("1") || type.equals("11") || type.equals("12")) {
                    context.output(outputTag1, jsonObject);
                } else if (type.equals("2") || type.equals("21") || type.equals("22")) {
                    context.output(outputTag2, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream1 = outputStream.getSideOutput(outputTag1);
        DataStream<JSONObject> stream2 = outputStream.getSideOutput(outputTag2);

        //定义OutputTag
        OutputTag<JSONObject> outputTag11 = new OutputTag<JSONObject>("stream11") {};
        OutputTag<JSONObject> outputTag12 = new OutputTag<JSONObject>("stream12") {};

        SingleOutputStreamOperator<JSONObject> outputStream1 = stream1.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("11")) {
                    context.output(outputTag11, jsonObject);
                } else if (type.equals("12")) {
                    context.output(outputTag12, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream11 = outputStream1.getSideOutput(outputTag11);
        DataStream<JSONObject> stream12 = outputStream1.getSideOutput(outputTag12);

        //定义OutputTag
        OutputTag<JSONObject> outputTag21 = new OutputTag<JSONObject>("stream21") {};
        OutputTag<JSONObject> outputTag22 = new OutputTag<JSONObject>("stream22") {};

        SingleOutputStreamOperator<JSONObject> outputStream2 = stream2.process(new ProcessFunction<JSONObject, JSONObject>() {

            @Override
            public void processElement(JSONObject jsonObject, Context context, Collector<JSONObject> collector)
                    throws Exception {
                String type = jsonObject.getString(TYPE);
                if (type.equals("21")) {
                    context.output(outputTag21, jsonObject);
                } else if (type.equals("22")) {
                    context.output(outputTag22, jsonObject);
                }
            }
        });

        DataStream<JSONObject> stream21 = outputStream2.getSideOutput(outputTag21);
        DataStream<JSONObject> stream22 = outputStream2.getSideOutput(outputTag22);

        //数据去向
        stream1.map(e -> {e.put("stream", "stream1");return e;}).print();
        stream11.map(e -> {e.put("stream", "stream11");return e;}).print();
        stream12.map(e -> {e.put("stream", "stream12");return e;}).print();
        stream2.map(e -> {e.put("stream", "stream2");return e;}).print();
        stream21.map(e -> {e.put("stream", "stream21");return e;}).print();
        stream22.map(e -> {e.put("stream", "stream22");return e;}).print();

        env.execute("SocketStreamTest");
    }
}

输出结果(与结果一致)：

3> {"data":"data1","stream":"stream1","type":"1","key":"001"}
4> {"data":"data11","stream":"stream11","type":"11","key":"001"}
4> {"data":"data11","stream":"stream1","type":"11","key":"001"}
5> {"data":"data12","stream":"stream12","type":"12","key":"001"}
5> {"data":"data12","stream":"stream1","type":"12","key":"001"}
6> {"data":"data2","stream":"stream2","type":"2","key":"002"}
7> {"data":"data21","stream":"stream21","type":"21","key":"002"}
7> {"data":"data21","stream":"stream2","type":"21","key":"002"}
8> {"data":"data22","stream":"stream22","type":"22","key":"002"}
8> {"data":"data22","stream":"stream2","type":"22","key":"002"}

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 212,294评论 6赞 493
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 90,493评论 3赞 385
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 157,790评论 0赞 348
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 56,595评论 1赞 284
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 65,718评论 6赞 386
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 49,906评论 1赞 290
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 39,053评论 3赞 410
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 37,797评论 0赞 268
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 44,250评论 1赞 303
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 36,570评论 2赞 327
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,711评论 1赞 341
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 34,388评论 4赞 332
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 40,018评论 3赞 316
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,796评论 0赞 21
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,023评论 1赞 266
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 46,461评论 2赞 360
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 43,595评论 2赞 350