聊聊flink的NetworkBufferPool

本文主要研究一下flink的NetworkBufferPool

BufferPoolFactory

flink-release-1.7.2/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java

public interface BufferPoolFactory {

    /**
     * Tries to create a buffer pool, which is guaranteed to provide at least the number of required
     * buffers.
     *
     * <p>The buffer pool is of dynamic size with at least <tt>numRequiredBuffers</tt> buffers.
     *
     * @param numRequiredBuffers
     *      minimum number of network buffers in this pool
     * @param maxUsedBuffers
     *      maximum number of network buffers this pool offers
     */
    BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException;

    /**
     * Tries to create a buffer pool with an optional owner, which is guaranteed to provide at least the
     * number of required buffers.
     *
     * <p>The buffer pool is of dynamic size with at least <tt>numRequiredBuffers</tt> buffers.
     *
     * @param numRequiredBuffers
     *      minimum number of network buffers in this pool
     * @param maxUsedBuffers
     *      maximum number of network buffers this pool offers
     *  @param owner
     *      the optional owner of this buffer pool to release memory when needed
     */
    BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers, Optional<BufferPoolOwner> owner) throws IOException;

    /**
     * Destroy callback for updating factory book keeping.
     */
    void destroyBufferPool(BufferPool bufferPool) throws IOException;

}
  • BufferPoolFactory定义了createBufferPool、destroyBufferPool方法;其中createBufferPool支持numRequiredBuffers、maxUsedBuffers、owner参数

NetworkBufferPool

flink-release-1.7.2/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java

public class NetworkBufferPool implements BufferPoolFactory {
    //......

    private static final Logger LOG = LoggerFactory.getLogger(NetworkBufferPool.class);

    private final int totalNumberOfMemorySegments;

    private final int memorySegmentSize;

    private final ArrayBlockingQueue<MemorySegment> availableMemorySegments;

    private volatile boolean isDestroyed;

    // ---- Managed buffer pools ----------------------------------------------

    private final Object factoryLock = new Object();

    private final Set<LocalBufferPool> allBufferPools = new HashSet<>();

    private int numTotalRequiredBuffers;

    /**
     * Allocates all {@link MemorySegment} instances managed by this pool.
     */
    public NetworkBufferPool(int numberOfSegmentsToAllocate, int segmentSize) {

        this.totalNumberOfMemorySegments = numberOfSegmentsToAllocate;
        this.memorySegmentSize = segmentSize;

        final long sizeInLong = (long) segmentSize;

        try {
            this.availableMemorySegments = new ArrayBlockingQueue<>(numberOfSegmentsToAllocate);
        }
        catch (OutOfMemoryError err) {
            throw new OutOfMemoryError("Could not allocate buffer queue of length "
                    + numberOfSegmentsToAllocate + " - " + err.getMessage());
        }

        try {
            for (int i = 0; i < numberOfSegmentsToAllocate; i++) {
                availableMemorySegments.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(segmentSize, null));
            }
        }
        catch (OutOfMemoryError err) {
            int allocated = availableMemorySegments.size();

            // free some memory
            availableMemorySegments.clear();

            long requiredMb = (sizeInLong * numberOfSegmentsToAllocate) >> 20;
            long allocatedMb = (sizeInLong * allocated) >> 20;
            long missingMb = requiredMb - allocatedMb;

            throw new OutOfMemoryError("Could not allocate enough memory segments for NetworkBufferPool " +
                    "(required (Mb): " + requiredMb +
                    ", allocated (Mb): " + allocatedMb +
                    ", missing (Mb): " + missingMb + "). Cause: " + err.getMessage());
        }

        long allocatedMb = (sizeInLong * availableMemorySegments.size()) >> 20;

        LOG.info("Allocated {} MB for network buffer pool (number of memory segments: {}, bytes per segment: {}).",
                allocatedMb, availableMemorySegments.size(), segmentSize);
    }

    @Override
    public BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException {
        return createBufferPool(numRequiredBuffers, maxUsedBuffers, Optional.empty());
    }

    @Override
    public BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers, Optional<BufferPoolOwner> owner) throws IOException {
        // It is necessary to use a separate lock from the one used for buffer
        // requests to ensure deadlock freedom for failure cases.
        synchronized (factoryLock) {
            if (isDestroyed) {
                throw new IllegalStateException("Network buffer pool has already been destroyed.");
            }

            // Ensure that the number of required buffers can be satisfied.
            // With dynamic memory management this should become obsolete.
            if (numTotalRequiredBuffers + numRequiredBuffers > totalNumberOfMemorySegments) {
                throw new IOException(String.format("Insufficient number of network buffers: " +
                                "required %d, but only %d available. The total number of network " +
                                "buffers is currently set to %d of %d bytes each. You can increase this " +
                                "number by setting the configuration keys '%s', '%s', and '%s'.",
                        numRequiredBuffers,
                        totalNumberOfMemorySegments - numTotalRequiredBuffers,
                        totalNumberOfMemorySegments,
                        memorySegmentSize,
                        TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION.key(),
                        TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN.key(),
                        TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX.key()));
            }

            this.numTotalRequiredBuffers += numRequiredBuffers;

            // We are good to go, create a new buffer pool and redistribute
            // non-fixed size buffers.
            LocalBufferPool localBufferPool =
                new LocalBufferPool(this, numRequiredBuffers, maxUsedBuffers, owner);

            allBufferPools.add(localBufferPool);

            try {
                redistributeBuffers();
            } catch (IOException e) {
                try {
                    destroyBufferPool(localBufferPool);
                } catch (IOException inner) {
                    e.addSuppressed(inner);
                }
                ExceptionUtils.rethrowIOException(e);
            }

            return localBufferPool;
        }
    }

    @Override
    public void destroyBufferPool(BufferPool bufferPool) throws IOException {
        if (!(bufferPool instanceof LocalBufferPool)) {
            throw new IllegalArgumentException("bufferPool is no LocalBufferPool");
        }

        synchronized (factoryLock) {
            if (allBufferPools.remove(bufferPool)) {
                numTotalRequiredBuffers -= bufferPool.getNumberOfRequiredMemorySegments();

                redistributeBuffers();
            }
        }
    }

    //......
}
  • NetworkBufferPool实现了BufferPoolFactory接口,它的构造器接收numberOfSegmentsToAllocate、segmentSize两个参数;构造器里头根据numberOfSegmentsToAllocate创建了availableMemorySegments这个ArrayBlockingQueue,然后通过MemorySegmentFactory.allocateUnpooledOffHeapMemory挨个创建MemorySegment添加到availableMemorySegments
  • createBufferPool方法创建的是LocalBufferPool(传递了自身NetworkBufferPool实例进去),然后添加到allBufferPools这个set中,同时增加numTotalRequiredBuffers;destroyBufferPool方法则从allBufferPools移除该bufferPool,同时减少numTotalRequiredBuffers
  • createBufferPool方法及destroyBufferPool方法会调用到redistributeBuffers方法,通过调用LocalBufferPool的setNumBuffers方法来调整buffer pool的大小

LocalBufferPool

flink-release-1.7.2/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java

class LocalBufferPool implements BufferPool {
    private static final Logger LOG = LoggerFactory.getLogger(LocalBufferPool.class);

    /** Global network buffer pool to get buffers from. */
    private final NetworkBufferPool networkBufferPool;

    /** The minimum number of required segments for this pool. */
    private final int numberOfRequiredMemorySegments;

    /**
     * The currently available memory segments. These are segments, which have been requested from
     * the network buffer pool and are currently not handed out as Buffer instances.
     *
     * <p><strong>BEWARE:</strong> Take special care with the interactions between this lock and
     * locks acquired before entering this class vs. locks being acquired during calls to external
     * code inside this class, e.g. with
     * {@link org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel#bufferQueue}
     * via the {@link #registeredListeners} callback.
     */
    private final ArrayDeque<MemorySegment> availableMemorySegments = new ArrayDeque<MemorySegment>();

    /**
     * Buffer availability listeners, which need to be notified when a Buffer becomes available.
     * Listeners can only be registered at a time/state where no Buffer instance was available.
     */
    private final ArrayDeque<BufferListener> registeredListeners = new ArrayDeque<>();

    /** Maximum number of network buffers to allocate. */
    private final int maxNumberOfMemorySegments;

    /** The current size of this pool. */
    private int currentPoolSize;

    /**
     * Number of all memory segments, which have been requested from the network buffer pool and are
     * somehow referenced through this pool (e.g. wrapped in Buffer instances or as available segments).
     */
    private int numberOfRequestedMemorySegments;

    private boolean isDestroyed;

    private final Optional<BufferPoolOwner> owner;

    /**
     * Local buffer pool based on the given <tt>networkBufferPool</tt> with a minimal number of
     * network buffers being available.
     *
     * @param networkBufferPool
     *      global network buffer pool to get buffers from
     * @param numberOfRequiredMemorySegments
     *      minimum number of network buffers
     */
    LocalBufferPool(NetworkBufferPool networkBufferPool, int numberOfRequiredMemorySegments) {
        this(networkBufferPool, numberOfRequiredMemorySegments, Integer.MAX_VALUE, Optional.empty());
    }

    /**
     * Local buffer pool based on the given <tt>networkBufferPool</tt> with a minimal and maximal
     * number of network buffers being available.
     *
     * @param networkBufferPool
     *      global network buffer pool to get buffers from
     * @param numberOfRequiredMemorySegments
     *      minimum number of network buffers
     * @param maxNumberOfMemorySegments
     *      maximum number of network buffers to allocate
     */
    LocalBufferPool(NetworkBufferPool networkBufferPool, int numberOfRequiredMemorySegments,
            int maxNumberOfMemorySegments) {
        this(networkBufferPool, numberOfRequiredMemorySegments, maxNumberOfMemorySegments, Optional.empty());
    }

    /**
     * Local buffer pool based on the given <tt>networkBufferPool</tt> and <tt>bufferPoolOwner</tt>
     * with a minimal and maximal number of network buffers being available.
     *
     * @param networkBufferPool
     *      global network buffer pool to get buffers from
     * @param numberOfRequiredMemorySegments
     *      minimum number of network buffers
     * @param maxNumberOfMemorySegments
     *      maximum number of network buffers to allocate
     *  @param owner
     *      the optional owner of this buffer pool to release memory when needed
     */
    LocalBufferPool(
        NetworkBufferPool networkBufferPool,
        int numberOfRequiredMemorySegments,
        int maxNumberOfMemorySegments,
        Optional<BufferPoolOwner> owner) {
        checkArgument(maxNumberOfMemorySegments >= numberOfRequiredMemorySegments,
            "Maximum number of memory segments (%s) should not be smaller than minimum (%s).",
            maxNumberOfMemorySegments, numberOfRequiredMemorySegments);

        checkArgument(maxNumberOfMemorySegments > 0,
            "Maximum number of memory segments (%s) should be larger than 0.",
            maxNumberOfMemorySegments);

        LOG.debug("Using a local buffer pool with {}-{} buffers",
            numberOfRequiredMemorySegments, maxNumberOfMemorySegments);

        this.networkBufferPool = networkBufferPool;
        this.numberOfRequiredMemorySegments = numberOfRequiredMemorySegments;
        this.currentPoolSize = numberOfRequiredMemorySegments;
        this.maxNumberOfMemorySegments = maxNumberOfMemorySegments;
        this.owner = owner;
    }

    //......

    @Override
    public void recycle(MemorySegment segment) {
        BufferListener listener;
        NotificationResult notificationResult = NotificationResult.BUFFER_NOT_USED;
        while (!notificationResult.isBufferUsed()) {
            synchronized (availableMemorySegments) {
                if (isDestroyed || numberOfRequestedMemorySegments > currentPoolSize) {
                    returnMemorySegment(segment);
                    return;
                } else {
                    listener = registeredListeners.poll();
                    if (listener == null) {
                        availableMemorySegments.add(segment);
                        availableMemorySegments.notify();
                        return;
                    }
                }
            }
            notificationResult = fireBufferAvailableNotification(listener, segment);
        }
    }

    private MemorySegment requestMemorySegment(boolean isBlocking) throws InterruptedException, IOException {
        synchronized (availableMemorySegments) {
            returnExcessMemorySegments();

            boolean askToRecycle = owner.isPresent();

            // fill availableMemorySegments with at least one element, wait if required
            while (availableMemorySegments.isEmpty()) {
                if (isDestroyed) {
                    throw new IllegalStateException("Buffer pool is destroyed.");
                }

                if (numberOfRequestedMemorySegments < currentPoolSize) {
                    final MemorySegment segment = networkBufferPool.requestMemorySegment();

                    if (segment != null) {
                        numberOfRequestedMemorySegments++;
                        return segment;
                    }
                }

                if (askToRecycle) {
                    owner.get().releaseMemory(1);
                }

                if (isBlocking) {
                    availableMemorySegments.wait(2000);
                }
                else {
                    return null;
                }
            }

            return availableMemorySegments.poll();
        }
    }

    //......
}
  • LocalBufferPool的构造器要求传入NetworkBufferPool,而其内部的requestMemorySegment方法,在availableMemorySegments为空且numberOfRequestedMemorySegments < currentPoolSize时,会调用networkBufferPool.requestMemorySegment()来申请MemorySegment;而recycle方法在numberOfRequestedMemorySegments > currentPoolSize时会归还MemorySegment到networkBufferPool,否则在BufferListener为null的时候会归还到availableMemorySegments

小结

  • BufferPoolFactory定义了createBufferPool、destroyBufferPool方法;其中createBufferPool支持numRequiredBuffers、maxUsedBuffers、owner参数;NetworkBufferPool实现了BufferPoolFactory接口,它的构造器接收numberOfSegmentsToAllocate、segmentSize两个参数;构造器里头根据numberOfSegmentsToAllocate创建了availableMemorySegments这个ArrayBlockingQueue,然后通过MemorySegmentFactory.allocateUnpooledOffHeapMemory挨个创建MemorySegment添加到availableMemorySegments
  • NetworkBufferPool的createBufferPool方法创建的是LocalBufferPool(传递了自身NetworkBufferPool实例进去),然后添加到allBufferPools这个set中,同时增加numTotalRequiredBuffers;destroyBufferPool方法则从allBufferPools移除该bufferPool,同时减少numTotalRequiredBuffers;createBufferPool方法及destroyBufferPool方法会调用到redistributeBuffers方法,通过调用LocalBufferPool的setNumBuffers方法来调整buffer pool的大小
  • LocalBufferPool的构造器要求传入NetworkBufferPool,而其内部的requestMemorySegment方法,在availableMemorySegments为空且numberOfRequestedMemorySegments < currentPoolSize时,会调用networkBufferPool.requestMemorySegment()来申请MemorySegment;而recycle方法在numberOfRequestedMemorySegments > currentPoolSize时会归还MemorySegment到networkBufferPool,否则在BufferListener为null的时候会归还到availableMemorySegments

doc

©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 210,978评论 6 490
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 89,954评论 2 384
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 156,623评论 0 345
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 56,324评论 1 282
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 65,390评论 5 384
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 49,741评论 1 289
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 38,892评论 3 405
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 37,655评论 0 266
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 44,104评论 1 303
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 36,451评论 2 325
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,569评论 1 340
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 34,254评论 4 328
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 39,834评论 3 312
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,725评论 0 21
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,950评论 1 264
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 46,260评论 2 360
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 43,446评论 2 348

推荐阅读更多精彩内容

  • 小芳: 你好. 你外婆总说我不像个舅舅,从没有做过一件舅舅该做的事,这让我很是困惑,我根本不知道,在常人心中的舅舅...
    慕容逸阅读 698评论 2 8
  • 芒格的多学科思维早几年前就接触了,昨天又翻看了格栅模型,或者叫多学科思维模型,结合以用为学的思想,我自问为什么不试...
    Y先生说阅读 346评论 0 0
  • 上午,在看公众号的时候,看到军报记者发布的那篇“军营以什么样的姿态迎接文职人员”时,心中很是羡慕。 ...
    小女子仗剑走天涯阅读 785评论 0 2
  • 只有活给自己看的人生,才能够剥离掉虚荣心、表演欲、自我感动的外壳,露出一点赤胆忠心。---题记 每年岁尾,都想写篇...
    落落_14c6阅读 256评论 0 1