CMSampleBuffer的使用

• 通过摄像头采样的视频数据。
• 通过mic采集的音频信息。

- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;

• AVAssetWriter保存视频AVAssetWriterInput

- (BOOL)appendSampleBuffer:(CMSampleBufferRef)sampleBuffer;

• 读取视频文件的输出流AVAssetReaderOutput

- (nullable CMSampleBufferRef)copyNextSampleBuffer CF_RETURNS_RETAINED;

• AVSampleBufferDisplayLayer 直接显示解码后的sampleBuffer

- (void)enqueueSampleBuffer:(CMSampleBufferRef)sampleBuffer;

• ARKit ARSessionObserver输出捕获的audioSampleBuffer

/**
 This is called when the session outputs a new audio sample buffer.
 @param session The session being run.
 @param audioSampleBuffer The captured audio sample buffer.
 */
- (void)session:(ARSession *)session didOutputAudioSampleBuffer:(CMSampleBufferRef)audioSampleBuffer;

• VTCompressionSession硬件编码作为VTCompressionOutputCallback输出

typedef void (*VTCompressionOutputCallback)(
        void * CM_NULLABLE outputCallbackRefCon,
        void * CM_NULLABLE sourceFrameRefCon, 
        OSStatus status, 
        VTEncodeInfoFlags infoFlags,
        CM_NULLABLE CMSampleBufferRef sampleBuffer );

创建压缩session

注意：

Clients of CMSampleBuffer must explicitly manage the retain count by calling CFRetain and CFRelease, even in processes using garbage collection.

CMSampleBuffer 数据结构

• CMSampleBufferRef 这是一个包含零个或多个解码后（未解码）特定媒体类型的样本（音频，视频，多路复用等）。
  API注释

A reference to a CMSampleBuffer, a CF object containing zero or more compressed (or uncompressed) samples of a particular media type (audio, video, muxed, etc).

• CaptureOutput输出video CMSampleBuffer,编码前内容

(lldb) po sampleBuffer
CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    buffer-level attachments:
        Orientation(P) = 1
        {Exif}   (P)  {//省略}

        DPIWidth  (P) = 72
        {TIFF}    (P) {//省略}

        DPIHeight (P) = 72
    formatDescription = {
    mediaType:'vide' 
    mediaSubType:'420f' 
    mediaSpecific: {
        codecType: '420f'       dimensions: 480 x 640 
    } 
    extensions: {{type = immutable dict, count = 6,
entries =>
    0 :{contents = "CVImageBufferYCbCrMatrix"} = {contents = "ITU_R_601_4"}
    1 : {contents = "CVImageBufferTransferFunction"} ={contents = "ITU_R_709_2"}
    2 : {contents = "CVBytesPerRow"} ={value = +724, type = kCFNumberSInt32Type}
    3 :{contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
    5 : {contents = "CVImageBufferColorPrimaries"} = {contents = "ITU_R_709_2"}
    6 : {contents = "CVImageBufferChromaLocationTopField"} = {contents = "Center"}
}
}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {6510691197875/1000000000 = 6510.691}, DTS = {INVALID}, duration = {INVALID}},
    }
    imageBuffer = 0x1c4134c80

(lldb) 

• 硬编码后的数据,相比编码前多了sampleAttachmentsArray，颜色空间是'avc1'

CMSampleBuffer 0x11be1fda0 retainCount: 
  1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription =  {
    mediaType:'vide' 
    mediaSubType:'avc1' 
    mediaSpecific: {
        codecType: 'avc1'       dimensions: 480 x 640 
    } 
  2 : {contents = "FormatName"} = H.264
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {65/1000 = 0.065}, DTS = {64/1000 = 0.064}, duration = {INVALID}},
    }
    sampleSizeArray[1] = {
        sampleSize = 212,
    }
    sampleAttachmentsArray[1] = {
        sample 0:
            NotSync    = true
            DependsOnOthers = true
    }
    dataBuffer = 0x1c410ac20

• CaptureOutput输出音频，buffer内容

CMSampleBuffer 0x125e41940 retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription =  {
    mediaType:'soun' 
    mediaSubType:'lpcm' 
    mediaSpecific: {
        ASBD: {
            mSampleRate: 44100.000000 
            mFormatID: 'lpcm' 
            mFormatFlags: 0xc 
            mBytesPerPacket: 2 
            mFramesPerPacket: 1 
            mBytesPerFrame: 2 
            mChannelsPerFrame: 1 
            mBitsPerChannel: 16     } 
        cookie: {(null)} 
        ACL: {(null)}
        FormatList Array: {(null)} 
    } 
    extensions: {(null)}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1024
    sampleTimingArray[1] = {
        {PTS = {10713597355917/1000000000 = 10713.597, rounded}, DTS = {INVALID}, duration = {1/44100 = 0.000}},
    }
    sampleSizeArray[1] = {
        sampleSize = 2,
    }
    dataBuffer = 0x1c4304650

• AVAssetReaderOutput读取MP4文件sampleBuffer内容

CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription = {

    mediaType:'vide' 
    mediaSubType:'BGRA' 
    mediaSpecific: {
        codecType: 'BGRA'   dimensions: 640 x 360 
    } 
    extensions: {{type = immutable dict, count = 9,
entries =>
    0 : {contents = "CVFieldCount"} = {value = +1, type = kCFNumberSInt32Type}
    1 : {contents = "CVImageBufferYCbCrMatrix"} = {contents = "YCbCrMatrix#5"}
    2 : {contents = "CVBytesPerRow"} ={value = +2560, type = kCFNumberSInt32Type}
    3 : {contents = "CVPixelAspectRatio"} = {type = immutable dict, count = 2,
entries =>
    1 : {contents = "HorizontalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
    2 : {contents = "VerticalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
}
    4 : {contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
    5 : {contents = "CVImageBufferColorPrimaries"} = SMPTE_C
    8 : {contents = "CVImageBufferTransferFunction"} = {contents = "TransferFunction#5"}
    10 : {contents = "CVImageBufferChromaLocationBottomField"} = Left
    12 : {contents = "CVImageBufferChromaLocationTopField"} = Left
}
}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {0/600 = 0.000}, DTS = {INVALID}, duration = {INVALID}},
    }
    imageBuffer = 0x1c4139280

• 以上可以看出未编码\解码后的CMSampleBuffer都包含了imageBuffer数据、formatDescription信息和sampleTimingArray信息。录像多了exif和tiff信息。

• CVPixelBuffer：编码前和解码后的图像数据结构.

• 无论是读取还是录制
  output的settings的kCVPixelBufferPixelFormatTypeKey字段 控制的mediaSubType输出的色彩空间，转换色彩空间过程有GPU参与，性能较高。可以按需设置。

CMSampleBuffer的创建及示例

- (void)appendVideoPixelBuffer:(CVPixelBufferRef)pixelBuffer withPresentationTime:(CMTime)presentationTime
{
    CMSampleBufferRef sampleBuffer = NULL;
    CMFormatDescriptionRef outputFormatDescription = NULL;
    CMVideoFormatDescriptionCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, &outputFormatDescription );


    CMSampleTimingInfo timingInfo = {0,};
    timingInfo.duration = kCMTimeInvalid;
    timingInfo.decodeTimeStamp = kCMTimeInvalid;
    timingInfo.presentationTimeStamp = presentationTime;

    OSStatus err = CMSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, true, NULL, NULL, outputFormatDescription, &timingInfo, &sampleBuffer );
    if ( sampleBuffer ) {
        // do some thing
        CFRelease( sampleBuffer );
    }
    else {
        NSString *exceptionReason = [NSString stringWithFormat:@"sample buffer create failed (%i)", (int)err];
        @throw [NSException exceptionWithName:NSInvalidArgumentException reason:exceptionReason userInfo:nil];
    }
}

• CMSampleBufferCreateReady与CMSampleBufferCreate相同，只是dataReady始终为true，因此不需要传递makeDataReadyCallback或refcon。