Android 视频中抓取图像 - MediaMetadataRetriever

最近在side project中遇到了需要从视频中抓取多张图片的需求。安卓已经提供了从视频获取预览图片的ThumbnailUtils, 但此类不能根据timestamp获取bitmap。

以下记录自己找出的解决方案。

需求

在本地视频根据时间戳(timestamp)抓取bitmap图像。

解决方案

MediaMetadataRetriever.getFrameAtTime()

配合使用这些flag可以达到不同的程度的时间精确度

OPTION_PREVIOUS_SYNC：前一个i-frame
OPTION_NEXT_SYNC：后一个 i-frame
OPTION_CLOSEST_SYNC：最近的i-frame，不管前后
OPTION_CLOSEST：最近的frame，不一定是i-frame。上面三个flag只需要decode一张frame即可（而且是i-frame）。但这个flag需要decode多张frame才能接近输入的timestamp，因此速度会慢些。

值得一提的是此类会用binder IBC给system media service发送请求，因此decoder是在系统服务中进行的，并非在我自己的app进程中。
方法调用时logcat可看到如下log

  918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process
  918 28062 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process
  918  1854 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process
  918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process

具体实现可参考：http://androidxref.com/9.0.0_r3/xref/frameworks/av/media/libmedia/mediametadataretriever.cpp#154

需要注意的坑

如果只看文档就会以为用getFrameAtTime + OPTION_CLOSEST就能非常精准的返回在输入timestamp附近的视频帧，但在不同device上跑过代码才发现其行为其实很不统一。:(

高端机基本都能做到预期效果，返回的bitmap在输入timestamp附近。但低端机型直接无视OPTION_CLOSEST 返回附近I-frame的bitmap，因此有时得到的图像跟输入的timestamp相差甚远。例如输入的是3.3sec，返回的有可能是2.8sec的i-frame图像。

出于好奇，我在低端机上做了个小实验。扫描一个视频文件的video timestamp，再把他们逐一用来调用getFrameAtTime，看看返回的bitmap有没有重复。

很明显的看到高端机基本无重复的bitmap，低端机却有大把大把的重复。猜测应该是厂商为了性能而只做了附近i-frame的解码就返回了。有点坑。：（

如果需要在所有机器上返回精准timestamp的bitmap，恐怕只能自己操作decoder了。估计非常复杂。

此外 API28+ 新增加了MediaMetadataRetriever.getFramesAtIndex() 。这应该是比pts更加稳定的抓取方式，毕竟index是连续的int。但不知低端机型会不会继续坑。


==高端
D/MediaMetadataRetrieverRunner: pts 33375    digest :ym71cu9iO1H94190FWVgeg==
D/MediaMetadataRetrieverRunner: pts 66625    digest :y9hG93MqABw3ILMta/noUg==
D/MediaMetadataRetrieverRunner: pts 100000   digest :0l2CkOYDNyxvZCfCeLuq9A==
D/MediaMetadataRetrieverRunner: pts 133375   digest :OHSVQQOxrO77mh9f7tTorQ==
D/MediaMetadataRetrieverRunner: pts 166625   digest :XZ8kDFxsdjgBkUzKl41dgA==
D/MediaMetadataRetrieverRunner: pts 200000   digest :0/13IPmw1gP9FLdpsJc/Wg==
D/MediaMetadataRetrieverRunner: pts 233375   digest :TeHChk9OWbb8nLbrFphlEg==
D/MediaMetadataRetrieverRunner: pts 266625   digest :GEzE3LtFzG88bT7WtMTbxA==
D/MediaMetadataRetrieverRunner: pts 300000   digest :QjdeoZYwYgNxV144kKLTNw==
D/MediaMetadataRetrieverRunner: pts 333375   digest :0DZCQer3BqMPfSYTJaafMg==
D/MediaMetadataRetrieverRunner: pts 366625   digest :mfjEap/awknngWIeyPAScg==
D/MediaMetadataRetrieverRunner: pts 400000   digest :Q1oSKau7n9boi0vDSFZJFA==
D/MediaMetadataRetrieverRunner: pts 433375   digest :kW0vjMra+boztvH+PXQrlw==
D/MediaMetadataRetrieverRunner: pts 466625   digest :a6C/BpHAcr4BxSDg7Gt4bQ==
D/MediaMetadataRetrieverRunner: pts 500000   digest :odAEN5ESKWrLmXwDu09arA==
D/MediaMetadataRetrieverRunner: pts 533375   digest :+zTTQsil/s6on0EVqFuH/Q==
D/MediaMetadataRetrieverRunner: pts 566625   digest :ZqsNHZd1r12UL7cFHdc9vw==
D/MediaMetadataRetrieverRunner: pts 600000   digest :CgUZWadeCe+S+e28C/4qkA==
D/MediaMetadataRetrieverRunner: pts 633375   digest :hUaJy2jWWa9RU4dR24Om7w==
D/MediaMetadataRetrieverRunner: pts 666625   digest :a5HlUYUJcz3xaQrn/hNsIg==
D/MediaMetadataRetrieverRunner: pts 700000   digest :YF70StAojixZxjk8epZL3w==
D/MediaMetadataRetrieverRunner: pts 733375   digest :rolF61sMxRMSP09ePtUGcQ==
D/MediaMetadataRetrieverRunner: pts 766625   digest :K+qZUFNP5EgzWPscmmnFew==
D/MediaMetadataRetrieverRunner: pts 800000   digest :d8iVGhHf3VpMn+vMnBdmng==
D/MediaMetadataRetrieverRunner: pts 833375   digest :cKREwaki8AJmOuSVa8Zvbw==
D/MediaMetadataRetrieverRunner: pts 866625   digest :b3QXkX+wTE1CuCe79JK7Ww==
D/MediaMetadataRetrieverRunner: pts 900000   digest :atzAyrZNcOf8Ghgf04lftw==
D/MediaMetadataRetrieverRunner: pts 933375   digest :hS9ukOCLMCobHplBeRNdOA==
D/MediaMetadataRetrieverRunner: pts 966625   digest :rZyr6vVt5ae+TiMMVTCRrg==




=====


==低端
D/MediaMetadataRetrieverRunner: pts 1601625      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1635000      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1668375      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1701750      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1735000      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1768375      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1801750      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1835125      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1868500      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1901875      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1935250      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1968625      digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 2002000      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2035375      digest :830odrRgw9UyAhNKeBUWAA==
D/MediaMetadataRetrieverRunner: pts 2068750      digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2102125      digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2135500      digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2168875      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2202250      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2235500      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2268875      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2302250      digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2335625      digest :h5ZNxcxO6WrbYFQPsknjnw==
D/MediaMetadataRetrieverRunner: pts 2369000      digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2402375      digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2435750      digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2469125      digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2502500      digest :2h341j90Scn3kRtA4Fr+IA==
D/MediaMetadataRetrieverRunner: pts 2535875      digest :2h341j90Scn3kRtA4Fr+IA==
D/MediaMetadataRetrieverRunner: pts 2569250      digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2602625      digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2636000      digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2669375      digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2702750      digest :OOdPxTb4/NWbUrC8HDzpng==

相关代码

public class MediaMetadataRetrieverRunner {

  private String TAG = MediaMetadataRetrieverRunner.class.getSimpleName();

  void scanAndDigest(String path) throws Exception {
    List<Long> ptsUsList = VideoPtsScanner.scanPts(path);
    Log.d(TAG, "pts list in micro sec: " + ptsUsList);

    MediaMetadataRetriever retriever = new MediaMetadataRetriever();
    retriever.setDataSource(path);

    for (long ptsUs : ptsUsList) {
      Bitmap bitmap = retriever.getFrameAtTime(ptsUs, MediaMetadataRetriever.OPTION_CLOSEST);

      if (bitmap == null) {
        Log.e(TAG, "got null in pts " + ptsUs);
        continue;
      }

      int size = bitmap.getAllocationByteCount();
      ByteBuffer buffer = ByteBuffer.allocateDirect(size);
      bitmap.copyPixelsToBuffer(buffer);
      buffer.flip();

      String digest = makeString(buffer);
      Log.d(TAG, "pts " + ptsUs + " \t digest :" + digest);
    }

    retriever.release();
  }

  private String makeString(ByteBuffer buffer) throws NoSuchAlgorithmException {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(buffer);
    byte[] digest = md5.digest();
    // covert into base64 string, since it's already provided in android
    return Base64.encodeToString(digest, Base64.DEFAULT);
  }

}

-----
public class VideoPtsScanner {

  public static List<Long> scanPts(String path) throws IOException {
    MediaExtractor mediaExtractor = new MediaExtractor();
    mediaExtractor.setDataSource(path);
    mediaExtractor.selectTrack(getVideoTrack(mediaExtractor));

    List<Long> ret = new ArrayList();
    long pts = -1;
    while ((pts = mediaExtractor.getSampleTime()) >= 0) {
      ret.add(pts);
      mediaExtractor.advance();
    }

    mediaExtractor.release();

    return ret;
  }

  private static int getVideoTrack(MediaExtractor extractor) {
    for (int i = 0; i < extractor.getTrackCount(); i++) {
      MediaFormat format = extractor.getTrackFormat(i);
      if (isVideo(format)) {
        return i;
      }
    }

    return -1;
  }

  private static boolean isVideo(MediaFormat format) {
    return format.getString(MediaFormat.KEY_MIME).toLowerCase().contains("video");
  }
}

Android 视频中抓取图像 - MediaMetadataRetriever

需求

解决方案

需要注意的坑

相关代码

推荐阅读更多精彩内容