1.Machine Learning 应用
image.png
2.CoreML 结构图
image.png
Vision: 高性能的图像分析和计算机视觉处理框架
Apply high-performance image analysis and computer vision techniques to identify faces, detect features, and classify scenes in images and video.
e.g.
- 物体追踪
- 面部识别
CroeML 优点
优势
CoreML 支持的模型类型
image.png
应用举例
image.png
3.CoreML 模型获取
可以从Apple machine-learning 页面获得,也可以从三方机构获取模型然后用苹果的 python 工具转换为coreml的模型.
https://developer.apple.com/machine-learning
convert 3rd to mlmodel
4.CoreML workflow
- 拖入 CoreML model
- 勾选 对应的 Target MemberShip, 自动生成需要的模型实体代码
- 同时可以在 Xcode 预览 模型的输入参数/输出参数
image.png
** 创建模型,传入值 得到输出结果 **
Demo1
FlowerClassifiler.mlmodel v3 type
let flowerModel = FlowerClassifier()
func caption(image: CVPixelBuffer) throws -> String {
let prediction = try self.flowerModel.prediction(flowerImage: image)
return prediciton.flowerType
}
Demo1.png
** 拖入训练模型的时候记得勾选target **
Demo2
使用 GoogLeNetPlaces.mlmodel<NNC> 用于图片识别;Detects the scene of an image from 205 categories such as airport, bedroom, forest, coast etc.
代码实现部分
- (NSString *)predictImageScene:(UIImage *)image {
GoogLeNetPlaces *model = [[GoogLeNetPlaces alloc] init];
NSError *error;
UIImage *scaledImage = [image scaleToSize:CGSizeMake(224, 224)];
CVPixelBufferRef buffer = [image pixelBufferFromCGImage:scaledImage];
GoogLeNetPlacesInput *input = [[GoogLeNetPlacesInput alloc] initWithSceneImage:buffer];
GoogLeNetPlacesOutput *output = [model predictionFromFeatures:input error:&error];
return output.sceneLabel;
}
相关 category
1.实现UIImage 和 CVPixelBuffer(Ref)的转换函数[Objective-C]
- (CVPixelBufferRef)pixelBufferFromCGImage:(UIImage *)originImage
- (CVPixelBufferRef)pixelBufferFromCGImage:(UIImage *)originImage {
CGImageRef image = originImage.CGImage;
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
nil];
CVPixelBufferRef pxbuffer = NULL;
CGFloat frameWidth = CGImageGetWidth(image);
CGFloat frameHeight = CGImageGetHeight(image);
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
frameWidth,
frameHeight,
kCVPixelFormatType_32ARGB,
(__bridge CFDictionaryRef) options,
&pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata,
frameWidth,
frameHeight,
8,
CVPixelBufferGetBytesPerRow(pxbuffer),
rgbColorSpace,
(CGBitmapInfo)kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextConcatCTM(context, CGAffineTransformIdentity);
CGContextDrawImage(context, CGRectMake(0,
0,
frameWidth,
frameHeight),
image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
- (UIImage *)scaleToSize:(CGSize)size {
UIGraphicsBeginImageContext(size);
[self drawInRect:CGRectMake(0, 0, size.width, size.height)];
UIImage* scaledImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return scaledImage;
}
swift 辅助代码
fileprivate func CreatePixelBufferFromImage(_ image: UIImage) -> CVPixelBuffer?{
let size = image.size
var pxbuffer : CVPixelBuffer?
let pixelBufferPool = createPixelBufferPool(224, 224, FourCharCode(kCVPixelFormatType_32BGRA), 2056) // Hard coded values for demo purposes.
let status = CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pixelBufferPool!, &pxbuffer)
guard (status == kCVReturnSuccess) else{
return nil
}
CVPixelBufferLockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0))
let pxdata = CVPixelBufferGetBaseAddress(pxbuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: pxdata,
width: Int(size.width),
height: Int(size.height),
bitsPerComponent: 8,
bytesPerRow: CVPixelBufferGetBytesPerRow(pxbuffer!),
space: rgbColorSpace,
bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue)
context?.translateBy(x: 0, y: image.size.height)
context?.scaleBy(x: 1.0, y: -1.0)
UIGraphicsPushContext(context!)
image.draw(in: CGRect(x: 0, y: 0, width: size.width, height: size.height))
UIGraphicsPopContext()
CVPixelBufferUnlockBaseAddress(pxbuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pxbuffer
}
```
fileprivate func createPixelBufferPool(_ width: Int32, _ height: Int32, _ pixelFormat: FourCharCode, _ maxBufferCount: Int32) -> CVPixelBufferPool? {
var outputPool: CVPixelBufferPool? = nil
let sourcePixelBufferOptions: NSDictionary = [kCVPixelBufferPixelFormatTypeKey: pixelFormat,
kCVPixelBufferWidthKey: width,
kCVPixelBufferHeightKey: height,
kCVPixelFormatOpenGLESCompatibility: true,
kCVPixelBufferIOSurfacePropertiesKey: NSDictionary()]
let pixelBufferPoolOptions: NSDictionary = [kCVPixelBufferPoolMinimumBufferCountKey: maxBufferCount]
CVPixelBufferPoolCreate(kCFAllocatorDefault, pixelBufferPoolOptions, sourcePixelBufferOptions, &outputPool)
return outputPool
}
** 简单总结: **
CoreML 就是屌. 利用Xcode 导入训练的模型(mlmodel),勾选 target membership, 根据模型描述传入输入值,得到输出结果. 你可以不懂机器学习,但是调用函数传递参数得到输出值展示 UI 你总会吧.
------
下一部分跟着 sessions 继续补充
1. NLP 的使用 Wednesday 9:00 AM
-------
2. Vision 的使用 Wednesday 3:10 PM
-----
连续追踪
支持图片类型
不需要你缩小 矫正方向
流: CVPixelBuffer CMSampleBuffer 获取文件是 nsurl
3. CoreML 深入使用 Thursday 9:00 AM
4. CoreML模型的转换