Skip to content
This repository was archived by the owner on Jan 24, 2024. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ App模型需要开启摄像头权限。启动App后,点击屏幕,将会出

## 快速安装

如果你想快速地体验PDCamera,可通过扫描以下二维码进行安装。成功识别二维码之后,会自动跳转到安装页面,点击“Install PDCamera”链接,App会自动下载并安装到你的iOS设备上。
如果你想快速地体验PDCamera,可通过扫描以下二维码进行安装。成功识别二维码之后,会自动跳转到安装页面,点击**“Install PDCamera”**链接,App会自动下载并安装到你的iOS设备上。

成功安装App后,你还需要安装如下步骤进一步设置:设置 → 通用 → 设备管理 → Baidu USA llc → 信任“Baidu USA llc”。
成功安装App后,你还需要按照如下步骤设置你的iOS设备:**设置 → 通用 → 设备管理 → Baidu USA llc → 信任“Baidu USA llc”**

<p align="center">
<img src="assets/qr_code_ios.png" width = "30%"/>
Expand All @@ -71,6 +71,8 @@ Github上面只维护了该Demo相关的源码文件和项目配置。用户可

VGG模型的识别精度高,但由于模型较大(104.3MB),需要占用较高的内存(\~800MB),并且识别速度慢(每帧~1.5秒),因此对设备的计算能力要求较高(iPhone6s以上),默认没有添加到项目中。用户也可自行下载[vgg\_ssd\_net.paddle](http://cloud.dlnel.org/filepub/?uuid=1116a5f3-7762-44b5-82bb-9954159cb5d4),添加到项目中,体验其高精度识别效果。

这里,我们使用的是**合并的模型**(merged model)。如何从配置文件(例如`config.py`)和训练得到的参数文件(例如`params_pass_0.tar.gz`)得的**合并的模型**文件,请参考[如何合并模型](../../../deployment/model/merge_config_parameters/README.cn.md)。

### 准备PaddlePaddle库

用户可按照[iOS平台编译指南](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_cn.md),拉取[Paddle](https://github.com/PaddlePaddle/Paddle)最新的源码,编译适用于iOS平台的PaddlePaddle库。在执行`make install`之后,PaddlePaddle库将会安装在`CMAKE_INSTALL_PREFIX`所指定的目录下。该目录包含如下子目录:
Expand Down
155 changes: 88 additions & 67 deletions Demo/iOS/AICamera/README.md
Original file line number Diff line number Diff line change
@@ -1,89 +1,68 @@
# PDCamera iOS Demo with SSD Model

- [Overview](#overview)
- [Pre-trained Models](#pre-trained-models)
- [Demo Screenshot](#demo-screenshot)
- [Fast Installation through QR Code](#fast-installation-through-qr-code)
- [Build from Source Code](#build-from-source-code)
- [Prepare Models](#prepare-models)
- [Prepare PaddlePaddle Inference Library](#prepare-paddlepaddle-inference-library)
- [Directory Tree](#directory-tree)
- [Integrate Paddle C Library to iOS Project](#integrate-paddle-c-library-to-ios-project)

## Overview

This iOS demo shows PaddlePaddle running SSD(Single Shot MultiBox Detector)Object detection on iOS devices locally and offline. It loads a pretrained model with PaddlePaddle and uses camera to capture images and call PaddlePaddle's inference ability to show detected objects to users.

You can look at SSD model architecture [here](https://github.com/PaddlePaddle/models/tree/develop/ssd) and a linux demo [here](https://github.com/PaddlePaddle/Mobile/tree/develop/Demo/linux)

### Pre-trained Models

## Download and run the app
`pascal_mobilenet_300_66` and `vgg_ssd_net` models can classify 20 objects.
`face_mobilenet_160_91` can only classify human's face.

To simply run the demo with iPhone/iPad, scan the QR code below, click "Install PDCamera" in the link and the app will be downloaded in the background.
After installed, go to Settings -> General -> Device Management -> Baidu USA llc -> Trust "Baidu USA llc"
| Model | Dimensions | Accuracy | Size |
| ------------------------ |:----------:| --------:|------:|
| [pascal\_mobilenet\_300\_66.paddle](http://cloud.dlnel.org/filepub/?uuid=39c325d9-b468-4940-ba47-d50c8ec5fd5b) | 300 x 300 | 66% | 23.2MB |
| [vgg\_ssd\_net.paddle](http://cloud.dlnel.org/filepub/?uuid=1116a5f3-7762-44b5-82bb-9954159cb5d4) | 300 x 300 | 71% | 104.3MB |
| [face\_mobilenet\_160\_91.paddle](http://cloud.dlnel.org/filepub/?uuid=038c1dbf-08b3-42a9-b2dc-efccd63859fb) | 160 x 160 | 91% | 18.4MB |

### Demo Screenshot

### QR code link
Simply tap on the screen to toggle settings.

<img src="assets/qr_code_ios.png" width = "20%" />

### Demo screenshot
- Models: Select Pascal MobileNet 300 or Face MobileNet 160, App will exit, need to launch to restart.
- Camera: Toggle Front/Back Camera. App will exit, need to launch to restart.
- Accuracy Threshold: Adjust threshold to filter more/less objects based on probability.
- Time Refresh Rate: Adjust the time to refresh bounding box more/less frequently.

<img src="assets/demo_screenshot.jpg" width = "30%" />
<p align="center">
<img src="assets/demo_main.jpeg" width = "25%" />
<img src="assets/demo_pascal.jpg" width = "25%" />
<img src="assets/demo_face.jpeg" width = "25%" /><br/>
Figure-1
</p>

Detected object will be highlighted as a bounding box with a classified object label and probability.

## Fast Installation through QR Code

## Classifications
`pascal_mobilenet_300_66` and `vgg_ssd_net` models can only classify following 20 objects:

- aeroplane
- bicycle
- background
- boat
- bottle
- bus
- car
- cat
- chair
- cow
- diningtable
- dog
- horse
- motorbike
- person
- pottedplant
- sheep
- sofa
- train
- tvmonitor

`face_mobilenet_160_91` can only classify human's face


## Settings

Simply tap on the screen to toggle settings

- Models: Select Pascal MobileNet 300 or Face MobileNet 160, App will exit, need to launch to restart.
- Camera: Toggle Front/Back Camera. App will exit, need to launch to restart.
- Accuracy Threshold: Adjust threshold to filter more/less objects based on probability
- Time Refresh Rate: Adjust the time to refresh bounding box more/less frequently
To simply run the demo with iPhone/iPad, scan the QR code below, click "Install PDCamera" in the link and the app will be downloaded in the background.
After installed, go to Settings -> General -> Device Management -> Baidu USA llc -> Trust "Baidu USA llc"

<p align="center">
<img src="assets/qr_code_ios.png" width = "30%"/>
</p>

## Development or modify
## Build from Source Code

Use latest XCode for development. This demo requires a camera for object detection, therefore you must use a device (iPhone or iPad) for development and testing. Simulators will not work as they cannot access camera.

For developers, feel free to use this as a reference to start a new project. This demo fully demonstrates how to integrate Paddle C Library to iOS and called from Swift.

Swift cannot directly call C API, in order to have client in Swift work, create Objective-C briding header and a Objective-C++ wrapper (.mm files) to access paddle APIs.


## Integrate Paddle C Library to iOS

-Follow this guide [Build PaddlePaddle for iOS](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_cn.md) to generate paddle libs(include, lib, third_party).
-Create a folder paddle-ios and add to project root. Put the 3 paddle libs folder under paddle-ios.
- Add the `include` directory to **Header Search Paths**
![image](https://user-images.githubusercontent.com/12538138/32491809-b215cf7a-c37d-11e7-87f8-3d45f07bc63e.png)

- Add the `Accelerate.framework` or `veclib.framework` to your project, if your PaddlePaddle is built with `IOS_USE_VECLIB_FOR_BLAS=ON`
- Add the libraries of paddle, `libpaddle_capi_layers.a` and `libpaddle_capi_engine.a`, and all the third party libraries to your project
<img src="https://user-images.githubusercontent.com/12538138/32492222-2ecef414-c37f-11e7-9913-b90fc88be10f.png" width = "30%" />

- Set `-force_load` for `libpaddle_capi_layers.a`
![image](https://user-images.githubusercontent.com/12538138/32492328-8504ebae-c37f-11e7-98b5-41615519fbb3.png)


## Download Models
### Prepare Models

Our models are too large to upload to Github. Create a model folder and add to project root. Download [face_mobilenet_160_91.paddle](http://cloud.dlnel.org/filepub/?uuid=038c1dbf-08b3-42a9-b2dc-efccd63859fb) and [pascal_mobilenet_300_66.paddle](http://cloud.dlnel.org/filepub/?uuid=39c325d9-b468-4940-ba47-d50c8ec5fd5b) to the model folder.

Expand All @@ -92,11 +71,53 @@ Note: Only runs on iPhone6s or above (iPhone 6 or below will crash due to memory
If you want to try it out, download [vgg_ssd_net.paddle](http://cloud.dlnel.org/filepub/?uuid=1116a5f3-7762-44b5-82bb-9954159cb5d4), then go to
XCode target -> Bulid Phases -> Copy Bundle Resources, click '+' to add vgg_ssd_net.paddle

### Prepare PaddlePaddle Inference Library

Follow this guide [Build PaddlePaddle for iOS](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_cn.md) to generate paddle libs(include, lib, third_party).
Create a folder paddle-ios and add to project root. Put the 3 paddle libs folder under paddle-ios.

### Directory Tree

```
$ git clone https://github.com/PaddlePaddle/Mobile.git
$ cd Mobile/Demo/iOS/AICamera
$ tree
.
├── AICamera # sources codes
├── PDCamera.xcodeproj
├── README.md
├── README_cn.md
├── assets
├── models # models
│   ├── face_mobilenet_160_91.paddle
│   ├── pascal_mobilenet_300_66.paddle
│   └── vgg_ssd_net.paddle
└── paddle-ios # PaddlePaddle inference library
├── include
├── lib
│   ├── libpaddle_capi_engine.a
│   ├── libpaddle_capi_layers.a
│   └── libpaddle_capi_whole.a
└── third_party
```

## Integrate Paddle C Library to iOS Project

## Accuracy
- Add the `include` directory to **Header Search Paths**

<p align="center">
<img src="https://user-images.githubusercontent.com/12538138/32491809-b215cf7a-c37d-11e7-87f8-3d45f07bc63e.png" width="90%">
</p>

- Add the `Accelerate.framework` or `veclib.framework` to your project, if your PaddlePaddle is built with `IOS_USE_VECLIB_FOR_BLAS=ON`
- Add the libraries of paddle, `libpaddle_capi_layers.a` and `libpaddle_capi_engine.a`, and all the third party libraries to your project

<p align="center">
<img src="https://user-images.githubusercontent.com/12538138/32492222-2ecef414-c37f-11e7-9913-b90fc88be10f.png" width = "30%">
</p>

- Set `-force_load` for `libpaddle_capi_layers.a`

| Model | Dimensions | Accuracy |
| ------------------------ |:----------:| --------:|
| face_mobilenet_160_91 | 160x160 | 91% |
| pascal_mobilenet_300_66 | 300x300 | 66% |
| vgg_ssd_net | 300x300 | 71% |
<p align="center">
<img src="https://user-images.githubusercontent.com/12538138/32492328-8504ebae-c37f-11e7-98b5-41615519fbb3.png" width="90%">
</p>
47 changes: 47 additions & 0 deletions README.cn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# 移动PaddlePaddle

[![Build Status](https://travis-ci.org/PaddlePaddle/Mobile.svg?branch=develop)](https://travis-ci.org/PaddlePaddle/Mobile)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](http://www.paddlepaddle.org/docs/develop/mobile/README.html)
[![Download](https://img.shields.io/badge/download-latest-brightgreen.svg?style=flat)](https://github.com/PaddlePaddle/Mobile/wiki)
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)

PaddlePaddle支持在移动设备上,使用训练好的模型进行离线推断。这里,我们主要介绍如何在移动设备上部署PaddlePaddle推断库,以及移动设备上可以使用到的一些优化方法。

## 构建PaddlePaddle库
PaddlePaddle可以通过原生编译、交叉编译的方式,构建多种移动平台上的推断库。

- [Android平台编译指南](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_android_cn.md)
- [iOS平台编译指南](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_cn.md)
- [Rapsberry Pi3平台编译指南](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_raspberry_cn.md)
- NVIDIA Driver PX2平台,采用原生编译的方式,可直接依照[PaddlePaddle源码编译指南](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/getstarted/build_and_install/build_from_source_cn.rst)进行编译

## 使用示例

- [命令行示例程序](./benchmark/tool/C/README.cn.md)
- [iOS示例应用:PDCamera](./Demo/iOS/AICamera/README.cn.md)

## 部署优化方法
移动端对接入库的大小通常都有要求,在编译PaddlePaddle库时,用户可以通过设置一些编译选项来进行优化。

- [如何构建最小的PaddlePaddle推断库](./deployment/library/build_for_minimum_size.md)

训练得到的模型,可在不降低或者轻微降低模型推断精度的前提下,进行一些变换,优化移动设备上的内存使用和执行效率。

- [合并网络中的BN层](./deployment/model/merge_batch_normalization/README.md)
- [压缩模型大小的rounding方法](./deployment/model/rounding/README.md)
- [如何合并模型](./deployment/model/merge_config_parameters/README.cn.md)
- INT8量化方法

## 模型压缩
基于PaddlePaddle框架,可以使用模型压缩训练进一步裁剪模型的大小。

- [Pruning稀疏化方法](./model_compression/pruning/README.md)

## 性能数据
我们列出一些移动设备上的性能测试数据,给用户参考和对比。

- [Mobilenet模型性能数据](./benchmark/README.md)
- ENet模型性能数据
- [DepthwiseConvolution优化效果](https://github.com/hedaoyuan/Function/blob/master/src/conv/README.md)

本教程由[PaddlePaddle](https://github.com/PaddlePaddle/Paddle)创作,采用[Apache-2.0 license](LICENSE)许可协议进行许可。
33 changes: 20 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,40 @@
# Mobile

[![Build Status](https://travis-ci.org/PaddlePaddle/Mobile.svg?branch=develop)](https://travis-ci.org/PaddlePaddle/Mobile)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](http://www.paddlepaddle.org/docs/develop/mobile/README.html)
[![Download](https://img.shields.io/badge/download-latest-brightgreen.svg?style=flat)](https://github.com/PaddlePaddle/Mobile/wiki)
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)

Here mainly describes how to deploy PaddlePaddle to the mobile end, as well as some deployment optimization methods and some benchmark.

## How to build PaddlePaddle for mobile
- Build PaddlePaddle for Android [[Chinese](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_android_cn.md)] [[English](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_android_en.md)]
- Build PaddlePaddle for IOS [[Chinese](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_cn.md)] [[English](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_en.md)]
- Build PaddlePaddle for Raspberry Pi3 [[Chinese](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_raspberry_cn.md)] [[English](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_raspberry_en.md)]
- Build PaddlePaddle for PX2
- [How to build PaddlePaddle mobile inference library with minimum size.](./deployment/library/build_for_minimum_size.md)
## Build PaddlePaddle
- [Build PaddlePaddle for Android](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_android_en.md)
- [Build PaddlePaddle for IOS](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_ios_en.md)
- [Build PaddlePaddle for Raspberry Pi3](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/mobile/cross_compiling_for_raspberry_en.md)
- Build PaddlePaddle for NVIDIA Driver PX2

## Demo
- [A command-line inference demo.](./benchmark/tool/C/README.md)
- [iOS demo of AICamera](./Demo/iOS/AICamera/README.md)
- [iOS demo of PDCamera](./Demo/iOS/AICamera/README.md)

## Deployment optimization methods
- [Merge batch normalization before deploying the model to the mobile.](./deployment/model/merge_batch_normalization/README.md)
- [Compress the model before deploying the model to the mobile.](./deployment/model/rounding/README.md)
- [Merge model config and parameter files into one file.](./deployment/model/merge_config_parameters/README.md)
- How to deploy int8 model in mobile inference with PaddlePaddle.
Optimization for the library:

- [How to build PaddlePaddle mobile inference library with minimum size.](./deployment/library/build_for_minimum_size.md)

Optimization for models:

- [Merge batch normalization layers](./deployment/model/merge_batch_normalization/README.md)
- [Compress the model based on rounding](./deployment/model/rounding/README.md)
- [Merge model's config and parameters](./deployment/model/merge_config_parameters/README.md)
- How to deploy int8 model in mobile inference with PaddlePaddle

## Model compression
- [How to use pruning to train smaller model](./model_compression/pruning/)
- [How to use pruning to train smaller model](./model_compression/pruning/README.md)

## PaddlePaddle mobile benchmark
- [Benchmark of Mobilenet](./benchmark/README.md)
- Benchmark of ENet
- [Benchmark of DepthwiseConvolution in PaddlePaddle](https://github.com/hedaoyuan/Function/blob/master/src/conv/README.md)
- [Benchmark of DepthwiseConvolution](https://github.com/hedaoyuan/Function/blob/master/src/conv/README.md)

This tutorial is contributed by [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) and licensed under the [Apache-2.0 license](LICENSE).
Loading