Skip to content

Commit ce51398

Browse files
IRONICBoIRONICBo
authored
[ARM plugin] GSoC 2022: Android demo application for ARM CPUs (#382)
* Using a single photo * Fix bugs and update README * Fix links * Move to arm plugin directory * Update to coco demo * Update demo result * Update to Java API 2.0 and fix issues * Update README for jar package * Add different color for COCO classes * Remove Proguard, icon files and gradle configs * Fix log and replace loadLibrary function * Fix format * Fix code style * Fix package * Fix style * Fix return style * Update to loadLibrary Co-authored-by: Bo <[email protected]>
1 parent 13bef9a commit ce51398

File tree

13 files changed

+667
-0
lines changed

13 files changed

+667
-0
lines changed

‎modules/arm_plugin/demos/coco_detection_android_demo/README.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Coco Detection Android Demo
2+
3+
![Running result](https://user-images.githubusercontent.com/47499836/189129594-2634e176-5a5b-4051-b713-ae9574a8c3da.png)
4+
5+
This is a demo for ARM processors Android devices. Using object detection model to reach the coco datasets' infomation. We use pre-trained models from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo): [ssdlite_mobilenet_v2](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssdlite_mobilenet_v2) for object detection in coco dataset, [efficientdet-d0-tf](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/efficientdet-d0-tf), [pelee-coco](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/pelee-coco).
6+
7+
The application reads frames from your device's camera or emulator's camera, and processes the network to detect the coco objects' locations and attributes. Then draw it on the image.
8+
9+
The current openvino engine does not currently support models in INT8 MIXed format for reasoning on ARM devices, but will support models in this format in the near future and will achieve better performance.
10+
11+
Otherwise, there is no difference between FP16 and FP32 for CPU, the plugin will convert it automatically to FP32.
12+
13+
## How to run it
14+
15+
### Build the OpenVINO libraries for Android
16+
17+
Before we run the demo on ARM Android phones, we need to prepare the libraries for ARM plugin and java bingings for OpenVINO. These libraries are built from Ubuntu 18.04, and it
18+
19+
- Set java environment
20+
21+
```bash
22+
sudo apt-get install -y openjdk-8-jdk
23+
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
24+
```
25+
26+
- Create work directory
27+
28+
```bash
29+
mkdir openvino_android
30+
export WORK_DIR="$(pwd)/openvino_android"
31+
cd $WORK_DIR
32+
```
33+
34+
- Clone `OpenVINO` and `OpenVINO Contrib` repositories (Use master branch for Java API 2.0).
35+
36+
```bash
37+
git clone --recurse https://github.com/openvinotoolkit/openvino.git "$WORK_DIR/openvino"
38+
git clone --recurse https://github.com/openvinotoolkit/openvino_contrib.git "$WORK_DIR/openvino_contrib"
39+
```
40+
41+
- Download Android NDK and set environment for it. (If you need proxy, you need to set specific url to XXX, or just remove `--no_https --proxy=http --proxy_host=XXX --proxy_port=XXX`)
42+
43+
```bash
44+
mkdir "$WORK_DIR/android-tools"
45+
wget https://dl.google.com/android/repository/commandlinetools-linux-7583922_latest.zip
46+
unzip commandlinetools-linux-7583922_latest.zip
47+
yes | ./cmdline-tools/bin/sdkmanager --sdk_root="$WORK_DIR/android-tools" --licenses --no_https --proxy=http --proxy_host=XXX --proxy_port=XXX
48+
./cmdline-tools/bin/sdkmanager --sdk_root="$WORK_DIR/android-tools" --install "ndk-bundle" --no_https --proxy=http --proxy_host=XXX --proxy_port=XXX
49+
```
50+
51+
- Build OpenVINO and ARM plugin for ARM64
52+
53+
```bash
54+
mkdir "$WORK_DIR/openvino_build" "$WORK_DIR/openvino_install"
55+
cmake -GNinja \
56+
-DVERBOSE_BUILD=ON \
57+
-DCMAKE_BUILD_TYPE=Release \
58+
-DCMAKE_TOOLCHAIN_FILE="$WORK_DIR/android-tools/ndk-bundle/build/cmake/android.toolchain.cmake" \
59+
-DANDROID_ABI=arm64-v8a \
60+
-DANDROID_STL=c++_shared \
61+
-DANDROID_PLATFORM=29 \
62+
-DENABLE_SAMPLES=ON \
63+
-DENABLE_INTEL_MYRIAD=OFF \
64+
-DENABLE_INTEL_MYRIAD_COMMON=OFF \
65+
-DBUILD_java_api=ON \
66+
-DTHREADING=SEQ \
67+
-DIE_EXTRA_MODULES="$WORK_DIR/openvino_contrib/modules" \
68+
-DARM_COMPUTE_SCONS_JOBS=$(nproc) \
69+
-DCMAKE_INSTALL_PREFIX="$WORK_DIR/openvino_install" \
70+
-B "$WORK_DIR/openvino_build" -S "$WORK_DIR/openvino"
71+
ninja -C "$WORK_DIR/openvino_build"
72+
ninja -C "$WORK_DIR/openvino_build" install
73+
```
74+
75+
The built results are in `$WORK_DIR/openvino_install/runtime/lib/aarch64`. We will use them later.
76+
77+
> Please confirm that your `plugins.xml` in `$WORK_DIR/openvino_install/runtime/lib/aarch64` contains plugin name `"CPU"`.
78+
79+
- Build Java API 2.0 for Android
80+
81+
```bash
82+
source $WORK_DIR/openvino_install/setupvars.sh
83+
cd $WORK_DIR/openvino_contrib/modules/java_api
84+
gradle build
85+
```
86+
87+
- Download and convert model "ssdlite_mobilenet_v2" [or pelee-coco, efficientdet-d0-tf] with Open Model Zoo
88+
89+
```bash
90+
git clone --depth 1 https://github.com/openvinotoolkit/open_model_zoo "$WORK_DIR/open_model_zoo"
91+
cd "$WORK_DIR/open_model_zoo/tools/downloader"
92+
python3 -m pip install -r requirements.in
93+
omz_downloader --name ssdlite_mobilenet_v2 --output_dir $WORK_DIR/open_model_zoo/tools/downloader
94+
omz_converter --name ssdlite_mobilenet_v2 --download_dir $WORK_DIR/open_model_zoo/tools/downloader --precision FP16
95+
```
96+
97+
### Import demo project on Android Studio
98+
99+
In this step, we will import demo project to infer object detection.
100+
101+
- Choose and download [Android Studio](https://developer.android.com/studio) on your PC.
102+
103+
- Clone latest branch of OpenVINO Contrib.
104+
105+
```bash
106+
git clone https://github.com/openvinotoolkit/openvino_contrib.git "$WORK_DIR/demo"
107+
```
108+
109+
- Select "File -> Open", and import demo project in `"$WORK_DIR/demo/openvino_contrib/modules/java_api/demos/coco_detection_android_demo"`.
110+
111+
- Copy libraries and model files to the corresponding folder.
112+
113+
1. Clone `"$WORK_DIR/openvino_contrib/modules/java_api/build/libs/java_api.jar"` to `app/libs` folder, and add it as library.
114+
2. Clone `"$WORK_DIR/openvino_install/runtime/lib/aarch64/*.so"` and `"$WORK_DIR/android-tools/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/arm64-v8a/libc++_shared.so"` to `"app/src/main/jniLibs/arm64-v8a"`
115+
3. Clone `"$WORK_DIR/openvino_install/runtime/lib/aarch64/plugins.xml"` to `"app/src/main/assets"`
116+
4. Copy `"$WORK_DIR/open_model_zoo/tools/downloader/intel/ssdlite_mobilenet_v2/FP16/ssdlite_mobilenet_v2.xml"`, `"$WORK_DIR/open_model_zoo/tools/downloader/intel/ssdlite_mobilenet_v2/FP16/ssdlite_mobilenet_v2.bin"` to `"app/src/main/assets"`
117+
118+
- Add OpenCV dependency to project
119+
120+
1. Download [OpenCV SDK for Android](https://github.com/opencv/opencv/releases/download/4.5.0/opencv-4.5.0-android-sdk.zip) and unpack it.
121+
2. Import OpenCV module: select "File -> New -> ImportModule", and sepcify a path to unpacked SDK and set module name to "ocv".
122+
3. Replace `compileSdkVersion 26`, `targetSdkVersion 26` to `compileSdkVersion 32`, `targetSdkVersion 32` in `"$WORK_DIR/coco_detection_android_demo/ocv/build.gradle"`
123+
124+
- Start a ARM-based Android Emulator.
125+
126+
1. Using `AVD Manager -> Create Virtual Device`, and choose one virtual device.
127+
2. Select a system image with `arm64-v8a`.
128+
129+
- Run it!
130+
131+
> The first time when you run the demo application on your device, your need to grant camera permission. Then run it again.

‎modules/arm_plugin/demos/coco_detection_android_demo/app/build.gradle‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
plugins {
2+
id 'com.android.application'
3+
}
4+
5+
android {
6+
compileSdk 32
7+
8+
defaultConfig {
9+
applicationId "org.intel.openvino"
10+
minSdk 23
11+
targetSdk 32
12+
versionCode 1
13+
versionName "1.0"
14+
15+
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
16+
}
17+
18+
buildTypes {
19+
release {
20+
minifyEnabled false
21+
}
22+
}
23+
compileOptions {
24+
sourceCompatibility JavaVersion.VERSION_1_8
25+
targetCompatibility JavaVersion.VERSION_1_8
26+
}
27+
}
28+
29+
dependencies {
30+
31+
implementation 'androidx.appcompat:appcompat:1.4.2'
32+
implementation 'com.google.android.material:material:1.6.1'
33+
implementation 'androidx.constraintlayout:constraintlayout:2.1.4'
34+
testImplementation 'junit:junit:4.+'
35+
androidTestImplementation 'androidx.test.ext:junit:1.1.3'
36+
androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
37+
38+
implementation "androidx.camera:camera-camera2:1.0.0"
39+
implementation "androidx.camera:camera-lifecycle:1.0.0"
40+
implementation "androidx.camera:camera-view:1.0.0-alpha25"
41+
implementation files('libs/java_api.jar')
42+
}

‎modules/arm_plugin/demos/coco_detection_android_demo/app/libs/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# libs
2+
3+
You need to copy the following files to this folder.
4+
5+
- `~/android_ov/openvino/bin/aarch64/Release/lib/inference_engine_java_api.jar`

‎modules/arm_plugin/demos/coco_detection_android_demo/app/src/main/AndroidManifest.xml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
3+
package="org.intel.openvino">
4+
<uses-permission android:name="android.permission.CAMERA"/>
5+
<!-- Grant the permission to external libraries -->
6+
<application
7+
android:allowBackup="true"
8+
android:label="@string/app_name"
9+
android:supportsRtl="true"
10+
android:extractNativeLibs="true"
11+
android:theme="@style/Theme.CocoDetectionAndroidDemo">
12+
<activity
13+
android:name=".MainActivity"
14+
android:exported="true"
15+
android:screenOrientation="landscape">
16+
<intent-filter>
17+
<action android:name="android.intent.action.MAIN" />
18+
19+
<category android:name="android.intent.category.LAUNCHER" />
20+
</intent-filter>
21+
</activity>
22+
</application>
23+
24+
</manifest>

‎modules/arm_plugin/demos/coco_detection_android_demo/app/src/main/assets/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# assets
2+
3+
You need to copy the following files to this folder.
4+
5+
- `$WORK_DIR/openvino_install/runtime/lib/aarch64/plugins.xml`
6+
- `$WORK_DIR/open_model_zoo/tools/downloader/intel/ssdlite_mobilenet_v2/FP16/ssdlite_mobilenet_v2.xml`
7+
- `$WORK_DIR/open_model_zoo/tools/downloader/intel/ssdlite_mobilenet_v2/FP16/ssdlite_mobilenet_v2.bin`

0 commit comments

Comments
 (0)