Skip to content

Commit 5e07647

Browse files
Ezra-YuEzra-Yu
authored
[Enhance] Reproduce mobileone training accuracy. (open-mmlab#1191)
* add switch hook and UTs * update doc * update doc * fix lint * fix ci * fix ci * fix typo * fix ci * update configs names * update configs * update configs * update links * update readme * update vis_scheduler * update metafile * update configs * rebase * fix ci * rebase
1 parent 49b13cf commit 5e07647

23 files changed

+459
-226
lines changed

‎configs/mobileone/mobileone-s0_8xb128_in1k.py renamed to ‎configs/_base_/schedules/imagenet_bs256_coslr_coswd_300e.py

Lines changed: 3 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,6 @@
1-
_base_ = [
2-
'../_base_/models/mobileone/mobileone_s0.py',
3-
'../_base_/datasets/imagenet_bs32_pil_resize.py',
4-
'../_base_/default_runtime.py'
5-
]
6-
7-
# dataset settings
8-
train_dataloader = dict(batch_size=128)
9-
val_dataloader = dict(batch_size=128)
10-
test_dataloader = dict(batch_size=128)
11-
12-
# schedule settings
1+
# optimizer
132
optim_wrapper = dict(
14-
optimizer=dict(type='SGD', lr=0.1, momentum=0.9, weight_decay=0.0001),
15-
paramwise_cfg=dict(bias_decay_mult=0., norm_decay_mult=0.),
16-
)
3+
optimizer=dict(type='SGD', lr=0.1, momentum=0.9, weight_decay=0.0001))
174

185
# learning policy
196
param_scheduler = [
@@ -50,7 +37,4 @@
5037

5138
# NOTE: `auto_scale_lr` is for automatically scaling LR,
5239
# based on the actual training batch size.
53-
auto_scale_lr = dict(base_batch_size=1024)
54-
55-
# runtime setting
56-
custom_hooks = [dict(type='EMAHook', momentum=5e-4, priority='ABOVE_NORMAL')]
40+
auto_scale_lr = dict(base_batch_size=256)

‎configs/mobileone/README.md

Lines changed: 116 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -4,122 +4,163 @@
44
55
<!-- [ALGORITHM] -->
66

7-
## Abstract
7+
## Introduction
88

9-
Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference time under 1 ms on an iPhone12 with 75.9% top-1 accuracy on ImageNet. We show that MobileOne achieves state-of-the-art performance within the efficient architectures while being many times faster on mobile. Our best model obtains similar performance on ImageNet as MobileFormer while being 38x faster. Our model obtains 2.3% better top-1 accuracy on ImageNet than EfficientNet at similar latency. Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device.
9+
Mobileone is proposed by apple and based on reparameterization. On the apple chips, the accuracy of the model is close to 0.76 on the ImageNet dataset when the latency is less than 1ms. Its main improvements based on [RepVGG](../repvgg) are fllowing:
10+
11+
- Reparameterization using Depthwise convolution and Pointwise convolution instead of normal convolution.
12+
- Removal of the residual structure which is not friendly to access memory.
1013

1114
<div align=center>
1215
<img src="https://user-images.githubusercontent.com/18586273/183552452-74657532-f461-48f7-9aa7-c23f006cdb07.png" width="40%"/>
1316
</div>
1417

15-
## Results and models
18+
## Abstract
1619

17-
### ImageNet-1k
20+
<details>
1821

19-
| Model | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) | Config | Download |
20-
| :------------: | :-----------------------------: | :----------------------------: | :-------: | :-------: | :--------------------------------------------------: | :-----------------------------------------------------: |
21-
| MobileOne-s0\* | 5.29（train) \| 2.08 (deploy) | 1.09 (train) \| 0.28 (deploy) | 71.36 | 89.87 | [config (train)](./mobileone-s0_8xb128_in1k.py) \| [config (deploy)](./deploy/mobileone-s0_deploy_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_3rdparty_in1k_20220915-007ae971.pth) |
22-
| MobileOne-s1\* | 4.83 (train) \| 4.76 (deploy) | 0.86 (train) \| 0.84 (deploy) | 75.76 | 92.77 | [config (train)](./mobileone-s1_8xb128_in1k.py) \| [config (deploy)](./deploy/mobileone-s1_deploy_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s1_3rdparty_in1k_20220915-473c8469.pth) |
23-
| MobileOne-s2\* | 7.88 (train) \| 7.88 (deploy) | 1.34 (train) \| 1.31 (deploy) | 77.39 | 93.63 | [config (train)](./mobileone-s2_8xb128_in1k.py) \|[config (deploy)](./deploy/mobileone-s2_deploy_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s2_3rdparty_in1k_20220915-ed2e4c30.pth) |
24-
| MobileOne-s3\* | 10.17 (train) \| 10.08 (deploy) | 1.95 (train) \| 1.91 (deploy) | 77.93 | 93.89 | [config (train)](./mobileone-s3_8xb128_in1k.py) \|[config (deploy)](./deploy/mobileone-s3_deploy_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s3_3rdparty_in1k_20220915-84d6a02c.pth) |
25-
| MobileOne-s4\* | 14.95 (train) \| 14.84 (deploy) | 3.05 (train) \| 3.00 (deploy) | 79.30 | 94.37 | [config (train)](./mobileone-s4_8xb128_in1k.py) \|[config (deploy)](./deploy/mobileone-s4_deploy_8xb128_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s4_3rdparty_in1k_20220915-ce9509ee.pth) |
22+
<summary>Show the paper's abstract</summary>
2623

27-
*Models with * are converted from the [official repo](https://github.com/apple/ml-mobileone). The config files of these models are only for validation. We don't ensure these config files' training accuracy and welcome you to contribute your reproduction results.*
24+
<br>
25+
Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference time under 1 ms on an iPhone12 with 75.9% top-1 accuracy on ImageNet. We show that MobileOne achieves state-of-the-art performance within the efficient architectures while being many times faster on mobile. Our best model obtains similar performance on ImageNet as MobileFormer while being 38x faster. Our model obtains 2.3% better top-1 accuracy on ImageNet than EfficientNet at similar latency. Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device.
26+
</br>
2827

29-
*Because the [official repo.](https://github.com/apple/ml-mobileone) does not give a strategy for training and testing, the test data pipline of [RepVGG](https://github.com/open-mmlab/mmclassification/tree/master/configs/repvgg) is used here, and the result is about 0.1 lower than that in the paper. Refer to [this issue](https://github.com/apple/ml-mobileone/issues/2).*
28+
</details>
3029

3130
## How to use
3231

33-
The checkpoints provided are all `training-time` models. Use the reparameterize tool to switch them to more efficient `inference-time` architecture, which not only has fewer parameters but also less calculations.
32+
The checkpoints provided are all `training-time` models. Use the reparameterize tool or `switch_to_deploy` interface to switch them to more efficient `inference-time` architecture, which not only has fewer parameters but also less calculations.
3433

35-
### Use tool
34+
<!-- [TABS-BEGIN] -->
3635

37-
Use provided tool to reparameterize the given model and save the checkpoint:
36+
**Predict image**
3837

39-
```bash
40-
python tools/convert_models/reparameterize_model.py ${CFG_PATH} ${SRC_CKPT_PATH} ${TARGET_CKPT_PATH}
38+
Use `classifier.backbone.switch_to_deploy()` interface to switch the MobileOne to a inference mode.
39+
40+
```python
41+
>>> import torch
42+
>>> from mmcls.apis import init_model, inference_model
43+
>>>
44+
>>> model = init_model('configs/mobileone/mobileone-s0_8xb32_in1k.py', 'https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth')
45+
>>> predict = inference_model(model, 'demo/demo.JPEG')
46+
>>> print(predict['pred_class'])
47+
sea snake
48+
>>> print(predict['pred_score'])
49+
0.4539405107498169
50+
>>>
51+
>>> # switch to deploy mode
52+
>>> model.backbone.switch_to_deploy()
53+
>>> predict_deploy = inference_model(model, 'demo/demo.JPEG')
54+
>>> print(predict_deploy['pred_class'])
55+
sea snake
56+
>>> print(predict_deploy['pred_score'])
57+
0.4539395272731781
4158
```
4259

43-
`${CFG_PATH}` is the config file path, `${SRC_CKPT_PATH}` is the source chenpoint file path, `${TARGET_CKPT_PATH}` is the target deploy weight file path.
60+
**Use the model**
4461

45-
For example:
62+
```python
63+
>>> import torch
64+
>>> from mmcls.apis import init_model
65+
>>>
66+
>>> model = init_model('configs/mobileone/mobileone-s0_8xb32_in1k.py', 'https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth')
67+
>>> inputs = torch.rand(1, 3, 224, 224).to(model.data_preprocessor.device)
68+
>>> # To get classification scores.
69+
>>> out = model(inputs)
70+
>>> print(out.shape)
71+
torch.Size([1, 1000])
72+
>>> # To extract features.
73+
>>> outs = model.extract_feat(inputs)
74+
>>> print(outs[0].shape)
75+
torch.Size([1, 768])
76+
>>>
77+
>>> # switch to deploy mode
78+
>>> model.backbone.switch_to_deploy()
79+
>>> out_deploy = model(inputs)
80+
>>> print(out.shape)
81+
torch.Size([1, 1000])
82+
>>> assert torch.allclose(out, out_deploy) # pass without error
83+
```
84+
85+
**Train/Test Command**
86+
87+
Place the ImageNet dataset to the `data/imagenet/` directory, or prepare datasets according to the [docs](https://mmclassification.readthedocs.io/en/1.x/user_guides/dataset_prepare.html#prepare-dataset).
88+
89+
Train:
4690

4791
```shell
48-
python ./tools/convert_models/reparameterize_model.py ./configs/mobileone/mobileone-s0_8xb128_in1k.py https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_3rdparty_in1k_20220811-db5ce29b.pth ./mobileone_s0_deploy.pth
92+
python tools/train.py configs/mobileone/mobileone-s0_8xb32_in1k.py
4993
```
5094

51-
To use reparameterized weights, the config file must switch to **the deploy config files**.
95+
Download Checkpoint:
5296

53-
```bash
54-
python tools/test.py ${Deploy_CFG} ${Deploy_Checkpoint} --metrics accuracy
97+
```shell
98+
wget https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth
5599
```
56100

57-
For example of using the reparameterized weights above:
101+
Test use unfused model:
58102

59103
```shell
60-
python ./tools/test.py ./configs/mobileone/deploy/mobileone-s0_deploy_8xb128_in1k.py mobileone_s0_deploy.pth --metrics accuracy
104+
python tools/test.py configs/mobileone/mobileone-s0_8xb32_in1k.py mobileone-s0_8xb32_in1k_20221110-0bc94952.pth
61105
```
62106

63-
### In the code
107+
Reparameterize checkpoint:
64108

65-
Use the API `switch_to_deploy` of `MobileOne` backbone to to switch to the deploy mode. Usually called like `backbone.switch_to_deploy()` or `classificer.backbone.switch_to_deploy()`.
109+
```shell
110+
python ./tools/convert_models/reparameterize_model.py ./configs/mobileone/mobileone-s0_8xb32_in1k.py mobileone-s0_8xb32_in1k_20221110-0bc94952.pth mobileone_s0_deploy.pth
111+
```
66112

67-
For Backbones:
113+
Test use fused model:
68114

69-
```python
70-
from mmcls.models import build_backbone
71-
import torch
115+
```shell
116+
python tools/test.py configs/mobileone/deploy/mobileone-s0_deploy_8xb32_in1k.py mobileone_s0_deploy.pth
117+
```
118+
119+
<!-- [TABS-END] -->
72120

73-
x = torch.randn( (1, 3, 224, 224) )
74-
backbone_cfg=dict(type='MobileOne', arch='s0')
75-
backbone = build_backbone(backbone_cfg)
76-
backbone.init_weights()
77-
backbone.eval()
78-
outs_ori = backbone(x)
121+
### Reparameterize Tool
79122

80-
backbone.switch_to_deploy()
81-
outs_dep = backbone(x)
123+
Use provided tool to reparameterize the given model and save the checkpoint:
82124

83-
for out1, out2 in zip(outs_ori, outs_dep):
84-
assert torch.allclose(out1, out2)
125+
```bash
126+
python tools/convert_models/reparameterize_model.py ${CFG_PATH} ${SRC_CKPT_PATH} ${TARGET_CKPT_PATH}
85127
```
86128

87-
For ImageClassifiers:
129+
`${CFG_PATH}` is the config file path, `${SRC_CKPT_PATH}` is the source chenpoint file path, `${TARGET_CKPT_PATH}` is the target deploy weight file path.
88130

89-
```python
90-
from mmcls.models import build_classifier
91-
import torch
92-
import numpy as np
93-
94-
cfg = dict(
95-
type='ImageClassifier',
96-
backbone=dict(
97-
type='MobileOne',
98-
arch='s0',
99-
out_indices=(3, ),
100-
),
101-
neck=dict(type='GlobalAveragePooling'),
102-
head=dict(
103-
type='LinearClsHead',
104-
num_classes=1000,
105-
in_channels=1024,
106-
loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
107-
topk=(1, 5),
108-
))
109-
110-
x = torch.randn( (1, 3, 224, 224) )
111-
classifier = build_classifier(cfg)
112-
classifier.init_weights()
113-
classifier.eval()
114-
y_ori = classifier(x, return_loss=False)
115-
116-
classifier.backbone.switch_to_deploy()
117-
y_dep = classifier(x, return_loss=False)
118-
119-
for y1, y2 in zip(y_ori, y_dep):
120-
assert np.allclose(y1, y2)
131+
For example:
132+
133+
```shell
134+
wget https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth
135+
python ./tools/convert_models/reparameterize_model.py ./configs/mobileone/mobileone-s0_8xb32_in1k.py mobileone-s0_8xb32_in1k_20221110-0bc94952.pth mobileone_s0_deploy.pth
121136
```
122137

138+
To use reparameterized weights, the config file must switch to [**the deploy config files**](./deploy/).
139+
140+
```bash
141+
python tools/test.py ${Deploy_CFG} ${Deploy_Checkpoint}
142+
```
143+
144+
For example of using the reparameterized weights above:
145+
146+
```shell
147+
python ./tools/test.py ./configs/mobileone/deploy/mobileone-s0_deploy_8xb32_in1k.py mobileone_s0_deploy.pth
148+
```
149+
150+
For more configurable parameters, please refer to the [API](https://mmclassification.readthedocs.io/en/1.x/api/generated/mmcls.models.backbones.MobileOne.html#mmcls.models.backbones.MobileOne).
151+
152+
## Results and models
153+
154+
### ImageNet-1k
155+
156+
| Model | Params(M) | Flops(G) | Top-1 (%) | Top-5 (%) | Config | Download |
157+
| :----------: | :-----------------------------: | :----------------------------: | :-------: | :-------: | :---------------------------------------------------: | :------------------------------------------------------: |
158+
| MobileOne-s0 | 5.29（train) \| 2.08 (deploy) | 1.09 (train) \| 0.28 (deploy) | 71.34 | 89.87 | [config (train)](./mobileone-s0_8xb32_in1k.py) \| [config (deploy)](./deploy/mobileone-s0_deploy_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s0_8xb32_in1k_20221110-0bc94952.json) |
159+
| MobileOne-s1 | 4.83 (train) \| 4.76 (deploy) | 0.86 (train) \| 0.84 (deploy) | 75.72 | 92.54 | [config (train)](./mobileone-s1_8xb32_in1k.py) \| [config (deploy)](./deploy/mobileone-s1_deploy_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s1_8xb32_in1k_20221110-ceeef467.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s1_8xb32_in1k_20221110-ceeef467.json) |
160+
| MobileOne-s2 | 7.88 (train) \| 7.88 (deploy) | 1.34 (train) \| 1.31 (deploy) | 77.37 | 93.34 | [config (train)](./mobileone-s2_8xb32_in1k.py) \|[config (deploy)](./deploy/mobileone-s2_deploy_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s2_8xb32_in1k_20221110-9c7ecb97.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s2_8xb32_in1k_20221110-9c7ecb97.json) |
161+
| MobileOne-s3 | 10.17 (train) \| 10.08 (deploy) | 1.95 (train) \| 1.91 (deploy) | 78.06 | 93.83 | [config (train)](./mobileone-s3_8xb32_in1k.py) \|[config (deploy)](./deploy/mobileone-s3_deploy_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s3_8xb32_in1k_20221110-c95eb3bf.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s3_8xb32_in1k_20221110-c95eb3bf.pth) |
162+
| MobileOne-s4 | 14.95 (train) \| 14.84 (deploy) | 3.05 (train) \| 3.00 (deploy) | 79.69 | 94.46 | [config (train)](./mobileone-s4_8xb32_in1k.py) \|[config (deploy)](./deploy/mobileone-s4_deploy_8xb32_in1k.py) | [model](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s4_8xb32_in1k_20221110-28d888cb.pth) \| [log](https://download.openmmlab.com/mmclassification/v0/mobileone/mobileone-s4_8xb32_in1k_20221110-28d888cb.pth) |
163+
123164
## Citation
124165

125166
```bibtex

‎configs/mobileone/deploy/mobileone-s0_deploy_8xb128_in1k.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎configs/mobileone/deploy/mobileone-s0_deploy_8xb32_in1k.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = ['../mobileone-s0_8xb32_in1k.py']
2+
3+
model = dict(backbone=dict(deploy=True))

‎configs/mobileone/deploy/mobileone-s1_deploy_8xb128_in1k.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎configs/mobileone/deploy/mobileone-s1_deploy_8xb32_in1k.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = ['../mobileone-s1_8xb32_in1k.py']
2+
3+
model = dict(backbone=dict(deploy=True))

‎configs/mobileone/deploy/mobileone-s2_deploy_8xb128_in1k.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎configs/mobileone/deploy/mobileone-s2_deploy_8xb32_in1k.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = ['../mobileone-s2_8xb32_in1k.py']
2+
3+
model = dict(backbone=dict(deploy=True))

‎configs/mobileone/deploy/mobileone-s3_deploy_8xb128_in1k.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎configs/mobileone/deploy/mobileone-s3_deploy_8xb32_in1k.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = ['../mobileone-s3_8xb32_in1k.py']
2+
3+
model = dict(backbone=dict(deploy=True))

0 commit comments

Comments
 (0)