-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathREADME.html
More file actions
292 lines (286 loc) · 28.3 KB
/
README.html
File metadata and controls
292 lines (286 loc) · 28.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>EnergyShield (ICCPS '23, Submission 251)</title>
<style>
/* From extension vscode.github */
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
.vscode-dark img[src$=\#gh-light-mode-only],
.vscode-light img[src$=\#gh-dark-mode-only] {
display: none;
}
</style>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex/dist/katex.min.css">
<link href="https://cdn.jsdelivr.net/npm/katex-copytex@latest/dist/katex-copytex.min.css" rel="stylesheet" type="text/css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Microsoft/vscode/extensions/markdown-language-features/media/markdown.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Microsoft/vscode/extensions/markdown-language-features/media/highlight.css">
<style>
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe WPC', 'Segoe UI', system-ui, 'Ubuntu', 'Droid Sans', sans-serif;
font-size: 14px;
line-height: 1.6;
}
</style>
<style>
.task-list-item {
list-style-type: none;
}
.task-list-item-checkbox {
margin-left: -20px;
vertical-align: middle;
pointer-events: none;
}
</style>
</head>
<body class="vscode-body vscode-light">
<h1 id="energyshield-iccps-23-submission-251">EnergyShield (ICCPS '23, Submission 251)</h1>
<p>EnergyShield is an Autonomous Driving System framework designed to save energy on-vehicle by wirelessly offloading NN computations to edge computers, while at the same time maintaining a formal guarantee of safety. In particular, EnergyShield uses a barrier function/controller shield to provide provably safe edge response times for NN offloading requests; on vehicle evaluation provides a safety fallback.</p>
<p>EnergyShield is described in the paper:</p>
<blockquote>
<p><em>EnergyShield: Provably-Safe Offloading of Neural Network Controllers for Energy Efficiency.</em><br>
Mohanad Odema, James Ferlez, Goli Vaisi, Yasser Shoukry and Mohammad Abdullah Al Faruque. ICCPS 2023: 14th ACM/IEEE International Conference on Cyber-Physical Systems.</p>
</blockquote>
<p>which is attached to this document, and hereafter referred to as [ES23].</p>
<p>[ES23] contains a number of experiments showing the efficacy of EnergyShield in the <a href="https://carla.org">Carla</a> simulation environment. This README describes how to replicate those results using code packaged in a <a href="https://docs.docker.com/engine/">Docker</a> image. In particular, this artifact reruns from scratch the following experiments from [ES23]:</p>
<p> <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation"><strong>Experiment 1 <em>[ES23, Section 5.2]</em></strong></a>: <em>Energy Efficiency and Safety Evaluation of EnergyShield through Carla Simulation runs</em></p>
<p> <a href="#4-experiment-2---performance-under-wireless-channel-variation"><strong>Experiment 2 <em>[ES23, Section 5.3]</em></strong></a> : <em>Performance Gains from EnergyShield given wireless channel variation</em></p>
<p> <a href="#5-experiment-3---comparison-between-multiple-controllers"><strong>Experiment 3 <em>[ES23, Section 5.4]</em></strong></a>: <em>EnergyShield generality to other DRL agents</em></p>
<p>For each of these experiments, this artifact re-generates both new raw data and the analogous plots shown in [ES23].</p>
<h2 id="contents">Contents</h2>
<ol start="0">
<li><a href="#0-terminology">Terminology</a></li>
<li><a href="#1-system-requirements">System Requirements</a></li>
<li><a href="#2-setup">Setup</a></li>
<li><a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1 - Energy Efficiency and Safety Evaluation</a></li>
<li><a href="#4-experiment-2---performance-under-wireless-channel-variation">Experiment 2 - Performance under wireless channel variation</a></li>
<li><a href="#5-experiment-3---comparison-between-multiple-controllers">Experiment 3 - Comparison Between Multiple Controllers</a></li>
<li><a href="#6-appendix">Appendix</a></li>
</ol>
<h2 id="0-terminology">0. Terminology</h2>
<p>This repeatability artifact uses <a href="https://docs.docker.com/engine/">Docker</a>. We will use the following terminology throughout:</p>
<ul>
<li>The <strong>HOST</strong> will refer to the system running Docker (e.g. your laptop).</li>
<li>The <strong>CONTAINER</strong> will refer to the "virtualized" system created by Docker (this is where the code from the artifact is run).</li>
</ul>
<p>Commands meant to be executed from a shell on the host or the container will be prefixed with one of the following comments, respectively:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
<span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
</code></pre>
<h2 id="1-system-requirements">1. System Requirements</h2>
<p><strong>HARDWARE (HOST):</strong></p>
<ol>
<li>An x86-64 CPU</li>
<li>32GB of RAM</li>
<li>An NVIDIA GPU with 8GB of VRAM <strong>(Geforce 20xx series or later; RTX 2070, GTX 2080Ti and V100 cards were tested)</strong>; headless GPUs will work (e.g. servers and Amazon EC2/Microsoft Azure instances)</li>
<li>At least 80GB of free disk space on the filesystem where Docker stores images (the filesystem containing <code>/var/lib/docker</code> <a href="https://docs.docker.com/config/daemon/#daemon-data-directory">by default</a>)</li>
<li>An internet connection that can download ~30GB of data</li>
</ol>
<p><strong>SOFTWARE (HOST):</strong></p>
<ol>
<li>Un-virtualized Linux operating system; headless and cloud (e.g. Amazon EC2/Microsoft Azure) installs will work (tested on Ubuntu 20.04, but any distribution that meets the remaining requirements should work)</li>
<li>Official <a href="https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html">Linux NVIDIA drivers</a> <strong>(version >= 515.76 is **<em>REQUIRED</em>**)</strong></li>
<li>A recent version of <a href="https://docs.docker.com/engine/">Docker Engine</a> <strong>(version >= 19.03)</strong>; also known as Docker Server but <strong>not</strong> <a href="https://docker.com">Docker Desktop</a></li>
<li>The <a href="https://github.com/NVIDIA/nvidia-docker">NVIDIA container toolkit</a></li>
<li>A recent version of <code>git</code> on the path</li>
<li>The <code>bash</code> shell installed in <code>/bin/bash</code></li>
<li>A user account that is a member of the <code>docker</code> group (see <a href="https://docs.docker.com/engine/install/linux-postinstall/">Docker post-install instructions</a>)</li>
<li>A user account that can run Docker containers in privileged mode (i.e. with the <a href="https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities"><code>--privileged</code> switch</a>)</li>
</ol>
<blockquote>
<p><strong>WARNING:</strong> NVIDIA driver version >=515.76 is a <strong>STRICT REQUIREMENT</strong>. This repeatability artifact <strong>WILL NOT WORK</strong> unless the host has official NVIDIA drivers version 515.76 or higher installed.</p>
</blockquote>
<h2 id="2-setup">2. Setup</h2>
<h3 id="i-host-paths"><em>(i) Host Paths</em></h3>
<p>Choose an install location on the host:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
INSTALL_LOCATION=/path/to/some/place/convenient
<span class="hljs-built_in">cd</span> <span class="hljs-string">"<span class="hljs-variable">$INSTALL_LOCATION</span>"</span>
</code></pre>
<br>
<br>
<p>EnergyShield will be installed on the <strong>host</strong> at:</p>
<pre><code class="language-Bash">ES_PATH=<span class="hljs-string">"<span class="hljs-variable">$INSTALL_LOCATION</span>/EnergyShield"</span>
</code></pre>
<blockquote>
<p><strong>NOTE:</strong> All subsequent <strong>HOST</strong> paths in this readme are assumed to be relative to <code>$ES_PATH</code> unless otherwise specified.</p>
</blockquote>
<h3 id="ii-installing-energyshieldbuilding-the-docker-image"><em>(ii) Installing EnergyShield/Building the Docker Image</em></h3>
<p>To install EnergyShield and build the EnergyShield Docker image, execute the following in a non-root Bash shell on the host (from <code>$INSTALL_LOCATION</code>):</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
<span class="hljs-built_in">cd</span> <span class="hljs-string">"<span class="hljs-variable">$INSTALL_LOCATION</span>"</span>
git <span class="hljs-built_in">clone</span> --recursive https://github.com/MohanadOdema/EnergyShield
<span class="hljs-built_in">cd</span> EnergyShield
./dockerbuild.sh <span class="hljs-comment"># WARNING: downloads ~30GB of data, and may take > 1 hour even after download!</span>
</code></pre>
<blockquote>
<p><strong>NOTE:</strong> These and all subsequent <strong>host</strong> commands should be run from a normal user account; e.g., not with <code>sudo</code>. This requires that the current user account has permissions to execute the <code>docker</code> command. See <a href="#1-system-requirements">Section 1, Software Requirements #6</a>.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> <code>dockerbuild.sh</code> only ever needs to be run once per host user account, unless an updated version is released.</p>
</blockquote>
<h3 id="iii-starting-a-docker-container"><em>(iii) Starting a Docker Container</em></h3>
<p>First, make sure you have successfully built an EnergyShield Docker image. See <a href="#ii-installing-energyshieldbuilding-the-docker-image">Section 2 <em>(ii)</em></a> above.</p>
<p>Then to start the EnergyShield Docker container, execute the following in a non-root Bash shell on the host (from <code>$ES_PATH</code>):</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
./dockerrun.sh --interactive --start-carla
</code></pre>
<p>This should place you at a Bash shell inside a container with EnergyShield installed. The container's Bash shell will have a prompt that looks like:</p>
<pre><code class="language-Bash">carla@ece2ade62bc5:~$
</code></pre>
<p>where <code>ece2ade62bc5</code> is a unique container id (i.e. yours will be different).
<br>
<br>
<br></p>
<blockquote>
<p><strong>WARNING:</strong> if you exit the container's Bash shell, then the <strong>container and all experiments will stop</strong>. You may restart the container with the <strong>host</strong> command (from a non-root Bash shell at <code>$ES_PATH</code>):</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
./dockerrun.sh --interactive --start-carla
</code></pre>
</blockquote>
<h3 id="iv-testing-the-container"><em>(iv) Testing the Container</em></h3>
<p>First, make sure you have an EnergyShield container running. See <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a> above.</p>
<p>Then from the <strong>container</strong>'s Bash shell execute:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
ps | grep Carla
</code></pre>
<p>You should see output listing two processes related to <a href="https://carla.org">Carla</a>:</p>
<pre><code class="language-Bash"> 45 pts/0 00:00:00 CarlaUE4.sh
52 pts/0 00:01:00 CarlaUE4-Linux-
</code></pre>
<blockquote>
<p><strong>WARNING:</strong> if the above command produces no output or output unlike the above, then Carla is not running, and this repeatability artifact <strong>WILL NOT WORK</strong>.</p>
<p><strong>FIX:</strong> Double check that your host system meets the requirements in <a href="#1-system-requirements">Section 1</a>, especially the NVIDIA driver requirements (incorrect NVIDIA drivers are usually what prevents Carla from starting). If your system meets these requirements, then try restarting the container using the following sequence of commands:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
<span class="hljs-comment"># Exit from the container if it's still running:</span>
<span class="hljs-built_in">exit</span>
</code></pre>
<p>And then in <code>$ES_PATH</code>:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< HOST COMMANDS >>></span>
./dockerrun.sh --remove <span class="hljs-comment"># Remove any existsing container</span>
./dockerrun.sh --interactive --start-carla
</code></pre>
</blockquote>
<br>
<br>
<br>
<h2 id="3-experiment-1---energy-efficiency-and-safety-evaluation">3. Experiment 1 - Energy Efficiency and Safety Evaluation</h2>
<p>In this experiment, we compared EnergyShield with purely on-vehicle NN controller evaluation, both in terms of energy consumption and safety; see [ES23], Section 5.2. This comparison was made using a single RL-trained NN controller driving a fixed track; safety entails avoiding randomly spawned stationary obstacles along this track. This artifact reuses the same track and NN controller from our experiment, but the obstacle locations and instantaneous wireless-link performance are randomized.</p>
<p>To rerun this experiment, ensure that you have an EnergyShield container running (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>), and execute the following commands in the <strong>container</strong>'s bash shell:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
<span class="hljs-built_in">cd</span> /home/carla/EnergyShield
<span class="hljs-comment"># Run Experiment 1 Carla simulation</span>
./scripts/run_exp1.sh NUM_EPS
</code></pre>
<!-- `model` is the name of the pretrained model directory(default: "casc_agent_1_new"). Users do not need to alter this argument for it indicates new simulation experiments using our pretrained RL model. -->
<p><code>NUM_EPS</code> is an optional argument specifying the number of episodes to run (default: <code>NUM_EPS=3</code>); an episode is defined as one run of the fixed track until either the vehicle completes the track or hits an obstacle. For [ES23], we ran this experiment for 35 episodes.</p>
<blockquote>
<p><strong>NOTE:</strong> Running this script with the default <code>NUM_EPS=3</code> takes around 1 hour on a workstation with 32 GB RAM and NVIDIA GPU 2070 RTX super.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> When the script finishes, you will be returned to a shell prompt in the container (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>). If the script is successfully running, status information will be output to the console regularly.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> Experiments can be run consecutively in the same container without interruption, or in between container restarts.</p>
</blockquote>
<blockquote>
<p><strong>WARNING:</strong> Occasionally, the experiment script may fail to connect to Carla even if Carla is running (see <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>). This is a <a href="https://github.com/carla-simulator/carla/issues/3430">known issue</a> in Carla on slow host machines; if it occurs, simply re-run the script above. If this fails, try restarting the container according to the directions in <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>.</p>
</blockquote>
<p>The primary outputs of this script are figures that summarize the energy savings and safety properties of EnergyShield (across the number of episodes specified by <code>NUM_EPS</code>). These figures can be found on the <strong>HOST</strong> at the following paths:</p>
<pre><code class="language-Bash"><span class="hljs-variable">$ES_PATH</span>/container_results/Fig5_Energy.pdf
<span class="hljs-variable">$ES_PATH</span>/container_results/Fig5_Safety.pdf
<span class="hljs-variable">$ES_PATH</span>/container_results/Fig6_traj_noise_False.pdf
<span class="hljs-variable">$ES_PATH</span>/container_results/Fig6_traj_noise_True.pdf
<span class="hljs-variable">$ES_PATH</span>/container_results/Fig7_Ergy_v_dist.pdf
</code></pre>
<p>Their filenames match them to the figures that appear in [ES23]. For reference, the figures from [ES23] are also available with the same filenames in <code>$ES_PATH/paper_results</code>.</p>
<p>This script also outputs the raw Carla simulation data of each episode (i.e. the simulation time-stamped positions, control actions, etc. of the vehicle); this data is placed in <code>$ES_PATH/container_results/raw_data</code>; the analogous raw data from our simulations can be found in <code>$ES_PATH/paper_results/raw_data</code> for comparison. The format and structure of this data is described in the subsequent <a href="#6-appendix">Appendix</a>.</p>
<p>Finally, Figure 7 (Energy vs. distance) is derived from some summary statistics of raw data noted above; these summaries are produced as several <code>.CSV</code> files output to <code>$ES_PATH/container_results/distance/*.csv</code>.</p>
<h2 id="4-experiment-2---performance-under-wireless-channel-variation">4. Experiment 2 - Performance under wireless channel variation</h2>
<p>In this experiment, we evaluated the energy savings provided by EnergyShield as a function of different wireless-link conditions to the edge; see [ES23], Section 5.3. This experiment uses mostly the same setup as Experiment 1, including the same NN controller and track. However, in this experiment, batches of episodes are run under five different simulated wireless-link conditions. As before, this artifact reuses the same track and NN controller from our experiment, but the obstacle locations and instantaneous wireless-link performance are randomized.</p>
<!-- In 5.3, Additional Carla simulations are conducted to evaluate EnergyShield's resilience under variations of wireless network conditions, which are represented in this paper by the parameters of channel throughput and queuing delays. To run these additional simulations, users can run the following script: -->
<p>To rerun this experiment, ensure that you have an EnergyShield container running (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>), and execute the following commands in the <strong>container</strong>'s bash shell</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
<span class="hljs-built_in">cd</span> /home/carla/EnergyShield
<span class="hljs-comment"># Run Experiment 2 Carla simulations:</span>
./scripts/run_exp2.sh NUM_EPS
</code></pre>
<p>The optional parameter <code>NUM_EPS</code> has a similar interpretation to <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a> (default <code>NUM_EPS=3</code>), but here it represents the number of episodes to run under each wireless-link condition (of which this experiment contains five).</p>
<blockquote>
<p><strong>NOTE:</strong> Running this script with the default <code>NUM_EPS=3</code> takes around 1 hour on a workstation with 32 GB RAM and NVIDIA GPU 2070 RTX super.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> When the script finishes, you will be returned to a shell prompt in the container (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>). If the script is successfully running, status information will be output to the console regularly.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> Experiments can be run consecutively in the same container without interruption, or in between container restarts.</p>
</blockquote>
<blockquote>
<p><strong>WARNING:</strong> Occasionally, the experiment script may fail to connect to Carla even if Carla is running (see <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>). This is a <a href="https://github.com/carla-simulator/carla/issues/3430">known issue</a> in Carla on slow host machines; if it occurs, simply re-run the script above. If this fails, try restarting the container according to the directions in <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>.</p>
</blockquote>
<p>The primary outputs of this script are figures that summarize the energy performance of EnergyShield under these different wireless conditions, including the number of local-execution fallbacks required. These figures can be found on the <strong>HOST</strong> at the following paths:
<br>
<br>
<br></p>
<pre><code class="language-Bash"><span class="hljs-variable">$ES_PATH</span>/container_results/Fig8_windows.pdf
<span class="hljs-variable">$ES_PATH</span>/container_results/Fig9_energy.pdf
</code></pre>
<p>As before, their filenames match them to the figures that appear in [ES23]. Likewise, the figures from [ES23] are also available with the same filenames in <code>$ES_PATH/paper_results</code>.</p>
<p>The raw data from this experiment is similarly placed in <code>$ES_PATH/container_results/raw_data</code>, and the analogous raw data from our simulations is in <code>$ES_PATH/paper_results/raw_data</code> for comparison. See the the subsequent <a href="#6-appendix">Appendix</a> for a description of the format and structure of this raw data.</p>
<h2 id="5-experiment-3---comparison-between-multiple-controllers">5. Experiment 3 - Comparison Between Multiple Controllers</h2>
<p>In this experiment, we again compared EnergyShield with purely on-vehicle NN controller evaluation, both in terms of energy consumption and safety; however, in this experiment, we compared this performance between different NN controllers (four total, including the one used in <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a>). Hence, this experiment effectively amounts to re-running <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a> on three additional RL-trained NN controllers; see [ES23], Section 5.4. As before, this artifact reuses the same track and NN controllers from our experiment, but the obstacle locations and instantaneous wireless-link performance are randomized.</p>
<blockquote>
<p><strong>NOTE:</strong> To get complete results you should run Experiment 1 first.</p>
</blockquote>
<p>To rerun this experiment, ensure that you have an EnergyShield container running (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>), and execute the following commands in the <strong>container</strong>'s bash shell:</p>
<pre><code class="language-Bash"><span class="hljs-comment"># <<< CONTAINER COMMANDS >>></span>
<span class="hljs-built_in">cd</span> /home/carla/EnergyShield
<span class="hljs-comment"># Run Experiment 3 generate model statistics</span>
./scripts/run_exp3.sh NUM_EPS
</code></pre>
<p><code>NUM_EPS</code> is an optional argument that has a similar interpretation to <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a> (default <code>NUM_EPS=3</code>), but here it represents the (common) number of episodes to be run on <em>each</em> of the other three agents.</p>
<blockquote>
<p><strong>NOTE:</strong> Running this script should take about three times as long as running <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a> for the same number of episodes.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> When the script finishes, you will be returned to a shell prompt in the container (see <a href="#iii-starting-a-docker-container">Section 2 <em>(iii)</em></a>). If the script is successfully running, status information will be output to the console regularly.</p>
</blockquote>
<blockquote>
<p><strong>NOTE:</strong> Experiments can be run consecutively in the same container without interruption, or in between container restarts. However, to incorporate the results of <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a>, do not <em>remove</em> the container between runs of <code>run_exp1.sh</code> and <code>run_exp3.sh</code>.</p>
</blockquote>
<br>
<br>
<br>
<blockquote>
<p><strong>WARNING:</strong> Occasionally, the experiment script may fail to connect to Carla even if Carla is running (see <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>). This is a <a href="https://github.com/carla-simulator/carla/issues/3430">known issue</a> in Carla on slow host machines; if it occurs, simply re-run the script above. If this fails, try restarting the container according to the directions in <a href="#iv-testing-the-container">Section 2 <em>(iv)</em></a>.</p>
</blockquote>
<p>The primary outputs of this script are a collection of <code>*.CSV</code> files containing summary data for each agent, as reported in Table 1 of [ES23]. This data includes average center deviance (CD), Track Completion Rate (TCR) and average energy consumption (E). These files are output as:</p>
<pre><code class="language-Bash"><span class="hljs-variable">$ES_PATH</span>/container_results/stats_for_casc_agent_1_new.csv
<span class="hljs-variable">$ES_PATH</span>/container_results/stats_for_casc_agent_2_new.csv
<span class="hljs-variable">$ES_PATH</span>/container_results/stats_for_casc_agent_3_new.csv
<span class="hljs-variable">$ES_PATH</span>/container_results/stats_for_casc_agent_4_new.csv
</code></pre>
<p>The number in each file name matches the data to the corresponding Controller 1 through 4 as indicated in Table 1. Since Controller 1 was used in <a href="#3-experiment-1---energy-efficiency-and-safety-evaluation">Experiment 1</a>, the summary data in <code>stats_for_casc_agent_1_new.csv</code> is derived from the output of <code>run_exp1.sh</code>.</p>
<p>The raw data for each additional agent is likewise output to <code>$ES_PATH/container_results/raw_data</code>, with one folder for each of these additional agents. See the <a href="#6-appendix">Appendix</a> for a description of the structure and format of this data.</p>
<h2 id="6-appendix">6. Appendix</h2>
<p>Raw Data</p>
<p>Under <code>$ES_PATH/container_results/raw_data/</code>, every directory describes the evaluation results for a set of episodes. The name suffix of each directory describes the experimental configuration setting these evaluations belong to. For instance:</p>
<ul>
<li>Directory <code>Town04_OPT_ResNet152_Shield2_early</code>: experiment1 evaluations for EnergyShield early mode at default wireless settings</li>
<li>Directory <code>Town04_OPT_ResNet152_Shield2_belay_10Mbps</code>: experiment2 evaluations for EnergyShield uniform mode when varying wireless channel throughput to 10Mbps</li>
</ul>
<p>Within each of these directories, four subdirectories exist to describe whether the experiments were conducted with/without gaussian noise and with/without safety filter activated. In each one of these subdirectories, we can find the experimental data in <code>*.csv</code> format as follows:</p>
<ul>
<li><code>plots/*.csv</code> files containing the individual episode's data incurred by the carla simulator when running each episode for the corresponding experimental setting.</li>
<li><code>valid_data.csv</code> is the file describing the final statistics for every episode within the corresponding experimental configuration</li>
</ul>
<p>Information in the former include simulation time, position, steering angle, relative obstacle position, wireless conditions, safety time window, offloading actions, and performance evaluations in terms of latency and energy - all calculated instantaneously for every simulation tick.</p>
<p>Information in the latter include statistics for each episode about total number of ticks, reward, distance traveled, speed, average latency, average energy per inference, average center deviance, and whether the agent has hit an obstacle/curb.</p>
<p>Distance data</p>
<p>For figure 7 (normalized energy variation vs distance (meters)), raw data is used to construct the tables in <code>$ES_PATH/container_results/distance/*.csv</code> which aggregate average normalized energy consumption and #occurrences across 1 m increments of 'distance from obstacle' parameter from all episodes within the subdirectory. Each of these tables is then used to construct a plot from the figure describing how this relation varies under different experimental settings.</p>
<script async src="https://cdn.jsdelivr.net/npm/katex-copytex@latest/dist/katex-copytex.min.js"></script>
</body>
</html>