Skip to content

Commit 96e0983

Browse files
tirankatharosada
andauthored
bpo-40280: Add limited Emscripten REPL (GH-32284)
Co-authored-by: Katie Bell <[email protected]>
1 parent faa1208 commit 96e0983

File tree

8 files changed

+428
-19
lines changed

8 files changed

+428
-19
lines changed

Makefile.pre.in

+9-2
Original file line numberDiff line numberDiff line change
@@ -807,15 +807,22 @@ $(DLLLIBRARY) libpython$(LDVERSION).dll.a: $(LIBRARY_OBJS)
807807
else true; \
808808
fi
809809

810-
# wasm32-emscripten build
810+
# wasm32-emscripten browser build
811811
# wasm assets directory is relative to current build dir, e.g. "./usr/local".
812812
# --preload-file turns a relative asset path into an absolute path.
813813

814814
$(WASM_STDLIB): $(srcdir)/Lib/*.py $(srcdir)/Lib/*/*.py \
815-
pybuilddir.txt $(srcdir)/Tools/wasm/wasm_assets.py
815+
pybuilddir.txt $(srcdir)/Tools/wasm/wasm_assets.py \
816+
python.html python.worker.js
816817
$(PYTHON_FOR_BUILD) $(srcdir)/Tools/wasm/wasm_assets.py \
817818
--builddir . --prefix $(prefix)
818819

820+
python.html: $(srcdir)/Tools/wasm/python.html python.worker.js
821+
@cp $(srcdir)/Tools/wasm/python.html $@
822+
823+
python.worker.js: $(srcdir)/Tools/wasm/python.worker.js
824+
@cp $(srcdir)/Tools/wasm/python.worker.js $@
825+
819826
##########################################################################
820827
# Build static libmpdec.a
821828
LIBMPDEC_CFLAGS=$(PY_STDMODULE_CFLAGS) $(CCSHARED) @LIBMPDEC_CFLAGS@
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Replace Emscripten's limited shell with Katie Bell's browser-ui REPL from
2+
python-wasm project.

Tools/wasm/README.md

+44-15
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,13 @@ emrun builddir/emscripten-browser/python.html
5555
or
5656

5757
```shell
58-
python3 -m http.server
58+
./Tools/wasm/wasm_webserver.py
5959
```
6060

61+
and open http://localhost:8000/builddir/emscripten-browser/python.html . This
62+
directory structure enables the *C/C++ DevTools Support (DWARF)* to load C
63+
and header files with debug builds.
64+
6165
### Cross compile to wasm32-emscripten for node
6266

6367
```
@@ -79,17 +83,17 @@ popd
7983
node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscripten-node/python.js
8084
```
8185

82-
## wasm32-emscripten limitations and issues
86+
# wasm32-emscripten limitations and issues
8387

84-
- Heap and stack are limited.
85-
- Most stdlib modules with a dependency on external libraries are missing:
86-
``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more.
87-
- Shared extension modules are not implemented yet. All extension modules
88-
are statically linked into the main binary.
89-
The experimental configure option ``--enable-wasm-dynamic-linking`` enables
90-
dynamic extensions.
91-
- Processes are not supported. System calls like fork, popen, and subprocess
92-
fail with ``ENOSYS`` or ``ENOSUP``.
88+
Emscripten before 3.1.8 has known bugs that can cause memory corruption and
89+
resource leaks. 3.1.8 contains several fixes for bugs in date and time
90+
functions.
91+
92+
## Network stack
93+
94+
- Python's socket module does not work with Emscripten's emulated POSIX
95+
sockets yet. Network modules like ``asyncio``, ``urllib``, ``selectors``,
96+
etc. are not available.
9397
- Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or
9498
``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported.
9599
- ``socketpair`` does not work.
@@ -98,8 +102,21 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri
98102
does not resolve to a real IP address. IPv6 is not available.
99103
- The ``select`` module is limited. ``select.select()`` crashes the runtime
100104
due to lack of exectfd support.
105+
106+
## processes, threads, signals
107+
108+
- Processes are not supported. System calls like fork, popen, and subprocess
109+
fail with ``ENOSYS`` or ``ENOSUP``.
101110
- Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction``
102111
are not available or do not work correctly. ``SIGTERM`` exits the runtime.
112+
- Keyboard interrupt (CTRL+C) handling is not implemented yet.
113+
- Browser builds cannot start new threads. Node's web workers consume
114+
extra file descriptors.
115+
- Resource-related functions like ``os.nice`` and most functions of the
116+
``resource`` module are not available.
117+
118+
## file system
119+
103120
- Most user, group, and permission related function and modules are not
104121
supported or don't work as expected, e.g.``pwd`` module, ``grp`` module,
105122
``os.setgroups``, ``os.chown``, and so on. ``lchown`` and `lchmod`` are
@@ -113,23 +130,35 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri
113130
and are disabled.
114131
- Large file support crashes the runtime and is disabled.
115132
- ``mmap`` module is unstable. flush (``msync``) can crash the runtime.
116-
- Resource-related functions like ``os.nice`` and most functions of the
117-
``resource`` module are not available.
133+
134+
## Misc
135+
136+
- Heap memory and stack size are limited. Recursion or extensive memory
137+
consumption can crash Python.
138+
- Most stdlib modules with a dependency on external libraries are missing,
139+
e.g. ``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more.
140+
- Shared extension modules are not implemented yet. All extension modules
141+
are statically linked into the main binary.
142+
The experimental configure option ``--enable-wasm-dynamic-linking`` enables
143+
dynamic extensions.
118144
- glibc extensions for date and time formatting are not available.
119145
- ``locales`` module is affected by musl libc issues,
120146
[bpo-46390](https://bugs.python.org/issue46390).
121147
- Python's object allocator ``obmalloc`` is disabled by default.
122148
- ``ensurepip`` is not available.
123149

124-
### wasm32-emscripten in browsers
150+
## wasm32-emscripten in browsers
125151

152+
- The interactive shell does not handle copy 'n paste and unicode support
153+
well.
126154
- The bundled stdlib is limited. Network-related modules,
127155
distutils, multiprocessing, dbm, tests and similar modules
128156
are not shipped. All other modules are bundled as pre-compiled
129157
``pyc`` files.
130158
- Threading is not supported.
159+
- In-memory file system (MEMFS) is not persistent and limited.
131160

132-
### wasm32-emscripten in node
161+
## wasm32-emscripten in node
133162

134163
Node builds use ``NODERAWFS``, ``USE_PTHREADS`` and ``PROXY_TO_PTHREAD``
135164
linker options.

Tools/wasm/python.html

+245
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta http-equiv="X-UA-Compatible" content="IE=edge">
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
7+
<meta name="author" content="Katie Bell">
8+
<meta name="description" content="Simple REPL for Python WASM">
9+
<title>wasm-python terminal</title>
10+
<link rel="stylesheet" href="https://unpkg.com/[email protected]/css/xterm.css" crossorigin/>
11+
<style>
12+
body {
13+
font-family: arial;
14+
max-width: 800px;
15+
margin: 0 auto
16+
}
17+
#code {
18+
width: 100%;
19+
height: 180px;
20+
}
21+
#info {
22+
padding-top: 20px;
23+
}
24+
.button-container {
25+
display: flex;
26+
justify-content: end;
27+
height: 50px;
28+
align-items: center;
29+
gap: 10px;
30+
}
31+
button {
32+
padding: 6px 18px;
33+
}
34+
</style>
35+
<script src="https://unpkg.com/[email protected]/lib/xterm.js" crossorigin></script>
36+
<script type="module">
37+
class WorkerManager {
38+
constructor(workerURL, standardIO, readyCallBack) {
39+
this.workerURL = workerURL
40+
this.worker = null
41+
this.standardIO = standardIO
42+
this.readyCallBack = readyCallBack
43+
44+
this.initialiseWorker()
45+
}
46+
47+
async initialiseWorker() {
48+
if (!this.worker) {
49+
this.worker = new Worker(this.workerURL)
50+
this.worker.addEventListener('message', this.handleMessageFromWorker)
51+
}
52+
}
53+
54+
async run(options) {
55+
this.worker.postMessage({
56+
type: 'run',
57+
args: options.args || [],
58+
files: options.files || {}
59+
})
60+
}
61+
62+
handleStdinData(inputValue) {
63+
if (this.stdinbuffer && this.stdinbufferInt) {
64+
let startingIndex = 1
65+
if (this.stdinbufferInt[0] > 0) {
66+
startingIndex = this.stdinbufferInt[0]
67+
}
68+
const data = new TextEncoder().encode(inputValue)
69+
data.forEach((value, index) => {
70+
this.stdinbufferInt[startingIndex + index] = value
71+
})
72+
73+
this.stdinbufferInt[0] = startingIndex + data.length - 1
74+
Atomics.notify(this.stdinbufferInt, 0, 1)
75+
}
76+
}
77+
78+
handleMessageFromWorker = (event) => {
79+
const type = event.data.type
80+
if (type === 'ready') {
81+
this.readyCallBack()
82+
} else if (type === 'stdout') {
83+
this.standardIO.stdout(event.data.stdout)
84+
} else if (type === 'stderr') {
85+
this.standardIO.stderr(event.data.stderr)
86+
} else if (type === 'stdin') {
87+
// Leave it to the terminal to decide whether to chunk it into lines
88+
// or send characters depending on the use case.
89+
this.stdinbuffer = event.data.buffer
90+
this.stdinbufferInt = new Int32Array(this.stdinbuffer)
91+
this.standardIO.stdin().then((inputValue) => {
92+
this.handleStdinData(inputValue)
93+
})
94+
} else if (type === 'finished') {
95+
this.standardIO.stderr(`Exited with status: ${event.data.returnCode}\r\n`)
96+
}
97+
}
98+
}
99+
100+
class WasmTerminal {
101+
102+
constructor() {
103+
this.input = ''
104+
this.resolveInput = null
105+
this.activeInput = false
106+
this.inputStartCursor = null
107+
108+
this.xterm = new Terminal(
109+
{ scrollback: 10000, fontSize: 14, theme: { background: '#1a1c1f' }, cols: 100}
110+
);
111+
112+
this.xterm.onKey((keyEvent) => {
113+
// Fix for iOS Keyboard Jumping on space
114+
if (keyEvent.key === " ") {
115+
keyEvent.domEvent.preventDefault();
116+
}
117+
});
118+
119+
this.xterm.onData(this.handleTermData)
120+
}
121+
122+
open(container) {
123+
this.xterm.open(container);
124+
}
125+
126+
handleReadComplete(lastChar) {
127+
this.resolveInput(this.input + lastChar)
128+
this.activeInput = false
129+
}
130+
131+
handleTermData = (data) => {
132+
if (!this.activeInput) {
133+
return
134+
}
135+
const ord = data.charCodeAt(0);
136+
let ofs;
137+
138+
// TODO: Handle ANSI escape sequences
139+
if (ord === 0x1b) {
140+
// Handle special characters
141+
} else if (ord < 32 || ord === 0x7f) {
142+
switch (data) {
143+
case "\r": // ENTER
144+
case "\x0a": // CTRL+J
145+
case "\x0d": // CTRL+M
146+
this.xterm.write('\r\n');
147+
this.handleReadComplete('\n');
148+
break;
149+
case "\x7F": // BACKSPACE
150+
case "\x08": // CTRL+H
151+
case "\x04": // CTRL+D
152+
this.handleCursorErase(true);
153+
break;
154+
}
155+
} else {
156+
this.handleCursorInsert(data);
157+
}
158+
}
159+
160+
handleCursorInsert(data) {
161+
this.input += data;
162+
this.xterm.write(data)
163+
}
164+
165+
handleCursorErase() {
166+
// Don't delete past the start of input
167+
if (this.xterm.buffer.active.cursorX <= this.inputStartCursor) {
168+
return
169+
}
170+
this.input = this.input.slice(0, -1)
171+
this.xterm.write('\x1B[D')
172+
this.xterm.write('\x1B[P')
173+
}
174+
175+
prompt = async () => {
176+
this.activeInput = true
177+
// Hack to allow stdout/stderr to finish before we figure out where input starts
178+
setTimeout(() => {this.inputStartCursor = this.xterm.buffer.active.cursorX}, 1)
179+
return new Promise((resolve, reject) => {
180+
this.resolveInput = (value) => {
181+
this.input = ''
182+
resolve(value)
183+
}
184+
})
185+
}
186+
187+
clear() {
188+
this.xterm.clear();
189+
}
190+
191+
print(message) {
192+
const normInput = message.replace(/[\r\n]+/g, "\n").replace(/\n/g, "\r\n");
193+
this.xterm.write(normInput);
194+
}
195+
}
196+
197+
const replButton = document.getElementById('repl')
198+
const clearButton = document.getElementById('clear')
199+
200+
window.onload = () => {
201+
const terminal = new WasmTerminal()
202+
terminal.open(document.getElementById('terminal'))
203+
204+
const stdio = {
205+
stdout: (s) => { terminal.print(s) },
206+
stderr: (s) => { terminal.print(s) },
207+
stdin: async () => {
208+
return await terminal.prompt()
209+
}
210+
}
211+
212+
replButton.addEventListener('click', (e) => {
213+
// Need to use "-i -" to force interactive mode.
214+
// Looks like isatty always returns false in emscripten
215+
pythonWorkerManager.run({args: ['-i', '-'], files: {}})
216+
})
217+
218+
clearButton.addEventListener('click', (e) => {
219+
terminal.clear()
220+
})
221+
222+
const readyCallback = () => {
223+
replButton.removeAttribute('disabled')
224+
clearButton.removeAttribute('disabled')
225+
}
226+
227+
const pythonWorkerManager = new WorkerManager('./python.worker.js', stdio, readyCallback)
228+
}
229+
</script>
230+
</head>
231+
<body>
232+
<h1>Simple REPL for Python WASM</h1>
233+
<div id="terminal"></div>
234+
<div class="button-container">
235+
<button id="repl" disabled>Start REPL</button>
236+
<button id="clear" disabled>Clear</button>
237+
</div>
238+
<div id="info">
239+
The simple REPL provides a limited Python experience in the browser.
240+
<a href="https://github.com/python/cpython/blob/main/Tools/wasm/README.md">
241+
Tools/wasm/README.md</a> contains a list of known limitations and
242+
issues. Networking, subprocesses, and threading are not available.
243+
</div>
244+
</body>
245+
</html>

0 commit comments

Comments
 (0)