Skip to content

Example extending docker-stacks? #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dirkcgrunwald opened this issue Mar 9, 2019 · 19 comments
Open

Example extending docker-stacks? #106

dirkcgrunwald opened this issue Mar 9, 2019 · 19 comments

Comments

@dirkcgrunwald
Copy link

I have a dockerfile for a jupyterlab that is customized to our undergrad computing environment.

I'd like to add a theia or coding.com version of Visual Studio IDE (as well as rstudio etc). I've been reading through the dox and jupyterserverproxy-openrefine which uses binder.

That said, it would be very useful to have an example that uses one of the docker-stacks examples with an extension to have a sample extension (e.g. theia, rstudio). Am I missing this in the documentation?
@betatim
Copy link
Member

betatim commented Mar 9, 2019

There is a repo with theia that you could look at https://github.com/betatim/theia-binder. You can use http://repo2docker.readthedocs.io/ to see what the dockerfile is that is used to build the image on BinderHub. Should give you ideas on how to apply it to other Dockerfiles.

I think a how-to on extending a specific flavour of docker images is maybe better placed in the repository of the docker-stacks?

@betatim
Copy link
Member

betatim commented Mar 9, 2019

I just had a god at running https://github.com/codercom/code-server inn a binder. It installs fine (get the binary and launch it from a terminal) but connecting to it timesout. Not sure what that is about :-/ It also uses a lot of CPU even when no one is trying to connect to it so, yeah :-/ Would be a cool demo.

@betatim
Copy link
Member

betatim commented Mar 9, 2019

I made https://github.com/betatim/vscode-binder/tree/master to experiment on this. Currently stuck because code-server serves some assets with an absolute path instead of a relative one. Waiting to see if they know how to fix that.

@dirkcgrunwald
Copy link
Author

The challenge is that when you're using docker-stacks for a zero-to-jupyterhub setup, you're planning on mounting /home/jovyan using a PV. In a binderhub, all the files are in the gitrepo, do the examples work fine.

The example theia configuration assumes that the package.json for theia is in the users directory -- but it's not because we've mounted a pre-existing PV on top of /home/jovyan when we deploy. Again, if you're using binder, then the directory/files will be in the user directory based on my trials.

My goal is to e.g. install theia in /opt/theia. So far, my unsuccessful attempt, which can start theia but wherein it just spins a loading icon forever is to modify the jupyter_theia_proxy to chdir() to /opt/theia and then start with a workplace argument of /home/jovyan. However, when running theia, the /opt/theia directory needs to be R/W to the user (jovyan).

My dockerfile is below.

FROM jupyter/minimal-notebook                                                                             
                                                                                                          
USER    root                                                                                              
COPY    theia /opt/theia                                                                                  
RUN     conda install python-language-server flake8 autopep8 && \                                         
        cd /opt/theia && \                                                                                
        jlpm && \                                                                                         
        jlpm theia build                                                                                  
                                                                                                          
RUN     conda install nbserverproxy && \                                                                  
        jupyter serverextension enable --py --sys-prefix nbserverproxy && \                               
        jupyter labextension install jupyterlab-server-proxy                                              
                                                                                                          
RUN     pip install jupyter-theia-proxy && \                                                              
        rm -rf /opt/conda/lib/python3.7/site-packages/jupyter_theia_proxy/__pycache__                     
                                                                                                          
COPY    p-theia/jupyter_theia_proxy /opt/conda/lib/python3.7/site-packages/jupyter_theia_proxy            
                                                                                                          
RUN     chown -R $NB_UID /opt/theia                                                                       
                                                                                                          
ENV     PATH=/opt/theia/node_modules/.bin:$PATH                                                           
                                                                                                          
USER    $NB_UID

and the jupyuter_theia_proxy/init.py file has been modified as such:

        os.chdir('/opt/theia')                                                                            
        return ['theia', 'start', '/home/jovyan', '--hostname=127.0.0.1', '--port=' + str(port)]

This still isn't working, which is why I'm looking for an example that works. The reason I'm asking in this repo is because the jupyter_theia_proxy package is here.

@dirkcgrunwald
Copy link
Author

I think this is a theia problem - I believe it's loading files using absolute path names, e.g.

[W 16:46:57.311 LabApp] 404 GET /min-maps/vs/loader.js.map (172.17.0.1) 3.32ms referer=None
[W 16:46:57.315 LabApp] 404 GET /min-maps/vs/editor/editor.main.js.map (172.17.0.1) 6.03ms referer=None
[W 16:46:57.317 LabApp] 404 GET /min-maps/vs/editor/editor.main.nls.js.map (172.17.0.1) 7.66ms referer=None

I tried packaging up Cloud9 ( see https://github.com/dirkcgrunwald/jupyter-cloud9-proxy ) but it has a similar problem -- opening the C9 tile launches e.g. http://localhost:5555/c9 and then cloud9 immediately opens http://localhost:5555/ide.htm (omitting the 'c9' prefix). This is a known problem with Cloud9 as I'm now aware ( c9/core#11 ) and would require modification to their code.

There's a similar problem with the http://coder.com version of VScode - the websocket address doesn't include the URL prefix ( coder/code-server#149 ). Looks like this might be fixed in their 1.32 release

@oscar6echo
Copy link

oscar6echo commented Apr 14, 2019 •
edited
Loading

I'm also trying to serve VS Code to my jhub users.
Following @betatim I am using package jupyter-proxy-server with custom package jupyter-code-proxy with the following init.py.

It does not crash nor show any error but the screen is blank (black in fact) and it just hangs.
Any idea if it is a code-server issue or a bad config on my part ?

import os
import shutil

def setup_code_server():
    """
    """
    def _code_server_command(port):
        # Make sure code-server is in $PATH
        full_path = shutil.which('/opt/code-server/code-server')
        if not full_path:
            raise FileNotFoundError('Can not find code-server executable in $PATH')
        return ['/opt/code-server/code-server',
                '--no-auth',
                '--allow-http',
                '--port', str(port),
                '/home/jovyan',
                ]

    return {
        'command': _code_server_command,
        'launcher_entry': {
            'title': 'VS Code',
            'icon_path': os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                      'assets',
                                      'logo-vscode.svg')
        }
    }

after I downloaded and unzipped code-server from code-server1.792-vsc1.33.1-linux-x64.tar.gz and put it as /opt/code-server/code-server in my single-user container.

The kube logs from the notebook-server are

...
I 2019-04-14 19:30:37.877 SingleUserNotebookApp log:158] 200 GET /user/ab/api/sessions?_=1555270237760 ([email protected]) 8.99ms
[I 2019-04-14 19:30:37.880 SingleUserNotebookApp log:158] 200 GET /user/ab/api/terminals?_=1555270237761 ([email protected]) 11.07ms
[I 2019-04-14 19:30:37.895 SingleUserNotebookApp log:158] 200 GET /user/ab/api/contents?type=directory&_=1555270237762 ([email protected]) 3.98ms
INFO  code-server v1.792-vsc1.33.1
INFO  Additional documentation: http://github.com/codercom/code-server
INFO  Initializing {"data-dir":"/home/jovyan/.local/share/code-server","extensions-dir":"/home/jovyan/.local/share/code-server/extensions","working-dir":"/home/jovyan","log-dir":"/home/jovyan/.cache/code-server/logs/20190414193047369"}
INFO  Starting shared process [1/5]...
INFO  Starting webserver... {"host":"0.0.0.0","port":"43093"}
WARN  No certificate specified. This could be insecure.
WARN  Documentation on securing your setup: https://github.com/codercom/code-server/blob/master/doc/security/ssl.md
WARN  Launched without authentication.
INFO   
INFO  Started (click the link below to open):
INFO  http://localhost:43093/
INFO   
WARN  Port scanning will not be available because netstat is not installed
[I 2019-04-14 19:30:47.547 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 705.66ms
[I 2019-04-14 19:30:47.571 SingleUserNotebookApp log:158] 200 GET /user/ab/code/main.css ([email protected]) 15.54ms
[I 2019-04-14 19:30:47.610 SingleUserNotebookApp log:158] 200 GET /user/ab/code/1-f2dd3d.bundle.js ([email protected]) 53.03ms
[I 2019-04-14 19:30:47.760 SingleUserNotebookApp log:158] 200 GET /user/ab/code/f2dd3d.bundle.js ([email protected]) 201.54ms
/opt/conda/lib/python3.7/site-packages/jupyter_server_proxy/websocket.py:89: RuntimeWarning: coroutine 'WebSocketHandler.get' was never awaited
  super().get(*args, **kwargs)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[I 2019-04-14 19:30:48.345 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 5.18ms
[I 2019-04-14 19:30:48.403 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.01ms
[I 2019-04-14 19:30:48.458 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.94ms
[I 2019-04-14 19:30:48.517 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.94ms
[I 2019-04-14 19:30:48.578 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.01ms
INFO  Connected to shared process
[I 2019-04-14 19:30:49.649 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 2.54ms
[I 2019-04-14 19:30:51.711 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.12ms
[I 2019-04-14 19:30:51.874 SingleUserNotebookApp log:158] 200 GET /user/ab/code/main.css ([email protected]) 15.56ms
[I 2019-04-14 19:30:54.787 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.10ms
[I 2019-04-14 19:30:57.866 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.15ms
[I 2019-04-14 19:31:00.972 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.90ms
[I 2019-04-14 19:31:04.179 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.15ms
[I 2019-04-14 19:31:07.318 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.91ms
[I 2019-04-14 19:31:10.609 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.95ms
[I 2019-04-14 19:31:14.562 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.92ms
[I 2019-04-14 19:31:18.149 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.93ms
WARN  Uncaught Exception:  Error: ENOENT: no such file or directory, scandir '/home/jovyan/.local/share/code-server/User/workspaceStorage'

Error: ENOENT: no such file or directory, scandir '/home/jovyan/.local/share/code-server/User/workspaceStorage'
    at Timeout.setTimeout [as _onTimeout] (/home/travis/build/codercom/code-server/packages/server/build/bootstrap-fork.js.gz:53331:31)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)

[I 2019-04-14 19:31:23.634 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.13ms
[I 2019-04-14 19:31:28.189 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.05ms
[I 2019-04-14 19:31:33.444 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.14ms
[I 2019-04-14 19:31:38.684 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 0.96ms
[I 2019-04-14 19:31:44.328 SingleUserNotebookApp log:158] 200 GET /user/ab/code/ ([email protected]) 1.06ms
...

Then the url http://0.0.0.0:8000/user/ab/code/shows a black screen.
And the web console complains about a failed websocket handshake.

 INFO  Loading IDE
f2dd3d.bundle.js:53  INFO  Starting socket [1/5]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Starting socket [2/5]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Starting socket [3/5]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Starting socket [4/5]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Starting socket [5/5]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Retrying socket in 1s
f2dd3d.bundle.js:53  INFO  Starting socket [6]...
f2dd3d.bundle.js:53 WebSocket connection to 'ws://0.0.0.0:8000/user/ab/code/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ f2dd3d.bundle.js:53
f2dd3d.bundle.js:53  INFO  Retrying socket in 2s

@dirkcgrunwald
Copy link
Author

dirkcgrunwald commented Apr 14, 2019 via email

@oscar6echo
Copy link

Thx but the code is quite garbled...
Maybe the same paste in the github issue, as opposed to email ?

@betatim
Copy link
Member

betatim commented Apr 15, 2019

https://github.com/betatim/vscode-binder/tree/master is how I setup things to have VS Code in a repo2docker compatible setup. It uses jupyter-server-proxy as well. It currently hangs with a black screen as well, if you look in nthe network inspector of your browser it is because it tries to load some files from the wrong location. I thought that had been fixed a while ago but apparently not. Maybe use a newer version of the vscode server package?

@betatim
Copy link
Member

betatim commented Apr 15, 2019

I updated vscode-binder and it works again.

@dirkcgrunwald
Copy link
Author

I put a sample Dockerfile that builds on jupyter/minimal-notebook at https://gist.github.com/dirkcgrunwald/d3d5669e291e8249f5b4f6ad8b6c1f46

One item not obvious was the need to roll back Tornado to a version prior to 6

That dockerfile builds and can be launched on my mac w/VScode starting up

@betatim
Copy link
Member

betatim commented Apr 15, 2019

@dirkcgrunwald do you know where the source to jupyter-codeserver-proxy is that you reference and install from PyPI's test instance?

@dirkcgrunwald
Copy link
Author

dirkcgrunwald commented Apr 15, 2019 via email

@Chrisjw42
Copy link

Chrisjw42 commented Oct 28, 2019 •
edited
Loading

I can confirm Dirk's method works on our K8s JupyterHub deployment from Dask's notebook dockerfile.

I should note that the bash file installs the conda packages from the above example.

EDIT: The issue is now, newer versions of code-server do not work with this method..

"""
FROM daskdev/dask-notebook:1.1.1

USER root
RUN sudo apt-get update -y -q && sudo apt-get install net-tools

COPY additional-installs.sh /tmp/additional-installs.sh
RUN chmod +x /tmp/installs.sh
USER $NB_UID
RUN /tmp/installs.sh

USER root

RUN jupyter serverextension enable --py --sys-prefix nbserverproxy &&
jupyter labextension install jupyterlab-server-proxy &&
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple jupyter-codeserver-proxy==1.0b1

RUN cd /opt &&
mkdir /opt/code-server &&
cd /opt/code-server &&
wget -qO- https://github.com/codercom/code-server/releases/download/2.1650-vsc1.39.2/code-server2.1650-vsc1.39.2-linux-x86_64.tar.gz | tar zxvf - --strip-components=1

ENV PATH=/opt/code-server:$PATH
"""

@vnijs
Copy link

vnijs commented Oct 29, 2019

@Chrisjw42 Does this start code server by clicking on a launcher in Jupyter? The reason I ask is that I have that working fine with previous versions of code server but for some reason with the last two releases code server won't launch anymore.

@Chrisjw42
Copy link

Chrisjw42 commented Oct 29, 2019 via email

@vnijs
Copy link

vnijs commented Oct 29, 2019

@Chrisjw42 Could you point me to the information about the default args you mentioned? Is this in code-server of jupyter-server-proxy?

@dirkcgrunwald
Copy link
Author

I saw that the coder/code setup changed the arg and confirmed that it causes problems. I'm going to see if there's an easy way to spot the vscode version and then use the new args.

To answer the question, yes, the vscode link appears as a tile in the notebooks pane (see below). I tried to figure out how to get it down with e.g. terminal and editor but ran out of time.

image

@vnijs
Copy link

vnijs commented Oct 29, 2019

Thanks for the comments @dirkcgrunwald. I think I figured out the issue with the code server arguments.

Question: Do you have any documentation on how to add the SQL kernel from your screenshot? I found https://github.com/pbugnion/jupyterlab-sql recently but I'm curious about what you are using

@vnijs vnijs mentioned this issue Nov 11, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dirkcgrunwald @betatim @oscar6echo @vnijs @Chrisjw42