Change in behavior launching ipython notebooks from anacona environment

[jhamman]

Cross posting this issue from ContinuumIO/anaconda-issues#461.

---

Previously, I have been able to run ipython notebooks from Anaconda environments using this workflow:

conda create -n test python=3.4 ipython-notebook
source activate test
ipython notebook

and then in a new notebook

import sys
print(sys.executable)

would yield

/Users/jhamman/anaconda/envs/test/bin/python

Recently, this behavior has changed (I've seen it on Mac OS and Linux for Python 2.7 and 3.4). Now I get the root anaconda python executable:

/Users/jhamman/anaconda/bin/python

Consequently, I'm not able to import/use packages in my environments from ipython notebooks. If I start an ipython session in the terminal, this problem does NOT exist. So this may just be an issue with the notebook kernels.

---

After discussion with @jasongrout, it seems like this change came in the 4.0 release.

@jasongrout and I have discussed methods for allowing the environment executable to be an available (default) kernel when launching a notebook from a virtual environment.

The first idea mentioned was ipython notebook --defaultkernel, where defaultkernel indicates that the user would like to use the python executable that is launching the notebook.

cc @minrk @jasongrout

xref: ContinuumIO/anaconda-issues#461 ipython/ipython#8880

[jdfreder]

I'm not familiar with Anaconda, because I do not use it. However, try python -m IPython kernelspec install-self --user, but replace python with the python from the conda env. I think this will make the IPython kernel launch in that env's Python, but when you remove the conda env, it may go bonkers. You may even get an error regarding "IPython" not existing, which would suggest conda wasn't copying IPython into the new env.

[jdfreder]

+1 to the defaultkernel idea

[jasongrout]

@jdfreder - I'm not sure your idea will work with Conda. Basically, to switch conda environments, you need to add the appropriate directory to your path. The thing about just invoking the python in the correct location is that anything else may use the wrong scripts since the path isn't set correctly.

By the way, I think defaultkernel is a terrible name, but I do think that making it very easy to have a user start up a notebook in an environment with the assumption that the python running the notebook will also run the kernel will lower the barrier a lot.

[damianavila]

First of all:

damian-S400CA :: ~ :: 
$ conda create -n test python=3.4 ipython-notebook --yes
Fetching package metadata: ......
Solving package specifications: .
Package plan for installation in environment /home/damian/miniconda3/envs/test:

The following NEW packages will be INSTALLED:

    certifi:          14.05.14-py34_0
    decorator:        4.0.4-py34_0   
    ipykernel:        4.0.3-py34_0   
    ipython:          4.0.0-py34_0   
    ipython-notebook: 4.0.4-py34_0   
    ipython_genutils: 0.1.0-py34_0   
    jinja2:           2.8-py34_0     
    jsonschema:       2.4.0-py34_0   
    jupyter_client:   4.0.0-py34_0   
    jupyter_core:     4.0.6-py34_0   
    libsodium:        0.4.5-0        
    markupsafe:       0.23-py34_0    
    mistune:          0.7.1-py34_0   
    nbconvert:        4.0.0-py34_0   
    nbformat:         4.0.0-py34_0   
    notebook:         4.0.5-py34_0   
    openssl:          1.0.1k-1       
    path.py:          8.1.2-py34_0   
    pexpect:          3.3-py34_0     
    pickleshare:      0.5-py34_0     
    pip:              7.1.2-py34_0   
    ptyprocess:       0.4-py34_0     
    pygments:         2.0.2-py34_0   
    python:           3.4.3-1        
    pyzmq:            14.7.0-py34_0  
    readline:         6.2-2          
    setuptools:       18.3.2-py34_0  
    simplegeneric:    0.8.1-py34_0   
    sqlite:           3.8.4.1-1      
    system:           5.8-2          
    terminado:        0.5-py34_0     
    tk:               8.5.18-0       
    tornado:          4.2.1-py34_0   
    traitlets:        4.0.0-py34_0   
    wheel:            0.26.0-py34_1  
    xz:               5.0.5-0        
    zeromq:           4.0.5-0        
    zlib:             1.2.8-0        

Linking packages ...
[      COMPLETE      ]|###################################################| 100%
#
# To activate this environment, use:
# $ source activate test
#
# To deactivate this environment, use:
# $ source deactivate
#
damian-S400CA :: ~ :: 
$ source activate test
discarding /home/damian/miniconda3/bin from PATH
prepending /home/damian/miniconda3/envs/test/bin to PATH
(test)damian-S400CA :: ~ :: 
$ ipython notebook
[W 18:50:16.003 NotebookApp] ipywidgets package not installed.  Widgets are unavailable.
[I 18:50:16.014 NotebookApp] The port 8888 is already in use, trying another random port.
[I 18:50:16.023 NotebookApp] Serving notebooks from local directory: /home/damian
[I 18:50:16.023 NotebookApp] 0 active kernels 
[I 18:50:16.023 NotebookApp] The IPython Notebook is running at: http://localhost:8889/
[I 18:50:16.024 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Created new window in existing browser session.
[I 18:50:28.001 NotebookApp] Creating new notebook in 
[W 18:50:29.476 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20151007185015 (127.0.0.1) 30.10ms referer=http://localhost:8889/notebooks/Untitled.ipynb?kernel_name=python3
[I 18:50:29.743 NotebookApp] Kernel started: 86bee2e1-171f-452f-9ba2-97f34e11f209
[I 18:52:29.719 NotebookApp] Saving file at /Untitled.ipynb

and in the notebook:

import sys
sys.executable
Out[1]:
'/home/damian/miniconda3/envs/test/bin/python3'

which is reasonable becuase the native kernel lives in the environment...

So... it is weird that your kernel is pointing to the root environment...

Do you have any kernels in the JUPYTER_DATA_DIR directory?

Also:

try python -m IPython kernelspec install-self --user, but replace python with the python from the conda env

Here @minrk explained a little bit more if you need more insight: ipython/ipython#8877

[jhamman]

Yes. I do have three kernels in the JUPYTER_DATA_DIR.

ls /Users/jhamman/Library/Jupyter/kernels/
bash/    python2/ python3/

so the python -m IPython kernelspec install-self --user "works" except it means I will have to continually be creating new kernels, each time I create a new conda environment.

[damianavila]

And I guess the kernel.json inside the python2 and python3 folders are pointing to root (can you post the content?)... try removing those folders (python2 and python3) and the native kernels should take place...

[minrk]

The notebook no longer relies on IPython, which provides one of many kernels. For convenience the notebook does, however, add the IPython kernel for the notebook server's env if no Python kernel is already registered. That IF is the source of frustration, if you want to be doing it all the time, though. The idea being that changing the env of the notebook server should not change the env of your kernels. Similarly, making an env with a kernel doesn't require installing the notebook in that env, just the kernel. This is my habit - I frequently make envs and install IPython in them, and register a kernelspec for the env. What I don't do is install the notebook server multiple times.

We could also make the in-env kernel available unconditionally with a different name, though. It's more special treatment of IPython than we wanted, but it may be appropriate to put this behavior back.

There's another hack you can do, and I'm not sure if/how we can make it more convenient to do so: Change the absolute path for Python in the kernel.json to just python instead of /path/to/bin/python3. If you do this, the kernel will always use the first python on PATH in the notebook server's process, which is generally the env's Python.

[jhamman]

@minrk - This is making sense to me now. I'm going to try a few things out and I will let you all know how it goes.

cc @jgaski @orianac

[michaelaye]

@minrk Are you using conda? Does that mean you have a notebook server installed in conda root and not elsewhere? If so, how do you get the PATHs right when you are working in an env? Pity this has become so brain-braking...

[minrk]

@michaelaye yes, I'm using conda, and the notebook (latest master) is in the root env. I also have an env called stable, which has stable versions of everything, and various others. I also have kernels in several conda and virtualenvs. If you do the naïve approach above, which just hardcodes sys.executable, the PATH env will not be modified. I have, however, been playing with a2km which fully activates envs, rather than just specifying the executable.

[michaelaye]

So, the kernel text is so simple, I just added what I needed into my jupyter --data-dir/kernels. Just one piece missing: You had to add the conda root into your general PATH env so that its Jupyter notebook is always findable, despite working in some conda env, correct?

[kerwin]

+1

[michaelaye]

as an update for anybody who struggles with this, it seems that the conda package nb_conda_kernels has established a quasi-default status, it automatically finds kernels in conda environments and offers them via the kernel menu in the jupyter notebook, so i'm not struggling with this anymore.

[Carreau]

Thanks @michaelaye for the update; it's true that nb_conda_kernels help a lot with this.

Do you think the issue can be closed ?

[michaelaye]

for me yes, but @jhamman should decide.

[Carreau]

for me yes, but @jhamman should decide.

Yes, I was imprecise in my question, I was addressing to everyone but it was unclear.
Thanks for pointing that out.

[mshap09]

When I run sys.executable in the command shell it returns: C:\Users\username\AppData\Local\Continuum\Anaconda2\envs\py27\python.exe

However, when I run sys.executable in Jupyter Notebook I get:
C:\Users\username\AppData\Local\Continuum\Anaconda2\python.exe

Then, when I run jupyter kernelspec list in the command shell it returns:
python2 C:\Users\username\AppData\Local\Continuum\Anaconda2\lib\site-packages\ipykernel\resources

But, this location does not have a JSON file in it. Any suggestions as to how I can get the two paths to match?

FYI, I am working in a Windows environment.

[takluyver]

Two options:

• Set up a kernelspec for that environment
• Or use Anaconda's nb_conda_kernels extension