Python

From CsWiki
Jump to: navigation, search

Currently there are 2 versions of python supported on the linux machines: python2.7 and python3.7.

Other versions might be installed but not supported. Some or most of the packages might not work with them.

A lot of packages are already installed globally and do not need to be installed manually. However if some packages are needed and missing there are several way to install them:

  • Global installation
  • virtualenv
  • pip --user

Global installation

Installations that benefits many users can be installed globally by the system. A request can be sent to system@cs.huji.ac.il.

Not all packages are simple to install globally, and version changes can be more complicated, so installation requests might be denied or delayed. As such, course TA's should request such installations BEFORE the start of the semester.

Some packages are installed using the modules system and aren't active by default, or have several versions installed. This includes packages like tensorflow and opencv.

virtualenv

In case a package can't be installed globally, the preferred method is using virtualenv. virtualenv allows creating an isolated (or semi-isolated) python environment which the user can install his own packages without requiring super-user privileges.

A new virtualenv should be created per platform/version/distribution/architecture where the code needs to run.

Creating virtualenv

virtualenv can take a lot of disk space, so it should be created on a lab storage space and not on the home directory.

To create a virtualenv in a lab's storage:

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv

To create with specific python version

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv --python python3

By default, the environment created will be isolated from all globally installed packages (even those installed by the modules system). To use globally installed packages, the virtualenv needs to be created with --system-site-packages:

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv --system-site-packages

Using virtualenv

To use the virtualenv, it needs to be activated:

source /cs/labs/<supervisor>/<login>/my-first-venv/bin/activate.csh

For users using sh, bash or zsh, or when writing sh or bash scripts:

. /cs/labs/<supervisor>/<login>/my-first-venv/bin/activate

After activating, any python related commands will work from the virtual env.

To install packages inside the virtualenv, it's best to use pip, but without the --user option:

pip install <package>

Packages that comes from source with a setup.py, should also work properly inside a virtualenv. I.e.:

python setup.py install

should work (without needing sudo).


To exit the virtualenv:

deactivate

Using virutalenv within PyCharm

  • Choose "File -> Settings"
  • Under "Project interpreter", click on the gear icon -> Add
  • Choose one of the existing virtualenvs

Once the virtualenv is added, it can be chosen in the interpreters list

Using virtualenv within Jupyter Notebook

Jupyter is installed on the system, so there's no need to install it using virtualenv. In case that you want to use the virtualenv within the jupyter notebook, you have to install "kernel". This is done as follow:

  1. Activate your virtual environment
  2. pip install ipykernel
  3. Within the virtual environment, add a new kernel:
ipython kernel install --user --name=MyVirtualEnv --display-name="My virtual Environment"
  1. Now the new kernel ("My virtual Environment") has to be shown in the jupyter notebook

Packages installation

pip

By default installing packages using pip will try to install them globally and will fail.

pip --user

This is the least preferred method to install python packages. It will install the packages in ~/.local/lib/python<version>/. There are several issues with this approach:

  1. It takes space in the valuable home directory. And usually users forget they installed packages there.
  2. It remains there and affects all python runs of the user. Once it gets out of date, users seldom understand where the bad package is coming from.
  3. The home directory is shared between several platforms: linux x86, linux ppc and mac os x. Installing a python package for one platform will usually not work on different platform or even on different version of the same platform (but they will try and fail to use them).

To avoid accidentally installing packages in such a way, it's suggested to make the directory not writable, i.e. run:

mkdir -p ~/.local/lib/python{2.7,3.7}/site-packages
chmod a-w ~/.local/lib/python{2.7,3.7}/site-packages

pip inside virtualenv

After activating a virtualenv, pip can be used normally to install packages there. e.g.

pip install that-package-I-want

To uninstall a package:

pip uninstall that-package-I-do-not-want

If the virtualenv was activated with the --system-site-packages option, and an upgrade of a globally installed package is wanted:

pip install --force-reinstall --ignore-installed that-package-I-want-upgraded

To create a new virtualenv with the same package as a different virtualenv (e.g. when building a new virtualenv for a different platform), one can use the freeze/requirements method. In the old virtualenv:

pip freeze > /somewhere/to/save/requirements.txt

In the new virtualenv:

pip install -r /somewhere/to/save/requirements.txt

Important: Don't use pip freeze in virtualenv created with the --system-site-packages option. As it will give a very long unhelpful list.

Installing packages within PyCharm

Adding packages within PyCharm is done by entering the Project settings->Project interpreter, and cliking on the + sign beside the packages list/ You have three options when trying to add python packages using this method:

  1. Installing globally - this option will fail as described on pip
  2. Installing using user space by checking the "Install to user's site packaged directory - this method is not recommended as described on pip --user
  3. Installing into virtualenv - this is the preferred method. It's done by using virtualenv as described above, and clicking on the + sign beside the packages list. The chosen package would be installed inside the virtualenv