Python

From CsWiki
Revision as of 17:42, 28 March 2018 by Irush (Talk | contribs) (Created page with " Currently there are 2 versions of python supported on the linux machines: '''python2.7''' and '''python3.5'''. Other versions might be installed but not supported. Some or m...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Currently there are 2 versions of python supported on the linux machines: python2.7 and python3.5.

Other versions might be installed but not supported. Some or most of the packages might not work with them.

A lot of packages are already installed globally and do not need to be installed manually. However if some packages are needed and missing there are several way to install them:

  • Global installation
  • virtualenv
  • pip --user

Global installation

Installations that benefits many users can be installed globally by the system. A request can be sent to system@cs.huji.ac.il.

Not all packages are simple to install globally, and version changes can be more complicated, so installation requests might be denied or delayed. As such, course TA's should request such installations BEFORE the start of the semester.

Some packages are installed using the modules system and aren't active by default, or have several versions installed. This includes packages like tensorflow and opencv.

virtualenv

In case a package can't be installed globally, the preferred method is using virtualenv. virtualenv allows creating an isolated (or semi-isolated) python environment which the user can install his own packages without requiring super-user.

A new virtualenv should be created per platform/version/architecture where the code needs to run.

Creating virtualenv

virtualenv can take a lot of disk space, so it should be created on a lab storage space and not on the home directory.

To create a virtualenv in a lab's storage:

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv

To create with specific python version

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv --python python3

By default, the environment created will be isolated from all globally installed packages (even those installed by the modules system). To use them:

virtualenv /cs/labs/<supervisor>/<login>/my-first-venv --system-site-packages

Using virtualenv

To use the virtualenv, it needs to be activated:

source /cs/labs/<supervisor>/<login>/my-first-venv/bin/activate.csh

For users using sh, bash or zsh, or withing sh or bash scripts:

. /cs/labs/<supervisor>/<login>/my-first-venv/bin/activate

After activating, any python related commands will work from the virtual env.

To install packages inside the virtualenv, it's best to use pip, but without the --user option.

To exit the virtualenv:

deactivate

pip

By default installing packages using pip will try to install them globally and will fail.

pip --user

This is the least preferred method to install python packages. It will install the packages in ~/.local/lib/python<version>/. There are several issues with this approach:

  1. It takes space in the valuable home directory. And usually users forget they installed packages there.
  2. It remains there and effects all python runs of the user. Once it gets out of date, users seldom understand where the bad package is coming from.
  3. The home directory is shared between several platforms: linux x86, linux ppc and mac os x. Installing a python package for one platform will usually not work on different platform or even on different version of the same platform (but they will try and fail to use them).

To avoid accidentally installing packages in such a way, it's suggested to make the directory not writable, i.e. run:

mkdir -p ~/.local/lib/python{2.7,3.5}/site-packages
chmod a-w ~/.local/lib/python{2.7,3.5}/site-packages

pip inside virtualenv

After activating a virtualenv, pip can be used normally to install packages there. e.g.

pip install that-package-I-want

To uninstall a package:

pip uninstall that-package-I-do-not-want

If the virtualenv was activated with the --system-site-packages option, and an upgrade of a globally installed package is wanted:

pip install --force-reinstall --ignore-installed that-package-I-want-upgraded

To create a new virtualenv with the same package as a different virtualenv (e.g. when building a new virtualenv for a different platform), one can use the freeze/requirements method. In the old virtualenv:

pip freeze > /somewhere/to/save/requirements.txt

In the new virtualenv:

pip install -r /somewhere/to/save/requirements.txt

Be careful not to use pip freeze in virtualenv created with the --system-site-packages option. As it will give a very long unhelpful list.