A standard Python environment directory structure (created by Anaconda)

# base python
/anaconda/

# user created environment
/anaconda/env/

Each python environment follow the similar structure (including anaconda's base one)

PATH/bin
    (include executable binary, like conda, pip, python_intepreter, etc)
PATH/lib
    /python_{version_number}
        /site-package (both conda and pip will install packages to this directory)

• Q: under conda environment, where does pip install packages to?

  • A: within activate conda environment's corresponding /lib/python{version}/site-package/ directory

• Q: if not using conda, where is Mac's original default python environment?

  • A: no idea ...

• Q: conda by default install environment to system directory /anaconda (same as //anaconda). Can I change it to user's local directory?

  • A: tbd

Anaconda

Create/change environment

Create environment

conda create --name snowflakes pandas
conda create --name bunnies python=3 astroid babel

create Python 3 environment

$ conda create -n py35 python=3.5 anaconda
$ source activate py35
$ source deactivate

Here, the ‘py35’ is the name of the environment you want to create, and ‘anaconda’ is the meta-package that includes all of the actual Python packages comprising the Anaconda distribution. When creating a new environment and installing Anaconda, you can specify the exact package and Python versions, for example, numpy=1.7 or python=3.5.

Create Python 2 environment

$ conda create -n py27 python=2.7 anaconda
$ source activate py27
$ source deactivate

TIP: Install all the programs you will want in this environment at the same time. Installing one program at a time can lead to dependency conflicts.

Switch environment

source activate snowflakes

List all environments

conda info --envs
conda env list

Clone environment

conda create --name flowers --clone snowflakes

Remove environment

conda remove --name flowers --all

Export environment file and create env from environment file

First switch to the environment you want to export, then:

conda env export > environment.yml
conda env create -f environment.yml

Find system's python package directory:

>>> import site; site.getsitepackages()
['/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/lib/site-python']

/usr/lib/python2.7/site-packages seems to be important ?

if I using sudo pip install virtualenv it will install virtualenv package into /usr/bin

Use pip to manage environment

Check package information (including the installed location)

pip show pandas

• ** pip is the recommended installer **
  Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state.
  Update pip:

pip install -U pip setuptools

Normal Install or update a package

To install the latest version of “SomeProject”:

pip install SomeProject

To install a specific version:

pip install SomeProject==1.4

To install greater than or equal to one version and less than another:

pip install SomeProject>=1,<2

To install a version that’s “compatible” with a certain version: [4]

pip install SomeProject~=1.4.2

In this case, this means to install any version “==1.4.*” version that’s also “>=1.4.2”.

To update a package:

pip install -U SomePackage
pip install --upgrade SomeProject

Source Distributions vs Wheels

• pip can install from either Source Distributions (sdist) or Wheels, but if both are present on PyPI, pip will prefer a compatible wheel.
• Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist), especially when a project contains compiled extensions.

Use pip with proxy setting

pip install --proxy user:pass@proxy:port works
pip search --proxy user:pass@proxy:port dont work

Installing to the User Site

To install packages that are isolated to the current user, use the --user flag:

pip install --user SomeProject

Requirements files

Install a list of requirements specified in a Requirements File.

pip install -r requirements.txt

Installing from a local src tree

Installing from local src in Development Mode, i.e. in such a way that the project appears to be installed, but yet is still editable from the src tree.

pip install -e <path>

You can also install normally from src

pip install <path>

Installing from local archives

Install a particular source archive file.

pip install ./downloads/SomeProject-1.0.4.tar.gz

Install from a local directory containing archives (and don’t check PyPI)

pip install --no-index --find-links=file:///local/dir/ SomeProject
pip install --no-index --find-links=/local/dir/ SomeProject
pip install --no-index --find-links=relative/dir/ SomeProject

virtualenv

$ virtualenv --no-site-packages env_name
source env_name/bin/activate

• say we create the virtual directory 'env_name':
  the python site-package will be under:
  env_name/lib/python2.7/site-packages/

Python Packaging Ecosystem

• A package is simply a directory with an init.py file inside it.
• distutils was created to install packages into the PYTHONPATH with little difficulty. The PYTHONPATH, also sys.path in code, is a list of locations to look for Python packages.
• if PYTHONPATH is set to /www/python:/opt/py, the search path will begin with ['', '/www/python', '/opt/py', ...].

>>> import sys
>>> sys.path
['',   # the current directory, represented by a null string
 '/usr/local/lib/python2.6',
 '/usr/local/lib/python2.6/site-packages',
 ...]
>>> import mypackage  # this package is at current directory
>>> mypackage.__file__
'mypackage/__init__.py'

• The convention way of manually installing packages is to put them in the .../site-packages/ directory. But one may want to install Python modules into some arbitrary directory.

• The most convenient way is to add a path configuration file to a directory that’s already in Python’s path, which could be the .../site-packages/ directory. Path configuration files have an extension of .pth, and each line must contain a single path that will be appended to sys.path.

• Finally, sys.path is just a regular Python list, so any Python application can modify it by adding or removing entries.

• Current python packaging tools

  • Distribute
  • pip, which replace easy_install
  • virtualenv

• Distribute is a collection of enhancements to the Python standard library module: distutils (for Python 2.3.5 and up on most platforms; 64-bit platforms require a minimum of Python 2.4) that allows you to more easily build and distribute Python packages, especially ones that have dependencies on other packages. Distribute was created because the Setuptools package is no longer maintained.

python directory structure

Linux

use default python

'/usr/lib/python2.7/site-packages', '/usr/lib/site-python'

use anaconda

use virtualenv

Mac

use default python

use anaconda

use virtualenv

Python Package

How to create Python Package (not very useful)

This is better : https://packaging.python.org/distributing/

Install Package

https://packaging.python.org/installing/

It’s important to note that the term “package” in this context is being used as a synonym for a distribution (i.e. a bundle of software to be installed), not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.

How to create Python Package