A comprehensive study of Python packages: Package construction and distribution

No public starting in: Python programming time

1. Why do you need to project distribution package?

Usually we used to use pip to install some third-party modules, the installation process is a simple reason, because the module developer for us silently as we did all the complicated work, and this process is 打包.

Packaging is your source code further encapsulation, and all project deployments are pre-arranged, so that users get after the box, do not have to worry about how to deploy the problem (if you do not want a bunch of shining deployment document manually, then).

Whether you're at work, or prepare to write a amateur can be uploaded to PyPI project, you have to learn how to package your project.

Python has developed so many years, project packaging tool has also been very mature. They are what it?

You may have heard of disutils, distutils, distutils2, setuptoolsand so on, it seems very familiar, but very strange, they are all to do with it?

2. ancestor package distribution: distutils

distutils Python is a standard library, from the name is easy to see that it is a distribution (distribute) tool (utlis), it is a tool to distribute packaged official development Python, all subsequent packaging tools, all of which are based on development of.

distutils The essence is written setup.py, it is a module distribution and installation guidance document.

So how do you write setup.py it? This is inside the content very much, I will conduct a detailed analysis in the back, please be patient and read on.

You may not have written setup.py, but you definitely have used setup.py to do some things, such as the following command, we often use it to install the module.

$ python setup.py install

This installation method is to install from source, the corresponding is the same I will be described later by installing binary packages.

3. distribution tools upgrade: setuptools

setuptoolsDistutils is an enhanced version, not included in the standard library. It extended a lot of features, can better help developers create and distribute Python packages. Most users will use the Python setuptools more advanced modules.

the distribute , maybe you have seen it in other places, mention here.

distribute a version of setuptools has a branch, the branch of the reason may be part of developers think setuptools development is too slow. But now, distribute and merge back in setuptools. Therefore, we can say that they are the same thing.

There is also a large package distribution tool is distutils2 , its attempt to try to take advantage of distutils, detuptools and distribute and became the Python standard library of standard tools. However, the plan did not achieve the intended purpose, and is already an abandoned project.

Therefore, setuptools is an excellent and reliable Python package installation and distribution tools.

So how does setuptools installed in a clean environment?

There are two main methods:

• Source Installation: Download the zip package unzip in https://pypi.org/project/setuptools/#files perform python setup.py installthe installation
• By directing install: Download the boot program, it can be used to download or update the latest version of setuptools

$ wget http://peak.telecommunity.com/dist/ez_setup.py

# 安装
$ python ez_setup.py

# 更新，以下两种任选
$ python ez_setup.py –U setuptools
$ pip install -U setuptools

4. easy_install Guide

When you have finished installing setuptools, it has a called easy_installthird-party management tools, which is a big improvement in its distinction of distutils.

Here briefly its use, although it has been used very less.

First, install the package

# 通过包名，从PyPI寻找最新版本，自动下载、编译、安装
$ easy_install pkg_name

# 通过包名从指定下载页寻找链接来安装或升级包
$ easy_install -f http://pythonpaste.org/package_index.html 

# 指定线上的包地址安装
$ easy_install http://example.com/path/to/MyPackage-1.2.3.tgz

# 从本地的 .egg 文件安装
$ easy_install xxx.egg

# 在安装时你可以添加额外的参数
指定安装目录：--install-dir=DIR, -d DIR
指定用户安装：--user

Furthermore is the upgrade package

# 从 pypi 中搜索并升级包
$ easy_install --upgrade pkg_name

# 指定版本进行升级
$ easy_install "SomePackage==2.0"

Finally, removal of the package

$ easy_install -m pkg_name

It should be noted that this deletion is only in easy-install.pth delete a file, it can not use this module in python, but the actual package is still in your computer, to remove completely, you need to manually delete the associated .egg and other documents.

By default, only downloaded from the easy_install pypi related packages, because of this source in a foreign country, the speed of the download package is not ideal, used pip friends will naturally wonder if easy_install can specify the source install it?

The answer is, I can.

Edit Profile /root/.pydistutils.cfg

[easy_install]
index-url=http://mirrors.aliyun.com/pypi/simple/
find-links=http://mirrors.aliyun.com/pypi/simple/

Above only describes some of the common methods easy_install, you want to know more, you can click on the official document: https: //setuptools.readthedocs.io/en/latest/easy_install.html

Sum up by saying: setuptools is a professional tool for official distribution package, if only from the point of view of the installation, its function is really simple. Its greater significance is the distribution of the package is useful, customized program is very high, we are now in progress with the release version of the package it.

The source packages and binary packages What is the difference?

Python distribution package can be divided into two types:

1. Source package in the manner of release

Source package installation process is the first extract, compile, and finally installed, so it is cross-platform, since each installation must be compiled, the relative binary package installation is installed slower.

Source nature of the package is a compressed package that common formats are:

1. Distributed as a binary form

The installation process eliminates the process of binary packages compiled directly extract the installation, the installation speed is faster than the source package.

Because compiled packages for different platforms can not be universal, so when released, subject to prior compiled package multiple platforms.

Common format binary packages are:

6. eggs and wheels What is the difference?

Egg format was introduced in 2004 by the setuptools, while Wheel format is defined by the PEP427 in 2012. Wheel appearance of a replacement for Egg, its essence is a zip package, which is now considered the standard Python format binary packages.

The following are the main difference between Wheel and the Egg:

• Wheel of PEP427 to have an official definition and Egg PEP is not defined
• Wheel is a distribution format, packed format. The Egg is both a distribution format, installation is also a run-time format, and can be directly import
• Wheel file will not contain .pyc file
• Wheel PEP376 use and compatible .dist-info directory, Egg using .egg-info directory
• Wheel has a richer naming rules.
• There is a version of the Wheel. Each file contains Wheel wheel versions of the specification and implementation of packaging
• Wheel sysconfig path type is managed in-house, and therefore also more likely to turn to other formats

wheel package can be installed by pip, but need to install the wheel module, and then use pip command.

$ pip install wheel
$ pip wheel --wheel-dir=/local/wheels pkg

7. Super explain in detail the preparation of setup.py?

Packaged distribution and most crucial step is to write setup.pythe file.

The following is an example of using a simple setup.py

from setuptools import setup, find_packages

setup(
    name="mytest",
    version="1.0",
    author="wangbm",
    author_email="[email protected]",
    description="Learn to Pack Python Module  -->公众号：Python编程时光",
    
    # 项目主页
    url="http://python-online.cn/", 
    
    # 你要安装的包，通过 setuptools.find_packages 找到当前目录下有哪些包
    packages=find_packages()
)

Next, I will gradually expand this setup function, adding more parameters, so that you can understand what things can do the setup function.

Programs classified information

classifiersParameter Description packet classification. Category list of all supported See: https: //pypi.org/pypi% 3Aaction = list_classifiers?

Example:

from setuptools import setup, find_packages

setup(
    classifiers = [
        # 发展时期,常见的如下
        #   3 - Alpha
        #   4 - Beta
        #   5 - Production/Stable
        'Development Status :: 3 - Alpha',

        # 开发的目标用户
        'Intended Audience :: Developers',

        # 属于什么类型
        'Topic :: Software Development :: Build Tools',

        # 许可证信息
        'License :: OSI Approved :: MIT License',

        # 目标 Python 版本
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
    ]
)

About distribute documents

from setuptools import setup, find_packages


setup(
    name="mytest",
    version="1.0",
    author="wangbm",
    author_email="[email protected]",
    description="Learn to Pack Python Module",
    url="http://python-online.cn/", 
    packages=find_packages(),
    
    # 安装过程中，需要安装的静态文件，如配置文件、service文件、图片等
    data_files=[
        ('', ['conf/*.conf']),
        ('/usr/lib/systemd/system/', ['bin/*.service']),
               ],
    
    # 希望被打包的文件
    package_data={
        '':['*.txt'],
        'bandwidth_reporter':['*.txt']
               },
    # 不打包某些文件
    exclude_package_data={
        'bandwidth_reporter':['*.txt']
               }
)

In addition to the above parameters, you can also use a named MANIFEST.infile to control the distribution of documents.

It follows is a MANIFEST.insample:

include *.txt
recursive-include examples *.txt *.py
prune examples/sample?/build

These configuration, sets out the following points

• Txt file name suffix at the root of all, will distribute
• examples directory and txt in the root directory, py files are distributed
• Path matching the examples / sample? / Build does not distribute

MANIFEST.in And needs to be placed under the top-level directory setup.py the same level, setuptools will automatically read the file.

About dependencies to download and install

from setuptools import setup, find_packages


setup(
    ...
    
    # 表明当前模块依赖哪些包，若环境中没有，则会从pypi中下载安装
    install_requires=['docutils>=0.3'],
    
    # setup.py 本身要依赖的包，这通常是为一些setuptools的插件准备的配置
    # 这里列出的包，不会自动安装。
    setup_requires=['pbr'],
    
    # 仅在测试时需要使用的依赖，在正常发布的代码中是没有用的。
    # 在执行python setup.py test时，可以自动安装这三个库，确保测试的正常运行。
    tests_require=[
        'pytest>=3.3.1',
        'pytest-cov>=2.5.1',
    ],
    
    # 用于安装setup_requires或tests_require里的软件包
    # 这些信息会写入egg的 metadata 信息中
    dependency_links=[
        "http://example2.com/p/foobar-1.0.tar.gz",
    ],
    
    # install_requires 在安装模块时会自动安装依赖包
    # 而 extras_require 不会，这里仅表示该模块会依赖这些包
    # 但是这些包通常不会使用到，只有当你深度使用模块时，才会用到，这里需要你手动安装
    extras_require={
        'PDF':  ["ReportLab>=1.2", "RXP"],
        'reST': ["docutils>=0.3"],
    }
)

On install_requires, there is a common representation of the following five:

1. 'argparse'Containing only the package name. This form only checks the existence of the package, do not check the version. Convenient, but not conducive to control risk.
2. 'setuptools==38.2.4'Specify version. This form of the risk to a minimum to ensure that the development, testing and deployment of the same version, there are no surprises. The disadvantage is not conducive to the update, each update will require changes to the code.
3. 'docutils >= 0.3'It is more commonly used form. When a library is more likely to trust, this form can be automatically kept up to date version.
4. 'Django >= 1.11, != 1.11.1, <= 2'This is a more complex form. As shown in this example, to ensure that the major version Django between 1.11 and 2, i.e. 1.11.x; and, excluded are known in version 1.11.1 (example only). For some large, complex database, this form is the most appropriate.
5. 'requests[security, socks] >= 2.18.4'It is the inclusion of additional optional dependent manner. Normal installation requests automatically installs its install_requiresdesignated dependent, but will not be installed securityand socksboth sets of dependence. The two sets are defined in dependence on its extras_requirein. This form, with depth when using certain libraries.

Restrictions on the installation environment

Some libraries are not in so Python versions are applicable, if a library is installed in a non-compatible Python environment, in theory, it should not be in use when the error, but it should make it fail in the installation process, the installation prompts ban .

This function can be used python_requiresto achieve.

setup(
    ...
    python_requires='>=2.7, <=3',
)

Generate executable file distribution

from setuptools import setup, find_packages


setup(
    name="mytest",
    version="1.0",
    author="wangbm",
    author_email="[email protected]",
    description="Learn to Pack Python Module",
    url="http://python-online.cn/", 
    packages=find_packages(),
   
    # 用来支持自动生成脚本，安装后会自动生成 /usr/bin/foo 的可执行文件
    # 该文件入口指向 foo/main.py 的main 函数
    entry_points={
        'console_scripts': [
            'foo = foo.main:main'
        ]
    },
    
    # 将 bin/foo.sh 和 bar.py 脚本，生成到系统 PATH中
    # 执行 python setup.py install 后
    # 会生成 如 /usr/bin/foo.sh 和 如 /usr/bin/bar.py
    scripts=['bin/foo.sh', 'bar.py']
)

Some of the scripts in the script above there shand pysuffix, so after installation, setuptools will move intact to / usr / bin and add executable permissions.

If you want to make some changes to these files again, remove the excess of such suffixes, you can do so

from setuptools.command.install_scripts import install_scripts

class InstallScripts(install_scripts):

    def run(self):
        setuptools.command.install_scripts.install_scripts.run(self)

        # Rename some script files
        for script in self.get_outputs():
            if basename.endswith(".py") or basename.endswith(".sh"):
                dest = script[:-3]
            else:
                continue
            print("moving %s to %s" % (script, dest))
            shutil.move(script, dest)

setup(
    ...
    scripts=['bin/foo.sh', 'bar.py'],
    
    cmdclass={
        "install_scripts": InstallScripts
    }
)

ext_modules

ext_modulesParameters used to construct the C and C ++ extensions extended package. Extension which is a list of instances, each of Extension example describes a separate expansion module, the expansion module may set the extension package name, header files, source files, libraries and their paths, macros and editing parameters. Such as:

setup(
    # other arguments here...
    ext_modules=[
        Extension('foo',
                  glob(path.join(here, 'src', '*.c')),
                  libraries = [ 'rt' ],
                  include_dirs=[numpy.get_include()])
    ]
)

Learn more about refer to: https: //docs.python.org/3.6/distutils/setupscript.html#preprocessor-options

setup.py many parameters, it is possible to write a setup.py without the aid of documentation did not seem so simple. To note, I compiled some common setup function parameters:

更多参数可见：https://setuptools.readthedocs.io/en/latest/setuptools.html

8. 打包辅助神器PBR 是什么？

pbr 是 setuptools 的辅助工具，最初是为 OpenStack 开发(https://launchpad.net/pbr)，基于d2to1。

pbr 会读取和过滤setup.cfg中的数据，然后将解析后的数据提供给 setup.py 作为参数。包含如下功能：

1. 从git中获取Version、AUTHORS and ChangeLog信息
2. Sphinx Autodoc。pbr 会扫描project，找到所有模块，生成stub files
3. Requirements。pbr会读取requirements.txt，生成setup函数需要的install_requires/tests_require/dependency_links

这里需要注意，在 requirements.txt 文件的头部可以使用：--index https://pypi.python.org/simple/，这一行把一个抽象的依赖声明如 requests==1.2.0 转变为一个具体的依赖声明 requests 1.2.0 from pypi.python.org/simple/

1. long_description。从README.rst, README.txt or README file中生成long_description参数

使用pbr很简单：

from setuptools import setup

setup(
    setup_requires=['pbr'],
    pbr=True,
)

使用pbr时，setup.cfg中有一些配置。在[files]中，有三个key：
packages:指定需要包含的包，行为类似于setuptools.find_packages
namespace_packages:指定namespace packages
data_files: 指定目的目录和源文件路径，一个示例：

[files]
data_files =
    etc/pbr = etc/pbr/*
    etc/neutron =
        etc/api-paste.ini
        etc/dhcp-agent.ini
    etc/init.d = neutron.init

[entry_points] 段跟 setuptools 的方式相同。

到此，我讲了三种编写使用 setup.py 的方法

• 使用命令行参数指定，一个一个将参数传递进去（极不推荐）
• 在 setup.py 中的setup函数中指定（推荐使用）
• 使用 pbr ，在 setup.cfg 中指定（易于管理，更推荐）

9. 如何使用 setup.py 构建包

1、构建源码发布包。

用于发布一个 Python 模块或项目，将源码打包成 tar.gz （用于 Linux 环境中）或者 zip 压缩包（用于 Windows 环境中）

$ python setup.py sdist

那这种包如何安装呢？

答案是，使用下一节即将介绍的 setuptools 中提供的 easy_install 工具。

$ easy_install xxx.tar.gz

使用 sdist 将根据当前平台创建默认格式的存档。在类 Unix 平台上，将创建后缀后为 .tar.gz 的 gzip 压缩的tar文件分发包，而在Windows上为 ZIP 文件。

当然，你也可以通过指定你要的发布包格式来打破这个默认行为

$ python setup.py sdist --formats=gztar,zip

你可以指定的格式有哪些呢？

创建一个压缩的tarball和一个zip文件。可用格式为：

对以上的格式，有几点需要注意一下：

• 在版本3.5中才添加了对 xztar 格式的支持
• zip 格式需要你事先已安装相应的模块：zip程序或zipfile模块（已成为Python的标准库）
• ztar 格式正在弃用，请尽量不要使用

另外，如果您希望归档文件的所有文件归root拥有，可以这样指定

python setup.py sdist --owner=root --group=root

2、构建二进制分发包。

在windows中我们习惯了双击 exe 进行软件的安装，Python 模块的安装也同样支持 打包成 exe 这样的二进制软件包。

$ python setup.py bdist_wininst

而在 Linux 中，大家也习惯了使用 rpm 来安装包，对此你可以使用这条命令实现 rpm 包的构建

$ python setup.py bdist_rpm

若你喜欢使用 easy_install 或者 pip 来安装离线包。你可以将其打包成 egg 包

$ python setup.py bdist_egg

若你的项目，需要安装多个平台下，既有 Windows 也有 Linux，按照上面的方法，多种格式我们要执行多次命令，为了方便，你可以一步到位，执行如下这条命令，即可生成多个格式的进制包

$ python setup.py bdist

10. 如何使用 setup.py 安装包

正常情况下，我们都是通过以上构建的源码包或者二进制包进行模块的安装。

但在编写 setup.py 的过程中，可能不能一步到位，需要多次调试，这时候如何测试自己写的 setup.py 文件是可用的呢？

这时候你可以使用这条命令，它会将你的模块安装至系统全局环境中

$ python setup.py install

如若你的项目还处于开发阶段，频繁的安装模块，也是一个麻烦事。

这时候你可以使用这条命令安装，该方法不会真正的安装包，而是在系统环境中创建一个软链接指向包实际所在目录。这边在修改包之后不用再安装就能生效，便于调试。

$ python setup.py develop

11. 如何发布包到 PyPi？

通过上面的学习，你一定已经学会了如何打包自己的项目，若你觉得自己开发的模块非常不错，想要 share 给其他人使用，你可以将其上传到 PyPi （Python Package Index）上，它是 Python 官方维护的第三方包仓库，用于统一存储和管理开发者发布的 Python 包。

如果要发布自己的包，需要先到 pypi 上注册账号。然后创建 ~/.pypirc 文件，此文件中配置 PyPI 访问地址和账号。如的.pypirc文件内容请根据自己的账号来修改。

典型的 .pypirc 文件

[distutils]
index-servers = pypi

[pypi]
username:xxx
password:xxx

然后使用这条命令进行信息注册，完成后，你可以在 PyPi 上看到项目信息。

$ python setup.py register

注册完了后，你还要上传源码包，别人才使用下载安装

$ python setup.py upload

或者也可以使用 twine 工具注册上传，它是一个专门用于与 pypi 进行交互的工具，详情可以参考官网：https://www.ctolib.com/twine.html，这里不详细讲了。

参考文章

• http://blog.konghy.cn/2018/04/29/setup-dot-py/
• https://note.qidong.name/2018/01/python-setup-requires/