Combining pipdeptree and freeze
Let’s see what happens when we use pipdeptree and freeze altogether,
Command:
$pipdeptree --freeze
Output:
altair==4.1.0 entrypoints==0.3 Jinja2==2.11.2 MarkupSafe==1.1.1 jsonschema==3.2.0 attrs==19.3.0 pyrsistent==0.16.0 six==1.15.0 setuptools==41.2.0 six==1.15.0 numpy==1.18.4 pandas==1.0.4 numpy==1.18.4 python-dateutil==2.8.1 six==1.15.0 pytz==2020.1 toolz==0.10.0 docutils==0.15.2 jmespath==0.10.0 Mako==1.1.3 MarkupSafe==1.1.1 opencv-python==4.2.0.34 numpy==1.18.4 pipdeptree==1.0.0 pip==20.2.1 scipy==1.5.2 numpy==1.18.4 urllib3==1.25.9
So, here we see that using pipdeptree along with freeze shows the output by combining the properties of both commands. So it looks like the output of pip freeze indicates which that package installed which another package, similar to pipdeptree but here indentation is used instead of a hyphen(-) to indicate tree.
Warnings in pipdeptree:
Commonly there occur two types of warnings while executing pipdeptree command, let us see them one by one.
1. Conflicting Dependencies: As the name suggests “conflicting dependency”, so is its relevance. Sometimes there is/are package(s) that are specified as a dependency of multiple packages with a different version, in this situation possible conflicting dependency warning arises. So, any package that’s specified as a dependency of multiple packages with a different version is considered as a possible conflicting dependency.
pipdeptree by default warns about possible conflicting dependencies.
Let us see one more example of pipdeptree:
Command:
$pipdeptree
Output:
Warning!!! Possibly conflicting dependencies found: * impacket==0.9.20 - ldap3 [required: ==2.5.1, installed: ?] - ldapdomaindump [required: >=0.9.0, installed: ?] ------------------------------------------------------------------------ alembic==1.0.11.dev0 attrs==18.2.0 dulwich==0.20.2 - certifi [required: Any, installed: 2018.11.29] - urllib3 [required: >=1.24.1, installed: 1.24.1] EditorConfig==0.12.1 Flask-Cors==3.0.8 - Flask [required: >=0.9, installed: 1.1.1] - Six [required: Any, installed: 1.13.0] Flask-Session==0.3.1 Flask-SocketIO==4.2.1 - Flask [required: >=0.9, installed: 1.1.1] - python-socketio [required: >=4.3.0, installed: 4.5.1] - python-engineio [required: >=3.9.0, installed: 3.12.1] - six [required: >=1.9.0, installed: 1.13.0] - six [required: >=1.9.0, installed: 1.13.0] google==2.0.1 - beautifulsoup4 [required: Any, installed: 4.8.0] html2text==2019.8.11 impacket==0.9.20 - ldap3 [required: ==2.5.1, installed: ?] - ldapdomaindump [required: >=0.9.0, installed: ?] ipython==5.8.0 - backports.shutil-get-terminal-size [required: Any, installed: 1.0.0] - pathlib2 [required: Any, installed: 2.3.5] - scandir [required: Any, installed: 1.10.0] - pexpect [required: Any, installed: 4.6.0]
pip doesn’t have a true dependency resolution yet. The warning is printed to stderr (Standard error) instead of stdout (Standard Output). To completely silence this warning use the -w silence or –warn silence flag It can also be made mode strict with –warn fail in which case the command will not only print the warnings to stderr but also exit with a non-zero status code. This could be useful if you want to fit this tool into your CI pipeline.
Note: The –warn flag was added in version 0.6.0. For older version, use –nowarn flag.
2. Circular Dependencies: This dependency occurs when two packages depend on each other. Suppose, package A depends upon package B and package B depends upon package A.
For this let us see one more example:
Command:
$pipdeptree
Output:
Warning!!! Cyclic dependencies found: - CircularDependencyA => CircularDependencyB => CircularDependencyA - CircularDependencyB => CircularDependencyA => CircularDependencyB ------------------------------------------------------------------------ wsgiref==0.1.2 argparse==1.2.1
Note: They are also printed to stderr and can be controlled using the –warn flag.
To find why a particular package is installed:
Now, we may sometimes want to know why a particular package is installed. Then we can use –reverse (or simply -r) flag for this. To find out what all packages require a particular package(s), it can be combined with –packages
flag as shown in following example:
Command:
$pipdeptree --reverse --packages MarkupSafe,numpy
Output:
MarkupSafe==1.1.1 - Jinja2==2.11.2 [requires: MarkupSafe>=0.23] - altair==4.1.0 [requires: jinja2] - Mako==1.1.3 [requires: MarkupSafe>=0.9.2] numpy==1.18.4 - altair==4.1.0 [requires: numpy] - opencv-python==4.2.0.34 [requires: numpy>=1.17.3] - pandas==1.0.4 [requires: numpy>=1.13.3] - altair==4.1.0 [requires: pandas>=0.18] - scipy==1.5.2 [requires: numpy>=1.14.5]
Using pipdeptree to write requirements.txt file:
If you wish to track only the top-level packages in your requirements.txt file, it’s possible to do so using pipdeptree by grep-ing only the top-level lines from the output,
Command:
$pipdeptree | grep -P '^\w+'
Output:
Lookupy==0.1 wsgiref==0.1.2 argparse==1.2.1 psycopg2==2.5.2 Flask-Script==0.6.6 alembic==0.6.2 ipython==2.0.0 slugify==0.0.1 redis==2.9.1
There is a problem here though. The output doesn’t mention anything about Lookupy being installed as an editable package (refer to the output of pip freeze above) and information about its source is lost. To fix this, pipdeptree must be run with a -f or –freeze flag
Command:
$pipdeptree -f --warn silence | grep -P '^[\w0-9\-=.]+'
Output:
-e git+git@github.com:naiquevin/lookupy.git@cdbe30c160e1c29802df75e145ea4ad903c05386#egg=Lookupy-master wsgiref==0.1.2 argparse==1.2.1 psycopg2==2.5.2 Flask-Script==0.6.6 alembic==0.6.2 ipython==2.0.0 slugify==0.0.1 redis==2.9.1
Command:
$ pipdeptree -f --warn silence | grep -P '^[\w0-9\-=.]+' > requirements.txt
The freeze flag will also not output the hyphens for child dependencies, so you could dump the complete output of pipdeptree -f to the requirements.txt file making the file human-friendly (due to indentations) as well as pip-friendly. (Take care of duplicate dependencies though)
Using pipdeptree with External tools:
pipdeptree uses flag –json to show output in JSON representation, as shown below:
Command:
$pipdeptree --json
Output:
[ { "package": { "key": "werkzeug", "package_name": "Werkzeug", "installed_version": "1.0.1" }, "dependencies": [] }, { "package": { "key": "urllib3", "package_name": "urllib3", "installed_version": "1.25.9" }, "dependencies": [] }, { "package": { "key": "pytz", "package_name": "pytz", "installed_version": "2020.1" }, "dependencies": [] }, { "package": { "key": "python-dateutil", "package_name": "python-dateutil", "installed_version": "2.8.1" }, "dependencies": [ { "key": "six", "package_name": "six", "installed_version": "1.15.0", "required_version": ">=1.5" } ] }, { "package": { "key": "pyrsistent", "package_name": "pyrsistent", "installed_version": "0.16.0" }, "dependencies": [ { "key": "six", "package_name": "six", "installed_version": "1.15.0", "required_version": null } ] }, { "package": { "key": "pipdeptree", "package_name": "pipdeptree", "installed_version": "1.0.0" }, "dependencies": [ { "key": "pip", "package_name": "pip", "installed_version": "20.2.1", "required_version": ">=6.0.0" } ] }, ]
Note: –json will output a flat list of all packages with their immediate dependencies. To obtain nested JSON, use –-json-tree (added in version 0.11.0).
Command:
$pipdeptree --json-tree
Output:
[ { "key": "altair", "package_name": "altair", "installed_version": "4.1.0", "required_version": "4.1.0", "dependencies": [ { "key": "entrypoints", "package_name": "entrypoints", "installed_version": "0.3", "required_version": "Any", "dependencies": [] }, { "key": "jinja2", "package_name": "jinja2", "installed_version": "2.11.2", "required_version": "Any", "dependencies": [ { "key": "markupsafe", "package_name": "MarkupSafe", "installed_version": "1.1.1", "required_version": ">=0.23", "dependencies": [] } ] }, { "key": "urllib3", "package_name": "urllib3", "installed_version": "1.25.9", "required_version": "1.25.9", "dependencies": [] } ]
Dependency tree of a Python Module
Generally, many Python packages are dependent on other packages but how do we know that on which packages is a module dependent?