The ecosystem of tools for packaging, testing, distributing, and testing Python projects is overwhelming, especially if you’re coming to Python from other languages.
In this post, we’ll survey the evolution of Python tooling from its inception in 1991 through the present day, touching on significant milestones. We’ll explain each tool’s benefits, the problems it addresses, and any configuration files you’ll find associated with it in Python projects.
After reading this post, you’ll have a richer understanding of the many setup, testing, and packaging patterns you’re likely to find in open-source Python projects and be better equipped to work with them. You’ll see how to migrate your project to a modern dependency management and packaging tool such as Poetry.
We’ll explain why we are converting the Pinecone Python client to Poetry, the new contributor-facing guides we’ve added to ease this migration, and how this makes the project easier to develop locally and even within Jupyter Notebooks and alongside popular libraries such as LangChain.
Along the way, we’ll take an unflinching look at some of the pain involved in developing Python projects to set the stage for understanding how the latest tooling can make developers’ lives easier.
Guido van Rossum released Python on February 20, 1991. At the time of this writing, that was 32 years ago. To understand why today’s preferred tools work the way they do, we must first look backward to understand the evolution of Python.
If you’ve ever worked with Python, you’re probably familiar with the following tools, patterns, digital duct tape, and the headaches that come with a scattered landscape of options.
Most readers familiar with Python have probably run pip install -r requirements.txt to tell the pip package manager binary to read a requirements text file, which lists the dependencies (external libraries) required by a given Python script.
pip install is probably the most common and straightforward way to get up and running with a given Python program. Now, consider why this typical pattern is fraught with complexity and suffering.
Until macOS Catalina (10.15), released in 2019, Python 2 was pre-installed on your OSX machine, requiring developers who needed Python 3 to install it separately. Folks in a hurry could accidentally execute commands against python instead of python3, leading to errors.
The two versions also had dependency conflicts because each maintained a separate site-packages, the directory where Python installs packages, leading to potential duplicate packages and version incompatibility on the same system.
The shebang lines at the top of Python scripts that signal which interpreter were equally confusing. #!/usr/bin/env python defaulted to Python 2, which could be problematic for code intended to be run by Python 3.
This confusion spread to the use of pip itself. You might have meant to pip3 install a given requirements.txt file because pip3 is not the same binary as pip. If you have Python 2 and Python 3 installed on your system, pip and pip3 will install packages in different locations to avoid conflicts.
However, if you only have Python 3 installed, pip might be an alias for pip3; in that case, they would install packages in the same location.
Plenty of shims, patches, and bandaids emerged to help address this pain. The python-is-python3 package explains: “This is a convenience package which ships a symlink to point the /usr/bin/python interpreter at the current default python3. It may improve compatibility with other modern systems, whilst breaking some obsolete or 3rd-party software.“
Pyenv is a tool for managing multiple versions of Python on a single machine, so you can quickly toggle between versions (which might be necessary to fix a bug in library A and then continue developing your project B - both of which use incompatible Python versions).
Pyenv is very convenient if you regularly need to toggle between Python versions, yet it adds complexity and may not play nicely with external scripts and tools that just want to call python.
Don’t forget that you need to modify your shell configuration to get pyenv to work correctly - and remember to do so across every machine you work on. It can also automatically read .python-version files if you include them in your project, which is a nice technical solution. But there’s also the human element; not everyone on your team may want to run Pyenv.
We’ve surveyed some of the gotchas in managing different versions of Python on a single machine. Let’s now take a step back and consider the evolution of tooling from the open-source Python project maintainer’s perspective.
At Pinecone, I’m fortunate to work with highly experienced developers. We're hiring if you’re looking to join a team with extensive engineering talent.
In a recent conversation with a colleague who has been using Python for 25 years, I was lamenting the many tools and patterns you need to grok to maintain Python projects. “It used to be a lot worse,” he explained.
You used to go and get your Python “eggs” from “The Cheese Shop,” which was the original name for the PyPi (Python Package Index) we know and use today; whenever you pip install a dependency.
The Cheese Shop referenced Monty Python’s The Cheese Shop sketch, featuring a Cheese Shop devoid of actual cheese. The reference was a tongue-in-cheek indication that the index was mostly empty yet had potential.
Here’s a chronological list of the methods for maintaining Python projects and their dependencies (as well as, in some cases, handling everyday needs such as packaging and isolation via tools like virtualenv):
pip library, enabling packaging management within that isolated environment.Downsides: Early adoption issues; complicates migration for older projects.
No perfect tool can solve everyone’s problems and cover all use cases neatly in a backward-compatible way. Humans write imperfect software and tooling, and we’ll likely continue doing so, at least for the foreseeable future.
Ultimately, engineering comes down to tradeoffs and choosing the best fit amongst those tradeoffs. We settled on Poetry to manage the Pinecone Python client, which you can use to create, upsert into, and query Pinecone vector database indexes, because we felt it provided the most benefits to our internal maintainers, customers, and community contributors.
Poetry will manage the Pinecone Python client soon once this pull request is merged. If you’re wondering how to migrate your existing Python project to use Poetry, you can see how we did it by reviewing that pull request.
Here’s why we chose Poetry:
import pinecone and B.) track those changes in git so you can open a pull request to contribute them back. Poetry supports dependency resolution and virtualenv management via a single interface, simplifying project setup for new contributors.pinecone-client and langchain are commonly pip installed into Jupyter Notebooks, such as our example notebooks demonstrating the latest AI patterns. In some cases, such as improving bottlenecks or adding features, maintainers must be able to modify both langchain and the pinecone-client that it wraps simultaneously and track their changes in git. Again, Poetry simplifies this workflow.Python is the predominant language for AI use cases, but as we’ve written here, we believe most folks underestimate JavaScript for AI application development.
Meanwhile, developers eagerly anticipate project Mojo, which will likely significantly improve the Python developer experience again.
Python has come a long way and continues receiving outstanding contributions and enhancements. We’re excited to continue shipping Python tooling and libraries as the developer experience improves.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。