I wanted to publish my Scrapy download handler to PyPi - UV made it incredibly easy. Here's how.
At some point, every scraping developer writes the same middleware twice.
The first time is in project A. It works well, so when project B comes along you copy it across. Then project C. Then a colleague asks for it. You email them the file. They make changes. Now there are four versions of the same code living in four different repositories, diverging slowly, none of them getting each other's bug fixes.
The solution is a package. Publish it once to the Python Package Index (PyPI) and every project that needs it can install it with pip install your-package. Updates go to everyone at once. The code has a home.
This guide walks through the full process using uv, a fast, modern Python toolchain that replaces pip, virtualenv, pip-tools, twine, and build with a single tool. We will write a reusable Scrapy download handler, structure it as a proper Python package, test it, and publish it to PyPI.
By the end, the package will be installable with:
uv add scrapy-random-delay
And usable in any Scrapy project with two lines in settings.py:
DOWNLOAD_HANDLERS = {
"http": "scrapy_random_delay.RandomDelayHandler",
"https": "scrapy_random_delay.RandomDelayHandler",
}
RANDOM_DELAY_RANGE = (0.5, 3.0) # seconds
Installing uv
If you do not have uv installed:
# macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# Or via pip if you prefer
pip install uv
Verify the installation:
uv --version
# uv 0.5.0 (or later)
uv is written in Rust and is dramatically faster than pip for dependency resolution and installation: typically 10 to 100 times faster. More importantly for this guide, it provides a complete project management workflow that makes creating and publishing packages significantly simpler than the traditional pip, build, and twine toolchain.
What we are building
A Scrapy download handler that adds a random delay drawn from a configurable range before each request, with optional per-domain configuration. It is realistic, self-contained, and exactly the kind of code worth packaging: useful across multiple projects but simple enough to understand completely.
How Scrapy download handlers work
A download handler is responsible for a URL scheme. Scrapy's default HTTP handler takes a request, makes the network call, and returns the response. A custom handler wraps this: add headers, impose a delay, swap the underlying HTTP library. The API is simple:
class MyHandler:
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings)
async def download_request(self, request, spider):
# return a Response
pass
def close(self):
pass
Our handler delegates actual HTTP work to Scrapy's built-in HTTPDownloadHandler and adds the delay logic around it.
Creating the project
uv init scaffolds a new package project in one command:
uv init --package scrapy-random-delay
cd scrapy-random-delay
The --package flag tells uv to create a proper importable package rather than a single-script project. The generated structure looks like this:
scrapy-random-delay/
├── src/
│ └── scrapy_random_delay/
│ └── __init__.py
├── pyproject.toml
└── README.md
uv uses the src/ layout by default, placing your package inside src/ so it cannot be accidentally imported from the project root during development, which can mask packaging errors.
Look at what uv generated in pyproject.toml:
[project]
name = "scrapy-random-delay"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[project.scripts]
scrapy-random-delay = "scrapy_random_delay:main"
[build-system]
requires = ["uv_build>=0.11.7,<0.12.0"]
build-backend = "uv_build"
Two things to note: the [project.scripts] entry is scaffolding you can delete, since the package exports a handler class rather than a command-line entry point. The build backend is uv_build, uv's own backend introduced in recent versions. We will fill in the rest of the metadata as we go.
Adding dependencies
Add Scrapy as a runtime dependency:
uv add scrapy
Add development dependencies — pytest and pytest-asyncio for testing:
uv add --dev pytest pytest-asyncio
uv writes these into pyproject.toml automatically and creates a uv.lock lockfile that pins exact versions for reproducible installs. You do not need to manually edit pyproject.toml for dependencies.
The resulting pyproject.toml dependencies section:
[project]
dependencies = [
"scrapy>=2.12",
]
[dependency-groups]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.24",
]
Writing the package
Add the handler module:
touch src/scrapy_random_delay/handler.py
# src/scrapy_random_delay/handler.py
import asyncio
import random
import logging
from typing import Tuple, Optional
from scrapy import Request
from scrapy.crawler import Crawler
from scrapy.http import Response
from scrapy.settings import Settings
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler as HTTPDownloadHandler
logger = logging.getLogger(__name__)
class RandomDelayHandler:
"""
A Scrapy download handler that adds a random delay before each request.
Settings
--------
RANDOM_DELAY_RANGE : tuple[float, float]
(min_seconds, max_seconds) delay range. Default: (0.5, 2.0)
RANDOM_DELAY_PER_DOMAIN : dict[str, tuple[float, float]]
Per-domain overrides. Keys are domain strings, values are (min, max) tuples.
Example: {"api.example.com": (1.0, 4.0), "cdn.example.com": (0.0, 0.1)}
RANDOM_DELAY_VERBOSE : bool
Log the actual delay chosen for each request. Default: False
"""
DEFAULT_RANGE: Tuple[float, float] = (0.5, 2.0)
def __init__(self, settings: Settings, crawler: Crawler):
self._settings = settings
self._range = self._parse_range(
settings.get("RANDOM_DELAY_RANGE", self.DEFAULT_RANGE)
)
self._per_domain: dict = settings.getdict("RANDOM_DELAY_PER_DOMAIN", {})
self._verbose: bool = settings.getbool("RANDOM_DELAY_VERBOSE", False)
self._delegate = HTTPDownloadHandler(settings, crawler)
@classmethod
def from_crawler(cls, crawler: Crawler) -> "RandomDelayHandler":
return cls(crawler.settings, crawler)
def _parse_range(self, value) -> Tuple[float, float]:
try:
low, high = float(value[0]), float(value[1])
except (TypeError, ValueError, IndexError) as e:
raise ValueError(
f"RANDOM_DELAY_RANGE must be a two-element sequence of numbers, got: {value!r}"
) from e
if low < 0 or high < 0:
raise ValueError("RANDOM_DELAY_RANGE values must be non-negative")
if low > high:
raise ValueError(
f"RANDOM_DELAY_RANGE minimum ({low}) must not exceed maximum ({high})"
)
return low, high
def _get_delay_range(self, request: Request) -> Tuple[float, float]:
from urllib.parse import urlparse
domain = urlparse(request.url).netloc
if domain in self._per_domain:
return self._parse_range(self._per_domain[domain])
return self._range
async def download_request(self, request: Request, spider) -> Response:
low, high = self._get_delay_range(request)
delay = random.uniform(low, high)
if self._verbose:
logger.debug(f"Random delay: {delay:.2f}s before {request.url}")
await asyncio.sleep(delay)
return await self._delegate.download_request(request, spider)
def close(self):
self._delegate.close()
Update __init__.py to export the handler and set the version:
# src/scrapy_random_delay/__init__.py
from scrapy_random_delay.handler import RandomDelayHandler
__version__ = "0.1.0"
__all__ = ["RandomDelayHandler"]
Writing the tests
Create the test directory:
mkdir tests
touch tests/__init__.py tests/conftest.py tests/test_handler.py
# tests/conftest.py
import pytest
from unittest.mock import MagicMock
from scrapy.http import Request, Response
from scrapy.utils.test import get_crawler
@pytest.fixture
def crawler():
return get_crawler()
@pytest.fixture
def make_request():
def _make(url="https://example.com/products", **kwargs):
return Request(url=url, **kwargs)
return _make
@pytest.fixture
def mock_response():
return MagicMock(spec=Response)
# tests/test_handler.py
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from scrapy.utils.test import get_crawler
from scrapy_random_delay import RandomDelayHandler
def make_handler(settings_dict: dict = None) -> RandomDelayHandler:
crawler = get_crawler(settings_dict=settings_dict or {})
with patch("scrapy_random_delay.handler.HTTPDownloadHandler"):
return RandomDelayHandler(crawler.settings, crawler)
class TestRangeValidation:
def test_default_range_applied_when_not_configured(self):
handler = make_handler()
assert handler._range == (0.5, 2.0)
def test_custom_range_accepted(self):
handler = make_handler({"RANDOM_DELAY_RANGE": (1.0, 5.0)})
assert handler._range == (1.0, 5.0)
def test_equal_values_accepted(self):
handler = make_handler({"RANDOM_DELAY_RANGE": (2.0, 2.0)})
assert handler._range == (2.0, 2.0)
def test_zero_range_accepted(self):
handler = make_handler({"RANDOM_DELAY_RANGE": (0.0, 0.0)})
assert handler._range == (0.0, 0.0)
def test_inverted_range_raises(self):
with pytest.raises(ValueError, match="minimum.*must not exceed maximum"):
make_handler({"RANDOM_DELAY_RANGE": (5.0, 1.0)})
def test_negative_range_raises(self):
with pytest.raises(ValueError, match="non-negative"):
make_handler({"RANDOM_DELAY_RANGE": (-1.0, 2.0)})
def test_invalid_type_raises(self):
with pytest.raises(ValueError, match="two-element sequence"):
make_handler({"RANDOM_DELAY_RANGE": "bad-value"})
class TestPerDomainOverrides:
def test_per_domain_range_returned_for_matching_domain(self, make_request):
handler = make_handler({
"RANDOM_DELAY_RANGE": (0.5, 1.0),
"RANDOM_DELAY_PER_DOMAIN": {"api.example.com": [2.0, 4.0]},
})
request = make_request(url="https://api.example.com/products")
assert handler._get_delay_range(request) == (2.0, 4.0)
def test_default_range_returned_for_unmatched_domain(self, make_request):
handler = make_handler({
"RANDOM_DELAY_RANGE": (0.5, 1.0),
"RANDOM_DELAY_PER_DOMAIN": {"api.example.com": [2.0, 4.0]},
})
request = make_request(url="https://other.example.com/products")
assert handler._get_delay_range(request) == (0.5, 1.0)
class TestDownloadRequest:
@pytest.mark.asyncio
async def test_delegates_to_inner_handler(self, make_request, mock_response):
handler = make_handler({"RANDOM_DELAY_RANGE": (0.0, 0.0)})
handler._delegate.download_request = AsyncMock(return_value=mock_response)
result = await handler.download_request(make_request(), MagicMock())
assert result is mock_response
handler._delegate.download_request.assert_awaited_once()
@pytest.mark.asyncio
async def test_delay_is_within_range(self, make_request, mock_response):
import time
handler = make_handler({"RANDOM_DELAY_RANGE": (0.05, 0.1)})
handler._delegate.download_request = AsyncMock(return_value=mock_response)
start = time.monotonic()
await handler.download_request(make_request(), MagicMock())
elapsed = time.monotonic() - start
assert 0.05 <= elapsed <= 0.5 # generous upper bound for slow CI
@pytest.mark.asyncio
async def test_verbose_mode_logs_delay(self, make_request, mock_response, caplog):
import logging
handler = make_handler({
"RANDOM_DELAY_RANGE": (0.0, 0.0),
"RANDOM_DELAY_VERBOSE": True,
})
handler._delegate.download_request = AsyncMock(return_value=mock_response)
with caplog.at_level(logging.DEBUG, logger="scrapy_random_delay.handler"):
await handler.download_request(make_request(), MagicMock())
assert "Random delay" in caplog.text
class TestClose:
def test_close_delegates_to_inner_handler(self):
handler = make_handler()
handler._delegate.close = MagicMock()
handler.close()
handler._delegate.close.assert_called_once()
Configure pytest in pyproject.toml by adding this section:
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
Run the tests with uv:
uv run pytest
uv run executes the command inside the project's virtual environment, which uv manages automatically. No source .venv/bin/activate needed.
Completing pyproject.toml
Fill in the project metadata. uv has already set the basics: add the rest:
[project]
name = "scrapy-random-delay"
version = "0.1.0"
description = "A Scrapy download handler that adds a configurable random delay before each request."
readme = "README.md"
license = { file = "LICENSE" }
requires-python = ">=3.9"
authors = [
{ name = "Your Name", email = "you@example.com" },
]
keywords = ["scrapy", "web-scraping", "download-handler", "rate-limiting"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Framework :: Scrapy",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Software Development :: Libraries :: Python Modules",
]
dependencies = [
"scrapy>=2.8",
]
[dependency-groups]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.24",
]
[project.urls]
Homepage = "https://github.com/yourname/scrapy-random-delay"
Documentation = "https://github.com/yourname/scrapy-random-delay#readme"
Repository = "https://github.com/yourname/scrapy-random-delay.git"
Issues = "https://github.com/yourname/scrapy-random-delay/issues"
[build-system]
requires = ["uv_build>=0.11.7,<0.12.0"]
build-backend = "uv_build"
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
The classifiers list is used by PyPI for filtering and discovery. The Framework :: Scrapy classifier ensures your package appears when someone browses the Scrapy ecosystem. Browse the full list at pypi.org/classifiers.
Writing the README
# scrapy-random-delay
A Scrapy download handler that adds a configurable random delay before each request.
Scrapy's built-in `DOWNLOAD_DELAY` setting adds a fixed delay between requests.
This package draws a delay from a uniform distribution between a minimum and maximum,
which produces more natural request timing and is configurable per domain.
## Installation
uv add scrapy-random-delay
# or
pip install scrapy-random-delay
## Usage
Add to your Scrapy `settings.py`:
DOWNLOAD_HANDLERS = {
"http": "scrapy_random_delay.RandomDelayHandler",
"https": "scrapy_random_delay.RandomDelayHandler",
}
RANDOM_DELAY_RANGE = (0.5, 3.0) # seconds (min, max). Default: (0.5, 2.0)
### Per-domain configuration
RANDOM_DELAY_PER_DOMAIN = {
"api.example.com": (1.0, 4.0), # slower for sensitive endpoints
"cdn.example.com": (0.0, 0.1), # faster for static assets
}
### Verbose logging
RANDOM_DELAY_VERBOSE = True # logs the delay chosen for each request
## Settings reference
| Setting | Type | Default | Description |
|---|---|---|---|
| `RANDOM_DELAY_RANGE` | `(float, float)` | `(0.5, 2.0)` | Min and max delay in seconds |
| `RANDOM_DELAY_PER_DOMAIN` | `dict` | `{}` | Per-domain delay overrides |
| `RANDOM_DELAY_VERBOSE` | `bool` | `False` | Log each delay to DEBUG |
## Compatibility
- Python 3.9+
- Scrapy 2.8+
## License
MIT
Publishing to PyPI
Create accounts
You need two accounts: test.pypi.org for the test registry, and pypi.org for the real registry that pip install and uv add use. Use the test registry first, since it resets periodically and will not pollute the real index with test uploads. Enable two-factor authentication on both, as PyPI requires it for publishing.
Publish to Test PyPI
uv has publishing built in via uv publish. No need to install twine or build separately: uv handles the build step automatically.
uv publish --publish-url https://test.pypi.org/legacy/
You will be prompted for your Test PyPI username (__token__) and your API token. Create a token at test.pypi.org/manage/account/token.
Verify the Test PyPI installation
uv add scrapy-random-delay --index https://test.pypi.org/simple/ --extra-index https://pypi.org/simple/
The --extra-index flag allows uv to find dependencies like scrapy from the real PyPI, since they will not be on Test PyPI.
Verify it works:
import scrapy_random_delay
print(scrapy_random_delay.__version__) # 0.1.0
from scrapy_random_delay import RandomDelayHandler
print(RandomDelayHandler)
Publish to the real PyPI
Once satisfied:
uv publish
Your package is live. Within a few minutes anyone can install it:
uv add scrapy-random-delay
# or
pip install scrapy-random-delay
Using a token stored in the environment
Rather than typing your token on every publish, store it in an environment variable:
export UV_PUBLISH_TOKEN=pypi-your-token-here
uv publish
Or configure it per-project in a .env file (never commit this):
# .env
UV_PUBLISH_TOKEN=pypi-your-token-here
# .gitignore
.env
Automating releases with GitHub Actions
Manually publishing works fine, but automating it means a new version ships by pushing a version tag, with no manual steps:
# .github/workflows/publish.yml
name: Publish to PyPI
on:
push:
tags:
- "v*" # triggers on tags like v0.1.0, v1.2.3
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Run tests
run: uv run pytest
publish:
needs: test
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write # required for trusted publishing
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Publish to PyPI
run: uv publish
# Uses trusted publishing via OIDC — no token stored in secrets needed
# Configure at pypi.org/manage/account/publishing/
The workflow is notably short compared to the traditional pip, build, and twine approach, since uv handles the build and publish in a single command.
Trusted publishing is worth setting up. It authenticates the GitHub Actions workflow directly with PyPI using OpenID Connect, so you never need to store a PyPI token as a GitHub secret. Configure it once at pypi.org under your account's Publishing settings, specifying your GitHub username, repository name, and workflow file name.
To cut a release:
# Update version in pyproject.toml and src/scrapy_random_delay/__init__.py
git commit -am "Release v0.2.0"
git tag v0.2.0
git push && git push --tags
The workflow runs, tests pass, the package publishes.
Bumping versions
uv does not yet have a built-in version bump command, but the process is straightforward. The version lives in two places: pyproject.toml and __init__.py, and both need updating together.
For a small package, doing this by hand is fine. For a larger project, bump2version or python-semantic-release automate the version increment, changelog update, git commit, and tag in one command:
uv add --dev bump2version
uv run bump2version patch # 0.1.0 → 0.1.1
uv run bump2version minor # 0.1.0 → 0.2.0
uv run bump2version major # 0.1.0 → 1.0.0
Configure it in pyproject.toml:
[tool.bumpversion]
current_version = "0.1.0"
commit = true
tag = true
[[tool.bumpversion.files]]
filename = "pyproject.toml"
[[tool.bumpversion.files]]
filename = "src/scrapy_random_delay/__init__.py"
Now bumping the version, committing, tagging, and triggering the publish workflow is a single command.
Testing against multiple Python versions
Add a test matrix to the workflow to catch version-specific issues before they reach users:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v4
- name: Run tests on Python ${{ matrix.python-version }}
run: uv run --python ${{ matrix.python-version }} pytest
uv handles Python version management directly: --python 3.9 downloads and uses that Python version without any additional tooling like pyenv.
Keeping your package maintainable
Pin your minimum Scrapy version. scrapy>=2.8 in dependencies sets a floor. Check which Scrapy version introduced the APIs you use and do not allow anything older. uv add scrapy will pick the current latest: tighten the floor to the oldest version you have tested against.
Keep a CHANGELOG. Users upgrading a dependency want to know if anything changed. Keep a CHANGELOG.md with an ## Unreleased section at the top that becomes the next version entry when you tag a release.
Be conservative with dependencies. Every package in dependencies becomes a constraint users must satisfy alongside their own dependencies. A Scrapy extension that depends on pandas and numpy is painful to integrate. Keep the list to what is genuinely required.
Handle deprecation cleanly. When you remove a setting or change an API, keep the old behavior working for at least one minor version and log a deprecation warning. Users who update without reading the changelog will thank you.
Lock files are for applications, not libraries. Commit uv.lock if this is an application. Do not commit it if this is a library, since your users' projects will resolve their own dependency versions and a committed lockfile does not help them.
The complete file tree
After following this guide, your project looks like this:
scrapy-random-delay/
├── src/
│ └── scrapy_random_delay/
│ ├── __init__.py # exports RandomDelayHandler, sets __version__
│ └── handler.py # the handler implementation
├── tests/
│ ├── __init__.py
│ ├── conftest.py # shared fixtures
│ └── test_handler.py # test suite
├── .github/
│ └── workflows/
│ └── publish.yml # automated test and publish workflow
├── pyproject.toml # project metadata, dependencies, tool config
├── uv.lock # locked dependency versions (omit for libraries)
├── README.md # installation and usage documentation
├── LICENSE # MIT or whichever license you choose
└── .gitignore # include .env, __pycache__, dist/, *.egg-info/
Next steps
Publishing a Scrapy extension to PyPI is the same process regardless of what the extension does: the download handler is just the example. The same pyproject.toml structure, the same uv publish command, and the same GitHub Actions workflow apply to middlewares, pipelines, item types, and extensions. If you want to go deeper on Scrapy's component system and understand where each type of extension fits, the Modern Scrapy Developer's Guide covers the full architecture from spiders through to deployment on Scrapy Cloud.