惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Stack Overflow Blog
Stack Overflow Blog
Y
Y Combinator Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
M
MIT News - Artificial intelligence
GbyAI
GbyAI
A
About on SuperTechFans
T
The Blog of Author Tim Ferriss
雷峰网
雷峰网
Blog — PlanetScale
Blog — PlanetScale
J
Java Code Geeks
IT之家
IT之家
Microsoft Azure Blog
Microsoft Azure Blog
V
V2EX
爱范儿
爱范儿
N
Netflix TechBlog - Medium
U
Unit 42
博客园 - 三生石上(FineUI控件)
WordPress大学
WordPress大学
博客园 - 叶小钗
G
Google Developers Blog
Jina AI
Jina AI
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
The GitHub Blog
The GitHub Blog
腾讯CDC

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
I Deployed 6 AI Systems Live — Here's What Actually Broke
DANISH ZULFIQAR · 2026-06-28 · via DEV Community

I Deployed 6 AI Systems Live — Here's What Actually Broke

A few weeks ago I wrote about the 5 bugs that cost me 60+ hours building 49 AI systems. Every one of those bugs lived inside the code itself wrong array layout, a renamed model class, a serialization mismatch.

This article is the second half of that story, and it taught me something more uncomfortable: code that runs perfectly on your machine can fail completely the moment it leaves your machine for reasons that have nothing to do with your code.

I took 6 of my pinned GitHub projects and deployed every one of them live on Streamlit Cloud. Locally, all 6 worked without a single error. Deploying them surfaced 5 failures I had never seen before, none of which were bugs in my logic.

Here they are, in the order I hit them.


Failure 1 — A Module That Existed Yesterday, Gone Today

My RAG chatbot used this import, unchanged for weeks:

from langchain.chains import ConversationalRetrievalChain

Locally: works. Deployed: instant crash.

ModuleNotFoundError: No module named 'langchain.chains'

The cause had nothing to do with my code. My local environment had an old, cached version of LangChain installed months ago. The deploy environment did a clean install and pulled whatever the latest version was at that moment and recent LangChain releases moved legacy chain classes like this one out of the core package entirely.

The fix that actually worked pin the exact version that still contains the class, rather than chasing the newest API pattern under deployment pressure:

langchain==0.3.7
langchain-community==0.3.7

The lesson: "it works on my machine" is frequently true specifically because your machine never reinstalled anything recently. A clean deploy environment has no such luxury it gets whatever is newest the moment it builds. Pin your versions before you ever need to debug this at 1 AM.


Failure 2 — A File That Exists, Until It Doesn't

My construction RAG project loads a prebuilt FAISS vector index from disk:

vectorstore = FAISS.load_local("faiss_index", embedding, allow_dangerous_deserialization=True)

Locally, instant load. Deployed, a raw crash deep inside FAISS's C++ binding with no clean Python traceback the kind of failure that gives you nothing to Google.

The actual cause: Git LFS. My index file had been quietly stored via Git LFS, which keeps a tiny text pointer in your git history instead of the real binary. Locally, my LFS client silently resolved that pointer into the real file, so I never noticed. The cloud platform's git clone fetched the pointer file only a few hundred bytes of text and handed that to FAISS, expecting a binary index.

The fix:

git lfs untrack "faiss_index/*"
git rm --cached faiss_index/index.faiss
git add faiss_index/index.faiss
git commit -m "Stop using Git LFS — commit as plain binary"
git push

The lesson: Git LFS is invisible exactly when it's working correctly on your machine. The only time you discover you were depending on it is the first time you deploy somewhere that doesn't support it.


Failure 3 — Two Different Size Limits Wearing the Same Error Message

Uploading an 83MB PyTorch model checkpoint through GitHub's website gave me this:

Yowza, that's a big file. Try again with a file smaller than 25MB.

I assumed GitHub simply couldn't take files that size. It can the website's drag-and-drop has a 25MB ceiling, but git push from the command line has a completely separate 100MB ceiling. Same platform, two different limits depending on which door you walk through.

The fix:

git add model.pth
git commit -m "Add trained model checkpoint"
git config --global http.postBuffer 157286400
git push

That postBuffer line matters specifically for files in the 50-100MB range without it, larger pushes can silently time out mid-transfer on a slower connection.

The lesson: a platform's documented limit and a UI's enforced limit are not always the same number. When something fails at a suspiciously round threshold, check whether you're hitting the actual platform limit or an arbitrary limit of the specific interface you happened to use.


Failure 4 — The Platform Changed Under Me, Without Asking

Midway through this deployment sprint, an app that had been working for days suddenly broke with a wall of import errors torchvision missing, dozens of warnings cascading from deep inside transformers.

Nothing in my code had changed. What had changed was the Python version my deploy platform silently selected newer than what I'd pinned, and several of my dependencies didn't yet have compatible builds for it.

The fix that actually held: I stopped trying to pin a specific Python version against a platform that wasn't reliably honoring the pin, and instead removed every heavy compiled dependency I didn't strictly need no torch, no transformers, no faiss for a project whose knowledge base was small enough to live directly in a prompt instead of a vector store. A requirements file with three lines:

streamlit==1.40.0
requests==2.32.3
python-dotenv==1.0.1

cannot break this way, because there is nothing in it with compiled platform-specific wheels to break.

The lesson: when a managed platform controls the runtime, the most resilient strategy is not fighting to pin every variable it's minimizing how many variables you depend on in the first place.


Failure 5 — A Push That Succeeds and Goes Nowhere You're Looking

The most disorienting failure of the five: git push reported complete success files written, no errors, a clean exit. The file was simply not visible anywhere on GitHub afterward.

git branch -a
* master
  remotes/origin/main

My repository had been created with a default main branch already on GitHub. I had been committing and pushing to master the entire time a branch that existed locally and now also existed remotely, sitting parallel to main, never appearing on the page I was checking.

The fix:

git push origin master:main

or, going forward, simply commit directly to whichever branch GitHub actually shows by default.

The lesson: a successful push confirms your laptop and the remote agree with each other. It confirms nothing about whether that destination is the one a human is looking at in a browser tab.


The Pattern Across All Five

None of these were logic bugs. My code was correct in every case. Every failure came from a gap between two environments that I had assumed were equivalent and were not:

  • my cached dependencies vs. a fresh install
  • my local LFS resolution vs. a clone that skips LFS
  • a UI's limit vs. a protocol's limit
  • the runtime I requested vs. the runtime I was actually given
  • the branch I was typing into vs. the branch being displayed

"Works locally" is a claim about one specific environment. Deployment is the process of discovering every assumption that claim was quietly resting on.


A Short Checklist For Next Time

Before deploying anything again, I now check:

  1. Are my dependency versions pinned to exact numbers, not ranges?
  2. Does anything in this repo rely on Git LFS — and does my deploy target support it?
  3. Are any committed files close to a platform's size limits, and which limit — UI or protocol?
  4. Can I remove a heavy compiled dependency instead of fighting to pin its version?
  5. Does git branch -a show exactly one branch I'm pushing to, with no silent second branch sitting beside it?

Five questions, thirty seconds, asked before the deploy instead of discovered after.


All 6 systems are live and open source:
🔗 github.com/Danish08654

If you've hit a deployment failure that had nothing to do with your actual code I'd like to hear it. Drop it in the comments.