Lessons from replacing a brittle chat integration with a lean, SSE-based LangChain flow

Introduction: A simple stack

Throughout my software development career, my focus has evolved from "architecture is most important" to "quality is everything" to "engineers are the backbone." While these views are different, they all share one common goal: building good software.

But there has always been one aspect that adds the most cost to the product lifecycle: deployment and maintenance. A team can build one great product, but if the architecture does not account for deployment and maintenance, that team may never build a second product. It will just keep fighting fires.

With that in mind, I started a new project to build an AI-integrated application that improves user workflows. The app must make decisions based on status, report information in a rich format, and provide a conversational interface.

For this project, I set a strict constraint: minimize the surface area. I wanted a robust analysis pipeline without the deployment tax of extra services. The goal was a clean, four-element ecosystem:

1. The Frontend: React (TypeScript), including a conversational interface
2. The Backend: Node.js / Express
3. The Database: Postgres (handling data, vector embeddings and job queuing via pg-boss)
4. The AI: A self-deployed LLM (Ollama + Qwen 2.5) for local processing and LangChain

With a tight delivery timeline, Cursor was the right tool to accelerate implementation. Cursor supported development well, until it hit a wall with the conversational interface using CopilotKit. I had to step in with more details and design guidance to successfully implement the interface.

What I tried

I did some research on the right components to implement my requirements and CopilotKit came up strong. I already had my LangChain endpoint ready and verified, so I instructed Cursor to use CopilotKit and add a chat interface to the frontend and integrate with my LangChain endpoint. Things did not work out very well and every iteration brought up new problems.

What kept breaking

• The primary challenge was that the runtime would not recognize my LLM settings from the environment. I had to hardcode the model into the runtime code.
• The data would not reach my LangChain endpoint. Cursor recommended an HTTP adapter and implemented it.
• Once LangChain processed the input and responded the client interface would not recognize the content.

Since CopilotKit is being actively improved, using the latest version was the first mistake. From what I read, every version works well as long as the front end is connected directly to the runtime. But once a new intermediate layer was introduced, the integration failed.

Why I switched to manual (guided) integration

I did not want to compromise on my requirement to reduce stack spread, so I reset to a previous stable point in my code and started building a custom integration with LangChain. This required three parts:

1. Chat interfaces
2. LangChain interface
3. A way to send back the response. SSE was the answer

As soon as I shifted to this design the implementation was quicker and smoother. Cursor built a chat interface with the inputs and customizations I asked for. Then the SSE layer ensured that communication between the server and client was based on standard protocols. The format translation between the client input and the LangChain interface was trivial.

This ensured that chat and tool responses transmitted to the client seamlessly. Now it was a matter of defining the protocol that would help the client choose between displaying the response as a chat message or using a client side component to display rich content. I did hit a few more challenges after that, but they were primarily due to differences in how commercial LLMs (Vertex, OpenAI) handled tools compared to local LLMs. That is a topic for another time.

Conclusion

Coding agents are amazing at speeding up development, improving test coverage, and even keeping interfaces documented. But some patterns are still too new for current agents. In this case, the limitations were easy to spot. In other cases, agents may produce code that works but introduce design choices that make long-term maintenance harder. Defining clear design criteria before implementation helps avoid those pitfalls.