前言 备受期待的DeepSeek V4终于发布了，相信很多人都想要看到这个模型在Coding任务中的实际表现。按传统来说，我们应该跑一个天气卡片，但是我一直觉得只有天气卡片完全反映不出来模型真正的能力，于是我决定来点创新的实战测试。 为保证本测试接近实际使用场景，测试全程使用Claude Code v2.1.86，官方正价API，模型映射配置如下所示： [!NOTE] 本次测试中DeepSeek V4所编写的源码可在 Raven95676/ds_v4_test: DeepSeek V4 Test - Codeberg.org 获取。真实环境测试仅保留Diff文件。 由于时间以及精力还有财力限制，测试可能片面，欢迎各位补充测试用例。 综上，让我们开始测试吧！ 希望测完了我不会变成负翁 独立项目从零生成测试 很多情况下，我们有了灵感都会让AI来开荒。我们首先来测试AI从零生成的能力。 天气卡片 首先是我们最为经典的天气卡片。 Prompt： 请编写一个单文件的HTML，实现一个现代化的天气卡片应用。所有的CSS和JS必须内联在这个HTML文件中，不允许引入外部的UI组件库。界面设计要求具备毛玻璃效果，包含天气图标、温度、地点、湿度、风速等元素，排版要求高水准的美感。 成果展示： 尽管没有要求，DeepSeek V4依然接入了实际可用的数据源，填入OpenWeatherMap API Key后： 算法可视化 [!NOTE] 本项测试使用 npm create vite@latest 的TypeScript+React+React Compiler模板创建了一个空项目，DeepSeek V4将在这个空项目的基础上开发。本项测试无额外预装依赖。 Prompt： 使用React+TypeScript编写一个算法可视化网站，包含排序算法可视化和寻路算法可视化两个模块，通过顶部导航标签页切换。使用纯React状态与CSS实现，不依赖任何第三方动画库或状态管理库。具体要求如下： ## 排序算法可视化（标签页1） 1. 随机生成包含20个整数的数组，用垂直柱状图展示。 2. 提供下拉菜单选择算法：冒泡排序、选择排序、插入排序、快速排序。 3. 排序过程逐步动画展示。 ## 寻路算法可视化（标签页2） 1. 显示一个20×20的网格。允许用户交互编辑网格。 2. 提供下拉菜单选择寻路算法：Dijkstra、A*、BFS、DFS。 3. 寻路过程逐步动画展示。 成果展示： 3D三角形旋转 [!NOTE] 本项测试使用 npm create vite@latest 的TypeScript+React+React Compiler模板创建了一个空项目，DeepSeek V4将在这个空项目的基础上开发。本项测试额外预装依赖 three @react-three/fiber @react-three/drei @types/three 。 Prompt： 使用React+TypeScript+React Three Fiber+Three.js编写一个3D场景。场景要求： 1. 场景中央显示一个彩色的3D等边三角形（红、绿、蓝），背景颜色为深灰色，带有简单的网格地面辅助线。 2. 三角形绕自身Y轴以恒定速度旋转。 3. 允许用户通过鼠标拖拽旋转整个场景视角，支持滚轮缩放。 成果展示： 爬虫编写 [!NOTE] 本项测试使用 uv init 创建了一个空项目，预装依赖 playwright ，启用 chrome-devtools MCP，选取盗版小说网站 https://m.bqgl.cc/look/7546/ 作为评估示例。测试行为严格限于非商业性的技术验证，且遵循低频请求原则。如相关权利人认为该示例侵犯了网站的合法权益，请与我联系。 Prompt： 此项目为Python爬虫，目标网址为https://m.snapd.net/read/165986/。请使用chrome-devtools MCP分析目标网站并完成项目，依赖管理使用uv，使用playwright进行抓取。有以下要求： 1. 解析出小说的所有章节列表，提取章节正文内容，清理多余的HTML标签，清理非小说正文的广告内容，保留纯文本段落。 2. 将所有章节按顺序合并保存到一个txt文件中，每章标题作为一行，空一行，接着是正文内容，然后两个空行分隔下一章。 3. 请求之间设置1秒的延迟，实现基本的错误处理和重试机制。 成果展示： 戴森云模拟 [!NOTE] 本项测试使用 npm create vite@latest 的TypeScript+React+React Compiler模板创建了一个空项目，DeepSeek V4将在这个空项目的基础上开发。本项测试额外预装依赖 three @react-three/fiber @react-three/drei @types/three ，启用 chrome-devtools MCP。 Prompt： 使用React+TypeScript+React Three Fiber+Three.js编写一个戴森云模拟场景。场景要求： 1. 场景中心有一颗发光恒星，发出暖黄色光芒，并带有光晕效果。 2. 恒星周围分布着大量小型蓝白色发光粒子，它们大致位于一个球壳范围内，形成不规则的云状结构。所有粒子以不同的角速度绕恒星中心公转，且有小幅度的随机径向偏移运动。 3. 允许用户通过鼠标拖拽旋转整个场景视角，支持滚轮缩放，右上角显示FPS等数据。 4. 添加基本的星场背景。 5. 小型蓝白色发光粒子需要允许用户通过滑块调整数量，数量范围在50-10000，请在不减少粒子数量的情况下保证性能。 请使用chrome-devtools MCP对成果进行验证。 成果展示： 真实环境测试 当然，我们不仅需要AI能够从零开荒，修复已有问题的能力也是很重要的。接下来我们测试DeepSeek V4在真实环境中的表现。 这个测试是选择一个已被修复的issue，然后分析AI是如何对这个问题进行修复的。 [!NOTE] 在实际应用中，不建议向开源项目贡献纯AI生成代码，尤其是在开源项目明确拒绝的情况下。 cpython github.com/python/cpython Crash on _ssl__SSLContext_load_cert_chain_impl (requests running w/ cert in multi-threading) 已打开 09:16AM - 26 May 25 UTC 已关闭 01:27PM - 08 Oct 25 UTC Conobi extension-modules type-crash topic-SSL # Crash report ### What happened? Hi. We've been investigating random crashes … of our FastAPI application for over 6 months, and we think we've found the culprit. When **calling requests with a custom cert (like in the code below) in a multi-threaded paradigm**, it can crashes. On some versions, like 3.12/3.13, it can in some case even block Python in a zombie state, where the process isn't killed but keep being hung. I tested on all the versions mentionned, and **the crash happened on all versions**. In the latest ones (>=3.12), it feels like I get more often double free. ```python import threading from concurrent.futures import ThreadPoolExecutor from datetime import datetime, timedelta from pathlib import Path import requests from cryptography import x509 from cryptography.hazmat.primitives import hashes, serialization from cryptography.hazmat.primitives.asymmetric import rsa from cryptography.hazmat.primitives.serialization import pkcs12 from cryptography.x509.oid import NameOID CERT_PEM = "client_cert.pem" KEY_PEM = "client_key.pem" PFX_FILE = "client_cert.pfx" CERT_PASSWORD = b"password" # For PFX export def generate_and_save_cert() -> None: """Generate RSA key and self-signed cert, save PEM and PFX. Can be commented out. """ key = rsa.generate_private_key( public_exponent=65537, key_size=2048, ) subject = issuer = x509.Name( [ x509.NameAttribute(NameOID.COUNTRY_NAME, "US"), x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "California"), x509.NameAttribute(NameOID.LOCALITY_NAME, "San Francisco"), x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Test Org"), x509.NameAttribute(NameOID.COMMON_NAME, "localhost"), ] ) cert = ( x509.CertificateBuilder() .subject_name(subject) .issuer_name(issuer) .public_key(key.public_key()) .serial_number(x509.random_serial_number()) .not_valid_before( datetime.now( datetime.utcnow() # for backward compatibility ) ) .not_valid_after( datetime.now( datetime.utcnow() # for backward compatibility ) + timedelta(days=365) ) .sign(key, hashes.SHA256()) ) Path(CERT_PEM).write_bytes(cert.public_bytes(serialization.Encoding.PEM)) Path(KEY_PEM).write_bytes( key.private_bytes( encoding=serialization.Encoding.PEM, format=serialization.PrivateFormat.TraditionalOpenSSL, encryption_algorithm=serialization.NoEncryption(), ) ) pfx = pkcs12.serialize_key_and_certificates( name=b"client", key=key, cert=cert, cas=None, encryption_algorithm=serialization.BestAvailableEncryption(CERT_PASSWORD), ) Path(PFX_FILE).write_bytes(pfx) def post_with_cert(url: str, idx: int) -> None: """Perform a POST request using PEM cert.""" try: response = requests.get( url, data={"test": f"thread-{idx}"}, cert=(CERT_PEM, KEY_PEM), # PEM files timeout=10, ) print(f"Thread {idx}: Status {response.status_code}") except Exception as exc: print(f"Thread {idx}: Exception {exc}") def main() -> None: generate_and_save_cert() url = "https://example.com" # <-- Change this! with ThreadPoolExecutor(max_workers=150) as executor: for i in range(150): executor.submit(post_with_cert, url, i) if __name__ == "__main__": main() ``` Here's the crash dump: ``` Core was generated by `/usr/local/bin/python3.15 foo.py'. Program terminated with signal SIGABRT, Aborted. #0 0x0000771454ecf624 in ?? () from /usr/lib/libc.so.6 [Current thread is 1 (Thread 0x7713720006c0 (LWP 87696))] #0 0x0000771454ecf624 in ?? () from /usr/lib/libc.so.6 #1 0x0000771454e75ba0 in raise () from /usr/lib/libc.so.6 #2 0x0000771454e5d582 in abort () from /usr/lib/libc.so.6 #3 0x0000771454e5e3bf in ?? () from /usr/lib/libc.so.6 #4 0x0000771454ed9765 in ?? () from /usr/lib/libc.so.6 #5 0x0000771454edbc8a in ?? () from /usr/lib/libc.so.6 #6 0x0000771454ede9ab in free () from /usr/lib/libc.so.6 #7 0x0000771453dc2eb5 in RSA_free () from /usr/lib/libcrypto.so.3 #8 0x0000771453d5ebd2 in ?? () from /usr/lib/libcrypto.so.3 #9 0x0000771453d5f498 in EVP_PKEY_free () from /usr/lib/libcrypto.so.3 #10 0x0000771454198e8a in ?? () from /usr/lib/libssl.so.3 #11 0x000077145419e4d6 in SSL_CTX_use_PrivateKey_file () from /usr/lib/libssl.so.3 #12 0x000077145429f899 in _ssl__SSLContext_load_cert_chain_impl (self=0x7714536f9b50, certfile=<optimized out>, keyfile=<optimized out>, password=<optimized out>) at ./Modules/_ssl.c:4148 #13 _ssl__SSLContext_load_cert_chain (self=0x7714536f9b50, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ./Modules/clinic/_ssl.c.h:1429 #14 0x0000586280553ca0 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x771454583ba0, args=0x7714540e89e0, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:169 #15 PyObject_Vectorcall (callable=0x771454583ba0, args=args@entry=0x771371ffe468, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327 #16 0x00005862806cda53 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:1619 #17 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #18 _PyEval_Vector (tstate=0x58629578de30, func=0x7714538ee090, locals=0x0, args=0x771452165530, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #19 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714538ee090, args=0x771452165530, nargsf=4, kwnames=0x7714521800b0) at ./Include/internal/pycore_call.h:169 #20 method_vectorcall (method=<optimized out>, args=0x771452165538, nargsf=<optimized out>, kwnames=0x7714521800b0) at Objects/classobject.c:64 #21 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x77145217da00, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:285 #22 _PyObject_Call (tstate=0x58629578de30, callable=0x77145217da00, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348 #23 PyObject_Call (callable=0x77145217da00, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373 #24 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654 #25 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #26 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361b1c0, locals=0x0, args=0x771452157730, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #27 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361b1c0, args=0x771452157730, nargsf=2, kwnames=0x771452164160) at ./Include/internal/pycore_call.h:169 #28 method_vectorcall (method=<optimized out>, args=0x771452157738, nargsf=<optimized out>, kwnames=0x771452164160) at Objects/classobject.c:64 #29 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452157680, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:285 #30 _PyObject_Call (tstate=0x58629578de30, callable=0x771452157680, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348 #31 PyObject_Call (callable=0x771452157680, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373 #32 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654 #33 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #34 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361be20, locals=0x0, args=0x771452157b70, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #35 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361be20, args=0x771452157b70, nargsf=2, kwnames=0x771452164700) at ./Include/internal/pycore_call.h:169 #36 method_vectorcall (method=<optimized out>, args=0x771452157b78, nargsf=<optimized out>, kwnames=0x771452164700) at Objects/classobject.c:64 #37 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452156b80, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:285 #38 _PyObject_Call (tstate=0x58629578de30, callable=0x771452156b80, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348 #39 PyObject_Call (callable=0x771452156b80, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373 #40 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654 #41 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #42 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361b8a0, locals=0x0, args=0x7714521553f0, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #43 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361b8a0, args=0x7714521553f0, nargsf=1, kwnames=0x771452127d60) at ./Include/internal/pycore_call.h:169 #44 method_vectorcall (method=<optimized out>, args=0x7714521553f8, nargsf=<optimized out>, kwnames=0x771452127d60) at Objects/classobject.c:64 #45 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452154c40, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:285 #46 _PyObject_Call (tstate=0x58629578de30, callable=0x771452154c40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348 #47 PyObject_Call (callable=0x771452154c40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373 #48 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654 #49 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #50 _PyEval_Vector (tstate=0x58629578de30, func=0x7714549d1170, locals=0x0, args=0x771371fff938, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #51 0x000058628055802b in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714549d1170, args=0x771371fff938, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:169 #52 method_vectorcall (method=<optimized out>, args=0x771371fffbd8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72 #53 0x00005862806f784c in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x771452154800, args=0x771371fffbd8, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:169 #54 context_run (self=0x7714521549c0, args=0x771371fffbd0, nargs=<optimized out>, kwnames=0x0) at Python/context.c:728 #55 0x00005862806ceefe in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:3764 #56 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119 #57 _PyEval_Vector (tstate=0x58629578de30, func=0x7714549d1220, locals=0x0, args=0x771371fffdd8, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961 #58 0x000058628055802b in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714549d1220, args=0x771371fffdd8, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:169 #59 method_vectorcall (method=<optimized out>, args=0x586280a26958 <_PyRuntime+90104>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72 #60 0x00005862807ed50c in thread_run (boot_raw=0x58629578ddf0) at ./Modules/_threadmodule.c:368 #61 0x000058628076d957 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242 #62 0x0000771454ecd70a in ?? () from /usr/lib/libc.so.6 #63 0x0000771454f51aac in ?? () from /usr/lib/libc.so.6 ``` Since I did run the test on multiple versions: - I did run the test on official Docker Python images (except 3.9/3.10/3.14/3.15) - My Docker `openssl version`: `OpenSSL 3.0.16 11 Feb 2025 (Library: OpenSSL 3.0.16 11 Feb 2025)` - My host `openssl version`: `OpenSSL 3.4.1 11 Feb 2025 (Library: OpenSSL 3.4.1 11 Feb 2025)` - For CPython main branch, here's my `python -VV`: `Python 3.15.0a0 (heads/main-dirty:1729468016, May 23 2025, 14:52:55) [GCC 14.2.1 20250207]` - My distrib: `Manjaro Linux 25.0.0` - `/proc/version`: `Linux version 6.11.5-lqx1-1-lqx (linux-lqx@archlinux) (gcc (GCC) 14.2.1 20240910, GNU ld (GNU Binutils) 2.43.0) #1 ZEN SMP PREEMPT Tue, 22 Oct 2024 15:40:56 +0000` ### CPython versions tested on: 3.10, 3.11, 3.12, 3.13, 3.14, CPython main branch, 3.9 ### Operating systems tested on: Linux ### Output from running 'python -VV' on the command line: _No response_ ### Linked PRs * gh-134724 * gh-137107 * gh-137126 Prompt：Issue原文 成果分析： 总体来说就是头痛医头，脚痛医脚。DeepSeek V4确实可以找到问题并应用修复，但是它只会在一个点上进行修复，修复并不完整，缺乏全局视角。考虑到大多数LLM都有这个问题，并且为了公平起见毫无上下文引导，我个人认为倒是可以给出一个合格的成绩，不过距离真的生产可用还是有一段距离的。 2 个帖子 - 2 位参与者 阅读完整话题