惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

N
News and Events Feed by Topic
爱范儿
爱范儿
Blog — PlanetScale
Blog — PlanetScale
The GitHub Blog
The GitHub Blog
C
Check Point Blog
小众软件
小众软件
I
InfoQ
罗磊的独立博客
H
Hackread – Cybersecurity News, Data Breaches, AI and More
Engineering at Meta
Engineering at Meta
酷 壳 – CoolShell
酷 壳 – CoolShell
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
Hugging Face - Blog
Hugging Face - Blog
博客园 - 三生石上(FineUI控件)
MyScale Blog
MyScale Blog
The Cloudflare Blog
Last Week in AI
Last Week in AI
腾讯CDC
Y
Y Combinator Blog
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
雷峰网
雷峰网
B
Blog
T
Tailwind CSS Blog
MongoDB | Blog
MongoDB | Blog
A
About on SuperTechFans
D
Docker
博客园 - 司徒正美
博客园_首页
Recent Announcements
Recent Announcements
D
DataBreaches.Net
阮一峰的网络日志
阮一峰的网络日志
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
G
Google Developers Blog
Microsoft Security Blog
Microsoft Security Blog
F
Fortinet All Blogs
Stack Overflow Blog
Stack Overflow Blog
aimingoo的专栏
aimingoo的专栏
N
Netflix TechBlog - Medium
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
博客园 - 聂微东
GbyAI
GbyAI
Jina AI
Jina AI
V
V2EX
Vercel News
Vercel News
IT之家
IT之家
WordPress大学
WordPress大学
M
MIT News - Artificial intelligence
NISL@THU
NISL@THU
V
Visual Studio Blog
C
Cybersecurity and Infrastructure Security Agency CISA

博客园 - MK2

cnpmcore 超大 JSON parse 性能优化 基于 Postgres 实现一个 Job Queue Graceful exit with cluster and pm Use Blanket.js instead of jscover 使用 connect-domain 捕获异步调用中出现的异常 Defense hash algorithm collision 防御hash算法冲突导致拒绝服务器 fibonacci(40) benchmark [nodejs]保证你的程序死了还能复活:forever and forever webui [nodejs]Buffer vs String Nodejs "Hello world" benchmark npm 资源库镜像 NodeBlog v0.2.0 更新说明 关于__proto__的链式记忆 NAE支持自定义域名了 Nodejs 离线文档下载 Nodejs HTTP请求的超时处理(Nodejs HTTP Client Request Timeout Handle) 在jQuery 1.5+ 的jqXHR上监听文件上传进度(xhr.upload) 关于Nodejs中Buffer释放的二三事 nodejs web开发入门: Simple-TODO Nodejs 实现版
nodejs读写大文件
MK2 · 2011-08-16 · via 博客园 - MK2

内存限制

大文件读写操作,由于内存限制问题,不要直接使用fs.readFilefs.writeFile
必须使用fs.ReadStreamfs.WriteStream 来对文件进行读写操作。

fs.ReadStream:上传大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function upload(url, uploadfile, callback) {
    var urlinfo = urlparse(url);
    var options = {
        method: 'POST',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var chunks = [], length = 0;
        res.on('data', function(chunk) {
            length += chunk.length;
            chunks.push(chunk);
        });
        res.on('end', function() {
            var buffer = new Buffer(length);
            // delay copy
            for(var i = 0, pos = 0, size = chunks.length; i < size; i++) {
                chunks[i].copy(buffer, pos);
                pos += chunks[i].length;
            }
            res.body = buffer;
            callback(null, res);
        });
    });
    var readstream = fs.createReadStream(uploadfile);
    readstream.on('data', function(chunk) {
        console.log('write', chunk.length);
        req.write(chunk);
    });
    readstream.on('end', function() {
        req.end();
    });
};

upload('http://weibo.com/', '/tmp/bigfile.pdf', function(err, res) {
    console.log(res.statusCode, res.headers);
});

fs.WriteStream:下载大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function download(url, savefile, callback) {
    console.log('download', url, 'to', savefile)
    var urlinfo = urlparse(url);
    var options = {
        method: 'GET',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var writestream = fs.createWriteStream(savefile);
        writestream.on('close', function() {
            callback(null, res);
        });
        res.pipe(writestream);
    });
    req.end();
};

download('http://weibo.com/', '/tmp/downfile.html', function(err, res) {
    console.log(res.statusCode, res.headers);
});

有爱

本文示例中为了凸显文件操作部分,异常处理部分代码已去除,请读者自行补全。
^_^ 希望本文对你有用