惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

T
The Blog of Author Tim Ferriss
Hugging Face - Blog
Hugging Face - Blog
F
Fortinet All Blogs
B
Blog
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Microsoft Security Blog
Microsoft Security Blog
Blog — PlanetScale
Blog — PlanetScale
月光博客
月光博客
腾讯CDC
小众软件
小众软件
G
Google Developers Blog
V
Visual Studio Blog
罗磊的独立博客
GbyAI
GbyAI
V
V2EX
大猫的无限游戏
大猫的无限游戏
H
Help Net Security
L
LangChain Blog
Engineering at Meta
Engineering at Meta
量子位
The GitHub Blog
The GitHub Blog
博客园 - 司徒正美
WordPress大学
WordPress大学
B
Blog RSS Feed

博客园 - MK2

cnpmcore 超大 JSON parse 性能优化 基于 Postgres 实现一个 Job Queue Graceful exit with cluster and pm Use Blanket.js instead of jscover 使用 connect-domain 捕获异步调用中出现的异常 Defense hash algorithm collision 防御hash算法冲突导致拒绝服务器 fibonacci(40) benchmark [nodejs]保证你的程序死了还能复活:forever and forever webui [nodejs]Buffer vs String Nodejs "Hello world" benchmark npm 资源库镜像 NodeBlog v0.2.0 更新说明 关于__proto__的链式记忆 NAE支持自定义域名了 Nodejs 离线文档下载 Nodejs HTTP请求的超时处理(Nodejs HTTP Client Request Timeout Handle) 在jQuery 1.5+ 的jqXHR上监听文件上传进度(xhr.upload) 关于Nodejs中Buffer释放的二三事 nodejs web开发入门: Simple-TODO Nodejs 实现版
nodejs读写大文件
MK2 · 2011-08-16 · via 博客园 - MK2

内存限制

大文件读写操作,由于内存限制问题,不要直接使用fs.readFilefs.writeFile
必须使用fs.ReadStreamfs.WriteStream 来对文件进行读写操作。

fs.ReadStream:上传大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function upload(url, uploadfile, callback) {
    var urlinfo = urlparse(url);
    var options = {
        method: 'POST',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var chunks = [], length = 0;
        res.on('data', function(chunk) {
            length += chunk.length;
            chunks.push(chunk);
        });
        res.on('end', function() {
            var buffer = new Buffer(length);
            // delay copy
            for(var i = 0, pos = 0, size = chunks.length; i < size; i++) {
                chunks[i].copy(buffer, pos);
                pos += chunks[i].length;
            }
            res.body = buffer;
            callback(null, res);
        });
    });
    var readstream = fs.createReadStream(uploadfile);
    readstream.on('data', function(chunk) {
        console.log('write', chunk.length);
        req.write(chunk);
    });
    readstream.on('end', function() {
        req.end();
    });
};

upload('http://weibo.com/', '/tmp/bigfile.pdf', function(err, res) {
    console.log(res.statusCode, res.headers);
});

fs.WriteStream:下载大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function download(url, savefile, callback) {
    console.log('download', url, 'to', savefile)
    var urlinfo = urlparse(url);
    var options = {
        method: 'GET',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var writestream = fs.createWriteStream(savefile);
        writestream.on('close', function() {
            callback(null, res);
        });
        res.pipe(writestream);
    });
    req.end();
};

download('http://weibo.com/', '/tmp/downfile.html', function(err, res) {
    console.log(res.statusCode, res.headers);
});

有爱

本文示例中为了凸显文件操作部分,异常处理部分代码已去除,请读者自行补全。
^_^ 希望本文对你有用