惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

博客园_首页
I
InfoQ
The Register - Security
The Register - Security
L
LangChain Blog
H
Help Net Security
The GitHub Blog
The GitHub Blog
S
Schneier on Security
博客园 - 【当耐特】
W
WeLiveSecurity
Attack and Defense Labs
Attack and Defense Labs
IT之家
IT之家
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
Google DeepMind News
Google DeepMind News
The Cloudflare Blog
H
Heimdal Security Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
Y
Y Combinator Blog
雷峰网
雷峰网
N
Netflix TechBlog - Medium
Security Archives - TechRepublic
Security Archives - TechRepublic
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
L
Lohrmann on Cybersecurity
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
T
The Exploit Database - CXSecurity.com
P
Privacy & Cybersecurity Law Blog
G
GRAHAM CLULEY
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
V
Visual Studio Blog
博客园 - 聂微东
PCI Perspectives
PCI Perspectives
Last Week in AI
Last Week in AI
A
Arctic Wolf
宝玉的分享
宝玉的分享
T
The Blog of Author Tim Ferriss
S
Secure Thoughts
T
Threat Research - Cisco Blogs
GbyAI
GbyAI
云风的 BLOG
云风的 BLOG
D
Darknet – Hacking Tools, Hacker News & Cyber Security
S
SegmentFault 最新的问题
SecWiki News
SecWiki News
月光博客
月光博客
大猫的无限游戏
大猫的无限游戏
Schneier on Security
Schneier on Security
P
Proofpoint News Feed
博客园 - Franky
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
AI
AI
Engineering at Meta
Engineering at Meta

博客园 - MK2

cnpmcore 超大 JSON parse 性能优化 基于 Postgres 实现一个 Job Queue Graceful exit with cluster and pm Use Blanket.js instead of jscover 使用 connect-domain 捕获异步调用中出现的异常 Defense hash algorithm collision 防御hash算法冲突导致拒绝服务器 fibonacci(40) benchmark [nodejs]保证你的程序死了还能复活:forever and forever webui [nodejs]Buffer vs String Nodejs "Hello world" benchmark npm 资源库镜像 NodeBlog v0.2.0 更新说明 关于__proto__的链式记忆 NAE支持自定义域名了 Nodejs 离线文档下载 Nodejs HTTP请求的超时处理(Nodejs HTTP Client Request Timeout Handle) 在jQuery 1.5+ 的jqXHR上监听文件上传进度(xhr.upload) 关于Nodejs中Buffer释放的二三事 nodejs web开发入门: Simple-TODO Nodejs 实现版
nodejs读写大文件
MK2 · 2011-08-16 · via 博客园 - MK2

内存限制

大文件读写操作,由于内存限制问题,不要直接使用fs.readFilefs.writeFile
必须使用fs.ReadStreamfs.WriteStream 来对文件进行读写操作。

fs.ReadStream:上传大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function upload(url, uploadfile, callback) {
    var urlinfo = urlparse(url);
    var options = {
        method: 'POST',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var chunks = [], length = 0;
        res.on('data', function(chunk) {
            length += chunk.length;
            chunks.push(chunk);
        });
        res.on('end', function() {
            var buffer = new Buffer(length);
            // delay copy
            for(var i = 0, pos = 0, size = chunks.length; i < size; i++) {
                chunks[i].copy(buffer, pos);
                pos += chunks[i].length;
            }
            res.body = buffer;
            callback(null, res);
        });
    });
    var readstream = fs.createReadStream(uploadfile);
    readstream.on('data', function(chunk) {
        console.log('write', chunk.length);
        req.write(chunk);
    });
    readstream.on('end', function() {
        req.end();
    });
};

upload('http://weibo.com/', '/tmp/bigfile.pdf', function(err, res) {
    console.log(res.statusCode, res.headers);
});

fs.WriteStream:下载大文件

var urlparse = require('url').parse
  , http = require('http')
  , fs = require('fs');

function download(url, savefile, callback) {
    console.log('download', url, 'to', savefile)
    var urlinfo = urlparse(url);
    var options = {
        method: 'GET',
        host: urlinfo.host,
        path: urlinfo.pathname
    };
    if(urlinfo.port) {
        options.port = urlinfo.port;
    }
    if(urlinfo.search) {
        options.path += urlinfo.search;
    }
    var req = http.request(options, function(res) {
        var writestream = fs.createWriteStream(savefile);
        writestream.on('close', function() {
            callback(null, res);
        });
        res.pipe(writestream);
    });
    req.end();
};

download('http://weibo.com/', '/tmp/downfile.html', function(err, res) {
    console.log(res.statusCode, res.headers);
});

有爱

本文示例中为了凸显文件操作部分,异常处理部分代码已去除,请读者自行补全。
^_^ 希望本文对你有用