惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Help Net Security
Help Net Security
S
Schneier on Security
Security Latest
Security Latest
C
Cyber Attacks, Cyber Crime and Cyber Security
博客园_首页
K
Kaspersky official blog
B
Blog
雷峰网
雷峰网
MyScale Blog
MyScale Blog
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
人人都是产品经理
人人都是产品经理
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
T
Threatpost
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Latest news
Latest news
J
Java Code Geeks
W
WeLiveSecurity
GbyAI
GbyAI
V
Vulnerabilities – Threatpost
S
Secure Thoughts
IT之家
IT之家
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
博客园 - 叶小钗
Engineering at Meta
Engineering at Meta
S
Security @ Cisco Blogs
罗磊的独立博客
M
MIT News - Artificial intelligence
Blog — PlanetScale
Blog — PlanetScale
Stack Overflow Blog
Stack Overflow Blog
月光博客
月光博客
P
Proofpoint News Feed
博客园 - 聂微东
云风的 BLOG
云风的 BLOG
N
Netflix TechBlog - Medium
D
DataBreaches.Net
S
SegmentFault 最新的问题
V
Visual Studio Blog
V
V2EX
The Register - Security
The Register - Security
N
News | PayPal Newsroom
aimingoo的专栏
aimingoo的专栏
C
Cybersecurity and Infrastructure Security Agency CISA
N
News and Events Feed by Topic
Hacker News: Ask HN
Hacker News: Ask HN
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
Google DeepMind News
Google DeepMind News
Webroot Blog
Webroot Blog
NISL@THU
NISL@THU
D
Darknet – Hacking Tools, Hacker News & Cyber Security

博客园 - xhan

提交给mysql java驱动的优化下个版本要发布了^_^ java mysql大数据量批量插入与流式读取分析 innodb next-key lock引发的死锁 jremoting的功能扩展点 java开源项目jremoting largest remainder method java impl 数据库单元测试 元数据编程实战_使用Emit运行时生成Protobuf编码类 发布个c#版的HandlerSocket客户端类库 redis入门系列文章广告贴 九 redis学习笔记之虚拟内存 八 redis学习笔记之主从复制 七 redis学习笔记之持久化 六 redis学习笔记之发布订阅 五 redis学习笔记之pipeline 四 redis学习笔记之事务 三 redis学习笔记之排序 二 redis学习笔记之数据类型 - xhan - 博客园 一 redis学习笔记之环境搭建
java 可伸缩阻塞队列实现
xhan · 2015-07-05 · via 博客园 - xhan

最近一年多写的最虐心的代码。必须好好复习java并发了。搞了一晚上终于测试都跑通过了,特此纪念,以资鼓励!

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * 实现可调整大小的阻塞队列,支持数据迁移平衡reader,writer读取吸入速度,达到最大吞吐
 * @author hanjie
 *
 */
public class RecordBuffer {

    public static final Record CLOSE_RECORD = new Record() {

        @Override
        public Object getColumnValue(String columnName) {
            // TODO Auto-generated method stub
            return null;
        }
    };
    
    public static final Record SWITCH_QUEUE_RECORD = new Record() {

        @Override
        public Object getColumnValue(String columnName) {
            // TODO Auto-generated method stub
            return null;
        }
    };
    
    public Lock switchingQueueLock = new ReentrantLock();
    public Condition readerSwitched = switchingQueueLock.newCondition();
    public Condition writerSwitched = switchingQueueLock.newCondition();
    public Condition switchFinished = switchingQueueLock.newCondition();
    
    public volatile boolean readerSwitchSuccess = true;
    public volatile boolean writerSwitchSuccess = true;
    public volatile boolean switchingQueue = false;
    public volatile boolean closed = false;
    private volatile ArrayBlockingQueue<Record> queue;
    private TaskCounter taskCounter;
    

    public RecordBuffer(TaskCounter taskCounter, int size) {
        this.queue = new ArrayBlockingQueue<Record>(size);
        this.taskCounter = taskCounter;
    }



    public void resize(int newSize) {
        try {
            
            if(closed){
                return;
            }
    
            switchingQueueLock.lock();
            try {
                //double check下,要不可能writer收到CLOSED_record已经 退出了。writerSwitched.await() 会hang住
                if(closed){
                    return;
                }
                this.switchingQueue = true;
        
                ArrayBlockingQueue<Record> oldQueue = queue;
                queue = new ArrayBlockingQueue<Record>(newSize);
                this.readerSwitchSuccess = false;
                this.writerSwitchSuccess = false;
                
                //先拯救下writer,可能writer刚好阻塞到take上,失败也没关系,说明老队列不空,writer不会阻塞到take
                oldQueue.offer(SWITCH_QUEUE_RECORD);

                while (!writerSwitchSuccess) {
                    writerSwitched.await();
                }
                //writer先切换队列,然后reader可能阻塞在最后一个put上,清空下老队列拯救reader,让它顺利醒来
                transferOldQueueRecordsToNewQueue(oldQueue);
                
                
                while (!readerSwitchSuccess) {
                    readerSwitched.await();
                }
                //前面的清空,刚好碰到reader要put最后一个,非阻塞式清空动作就有残留最后一个put
                transferOldQueueRecordsToNewQueue(oldQueue);
                
                this.switchingQueue = false;
                this.switchFinished.signalAll();

            } finally {
                switchingQueueLock.unlock();
            }

        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private void transferOldQueueRecordsToNewQueue(ArrayBlockingQueue<Record> oldQueue)
            throws InterruptedException {
        List<Record> oldRecords = new ArrayList<Record>(oldQueue.size());
        Record record = null;
        while ((record = oldQueue.poll()) != null) {
            oldRecords.add(record);
        }
        // 转移老队列剩下的记录到新队列
        for (int i = 0; i < oldRecords.size(); i++) {
            queue.put(oldRecords.get(i));
        }
    }

    public void close() {
        this.closed = true;
        switchingQueueLock.lock();
        try {
            //如果正在切换队列, 等切换做完才能,发送最后一个CLOSE
            while (switchingQueue) {
                switchFinished.await();
            }
            
            this.queue.put(CLOSE_RECORD);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
        finally{
            switchingQueueLock.unlock();
        }
    }

    public void put(Record record) {
        try {
            
            if (!queue.offer(record)) {
                taskCounter.incrBufferFullCount();
                if (!readerSwitchSuccess) {
                    notifyReaderSwitchSuccess();
                }
                queue.put(record);
            }
            taskCounter.incrReadCount();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private void notifyReaderSwitchSuccess() {
        System.out.println("reader switch");
        switchingQueueLock.lock();
        try {
            readerSwitchSuccess = true;
            readerSwitched.signalAll();
        } finally {
            switchingQueueLock.unlock();
        }
    }

    public Record take() {
        try {
            
            Record record = queue.poll();
            //如果拿到了切换记录,则切换队列重试
            if(record == SWITCH_QUEUE_RECORD){
                if (!writerSwitchSuccess) {
                    notifyWriterSwitchSuccess();
                }
                record = queue.poll();
            }
            
            if (record == null) {
                taskCounter.incrBufferEmptyCount();
                
                //调用take先检查是否正在切换,保证拿到新的队列
                if (!writerSwitchSuccess) {
                    notifyWriterSwitchSuccess();
                }
                record = queue.take();
                //如果很不幸刚好在take阻塞时候,切换,只能发送一个切换记录将其唤醒
                if(record == SWITCH_QUEUE_RECORD){
                    if (!writerSwitchSuccess) {
                        notifyWriterSwitchSuccess();
                    }
                    record = queue.take();
                }
            }
            if (record == CLOSE_RECORD) {
                if (!writerSwitchSuccess) {
                    notifyWriterSwitchSuccess();
                }
                return null;
            }
            taskCounter.incrWriteCount();
            return record;
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private void notifyWriterSwitchSuccess() {

        System.out.println("writer switch");
        switchingQueueLock.lock();
        try {
            writerSwitchSuccess = true;
            writerSwitched.signalAll();
        } finally {
            switchingQueueLock.unlock();
        }

    }

}