With the rapid development of blockchain, Byzantine fault-tolerant protocols have attracted revived interest recently. To overcome the theoretical bounds of Byzantine fault tolerance, many protocols attempt to use Trusted Execution Environment (TEE) to prevent equivocation and improve performance and scalability. However, due to the broken quorum intersection assumption caused by the reduction of the replica number, the improvement is mostly at the cost of increased communication complexity which prevents existing TEE-based protocols to be applied to large-scale blockchain systems. In this paper, we propose TBFT, an efficient Byzantine fault-tolerant protocol in the partial synchrony setting, which has O(n) message complexity in both normal-case and view-change. Compared to previous protocols, TBFT uses TEE-assisted primitives to limit more types of malicious behaviors of replicas rather than preventing equivocation only, thereby reducing the latency and communication complexity of clients and replicas. Besides, we also introduce lightweight cryptographic primitives including a novel leader election mechanism and an efficient voting message aggregation mechanism for better security and performance. We evaluate TBFT via systematic analysis and experiments, and the results show that TBFT has better performance and scalability compared to other protocols.