Smart contract is the building block of blockchain systems that enables automated peer-to-peer transactions and decentralized services. With the increasing popularity of smart contracts, blockchain systems, in particular Ethereum, have been the "paradise" of versatile fraud activities in which Ponzi, Honeypot and Phishing are the prominent ones. Formal verification and symbolic analysis have been employed to combat these destructive scams by analyzing the codes and function calls, yet the vulnerability of each \emph{individual} scam should be predefined discreetly. In this work, we present SCSGuard, a novel deep learning scam detection framework that harnesses the automatically extractable bytecodes of smart contracts as their new features. We design a GRU network with attention mechanism to learn from the \emph{N-gram bytecode} patterns, and determines whether a smart contract is fraudulent or not. Our framework is advantageous over the baseline algorithms in three aspects. Firstly, SCSGuard provides a unified solution to different scam genres, thus relieving the need of code analysis skills. Secondly, the inference of SCSGuard is faster than the code analysis by several order of magnitudes. Thirdly, experimental results manifest that SCSGuard achieves high accuracy (0.92$\sim$0.94), precision (0.94$\sim$0.96\%) and recall (0.97$\sim$0.98) for both Ponzi and Honeypot scams under similar settings, and is potentially useful to detect new Phishing smart contracts.