With more than 500 million daily tweets from over 330 million active users, Twitter constantly attracts malicious users aiming to carry out phishing and malware-related attacks against its user base. It therefore becomes of paramount importance to assess the effectiveness of Twitter's use of blacklists in protecting its users from such threats. We collected more than 182 million public tweets containing URLs from Twitter's Stream API over a 2-month period and compared these URLs against 3 popular phishing, social engineering, and malware blacklists, including Google Safe Browsing (GSB). We focus on the delay period between an attack URL first being tweeted to appearing on a blacklist, as this is the timeframe in which blacklists do not warn users, leaving them vulnerable. Experiments show that, whilst GSB is effective at blocking a number of social engineering and malicious URLs within 6 hours of being tweeted, a significant number of URLs go undetected for at least 20 days. For instance, during one month, we discovered 4,930 tweets containing URLs leading to social engineering websites that had been tweeted to over 131 million Twitter users. We also discovered 1,126 tweets containing 376 blacklisted Bitly URLs that had a combined total of 991,012 clicks, posing serious security and privacy threats. In addition, an equally large number of URLs contained within public tweets remain in GSB for at least 150 days, raising questions about potential false positives in the blacklist. We also provide evidence to suggest that Twitter may no longer be using GSB to protect its users.