Sentiment polarity of tweets, blog posts or product reviews has become highly attractive and is utilized in recommender systems, market predictions, business intelligence and more. Deep learning techniques are becoming top performers on analyzing such texts. There are however several problems that need to be solved for efficient use of deep neural networks on text mining and text polarity analysis. First, deep neural networks need to be fed with data sets that are big in size as well as properly labeled. Second, there are various uncertainties regarding the use of word embedding vectors: should they be generated from the same data set that is used to train the model or it is better to source them from big and popular collections? Third, to simplify model creation it is convenient to have generic neural network architectures that are effective and can adapt to various texts, encapsulating much of design complexity. This thesis addresses the above problems to provide methodological and practical insights for utilizing neural networks on sentiment analysis of texts and achieving state of the art results. Regarding the first problem, the effectiveness of various crowdsourcing alternatives is explored and two medium-sized and emotion-labeled song data sets are created utilizing social tags. To address the second problem, a series of experiments with large text collections of various contents and domains were conducted, trying word embeddings of various parameters. Regarding the third problem, a series of experiments involving convolution and max-pooling neural layers were conducted. Combining convolutions of words, bigrams, and trigrams with regional max-pooling layers in a couple of stacks produced the best results. The derived architecture achieves competitive performance on sentiment polarity analysis of movie, business and product reviews.