A large number of computational scientific research projects make use of open source software packages. However, the development process of such tools frequently differs from conventional software development; partly because of the nature of research, where the problems being addressed are not always fully understood; partly because the majority of the development is often carried out by scientists with limited experience and exposure to best practices of software engineering. Often the software development suffers from the pressure to publish scientific results and that credit for software development is limited in comparison. Fundamental components of software engineering like modular and reusable design, validation, documentation, and software integration as well as effective maintenance and user support tend to be disregarded due to lack of resources and qualified specialists. Thus innovative developments are often hindered by steep learning curves required to master development for legacy software packages full of ad hoc solutions. The growing complexity of research, however, requires suitable and maintainable computational tools, resulting in a widening gap between the potential users (often growing in number) and contributors to the development of such a package. In this paper we share our experiences aiming to improve the situation by training particularly young scientists, through disseminating our own experiences at contributing to open source software packages and practicing key components of software engineering adapted for scientists and scientific software development. Specifically we summarize the outcome of the Workshop in Advanced Techniques for Scientific Programming and Collaborative Development of Open Source Software Packages run at the Abdus Salam International Centre for Theoretical Physics in March 2013, and discuss our conclusions for future efforts.