Stringent latency requirements in advanced Internet of Things (IoT) applications as well as an increased load on cloud data centers have prompted a move towards a more decentralized approach, bringing storage and processing of IoT data closer to the end-devices through the deployment of multi-purpose IoT gateways. However, the resource constrained nature and diversity of these gateways pose a challenge in developing applications that can be deployed widely. This challenge can be overcome with containerization, a form of lightweight virtualization, bringing support for a wide range of hardware architectures and operating system agnostic deployment of applications on IoT gateways. This paper discusses the architectural aspects of containerization, and studies the suitability of available containerization tools for multi-container deployment in the context of IoT gateways. We present containerization in the context of AGILE, a multi-container and micro-service based open source framework for IoT gateways, developed as part of a Horizon 2020 project. Our study of containerized services to perform common gateway functions like device discovery, data management and cloud integration among others, reveal the advantages of having a containerized environment for IoT gateways with regard to use of base image hierarchies and image layering for in-container and cross-container performance optimizations. We illustrate these results in a set of benchmark experiments in this paper.