A social behavior analysis is used to study how a group of people interacts with another group. The analysis helps to understand how social behavior leads to its consequences such as what business decision is made after a businessmen's meeting. In this paper, we focus on visual human motion analysis which is one important component of social behavior analysis. Human motion analysis in visual surveillance usually tracks the motion of an individual or a group of people, yet social behavior is usually neglected. This paper first delivers a literature survey of visual surveillance, with emphasis on aspects of human motion analysis and social behavior. Second, it offers a perspective for social behavior analysis in an intelligent visual human monitoring system. Third, a social interaction is induced by a social behavior between two persons, one person and one group, or two groups, hence three general scenarios of social interactions are outlined for future theoretical development. The proposed human monitoring system enables the generation of valuable quantified information of social interactions. It provides an objective approach to evaluate performance of a human organization such as a company or a school. Finally, it can raise the awareness of researchers to further explore the field of social behavior analysis.