In this work, we introduce the \emph{interval permutation segment (IP-SEG)} model that naturally generalizes the geometric intersection models of interval and permutation graphs. We study properties of two graph classes that arise from the IP-SEG model and present a family of forbidden subgraphs for these classes. In addition, we present polynomial algorithms for the clique and independent set problems on these classes, when the model is given as part of the input.