As the number of blockchain platforms continues to grow, the independence of these networks poses challenges for transferring assets and information across chains. Cross-chain bridge technology has emerged to address this issue, establishing communication protocols to facilitate cross-chain interaction of assets and information, thereby enhancing user experience. However, the complexity of cross-chain transactions increases the difficulty of security regulation, rendering traditional single-chain detection methods inadequate for cross-chain scenarios. Therefore, understanding cross-chain transaction semantics is crucial, as it forms the foundation for cross-chain security detection tasks. Although there are existing methods for extracting transaction semantics specifically for single chains, these approaches often overlook the unique characteristics of cross-chain scenarios, limiting their applicability. This paper introduces XSema, a novel cross-chain semantic extraction framework grounded in asset transfer and message-passing, designed specifically for cross-chain contexts. Experimental results demonstrate that XSema effectively distinguishes between cross-chain and non-cross-chain transactions, surpassing existing methods by over 9% for the generality metric and over 10% for the generalization metric. Furthermore, we analyze the underlying asset transfer patterns and message-passing event logs associated with cross-chain transactions. We offer new insights into the coexistence of multiple blockchains and the cross-chain ecosystem.