Intelligent reflecting surfaces (IRSs) have the potential to transform wireless communication channels into smart reconfigurable propagation environments. To realize this new paradigm, the passive IRSs have to be large, especially for communication in far-field scenarios, so that they can compensate for the large end-to-end path-loss, which is caused by the multiplication of the individual path-losses of the transmitter-to-IRS and IRS-to-receiver channels. However, optimizing a large number of sub-wavelength IRS elements imposes a significant challenge for online transmission. To address this issue, in this paper, we develop a physics-based model and a scalable optimization framework for large IRSs. The basic idea is to partition the IRS unit cells into several subsets, referred to as tiles, model the impact of each tile on the wireless channel, and then optimize each tile in two stages, namely an offline design stage and an online optimization stage. For physics-based modeling, we borrow concepts from the radar literature, model each tile as an anomalous reflector, and derive its impact on the wireless channel for a given phase shift by solving the corresponding integral equations for the electric and magnetic vector fields. In the offline design stage, the IRS unit cells of each tile are jointly designed for the support of different transmission modes, where each transmission mode effectively corresponds to a given configuration of the phase shifts that the unit cells of the tile apply to an impinging electromagnetic wave. In the online optimization stage, the best transmission mode of each tile is selected such that a desired quality-of-service (QoS) criterion is maximized. We show that the proposed modeling and optimization framework can be used to efficiently optimize large IRSs comprising thousands of unit cells.