Silicon-based analog neural networks physically embody the ideal neural network model in an approximate way. We show that by retraining the neural network using a physics-informed hardware-aware model one can fully recover the inference accuracy of the ideal network model even in the presence of significant non-idealities. This is way more promising for scalability and integration density than the default option of improving the fidelity of the analog neural network at the cost of significant energy, area, and design overhead, through extensive calibration and conservative analog design. We first present a physics-informed hardware-aware model for a time-domain vector-matrix multiplier implemented with single-transistor floating-gate memory cells that explicitly accounts for two dominant non-idealities of the physical implementation - capacitive crosstalk and bit-line voltage drop - and integrates seamlessly with modern deep-learning workflows. The model discretizes each operation into adaptive time slots, processes activation patterns in parallel, and accumulates their contributions to predict effective multiplier outputs. Using measurements from a 16x16 silicon array, we calibrate the model, show that crosstalk is layout-dependent and often dominant, and introduce an improved weight-extraction procedure that doubles signal-to-error ratio versus an ideal vector-matrix multiplier model. Finally, we show that by training silicon-based analog neural networks using an hardware-aware model in the forward pass we can recover the accuracy of the ideal software networks across three architectures -- custom MLP on low-resolution MNIST, LeNet-5 on MNIST, and a VGG-style CNN on CIFAR-10 - establishing a complete design-to-deployment workflow for time-domain analog neuromorphic chips.