Wrist accelerometers for assessing hallmark measures of physical activity (PA) are rapidly growing with the advent of smartwatch technology. Given the growing popularity of wrist-worn accelerometers, there needs to be a rigorous evaluation for recognizing (PA) type and estimating energy expenditure (EE) across the lifespan. Participants (66% women, aged 20-89 yrs) performed a battery of 33 daily activities in a standardized laboratory setting while a tri-axial accelerometer collected data from the right wrist. A portable metabolic unit was worn to measure metabolic intensity. We built deep learning networks to extract spatial and temporal representations from the time-series data, and used them to recognize PA type and estimate EE. The deep learning models resulted in high performance; the F1 score was: 0.82, 0.81, and 95 for recognizing sedentary, locomotor, and lifestyle activities, respectively. The root mean square error was 1.1 (+/-0.13) for the estimation of EE.