Research papers on pose estimation show impressive accuracy numbers. Production apps on consumer devices tell a different story. Here's what I learned shipping real-time pose estimation to thousands of users across 22 sports.

I built SportsReflector, an AI coaching app that analyzes athletic form using Apple's Vision framework on-device. The app runs pose estimation at 30fps during live sessions and frame-by-frame during video analysis. This article covers the gap between what the documentation promises and what actually works when real users point their iPhones at themselves in gyms, courts, and living rooms.

The Model Options on iOS
Apple gives you three paths for pose estimation on iOS:

VNDetectHumanBodyPoseRequest (Vision framework)
Extracts 19 body points. Runs on the Neural Engine. No custom model needed. This is what most developers should use.

CreateML trained custom model
Train your own pose model with labeled data. More control over which points you detect. Requires training data which is expensive to create for sports.

Third-party CoreML models (MoveNet, BlazePose, PoseNet)
Converted from TensorFlow/PyTorch. More keypoints (33 with BlazePose vs 19 with Vision). Harder to optimize for Neural Engine. Often slower than Apple's native model.

What I chose and why:
Apple's VNDetectHumanBodyPoseRequest. The 19 keypoints are sufficient for scoring form in every sport I've tested. The performance is dramatically better than converted third-party models because Apple optimized it specifically for their Neural Engine silicon. On an iPhone 13 or newer, single-frame inference is 8-12ms — fast enough for 60fps processing with headroom.

The third-party models give you more keypoints but at 2-3x the inference cost. For a research project where accuracy matters more than speed, use BlazePose. For a consumer app where smooth real-time performance matters more than marginal accuracy, use Apple's native model.

What the Documentation Doesn't Tell You
Problem 1: Confidence scores vary wildly by body position
Apple's pose estimation returns a confidence score (0.0-1.0) for each keypoint. The documentation suggests filtering points below 0.5 confidence. In practice, this threshold is too aggressive for many athletic movements.

During a squat at the bottom position, hip keypoints regularly drop to 0.3-0.4 confidence because the thighs occlude the hip joint from the camera's perspective. During a boxing combination, the rear hand drops below 0.3 when it's behind the torso. During a tennis serve at trophy position, the tossing arm's wrist confidence drops when it crosses behind the head.

The fix was sport-specific confidence thresholds. For squats, I accept hip keypoints down to 0.2 and interpolate position from adjacent frames when confidence is low. For boxing, I track the rear hand's last known position and predict its current position when occluded. For tennis, I use temporal smoothing across 5-frame windows to maintain continuity through low-confidence phases.

What I'd tell other developers: don't use a single confidence threshold globally. Calibrate per body part and per activity type. The wrist needs different thresholds than the shoulder, and both need different thresholds during a squat vs a sprint.

Problem 2: Camera angle dramatically affects accuracy

The documentation shows pose estimation working with a straight-on camera view. Users don't set up cameras at optimal angles. They prop their phone against a water bottle on the gym floor. They lean it against a wall at a 30-degree angle. Their training partner holds it at head height. Their tripod is behind them.
Accuracy degrades significantly at angles beyond 30 degrees from perpendicular. Side views work well for sagittal-plane movements (squats, deadlifts, running gait). Front views work for frontal-plane movements (lateral raises, jumping jacks). But most users don't know which angle to use for which exercise.
The fix was adding a camera setup guide that shows the user exactly where to place their phone for each exercise type, plus an automatic angle quality check that warns users when their camera angle is suboptimal before they start recording.
What I'd tell other developers: never assume optimal camera placement. Your users will find every possible bad angle. Build angle detection into your pipeline and guide users toward good placement before processing.
Problem 3: Lighting conditions in gyms are terrible
Research papers evaluate pose estimation under controlled lighting. Gyms have mixed lighting — overhead fluorescents, natural light from windows, mirror reflections creating secondary light sources, and dark corners near cable machines.
Low-light conditions cause two problems. Frame noise reduces keypoint confidence. And auto-exposure adjustments cause momentary brightness shifts that confuse frame-to-frame tracking.
The fix was pre-processing frames with adaptive histogram equalization before feeding them to the pose estimator, plus locking camera exposure after the initial setup phase to prevent mid-recording brightness shifts.
What I'd tell other developers: test your pose estimation in the worst lighting you can find. If it works under a single dim bulb with mirror reflections, it'll work everywhere.
Problem 4: Clothing and equipment cause occlusion
Loose clothing (baggy gym shorts, hoodies, wide-leg joggers) hides joint positions. The pose estimator can't see the knee through baggy shorts, so it guesses — often incorrectly.
Equipment compounds this. A barbell across the shoulders during a squat occludes the neck and upper back keypoints. A tennis racket in the hand confuses wrist detection. Boxing gloves change hand proportions that the model expects.
There's no clean fix for this. The mitigations are temporal smoothing (use the trajectory of the joint over multiple frames to predict position during occlusion), anatomical constraints (the knee can't be above the hip during a squat, so cap estimates to physiologically possible ranges), and user guidance (suggest form-fitting clothing in the onboarding flow).
What I'd tell other developers: your pose estimation will fail on some percentage of users due to clothing and equipment. Build graceful degradation — lower confidence scores should trigger warnings to the user rather than wildly incorrect analysis.
Problem 5: Multiple people in frame
Gym environments frequently have other people in the background. The pose estimator detects all of them. If you're not careful, your analysis might score the person walking behind your user instead of your user.
The fix was implementing a "primary subject" tracking system. On the first frame, detect all bodies and select the largest (closest to camera) as the primary subject. Track that subject's position across frames using centroid tracking. Ignore all other detected bodies. If the primary subject disappears (walks out of frame), pause analysis and prompt the user to reposition.

What I'd tell other developers: always implement subject isolation if your app runs in environments with multiple people. Single-person pose estimation in multi-person environments is a pipeline problem, not a model problem.

Performance Optimization That Actually Matters
Frame skipping for battery life
Running pose estimation at 30fps continuously drains battery fast. For live AR feedback, you need every frame. For video analysis (deferred path), you don't.

For deferred analysis, I process every 3rd frame (10fps effective) for initial pass, then re-process key frames (phase transitions, lowest position in squat, peak extension in serve) at full resolution. This reduces processing time by 60% with negligible accuracy loss for scoring.

For live AR during workouts, I dynamically adjust processing frequency based on battery level. Above 50% battery, process every frame. Below 50%, process every 2nd frame. Below 20%, process every 3rd frame and show a battery warning.
Memory management during long sessions

A 60-minute workout session at 30fps generates 108,000 frames. Storing pose data for every frame would consume hundreds of megabytes. The fix is streaming analysis — process each frame, extract metrics, store only the aggregate metrics (per-rep scores, phase timestamps, anomaly flags), and discard the raw pose data.
For video analysis where users want to replay with skeleton overlay, store keypoints for key frames only (phase transitions, anomalies) and interpolate between them during playback.
Neural Engine vs GPU vs CPU
Apple's Neural Engine is fastest for pose estimation but isn't always available — it's shared with other system processes. The Vision framework automatically falls back to GPU or CPU when the Neural Engine is busy.
The problem: inference time varies 3-5x between Neural Engine (8ms) and CPU fallback (30-40ms). If your real-time pipeline assumes consistent 8ms inference, CPU fallback frames cause visible stuttering in the AR overlay.
The fix was building a frame budget system that measures actual inference time per frame and dynamically adjusts overlay rendering complexity. Fast inference: full skeleton with joint angles and color-coded feedback. Slow inference: simplified skeleton with key joints only. The user sees smooth animation regardless of which processor handles inference.
The Results
After shipping these optimizations, the production performance profile looks like this:

Real-time AR overlay: 30fps sustained on iPhone 12+, 60fps on iPhone 13 Pro+
Video analysis: 10-15 seconds for a 30-second clip
Battery consumption during 60-minute AR workout: approximately 15-20% battery drain
Crash rate related to pose estimation: 0.0% over 30 days (error boundaries catch all edge cases)
User-reported accuracy satisfaction: tracked through review sentiment

The gap between research demos and production apps is significant. Research optimizes for accuracy on clean datasets. Production optimizes for resilience across terrible camera angles, bad lighting, occluded joints, multiple subjects, and variable processing budgets. Understanding this gap early would have saved me months.

SportsReflector is available on the iOS App Store. Built with Apple's Vision framework, CoreML, and ARKit. If you're shipping pose estimation in production and dealing with similar issues, I'd love to compare notes.