Couldn’t attend Transform 2022? Check out all the top sessions in our on-demand library now! Look here.
While you’re doing push-ups, squats or abs, lifting dumbbells, jumping or stretching, a device on your TV follows you during your workout.
You are tracked on your form, your completion of an exercise (or lack thereof); you get recommendations on which cardio, body weight, strength or yoga workout to do next; and you can work on achievement badges.
This is the next level home fitness experience powered by Peloton Guide, a TV-mounted camera-based exercise device and system powered by computer vision, artificial intelligence (AI), advanced algorithms and synthetic data.
Sanjay Nichani, leader of Peloton’s computer vision group, discussed the technology’s development — and continuous improvement — in a live stream this week on Transform 2022.
AI-driven motivation
Peloton Guide’s computer vision tracks members and recognizes their activity, gives them recognition for completed moves, provides recommendations and real-time feedback. A “self-mode” mechanism also allows users to pan and zoom their device to view themselves on the screen and make sure they show the correct shape.
Nicani underlined the power of metric accountability when it comes to fitness, saying that “understanding and progress are very motivating.”
Achieving the Peloton Guide’s final commercial product was an “iterative process,” he said. The original goal of AI is to “start up quickly” by sourcing small amounts of custom data and combining it with open-source data.
Once a model is developed and implemented, detailed analysis, evaluation and telemetry are applied to continuously improve the system and make “targeted improvements,” Nicani said.
The machine learning (ML) flywheel “all starts with data,” he said. Peloton developers used real data supplemented with “a heavy dose of synthetic data,” creating data sets using nomenclature specific to exercises and postures combined with appropriate reference material.
Development teams also applied pose estimation and matching, accuracy recognition models and optical flow, what Nicani called a “classic computer vision technique.”
Various Attributes Affecting Computer Vision
One of the challenges of computer vision, Nichani said, is the “wide variety of attributes to take into account.”
This includes:
- Environmental features: background (walls, floors, furniture, windows); lighting, shadows, reflections; other people or animals in line of sight; equipment being used.
- Member Features: gender, skin color, body type, fitness level and clothing.
- Geometric Attributes: Camera user placement; camera mounting height and tilt; lid orientation and distance from the camera.
Peloton developers have conducted extensive field testing to enable edge cases and built in an ability that “push” users if the camera can’t distinguish them due to a number of factors, Nicani said.
The Bias Challenge
Fairness and inclusiveness are both paramount to the process of developing AI models, Nicani said.
The first step to reducing bias in models is to ensure that data is diverse and has enough values for different attributes for training and testing, he said.
Still, he noted, “a diverse dataset alone does not make for unbiased systems. Bias tends to creep in, in deep learning models, even if the data is unbiased.”
Through Peloton’s process, all source data is tagged with attributes. This allows models to measure performance across “different segments of attributes” so that no bias is observed in models before they go into production, explains Nicani.
If bias is detected, it is addressed – and ideally corrected – through the flywheel process and deep dive analysis. Nichani said Peloton developers observe an “equality of opportunity” fairness metric.
That is, “for a given label and attribute, a classifier predicts that label equally for all values of that attribute.”
For example, to predict whether a member is doing a crossbody curl, a squat, or a dumbbell swing, models were built to take into account characteristics of body type (“underweight”, “average”, “overweight”) and skin color based on the Fitzpatrick classification – which, while widely accepted for classifying skin color, still has a few in particular limits
Still, any challenges are more than offset by significant opportunities, Nicani said. AI has many implications for home fitness – from personalization to accountability to convenience (e.g. voice commands), to guidance to overall engagement.
Providing insights and metrics can improve a user’s performance “and really incentivize them to do more,” said Nicani. Peloton strives to provide personalized gaming experiences “so you don’t look at the clock when you’re training.”
Watch the full length talk of Transform 2022.
