Here's a problem: I'm pretty sure you'd get just as good (probably superior) results with a simpler, faster out-of-the-box algorithm like logistic regression, random forests or boosted trees (depending on the amount of data). This just isn't a problem that seems well suited for neural networks.
Gyroscope/accelerometer data might seem like the kind of extremely-noisy, high-dimensional data well-suited for neural nets, but 1) you don't have a lot of data, and 2) intuitively the data isn't actually that noisy; my guess is the data separates out quite nicely (not necessarily linearly).
I agree and disagree. I would definitely start with a simpler algorithm like logistic regression or even trees and use temporal features like power at different frequencies by taking an FFT. I would even add time lagged features. After that I'd graduate to a single hidden layer MLP.
If I had tons of data (a lot of different people using it), I'd experiment with LSTM neural networks because I think the temporal information is crucial for determining a movement.
I think a killer app would be aimed at weight lifters. If you go to the gym, the serious weight lifters record their reps and weight for each exercise. The app would utilize the iWatch or some other wearable and detect the exercise and count the reps. Then it would prompt the user for how much weight they used.
You're right on the ball. I'm indeed using temporal features to classify, along with various other statistical ones.
The repetition counting in my next aim. Once apple opens up the gyro on the Watch, I think there could be a good chance to get something like this out the door.
I'm still pretty sure you'd get superior results with other standard techniques for time series data (HMMs, conditional random fields, etc.). You really need a crapload of data to train an RNN well.
You're right that I could get just as good results with other potentially faster algorithms but those algorithms don't afford me the ability (especially decision trees) to personalise.
The system is designed to be personalised. So for example if it's not doing well in recognising your squats for instance, you can immediately 'show' it what they look like. A neural network can immediately integrate that, and spit out great accuracy levels. Whilst i'd have to reconstruct a decision tree each time.
I never even considered logistic regression for this problem. Now that you've pointed it out, I'm going to look into.
Neural networks aren't really core part of the system. As long as I can get good + quick results, I'm really open to swapping out the learning algorithm.
"Wouldn’t a better approach be to initially train the neural network on more than just one person?
That is a great point! In fact Microsoft’s research arm published a paper last year doing exactly that. Although they achieved great results, their initial training cohort consisted of 94 participants! I, as a one man team, can’t possibly duplicate that. This is why I created a system that can adapt, eliminating the need of Microsoft level resources."
--
Rather than using an accelerometer (OP attached an old iPhone to his person while doing the movements) - you could plausibly point a camera and try to use video as the input.
If that pans out the next step would be using Mechanical Turk to cheaply and quickly build the initial training set (no pun intended) using publicly available videos of people working out.
That's an interesting point. The problem here would be that video input would not provide accel+gryo motion information, which is what the network needs in order to learn.
If we instead simply used video information to track exercises the problem would be scaling that to consumers. They'd require an external camera to watch them.
But the idea of using Mechanical Turk is quite smart. It'd help me get varied form - especially if I can get them to wear a band/phone that has the sensors.
And this what happens when you bring a Computer Scientist into the gym! One question: how well would a neural network recognize a more complex and involved exercise like Kettlebell Turkish Getup compared to an exercise with simple movement like Push-ups?
That's a good question! Turkish Getup's are very complex like you say. Given my current setup and window length, it's unlikely that it'll recognise them, because one rep is very long. Maybe if I did exercise based window lengths it could work better? I'm honestly not sure. But that is something to try out!
That's awesome!
I didn't think this would work but it does. It would be amazing to combine this with a training tracking app. So far the data entry has always been really annoying so only machine exercises could really be captured properly. Really exciting stuff. I think this has a lot of potential for a startup!
This might be a good chance to ask something that's been bothering me -- is there a good training log format for weightlifting?
I can find apps for recording training data, but they all use their own formats. Anyway, if there is one it would be cool if this program could use it as output, instead of making up its own ad-hoc format.
Actually thought about an app that would do this on a smartwatch. I would definitely buy a smartwatch if I could have this level of accuracy of body movement tracking.
Hi. FocusMotion here. You you download and play with our SDK that works on any wearable device with an open accelerometer (Apple Watch, Android Wear, Microsoft Band, Pebble). We track 50 exercises, auto classify 22, and there's a machine learning tool that let's you add new movements to the system and improve it over time.
The article says the second phone is just for accelerometer data, so it could be replaced by some sort of Bluetooth device. I'm sure one of the many fitness trackers on the market provide real time data.
Not truly open, but we have an SDK similar to this that you can play with.
It works with any open accelerometer (Apple Watch, Android Wear, Microsoft Band, Pebble). We track 50 exercises, auto classify 22, and there's a machine learning tool that let's you add new movements to the system and improve it over time.
Gyroscope/accelerometer data might seem like the kind of extremely-noisy, high-dimensional data well-suited for neural nets, but 1) you don't have a lot of data, and 2) intuitively the data isn't actually that noisy; my guess is the data separates out quite nicely (not necessarily linearly).
If I had tons of data (a lot of different people using it), I'd experiment with LSTM neural networks because I think the temporal information is crucial for determining a movement.
I think a killer app would be aimed at weight lifters. If you go to the gym, the serious weight lifters record their reps and weight for each exercise. The app would utilize the iWatch or some other wearable and detect the exercise and count the reps. Then it would prompt the user for how much weight they used.
The repetition counting in my next aim. Once apple opens up the gyro on the Watch, I think there could be a good chance to get something like this out the door.
The system is designed to be personalised. So for example if it's not doing well in recognising your squats for instance, you can immediately 'show' it what they look like. A neural network can immediately integrate that, and spit out great accuracy levels. Whilst i'd have to reconstruct a decision tree each time.
Neural networks aren't really core part of the system. As long as I can get good + quick results, I'm really open to swapping out the learning algorithm.
--
Rather than using an accelerometer (OP attached an old iPhone to his person while doing the movements) - you could plausibly point a camera and try to use video as the input.
If that pans out the next step would be using Mechanical Turk to cheaply and quickly build the initial training set (no pun intended) using publicly available videos of people working out.
If we instead simply used video information to track exercises the problem would be scaling that to consumers. They'd require an external camera to watch them.
But the idea of using Mechanical Turk is quite smart. It'd help me get varied form - especially if I can get them to wear a band/phone that has the sensors.
Ah, now I understand why it took so long for someone to do this! ;)
Also, identifying good form--that's an app that could help a lot of amateur athletes.
I can find apps for recording training data, but they all use their own formats. Anyway, if there is one it would be cool if this program could use it as output, instead of making up its own ad-hoc format.
If there is a standard out there, I'll definitely look at integrating that for the output.
www.focusmotion.io
It works with any open accelerometer (Apple Watch, Android Wear, Microsoft Band, Pebble). We track 50 exercises, auto classify 22, and there's a machine learning tool that let's you add new movements to the system and improve it over time.
www.focusmotion.io