Remote multiplayer interactive physical gaming with mobile computing devices
Methods and systems for remote real-time or asynchronous multiplayer gameplay using mobile devices are disclosed. The methods and systems include steps of mounting mobile computing devices in gaming areas associated with users; initiating a game between the users; analyzing movements of the users, ball(s), and/or other game equipment of the game by performing a computer vision algorithm on one or more frames of an input video of the game; and providing feedback to the users, where feedback may include instructions to be performed by the users and player analytics. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps.
1. A method for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, comprising:
joining a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;
capturing a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on a first mobile computing device in a first gaming area associated with the first player;
extracting a first posture flow of the first player from the first game video, by performing a computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;
generating a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;
receiving a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of the second player in a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and
generating a feedback to the first player based on at least one of the first player analytic and the second player analytic.
2. The method of claim 1 , wherein the second gaming area is geographically different from the first gaming area.
3. The method of claim 1 , wherein the first play and the second play of the multiplayer interactive physical sports game are asynchronous in time.
4. The method of claim 1 , further comprising:
receiving a user input from the first player to initiate the multiplayer interactive physical sports game, wherein the user input is selected from the group consisting of a voice input of a predetermined phrase and a gesture input of a predetermined gesture sequence to the first mobile computing device.
5. The method of claim 1 , further comprising:
detecting a shot attempt from the first game video of the multiplayer interactive physical sports game, wherein the multiplayer interactive physical sports game is a ball game, and wherein the generating of the first player analytic is based on an outcome of the shot attempt.
6. The method of claim 1 , further comprising:
providing the feedback to the first player using an audio output.
7. The method of claim 1 , wherein the generating of the feedback is live during the first play of the multiplayer interactive physical sports game.
8. The method of claim 1 , wherein the feedback to the first player are instructions for actions to be performed by the first player.
9. The method of claim 1 , further comprising:
determining a foot location of the first player in the first gaming area from the first game video, wherein the generating of the feedback is further based on the foot location of the first player.
10. A system for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, comprising:
at least one processor on a first mobile computing device; and
a non-transitory physical medium for storing program code accessible by the at least one processor, the program code when executed by the processor causes the processor to:
join a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;
capture a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on the first mobile computing device in a first gaming area associated with the first player;
extract a first posture flow of the first player from the first game video, by performing a first computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;
generate a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;
receive a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and
generate a feedback to the first player based on at least one of the first player analytic and the second player analytic.
11. The system of claim 10 , wherein the second gaming area is geographically different from the first gaming area.
12. The system of claim 10 , wherein the first play and the second play of the multiplayer interactive physical sports game are asynchronous in time.
13. The system of claim 10 , wherein the program code when executed by the processor further causes the processor to:
receive a user input from the first player to initiate the multiplayer interactive physical sports game, wherein the user input is selected from the group consisting of a voice input of a predetermined phrase and a gesture input of a predetermined gesture sequence to the first mobile computing device.
14. The system of claim 10 , wherein the program code when executed by the processor further causes the processor to:
detect a shot attempt from the first game video of the multiplayer interactive physical sports game, wherein the multiplayer interactive physical sports game is a ball game, and wherein the generating of the first player analytic is based on an outcome of the shot attempt.
15. The system of claim 10 , wherein the program code when executed by the processor further causes the processor to:
provide the feedback to the first player using an audio output.
16. The system of claim 10 , wherein the generating of the feedback is live during the first play of the multiplayer interactive physical sports game.
17. The system of claim 10 , wherein the feedback to the first player are instructions for actions to be performed by the first player.
18. A non-transitory physical storage medium for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, the storage medium comprising program code stored thereon, and the program code when executed by a processor causes the processor to:
join a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;
capture a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on a first mobile computing device in a first gaming area associated with the first player;
extract a first posture flow of the first player from the first game video, by performing a first computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;
generate a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;
receive a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and
generate a feedback to the first player based on at least one of the first player analytic and the second player analytic.