IP Library › Granted Patent US 10,200,659
Granted Patent B2
US 10,200,659 · App. 15/056,922 · Granted Feb 5, 2019

Collaborative camera viewpoint control for interactive telepresence

Inventors: Nikolai Smolyanskiy (Seattle, WA); Zhengyou Zhang (Bellevue, WA); Vikram R. Dendi (Redmond, WA); Michael Hall (Snohomish, WA)
Assignee: Microsoft Technology Licensing, LLC
H04N7/185G02B27/017G05D1/0016G06Q10/101G06T19/006H04N5/23296G02B2027/0178
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,200,659
App. No.
15/056,922
Granted
Feb 5, 2019
Kind
B2
Abstract

In embodiments of collaborative camera viewpoint control for interactive telepresence, a system includes a vehicle that travels based on received travel instructions, and the vehicle includes a camera system of multiple cameras that each capture video of an environment in which the vehicle travels from different viewpoints. Viewing devices receive the video of the environment from the different viewpoints, where the video of the environment from a selected one of the viewpoints is displayable to users of the viewing devices. Controller devices that are associated with the viewing devices can each receive a user input as a proposed travel instruction for the vehicle based on the selected viewpoint of the video that is displayed on the viewing devices. A trajectory planner receives the proposed travel instructions initiated via the controller devices, and generates a consensus travel instruction for the vehicle based on the proposed travel instructions.

Claims (55)

1. A system implemented for collaborative camera viewpoint control, the system comprising:

a trajectory planner configured to:

receive, from a plurality of controller devices, proposed camera viewpoints;

weight each proposed camera viewpoint based on a user of a controller device associated with the each proposed camera viewpoint;

receive, from the plurality of controller devices, proposed travel instructions for a vehicle;

weight each proposed travel instruction based on the user of the controller device associated with the each proposed travel instruction;

generate a consensus travel instruction for the vehicle based on the weighted proposed travel instructions;

generate a consensus camera viewpoint for the vehicle based on the weighted proposed camera viewpoints; and

communicate the consensus travel instruction and the consensus camera viewpoint to the vehicle.

2. The system as recited in claim 1 , wherein the trajectory planner is further configured to:

rank different viewpoints of an environment of the vehicle from multiple cameras of the vehicle; and

provide the ranked different viewpoints to the plurality of controller devices, wherein the received proposed camera viewpoints are based on a selection of a ranked different viewpoint.

3. The system as recited in claim 1 , wherein the vehicle comprises:

the trajectory planner;

a camera system of multiple cameras configured to capture video of an environment in which the vehicle travels from the different viewpoints; and

a travel control system configured to control the travel of the vehicle based on the consensus travel instruction.

4. The system as recited in claim 1 , wherein the trajectory planner is implemented with a voting system to generate the consensus travel instruction based on each of the proposed travel instructions counted as a user vote to direct the travel of the vehicle.

5. The system as recited in claim 3 , wherein the vehicle is further configured to:

transmit to a first group of viewing devices video of the environment from a first camera having a first viewpoint;

transmit to a second group of viewing devices video of the environment from at least a second camera having a second viewpoint; and

the trajectory planner configured to receive the proposed travel instructions based on the different viewpoints of the environment.

6. The system as recited in claim 1 , wherein the trajectory planner is remote from the vehicle.

7. The system as recited in claim 1 , wherein the vehicle is at least one of a drone, a submersible, a land rover, a mobilized computing device, a human camera carrier, or a camera transport.

8. The system as recited in claim 1 , wherein the proposed travel instructions comprise at least one of stop, brake, start, accelerate, turn-off the vehicle, turn-on the vehicle, travel up, travel down, turn left, turn right, turn around, rotate, travel forward, or travel back.

9. A method for collaborative camera viewpoint control implemented by a vehicle, the method comprising:

receiving, from a plurality of controller devices, proposed camera viewpoints;

weighting each proposed camera viewpoint based on a user of a controller device associated with the each proposed camera viewpoint;

receiving, from the plurality of controller devices, proposed travel instructions for a vehicle;

weighting each proposed travel instruction based on the user of the controller device associated with the each proposed travel instruction;

generating a consensus travel instruction for the vehicle based on the weighted proposed travel instructions;

generating a consensus camera viewpoint for the vehicle based on the weighted proposed camera viewpoints; and

communicating the consensus travel instruction and the consensus camera viewpoint to the vehicle.

10. The method as recited in claim 9 , further comprising:

ranking the different viewpoints of an environment of the vehicle from multiple cameras of the vehicle; and

providing the ranked different viewpoints to the plurality of controller devices, wherein the received proposed camera viewpoints are based on a selection of a ranked different viewpoint.

11. The system as recited in claim 1 , further comprising a voting system configured to count user votes based on the received proposed camera viewpoints, wherein the consensus camera viewpoint instruction is determined based on the user votes.

12. The system as recited in claim 11 , further comprising a policy system configured to determine the weight for each proposed camera viewpoint based on the user of the controller device associated with the each proposed camera viewpoint.

13. The system as recited in claim 12 , wherein the weight for each proposed camera viewpoint is determined based on a user factor.

14. The system as recited in claim 13 , wherein the user factor is skill level of the user with a type of the vehicle.

15. The system as recited in claim 12 , wherein the policy system is further configured to override a proposed travel instruction such that the overrode proposed travel instruction is not the consensus travel instruction.

16. The system as recited in claim 15 , wherein the policy system is further configured to:

determine the vehicle is flying, wherein the overrode proposed travel instruction is to shut off the vehicle.

17. The method as recited in claim 9 , further comprising further counting user votes based on the received proposed camera viewpoints, wherein the consensus camera viewpoint instruction is determined based on the user votes.

18. The method as recited in claim 17 , determining the weight for each proposed camera viewpoint based on the user of the controller device associated with the each proposed camera viewpoint.

19. One or more non-transitory computer-readable media storing processor-executable instructions that cause one or more processors to perform operations that provides collaborative camera viewpoint control implemented by a vehicle, the operations comprising:

receiving, from a plurality of controller devices, proposed camera viewpoints;

weighting each proposed camera viewpoint based on a user of a controller device associated with the each proposed camera viewpoint;

receiving, from the plurality of controller devices, proposed travel instructions for a vehicle;

weighting each proposed travel instruction based on the user of the controller device associated with the each proposed travel instruction;

generating a consensus travel instruction for the vehicle based on the weighted proposed travel instructions;

generating a consensus camera viewpoint for the vehicle based on the weighted proposed camera viewpoints; and

communicating the consensus travel instruction and the consensus camera viewpoint to the vehicle.

20. The one more non-transitory computer-readable media as recited in claim 19 , wherein the operations further comprise:

counting user votes based on the received proposed camera viewpoints, wherein the consensus camera viewpoint instruction is determined based on the user votes; and

determining the weight for each proposed camera viewpoint based on the user of the controller device associated with the each proposed camera viewpoint.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 8, 2016
From: SMOLYANSKIY, NIKOLAI; ZHANG, ZHENGYOU; DENDI, VIKRAM R.; HALL, MICHAEL
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 037924/0903 →
Continuity (1)
Related Publication 20170251180A1 · Aug 31, 2017
Cited By (2)
US 12,197,236 US 12,743,091