Systems and methods for reducing the likelihood of rollover for an e-pallet using camera based on bank angle estimation

1 . A method for reducing the likelihood of rollover for an e-pallet, the method comprising:

obtaining, via one or more cameras of the e-pallet, camera data as to a user of the e-pallet that is external to the e-pallet, wherein the camera data includes an image in which the user is depicted;

determining, via a processor using the camera data, a relative orientation of the user;

determining, via the processor, an e-pallet vertical vector based on the camera data;

determining, via the processor, a user vertical vector based on the relative orientation of the user, wherein determining the user vertical vector includes detecting two or more key points associated with the user within the camera data and extending vectors between pairs of the two or more key points;

determining, via the processor, a bank angle of the e-pallet based on the e-pallet vertical vector and the user vertical vector; and

taking a control action for the e-pallet, in accordance with instructions provided by the processor, based on the bank angle, wherein the control action includes controlling movement of the e-pallet by selectively controlling a speed of the e-pallet and/or a yaw rate of the e-pallet, in accordance with the instructions provided by the processor, to remain below a preprogrammed safety threshold based on the bank angle.

2 . The method of claim 1 , wherein the step of determining the e-pallet vertical vector based on the camera data comprises:

obtaining, via the one or more cameras, the image in which the user is depicted, wherein the image includes a frame in which the user is depicted; and

determining, via the processor, the e-pallet vertical vector based on a vertical edge of the frame of the image.

3 . The method of claim 1 , wherein the step of determining the user vertical vector based on the relative orientation of the user comprises, via the processor:

determining the user vertical vector based on the two or more key points using a machine learning algorithm that is pre-trained with a plurality of training images containing various users at various positions and orientations.

4 . The method of claim 1 , wherein the step of taking the control action comprises:

sounding a notification or alarm in response to a determination that the speed and/or the yaw rate exceeds the safety threshold in view of the bank angle.

5 . The method of claim 1 , wherein determining the bank angle of the e-pallet based on the e-pallet vertical vector and the user vertical vector comprises, via the processor:

determining a first difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a first time to calculate a first bank angle;

determining a second difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a second time to calculate a second bank angle; and

determining an estimated bank angle over time based on the first bank angle and the second bank angle.

6 . The method of claim 1 , further comprising, via the processor:

determining a reliability of the bank angle based on a bank angle reliability threshold associated with a signal-to-noise ratio of the camera data; and

taking the control action for the e-pallet, in accordance with the instructions provided by the processor, based on the bank angle and the reliability of the bank angle.

7 . The method of claim 1 , wherein the image depicts the user standing or walking within view of the e-pallet, the two or more key points are associated with corresponding anatomical parts of the user, and the method includes receiving, via the processor, commands from the user.

8 . A system for reducing the likelihood of rollover for an e-pallet, the system comprising:

one or more cameras configured to obtain camera data as to a user of the e-pallet that is external to the e-pallet, wherein the camera data includes an image in which the user is depicted; and

a processor coupled to the one or more cameras and configured to at least facilitate:

determining, using the camera data, a relative orientation of the user;

determining an e-pallet vertical vector based on the camera data;

determining a user vertical vector based on the relative orientation of the user, wherein determining the user vertical vector includes detecting two or more key points associated with the user within the camera data and extending vectors between pairs of the two or more key points;

determining a bank angle of the e-pallet based on the e-pallet vertical vector and the user vertical vector; and

taking a control action for the e-pallet, in accordance with instructions provided by the processor, based on the bank angle, wherein the control action includes controlling movement of the e-pallet by selectively controlling a speed of the e-pallet and/or a yaw rate of the e-pallet, in accordance with the instructions provided by the processor, to remain below a preprogrammed safety threshold based on the bank angle.

9 . The system of claim 8 , wherein determining the e-pallet vertical vector based on the camera data comprises:

obtaining, via the one or more cameras, the image in which the user is depicted, wherein the image includes a frame in which the user is depicted; and

determining the e-pallet vertical vector based on a vertical edge of the frame of the image.

10 . The system of claim 8 , wherein determining the user vertical vector based on the relative orientation of the user comprises:

determining the user vertical vector based on the two or more key points using a machine learning algorithm that is pre-trained with a plurality of training images containing various users at various positions and orientations.

11 . The system of claim 8 , wherein taking the control action comprises:

sounding a notification or alarm in response to a determination that the speed and/or the yaw rate exceeds the safety threshold in view of the bank angle.

12 . The system of claim 8 , wherein determining the bank angle of the e-pallet based on the e-pallet vertical vector and the user vertical vector comprises:

determining a first difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a first time to calculate a first bank angle;

determining a second difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a second time to calculate a second bank angle; and

determining an estimated bank angle over time based on the first bank angle and the second bank angle.

13 . The system of claim 8 , wherein the processor is configured to facilitate:

determining a reliability of the bank angle based on a bank angle reliability threshold associated with a signal-to-noise ratio of the camera data; and

taking the control action for the e-pallet, in accordance with the instructions provided by the processor, based on the bank angle and the reliability of the bank angle.

14 . The system of claim 8 , wherein the image depicts the user standing or walking within view of the e-pallet, the two or more key points are associated with corresponding anatomical parts of the user, and the processor is configured to facilitate receiving commands from the user.

15 . The system of claim 8 , wherein the image depicts the user standing or walking within view of the e-pallet, the two or more key points are associated with corresponding visually identifiable markings on an article of clothing worn by the user, and the processor is configured to facilitate receiving commands from the user.

16 . An e-pallet device comprising:

a body;

one or more cameras disposed on or within the body and configured to obtain camera data as to a user of the e-pallet device that is external to the body, wherein the camera data includes an image having a frame in which the user is depicted; and

a processor that is coupled to the one or more cameras, the processor configured to at least facilitate:

determining, using the camera data, a relative orientation of the user;

determining an e-pallet vertical vector based on an edge of the frame of the image;

determining a user vertical vector based on the relative orientation of the user, wherein determining the user vertical vector includes detecting two or more key points associated with the user within the camera data and extending vectors between pairs of the two or more key points and determining the user vertical vector based on one or more of the body vectors;

determining a bank angle of the body based on the e-pallet vertical vector and the user vertical vector; and

controlling movement of the body by selectively controlling a speed of the body and/or a yaw rate of the body, in accordance with instructions provided by the processor, to remain below a preprogrammed safety threshold based on the bank angle.

17 . The e-pallet device of claim 16 , wherein determining the user vertical vector based on the relative orientation of the user comprises:

determining the user vertical vector based on the two or more key points using a machine learning algorithm that is pre-trained with a plurality of training images containing various users at various positions and orientations.

18 . The e-pallet device of claim 16 , wherein determining the bank angle of the body based on the e-pallet vertical vector and the user vertical vector comprises:

determining a first difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a first time to calculate a first bank angle;

determining a second difference between the e-pallet vertical vector and the user vertical vector based on the camera data at a second time to calculate a second bank angle; and

determining an estimated bank angle over time based on the first bank angle and the second bank angle.

19 . The e-pallet device of claim 16 , wherein the processor is configured to facilitate:

determining a reliability of the bank angle based on a bank angle reliability threshold associated with a signal-to-noise ratio of the camera data; and

controlling the movement of the body, in accordance with instructions provided by the processor, based on the bank angle and the reliability of the bank angle.

20 . The e-pallet device of claim 16 , wherein the image depicts the user standing or walking within view of the e-pallet, the two or more key points are associated with corresponding anatomical parts of the user or associated with corresponding visually identifiable markings on an article of clothing worn by the user, and the processor is configured to facilitate receiving commands from the user.