Methodology of building a multi task model into a single model
The system includes a processor and a memory. The processor is configured to receive data from a camera, determine a plurality of heat maps using a multitask model, and provide the plurality of heat maps. The memory is coupled to the processor and configured to provide the processor with instructions.
1. A system, comprising:
a video event recorder, comprising:
a camera; and
a processor configured to:
receive data from the camera using an input image interface;
determine, using the data and a multitask model, a plurality of heat maps, wherein the multitask model is trained using preprocessed data, wherein the preprocessed data is related to one task and one heat map output, wherein determining the plurality of heat maps comprises determining a horizon heat map comprising horizon pixels; and
provide the plurality of heat maps; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the data from the camera comprises video data.
3. The system of claim 1 , wherein the data from the camera comprises image data.
4. The system of claim 1 , wherein the horizon heat map is post processed by one or more of thresholding, determining a number of highest value points, fitting a line through a number of points, determining an angle of a horizon line as a horizon annotation, and/or determining an offset location of a horizon line as a horizon annotation.
5. The system of claim 1 , wherein preprocessed data includes horizontal flipping of data, cropping of data, changing contrast of data, changing brightness of data, changing saturation of data, changing hue of data, resizing of image data, cutting out image data, shifting image data vertically, and/or shifting image data horizontally.
6. The system of claim 1 , wherein the data from the camera is encoded by down sampling into an encoded tensor, and wherein the encoded tensor is decoded by up sampling into a decoded tensor.
7. The system of claim 1 , further comprising an other processor for performing post processing of the plurality of heat maps.
8. The system of claim 1 , further comprising a vehicle camera, wherein the processor is further configured to receive other data from the vehicle camera via a vehicle data bus interface, and wherein determining the plurality of heat maps uses the other data.
9. The system of claim 1 , wherein the processor is further configured to determine driver assistance alerts using the plurality of heat maps.
10. The system of claim 1 , wherein the multitask model determines an object heat map.
11. The system of claim 10 , wherein the object heat map comprises one or more of: a vehicle object heat map, a pedestrian object heat map, a traffic light object heat map, a cyclist object heat map, a stop sign object heat map, an x object scale, a y object scale, an x object offset, and/or a y object offset.
12. The system of claim 10 , wherein the object heat map is post processed to generate an object location and an object box.
13. The system of claim 1 , wherein the multitask model determines a lane heat map.
14. The system of claim 13 , wherein the lane heat map comprises one or more of: a lane location heat map, one or two vertical affinity fields for lanes, and/or a horizontal affinity filed for lanes.
15. The system of claim 14 , wherein the lane heat map is post processed by one or more of thresholding the lane heat map to determine lane areas, determining a vector field as a lane annotation, determining a change in vector field direction, determining a vertical or a near vertical vector associated with a location in the lane heat map to cluster the lanes areas, determining a change in an image row of a vector field direction to identify a lane edge point, and/or fitting a cubic spline to sets of lane edge points as a lane annotation.
16. A method, comprising:
receiving, by a processor of a video event recorder, data from a camera of the video event recorder;
determining, using the data and a multitask model executing on the processor, a plurality of heat maps, wherein the multitask model is trained using preprocessed data, wherein the preprocessed data is related to one task and one heat map output, wherein determining the plurality of heat maps comprises determining a horizon heat map comprising horizon pixels; and
providing the plurality of heat maps.
17. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving, by a processor of a video event recorder, data from a camera of the video event recorder;
determining, using the data and a multitask model executing on the processor, a plurality of heat maps, wherein the multitask model is trained using preprocessed data, wherein the preprocessed data is related to one task and one heat map output, wherein determining the plurality of heat maps comprises determining a horizon heat map comprising horizon pixels; and
providing the plurality of heat maps.
18. A system, comprising:
a video event recorder, comprising:
a vehicle camera;
a camera; and
a processor configured to:
receive data from the camera using an input image interface;
receive other data from the vehicle camera via a vehicle data bus interface;
determine, using the data, the other data, and a multitask model, a plurality of heat maps, wherein the multitask model is trained using preprocessed data related to one task and one heat map output; and
provide the plurality of heat maps; and
a memory coupled to the processor and configured to provide the processor with instructions.
19. The system of claim 18 , wherein the processor is further configured to determine driver assistance alerts using the plurality of heat maps.
20. A method, comprising:
receiving, by a processor of a video event recorder, data from a camera of the video event recorder;
receiving other data from a vehicle camera via a vehicle data bus interface;
determining, using the data, the other data, and a multitask model executing on a processor, a plurality of heat maps, wherein the multitask model is trained using preprocessed data related to one task and one heat map output; and
providing the plurality of heat maps.
21. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving, by a processor of a video event recorder, data from a camera of the video event recorder;
receiving other data from a vehicle camera via a vehicle data bus interface;
determining, using the data, the other data, and a multitask model executing on a processor, a plurality of heat maps, wherein the multitask model is trained using preprocessed data related to one task and one heat map output; and
providing the plurality of heat maps.