Real-time data pipeline techniques for improving a fast weather forecasting system
The system as described collects and utilizes weather data sensor information in order to rapidly collect and update weather forecasts using real-time weather data collected at high rates of frequency, and use this collected high frequency weather data to rapidly correct and update the weather forecasts generated by the system.
1. A computerized method of calculating location-specific weather data, the computerized method comprising:
receiving, by a communications interface in electronic communication with a first data source, radio frequency link data for at least one radio frequency link, the radio frequency link data including a signal attenuation measurement and at least one link location identifier that identifies radio frequency link endpoints;
pre-processing, with at least one processor, the radio frequency link data, thereby producing pre-processed radio frequency link attenuation data having an attenuation measurement corresponding to the signal attenuation measurement and a link geolocation corresponding to the at least one link location identifier;
providing, with the at least one processor, the pre-processed radio frequency link attenuation data to a first data store, the first data store storing the pre-processed radio frequency link attenuation data; and
processing, on a schedule with the at least one processor, the pre-processed radio frequency link attenuation data using a predefined data transform, the predefined data transform calculating first location-specific weather data based on the pre-processed radio frequency link attenuation data and the link geolocation,
wherein pre-processing of the radio frequency link data and processing of the pre-processed radio frequency link attenuation data include associating the pre-processed radio frequency link attenuation data with a cadence instance Mi of a cadence cycle comprising a series of repeating cadence instances configured to asynchronously collect radio frequency link data and pre-process the collected radio frequency link data into weather parameter data, the cadence cycle having a time delay between cadence instances.
2. The method of claim 1 wherein the at least one radio frequency link comprises a terrestrial microwave link.
3. The method of claim 1 wherein the at least one radio frequency link comprises a satellite link.
4. The method of claim 1 further comprising:
receiving from a second data source, second location-specific weather data;
storing the second location-specific weather data; and
processing the first location-specific weather data and the second location-specific weather data to produce blended weather data.
5. The method of claim 4 wherein the blended weather data includes at least a precipitation intensity for a plurality of tiles, each tile corresponding to a geographic region.
6. The method of claim 4 further comprising:
receiving the blended weather data; and
processing the blended weather data to produce processed blended weather data usable by an on-demand information product.
7. The method of claim 6 wherein the processed blended weather data includes map data usable to display a map on a viewing device.
8. The method of claim 6 wherein the on-demand information product comprises a real-time precipitation map.
9. The method of claim 4 further comprising producing forecast data based on the blended weather data.
10. The method of claim 9 further comprising:
receiving the forecast weather data;
producing, based on the forecast weather data, second forecast weather data; and
storing the second forecast weather data.
11. The method of claim 4 further comprising calculating forecast weather data based upon the blended weather data.
12. The method of claim 1 wherein the radio frequency link data comprises terrestrial wireless network data including wireless network topology information and wireless link attenuation information.
13. The method of claim 1 wherein processing of the pre-processed radio frequency link data is successively performed for multiple cadence instances of the cadence cycle.
14. The method of claim 1 wherein the time delay is less than five minutes.
15. The method of claim 1 wherein the time delay is less than fifteen minutes.
16. The method of claim 1 further including storing the pre-processed radio frequency link attenuation data successively for multiple cadence instances of the cadence cycle.
17. The method of claim 1 wherein each cadence instance includes one or more tile layers, at least one tile layer representing pre-processed radio frequency link data.
18. The method of claim 1 further comprising:
calculating forecast weather data based upon the first location-specific weather data; and
storing the forecast weather data.
19. A computerized system for calculating location-specific weather data, the computerized system comprising:
at least one processor configured to receive radio frequency link data including a signal attenuation measurement and at least one link location identifier that identifies link endpoints, the at least one processor pre-processing the radio frequency link data to create pre-processed radio frequency link attenuation data having a link attenuation measurement corresponding to a geolocation corresponding to the at least one link location identifier;
the at least one processor being further configured to process the pre-processed radio frequency link attenuation data using a predefined data transform, the predefined data transform calculating first location-specific weather data based on the pre-processed radio frequency link attenuation data and the at least one link location identifier,
wherein pre-processing of the radio frequency link data and processing of the pre-processed radio frequency link attenuation data include associating the pre-processed radio frequency link attenuation data with a cadence instance M i of a cadence cycle comprising a series of repeating cadence instances configured to asynchronously collect radio frequency link data and pre-process the collected radio frequency link data into weather parameter data, the cadence cycle having a time delay between cadence instances.
20. The system of claim 19 wherein the at least one processor is further configured to receive second location-specific weather data and process the first and second location-specific weather data to produce blended weather data.
21. The computerized system of claim 19 wherein the at least one radio frequency link comprises a terrestrial microwave link.
22. The computerized system of claim 19 wherein the at least one radio frequency link comprises a satellite link.
23. The computerized system of claim 19 wherein each cadence instance includes one or more tile layers, at least one tile layer representing pre-processed radio frequency link data.