Filtering of high frequency time series data
View Patent ↗The present invention is a method and apparatus for filtering high frequency time series data using a variety of techniques implemented on a computer. The techniques are directed to detecting and eliminating data errors such as the decimal error, monotonic series of quotes, long series of repeated quotes, scaling changes, and domain errors. Further, by means of comparison with nearby quotes in the time series, the techniques are also able to evaluate the credibility of the quotes.
1. A method of filtering time series financial data comprising the steps of:
testing said data for decimal error;
testing said data for scaling error;
testing said data for domain error;
testing for credibility of said data that passes the tests for decimal error, scaling error and domain error by comparing nearby data in the time series; and
rejecting by a computer an item of data that fails the testing for decimal error, scaling error, domain error and credibility.
2. The method of claim 1 further comprising the step of testing for a monotonic series of quotes in the time series data and rejecting such quotes when detected.
3. The method of claim 1 further comprising the step of testing for a long series of repeated quotes in the time series data and rejecting such quotes when detected.
4. The method of claim 1 wherein the step of testing said data for decimal error comprises the step of testing if an absolute value of a difference between a new quote and a previous quote in the time series data is within a predetermined value of the next power of ten.
5. The method of claim 4 wherein the step of testing said data for decimal error further comprises the step of testing if a time interval between the new quote and the previous quote is less than a predetermined time.
6. The method of claim 5 wherein the predetermined time is 70 minutes.
7. The method of claim 1 wherein the time series data is a series of quotes and the step of testing for decimal error comprises the steps of:
testing for a decimal error in a quote,
computing a corrected quote if a decimal error is detected, and
testing the corrected quote for validity.
8. The method of claim 1 wherein the time series data is a series of quotes and the step of testing for decimal error comprises the steps of:
testing for a decimal error in a quote,
computing a corrected quote if a decimal error is detected,
testing the corrected quote for credibility, and
comparing the credibility of the corrected quote with the credibility of the quote in which the decimal error was detected.
9. The method of claim 1 wherein the step of testing said data for domain error comprises the step of testing for an illegal value of the time series data.
10. The method of filtering time series data of claim 1 wherein the time series data is a series of quotes and the quotes are tested for credibility relative to the quotes within a time window.
11. A method of filtering a time series of quotes comprising the steps of:
testing said quotes for decimal error,
testing for credibility of said quotes by comparing nearby quotes in the time series,
rejecting by a computer a quote that fails the tests for decimal error and credibility,
testing if a ratio of a new quote and a previous quote lies within a predetermined range; and
if the ratio does not lie within the predetermined range, changing the ratio by a power of ten until the changed ratio lies within the predetermined range.
12. The method of claim 11 further comprising the step of testing said quotes for at least one of scaling error and domain error.
13. The method of claim 11 further comprising the step of testing for a monotonic series of quotes in the time series of quotes and rejecting such quotes when detected.
14. The method of claim 11 further comprising the step of testing for a long series of repeated quotes in the time series of quotes and rejecting such quotes when detected.
15. The method of claim 11 wherein the quotes are tested for credibility relative to the quotes within a time window.
16. The method of claim 11 wherein the step of testing said quotes for decimal error comprises the step of testing if an absolute value of a difference between a new quote and a previous quote in the time series of quotes is within a predetermined value of the next power of ten.
17. The method of claim 11 wherein the step of testing said quotes for decimal error further comprises the step of testing if a time interval between the new quote and the previous quote is less than a predetermined time.
18. The method of claim 11 wherein the step of testing for decimal error comprises the steps of:
testing for a decimal error in a quote,
computing a corrected quote if a decimal error is detected, and
testing the corrected quote for validity.
19. The method of claim 11 wherein the step of testing for decimal error comprises the steps of:
testing for a decimal error in a quote,
computing a corrected quote if a decimal error is detected,
testing the corrected quote for credibility, and
comparing the credibility of the corrected quote with the credibility of the original quote.
20. The method of claim 1 wherein the time series data is a series of quotes and the step of testing said data for scaling error comprises the steps of:
testing if a ratio of a new quote and a previous quote lies within a predetermined range; and
if the ratio does not lie within the predetermined range, changing the ratio by a power of ten until the changed ratio lies within the predetermined range.
21. The method of claim 20 wherein the range is between √0.1 and √10.
22. The method of claim 11 further comprising the step of testing for an illegal value of quotes in the time series of quotes.
23. The method of claim 11 , wherein the range is between √0.1 and √10.
24. The method of claim 1 wherein rejecting is made by identifying an item of data as bad.
25. The method of claim 1 wherein rejecting is made by eliminating from the time series data an item of data that is bad.
26. The method of claim 11 wherein rejecting is made by identifying a quote as bad.
27. The method of claim 11 wherein rejecting is made by eliminating from the time series of quotes a quote that is bad.
28. A method of filtering time series financial data comprising:
assigning a numerical value of a credibility measure to each datum in a time series of quotes or transaction prices of financial instruments, and
filtering out with a computer those data in the time series of quotes or transaction prices for which the assigned numerical value does not meet a predetermined value.
29. The method of claim 28 further comprising the preliminary step of detecting and correcting decimal errors in the time series of financial data.
30. The method of claim 28 further comprising the preliminary step of detecting and correcting scaling errors in the time series of financial data.
31. The method of claim 28 further comprising the preliminary step of eliminating domain errors in the time series of financial data.
32. The method of claim 28 wherein each quote or transaction price accumulates an additive Trust Capital based on various criteria resulting in a numerical quantity ranging from −Infinity to Infinity.
33. The method of claim 32 wherein the numerical value of the Credibility measure is computed from the Trust Capital as a monotonic mapping from the domain [−Infinity, Infinity] to the range [0,1].
34. The method of claim 28 wherein each quote or transaction price is assigned a credibility measure having a numerical value ranging from 0 to 1.
35. The method of claim 28 wherein the temporal distance of quotes is computed in a modified time scale in which epochs known a priori to be of high activity are contracted with respect to physical time and epochs known a priori to be of low activity are expanded with respect to physical time and wherein a normalization is applied such that the total hours per day or per week for the modified time scale matches physical time.
36. The method of claim 28 wherein the numerical value of the credibility measure, C(T), of a datum x is determined by
C
(
T
)
=
1
2
+
T
2
1
+
T
2
(
4.1
)
where
T
=
1
-
x
^
i
2
k
(
4.6
)
(
4.7
)
and
x
^
i
=
x
-
x
_
EMA
[
t
;
(
x
-
x
_
)
2
]
(
4.3
)
where x is the exponential moving average (EMA) of the data in the series of quotes or transaction preces in the time period t.