Condition detection and reporting in complex systems
Maintaining consistency and freshness of information about an operational system, assuring consistent actions by system actors, assuring that system elements use only a single global status of the system for any particular status time, assuring that each pair of elements acts only upon consistent status values, assuring that system elements operate only with status values measured sufficiently recently to be reliable. Information collectors respond to status values. Information containers respond to information collectors, maintaining status values and metadata indicating whether those status values are reliable, or are consistent with respect to known correct values. Information conditions respond to information collectors or information containers, maintaining logical consistency with a unified global status. System actors respond to information containers or information conditions, acting logically consistent with that global status. System actors might include those which notify monitors or operators, and those which modify the system to detect and correct operation gone awry.
1. A system including
one or more collectors, said collectors each disposed to operate independently to maintain a status value relating to said system, wherein at least one said status value is disposed to change during operation of said system;
one or more containers, said containers each disposed to operate independently in response to a collector, and each maintaining both said status value and metadata indicating whether said status value can be reliably used;
a physical medium including global status information accessible by a computing device, said global status information indicating a single global status of said system;
one or more conditions, said conditions disposed to operate on said computing device sequentially and consistently with said single global status, each responsive to one or more containers or conditions, and each associated with an output; and
one or more actors, said actors disposed to operate on said computing device sequentially and consistently with said single global status, each responsive to one or more conditions, at least one of said actors disposed to operate to alter said single global status;
a RAID storage system, said storage system including one or more disk drives;
wherein at least one particular said actor is disposed to restripe or recompute parity, in response to a presence of a newly available and reliable disk drive;
wherein
said collectors and containers can operate independently to provide data relating to said single global status at speeds independently associated with each said collector or container; and
said conditions and actors can rely on a consistency of said single global status to operate on said computing device at a speed independent of any collector or container, during operation of said system.
2. A system as in claim 1 , including
a RAID storage system, said storage system including one or more disk drives;
wherein at least one particular said condition has an output responsive to a disk drive being unavailable or unreliable.
3. A system as in claim 1 , including
a storage system, said storage system including one or more disk drives;
wherein at least one particular said actor is disposed to distribute or redistribute load for said storage system.
4. A system as in claim 1 , wherein
at least one particular said actor is disposed to alter data retrieved from said collectors and containers, when said data is deemed inaccurate, with alternative data deemed accurate.
5. A system as in claim 1 , wherein
at least one particular said condition includes a system parameter, an output of said particular condition being responsive to a comparison of said system parameter with a result of a calculation relating to at least one of: said status value, said metadata.
6. A system as in claim 1 , wherein
at least one particular said container includes a time value, said metadata being responsive to a comparison of said time value with a current time.
7. A system as in claim 1 , wherein
at least one particular said container includes validity information, said metadata being responsive to a comparison of said validity information with said status value associated with said particular container.
8. A system as in claim 1 , wherein
one or more said actors are disposed to notify an element external to said system.
9. A system as in claim 1 , wherein
one or more said collectors includes a physical sensor.
10. A system as in claim 1 , wherein
said collectors can be pairwise inconsistent;
said containers can be pairwise inconsistent; and
said conditions are pairwise consistent.
11. A system as in claim 1 , wherein
said metadata includes whether said status value is fresh.
12. A system as in claim 1 , wherein
said status values indicate an absence or presence of one or more of:
a disk failure, a file system error, a hardware error, a page fault or page trap, a parity error, a stimulus, an alarm, an interrupt.
13. A system as in claim 1 , wherein
said system includes a cluster; and
at least one particular said actor is disposed to shut down said cluster in response to information indicating a power failure or a limit to UPS runtime.
14. A system as in claim 1 , wherein
said system includes a cluster; and
at least one particular said actor is disposed to synchronize time values at more than one node in said cluster.
15. A system as in claim 1 , wherein
two or more said collectors are disposed to operate in response to distinct stimuli.
16. Apparatus including
a cluster computing system;
a processor in said cluster computing system, said processor coupled to instructions interpretable by a computing device, said instructions directing said processor to implement
one or more collectors, said collectors each disposed to operate independently to maintain a status value relating to said system, wherein at least one said status value is disposed to change during operation of said system,
one or more containers, said containers each disposed to operate independently in response to a collector, and each maintaining both said status value and metadata indicating whether said status value can be reliably used,
a physical medium including global status information accessible by a said processor, said global status information indicating a single global status of said system,
one or more conditions, said conditions disposed to operate on said processor sequentially and consistently with said single global status, each responsive to one or more containers or conditions, and each associated with an output,
one or more actors, said actors disposed to operate on said processor sequentially and consistently with said single global status, each responsive to one or more conditions, at least one of said actors disposed to operate to alter said single global status; and
a RAID storage system, said storage system including one or more disk drives, wherein at least one particular said actor is disposed to restripe or recompute parity, in response to a presence of a newly available and reliable disk drive;
wherein said collectors and containers can operate independently to provide data relating to said single global status at speeds independently associated with each said collector or container;
said conditions and actors can rely on a consistency of said single global status to operate on said computing device at a speed independent of any collector or container, during operation of said system;
said collectors and containers can operate independently in said cluster computing system to provide data relating to said single global status at speeds independently associated with each said collector or container; and
said conditions and actors can rely on a consistency of said single global status at said processor to operate on said computing device at a speed independent of any collector or container, during operation of said system.