Evaluation device, evaluation system, and evaluation method

1. An evaluation device for evaluating security of an electronic control system in which a plurality of electronic control units are connected to a bus used for communication, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation device comprising:

a recording medium that holds attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

a transmitter that transmits the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

a monitor that monitors a physical amount changed by operation of the actuator; and

an evaluator that makes the evaluation on basis of the physical amount changed by operation of the actuator obtained by the monitor when the transmitter transmits the plurality of frames to the bus, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, wherein

the evaluator further counts a number times the defense has failed and a number of times the defense has succeeded,

when the number of times the defense has failed is above a predetermined threshold, or the percentage of times the defense has failed exceeds a predetermined percentage, the evaluator determines that there is an absence of attack resistance, and,

when the number of times the defense has failed is below the predetermined threshold, or the percentage of times the defense has failed falls below the predetermined percentage, the evaluator determines that there is a presence of attack resistance.

2. The evaluation device according to claim 1 , wherein

the plurality of electronic control units perform communication over the bus in accordance with a controller area network (CAN) protocol.

3. The evaluation device according to claim 1 , wherein

the monitor detects, as the monitoring of the actuator, transmission of a control instruction frame to the bus, the control instruction frame instructing one of the plurality of electronic control units to control the actuator; and

the evaluator makes the evaluation so that an evaluation result differs depending on whether or not transmission of the control instruction frame over the bus has been detected by the monitor within a certain period after transmission of one or more of the plurality of frames from the transmitter to the bus.

4. The evaluation device according to claim 1 , wherein

the monitor detects, as the monitoring of the actuator, input of a control signal to the actuator from one of the plurality of electronic control units; and

the evaluator makes the evaluation so that an evaluation result differs depending on whether or not input of the control signal to the actuator has been detected by the monitor within a certain period after transmission of one or more of the plurality of frames from the transmitter to the bus.

5. The evaluation device according to claim 1 , wherein

the monitor detects, as the monitoring of the actuator, operation of the actuator; and

the evaluator makes the evaluation so that an evaluation result differs depending on whether or not operation of the actuator has been detected by the monitor within a certain period after transmission of one or more of the plurality of frames from the transmitter to the bus.

6. The evaluation device according to claim 5 , wherein

the actuator has a computer that executes a program simulating operation of the actuator; and

the monitor detects operation of the actuator by observing a change of predetermined data concerning the program in the computer.

7. The evaluation device according to claim 1 , wherein

the attack procedure information further indicates a transmission interval between the plurality of frames; and

the transmitter transmits the plurality of frames to the bus in accordance with the transmission order and the transmission interval indicated by the attack procedure information.

8. The evaluation device according to claim 1 , wherein

the evaluator outputs, as an evaluation result, information indicating whether or not the electronic control system has attack resistance.

9. The evaluation device according to claim 1 , wherein

the transmitter repeats, plural times, an attack pattern of transmitting the plurality of frames to the bus in the transmission order indicated by the attack procedure information; and

the evaluator makes the evaluation so that an evaluation result differs depending on whether or not the monitoring result has been changed by the repetition of the attack pattern.

10. An evaluation system for evaluating security of an electronic control system including a plurality of electronic control units that perform communication over a bus, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation system comprising:

a recording medium that holds attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

a transmitter that transmits the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

a monitor that monitors a physical amount changed by operation of the actuator; and

an evaluator that makes the evaluation on basis of the physical amount changed by operation of the actuator obtained by the monitor when the transmitter transmits the plurality of frames to the bus, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, and wherein

the evaluator further counts a number times the defense has failed and a number of times the defense has succeeded,

when the number of times the defense has failed is above a predetermined threshold, or the percentage of times the defense has failed exceeds a predetermined percentage, the evaluator determines that there is an absence of attack resistance, and,

when the number of times the defense has failed is below the predetermined threshold, or the percentage of times the defense has failed falls below the predetermined percentage, the evaluator determines that there is a presence of attack resistance.

11. An evaluation method for evaluating security of an electronic control system including a plurality of electronic control units that perform communication over a bus, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation method comprising:

holding attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

transmitting with a transmitter the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

monitoring with a monitor a physical amount changed by operation of the actuator; and

making an evaluation with an evaluator on basis of the physical amount changed by operation of the actuator obtained by the monitoring, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, and wherein

the evaluator further counts a number times the defense has failed and a number of times the defense has succeeded,

when the number of times the defense has failed is above a predetermined threshold, or the percentage of times the defense has failed exceeds a predetermined percentage, the evaluator determines that there is an absence of attack resistance, and,

when the number of times the defense has failed is below the predetermined threshold, or the percentage of times the defense has failed falls below the predetermined percentage, the evaluator determines that there is a presence of attack resistance.

12. The evaluation method according to claim 11 , wherein

the plurality of electronic control units perform communication over the bus in accordance with controller area network (CAN) protocol; and

the evaluation method further comprises making the evaluation so that an evaluation result differs depending on whether or not transmission of a control instruction frame to the bus has been detected within a certain period after transmission of one or more of the plurality of frames to the bus, the control instruction frame instructing one of the plurality of electronic control units to control the actuator.

13. An evaluation device for evaluating security of an electronic control system in which a plurality of electronic control units are connected to a bus used for communication, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation device comprising:

a recording medium that holds attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

a transmitter that transmits the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

a monitor that monitors a physical amount changed by operation of the actuator; and

an evaluator that makes the evaluation on basis of the physical amount changed by operation of the actuator obtained by the monitor when the transmitter transmits the plurality of frames to the bus, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, wherein

the defense includes a plurality of defense functions,

the monitor further monitors whether or not each of a plurality of defense functions works,

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work falls below a predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work exceeds the predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded.

14. An evaluation system for evaluating security of an electronic control system including a plurality of electronic control units that perform communication over a bus, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation system comprising:

a recording medium that holds attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

a transmitter that transmits the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

a monitor that monitors a physical amount changed by operation of the actuator; and

an evaluator that makes the evaluation on basis of the physical amount changed by operation of the actuator obtained by the monitor when the transmitter transmits the plurality of frames to the bus, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, and wherein

the defense includes a plurality of defense functions,

the monitor further monitors whether or not each of a plurality of defense functions works,

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work falls below a predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work exceeds the predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded.

15. An evaluation method for evaluating security of an electronic control system including a plurality of electronic control units that perform communication over a bus, the electronic control system including an actuator controlled by any of the plurality of electronic control units and having a defense against an attack of at least one unauthorized frame against the actuator, the evaluation method comprising:

holding attack procedure information indicative of contents of a plurality of frames which includes the at least one unauthorized frame to attack the actuator and of a transmission order of the plurality of frames;

transmitting with a transmitter the plurality of frames to the bus in the transmission order indicated by the attack procedure information;

monitoring with a monitor a physical amount changed by operation of the actuator; and

making an evaluation with an evaluator on basis of the physical amount changed by operation of the actuator obtained by the monitoring, wherein

when the transmitter transmits the at least one unauthorized frame and the actuator operates according to the least one unauthorized frame to change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the actuator does not operate according to the at least one unauthorized frame to not change the physical amount monitored by the monitor, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded, and wherein

the defense includes a plurality of defense functions,

the monitor further monitors whether or not each of a plurality of defense functions works,

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work falls below a predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has failed, and

when the transmitter transmits the at least one unauthorized frame and the percentage of times the defense functions work exceeds the predetermined percentage, the evaluator determines that the defense of the electronic control system against the attack of the at least one unauthorized frame on the actuator has succeeded.