Schedule creation method, schedule creating device, substrate processing apparatus, substrate processing system, and storage medium

1 . A substrate processing apparatus comprising:

respective elements relating to a substrate processing sequence; and

a schedule creating device that creates through machine learning a time schedule for the respective elements, wherein

the schedule creating device includes at least one processor and a non-transitory storage medium that stores therein a program;

the processor creates, by executing the program, the time schedule by executing a learning step multiple times;

the learning step is processing of: sequentially placing patterns in a timetable for defining the time schedule for the elements of the substrate processing apparatus, the patterns being prepared in advance and each indicating a procedure in the substrate processing sequence; and acquiring an evaluation value after all the patterns have been placed in the timetable;

the processor executes the program so as to:

acquire one or more placeable patterns that are allowed to be placed in the timetable from among the patterns based on a prescribed constraint condition;

predict and select through the machine leaning a pattern that makes an evaluation value maximum from among the one or more placeable patterns; and

update the timetable by placing the selected pattern in the timetable,

after all the patterns prepared in advance have been placed in the timetable, the processor executes the program so as to:

specify a last time at which the substrate is processed last and acquire a final reward wherein the earlier the last time is, the larger the final reward is; and

calculate the evaluation value based on a reward including the final reward, and

the processor controls the respective elements based on the created time schedule.

2 . The substrate processing apparatus according to claim 1 , wherein

the constraint condition includes a condition for acquiring from among the patterns a succeedingly placeable pattern that is allowed to be placed next according to an order of the substrate processing sequence.

3 . The substrate processing apparatus according to claim 1 , wherein

the constraint condition includes a condition for prohibiting execution of an operation that the substrate processing apparatus is physically disabled to perform.

4 . The substrate processing apparatus according to claim 1 , wherein

the elements of the substrate processing apparatus include a plurality of processing sections that each perform substrate processing,

the final reward includes a specific final reward that is a positive reward, and

the acquiring a final reward includes:

specifying a last processing section that is a processing section of the processing sections that performs the substrate processing last;

acquiring a time at which the last processing section performs the substrate processing last from among times indicated by the timetable; and

acquiring the specific final reward based on the acquired time.

5 . The substrate processing apparatus according to claim 1 , wherein

the elements of the substrate processing apparatus includes a plurality of processing sections that each perform substrate processing,

the final reward includes a specific final reward that is a positive reward, and

the processor executes the program so as to:

acquire first times and second times from times indicated by the timetable, the first times each being a time at which a corresponding one of the processing sections performs the substrate processing first, the second times each being a time at which a corresponding one of the processing sections performs the substrate processing last;

acquire elapsed times from the first times to the second times of the respective processing sections; and

acquire the specific final reward based on a distribution of the elapsed times.

6 . The substrate processing apparatus according to claim 1 , wherein

the reward further includes an intermediate reward, and

the processor further executes, by executing the program:

determining, after the updating the timetable by placing the selected pattern in the timetable, whether or not a pattern of the patterns that is placed latest in the timetable meets a prescribed intermediate reward grant condition; and

acquiring the intermediate reward when it is determined that the pattern placed latest meets the prescribed intermediate reward grant condition.

7 . The substrate processing apparatus according to claim 6 , wherein, in calculating the evaluation value, the processor executes, by executing the program:

acquiring a time corresponding to a position of the pattern that meets the prescribed intermediate reward grant condition from among times indicated by the timetable;

determining whether or not to allocate a discount rate to the intermediate reward based on the acquired time; and

calculating, when it is determined to allocate the discount rate to the intermediate reward, a value of the discount rate according to the acquired time.

8 . The substrate processing apparatus according to claim 6 , wherein

the elements of the substrate processing apparatus include a transport robot capable of transporting a plurality of substrates simultaneously,

the intermediate reward includes a specific intermediate reward that is a positive reward,

the intermediate reward grant condition includes a specific intermediate reward grant condition corresponding to the specific intermediate reward, and

the specific intermediate reward grant condition indicates simultaneous transport of mutually different substrates by the transport robot.

9 . The substrate processing apparatus according to claim 6 , wherein

the elements of the substrate processing apparatus include a processing section that performs substrate processing,

the intermediate reward includes a specific intermediate reward that is a positive reward,

the intermediate reward grant condition includes a specific intermediate reward grant condition corresponding to the specific intermediate reward, and

the specific intermediate reward grant condition indicates bringing-in of a substrate into the processing section directly after bringing-out of another substrate from the processing section.

10 . The substrate processing apparatus according to claim 6 , wherein

the patterns includes a first pattern and a second pattern,

of times indicated by the timetable, a time corresponding to a position at which the first pattern is placed is earlier than a time at which the second pattern is placed,

the intermediate reward includes a specific intermediate reward that is a negative reward,

the intermediate reward grant condition includes a specific intermediate reward grant condition corresponding to the specific intermediate reward, and

the specific intermediate reward grant condition indicates placement of the first pattern in the timetable after placement of the second pattern in the timetable.

11 . A substrate processing system comprising:

a substrate processing apparatus including respective elements relating to a substrate processing sequence; and

a schedule creating device that creates through machine learning a time schedule for the respective elements, wherein

the schedule creating device includes at least one processor and a non-transitory storage medium that stores therein a program,

the processor creates, by executing the program, the time schedule by executing a learning step multiple times,

the learning step is processing of: sequentially placing patterns in a timetable for defining the time schedule for the elements of the substrate processing apparatus, the patterns being prepared in advance and each indicating a procedure in the substrate processing sequence; and acquiring an evaluation value after all the patterns have been placed in the timetable,

the processor executes the program so as to;

acquire one or more placeable patterns that are allowed to be placed in the timetable from among the patterns based on a prescribed constraint condition;

predict and select through the machine leaning a pattern that makes an evaluation value maximum from among the one or more placeable patterns; and

update the timetable by placing the selected pattern in the timetable;

after all the patterns prepared in advance have been placed in the timetable, the processor executes the program so as to:

specify a last time at which the substrate is processed last and acquire a final reward wherein the earlier the last time is, the larger the final reward is; and

calculate the evaluation value based on a reward including the final reward,

the substrate processing apparatus further includes:

a receiving section configured to receive the time schedule from the schedule creating device; and

at least one processor configured to control the respective elements based on the time schedule received by the receiving section.