Data caching method, system and device in AI cluster, and computer medium
View Patent ↗The present application discloses a method for caching data in AI cluster, system, apparatus and computer medium. The method comprises: determining target data set to be cached; acquiring weight value of the target data set on each of a plurality of cluster nodes of the AI cluster; determining a target cluster node for caching the target data set; acquiring a target shortest path from remaining cluster nodes of the AI cluster to the target cluster node, and a predecessor node of the target cluster node in the target shortest path, wherein the remaining cluster nodes comprise nodes of the AI cluster except the target cluster node; determining, based on the weight value, the target shortest path and the predecessor node, a cache path for caching the target data set to the target cluster node, to cache the target data set to the target cluster node according to the cache path.
1 . A method for caching data in an AI cluster, comprising:
determining a target data set to be cached;
acquiring a weight value of the target data set on each of a plurality of cluster nodes of the AI cluster;
determining a target cluster node for caching the target data set;
acquiring a target shortest path from remaining cluster nodes of the AI cluster to the target cluster node, and a predecessor node of the target cluster node in the target shortest path, wherein the remaining cluster nodes comprise nodes of the AI cluster except the target cluster node;
determining, based on the weight value, the target shortest path and the predecessor node, a cache path for caching the target data set to the target cluster node, to cache the target data set to the target cluster node according to the cache path-;
wherein the step of, acquiring the target shortest path from the remaining cluster nodes of the AI cluster to the target cluster node, and the predecessor node of the target cluster node in the target shortest path comprises:
determining a first node set, wherein the first node set is configured for storing a first type of cluster node with a known target shortest path between the target cluster node;
determining a second node set, wherein the second node set is configured for storing a second type of cluster node of the AI cluster except the first node set;
determining a first shortest path between each of the second type cluster nodes and the target cluster node;
taking the second type cluster node corresponding to the first shortest path with the smallest value as a cluster node to be determined;
for each of the second type cluster nodes, determining a shortest path between the second type cluster node and the cluster node to be determined, and determining a sum of the first shortest path and the second shortest path corresponding to the node to be determined; under the condition that the first shortest path corresponding to the second type cluster node is less than the sum, updating the target shortest path of the second type cluster node as the first shortest path corresponding to the second type cluster node; under the condition that the first shortest path corresponding to the second type cluster node is greater than the sum, updating the target shortest path of the second type cluster node as the sum, and regarding the predecessor node of the target cluster node in the shortest path corresponding to the second type cluster node is the cluster node to be determined;
updating the cluster node to be determined as the first type cluster node;
judging whether the first node set contains all cluster nodes, under the condition of no, returning to the step of taking the second type cluster node corresponding to the first shortest path with the smallest value as the cluster node to be determined, and under the condition of yes, ending.
2 . The method according to claim 1 , wherein the step of, acquiring the weight value of the target data set on each of the plurality of cluster nodes of the AI cluster comprises:
analyzing the type of the cluster node for each of the plurality of the cluster nodes of the AI cluster;
under the condition that the cluster node is a management node, determining the total number of the cluster nodes of the AI cluster, and determining the total number of data sets on a shared storage node of the AI cluster;
determining a product value of the total number of the cluster nodes and the total number of the data sets as the weight value of the management node.
3 . The method according to claim 2 , wherein after analyzing the type of the cluster node, the method further comprises:
under the condition that the cluster node is a non-management node, judging whether the target data set is stored on the cluster node; and
under the condition that the target data set is not stored on the cluster node, determining that the weight value of the cluster node is infinity.
4 . The method according to claim 3 , wherein after judging whether the target data set is stored on the cluster node, the method further comprises:
under the condition that the target data set is stored on the cluster node, determining the number of first type tasks that the target data set is pulled by the cluster node, determining the number of second type tasks that the target data set is pulled out of the cluster node, and determining a sum of the number of the first type tasks, the number of the second type tasks and 1 as the weight value of the cluster node.
5 . The method according to claim 2 , wherein in the step of, under the condition that the cluster node is the management node, determining the total number of the cluster nodes of the AI cluster, and determining the total number of data sets on the shared storage node of the AI cluster,
wherein the management node refers to a node of the AI cluster having a management function.
6 . The method according to claim 2 , wherein in the step of, under the condition that the cluster node is the management node, determining the total number of the cluster nodes of the AI cluster, and determining the total number of data sets on the shared storage node of the AI cluster,
wherein the shared storage node refers to a node that data is capable to be shared by all cluster nodes of the AI cluster.
7 . The method according to claim 1 , wherein the step of, determining the target cluster node for caching the target data set comprises:
the corresponding target cluster node is capable to be determined according to a cache instruction sent by a user.
8 . The method according to claim 1 , wherein in the step of, acquiring the target shortest path from the remaining cluster nodes of the AI cluster to the target cluster node, and the predecessor node of the target cluster node in the target shortest path,
wherein the predecessor node refers to a cluster node on the shortest path from the other cluster nodes to the target cluster node, and before the target cluster node.
9 . The method according to claim 1 , wherein the step of, determining, based on the weight value, the target shortest path and the predecessor node, the cache path for caching the target data set to the target cluster node, to cache the target data set to the target cluster node according to the cache path comprises:
regarding other cluster nodes with the smallest weight value as a transmission node for transmitting the target data set, and transmitting the target data set to the target cluster node according to the target shortest path of the transmission node and the predecessor node.
10 . An apparatus for data caching in an AI cluster, comprising:
a memory, configured for storing a computer program;
a processor, configured for, when executing the computer program, implementing the steps of the method for caching data in the AI cluster according to claim 1 .
11 . The apparatus for data caching in an AI cluster according to claim 10 , wherein the step of, acquiring the weight value of the target data set on each of the plurality of cluster nodes of the AI cluster comprises:
analyzing the type of the cluster node for each of the plurality of the cluster nodes of the AI cluster;
under the condition that the cluster node is a management node, determining the total number of the cluster nodes of the AI cluster, and determining the total number of data sets on a shared storage node of the AI cluster;
determining a product value of the total number of the cluster nodes and the total number of the data sets as the weight value of the management node.
12 . The apparatus for data caching in an AI cluster according to claim 11 , wherein after analyzing the type of the cluster node, the method further comprises:
under the condition that the cluster node is a non-management node, judging whether the target data set is stored on the cluster node; and
under the condition that the target data set is not stored on the cluster node, determining that the weight value of the cluster node is infinity.
13 . The apparatus for data caching in an AI cluster according to claim 12 , wherein after judging whether the target data set is stored on the cluster node, the method further comprises:
under the condition that the target data set is stored on the cluster node, determining the number of first type tasks that the target data set is pulled by the cluster node, determining the number of second type tasks that the target data set is pulled out of the cluster node, and determining a sum of the number of the first type tasks, the number of the second type tasks and 1 as the weight value of the cluster node.
14 . The apparatus for data caching in an AI cluster according to claim 11 , wherein in the step of, under the condition that the cluster node is the management node, determining the total number of the cluster nodes of the AI cluster, and determining the total number of data sets on the shared storage node of the AI cluster,
wherein the management node refers to a node of the AI cluster having a management function.
15 . The apparatus for data caching in an AI cluster according to claim 10 , wherein the step of, determining the target cluster node for caching the target data set comprises:
the corresponding target cluster node is capable to be determined according to a cache instruction sent by a user.
16 . The apparatus for data caching in an AI cluster according to claim 10 , wherein in the step of, acquiring the target shortest path from the remaining cluster nodes of the AI cluster to the target cluster node, and the predecessor node of the target cluster node in the target shortest path,
wherein the predecessor node refers to a cluster node on the shortest path from the other cluster nodes to the target cluster node, and before the target cluster node.
17 . The apparatus for data caching in an AI cluster according to claim 10 , wherein the step of, determining, based on the weight value, the target shortest path and the predecessor node, the cache path for caching the target data set to the target cluster node, to cache the target data set to the target cluster node according to the cache path comprises:
regarding other cluster nodes with the smallest weight value as a transmission node for transmitting the target data set, and transmitting the target data set to the target cluster node according to the target shortest path of the transmission node and the predecessor node.
18 . A non-transitory computer-readable storage medium, wherein a computer program is stored in the non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for data caching in an AI cluster according to claim 1 are implemented.