Systems and methods for efficient content management and distribution for video game development
Systems and methods for distributing an image file and a plurality of data chunks in a video game development environment include at least one game studio that has a plurality of client computing devices and at least one cache node in data communication with each other over a first network local to the at least one game studio. The plurality of client computing devices are in data communication, through the at least one cache node, with remotely located metadata and object storage servers over a wide area network. The image file includes a manifest of one or more game files and of the plurality of data chunks. The image file is referenced by a hash of data indicative of the image file while each of the plurality of data chunks is referenced by a hash of data indicative of each of the plurality of data chunks.
1 . A computer-implemented method of performing data cleanup in a game development environment comprising at least one game studio having a plurality of client computing devices and at least one cache node in data communication with each other over a first network that is local to the at least one game studio, wherein the at least one cache node is in data communication with remotely located first and second servers over a second network external to the game studio, wherein the first and second servers, the at least one cache node and each of the plurality of client computing devices are configured to store one or more image files, a plurality of data chunks associated with each of the one or more image files and one or more labels referencing each of the one or more image files, and wherein the first server is configured to execute a first module, the second server is configured to execute a second module, the at least one cache node is configured to execute a third module and each of the plurality of client computing devices is configured to execute a fourth module, the method comprising:
instructing the first, second, third and fourth modules to cause a respective processor in the first server, the second server, the at least cache node and each of the plurality of client computing devices to:
determine at least one first label that has expired;
delete the at least one first label;
determine at least one first image file that is not referenced by any of the remaining one or more labels;
delete the at least one first image file;
determine one or more first data chunks that are not associated with any of the remaining one or more image files; and
delete the one or more first data chunks.
2 . The computer-implemented method of claim 1 , wherein each of the one or more labels has an attribute defining a period of time after which said each of the one or more labels expires, and wherein the attribute is automatically marked as expired at the end of the period of time.
3 . The computer-implemented method of claim 1 , wherein each of the one or more labels is an alphanumeric data structure comprising a form of:
namespace/name: tag.
4 . The computer-implemented method of claim 1 , wherein more than one of the one or more labels reference an image file of the one or more image files.
5 . The computer-implemented method of claim 1 , wherein each of the one or more image files includes a manifest of one or more game files and a plurality of data chunks of each of the one or more game files, wherein each of the one or more game files is referenced by a hash of data indicative of said each of the one or more game files, and wherein each of the plurality of data chunks is referenced by a hash of data indicative of said each of the plurality of data chunks.
6 . The computer-implemented method of claim 1 , wherein the first server is configured to store the one or more image files and the second server is configured to store the plurality of data chunks.
7 . The computer-implemented method of claim 1 , wherein the instruction is executed automatically on a predefined periodic basis by the first, second, third and fourth modules.
8 . The computer-implemented method of claim 1 , wherein determining the one or more first data chunks that are not associated with any of the remaining one or more image files comprises executing a command through a command-line interface generated by a client-side content management module.
9 . The computer-implemented method of claim 1 , wherein deleting the one or more first data chunks comprises reference counting the one or more first data chunks prior to deletion.
10 . The computer-implemented method of claim 1 , further comprising querying a database to identify the at least one first image file that is not referenced by any of the remaining one or more labels.
11 . The computer-implemented method of claim 1 , wherein deleting the one or more first data chunks comprises inserting identifiers of the one or more first data chunks into a delete queue and processing the queue using multiple threads.
12 . The computer-implemented method of claim 1 , wherein deleting the one or more first data chunks comprises issuing an HTTP DELETE command to an object storage server.
13 . The computer-implemented method of claim 1 , wherein determining the one or more first data chunks comprises comparing hashes of the one or more first data chunks against hashes stored in a database.
14 . The computer-implemented method of claim 1 , wherein deleting the at least one first image file comprises automatically deleting an image file that is no longer referenced by any labels.
15 . The computer-implemented method of claim 1 , wherein each of the plurality of client computing devices is configured to store the one or more image files in an images folder and the plurality of data chunks in an objects folder separate from the images folder.
16 . The computer-implemented method of claim 1 , wherein each client of the plurality of client computing devices is further configured to store mounts corresponding to each of the one or more image files in a mounts folder, wherein each of the mounts comprises at least one link to a corresponding one of the plurality of data chunks.
17 . The computer-implemented method of claim 1 , wherein determining the at least one first image file comprises executing a command configured to operate on a metadata server on a predefined periodic basis.
18 . The computer-implemented method of claim 1 , wherein determining the at least one first label that has expired comprises identifying labels having a time-to-live attribute marked as expired.
19 . The computer-implemented method of claim 1 , wherein determining the at least one first label that has expired comprises automatically marking a time-to-live attribute as expired when a defined period elapses.
20 . The computer-implemented method of claim 1 , wherein determining the at least one first label comprises processing a list of labels stored on at least one of the plurality of the client computing devices that references image hashes.