IP Library › Granted Patent US 12711393

Granted Patent B2

US 12711393 · App. 18/028,765 · Granted Aug 18, 2026

Collaborative training with compressed transmissions

Inventors: Di Wu (St Andrews, GB); Blesson Varghese (St Andrews, GB); Philip Rodgers (London, GB); Rehmat Ullah (St Andrews, GB); Peter Kilpatrick (Belfast, GB); Ivor Spence (Belfast, GB)

Assignee: RAKUTEN MOBILE, INC.

G06N3/098 G06N3/0455

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12711393
App. No.: 18/028,765
Granted: Aug 18, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Collaborative training with compressed transmissions is performed by partitioning a plurality of layers of a neural network model into a device partition and a server partition, combining a plurality of encoding layers of an auto-encoder neural network with the device partition, wherein a largest encoding layer among the plurality of encoding layers is adjacent a layer of the device partition bordering the server partition, combining a plurality of decoding layers of the auto-encoder neural network with the server partition, wherein a largest decoding layer among the plurality of decoding layers is adjacent a layer of the server partition bordering the device partition, transmitting, to a computation device, the device partition combined with the plurality of encoding layers, and training, collaboratively with the computation device through a network, the neural network model.

Claims (75)

1 . A non-transitory computer-readable medium including instructions executable by a server to cause the server to perform operations comprising:

partitioning a plurality of layers of a neural network model into a device partition to be trained by a computation device and a server partition to be trained by the server;

combining a plurality of encoding layers of an auto-encoder neural network with the device partition, wherein a largest encoding layer among the plurality of encoding layers is adjacent a layer of the device partition bordering the server partition;

combining a plurality of decoding layers of the auto-encoder neural network with the server partition, wherein a largest decoding layer among the plurality of decoding layers is adjacent a layer of the server partition bordering the device partition;

transmitting, to the computation device, the device partition combined with the plurality of encoding layers; and

training, collaboratively with the computation device through a network, the neural network model by

receiving, from the computation device after the computation device applies the device partition to a set of data samples to obtain a set of activations and applies the plurality of encoding layers to the set of activations to obtain a set of compressed activations, the set of compressed activations output from the plurality of encoding layers,

applying the plurality of decoding layers to the set of compressed activations to obtain a set of activations,

applying the server partition to the set of activations to obtain a set of output instances,

applying a loss function relating activations to output instances to each output instance among the current set of output instances to obtain a set of loss values,

computing a set of gradient vectors for each layer of the server partition, including a set of gradient vectors of a layer bordering the device partition, based on the set of loss values,

updating weight values of the server partition based on the set of gradient vectors for each layer of the server partition, and

transmitting, to the computation device, the set of gradient vectors of the layer bordering the device partition,

wherein the computation device computes a set of gradient vectors for each layer of the device partition, based on the set of gradient vectors of the layer of the server partition bordering the device partition, and updates weight values of the device partition based on the set of gradient vectors for each layer of the device partition.

2 . The computer-readable medium of claim 1 , wherein the operations further comprise:

training, collaboratively with the computation device through the network, the auto-encoder neural network before training the neural network model.

3 . The computer-readable medium of claim 1 , wherein the operations further comprise:

combining the plurality of decoding layers of the auto-encoder neural network with the device partition, wherein a smallest decoding layer among the plurality of decoding layers is adjacent a smallest layer among the plurality of encoding layers;

transmitting, to a computation device, the device partition combined with the plurality of encoding layers and the plurality of decoding layers; and

receiving, from the computation device, the plurality of decoding layers before combining the plurality of decoding layers with the server partition.

4 . The computer-readable medium of claim 1 , wherein the training the neural network model further includes:

dequantizing the set of compressed activations by increasing the bit-width of each compressed activation among the set of compressed activations.

5 . The computer-readable medium of claim 1 , wherein the operations further comprise:

performing a plurality of iterations of the training;

receiving the device partition from the computation device; and

combining the device partition with the server partition to obtain an updated neural network model.

6 . The computer-readable medium of claim 1 , wherein the receiving the set of compressed activations includes receiving a set of labels from the computation device.

7 . A non-transitory computer-readable medium including instructions executable by a computation device to cause the computation device to perform operations comprising:

receiving, from a server, a device partition of a neural network model combined with a plurality of encoding layers of an auto-encoder neural network, the neural network model including a plurality of layers partitioned into the device partition to be trained by the computation device and a server partition to be trained by the server, wherein a largest encoding layer among the plurality of encoding layers is adjacent a layer of the device partition bordering the server partition; and

training, collaboratively with the server through a network, the neural network model by applying the device partition to a set of data samples to obtain a set of activations, applying the plurality of encoding layers to the set of activations to obtain a set of compressed activations,

transmitting, to the server, the set of compressed activations,

receiving, from the server after the server applies the plurality of decoding layers to the set of compressed activations to obtain a set of activations, applies the server partition to the set of activations to obtain a set of output instances, applies a loss function relating activations to output instances to each output instance among the current set of output instances to obtain a set of loss values, and computes, based on the set of loss values, a set of gradient vectors for each layer of the server partition, including a set of gradient vectors of a layer bordering the device partition, the set of gradient vectors of a layer of the server partition bordering the device partition,

computing a set of gradient vectors for each layer of the device partition, based on the set of gradient vectors of the layer of the server partition bordering the device partition, and

updating weight values of the device partition based on the set of gradient vectors for each layer of the device partition,

wherein the server updates weight values of the server partition based on the set of gradient vectors for each layer of the server partition.

8 . The computer-readable medium of claim 7 , wherein the operations further comprise:

training, collaboratively with the server through the network, the auto-encoder neural network before training the neural network model.

9 . The computer-readable medium of claim 7 , wherein the operations further comprise:

receiving, from the server, the device partition combined with the plurality of encoding layers and a plurality of decoding layers of the auto-encoder neural network;

training the auto-encoder neural network before training the neural network model;

removing the plurality of decoding layers from the device partition and the plurality of encoding layers; and

transmitting, to the server, the plurality of decoding layers.

10 . The computer-readable medium of claim 7 , wherein the training the neural network model further includes:

quantizing the set of activations by decreasing the bit-width of each activation among the set of activations.

11 . The computer-readable medium of claim 7 , wherein the operations further comprise:

performing a plurality of iterations of the training;

removing the plurality of encoding layers from the device partition; and

transmitting the device partition to the server.

12 . The computer-readable medium of claim 7 , wherein the transmitting the set of compressed activations includes transmitting a set of labels to the server.

13 . A method comprising:

partitioning a plurality of layers of a neural network model into a device partition to be trained by a computation device and a server partition to be trained by a server;