Granted Patent B2

US 12,737,590 · App. 18/533,830 · Granted Sep 15, 2026

Inventors: Julian Röttger Büchel (Zürich, CH); Irem Boybat Kara (Adliswil, CH); Abbas Rahimi (Rüschlikon, CH); Athanasios Vasilopoulos (Kilchberg, CH); Manuel Le Gallo-Bourdeau (Horgen, CH); Abu Sebastian (Adliswil, CH)

Assignee: International Business Machines Corporation

G06N3/045

Abstract

Claims (52)

1 . A computer-implemented method for routing acceleration in mixture of experts (MoE) ensembles, the method comprising:

receiving input data at a router, wherein the router comprises a plurality of matrix-vector-multiplier (MVM) tiles;

generating a plurality of output vectors by applying a routing function to the input data, wherein each output vector in the plurality of output vectors is associated with one or more respective tiles or pairs of tiles in the plurality of MVM tiles;

determining a plurality of sub-vectors in the plurality of output vectors, wherein each sub-vector in the plurality of sub-vectors is associated with a respective output vector in the plurality of output vectors, and merging the plurality of sub-vectors into an element vector;

generating a probability distribution vector by applying a Softmax function to the element vector and determining one or more largest elements of the probability distribution; and

configuring the router based on the one or more largest elements of the probability distribution,

wherein configuring the router further comprises:

initializing the router using pre-trained weights;

retraining the router using the one or more largest elements of the probability distribution for one or more epochs to produce updated weights; and

scaling the router using the updated weights.

2 . The method of claim 1 , wherein configuring the router further comprises:

determining a mean optimal scale of the router based on a mean of the one or more largest elements of the probability distribution and one or more approximated largest element values; and

scaling the router using the mean optimal scale.

3 . The method of claim 1 , wherein the plurality of MVM tiles may comprise one or more MVM tiles in a single-layer MVM.

4 . The method of claim 1 , wherein the plurality of MVM tiles may comprise one or more pairs of MVM tiles in a multi-layer MVM.

5 . The method of claim 1 , wherein the routing function is executed using analogue in-memory computing.

6 . The method of claim 1 , wherein applying the Softmax function, determining the plurality of sub-vectors in the plurality of output vectors, merging the plurality of sub-vectors into the element vector, and determining the one or more largest elements of the probability distribution are performed using a digital circuit.

7 . A computer system for routing acceleration in mixture of experts (MoE) ensembles, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:

receiving input data at a router, wherein the router comprises a plurality of matrix-vector-multiplier (MVM) tiles;

generating a plurality of output vectors by applying a routing function to the input data, wherein each output vector in the plurality of output vectors is associated with one or more respective tiles or pairs of tiles in the plurality of MVM tiles;

determining a plurality of sub-vectors in the plurality of output vectors, wherein each sub-vector in the plurality of sub-vectors is associated with a respective output vector in the plurality of output vectors, and merging the plurality of sub-vectors into an element vector,

generating a probability distribution vector by applying a Softmax function to the element vector and determining one or more largest elements of the probability distribution; and

configuring the router based on the one or more largest elements of the probability distribution,

wherein configuring the router further comprises:

initializing the router using pre-trained weights;

retraining the router using the one or more largest elements of the probability distribution for one or more epochs to produce updated weights; and

scaling the router using the updated weights.

8 . The computer system of claim 7 , wherein configuring the router further comprises:

determining a mean optimal scale of the router based on a mean of the one or more largest elements of the probability distribution and one or more approximated largest element values; and

scaling the router using the mean optimal scale.

9 . The computer system of claim 7 , wherein the plurality of MVM tiles may comprise one or more MVM tiles in a single-layer MVM.

10 . The computer system of claim 7 , wherein the plurality of MVM tiles may comprise one or more pairs of MVM tiles in a multi-layer MVM.

11 . The computer system of claim 7 , wherein the routing function is executed using analogue in-memory computing.

12 . The computer system of claim 7 , wherein applying the Softmax function, determining the plurality of sub-vectors in the plurality of output vectors, merging the plurality of sub-vectors into the element vector, and determining the one or more largest elements of the probability distribution are performed using a digital circuit.

13 . A computer program product for routing acceleration in mixture of experts ensembles (MoE), the computer program product comprising:

one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:

receiving input data at a router, wherein the router comprises a plurality of matrix-vector-multiplier (MVM) tiles;

generating a plurality of output vectors by applying a routing function to the input data, wherein each output vector in the plurality of output vectors is associated with one or more respective tiles or pairs of tiles in the plurality of MVM tiles;

determining a plurality of sub-vectors in the plurality of output vectors, wherein each sub-vector in the plurality of sub-vectors is associated with a respective output vector in the plurality of output vectors, and merging the plurality of sub-vectors into an element vector;

generating a probability distribution vector by applying a Softmax function to the element vector and determining one or more largest elements of the probability distribution; and

configuring the router based on the one or more largest elements of the probability distribution,

wherein configuring the router further comprises:

initializing the router using pre-trained weights;

retraining the router using the one or more largest elements of the probability distribution for one or more epochs to produce updated weights; and

scaling the router using the updated weights.

14 . The computer program product of claim 13 , wherein configuring the router further comprises:

determining a mean optimal scale of the router based on a mean of the one or more largest elements of the probability distribution and one or more approximated largest element values; and

scaling the router using the mean optimal scale.

15 . The computer program product of claim 13 , wherein the plurality of MVM tiles may comprise one or more MVM tiles in a single-layer MVM.

16 . The computer program product of claim 13 , wherein the plurality of MVM tiles may comprise one or more pairs of MVM tiles in a multi-layer MVM.

17 . The computer program product of claim 13 , wherein the routing function is executed using analogue in-memory computing and wherein applying the Softmax function, determining the plurality of sub-vectors in the plurality of output vectors, merging the plurality of sub-vectors into the element vector, and determining the one or more largest elements of the probability distribution are performed using a digital circuit.

Assignments (1)

Continuity (1)

References Cited (19)