OpenAI Opco, LLC

A hardware accelerator is provided for dot-product based lookup algorithms including hardware accelerators for transformer key-value. The accelerator includes memory stacks that include memory banks (e.g., DRAM) fabricated on respective memory dies. The memory banks can be connected by through silicon vias (TSVs). The accelerator also includes arithmetic logic units configured on a processor die, and that include dedicated circuitry for the dot-product based lookup algorithm. The processor die can be arranged below the memory stacks to minimize the wire lengths from the ALUs to the memory stacks to reduce energy consumption due to moving bits between the ALUs and the memory stacks. The length of the wires from the ALUs to the memory stacks can be less than 1 mm (e.g., as short as 0.05 mm). The accelerators can also include processing in memory ALUs that include dedicated circuitry for the dot-product based lookup algorithm

A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.

Systems, apparatuses, and methods are disclosed for improved matrix-vector operations in accelerators that may be useful or heavy AI training and inference workloads. The disclosed technology provides arrangements that permit more efficient computation by, for example, performing calculations without repeated conversion between numeric domains. In some implementations, a compute-in-memory (CIM) macro is configured to perform a vector matrix multiplication (VMM) operation between a vector of activation values and a matrix of weight values, and a mode decoding unit is configured to provide a mode of the VMM operation according to a first floating point format of the activation values and a second floating point format of the weight values. A column cell of the CIM macro can be configured to output a product between an activation value and a weight value using a half adder and multiple multiplexers that provide selections to a full adder based on control signals.

Systems, apparatuses, and methods are disclosed for improved matrix-vector operations in accelerators that may be useful or heavy AI training and inference workloads. The disclosed technology provides arrangements that permit more efficient computation by, for example, performing calculations without repeated conversion between numeric domains. In some implementations, a compute-in-memory (CIM) macro is configured to perform a vector matrix multiplication (VMM) operation between a vector of activation values and a matrix of weight values in floating point formats. The CIM macro has a functional block configured to align mantissa bits of primitive products between the activation values and the weight values by shifting the mantissa bits and an adder tree configured to output an accumulation value of the primitive products in an integer format by adding the shifted mantissa bits.

A system including a memory and hardware compute logic is described. The memory includes memory cells storing weights corresponding to a matrix. Hardware compute logic is coupled with the memory cells. The hardware compute logic is configured to perform a vector-matrix multiplication (VMM) for the matrix and for a matrix transpose for the weights being stationary for the memory cells.