MLIR

Multi-Level IR Compiler Framework

'amdgpu' Dialect

The AMDGPU dialect provides wrappers around AMD-specific functionality and LLVM intrinsics. These wrappers should be used in conjunction with more generic dialects, such as gpu and vector, when generating LLVM IR that will eventually be executed on AMD hardware.

Operation definition 

amdgpu.lds_barrier (::mlir::amdgpu::LDSBarrierOp) 

Barrier that includes a wait for LDS memory operations.

Syntax:

operation ::= `amdgpu.lds_barrier` attr-dict

amdgpu.lds_barrier is both a barrier (all workitems in a workgroup must reach the barrier before any of them may proceed past it) and a wait for all operations that affect the Local Data Store (LDS) issued from that wrokgroup to complete before the workgroup may continue. Since the LDS is per-workgroup memory, this barrier may be used, for example, to ensure all workitems have written data to LDS before any workitem attempts to read from it.

Note that lds_barrier does not force reads to or from global memory to complete before execution continues. Therefore, it should be used when operations on global memory can be issued far in advance of when their results are used (for example, by writing them to LDS).

amdgpu.mfma (::mlir::amdgpu::MFMAOp) 

MLIR wrapper for CDNA mfma instructions

Syntax:

operation ::= `amdgpu.mfma` $sourceA `*` $sourceB `+` $destC
              attr-dict
              `blgp` `=` $blgp
              `:` type($sourceA) `,` type($destC)

The amdgpu.mfma op is an MLIR wrapper around intrinsics for various mfma instructions in the CDNA architecture, which perform multiple outer products in order to allow fast matrix multiplication.

The wrapper will select an appropriate mfma instruction, if one is available, based on the provided m, k, n, and nBlks attributes, along with the types of the source and destination arguments.

For information on the layouts of the input and output matrces (which are stored in sourceA, sourceB, destC, and destD), see the CDNA ISA documentation.

The cbsz, abid, and blgp parameters control how the lanes of the wave are permuted when matrix data is being loaded: blgp can be any number of fixed permutations, cbsz specifies the log_2 of the number of chunks the lanes holding sourceA are split into, and abid selects one of those chunks.

Note, this wrapper allows specifying vector<4Kxi8> arguments to MFMA intrinsics that take an integer type of width 4K. For example, one can provide a vector<4xi8> as an argument to an MFMA instruction that logically takes 4 i8s but whose intrinsics are specified to take an i32. In these cases, the bytes in the vector will be concatenated in little-endian order (that is, v[0] will go to arg[7:0], v[1] to arg[15:8] and so on).

The negateA, negateB, and negateC flags are only supported for double-precision operations on gfx940+.

Interfaces: NoSideEffect (MemoryEffectOpInterface)

Effects: MemoryEffects::Effect{}

Attributes: 

AttributeMLIR TypeDescription
m::mlir::IntegerAttr32-bit signless integer attribute
n::mlir::IntegerAttr32-bit signless integer attribute
k::mlir::IntegerAttr32-bit signless integer attribute
blocks::mlir::IntegerAttr32-bit signless integer attribute
cbsz::mlir::IntegerAttr32-bit signless integer attribute
abid::mlir::IntegerAttr32-bit signless integer attribute
blgp::mlir::amdgpu::MFMAPermBAttrThe possible permutations of the lanes storing B available in an MFMA
reducePrecision::mlir::UnitAttrunit attribute
negateA::mlir::UnitAttrunit attribute
negateB::mlir::UnitAttrunit attribute
negateC::mlir::UnitAttrunit attribute

Operands: 

OperandDescription
sourceA32-bit float or 64-bit float or 32-bit signless integer or 64-bit signless integer or vector of 32-bit float values of length 2 or vector of 16-bit float values of length 4 or vector of bfloat16 type values of length 2/4 or vector of 8-bit signless integer values of length 4/8
sourceB32-bit float or 64-bit float or 32-bit signless integer or 64-bit signless integer or vector of 32-bit float values of length 2 or vector of 16-bit float values of length 4 or vector of bfloat16 type values of length 2/4 or vector of 8-bit signless integer values of length 4/8
destC64-bit float or vector of 32-bit float values of length 4/16/32 or vector of 32-bit signless integer values of length 4/16/32 or vector of 64-bit float values of length 4

Results: 

ResultDescription
destD64-bit float or vector of 32-bit float values of length 4/16/32 or vector of 32-bit signless integer values of length 4/16/32 or vector of 64-bit float values of length 4

amdgpu.raw_buffer_atomic_fadd (::mlir::amdgpu::RawBufferAtomicFaddOp) 

Raw Buffer Floating-point Atomic Add (MI-* only)

Syntax:

operation ::= `amdgpu.raw_buffer_atomic_fadd` attr-dict $value `->` $memref `[` $indices `]`
              (`sgprOffset` $sgprOffset^)? `:`
              type($value) `->` type($memref) `,` type($indices)

The amdgpu.raw_buffer_atomic_fadd op is a wrapper around the buffer-based atomic floating point addition available on the MI-* series of AMD GPUs.

The index into the buffer is computed as for memref.store with the addition of indexOffset (which is used to aid in emitting vectorized code) and, if present sgprOffset (which is added after bounds checks and includes any non-zero offset on the memref type).

All indexing components are given in terms of the memref’s element size, not the byte lengths required by the intrinsic.

Out of bounds atomic operations are ignored in hardware.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

Traits: AttrSizedOperandSegments

Attributes: 

AttributeMLIR TypeDescription
boundsCheck::mlir::BoolAttrbool attribute
indexOffset::mlir::IntegerAttr32-bit signless integer attribute

Operands: 

OperandDescription
value32-bit float
memrefmemref of any type values
indices32-bit signless integer
sgprOffset32-bit signless integer

amdgpu.raw_buffer_load (::mlir::amdgpu::RawBufferLoadOp) 

Raw Buffer load, exposing GCN features

Syntax:

operation ::= `amdgpu.raw_buffer_load` attr-dict $memref `[` $indices `]`
              (`sgprOffset` $sgprOffset^)? `:`
              type($memref) `,` type($indices) `->` type($value)

The amdgpu.raw_buffer_load op is a wrapper around the buffer load intrinsics available on AMD GPUs, including extensions in newer GPUs.

The index into the buffer is computed as for memref.load with the additon of indexOffset and sgprOffset (which may or may not be considered in bounds checks and includes any offset present on the memref type if it’s non-zero).

All indices and offsets are in units of the memref’s data type and are converted to bytes during lowering.

When a load is out of bounds, the instruction returns zero. Partially-out of bounds have chipset-dependent behavior: whether reading 2 elements starting at index 7 of a memref<8xf32> returns the last element in the first vector component depends on the architecture.

The memref struct is converted into a buffer resource (a V#) and the arguments are translated to intrinsic arguments as follows:

  • The base address of the buffer is the base address of the memref
  • The stride is 0 to enable raw mode
  • The number of records is the size of the memref, in bytes In the case of dynamically-shaped memrefs, this is computed at runtime as max_d (size(d) * stride(d)) * sizeof(elementType(memref))
  • The offset enable bit is 1, the index enable bit is 0.
  • The thread ID addition bit is off
  • If boundsCheck is false and the target chipset is RDNA, OOB_SELECT is set to 2 to disable bounds checks, otherwise it is 3
  • The cache coherency bits are off

Traits: AttrSizedOperandSegments

Attributes: 

AttributeMLIR TypeDescription
boundsCheck::mlir::BoolAttrbool attribute
indexOffset::mlir::IntegerAttr32-bit signless integer attribute

Operands: 

OperandDescription
memrefmemref of any type values
indices32-bit signless integer
sgprOffset32-bit signless integer

Results: 

ResultDescription
valuebfloat16 type or 16-bit float or 32-bit float or 32-bit signless integer or 8-bit signless integer or vector of 32-bit float or 32-bit signless integer values of length 2/4 or vector of 16-bit float or bfloat16 type values of length 2/4/8 or vector of 8-bit signless integer values of length 2/4/8/16

amdgpu.raw_buffer_store (::mlir::amdgpu::RawBufferStoreOp) 

Raw Buffer Store, exposing GCN features

Syntax:

operation ::= `amdgpu.raw_buffer_store` attr-dict $value `->` $memref `[` $indices `]`
              (`sgprOffset` $sgprOffset^)? `:`
              type($value) `->` type($memref) `,` type($indices)

The amdgpu.raw_buffer_store op is a wrapper around the buffer store intrinsics available on AMD GPUs, including extensions in newer GPUs.

The store index is computed as in memref.store with the addition of indexOffset (which is included for uniformity with atomics and may be useful when writing vectorized code) and sgprOffset (which is added after bounds checks and implicitly includes the offset of the memref type if non-zero). All index components are in terms of the elements of the memref, not bytes, and are scaled up appropriately.

Out of bounds stores are ignored in hardware. Wthether a vector write that includes some in-bounds and soeme out-of-bounds components is partically completed is chipset-dependent.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

Traits: AttrSizedOperandSegments

Attributes: 

AttributeMLIR TypeDescription
boundsCheck::mlir::BoolAttrbool attribute
indexOffset::mlir::IntegerAttr32-bit signless integer attribute

Operands: 

OperandDescription
valuebfloat16 type or 16-bit float or 32-bit float or 32-bit signless integer or 8-bit signless integer or vector of 32-bit float or 32-bit signless integer values of length 2/4 or vector of 16-bit float or bfloat16 type values of length 2/4/8 or vector of 8-bit signless integer values of length 2/4/8/16
memrefmemref of any type values
indices32-bit signless integer
sgprOffset32-bit signless integer

Attribute definition 

MFMAPermBAttr 

The possible permutations of the lanes storing B available in an MFMA

Syntax:

!amdgpu.mfma_perm_b<
  ::mlir::amdgpu::MFMAPermB   # value
>

Parameters: 

ParameterC++ typeDescription
value::mlir::amdgpu::MFMAPermBan enum of type MFMAPermB