MLIR  18.0.0git
Namespaces | Classes | Typedefs | Enumerations | Functions
mlir::sparse_tensor Namespace Reference

Namespaces

 detail
 
 ir_detail
 

Classes

class  StorageLayout
 Provides methods to access fields of a sparse tensor with the given encoding. More...
 
class  SparseTensorType
 A wrapper around RankedTensorType, which has three goals: More...
 
struct  SparsifierOptions
 Options for the "sparsifier" pipeline. More...
 
struct  TensorExp
 Tensor expression. Represents an MLIR expression in tensor index notation. More...
 
struct  LatPoint
 Lattice point. More...
 
class  Merger
 A class to handle all iteration lattice operations. More...
 
struct  Element
 An element of a sparse tensor in coordinate-scheme representation (i.e., a pair of coordinates and value). More...
 
struct  ElementLT
 Closure object for operator< on Element with a given rank. More...
 
class  SparseTensorCOO
 A memory-resident sparse tensor in coordinate-scheme representation (a collection of Elements). More...
 
class  SparseTensorReader
 This class abstracts over the information stored in file headers, as well as providing the buffers and methods for parsing those headers. More...
 
class  MapRef
 A class for capturing the sparse tensor type map with a compact encoding. More...
 
class  SparseTensorStorageBase
 Abstract base class for SparseTensorStorage<P,C,V>. More...
 
class  SparseTensorStorage
 A memory-resident sparse tensor using a storage scheme based on per-level sparse/dense annotations. More...
 
class  CodegenEnv
 The code generation environment class aggregates a number of data structures that are needed during the code generation phase of sparsification. More...
 
class  FuncCallOrInlineGenerator
 A helper class to simplify lowering operations with/without function calls. More...
 
class  IterationGraphSorter
 
class  LoopEmitter
 
class  SparseTensorSpecifier
 
class  SparseTensorDescriptorImpl
 A helper class around an array of values that corresponds to a sparse tensor. More...
 
class  SparseTensorDescriptor
 Uses ValueRange for immutable descriptors. More...
 
class  MutSparseTensorDescriptor
 Using SmallVector for mutable descriptor allows users to reuse it as a tmp buffers to append value for some special cases, though users should be responsible to restore the buffer to legal states after their use. More...
 
class  SparsificationAndBufferizationPass
 A pass that lowers tensor ops to memref ops, regardless of whether they are dense or sparse. More...
 

Typedefs

using index_type = uint64_t
 This type is used in the public API at all places where MLIR expects values with the built-in type "index". More...
 
using complex64 = std::complex< double >
 
using complex32 = std::complex< float >
 
using Dimension = uint64_t
 The type of dimension identifiers and dimension-ranks. More...
 
using Level = uint64_t
 The type of level identifiers and level-ranks. More...
 
using Size = int64_t
 The type for individual components of a compile-time shape, including the value ShapedType::kDynamic (for shapes). More...
 
using FieldIndex = unsigned
 The type of field indices. More...
 
using TensorId = unsigned
 Tensor identifiers, chosen to be the BlockArgument::getArgNumber of the value passed to Merger::buildTensorExp. More...
 
using LoopId = unsigned
 Loop identifiers. More...
 
using TensorLoopId = unsigned
 A compressed representation of std::pair<TensorId, LoopId>. More...
 
using ExprId = unsigned
 TensorExp identifiers. More...
 
using LatPointId = unsigned
 LatPoint identifiers. More...
 
using LatSetId = unsigned
 LatSet identifiers. More...
 
using LvlLTPair = std::pair< Level, LevelType >
 A pair of level and its corresponding LevelType of a tensor. More...
 
using LoopCoeffPair = std::pair< LoopId, unsigned >
 A pair of loop id and its coefficients. More...
 
using LoopOrd = unsigned
 The position of a loop in the loop-stack, or the position of a LoopId in a topologically-sorted list of LoopIds. More...
 
using TensorLevel = unsigned
 

Enumerations

enum class  OverheadType : uint32_t {
  kIndex = 0 , kU64 = 1 , kU32 = 2 , kU16 = 3 ,
  kU8 = 4
}
 Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor. More...
 
enum class  PrimaryType : uint32_t {
  kF64 = 1 , kF32 = 2 , kF16 = 3 , kBF16 = 4 ,
  kI64 = 5 , kI32 = 6 , kI16 = 7 , kI8 = 8 ,
  kC64 = 9 , kC32 = 10
}
 Encoding of the elemental type, for "overloading" @newSparseTensor. More...
 
enum class  Action : uint32_t {
  kEmpty = 0 , kEmptyForward = 1 , kFromCOO = 2 , kFromReader = 4 ,
  kToCOO = 5 , kPack = 7 , kSortCOOInPlace = 8
}
 The actions performed by @newSparseTensor. More...
 
enum class  LevelType : uint8_t {
  Undef = 0 , Dense = 4 , Compressed = 8 , CompressedNu = 9 ,
  CompressedNo = 10 , CompressedNuNo = 11 , Singleton = 16 , SingletonNu = 17 ,
  SingletonNo = 18 , SingletonNuNo = 19 , LooseCompressed = 32 , LooseCompressedNu = 33 ,
  LooseCompressedNo = 34 , LooseCompressedNuNo = 35 , TwoOutOfFour = 64
}
 This enum defines all the sparse representations supportable by the SparseTensor dialect. More...
 
enum class  LevelFormat : uint8_t {
  Dense = 4 , Compressed = 8 , Singleton = 16 , LooseCompressed = 32 ,
  TwoOutOfFour = 64
}
 This enum defines all supported storage format without the level properties. More...
 
enum class  LevelPropertyNondefault : uint8_t { Nonunique = 1 , Nonordered = 2 }
 This enum defines all the nondefault properties for storage formats. More...
 
enum class  SparseTensorFieldKind : uint32_t { StorageSpec = 0 , PosMemRef = static_cast<uint32_t>(StorageSpecifierKind::PosMemSize) , CrdMemRef = static_cast<uint32_t>(StorageSpecifierKind::CrdMemSize) , ValMemRef = static_cast<uint32_t>(StorageSpecifierKind::ValMemSize) }
 ===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields. More...
 
enum class  EmitCInterface : bool { Off = false , On = true }
 Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall(). More...
 
enum class  SortMask : unsigned {
  kIncludeDenseOutput = 0x1 , kIncludeDenseInput = 0x2 , kIncludeAll = 0x7 , kIncludeDense = 0x3 ,
  kSparseOnly = 0x0
}
 Iteration graph sorting. More...
 
enum class  ExpArity { kNullary , kUnary , kBinary }
 

Functions

constexpr bool isFloatingPrimaryType (PrimaryType valTy)
 
constexpr bool isIntegralPrimaryType (PrimaryType valTy)
 
constexpr bool isRealPrimaryType (PrimaryType valTy)
 
constexpr bool isComplexPrimaryType (PrimaryType valTy)
 
constexpr const char * toMLIRString (LevelType lt)
 Returns string representation of the given dimension level type. More...
 
constexpr bool isValidLT (LevelType lt)
 Check that the LevelType contains a valid (possibly undefined) value. More...
 
constexpr bool isUndefLT (LevelType lt)
 Check if the LevelType is the special undefined value. More...
 
constexpr bool isDenseLT (LevelType lt)
 Check if the LevelType is dense (regardless of properties). More...
 
constexpr bool isCompressedLT (LevelType lt)
 Check if the LevelType is compressed (regardless of properties). More...
 
constexpr bool isSingletonLT (LevelType lt)
 Check if the LevelType is singleton (regardless of properties). More...
 
constexpr bool isLooseCompressedLT (LevelType lt)
 Check if the LevelType is loose compressed (regardless of properties). More...
 
constexpr bool is2OutOf4LT (LevelType lt)
 Check if the LevelType is 2OutOf4 (regardless of properties). More...
 
constexpr bool isWithPosLT (LevelType lt)
 Check if the LevelType needs positions array. More...
 
constexpr bool isWithCrdLT (LevelType lt)
 Check if the LevelType needs coordinates array. More...
 
constexpr bool isOrderedLT (LevelType lt)
 Check if the LevelType is ordered (regardless of storage format). More...
 
constexpr bool isUniqueLT (LevelType lt)
 Check if the LevelType is unique (regardless of storage format). More...
 
constexpr std::optional< LevelFormatgetLevelFormat (LevelType lt)
 Convert a LevelType to its corresponding LevelFormat. More...
 
constexpr std::optional< LevelTypebuildLevelType (LevelFormat lf, bool ordered, bool unique)
 Convert a LevelFormat to its corresponding LevelType with the given properties. More...
 
constexpr uint64_t encodeDim (uint64_t i, uint64_t cf, uint64_t cm)
 Bit manipulations for affine encoding. More...
 
constexpr uint64_t encodeLvl (uint64_t i, uint64_t c, uint64_t ii)
 
constexpr bool isEncodedFloor (uint64_t v)
 
constexpr bool isEncodedMod (uint64_t v)
 
constexpr bool isEncodedMul (uint64_t v)
 
constexpr uint64_t decodeIndex (uint64_t v)
 
constexpr uint64_t decodeConst (uint64_t v)
 
constexpr uint64_t decodeMulc (uint64_t v)
 
constexpr uint64_t decodeMuli (uint64_t v)
 
template<typename T >
RankedTensorType getRankedTensorType (T &&t)
 Convenience method to abbreviate casting getType(). More...
 
template<typename T >
MemRefType getMemRefType (T &&t)
 Convenience method to abbreviate casting getType(). More...
 
SparseTensorEncodingAttr getSparseTensorEncoding (Type type)
 Convenience method to get a sparse encoding attribute from a type. More...
 
bool hasAnySparseOperand (Operation *op)
 Returns true iff MLIR operand has any sparse operand. More...
 
bool hasAnySparseResult (Operation *op)
 Returns true iff MLIR operand has any sparse result. More...
 
bool hasAnySparseOperandOrResult (Operation *op)
 Returns true iff MLIR operand has any sparse operand or result. More...
 
bool hasAnyNonIdentityOperandsOrResults (Operation *op)
 Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps. More...
 
AffineMap inferLvlToDim (AffineMap dimToLvl, MLIRContext *context)
 Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails. More...
 
AffineMap inverseBlockSparsity (AffineMap dimToLvl, MLIRContext *context)
 Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases. More...
 
SmallVector< unsigned > getBlockSize (AffineMap dimToLvl)
 Given the dimToLvl map, returns the block sizes in a vector. More...
 
bool isBlockSparsity (AffineMap dimToLvl)
 Given the dimToLvl map, returns if it's block sparsity. More...
 
Dimension toDim (SparseTensorEncodingAttr enc, Level l)
 Convenience method to translate the given level to the corresponding dimension. More...
 
Level toLvl (SparseTensorEncodingAttr enc, Dimension d)
 Convenience method to translate the given dimension to the corresponding level. More...
 
StorageSpecifierKind toSpecifierKind (SparseTensorFieldKind kind)
 
SparseTensorFieldKind toFieldKind (StorageSpecifierKind kind)
 
unsigned getNumFieldsFromEncoding (SparseTensorEncodingAttr enc)
 
unsigned getNumDataFieldsFromEncoding (SparseTensorEncodingAttr enc)
 
void foreachFieldInSparseTensor (SparseTensorEncodingAttr enc, llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)> callback)
 
void foreachFieldAndTypeInSparseTensor (SparseTensorType, llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>)
 
SparseTensorType getSparseTensorType (Value val)
 Convenience methods to obtain a SparseTensorType from a Value. More...
 
std::optional< SparseTensorTypetryGetSparseTensorType (Value val)
 
void buildSparsifier (OpPassManager &pm, const SparsifierOptions &options)
 Adds the "sparsifier" pipeline to the OpPassManager. More...
 
void registerSparseTensorPipelines ()
 Registers all pipelines for the sparse_tensor dialect. More...
 
void registerTransformDialectExtension (DialectRegistry &registry)
 
void registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)
 
OverheadType overheadTypeEncoding (unsigned width)
 Converts an overhead storage bitwidth to its internal type-encoding. More...
 
OverheadType overheadTypeEncoding (Type tp)
 Converts an overhead storage type to its internal type-encoding. More...
 
Type getOverheadType (Builder &builder, OverheadType ot)
 Converts the internal type-encoding for overhead storage to an mlir::Type. More...
 
OverheadType posTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for position overhead storage. More...
 
OverheadType crdTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for coordinate overhead storage. More...
 
StringRef overheadTypeFunctionSuffix (OverheadType ot)
 Convert OverheadType to its function-name suffix. More...
 
StringRef overheadTypeFunctionSuffix (Type overheadTp)
 Converts an overhead storage type to its function-name suffix. More...
 
PrimaryType primaryTypeEncoding (Type elemTp)
 Converts a primary storage type to its internal type-encoding. More...
 
StringRef primaryTypeFunctionSuffix (PrimaryType pt)
 Convert PrimaryType to its function-name suffix. More...
 
StringRef primaryTypeFunctionSuffix (Type elemTp)
 Converts a primary storage type to its function-name suffix. More...
 
Value genCast (OpBuilder &builder, Location loc, Value value, Type dstTy)
 Add type casting between arith and index types when needed. More...
 
Value genScalarToTensor (OpBuilder &builder, Location loc, Value elem, Type dstTp)
 Add conversion from scalar to given type (possibly a 0-rank tensor). More...
 
Value genIndexLoad (OpBuilder &builder, Location loc, Value mem, Value s)
 Generates a pointer/index load from the sparse storage scheme. More...
 
TypedAttr getOneAttr (Builder &builder, Type tp)
 Generates a 1-valued attribute of the given type. More...
 
Value genIsNonzero (OpBuilder &builder, Location loc, Value v)
 Generates the comparison v != 0 where v is of numeric type. More...
 
void genReshapeDstShape (OpBuilder &builder, Location loc, SmallVectorImpl< Value > &dstShape, ArrayRef< Value > srcShape, ArrayRef< Size > staticDstShape, ArrayRef< ReassociationIndices > reassociation)
 Computes the shape of destination tensor of a reshape operator. More...
 
void reshapeCvs (OpBuilder &builder, Location loc, ArrayRef< ReassociationIndices > reassociation, ValueRange srcSizes, ValueRange srcCvs, ValueRange dstSizes, SmallVectorImpl< Value > &dstCvs)
 Reshape coordinates during a reshaping operation. More...
 
FlatSymbolRefAttr getFunc (ModuleOp module, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Returns a function reference (first hit also inserts into module). More...
 
func::CallOp createFuncCall (OpBuilder &builder, Location loc, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Creates a CallOp to the function reference returned by getFunc() in the builder's module. More...
 
Type getOpaquePointerType (MLIRContext *ctx)
 Returns the equivalent of void* for opaque arguments to the execution engine. More...
 
Type getOpaquePointerType (Builder &builder)
 
Value genAlloca (OpBuilder &builder, Location loc, Value sz, Type tp)
 Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>). More...
 
Value genAlloca (OpBuilder &builder, Location loc, unsigned sz, Type tp, bool staticShape=false)
 Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true). More...
 
Value genAllocaScalar (OpBuilder &builder, Location loc, Type tp)
 Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>. More...
 
Value allocaBuffer (OpBuilder &builder, Location loc, ValueRange values)
 Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer). More...
 
Value allocDenseTensor (OpBuilder &builder, Location loc, RankedTensorType tensorTp, ValueRange sizes)
 Generates code to allocate a buffer of the given type, and zero initialize it. More...
 
void deallocDenseTensor (OpBuilder &builder, Location loc, Value buffer)
 Generates code to deallocate a dense buffer. More...
 
void sizesFromSrc (OpBuilder &builder, SmallVectorImpl< Value > &sizes, Location loc, Value src)
 Populates given sizes array from dense tensor or sparse tensor constant. More...
 
MemRefType get1DMemRefType (Type etp, bool withLayout)
 Generates a 1D MemRefType with a dynamic size. More...
 
OperationgetTop (Operation *op)
 Scans to top of generated loop. More...
 
void foreachInSparseConstant (OpBuilder &builder, Location loc, SparseElementsAttr attr, AffineMap order, function_ref< void(ArrayRef< Value >, Value)> callback)
 Iterate over a sparse constant, generates constantOp for value and coordinates. More...
 
SmallVector< ValueloadAll (OpBuilder &builder, Location loc, size_t size, Value mem, size_t offsetIdx=0, Value offsetVal=Value())
 Loads size-many values from the memref, which must have rank-1 and size greater-or-equal to size. More...
 
void storeAll (OpBuilder &builder, Location loc, Value mem, ValueRange vs, size_t offsetIdx=0, Value offsetVal=Value())
 Stores all the values of vs into the memref mem, which must have rank-1 and size greater-or-equal to vs.size(). More...
 
TypedValue< BaseMemRefTypegenToMemref (OpBuilder &builder, Location loc, Value tensor)
 
Value genToPositions (OpBuilder &builder, Location loc, Value tensor, Level lvl)
 Infers the result type and generates ToPositionsOp. More...
 
Value genToCoordinates (OpBuilder &builder, Location loc, Value tensor, Level lvl, Level cooStart)
 Infers the result type and generates ToCoordinatesOp. More...
 
Value genToCoordinatesBuffer (OpBuilder &builder, Location loc, Value tensor)
 Infers the result type and generates ToCoordinatesBufferOp. More...
 
Value genToValues (OpBuilder &builder, Location loc, Value tensor)
 Infers the result type and generates ToValuesOp. More...
 
Value genValMemSize (OpBuilder &builder, Location loc, Value tensor)
 Generates code to retrieve the values size for the sparse tensor. More...
 
Value createOrFoldSliceOffsetOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
 Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known. More...
 
Value createOrFoldSliceStrideOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
 Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known. More...
 
Value genReader (OpBuilder &builder, Location loc, SparseTensorType stt, Value tensor, SmallVectorImpl< Value > &dimSizesValues, Value &dimSizesBuffer)
 Generates code that opens a reader and sets the dimension sizes. More...
 
Value genMapBuffers (OpBuilder &builder, Location loc, SparseTensorType stt, ArrayRef< Value > dimSizesValues, Value dimSizesBuffer, SmallVectorImpl< Value > &lvlSizesValues, Value &dim2lvlBuffer, Value &lvl2dimBuffer)
 Generates code to set up the buffer parameters for a map. More...
 
Value constantZero (OpBuilder &builder, Location loc, Type tp)
 Generates a 0-valued constant of the given type. More...
 
Value constantOne (OpBuilder &builder, Location loc, Type tp)
 Generates a 1-valued constant of the given type. More...
 
Value constantIndex (OpBuilder &builder, Location loc, int64_t i)
 Generates a constant of index type. More...
 
Value constantI64 (OpBuilder &builder, Location loc, int64_t i)
 Generates a constant of i64 type. More...
 
Value constantI32 (OpBuilder &builder, Location loc, int32_t i)
 Generates a constant of i32 type. More...
 
Value constantI16 (OpBuilder &builder, Location loc, int16_t i)
 Generates a constant of i16 type. More...
 
Value constantI8 (OpBuilder &builder, Location loc, int8_t i)
 Generates a constant of i8 type. More...
 
Value constantI1 (OpBuilder &builder, Location loc, bool b)
 Generates a constant of i1 type. More...
 
Value constantAction (OpBuilder &builder, Location loc, Action action)
 Generates a constant of the given Action. More...
 
Value constantOverheadTypeEncoding (OpBuilder &builder, Location loc, unsigned width)
 Generates a constant of the internal type-encoding for overhead storage. More...
 
Value constantPosTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for position overhead storage. More...
 
Value constantCrdTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for coordinate overhead storage. More...
 
Value constantPrimaryTypeEncoding (OpBuilder &builder, Location loc, Type elemTp)
 Generates a constant of the internal type-encoding for primary storage. More...
 
Value constantLevelTypeEncoding (OpBuilder &builder, Location loc, LevelType lt)
 Generates a constant of the internal dimension level type encoding. More...
 
bool isZeroRankedTensorOrScalar (Type type)
 
UnrealizedConversionCastOp getTuple (Value tensor)
 Returns the "tuple" value of the adapted tensor. More...
 
Value genTuple (OpBuilder &builder, Location loc, Type tp, ValueRange values)
 Packs the given values as a "tuple" value. More...
 
Value genTuple (OpBuilder &builder, Location loc, SparseTensorDescriptor desc)
 
SparseTensorDescriptor getDescriptorFromTensorTuple (Value tensor)
 
MutSparseTensorDescriptor getMutDescriptorFromTensorTuple (Value tensor, SmallVectorImpl< Value > &fields)
 
static bool containsSparseTensor (TypeRange types)
 Return true if one of the given types is a sparse tensor type. More...
 
static ExpArity getExpArity (TensorExp::Kind k)
 
static const char * kindToOpSymbol (TensorExp::Kind kind)
 
static bool isAdmissibleBranchExp (Operation *op, Block *block, Value v)
 Ensures that the sparsifier can generate code for expression. More...
 
static bool isAdmissibleBranch (Operation *op, Region &region)
 Ensures that the sparsifier can generate code for branch. More...
 
static Value insertYieldOp (RewriterBase &rewriter, Location loc, Region &region, ValueRange vals)
 
static Value buildUnaryPresent (RewriterBase &rewriter, Location loc, Operation *op, Value v0)
 
static Value buildBinaryOverlap (RewriterBase &rewriter, Location loc, Operation *op, Value v0, Value v1)
 

Typedef Documentation

◆ complex32

using mlir::sparse_tensor::complex32 = typedef std::complex<float>

Definition at line 78 of file Enums.h.

◆ complex64

using mlir::sparse_tensor::complex64 = typedef std::complex<double>

Definition at line 77 of file Enums.h.

◆ Dimension

using mlir::sparse_tensor::Dimension = typedef uint64_t

The type of dimension identifiers and dimension-ranks.

Definition at line 35 of file SparseTensor.h.

◆ ExprId

using mlir::sparse_tensor::ExprId = typedef unsigned

TensorExp identifiers.

These are allocated by Merger::addExp, and serve as unique identifiers for the corresponding TensorExp object.

Definition at line 48 of file Merger.h.

◆ FieldIndex

using mlir::sparse_tensor::FieldIndex = typedef unsigned

The type of field indices.

This alias is to help code be more self-documenting; unfortunately it is not type-checked, so it only provides documentation rather than doing anything to prevent mixups.

Definition at line 107 of file SparseTensorStorageLayout.h.

◆ index_type

using mlir::sparse_tensor::index_type = typedef uint64_t

This type is used in the public API at all places where MLIR expects values with the built-in type "index".

For now, we simply assume that type is 64-bit, but targets with different "index" bitwidths should link with an alternatively built runtime support library.

Definition at line 46 of file Enums.h.

◆ LatPointId

using mlir::sparse_tensor::LatPointId = typedef unsigned

LatPoint identifiers.

These are allocated by Merger::addLat, and serve as unique identifiers for the corresponding LatPoint object.

Definition at line 52 of file Merger.h.

◆ LatSetId

using mlir::sparse_tensor::LatSetId = typedef unsigned

LatSet identifiers.

These are allocated by Merger::addSet (and by other methods calling that one), and serve as unique identifiers for the corresponding SmallVector<LatPointId> object.

Definition at line 57 of file Merger.h.

◆ Level

using mlir::sparse_tensor::Level = typedef uint64_t

The type of level identifiers and level-ranks.

Definition at line 38 of file SparseTensor.h.

◆ LoopCoeffPair

using mlir::sparse_tensor::LoopCoeffPair = typedef std::pair<LoopId, unsigned>

A pair of loop id and its coefficients.

E.g., for affine expression in the affine map 2 * d0, loop id = 0, coefficient = 2.

Definition at line 64 of file Merger.h.

◆ LoopId

using mlir::sparse_tensor::LoopId = typedef unsigned

Loop identifiers.

Definition at line 38 of file Merger.h.

◆ LoopOrd

using mlir::sparse_tensor::LoopOrd = typedef unsigned

The position of a loop in the loop-stack, or the position of a LoopId in a topologically-sorted list of LoopIds.

Although this type may have the same cardinality as LoopId, it must not be confused with that type. The LoopId type is used by the Merger as a unique identifier for loop-variables, regardless of the ordering of those loops. Whereas the LoopOrd type is used by the LoopEmitter (and CodegenEnv) to refer to the actual order in which loops are generated.

TODO: further explicate the correspondences between these various types. In particular, since the $dim argument to linalg::IndexOp is a De Bruijn index, it seems like that should correspond to LoopOrd, and yet the Merger has that correspond with LoopId instead. In addition LoopEmitter::genAffine has AffineDimExpr::position correspond to LoopId, however it is unclear what the providence of those AffineDimExpr is.

Definition at line 43 of file LoopEmitter.h.

◆ LvlLTPair

using mlir::sparse_tensor::LvlLTPair = typedef std::pair<Level, LevelType>

A pair of level and its corresponding LevelType of a tensor.

Definition at line 60 of file Merger.h.

◆ Size

using mlir::sparse_tensor::Size = typedef int64_t

The type for individual components of a compile-time shape, including the value ShapedType::kDynamic (for shapes).

Definition at line 42 of file SparseTensor.h.

◆ TensorId

using mlir::sparse_tensor::TensorId = typedef unsigned

Tensor identifiers, chosen to be the BlockArgument::getArgNumber of the value passed to Merger::buildTensorExp.

Definition at line 35 of file Merger.h.

◆ TensorLevel

using mlir::sparse_tensor::TensorLevel = typedef unsigned

Definition at line 46 of file LoopEmitter.h.

◆ TensorLoopId

using mlir::sparse_tensor::TensorLoopId = typedef unsigned

A compressed representation of std::pair<TensorId, LoopId>.

The compression scheme is such that this also serves as an index into the bitvector stored in LatPoint (since that bitvector is just the implementation for a set of TensorLoopId values).

Definition at line 44 of file Merger.h.

Enumeration Type Documentation

◆ Action

enum mlir::sparse_tensor::Action : uint32_t
strong

The actions performed by @newSparseTensor.

Enumerator
kEmpty 
kEmptyForward 
kFromCOO 
kFromReader 
kToCOO 
kPack 
kSortCOOInPlace 

Definition at line 145 of file Enums.h.

◆ EmitCInterface

Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall().

Enumerator
Off 
On 

Definition at line 36 of file CodegenUtils.h.

◆ ExpArity

Enumerator
kNullary 
kUnary 
kBinary 

Definition at line 22 of file Merger.cpp.

◆ LevelFormat

enum mlir::sparse_tensor::LevelFormat : uint8_t
strong

This enum defines all supported storage format without the level properties.

Enumerator
Dense 
Compressed 
Singleton 
LooseCompressed 
TwoOutOfFour 

Definition at line 187 of file Enums.h.

◆ LevelPropertyNondefault

This enum defines all the nondefault properties for storage formats.

Enumerator
Nonunique 
Nonordered 

Definition at line 196 of file Enums.h.

◆ LevelType

enum mlir::sparse_tensor::LevelType : uint8_t
strong

This enum defines all the sparse representations supportable by the SparseTensor dialect.

We use a lightweight encoding to encode both the "format" per se (dense, compressed, singleton, loose_compressed, two-out-of-four) as well as the "properties" (ordered, unique). The encoding is chosen for performance of the runtime library, and thus may change in future versions; consequently, client code should use the predicate functions defined below, rather than relying on knowledge about the particular binary encoding.

The Undef "format" is a special value used internally for cases where we need to store an undefined or indeterminate LevelType. It should not be used externally, since it does not indicate an actual/representable format.

Enumerator
Undef 
Dense 
Compressed 
CompressedNu 
CompressedNo 
CompressedNuNo 
Singleton 
SingletonNu 
SingletonNo 
SingletonNuNo 
LooseCompressed 
LooseCompressedNu 
LooseCompressedNo 
LooseCompressedNuNo 
TwoOutOfFour 

Definition at line 168 of file Enums.h.

◆ OverheadType

enum mlir::sparse_tensor::OverheadType : uint32_t
strong

Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor.

Enumerator
kIndex 
kU64 
kU32 
kU16 
kU8 

Definition at line 50 of file Enums.h.

◆ PrimaryType

enum mlir::sparse_tensor::PrimaryType : uint32_t
strong

Encoding of the elemental type, for "overloading" @newSparseTensor.

Enumerator
kF64 
kF32 
kF16 
kBF16 
kI64 
kI32 
kI16 
kI8 
kC64 
kC32 

Definition at line 81 of file Enums.h.

◆ SortMask

enum mlir::sparse_tensor::SortMask : unsigned
strong

Iteration graph sorting.

Enumerator
kIncludeDenseOutput 
kIncludeDenseInput 
kIncludeAll 
kIncludeDense 
kSparseOnly 

Definition at line 25 of file IterationGraphSorter.h.

◆ SparseTensorFieldKind

===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields.

Note that every memref with ? size actually behaves as a "vector", i.e. the stored size is the capacity and the used size resides in the storage_specifier struct.

struct { ; per-level l: ; if dense: <nothing> ; if compressed: memref<? x pos> positions ; positions for level l memref<? x crd> coordinates ; coordinates for level l ; if loose-compressed: memref<? x pos> positions ; lo/hi position pairs for level l memref<? x crd> coordinates ; coordinates for level l ; if singleton/2-out-of-4: memref<? x crd> coordinates ; coordinates for level l

memref<? x eltType> values ; values

struct sparse_tensor.storage_specifier { array<rank x int> lvlSizes ; sizes/cardinalities for each level array<n x int> memSizes; ; sizes/lengths for each data memref } };

In addition, for a "trailing COO region", defined as a compressed level followed by one or more singleton levels, the default SOA storage that is inherent to the TACO format is optimized into an AOS storage where all coordinates of a stored element appear consecutively. In such cases, a special operation (sparse_tensor.coordinates_buffer) must be used to access the AOS coordinates array. In the code below, the method getCOOStart is used to find the start of the "trailing COO region".

If the sparse tensor is a slice (produced by tensor.extract_slice operation), instead of allocating a new sparse tensor for it, it reuses the same sets of MemRefs but attaching a additional set of slicing-metadata for per-dimension slice offset and stride.

Examples.

#CSR storage of 2-dim matrix yields memref<?xindex> ; positions-1 memref<?xindex> ; coordinates-1 memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

#COO storage of 2-dim matrix yields memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

Slice on #COO storage of 2-dim matrix yields ;; Inherited from the original sparse tensors memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>, ; lvl0, lvl1, 3xsizes ;; Extra slicing-metadata array<2 x i64>, array<2 x i64>)>) ; dim offset, dim stride.

===-------------------------------------------------------------------—===//

Enumerator
StorageSpec 
PosMemRef 
CrdMemRef 
ValMemRef 

Definition at line 87 of file SparseTensorStorageLayout.h.

Function Documentation

◆ allocaBuffer()

Value mlir::sparse_tensor::allocaBuffer ( OpBuilder builder,
Location  loc,
ValueRange  values 
)

Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer).

Definition at line 380 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and genAlloca().

Referenced by genMapBuffers(), and genReader().

◆ allocDenseTensor()

Value mlir::sparse_tensor::allocDenseTensor ( OpBuilder builder,
Location  loc,
RankedTensorType  tensorTp,
ValueRange  sizes 
)

Generates code to allocate a buffer of the given type, and zero initialize it.

If the buffer type has any dynamic sizes, then the sizes parameter should be as filled by sizesFromPtr(); that way we can reuse the genDimSizeCall() results generated by sizesFromPtr().

Definition at line 392 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), and mlir::get().

◆ buildBinaryOverlap()

static Value mlir::sparse_tensor::buildBinaryOverlap ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0,
Value  v1 
)
static

Definition at line 1524 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ buildLevelType()

constexpr std::optional<LevelType> mlir::sparse_tensor::buildLevelType ( LevelFormat  lf,
bool  ordered,
bool  unique 
)
constexpr

Convert a LevelFormat to its corresponding LevelType with the given properties.

Returns std::nullopt when the properties are not applicable for the input level format.

Definition at line 314 of file Enums.h.

References isValidLT().

Referenced by mlir::sparse_tensor::SparseTensorType::getCOOType(), and getNormalizedEncodingForSpecifier().

◆ buildSparsifier()

void mlir::sparse_tensor::buildSparsifier ( OpPassManager pm,
const SparsifierOptions options 
)

◆ buildUnaryPresent()

static Value mlir::sparse_tensor::buildUnaryPresent ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0 
)
static

Definition at line 1510 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ constantAction()

Value mlir::sparse_tensor::constantAction ( OpBuilder builder,
Location  loc,
Action  action 
)
inline

Generates a constant of the given Action.

Definition at line 391 of file CodegenUtils.h.

References constantI32().

◆ constantCrdTypeEncoding()

Value mlir::sparse_tensor::constantCrdTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for coordinate overhead storage.

Definition at line 411 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantI1()

Value mlir::sparse_tensor::constantI1 ( OpBuilder builder,
Location  loc,
bool  b 
)
inline

Generates a constant of i1 type.

Definition at line 386 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by createEqCompare(), createPartitionFunc(), endIf(), endLoop(), genCompressed(), genIf(), and genSparseReducedAffineCond().

◆ constantI16()

Value mlir::sparse_tensor::constantI16 ( OpBuilder builder,
Location  loc,
int16_t  i 
)
inline

Generates a constant of i16 type.

Definition at line 376 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantI32()

Value mlir::sparse_tensor::constantI32 ( OpBuilder builder,
Location  loc,
int32_t  i 
)
inline

Generates a constant of i32 type.

Definition at line 371 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantAction(), constantOverheadTypeEncoding(), and constantPrimaryTypeEncoding().

◆ constantI64()

Value mlir::sparse_tensor::constantI64 ( OpBuilder builder,
Location  loc,
int64_t  i 
)
inline

Generates a constant of i64 type.

Definition at line 366 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by createQuickSortFunc().

◆ constantI8()

Value mlir::sparse_tensor::constantI8 ( OpBuilder builder,
Location  loc,
int8_t  i 
)
inline

Generates a constant of i8 type.

Definition at line 381 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantLevelTypeEncoding().

◆ constantIndex()

Value mlir::sparse_tensor::constantIndex ( OpBuilder builder,
Location  loc,
int64_t  i 
)
inline

◆ constantLevelTypeEncoding()

Value mlir::sparse_tensor::constantLevelTypeEncoding ( OpBuilder builder,
Location  loc,
LevelType  lt 
)
inline

Generates a constant of the internal dimension level type encoding.

Definition at line 424 of file CodegenUtils.h.

References constantI8().

◆ constantOne()

Value mlir::sparse_tensor::constantOne ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 1-valued constant of the given type.

This supports all the same types as constantZero.

Definition at line 350 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), getOneAttr(), and mlir::Builder::getZeroAttr().

Referenced by createFor(), and createPartitionFunc().

◆ constantOverheadTypeEncoding()

Value mlir::sparse_tensor::constantOverheadTypeEncoding ( OpBuilder builder,
Location  loc,
unsigned  width 
)
inline

Generates a constant of the internal type-encoding for overhead storage.

Definition at line 396 of file CodegenUtils.h.

References constantI32(), and overheadTypeEncoding().

Referenced by constantCrdTypeEncoding(), and constantPosTypeEncoding().

◆ constantPosTypeEncoding()

Value mlir::sparse_tensor::constantPosTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for position overhead storage.

Definition at line 404 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantPrimaryTypeEncoding()

Value mlir::sparse_tensor::constantPrimaryTypeEncoding ( OpBuilder builder,
Location  loc,
Type  elemTp 
)
inline

Generates a constant of the internal type-encoding for primary storage.

Definition at line 417 of file CodegenUtils.h.

References constantI32(), and primaryTypeEncoding().

Referenced by genReader().

◆ constantZero()

Value mlir::sparse_tensor::constantZero ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 0-valued constant of the given type.

In addition to the scalar types (ComplexType, FloatType, IndexType, IntegerType), this also works for RankedTensorType and VectorType (for which it generates a constant DenseElementsAttr of zeros).

Definition at line 339 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), and mlir::Builder::getZeroAttr().

Referenced by allocDenseTensor(), allocSchemeForRank(), createAllocation(), createAllocFields(), createFor(), genEndInsert(), genIsNonzero(), and mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ containsSparseTensor()

static bool mlir::sparse_tensor::containsSparseTensor ( TypeRange  types)
static

Return true if one of the given types is a sparse tensor type.

Definition at line 40 of file SparsificationAndBufferizationPass.cpp.

References getSparseTensorEncoding().

◆ crdTypeEncoding()

OverheadType mlir::sparse_tensor::crdTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for coordinate overhead storage.

Definition at line 76 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ createFuncCall()

func::CallOp mlir::sparse_tensor::createFuncCall ( OpBuilder builder,
Location  loc,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Creates a CallOp to the function reference returned by getFunc() in the builder's module.

Definition at line 343 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::OpBuilder::getBlock(), getFunc(), mlir::Operation::getParentOfType(), and mlir::Block::getParentOp().

Referenced by dumpIndexMemRef(), and genReader().

◆ createOrFoldSliceOffsetOp()

Value mlir::sparse_tensor::createOrFoldSliceOffsetOp ( OpBuilder builder,
Location  loc,
Value  tensor,
Dimension  dim 
)

Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 596 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceOffset().

◆ createOrFoldSliceStrideOp()

Value mlir::sparse_tensor::createOrFoldSliceStrideOp ( OpBuilder builder,
Location  loc,
Value  tensor,
Dimension  dim 
)

Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 606 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceStride().

◆ deallocDenseTensor()

void mlir::sparse_tensor::deallocDenseTensor ( OpBuilder builder,
Location  loc,
Value  buffer 
)

Generates code to deallocate a dense buffer.

Definition at line 409 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

◆ decodeConst()

constexpr uint64_t mlir::sparse_tensor::decodeConst ( uint64_t  v)
constexpr

Definition at line 555 of file Enums.h.

◆ decodeIndex()

constexpr uint64_t mlir::sparse_tensor::decodeIndex ( uint64_t  v)
constexpr

Definition at line 554 of file Enums.h.

◆ decodeMulc()

constexpr uint64_t mlir::sparse_tensor::decodeMulc ( uint64_t  v)
constexpr

Definition at line 556 of file Enums.h.

◆ decodeMuli()

constexpr uint64_t mlir::sparse_tensor::decodeMuli ( uint64_t  v)
constexpr

Definition at line 557 of file Enums.h.

◆ encodeDim()

constexpr uint64_t mlir::sparse_tensor::encodeDim ( uint64_t  i,
uint64_t  cf,
uint64_t  cm 
)
constexpr

Bit manipulations for affine encoding.

Note that because the indices in the mappings refer to dimensions and levels (and not the sizes of these dimensions and levels), the 64-bit encoding gives ample room for a compact encoding of affine operations in the higher bits. Pure permutations still allow for 60-bit indices. But non-permutations reserve 20-bits for the potential three components (index i, constant, index ii).

The compact encoding is as follows:

0xffffffffffffffff |0000 | 60-bit idx| e.g. i |0001 floor| 20-bit const|20-bit idx| e.g. i floor c |0010 mod | 20-bit const|20-bit idx| e.g. i mod c |0011 mul |20-bit idx|20-bit const|20-bit idx| e.g. i + c * ii

This encoding provides sufficient generality for currently supported sparse tensor types. To generalize this more, we will need to provide a broader encoding scheme for affine functions. Also, the library encoding may be replaced with pure "direct-IR" code in the future.

Definition at line 531 of file Enums.h.

Referenced by genMapBuffers().

◆ encodeLvl()

constexpr uint64_t mlir::sparse_tensor::encodeLvl ( uint64_t  i,
uint64_t  c,
uint64_t  ii 
)
constexpr

Definition at line 543 of file Enums.h.

Referenced by genMapBuffers().

◆ foreachFieldAndTypeInSparseTensor()

void mlir::sparse_tensor::foreachFieldAndTypeInSparseTensor ( SparseTensorType  stt,
llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>  callback 
)

◆ foreachFieldInSparseTensor()

void mlir::sparse_tensor::foreachFieldInSparseTensor ( SparseTensorEncodingAttr  enc,
llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)>  callback 
)
inline

◆ foreachInSparseConstant()

void mlir::sparse_tensor::foreachInSparseConstant ( OpBuilder builder,
Location  loc,
SparseElementsAttr  attr,
AffineMap  order,
function_ref< void(ArrayRef< Value >, Value)>  callback 
)

Iterate over a sparse constant, generates constantOp for value and coordinates.

E.g., sparse<[ [0], [28], [31] ], [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] > => c1 = arith.constant 0 v1 = complex.constant (5.13, 2.0) callback({c1}, v1)

c2 = arith.constant 28 v2 = complex.constant (3.0, 4.0) callback({c2}, v2)

c3 = arith.constant 31 v3 = complex.constant (5.0, 6.0) callback({c3}, v3)

Definition at line 432 of file CodegenUtils.cpp.

References mlir::AffineMap::compose(), mlir::Builder::getMultiDimIdentityMap(), mlir::AffineMap::getNumResults(), and getRankedTensorType().

◆ genAlloca() [1/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
unsigned  sz,
Type  tp,
bool  staticShape = false 
)

Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true).

Definition at line 360 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), genAlloca(), and mlir::get().

◆ genAlloca() [2/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
Value  sz,
Type  tp 
)

Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>).

Definition at line 369 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

Referenced by allocaBuffer(), allocSlicePosBuf(), and genAlloca().

◆ genAllocaScalar()

Value mlir::sparse_tensor::genAllocaScalar ( OpBuilder builder,
Location  loc,
Type  tp 
)

Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>.

Definition at line 375 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

◆ genCast()

Value mlir::sparse_tensor::genCast ( OpBuilder builder,
Location  loc,
Value  value,
Type  dstTy 
)

Add type casting between arith and index types when needed.

Definition at line 151 of file CodegenUtils.cpp.

References mlir::convertScalarToDtype(), mlir::OpBuilder::create(), and mlir::Value::getType().

Referenced by genCompressed(), genLoad(), genScalarToTensor(), and genStore().

◆ genIndexLoad()

Value mlir::sparse_tensor::genIndexLoad ( OpBuilder builder,
Location  loc,
Value  mem,
Value  s 
)

Generates a pointer/index load from the sparse storage scheme.

Narrower data types need to be zero extended before casting the value into the index type used for looping and indexing.

Definition at line 177 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::Builder::getI64Type(), mlir::Builder::getIndexType(), mlir::Type::getIntOrFloatBitWidth(), and mlir::Value::getType().

Referenced by mlir::sparse_tensor::LoopEmitter::enterFilterLoopOverTensorAtLvl(), genSparseReducedAffineCond(), loadSlicePos(), loadSlicePosPtr(), and loadSlicePosTupleNum().

◆ genIsNonzero()

Value mlir::sparse_tensor::genIsNonzero ( OpBuilder builder,
mlir::Location  loc,
Value  v 
)

Generates the comparison v != 0 where v is of numeric type.

For floating types, we use the "unordered" comparator (i.e., returns true if v is NaN).

Definition at line 204 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), mlir::Value::getType(), and mlir::Type::isIntOrIndex().

◆ genMapBuffers()

Value mlir::sparse_tensor::genMapBuffers ( OpBuilder builder,
Location  loc,
SparseTensorType  stt,
ArrayRef< Value dimSizesValues,
Value  dimSizesBuffer,
SmallVectorImpl< Value > &  lvlSizesValues,
Value dim2lvlBuffer,
Value lvl2dimBuffer 
)

◆ genReader()

Value mlir::sparse_tensor::genReader ( OpBuilder builder,
Location  loc,
SparseTensorType  stt,
Value  tensor,
SmallVectorImpl< Value > &  dimSizesValues,
Value dimSizesBuffer 
)

◆ genReshapeDstShape()

void mlir::sparse_tensor::genReshapeDstShape ( OpBuilder builder,
Location  loc,
SmallVectorImpl< Value > &  dstShape,
ArrayRef< Value srcShape,
ArrayRef< Size staticDstShape,
ArrayRef< ReassociationIndices reassociation 
)

Computes the shape of destination tensor of a reshape operator.

This is only used when operands have dynamic shape. The shape of the destination is stored into dstShape.

Definition at line 219 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and product().

◆ genScalarToTensor()

Value mlir::sparse_tensor::genScalarToTensor ( OpBuilder builder,
Location  loc,
Value  elem,
Type  dstTp 
)

Add conversion from scalar to given type (possibly a 0-rank tensor).

Definition at line 166 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), and genCast().

◆ genToCoordinates()

Value mlir::sparse_tensor::genToCoordinates ( OpBuilder builder,
Location  loc,
Value  tensor,
Level  lvl,
Level  cooStart 
)

Infers the result type and generates ToCoordinatesOp.

If the level is within a COO region, the result type is a memref with unknown stride and offset. Otherwise, the result type is a memref without any specified layout.

Definition at line 566 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), mlir::Builder::getIndexAttr(), and getSparseTensorType().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ genToCoordinatesBuffer()

Value mlir::sparse_tensor::genToCoordinatesBuffer ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Infers the result type and generates ToCoordinatesBufferOp.

Definition at line 575 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), and getSparseTensorType().

◆ genToMemref()

TypedValue< BaseMemRefType > mlir::sparse_tensor::genToMemref ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Definition at line 550 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), and mlir::Value::getType().

◆ genToPositions()

Value mlir::sparse_tensor::genToPositions ( OpBuilder builder,
Location  loc,
Value  tensor,
Level  lvl 
)

Infers the result type and generates ToPositionsOp.

Definition at line 557 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), mlir::Builder::getIndexAttr(), and getSparseTensorType().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ genToValues()

Value mlir::sparse_tensor::genToValues ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Infers the result type and generates ToValuesOp.

Definition at line 583 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), and getRankedTensorType().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ genTuple() [1/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
SparseTensorDescriptor  desc 
)
inline

◆ genTuple() [2/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
Type  tp,
ValueRange  values 
)
inline

Packs the given values as a "tuple" value.

Definition at line 237 of file SparseTensorDescriptor.h.

References mlir::OpBuilder::create().

Referenced by genTuple(), and mlir::SparseTensorTypeToBufferConverter::SparseTensorTypeToBufferConverter().

◆ genValMemSize()

Value mlir::sparse_tensor::genValMemSize ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Generates code to retrieve the values size for the sparse tensor.

Definition at line 591 of file CodegenUtils.cpp.

References getDescriptorFromTensorTuple(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getValMemSize().

◆ get1DMemRefType()

MemRefType mlir::sparse_tensor::get1DMemRefType ( Type  etp,
bool  withLayout 
)
inline

Generates a 1D MemRefType with a dynamic size.

When withLayout is set, the returned memref has a layout has unknown strides and offsets. Otherwise, a memref with a standard unit stride zero offset layout is returned.

Definition at line 234 of file CodegenUtils.h.

References mlir::get(), and mlir::Type::getContext().

Referenced by genToCoordinates(), genToCoordinatesBuffer(), genToPositions(), and genToValues().

◆ getBlockSize()

SmallVector< unsigned > mlir::sparse_tensor::getBlockSize ( AffineMap  dimToLvl)

Given the dimToLvl map, returns the block sizes in a vector.

For instance, a 2x3 block will return [2, 3]. Unblocked dimension i will return 0, and i floordiv 1, i mod 1 will return 1. Therefore, the example below will return [0, 1]. map = ( i, j ) -> ( i : dense, j floordiv 1 : compressed, j mod 1 : dense ) Only valid block sparsity will be accepted.

Definition at line 838 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getResults(), isBlockSparsity(), and mlir::Mod.

◆ getDescriptorFromTensorTuple()

SparseTensorDescriptor mlir::sparse_tensor::getDescriptorFromTensorTuple ( Value  tensor)
inline

Definition at line 248 of file SparseTensorDescriptor.h.

References getTuple().

Referenced by genValMemSize().

◆ getExpArity()

static ExpArity mlir::sparse_tensor::getExpArity ( TensorExp::Kind  k)
static

Definition at line 28 of file Merger.cpp.

References mlir::sparse_tensor::TensorExp::kAbsC, mlir::sparse_tensor::TensorExp::kAbsF, mlir::sparse_tensor::TensorExp::kAbsI, mlir::sparse_tensor::TensorExp::kAddC, mlir::sparse_tensor::TensorExp::kAddF, mlir::sparse_tensor::TensorExp::kAddI, mlir::sparse_tensor::TensorExp::kAndI, mlir::sparse_tensor::TensorExp::kBinary, kBinary, mlir::sparse_tensor::TensorExp::kBinaryBranch, mlir::sparse_tensor::TensorExp::kBitCast, mlir::sparse_tensor::TensorExp::kCastFS, mlir::sparse_tensor::TensorExp::kCastFU, mlir::sparse_tensor::TensorExp::kCastIdx, mlir::sparse_tensor::TensorExp::kCastS, mlir::sparse_tensor::TensorExp::kCastSF, mlir::sparse_tensor::TensorExp::kCastU, mlir::sparse_tensor::TensorExp::kCastUF, mlir::sparse_tensor::TensorExp::kCeilF, mlir::sparse_tensor::TensorExp::kCIm, mlir::sparse_tensor::TensorExp::kCmpF, mlir::sparse_tensor::TensorExp::kCmpI, mlir::sparse_tensor::TensorExp::kCRe, mlir::sparse_tensor::TensorExp::kDenseOp, mlir::sparse_tensor::TensorExp::kDivC, mlir::sparse_tensor::TensorExp::kDivF, mlir::sparse_tensor::TensorExp::kDivS, mlir::sparse_tensor::TensorExp::kDivU, mlir::sparse_tensor::TensorExp::kExpm1C, mlir::sparse_tensor::TensorExp::kExpm1F, mlir::sparse_tensor::TensorExp::kExtF, mlir::sparse_tensor::TensorExp::kFloorF, mlir::sparse_tensor::TensorExp::kInvariant, mlir::sparse_tensor::TensorExp::kLog1pC, mlir::sparse_tensor::TensorExp::kLog1pF, mlir::sparse_tensor::TensorExp::kLoopVar, mlir::sparse_tensor::TensorExp::kMulC, mlir::sparse_tensor::TensorExp::kMulF, mlir::sparse_tensor::TensorExp::kMulI, mlir::sparse_tensor::TensorExp::kNegC, mlir::sparse_tensor::TensorExp::kNegF, mlir::sparse_tensor::TensorExp::kNegI, kNullary, mlir::sparse_tensor::TensorExp::kOrI, mlir::sparse_tensor::TensorExp::kReduce, mlir::sparse_tensor::TensorExp::kSelect, mlir::sparse_tensor::TensorExp::kShlI, mlir::sparse_tensor::TensorExp::kShrS, mlir::sparse_tensor::TensorExp::kShrU, mlir::sparse_tensor::TensorExp::kSinC, mlir::sparse_tensor::TensorExp::kSinF, mlir::sparse_tensor::TensorExp::kSqrtC, mlir::sparse_tensor::TensorExp::kSqrtF, mlir::sparse_tensor::TensorExp::kSubC, mlir::sparse_tensor::TensorExp::kSubF, mlir::sparse_tensor::TensorExp::kSubI, mlir::sparse_tensor::TensorExp::kSynZero, mlir::sparse_tensor::TensorExp::kTanhC, mlir::sparse_tensor::TensorExp::kTanhF, mlir::sparse_tensor::TensorExp::kTensor, mlir::sparse_tensor::TensorExp::kTruncF, mlir::sparse_tensor::TensorExp::kTruncI, mlir::sparse_tensor::TensorExp::kUnary, kUnary, and mlir::sparse_tensor::TensorExp::kXorI.

Referenced by mlir::sparse_tensor::Merger::expContainsTensor(), and mlir::sparse_tensor::Merger::hasNegateOnOut().

◆ getFunc()

FlatSymbolRefAttr mlir::sparse_tensor::getFunc ( ModuleOp  module,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Returns a function reference (first hit also inserts into module).

Sets the "_emit_c_interface" on the function declaration when requested, so that LLVM lowering generates a wrapper function that takes care of ABI complications with passing in and returning MemRefs to C functions.

Definition at line 323 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), mlir::ValueRange::getTypes(), and mlir::Operation::setAttr().

Referenced by createFuncCall().

◆ getLevelFormat()

constexpr std::optional<LevelFormat> mlir::sparse_tensor::getLevelFormat ( LevelType  lt)
constexpr

Convert a LevelType to its corresponding LevelFormat.

Returns std::nullopt when input lt is Undef.

Definition at line 305 of file Enums.h.

References Undef.

Referenced by getNormalizedEncodingForSpecifier().

◆ getMemRefType()

template<typename T >
MemRefType mlir::sparse_tensor::getMemRefType ( T &&  t)
inline

Convenience method to abbreviate casting getType().

Definition at line 82 of file SparseTensor.h.

Referenced by getMangledSortHelperFuncName(), and isMatchingWidth().

◆ getMutDescriptorFromTensorTuple()

MutSparseTensorDescriptor mlir::sparse_tensor::getMutDescriptorFromTensorTuple ( Value  tensor,
SmallVectorImpl< Value > &  fields 
)
inline

Definition at line 255 of file SparseTensorDescriptor.h.

References getTuple().

◆ getNumDataFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumDataFieldsFromEncoding ( SparseTensorEncodingAttr  enc)
inline

◆ getNumFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumFieldsFromEncoding ( SparseTensorEncodingAttr  enc)
inline

◆ getOneAttr()

mlir::TypedAttr mlir::sparse_tensor::getOneAttr ( Builder builder,
Type  tp 
)

Generates a 1-valued attribute of the given type.

This supports all the same types as getZeroAttr; however, unlike getZeroAttr, for unsupported types we raise llvm_unreachable rather than returning a null attribute.

Definition at line 189 of file CodegenUtils.cpp.

References mlir::DenseElementsAttr::get(), mlir::Builder::getFloatAttr(), mlir::Builder::getIndexAttr(), and mlir::Builder::getIntegerAttr().

Referenced by constantOne().

◆ getOpaquePointerType() [1/2]

Type mlir::sparse_tensor::getOpaquePointerType ( Builder builder)

Definition at line 356 of file CodegenUtils.cpp.

References mlir::Builder::getContext(), and getOpaquePointerType().

◆ getOpaquePointerType() [2/2]

Type mlir::sparse_tensor::getOpaquePointerType ( MLIRContext ctx)

Returns the equivalent of void* for opaque arguments to the execution engine.

Definition at line 352 of file CodegenUtils.cpp.

References mlir::get().

Referenced by genReader(), and getOpaquePointerType().

◆ getOverheadType()

Type mlir::sparse_tensor::getOverheadType ( Builder builder,
OverheadType  ot 
)

Converts the internal type-encoding for overhead storage to an mlir::Type.

Definition at line 54 of file CodegenUtils.cpp.

References mlir::Builder::getIndexType(), mlir::Builder::getIntegerType(), kIndex, kU16, kU32, kU64, and kU8.

◆ getRankedTensorType()

template<typename T >
RankedTensorType mlir::sparse_tensor::getRankedTensorType ( T &&  t)
inline

Convenience method to abbreviate casting getType().

Definition at line 74 of file SparseTensor.h.

Referenced by foreachInSparseConstant(), genToValues(), and mlir::sparse_tensor::LoopEmitter::initialize().

◆ getSparseTensorEncoding()

SparseTensorEncodingAttr mlir::sparse_tensor::getSparseTensorEncoding ( Type  type)

◆ getSparseTensorType()

SparseTensorType mlir::sparse_tensor::getSparseTensorType ( Value  val)
inline

◆ getTop()

Operation * mlir::sparse_tensor::getTop ( Operation op)

Scans to top of generated loop.

Definition at line 422 of file CodegenUtils.cpp.

◆ getTuple()

UnrealizedConversionCastOp mlir::sparse_tensor::getTuple ( Value  tensor)
inline

Returns the "tuple" value of the adapted tensor.

Definition at line 232 of file SparseTensorDescriptor.h.

References mlir::Value::getDefiningOp().

Referenced by flattenOperands(), getDescriptorFromTensorTuple(), and getMutDescriptorFromTensorTuple().

◆ hasAnyNonIdentityOperandsOrResults()

bool mlir::sparse_tensor::hasAnyNonIdentityOperandsOrResults ( Operation op)

Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps.

Definition at line 885 of file SparseTensorDialect.cpp.

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseOperand()

bool mlir::sparse_tensor::hasAnySparseOperand ( Operation op)
inline

Returns true iff MLIR operand has any sparse operand.

Definition at line 93 of file SparseTensor.h.

◆ hasAnySparseOperandOrResult()

bool mlir::sparse_tensor::hasAnySparseOperandOrResult ( Operation op)
inline

Returns true iff MLIR operand has any sparse operand or result.

Definition at line 107 of file SparseTensor.h.

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseResult()

bool mlir::sparse_tensor::hasAnySparseResult ( Operation op)
inline

Returns true iff MLIR operand has any sparse result.

Definition at line 100 of file SparseTensor.h.

◆ inferLvlToDim()

AffineMap mlir::sparse_tensor::inferLvlToDim ( AffineMap  dimToLvl,
MLIRContext context 
)

Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails.

Definition at line 770 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getNumSymbols(), inverseBlockSparsity(), mlir::inversePermutation(), isBlockSparsity(), and mlir::AffineMap::isPermutation().

Referenced by genReplaceDimToLvlMap().

◆ insertYieldOp()

static Value mlir::sparse_tensor::insertYieldOp ( RewriterBase rewriter,
Location  loc,
Region region,
ValueRange  vals 
)
static

◆ inverseBlockSparsity()

AffineMap mlir::sparse_tensor::inverseBlockSparsity ( AffineMap  dimToLvl,
MLIRContext context 
)

Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases.

Asserts on failure (so only use when known to succeed).

Definition at line 785 of file SparseTensorDialect.cpp.

References mlir::Add, mlir::FloorDiv, mlir::AffineMap::get(), mlir::getAffineBinaryOpExpr(), mlir::getAffineDimExpr(), mlir::AffineMap::getNumResults(), mlir::AffineMap::getResult(), mlir::Mod, and mlir::Mul.

Referenced by inferLvlToDim().

◆ is2OutOf4LT()

constexpr bool mlir::sparse_tensor::is2OutOf4LT ( LevelType  lt)
constexpr

◆ isAdmissibleBranch()

static bool mlir::sparse_tensor::isAdmissibleBranch ( Operation op,
Region region 
)
static

Ensures that the sparsifier can generate code for branch.

Definition at line 1242 of file Merger.cpp.

◆ isAdmissibleBranchExp()

static bool mlir::sparse_tensor::isAdmissibleBranchExp ( Operation op,
Block block,
Value  v 
)
static

Ensures that the sparsifier can generate code for expression.

Definition at line 1222 of file Merger.cpp.

◆ isBlockSparsity()

bool mlir::sparse_tensor::isBlockSparsity ( AffineMap  dimToLvl)

Given the dimToLvl map, returns if it's block sparsity.

Definition at line 855 of file SparseTensorDialect.cpp.

References mlir::FloorDiv, mlir::AffineMap::getResults(), and mlir::Mod.

Referenced by getBlockSize(), and inferLvlToDim().

◆ isComplexPrimaryType()

constexpr bool mlir::sparse_tensor::isComplexPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 140 of file Enums.h.

References kC32, and kC64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isCompressedLT()

constexpr bool mlir::sparse_tensor::isCompressedLT ( LevelType  lt)
constexpr

◆ isDenseLT()

constexpr bool mlir::sparse_tensor::isDenseLT ( LevelType  lt)
constexpr

◆ isEncodedFloor()

constexpr bool mlir::sparse_tensor::isEncodedFloor ( uint64_t  v)
constexpr

Definition at line 551 of file Enums.h.

◆ isEncodedMod()

constexpr bool mlir::sparse_tensor::isEncodedMod ( uint64_t  v)
constexpr

Definition at line 552 of file Enums.h.

◆ isEncodedMul()

constexpr bool mlir::sparse_tensor::isEncodedMul ( uint64_t  v)
constexpr

Definition at line 553 of file Enums.h.

◆ isFloatingPrimaryType()

constexpr bool mlir::sparse_tensor::isFloatingPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 128 of file Enums.h.

References kBF16, and kF64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isIntegralPrimaryType()

constexpr bool mlir::sparse_tensor::isIntegralPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 132 of file Enums.h.

References kI64, and kI8.

◆ isLooseCompressedLT()

constexpr bool mlir::sparse_tensor::isLooseCompressedLT ( LevelType  lt)
constexpr

◆ isOrderedLT()

constexpr bool mlir::sparse_tensor::isOrderedLT ( LevelType  lt)
constexpr

Check if the LevelType is ordered (regardless of storage format).

Definition at line 294 of file Enums.h.

Referenced by mlir::sparse_tensor::SparseTensorType::isOrderedLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isOrderedLvl().

◆ isRealPrimaryType()

constexpr bool mlir::sparse_tensor::isRealPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 136 of file Enums.h.

References kF64, and kI8.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isSingletonLT()

constexpr bool mlir::sparse_tensor::isSingletonLT ( LevelType  lt)
constexpr

◆ isUndefLT()

constexpr bool mlir::sparse_tensor::isUndefLT ( LevelType  lt)
constexpr

Check if the LevelType is the special undefined value.

Definition at line 250 of file Enums.h.

References Undef.

Referenced by findAffine(), findDepIdxSet(), genIf(), mlir::sparse_tensor::ir_detail::LvlSpec::LvlSpec(), startLoopSeq(), and translateBitsToTidLvlPairs().

◆ isUniqueLT()

constexpr bool mlir::sparse_tensor::isUniqueLT ( LevelType  lt)
constexpr

Check if the LevelType is unique (regardless of storage format).

Definition at line 299 of file Enums.h.

Referenced by mlir::sparse_tensor::SparseTensorType::isUniqueLvl(), mlir::sparse_tensor::SparseTensorStorageBase::isUniqueLvl(), and translateBitsToTidLvlPairs().

◆ isValidLT()

constexpr bool mlir::sparse_tensor::isValidLT ( LevelType  lt)
constexpr

Check that the LevelType contains a valid (possibly undefined) value.

Definition at line 239 of file Enums.h.

Referenced by buildLevelType(), mlir::sparse_tensor::ir_detail::LvlSpec::LvlSpec(), mlir::sparse_tensor::ir_detail::LvlTypeParser::parseLvlType(), and mlir::sparse_tensor::Merger::setLevelAndType().

◆ isWithCrdLT()

constexpr bool mlir::sparse_tensor::isWithCrdLT ( LevelType  lt)
constexpr

Check if the LevelType needs coordinates array.

Definition at line 288 of file Enums.h.

References is2OutOf4LT(), isCompressedLT(), isLooseCompressedLT(), and isSingletonLT().

Referenced by mlir::sparse_tensor::StorageLayout::foreachField(), and mlir::sparse_tensor::SparseTensorType::isWithCrd().

◆ isWithPosLT()

constexpr bool mlir::sparse_tensor::isWithPosLT ( LevelType  lt)
constexpr

Check if the LevelType needs positions array.

Definition at line 283 of file Enums.h.

References isCompressedLT(), and isLooseCompressedLT().

Referenced by mlir::sparse_tensor::StorageLayout::foreachField(), and mlir::sparse_tensor::SparseTensorType::isWithPos().

◆ isZeroRankedTensorOrScalar()

bool mlir::sparse_tensor::isZeroRankedTensorOrScalar ( Type  type)
inline

Definition at line 429 of file CodegenUtils.h.

Referenced by mlir::sparse_tensor::LoopEmitter::initialize().

◆ kindToOpSymbol()

static const char* mlir::sparse_tensor::kindToOpSymbol ( TensorExp::Kind  kind)
static

Definition at line 692 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::dumpExp().

◆ loadAll()

SmallVector< Value > mlir::sparse_tensor::loadAll ( OpBuilder builder,
Location  loc,
size_t  size,
Value  mem,
size_t  offsetIdx = 0,
Value  offsetVal = Value() 
)

Loads size-many values from the memref, which must have rank-1 and size greater-or-equal to size.

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value after loading.

Definition at line 507 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::Value::getType().

◆ overheadTypeEncoding() [1/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( Type  tp)

Converts an overhead storage type to its internal type-encoding.

Definition at line 46 of file CodegenUtils.cpp.

References mlir::Type::isIndex(), kIndex, and overheadTypeEncoding().

◆ overheadTypeEncoding() [2/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( unsigned  width)

Converts an overhead storage bitwidth to its internal type-encoding.

Definition at line 30 of file CodegenUtils.cpp.

References kIndex, kU16, kU32, kU64, and kU8.

Referenced by constantOverheadTypeEncoding(), crdTypeEncoding(), overheadTypeEncoding(), overheadTypeFunctionSuffix(), and posTypeEncoding().

◆ overheadTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( OverheadType  ot)

Convert OverheadType to its function-name suffix.

Definition at line 88 of file CodegenUtils.cpp.

References CASE, kIndex, and MLIR_SPARSETENSOR_FOREVERY_FIXED_O.

Referenced by overheadTypeFunctionSuffix().

◆ overheadTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( Type  overheadTp)

Converts an overhead storage type to its function-name suffix.

Definition at line 101 of file CodegenUtils.cpp.

References overheadTypeEncoding(), and overheadTypeFunctionSuffix().

◆ posTypeEncoding()

OverheadType mlir::sparse_tensor::posTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for position overhead storage.

Definition at line 71 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ primaryTypeEncoding()

PrimaryType mlir::sparse_tensor::primaryTypeEncoding ( Type  elemTp)

Converts a primary storage type to its internal type-encoding.

Definition at line 105 of file CodegenUtils.cpp.

References mlir::Type::isBF16(), mlir::Type::isF16(), mlir::Type::isF32(), mlir::Type::isF64(), mlir::Type::isInteger(), kBF16, kC32, kC64, kF16, kF32, kF64, kI16, kI32, kI64, and kI8.

Referenced by constantPrimaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( PrimaryType  pt)

Convert PrimaryType to its function-name suffix.

Definition at line 132 of file CodegenUtils.cpp.

References CASE, and MLIR_SPARSETENSOR_FOREVERY_V.

Referenced by primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( Type  elemTp)

Converts a primary storage type to its function-name suffix.

Definition at line 143 of file CodegenUtils.cpp.

References primaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels ( DialectRegistry registry)

◆ registerSparseTensorPipelines()

void mlir::sparse_tensor::registerSparseTensorPipelines ( )

Registers all pipelines for the sparse_tensor dialect.

At present, this includes only "sparsifier".

Definition at line 111 of file SparseTensorPipelines.cpp.

References buildSparsifier().

Referenced by mlir::registerAllPasses().

◆ registerTransformDialectExtension()

void mlir::sparse_tensor::registerTransformDialectExtension ( DialectRegistry registry)

◆ reshapeCvs()

void mlir::sparse_tensor::reshapeCvs ( OpBuilder builder,
Location  loc,
ArrayRef< ReassociationIndices reassociation,
ValueRange  srcSizes,
ValueRange  srcCvs,
ValueRange  dstSizes,
SmallVectorImpl< Value > &  dstCvs 
)

Reshape coordinates during a reshaping operation.

Definition at line 275 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().

◆ sizesFromSrc()

void mlir::sparse_tensor::sizesFromSrc ( OpBuilder builder,
SmallVectorImpl< Value > &  sizes,
Location  loc,
Value  src 
)

Populates given sizes array from dense tensor or sparse tensor constant.

Definition at line 414 of file CodegenUtils.cpp.

References mlir::linalg::createOrFoldDimOp(), mlir::sparse_tensor::SparseTensorType::getDimRank(), and getSparseTensorType().

Referenced by concatSizesFromInputs().

◆ storeAll()

void mlir::sparse_tensor::storeAll ( OpBuilder builder,
Location  loc,
Value  mem,
ValueRange  vs,
size_t  offsetIdx = 0,
Value  offsetVal = Value() 
)

Stores all the values of vs into the memref mem, which must have rank-1 and size greater-or-equal to vs.size().

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value before storing.

Definition at line 529 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and mlir::Value::getType().

◆ toDim()

Dimension mlir::sparse_tensor::toDim ( SparseTensorEncodingAttr  enc,
Level  l 
)

Convenience method to translate the given level to the corresponding dimension.

Requires: enc has a permuted dim2lvl map and 0 <= l < lvlRank.

Definition at line 895 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

Referenced by genSliceOffset(), and genSliceStride().

◆ toFieldKind()

SparseTensorFieldKind mlir::sparse_tensor::toFieldKind ( StorageSpecifierKind  kind)
inline

Definition at line 99 of file SparseTensorStorageLayout.h.

◆ toLvl()

Level mlir::sparse_tensor::toLvl ( SparseTensorEncodingAttr  enc,
Dimension  d 
)

Convenience method to translate the given dimension to the corresponding level.

Requires: enc has a permuted dim2lvl map and 0 <= d < dimRank.

Definition at line 904 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

◆ toMLIRString()

constexpr const char* mlir::sparse_tensor::toMLIRString ( LevelType  lt)
constexpr

Returns string representation of the given dimension level type.

Definition at line 202 of file Enums.h.

References Compressed, CompressedNo, CompressedNu, CompressedNuNo, Dense, LooseCompressed, LooseCompressedNo, LooseCompressedNu, LooseCompressedNuNo, Singleton, SingletonNo, SingletonNu, SingletonNuNo, TwoOutOfFour, and Undef.

Referenced by mlir::sparse_tensor::Merger::dumpBits().

◆ toSpecifierKind()

StorageSpecifierKind mlir::sparse_tensor::toSpecifierKind ( SparseTensorFieldKind  kind)
inline

Definition at line 94 of file SparseTensorStorageLayout.h.

◆ tryGetSparseTensorType()

std::optional<SparseTensorType> mlir::sparse_tensor::tryGetSparseTensorType ( Value  val)
inline

Definition at line 347 of file SparseTensorType.h.

References mlir::Value::getType().