MLIR  17.0.0git
Namespaces | Classes | Typedefs | Enumerations | Functions
mlir::sparse_tensor Namespace Reference

Namespaces

 detail
 

Classes

struct  SparseCompilerOptions
 Options for the "sparse-compiler" pipeline. More...
 
struct  Children
 Children subexpressions of tensor operations. More...
 
struct  TensorExp
 Tensor expression. Represents a MLIR expression in tensor index notation. More...
 
struct  LatPoint
 Lattice point. More...
 
class  Merger
 A class to handle all iteration lattice operations. More...
 
struct  Element
 An element of a sparse tensor in coordinate-scheme representation (i.e., a pair of indices and value). More...
 
struct  ElementLT
 Closure object for operator< on Element with a given rank. More...
 
class  SparseTensorCOO
 A memory-resident sparse tensor in coordinate-scheme representation (a collection of Elements). More...
 
class  SparseTensorReader
 This class abstracts over the information stored in file headers, as well as providing the buffers and methods for parsing those headers. More...
 
class  SparseTensorEnumeratorBase
 A (higher-order) function object for enumerating the elements of some SparseTensorStorage under a permutation. More...
 
class  SparseTensorStorageBase
 Abstract base class for SparseTensorStorage<P,I,V>. More...
 
class  SparseTensorEnumerator
 
class  SparseTensorStorage
 A memory-resident sparse tensor using a storage scheme based on per-dimension sparse/dense annotations. More...
 
class  SparseTensorNNZ
 Statistics regarding the number of nonzero subtensors in a source tensor, for direct sparse=>sparse conversion a la https://arxiv.org/abs/2001.02609. More...
 
class  CodegenEnv
 The code generation environment class aggregates a number of data structures that are needed during the code generation phase of sparsification. More...
 
class  LoopEmitter
 
class  StorageLayout
 Provides methods to access fields of a sparse tensor with the given encoding. More...
 
class  SparseTensorSpecifier
 
class  SparseTensorDescriptorImpl
 A helper class around an array of values that corresponding to a sparse tensor, provides a set of meaningful APIs to query and update a particular field in a consistent way. More...
 
class  SparseTensorDescriptor
 Uses ValueRange for immuatable descriptors;. More...
 
class  MutSparseTensorDescriptor
 Uses SmallVectorImpl<Value> & for mutable descriptors. More...
 
class  SparsificationAndBufferizationPass
 A pass that lowers tensor ops to memref ops, regardless of whether they are dense or sparse. More...
 

Typedefs

using index_type = uint64_t
 This type is used in the public API at all places where MLIR expects values with the built-in type "index". More...
 
using complex64 = std::complex< double >
 
using complex32 = std::complex< float >
 
template<typename V >
using ElementConsumer = const std::function< void(const std::vector< uint64_t > &, V)> &
 The type of callback functions which receive an element. More...
 

Enumerations

enum class  OverheadType : uint32_t {
  kIndex = 0 , kU64 = 1 , kU32 = 2 , kU16 = 3 ,
  kU8 = 4
}
 Encoding of overhead types (both pointer overhead and indices overhead), for "overloading" @newSparseTensor. More...
 
enum class  PrimaryType : uint32_t {
  kF64 = 1 , kF32 = 2 , kF16 = 3 , kBF16 = 4 ,
  kI64 = 5 , kI32 = 6 , kI16 = 7 , kI8 = 8 ,
  kC64 = 9 , kC32 = 10
}
 Encoding of the elemental type, for "overloading" @newSparseTensor. More...
 
enum class  Action : uint32_t {
  kEmpty = 0 , kFromCOO = 2 , kSparseToSparse = 3 , kEmptyCOO = 4 ,
  kToCOO = 5 , kToIterator = 6
}
 The actions performed by @newSparseTensor. More...
 
enum class  DimLevelType : uint8_t {
  Undef = 0 , Dense = 4 , Compressed = 8 , CompressedNu = 9 ,
  CompressedNo = 10 , CompressedNuNo = 11 , Singleton = 16 , SingletonNu = 17 ,
  SingletonNo = 18 , SingletonNuNo = 19
}
 This enum defines all the sparse representations supportable by the SparseTensor dialect. More...
 
enum class  LevelFormat : uint8_t { Dense = 4 , Compressed = 8 , Singleton = 16 }
 This enum defines all the storage formats supported by the sparse compiler, without the level properties. More...
 
enum  Kind {
  kTensor = 0 , kInvariant , kIndex , kIndex = 0 ,
  kAbsF , kAbsC , kAbsI , kCeilF ,
  kFloorF , kSqrtF , kSqrtC , kExpm1F ,
  kExpm1C , kLog1pF , kLog1pC , kSinF ,
  kSinC , kTanhF , kTanhC , kNegF ,
  kNegC , kNegI , kTruncF , kExtF ,
  kCastFS , kCastFU , kCastSF , kCastUF ,
  kCastS , kCastU , kCastIdx , kTruncI ,
  kCIm , kCRe , kBitCast , kBinaryBranch ,
  kUnary , kSelect , kMulF , kMulC ,
  kMulI , kDivF , kDivC , kDivS ,
  kDivU , kAddF , kAddC , kAddI ,
  kSubF , kSubC , kSubI , kAndI ,
  kOrI , kXorI , kShrS , kShrU ,
  kShlI , kBinary , kReduce
}
 Tensor expression kind. More...
 
enum class  EmitCInterface : bool { Off = false , On = true }
 Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall(). More...
 
enum class  SparseTensorFieldKind : uint32_t { StorageSpec = 0 , PtrMemRef = 1 , IdxMemRef = 2 , ValMemRef = 3 }
 
enum class  ExpArity { kNullary , kUnary , kBinary }
 

Functions

constexpr bool isFloatingPrimaryType (PrimaryType valTy)
 
constexpr bool isIntegralPrimaryType (PrimaryType valTy)
 
constexpr bool isRealPrimaryType (PrimaryType valTy)
 
constexpr bool isComplexPrimaryType (PrimaryType valTy)
 
std::string toMLIRString (DimLevelType dlt)
 Returns string representation of the given dimension level type. More...
 
constexpr bool isValidDLT (DimLevelType dlt)
 Check that the DimLevelType contains a valid (possibly undefined) value. More...
 
constexpr bool isUndefDLT (DimLevelType dlt)
 Check if the DimLevelType is the special undefined value. More...
 
constexpr bool isDenseDLT (DimLevelType dlt)
 Check if the DimLevelType is dense. More...
 
constexpr bool isCompressedDLT (DimLevelType dlt)
 Check if the DimLevelType is compressed (regardless of properties). More...
 
constexpr bool isSingletonDLT (DimLevelType dlt)
 Check if the DimLevelType is singleton (regardless of properties). More...
 
constexpr bool isOrderedDLT (DimLevelType dlt)
 Check if the DimLevelType is ordered (regardless of storage format). More...
 
constexpr bool isUniqueDLT (DimLevelType dlt)
 Check if the DimLevelType is unique (regardless of storage format). More...
 
constexpr std::optional< LevelFormatgetLevelFormat (DimLevelType dlt)
 Convert a DimLevelType to its corresponding LevelFormat. More...
 
constexpr std::optional< DimLevelTypegetDimLevelType (LevelFormat lf, bool ordered, bool unique)
 Convert a LevelFormat to its corresponding DimLevelType with the given properties. More...
 
template<typename T >
RankedTensorType getRankedTensorType (T t)
 Convenience method to abbreviate casting getType(). More...
 
template<typename T >
MemRefType getMemRefType (T t)
 Convenience method to abbreviate casting getType(). More...
 
SparseTensorEncodingAttr getSparseTensorEncoding (Type type)
 Convenience method to get a sparse encoding attribute from a type. More...
 
bool isUniqueCOOType (RankedTensorType tp)
 Returns true iff the given type is a type for a COO tensor with the last dimension level type being unique. More...
 
unsigned getCOOStart (SparseTensorEncodingAttr enc)
 Returns the starting dimension for a trailing COO region that spans across at least two dimensions. More...
 
DimLevelType getDimLevelType (SparseTensorEncodingAttr enc, uint64_t d)
 
DimLevelType getDimLevelType (RankedTensorType type, uint64_t d)
 
bool isDenseDim (RankedTensorType type, uint64_t d)
 Convenience function to test for dense dimension (0 <= d < rank). More...
 
bool isCompressedDim (RankedTensorType type, uint64_t d)
 Convenience function to test for compressed dimension (0 <= d < rank). More...
 
bool isSingletonDim (RankedTensorType type, uint64_t d)
 Convenience function to test for singleton dimension (0 <= d < rank). More...
 
bool isDenseDim (SparseTensorEncodingAttr enc, uint64_t d)
 Convenience function to test for dense dimension (0 <= d < rank). More...
 
bool isCompressedDim (SparseTensorEncodingAttr enc, uint64_t d)
 Convenience function to test for compressed dimension (0 <= d < rank). More...
 
bool isSingletonDim (SparseTensorEncodingAttr enc, uint64_t d)
 Convenience function to test for singleton dimension (0 <= d < rank). More...
 
bool isOrderedDim (RankedTensorType type, uint64_t d)
 Convenience function to test for ordered property in the given dimension (0 <= d < rank). More...
 
bool isUniqueDim (RankedTensorType type, uint64_t d)
 Convenience function to test for unique property in the given dimension (0 <= d < rank). More...
 
uint64_t toOrigDim (SparseTensorEncodingAttr enc, uint64_t d)
 
uint64_t toStoredDim (SparseTensorEncodingAttr enc, uint64_t d)
 
uint64_t toOrigDim (RankedTensorType type, uint64_t d)
 Convenience method to translate the given stored dimension to the original dimension (0 <= d < rank). More...
 
uint64_t toStoredDim (RankedTensorType type, uint64_t d)
 Convenience method to translate the given original dimension to the stored dimension (0 <= d < rank). More...
 
void buildSparseCompiler (OpPassManager &pm, const SparseCompilerOptions &options)
 Adds the "sparse-compiler" pipeline to the OpPassManager. More...
 
void registerSparseTensorPipelines ()
 Registers all pipelines for the sparse_tensor dialect. More...
 
void registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)
 
template<typename V >
void writeExtFROSTT (const SparseTensorCOO< V > &coo, const char *filename)
 Writes the sparse tensor to filename in extended FROSTT format. More...
 
OverheadType overheadTypeEncoding (unsigned width)
 Converts an overhead storage bitwidth to its internal type-encoding. More...
 
OverheadType overheadTypeEncoding (Type tp)
 Converts an overhead storage type to its internal type-encoding. More...
 
Type getOverheadType (Builder &builder, OverheadType ot)
 Converts the internal type-encoding for overhead storage to an mlir::Type. More...
 
OverheadType pointerOverheadTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for pointer overhead storage. More...
 
OverheadType indexOverheadTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for index overhead storage. More...
 
Type getPointerOverheadType (Builder &builder, SparseTensorEncodingAttr enc)
 Returns the mlir::Type for pointer overhead storage. More...
 
Type getIndexOverheadType (Builder &builder, SparseTensorEncodingAttr enc)
 Returns the mlir::Type for index overhead storage. More...
 
StringRef overheadTypeFunctionSuffix (OverheadType ot)
 Convert OverheadType to its function-name suffix. More...
 
StringRef overheadTypeFunctionSuffix (Type overheadTp)
 Converts an overhead storage type to its function-name suffix. More...
 
PrimaryType primaryTypeEncoding (Type elemTp)
 Converts a primary storage type to its internal type-encoding. More...
 
StringRef primaryTypeFunctionSuffix (PrimaryType pt)
 Convert PrimaryType to its function-name suffix. More...
 
StringRef primaryTypeFunctionSuffix (Type elemTp)
 Converts a primary storage type to its function-name suffix. More...
 
Attribute getOneAttr (Builder &builder, Type tp)
 Generates a 1-valued attribute of the given type. More...
 
Value genIsNonzero (OpBuilder &builder, Location loc, Value v)
 Generates the comparison v != 0 where v is of numeric type. More...
 
void genReshapeDstShape (Location loc, PatternRewriter &rewriter, SmallVectorImpl< Value > &dstShape, ArrayRef< Value > srcShape, ArrayRef< int64_t > staticDstShape, ArrayRef< ReassociationIndices > reassociation)
 Computes the shape of destination tensor of a reshape operator. More...
 
void translateIndicesArray (OpBuilder &builder, Location loc, ArrayRef< ReassociationIndices > reassociation, ValueRange srcIndices, ArrayRef< Value > srcShape, ArrayRef< Value > dstShape, SmallVectorImpl< Value > &dstIndices)
 Translate indices during a reshaping operation. More...
 
FlatSymbolRefAttr getFunc (ModuleOp module, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Returns a function reference (first hit also inserts into module). More...
 
func::CallOp createFuncCall (OpBuilder &builder, Location loc, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Creates a CallOp to the function reference returned by getFunc() in the builder's module. More...
 
Type getOpaquePointerType (OpBuilder &builder)
 Returns the equivalent of void* for opaque arguments to the execution engine. More...
 
Value genAlloca (OpBuilder &builder, Location loc, Value sz, Type tp)
 Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>). More...
 
Value genAlloca (OpBuilder &builder, Location loc, unsigned sz, Type tp, bool staticShape=false)
 Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true). More...
 
Value genAllocaScalar (OpBuilder &builder, Location loc, Type tp)
 Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>. More...
 
Value allocaBuffer (OpBuilder &builder, Location loc, ValueRange values)
 Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer). More...
 
Value allocDenseTensor (OpBuilder &builder, Location loc, RankedTensorType tensorTp, ValueRange sizes)
 Generates code to allocate a buffer of the given type, and zero initialize it. More...
 
void deallocDenseTensor (OpBuilder &builder, Location loc, Value buffer)
 Generates code to deallocate a dense buffer. More...
 
Value genValueForDense (OpBuilder &builder, Location loc, Value tensor, ValueRange ivs)
 Generates the code to read the value from tensor[ivs]. More...
 
void genDenseTensorOrSparseConstantIterLoop (OpBuilder &builder, Location loc, Value src, unsigned rank, function_ref< void(OpBuilder &, Location, Value, ValueRange)> bodyBuilder)
 Generates the loop structure to iterate over a dense tensor or a sparse tensor constant to support the lowering of dense-to-sparse convert operator. More...
 
void sizesFromSrc (OpBuilder &builder, SmallVectorImpl< Value > &sizes, Location loc, Value src)
 Populates given sizes array from dense tensor or sparse tensor constant. More...
 
MemRefType get1DMemRefType (Type etp, bool withLayout)
 Generates a 1D MemRefType with a dynamic size. More...
 
OperationgetTop (Operation *op)
 Scans to top of generated loop. More...
 
void foreachInSparseConstant (Location loc, RewriterBase &rewriter, SparseElementsAttr attr, function_ref< void(ArrayRef< Value >, Value)> callback)
 Iterate over a sparse constant, generates constantOp for value and indices. More...
 
void storeIndices (OpBuilder &builder, Location loc, unsigned rank, Value ind, ValueRange ivs, unsigned offsetDim=0, Value offset=Value())
 Converts the vector indices and store it into the memory pointed by ind, apply (optional) offset on offsetDim. More...
 
Value reshapeValuesToLevels (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc, const SmallVectorImpl< Value > &dimSizes, Value valuesBuffer, Value idxBuffer)
 Reshapes the linear values buffer for an annotated all dense sparse tensor to match the shape of the corresponding dense tensor to support direct access of the buffer through indices. More...
 
Value constantZero (OpBuilder &builder, Location loc, Type tp)
 Generates a 0-valued constant of the given type. More...
 
Value constantOne (OpBuilder &builder, Location loc, Type tp)
 Generates a 1-valued constant of the given type. More...
 
Value constantIndex (OpBuilder &builder, Location loc, int64_t i)
 Generates a constant of index type. More...
 
Value constantI32 (OpBuilder &builder, Location loc, int32_t i)
 Generates a constant of i32 type. More...
 
Value constantI16 (OpBuilder &builder, Location loc, int16_t i)
 Generates a constant of i16 type. More...
 
Value constantI8 (OpBuilder &builder, Location loc, int8_t i)
 Generates a constant of i8 type. More...
 
Value constantI1 (OpBuilder &builder, Location loc, bool b)
 Generates a constant of i1 type. More...
 
Value constantAction (OpBuilder &builder, Location loc, Action action)
 Generates a constant of the given Action. More...
 
Value constantOverheadTypeEncoding (OpBuilder &builder, Location loc, unsigned width)
 Generates a constant of the internal type-encoding for overhead storage. More...
 
Value constantPointerTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for pointer overhead storage. More...
 
Value constantIndexTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for index overhead storage. More...
 
Value constantPrimaryTypeEncoding (OpBuilder &builder, Location loc, Type elemTp)
 Generates a constant of the internal type-encoding for primary storage. More...
 
Value constantDimLevelTypeEncoding (OpBuilder &builder, Location loc, DimLevelType dlt)
 Generates a constant of the internal dimension level type encoding. More...
 
bool isZeroRankedTensorOrScalar (Type type)
 
Value genToPointers (OpBuilder &builder, Location loc, Value tensor, uint64_t d)
 Infers the result type and generates ToPointersOp. More...
 
Value genToIndices (OpBuilder &builder, Location loc, Value tensor, uint64_t d, uint64_t cooStart)
 Infers the result type and generates ToIndicesOp. More...
 
Value genToValues (OpBuilder &builder, Location loc, Value tensor)
 Infers the result type and generates ToValuesOp. More...
 
Value genValMemSize (OpBuilder &builder, Location loc, Value tensor)
 Generates code to retrieve the values size for the sparse tensor. More...
 
void foreachFieldInSparseTensor (SparseTensorEncodingAttr, llvm::function_ref< bool(unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>)
 For each field that will be allocated for the given sparse tensor encoding, calls the callback with the corresponding field index, field kind, dimension (for sparse tensor level memrefs) and dimlevelType. More...
 
void foreachFieldAndTypeInSparseTensor (RankedTensorType, llvm::function_ref< bool(Type, unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>)
 Same as above, except that it also builds the Type for the corresponding field. More...
 
unsigned getNumFieldsFromEncoding (SparseTensorEncodingAttr enc)
 Gets the total number of fields for the given sparse tensor encoding. More...
 
unsigned getNumDataFieldsFromEncoding (SparseTensorEncodingAttr enc)
 Gets the total number of data fields (index arrays, pointer arrays, and a value array) for the given sparse tensor encoding. More...
 
StorageSpecifierKind toSpecifierKind (SparseTensorFieldKind kind)
 
SparseTensorFieldKind toFieldKind (StorageSpecifierKind kind)
 
UnrealizedConversionCastOp getTuple (Value tensor)
 Returns the "tuple" value of the adapted tensor. More...
 
Value genTuple (OpBuilder &builder, Location loc, Type tp, ValueRange values)
 Packs the given values as a "tuple" value. More...
 
Value genTuple (OpBuilder &builder, Location loc, SparseTensorDescriptor desc)
 
SparseTensorDescriptor getDescriptorFromTensorTuple (Value tensor)
 
MutSparseTensorDescriptor getMutDescriptorFromTensorTuple (Value tensor, SmallVectorImpl< Value > &fields)
 
static bool containsSparseTensor (TypeRange types)
 Return true if one of the given types is a sparse tensor type. More...
 
static ExpArity getExpArity (Kind k)
 
static const char * kindToOpSymbol (Kind kind)
 
static bool isAdmissibleBranchExp (Operation *op, Block *block, Value v)
 Ensures that sparse compiler can generate code for expression. More...
 
static bool isAdmissibleBranch (Operation *op, Region &region)
 Ensures that sparse compiler can generate code for branch. More...
 
static Value insertYieldOp (RewriterBase &rewriter, Location loc, Region &region, ValueRange vals)
 
static Value buildUnaryPresent (RewriterBase &rewriter, Location loc, Operation *op, Value v0)
 
static Value buildBinaryOverlap (RewriterBase &rewriter, Location loc, Operation *op, Value v0, Value v1)
 

Typedef Documentation

◆ complex32

using mlir::sparse_tensor::complex32 = typedef std::complex<float>

Definition at line 78 of file Enums.h.

◆ complex64

using mlir::sparse_tensor::complex64 = typedef std::complex<double>

Definition at line 77 of file Enums.h.

◆ ElementConsumer

template<typename V >
using mlir::sparse_tensor::ElementConsumer = typedef const std::function<void(const std::vector<uint64_t> &, V)> &

The type of callback functions which receive an element.

We avoid packaging the coordinates and value together as an Element object because this helps keep code somewhat cleaner.

Definition at line 75 of file COO.h.

◆ index_type

using mlir::sparse_tensor::index_type = typedef uint64_t

This type is used in the public API at all places where MLIR expects values with the built-in type "index".

For now, we simply assume that type is 64-bit, but targets with different "index" bit widths should link with an alternatively built runtime support library.

Definition at line 46 of file Enums.h.

Enumeration Type Documentation

◆ Action

enum mlir::sparse_tensor::Action : uint32_t
strong

The actions performed by @newSparseTensor.

Enumerator
kEmpty 
kFromCOO 
kSparseToSparse 
kEmptyCOO 
kToCOO 
kToIterator 

Definition at line 124 of file Enums.h.

◆ DimLevelType

enum mlir::sparse_tensor::DimLevelType : uint8_t
strong

This enum defines all the sparse representations supportable by the SparseTensor dialect.

We use a lightweight encoding to encode both the "format" per se (dense, compressed, singleton) as well as the "properties" (ordered, unique). The encoding is chosen for performance of the runtime library, and thus may change in future versions; consequently, client code should use the predicate functions defined below, rather than relying on knowledge about the particular binary encoding.

The Undef "format" is a special value used internally for cases where we need to store an undefined or indeterminate DimLevelType. It should not be used externally, since it does not indicate an actual/representable format.

Enumerator
Undef 
Dense 
Compressed 
CompressedNu 
CompressedNo 
CompressedNuNo 
Singleton 
SingletonNu 
SingletonNo 
SingletonNuNo 

Definition at line 148 of file Enums.h.

◆ EmitCInterface

Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall().

Enumerator
Off 
On 

Definition at line 35 of file CodegenUtils.h.

◆ ExpArity

Enumerator
kNullary 
kUnary 
kBinary 

Definition at line 22 of file Merger.cpp.

◆ Kind

Tensor expression kind.

Enumerator
kTensor 
kInvariant 
kIndex 
kIndex 
kAbsF 
kAbsC 
kAbsI 
kCeilF 
kFloorF 
kSqrtF 
kSqrtC 
kExpm1F 
kExpm1C 
kLog1pF 
kLog1pC 
kSinF 
kSinC 
kTanhF 
kTanhC 
kNegF 
kNegC 
kNegI 
kTruncF 
kExtF 
kCastFS 
kCastFU 
kCastSF 
kCastUF 
kCastS 
kCastU 
kCastIdx 
kTruncI 
kCIm 
kCRe 
kBitCast 
kBinaryBranch 
kUnary 
kSelect 
kMulF 
kMulC 
kMulI 
kDivF 
kDivC 
kDivS 
kDivU 
kAddF 
kAddC 
kAddI 
kSubF 
kSubC 
kSubI 
kAndI 
kOrI 
kXorI 
kShrS 
kShrU 
kShlI 
kBinary 
kReduce 

Definition at line 26 of file Merger.h.

◆ LevelFormat

enum mlir::sparse_tensor::LevelFormat : uint8_t
strong

This enum defines all the storage formats supported by the sparse compiler, without the level properties.

Enumerator
Dense 
Compressed 
Singleton 

Definition at line 163 of file Enums.h.

◆ OverheadType

enum mlir::sparse_tensor::OverheadType : uint32_t
strong

Encoding of overhead types (both pointer overhead and indices overhead), for "overloading" @newSparseTensor.

Enumerator
kIndex 
kU64 
kU32 
kU16 
kU8 

Definition at line 50 of file Enums.h.

◆ PrimaryType

enum mlir::sparse_tensor::PrimaryType : uint32_t
strong

Encoding of the elemental type, for "overloading" @newSparseTensor.

Enumerator
kF64 
kF32 
kF16 
kBF16 
kI64 
kI32 
kI16 
kI8 
kC64 
kC32 

Definition at line 81 of file Enums.h.

◆ SparseTensorFieldKind

Enumerator
StorageSpec 
PtrMemRef 
IdxMemRef 
ValMemRef 

Definition at line 54 of file SparseTensorStorageLayout.h.

Function Documentation

◆ allocaBuffer()

Value mlir::sparse_tensor::allocaBuffer ( OpBuilder builder,
Location  loc,
ValueRange  values 
)

Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer).

Definition at line 395 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and genAlloca().

◆ allocDenseTensor()

Value mlir::sparse_tensor::allocDenseTensor ( OpBuilder builder,
Location  loc,
RankedTensorType  tensorTp,
ValueRange  sizes 
)

Generates code to allocate a buffer of the given type, and zero initialize it.

If the buffer type has any dynamic sizes, then the sizes parameter should be as filled by sizesFromPtr(); that way we can reuse the genDimSizeCall() results generated by sizesFromPtr().

Definition at line 407 of file CodegenUtils.cpp.

References constantZero(), and mlir::OpBuilder::create().

◆ buildBinaryOverlap()

static Value mlir::sparse_tensor::buildBinaryOverlap ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0,
Value  v1 
)
static

Definition at line 1235 of file Merger.cpp.

References mlir::Region::empty(), and insertYieldOp().

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ buildSparseCompiler()

void mlir::sparse_tensor::buildSparseCompiler ( OpPassManager pm,
const SparseCompilerOptions options 
)

◆ buildUnaryPresent()

static Value mlir::sparse_tensor::buildUnaryPresent ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0 
)
static

Definition at line 1221 of file Merger.cpp.

References mlir::Region::empty(), and insertYieldOp().

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ constantAction()

Value mlir::sparse_tensor::constantAction ( OpBuilder builder,
Location  loc,
Action  action 
)
inline

Generates a constant of the given Action.

Definition at line 299 of file CodegenUtils.h.

References constantI32().

◆ constantDimLevelTypeEncoding()

Value mlir::sparse_tensor::constantDimLevelTypeEncoding ( OpBuilder builder,
Location  loc,
DimLevelType  dlt 
)
inline

Generates a constant of the internal dimension level type encoding.

Definition at line 332 of file CodegenUtils.h.

References constantI8().

◆ constantI1()

Value mlir::sparse_tensor::constantI1 ( OpBuilder builder,
Location  loc,
bool  b 
)
inline

Generates a constant of i1 type.

Definition at line 294 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by createEqCompare(), createLessThanCompare(), genIf(), and genInsertionStore().

◆ constantI16()

Value mlir::sparse_tensor::constantI16 ( OpBuilder builder,
Location  loc,
int16_t  i 
)
inline

Generates a constant of i16 type.

Definition at line 284 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantI32()

Value mlir::sparse_tensor::constantI32 ( OpBuilder builder,
Location  loc,
int32_t  i 
)
inline

Generates a constant of i32 type.

Definition at line 279 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantAction(), constantOverheadTypeEncoding(), and constantPrimaryTypeEncoding().

◆ constantI8()

Value mlir::sparse_tensor::constantI8 ( OpBuilder builder,
Location  loc,
int8_t  i 
)
inline

Generates a constant of i8 type.

Definition at line 289 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantDimLevelTypeEncoding().

◆ constantIndex()

Value mlir::sparse_tensor::constantIndex ( OpBuilder builder,
Location  loc,
int64_t  i 
)
inline

◆ constantIndexTypeEncoding()

Value mlir::sparse_tensor::constantIndexTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for index overhead storage.

Definition at line 319 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantOne()

Value mlir::sparse_tensor::constantOne ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 1-valued constant of the given type.

This supports all the same types as constantZero.

Definition at line 263 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Builder::getArrayAttr(), getOneAttr(), and mlir::Builder::getZeroAttr().

◆ constantOverheadTypeEncoding()

Value mlir::sparse_tensor::constantOverheadTypeEncoding ( OpBuilder builder,
Location  loc,
unsigned  width 
)
inline

Generates a constant of the internal type-encoding for overhead storage.

Definition at line 304 of file CodegenUtils.h.

References constantI32(), and overheadTypeEncoding().

Referenced by constantIndexTypeEncoding(), and constantPointerTypeEncoding().

◆ constantPointerTypeEncoding()

Value mlir::sparse_tensor::constantPointerTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for pointer overhead storage.

Definition at line 312 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantPrimaryTypeEncoding()

Value mlir::sparse_tensor::constantPrimaryTypeEncoding ( OpBuilder builder,
Location  loc,
Type  elemTp 
)
inline

Generates a constant of the internal type-encoding for primary storage.

Definition at line 325 of file CodegenUtils.h.

References constantI32(), and primaryTypeEncoding().

◆ constantZero()

Value mlir::sparse_tensor::constantZero ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 0-valued constant of the given type.

In addition to the scalar types (ComplexType, `FloatType, IndexType, IntegerType), this also works for RankedTensorType and VectorType (for which it generates a constant DenseElementsAttr of zeros).

Definition at line 252 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Builder::getArrayAttr(), and mlir::Builder::getZeroAttr().

Referenced by allocDenseTensor(), genBuffers(), genInsertionLoad(), and genIsNonzero().

◆ containsSparseTensor()

static bool mlir::sparse_tensor::containsSparseTensor ( TypeRange  types)
static

Return true if one of the given types is a sparse tensor type.

Definition at line 32 of file SparsificationAndBufferizationPass.cpp.

References getSparseTensorEncoding().

Referenced by mlir::sparse_tensor::SparsificationAndBufferizationPass::runDenseBufferization().

◆ createFuncCall()

func::CallOp mlir::sparse_tensor::createFuncCall ( OpBuilder builder,
Location  loc,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Creates a CallOp to the function reference returned by getFunc() in the builder's module.

Definition at line 362 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::OpBuilder::getBlock(), getFunc(), mlir::Operation::getParentOfType(), and mlir::Block::getParentOp().

◆ deallocDenseTensor()

void mlir::sparse_tensor::deallocDenseTensor ( OpBuilder builder,
Location  loc,
Value  buffer 
)

Generates code to deallocate a dense buffer.

Definition at line 424 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

◆ foreachFieldAndTypeInSparseTensor()

void mlir::sparse_tensor::foreachFieldAndTypeInSparseTensor ( RankedTensorType  rType,
llvm::function_ref< bool(Type, unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>  callback 
)

Same as above, except that it also builds the Type for the corresponding field.

Definition at line 181 of file SparseTensorStorageLayout.cpp.

References foreachFieldInSparseTensor(), getSparseTensorEncoding(), IdxMemRef, PtrMemRef, StorageSpec, and ValMemRef.

Referenced by convertSparseTensorType().

◆ foreachFieldInSparseTensor()

void mlir::sparse_tensor::foreachFieldInSparseTensor ( SparseTensorEncodingAttr  enc,
llvm::function_ref< bool(unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>  callback 
)

For each field that will be allocated for the given sparse tensor encoding, calls the callback with the corresponding field index, field kind, dimension (for sparse tensor level memrefs) and dimlevelType.

The field index always starts with zero and increments by one between two callback invocations. Ideally, all other methods should rely on this function to query a sparse tensor fields instead of relying on ad-hoc index computation.

Definition at line 138 of file SparseTensorStorageLayout.cpp.

References getCOOStart(), getDimLevelType(), IdxMemRef, isCompressedDLT(), isDenseDLT(), isSingletonDLT(), kDataFieldStartingIdx, PtrMemRef, RETURN_ON_FALSE, StorageSpec, Undef, and ValMemRef.

Referenced by foreachFieldAndTypeInSparseTensor(), mlir::sparse_tensor::StorageLayout::foreachFieldInSparseTensor(), getNumDataFieldsFromEncoding(), and getNumFieldsFromEncoding().

◆ foreachInSparseConstant()

void mlir::sparse_tensor::foreachInSparseConstant ( Location  loc,
RewriterBase rewriter,
SparseElementsAttr  attr,
function_ref< void(ArrayRef< Value >, Value)>  callback 
)

Iterate over a sparse constant, generates constantOp for value and indices.

E.g., sparse<[ [0], [28], [31] ], [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] > => c1 = arith.constant 0 v1 = complex.constant (5.13, 2.0) callback({c1}, v1)

c2 = arith.constant 28 v2 = complex.constant (3.0, 4.0) callback({c2}, v2)

c3 = arith.constant 31 v3 = complex.constant (5.0, 6.0) callback({c3}, v3)

Definition at line 501 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::DenseElementsAttr::getValues(), and mlir::DenseElementsAttr::size().

Referenced by genForeachOnSparseConstant().

◆ genAlloca() [1/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
unsigned  sz,
Type  tp,
bool  staticShape = false 
)

Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true).

Definition at line 375 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and genAlloca().

◆ genAlloca() [2/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
Value  sz,
Type  tp 
)

Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>).

Definition at line 384 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

Referenced by allocaBuffer(), and genAlloca().

◆ genAllocaScalar()

Value mlir::sparse_tensor::genAllocaScalar ( OpBuilder builder,
Location  loc,
Type  tp 
)

Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>.

Definition at line 390 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

◆ genDenseTensorOrSparseConstantIterLoop()

void mlir::sparse_tensor::genDenseTensorOrSparseConstantIterLoop ( OpBuilder builder,
Location  loc,
Value  src,
unsigned  rank,
function_ref< void(OpBuilder &, Location, Value, ValueRange)>  bodyBuilder 
)

Generates the loop structure to iterate over a dense tensor or a sparse tensor constant to support the lowering of dense-to-sparse convert operator.

Definition at line 441 of file CodegenUtils.cpp.

References mlir::scf::buildLoopNest(), constantIndex(), mlir::linalg::createOrFoldDimOp(), genIndexAndValueForDense(), genIndexAndValueForSparse(), and genSplitSparseConstant().

◆ genIsNonzero()

Value mlir::sparse_tensor::genIsNonzero ( OpBuilder builder,
mlir::Location  loc,
Value  v 
)

Generates the comparison v != 0 where v is of numeric type.

For floating types, we use the "unordered" comparator (i.e., returns true if v is NaN).

Definition at line 222 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Value::getType(), mlir::Type::isa(), and mlir::Type::isIntOrIndex().

Referenced by genValueForDense().

◆ genReshapeDstShape()

void mlir::sparse_tensor::genReshapeDstShape ( Location  loc,
PatternRewriter rewriter,
SmallVectorImpl< Value > &  dstShape,
ArrayRef< Value srcShape,
ArrayRef< int64_t >  staticDstShape,
ArrayRef< ReassociationIndices reassociation 
)

Computes the shape of destination tensor of a reshape operator.

This is only used when operands have dynamic shape. The shape of the destination is stored into dstShape.

Definition at line 237 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().

◆ genToIndices()

Value mlir::sparse_tensor::genToIndices ( OpBuilder builder,
Location  loc,
Value  tensor,
uint64_t  d,
uint64_t  cooStart 
)

Infers the result type and generates ToIndicesOp.

If the dim is within a COO region, the result type is a memref with unknown stride and offset. Otherwise, the result type is a memref without any specified layout.

Definition at line 576 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), mlir::Builder::getIndexAttr(), getIndexOverheadType(), getRankedTensorType(), and getSparseTensorEncoding().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ genToPointers()

Value mlir::sparse_tensor::genToPointers ( OpBuilder builder,
Location  loc,
Value  tensor,
uint64_t  d 
)

◆ genToValues()

Value mlir::sparse_tensor::genToValues ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Infers the result type and generates ToValuesOp.

Definition at line 586 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), get1DMemRefType(), and getRankedTensorType().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ genTuple() [1/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
SparseTensorDescriptor  desc 
)
inline

◆ genTuple() [2/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
Type  tp,
ValueRange  values 
)
inline

Packs the given values as a "tuple" value.

Definition at line 386 of file SparseTensorStorageLayout.h.

References mlir::OpBuilder::create().

Referenced by genTuple(), and mlir::SparseTensorTypeToBufferConverter::SparseTensorTypeToBufferConverter().

◆ genValMemSize()

Value mlir::sparse_tensor::genValMemSize ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Generates code to retrieve the values size for the sparse tensor.

Definition at line 594 of file CodegenUtils.cpp.

References getDescriptorFromTensorTuple(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getValMemSize().

◆ genValueForDense()

Value mlir::sparse_tensor::genValueForDense ( OpBuilder builder,
Location  loc,
Value  tensor,
ValueRange  ivs 
)

Generates the code to read the value from tensor[ivs].

The generated code looks like the following and the insertion point after this routine is inside the if-then branch behind the assignment to ind. if (tensor[ivs] != 0) insert_point

Definition at line 429 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), genIsNonzero(), and mlir::OpBuilder::setInsertionPointToStart().

Referenced by genIndexAndValueForDense().

◆ get1DMemRefType()

MemRefType mlir::sparse_tensor::get1DMemRefType ( Type  etp,
bool  withLayout 
)
inline

Generates a 1D MemRefType with a dynamic size.

When withLayout is set, the returned memref has a layout has unknown strides and offsets. Otherwise, a memref with a standard unit stride zero offset layout is returned.

Definition at line 190 of file CodegenUtils.h.

References mlir::Type::getContext().

Referenced by genToIndices(), genToPointers(), and genToValues().

◆ getCOOStart()

unsigned mlir::sparse_tensor::getCOOStart ( SparseTensorEncodingAttr  enc)

Returns the starting dimension for a trailing COO region that spans across at least two dimensions.

If no such COO region is found, returns the rank of the tensor.

Definition at line 431 of file SparseTensorDialect.cpp.

References isCOOType().

Referenced by foreachFieldInSparseTensor(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getAOSMemRef(), mlir::sparse_tensor::StorageLayout::getFieldIndexAndStride(), mlir::sparse_tensor::SparseTensorDescriptor::getIdxMemRefOrView(), and mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ getDescriptorFromTensorTuple()

SparseTensorDescriptor mlir::sparse_tensor::getDescriptorFromTensorTuple ( Value  tensor)
inline

Definition at line 397 of file SparseTensorStorageLayout.h.

References getTuple().

Referenced by genValMemSize().

◆ getDimLevelType() [1/3]

constexpr std::optional<DimLevelType> mlir::sparse_tensor::getDimLevelType ( LevelFormat  lf,
bool  ordered,
bool  unique 
)
constexpr

Convert a LevelFormat to its corresponding DimLevelType with the given properties.

Returns std::nullopt when the properties are not applicable for the input level format. TODO: factor out a new LevelProperties type so we can add new properties without changing this function's signature

Definition at line 257 of file Enums.h.

References isValidDLT().

Referenced by computeIterationGraph(), findSparseAnnotations(), foreachFieldInSparseTensor(), genConstantDenseAddressFromLevel(), getDimLevelType(), getNormalizedEncodingForSpecifier(), getNumCompoundAffineOnSparseDims(), isCompressedDim(), isCOOType(), isDenseDim(), isOrderedDim(), isSingletonDim(), isUniqueDim(), mlirSparseTensorEncodingAttrGetDimLevelType(), mlirSparseTensorEncodingGetNumDimLevelTypes(), and translateBitsToTidDimPairs().

◆ getDimLevelType() [2/3]

DimLevelType mlir::sparse_tensor::getDimLevelType ( RankedTensorType  type,
uint64_t  d 
)
inline

Definition at line 84 of file SparseTensor.h.

References getDimLevelType(), and getSparseTensorEncoding().

◆ getDimLevelType() [3/3]

DimLevelType mlir::sparse_tensor::getDimLevelType ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)
inline

Definition at line 75 of file SparseTensor.h.

References Dense.

◆ getExpArity()

static ExpArity mlir::sparse_tensor::getExpArity ( Kind  k)
static

◆ getFunc()

FlatSymbolRefAttr mlir::sparse_tensor::getFunc ( ModuleOp  module,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Returns a function reference (first hit also inserts into module).

Sets the "_emit_c_interface" on the function declaration when requested, so that LLVM lowering generates a wrapper function that takes care of ABI complications with passing in and returning MemRefs to C functions.

Definition at line 342 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::ValueRange::getTypes(), and mlir::Operation::setAttr().

Referenced by createFuncCall().

◆ getIndexOverheadType()

Type mlir::sparse_tensor::getIndexOverheadType ( Builder builder,
SparseTensorEncodingAttr  enc 
)

Returns the mlir::Type for index overhead storage.

Definition at line 135 of file CodegenUtils.cpp.

References getOverheadType(), and indexOverheadTypeEncoding().

Referenced by genToIndices().

◆ getLevelFormat()

constexpr std::optional<LevelFormat> mlir::sparse_tensor::getLevelFormat ( DimLevelType  dlt)
constexpr

Convert a DimLevelType to its corresponding LevelFormat.

Returns std::nullopt when input dlt is Undef.

Definition at line 245 of file Enums.h.

References Undef.

Referenced by getNormalizedEncodingForSpecifier().

◆ getMemRefType()

template<typename T >
MemRefType mlir::sparse_tensor::getMemRefType ( t)
inline

Convenience method to abbreviate casting getType().

Definition at line 49 of file SparseTensor.h.

Referenced by getMangledSortHelperFuncName(), isMatchingWidth(), matchAndRewriteSortOp(), and reshapeValuesToLevels().

◆ getMutDescriptorFromTensorTuple()

MutSparseTensorDescriptor mlir::sparse_tensor::getMutDescriptorFromTensorTuple ( Value  tensor,
SmallVectorImpl< Value > &  fields 
)
inline

Definition at line 403 of file SparseTensorStorageLayout.h.

References getTuple().

◆ getNumDataFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumDataFieldsFromEncoding ( SparseTensorEncodingAttr  enc)

Gets the total number of data fields (index arrays, pointer arrays, and a value array) for the given sparse tensor encoding.

Definition at line 232 of file SparseTensorStorageLayout.cpp.

References foreachFieldInSparseTensor(), getNumFieldsFromEncoding(), and kDataFieldStartingIdx.

◆ getNumFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumFieldsFromEncoding ( SparseTensorEncodingAttr  enc)

◆ getOneAttr()

mlir::Attribute mlir::sparse_tensor::getOneAttr ( Builder builder,
Type  tp 
)

Generates a 1-valued attribute of the given type.

This supports all the same types as getZeroAttr; however, unlike getZeroAttr, for unsupported types we raise llvm_unreachable rather than returning a null attribute.

Definition at line 207 of file CodegenUtils.cpp.

References mlir::Type::cast(), mlir::Type::dyn_cast(), mlir::DenseElementsAttr::get(), mlir::Builder::getFloatAttr(), mlir::Builder::getIndexAttr(), mlir::Builder::getIntegerAttr(), and mlir::Type::isa().

Referenced by constantOne().

◆ getOpaquePointerType()

Type mlir::sparse_tensor::getOpaquePointerType ( OpBuilder builder)

Returns the equivalent of void* for opaque arguments to the execution engine.

Definition at line 371 of file CodegenUtils.cpp.

References mlir::Builder::getI8Type().

◆ getOverheadType()

Type mlir::sparse_tensor::getOverheadType ( Builder builder,
OverheadType  ot 
)

Converts the internal type-encoding for overhead storage to an mlir::Type.

Definition at line 104 of file CodegenUtils.cpp.

References mlir::Builder::getIndexType(), mlir::Builder::getIntegerType(), kIndex, kU16, kU32, kU64, and kU8.

Referenced by getIndexOverheadType(), and getPointerOverheadType().

◆ getPointerOverheadType()

Type mlir::sparse_tensor::getPointerOverheadType ( Builder builder,
SparseTensorEncodingAttr  enc 
)

Returns the mlir::Type for pointer overhead storage.

Definition at line 130 of file CodegenUtils.cpp.

References getOverheadType(), and pointerOverheadTypeEncoding().

Referenced by genToPointers().

◆ getRankedTensorType()

template<typename T >
RankedTensorType mlir::sparse_tensor::getRankedTensorType ( t)
inline

Convenience method to abbreviate casting getType().

Definition at line 43 of file SparseTensor.h.

Referenced by genToIndices(), genToPointers(), genToValues(), getTypeRank(), and mlir::sparse_tensor::LoopEmitter::initialize().

◆ getSparseTensorEncoding()

SparseTensorEncodingAttr mlir::sparse_tensor::getSparseTensorEncoding ( Type  type)

◆ getTop()

Operation * mlir::sparse_tensor::getTop ( Operation op)

Scans to top of generated loop.

Definition at line 491 of file CodegenUtils.cpp.

References mlir::Operation::getParentOp().

◆ getTuple()

UnrealizedConversionCastOp mlir::sparse_tensor::getTuple ( Value  tensor)
inline

Returns the "tuple" value of the adapted tensor.

Definition at line 381 of file SparseTensorStorageLayout.h.

References mlir::Value::getDefiningOp().

Referenced by getDescriptorFromTensorTuple(), and getMutDescriptorFromTensorTuple().

◆ indexOverheadTypeEncoding()

OverheadType mlir::sparse_tensor::indexOverheadTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for index overhead storage.

Definition at line 126 of file CodegenUtils.cpp.

References overheadTypeEncoding().

Referenced by getIndexOverheadType().

◆ insertYieldOp()

static Value mlir::sparse_tensor::insertYieldOp ( RewriterBase rewriter,
Location  loc,
Region region,
ValueRange  vals 
)
static

◆ isAdmissibleBranch()

static bool mlir::sparse_tensor::isAdmissibleBranch ( Operation op,
Region region 
)
static

Ensures that sparse compiler can generate code for branch.

Definition at line 1019 of file Merger.cpp.

References mlir::Region::empty(), mlir::Region::front(), mlir::Operation::getOperand(), mlir::Block::getTerminator(), and isAdmissibleBranchExp().

◆ isAdmissibleBranchExp()

static bool mlir::sparse_tensor::isAdmissibleBranchExp ( Operation op,
Block block,
Value  v 
)
static

Ensures that sparse compiler can generate code for expression.

Definition at line 999 of file Merger.cpp.

References mlir::Value::dyn_cast(), mlir::Operation::getBlock(), mlir::Value::getDefiningOp(), mlir::Operation::getNumOperands(), and mlir::Operation::getOperand().

Referenced by isAdmissibleBranch().

◆ isComplexPrimaryType()

constexpr bool mlir::sparse_tensor::isComplexPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 119 of file Enums.h.

References kC32, and kC64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isCompressedDim() [1/2]

bool mlir::sparse_tensor::isCompressedDim ( RankedTensorType  type,
uint64_t  d 
)
inline

Convenience function to test for compressed dimension (0 <= d < rank).

Definition at line 94 of file SparseTensor.h.

References getDimLevelType(), and isCompressedDLT().

Referenced by isCOOType().

◆ isCompressedDim() [2/2]

bool mlir::sparse_tensor::isCompressedDim ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)
inline

Convenience function to test for compressed dimension (0 <= d < rank).

Definition at line 109 of file SparseTensor.h.

References getDimLevelType(), and isCompressedDLT().

◆ isCompressedDLT()

constexpr bool mlir::sparse_tensor::isCompressedDLT ( DimLevelType  dlt)
constexpr

◆ isDenseDim() [1/2]

bool mlir::sparse_tensor::isDenseDim ( RankedTensorType  type,
uint64_t  d 
)
inline

Convenience function to test for dense dimension (0 <= d < rank).

Definition at line 89 of file SparseTensor.h.

References getDimLevelType(), and isDenseDLT().

◆ isDenseDim() [2/2]

bool mlir::sparse_tensor::isDenseDim ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)
inline

Convenience function to test for dense dimension (0 <= d < rank).

Definition at line 104 of file SparseTensor.h.

References getDimLevelType(), and isDenseDLT().

◆ isDenseDLT()

constexpr bool mlir::sparse_tensor::isDenseDLT ( DimLevelType  dlt)
constexpr

◆ isFloatingPrimaryType()

constexpr bool mlir::sparse_tensor::isFloatingPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 107 of file Enums.h.

References kBF16, and kF64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isIntegralPrimaryType()

constexpr bool mlir::sparse_tensor::isIntegralPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 111 of file Enums.h.

References kI64, and kI8.

◆ isOrderedDim()

bool mlir::sparse_tensor::isOrderedDim ( RankedTensorType  type,
uint64_t  d 
)
inline

Convenience function to test for ordered property in the given dimension (0 <= d < rank).

Definition at line 124 of file SparseTensor.h.

References getDimLevelType(), and isOrderedDLT().

◆ isOrderedDLT()

constexpr bool mlir::sparse_tensor::isOrderedDLT ( DimLevelType  dlt)
constexpr

Check if the DimLevelType is ordered (regardless of storage format).

Definition at line 234 of file Enums.h.

Referenced by isAllDimOrdered(), isOrderedDim(), and mlir::sparse_tensor::SparseTensorStorageBase::isOrderedLvl().

◆ isRealPrimaryType()

constexpr bool mlir::sparse_tensor::isRealPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 115 of file Enums.h.

References kF64, and kI8.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isSingletonDim() [1/2]

bool mlir::sparse_tensor::isSingletonDim ( RankedTensorType  type,
uint64_t  d 
)
inline

Convenience function to test for singleton dimension (0 <= d < rank).

Definition at line 99 of file SparseTensor.h.

References getDimLevelType(), and isSingletonDLT().

Referenced by isCOOType().

◆ isSingletonDim() [2/2]

bool mlir::sparse_tensor::isSingletonDim ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)
inline

Convenience function to test for singleton dimension (0 <= d < rank).

Definition at line 114 of file SparseTensor.h.

References getDimLevelType(), and isSingletonDLT().

◆ isSingletonDLT()

constexpr bool mlir::sparse_tensor::isSingletonDLT ( DimLevelType  dlt)
constexpr

◆ isUndefDLT()

constexpr bool mlir::sparse_tensor::isUndefDLT ( DimLevelType  dlt)
constexpr

Check if the DimLevelType is the special undefined value.

Definition at line 208 of file Enums.h.

References Undef.

Referenced by computeIterationGraph(), findAffine(), genIf(), isAdmissibleTensorExp(), and translateBitsToTidDimPairs().

◆ isUniqueCOOType()

bool mlir::sparse_tensor::isUniqueCOOType ( RankedTensorType  tp)

Returns true iff the given type is a type for a COO tensor with the last dimension level type being unique.

Definition at line 426 of file SparseTensorDialect.cpp.

References getSparseTensorEncoding(), and isCOOType().

◆ isUniqueDim()

bool mlir::sparse_tensor::isUniqueDim ( RankedTensorType  type,
uint64_t  d 
)
inline

Convenience function to test for unique property in the given dimension (0 <= d < rank).

Definition at line 130 of file SparseTensor.h.

References getDimLevelType(), and isUniqueDLT().

◆ isUniqueDLT()

constexpr bool mlir::sparse_tensor::isUniqueDLT ( DimLevelType  dlt)
constexpr

Check if the DimLevelType is unique (regardless of storage format).

Definition at line 239 of file Enums.h.

Referenced by isCOOType(), isUniqueDim(), and mlir::sparse_tensor::SparseTensorStorageBase::isUniqueLvl().

◆ isValidDLT()

constexpr bool mlir::sparse_tensor::isValidDLT ( DimLevelType  dlt)
constexpr

Check that the DimLevelType contains a valid (possibly undefined) value.

Definition at line 198 of file Enums.h.

Referenced by getDimLevelType(), and mlir::sparse_tensor::Merger::setDimAndDimLevelType().

◆ isZeroRankedTensorOrScalar()

bool mlir::sparse_tensor::isZeroRankedTensorOrScalar ( Type  type)
inline

Definition at line 337 of file CodegenUtils.h.

References mlir::Type::dyn_cast().

Referenced by mlir::sparse_tensor::LoopEmitter::initialize().

◆ kindToOpSymbol()

static const char* mlir::sparse_tensor::kindToOpSymbol ( Kind  kind)
static

◆ overheadTypeEncoding() [1/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( Type  tp)

Converts an overhead storage type to its internal type-encoding.

Definition at line 96 of file CodegenUtils.cpp.

References mlir::Type::dyn_cast(), mlir::Type::isIndex(), kIndex, and overheadTypeEncoding().

◆ overheadTypeEncoding() [2/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( unsigned  width)

Converts an overhead storage bitwidth to its internal type-encoding.

Definition at line 80 of file CodegenUtils.cpp.

References kIndex, kU16, kU32, kU64, and kU8.

Referenced by constantOverheadTypeEncoding(), indexOverheadTypeEncoding(), overheadTypeEncoding(), overheadTypeFunctionSuffix(), and pointerOverheadTypeEncoding().

◆ overheadTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( OverheadType  ot)

Convert OverheadType to its function-name suffix.

Definition at line 144 of file CodegenUtils.cpp.

References CASE, kIndex, and MLIR_SPARSETENSOR_FOREVERY_FIXED_O.

Referenced by overheadTypeFunctionSuffix().

◆ overheadTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( Type  overheadTp)

Converts an overhead storage type to its function-name suffix.

Definition at line 157 of file CodegenUtils.cpp.

References overheadTypeEncoding(), and overheadTypeFunctionSuffix().

◆ pointerOverheadTypeEncoding()

OverheadType mlir::sparse_tensor::pointerOverheadTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for pointer overhead storage.

Definition at line 121 of file CodegenUtils.cpp.

References overheadTypeEncoding().

Referenced by getPointerOverheadType().

◆ primaryTypeEncoding()

PrimaryType mlir::sparse_tensor::primaryTypeEncoding ( Type  elemTp)

Converts a primary storage type to its internal type-encoding.

Definition at line 161 of file CodegenUtils.cpp.

References mlir::Type::dyn_cast(), mlir::Type::isBF16(), mlir::Type::isF16(), mlir::Type::isF32(), mlir::Type::isF64(), mlir::Type::isInteger(), kBF16, kC32, kC64, kF16, kF32, kF64, kI16, kI32, kI64, and kI8.

Referenced by constantPrimaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( PrimaryType  pt)

Convert PrimaryType to its function-name suffix.

Definition at line 188 of file CodegenUtils.cpp.

References CASE, and MLIR_SPARSETENSOR_FOREVERY_V.

Referenced by primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( Type  elemTp)

Converts a primary storage type to its function-name suffix.

Definition at line 199 of file CodegenUtils.cpp.

References primaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels ( DialectRegistry registry)

◆ registerSparseTensorPipelines()

void mlir::sparse_tensor::registerSparseTensorPipelines ( )

Registers all pipelines for the sparse_tensor dialect.

At present, this includes only "sparse-compiler".

Definition at line 94 of file SparseTensorPipelines.cpp.

References buildSparseCompiler().

Referenced by mlir::registerAllPasses().

◆ reshapeValuesToLevels()

Value mlir::sparse_tensor::reshapeValuesToLevels ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc,
const SmallVectorImpl< Value > &  dimSizes,
Value  valuesBuffer,
Value  idxBuffer 
)

Reshapes the linear values buffer for an annotated all dense sparse tensor to match the shape of the corresponding dense tensor to support direct access of the buffer through indices.

Definition at line 546 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::Builder::getIndexType(), getMemRefType(), storeIndices(), and toOrigDim().

◆ sizesFromSrc()

void mlir::sparse_tensor::sizesFromSrc ( OpBuilder builder,
SmallVectorImpl< Value > &  sizes,
Location  loc,
Value  src 
)

Populates given sizes array from dense tensor or sparse tensor constant.

Definition at line 483 of file CodegenUtils.cpp.

References mlir::Type::cast(), mlir::linalg::createOrFoldDimOp(), and mlir::Value::getType().

Referenced by concatSizesFromInputs().

◆ storeIndices()

void mlir::sparse_tensor::storeIndices ( OpBuilder builder,
Location  loc,
unsigned  rank,
Value  ind,
ValueRange  ivs,
unsigned  offsetDim = 0,
Value  offset = Value() 
)

Converts the vector indices and store it into the memory pointed by ind, apply (optional) offset on offsetDim.

Definition at line 534 of file CodegenUtils.cpp.

References constantIndex(), and mlir::OpBuilder::create().

Referenced by haveNoReadsAfterWriteExceptSameIndex(), and reshapeValuesToLevels().

◆ toFieldKind()

SparseTensorFieldKind mlir::sparse_tensor::toFieldKind ( StorageSpecifierKind  kind)
inline

◆ toMLIRString()

std::string mlir::sparse_tensor::toMLIRString ( DimLevelType  dlt)
inline

Returns string representation of the given dimension level type.

Definition at line 170 of file Enums.h.

References Compressed, CompressedNo, CompressedNu, CompressedNuNo, Dense, Singleton, SingletonNo, SingletonNu, SingletonNuNo, and Undef.

Referenced by mlir::sparse_tensor::Merger::dumpBits(), and parseDLT().

◆ toOrigDim() [1/2]

uint64_t mlir::sparse_tensor::toOrigDim ( RankedTensorType  type,
uint64_t  d 
)

Convenience method to translate the given stored dimension to the original dimension (0 <= d < rank).

Definition at line 470 of file SparseTensorDialect.cpp.

References getSparseTensorEncoding(), and toOrigDim().

◆ toOrigDim() [2/2]

uint64_t mlir::sparse_tensor::toOrigDim ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)

◆ toSpecifierKind()

StorageSpecifierKind mlir::sparse_tensor::toSpecifierKind ( SparseTensorFieldKind  kind)
inline

Definition at line 98 of file SparseTensorStorageLayout.h.

References StorageSpec.

◆ toStoredDim() [1/2]

uint64_t mlir::sparse_tensor::toStoredDim ( RankedTensorType  type,
uint64_t  d 
)

Convenience method to translate the given original dimension to the stored dimension (0 <= d < rank).

Definition at line 475 of file SparseTensorDialect.cpp.

References getSparseTensorEncoding(), and toStoredDim().

◆ toStoredDim() [2/2]

uint64_t mlir::sparse_tensor::toStoredDim ( SparseTensorEncodingAttr  enc,
uint64_t  d 
)

Definition at line 455 of file SparseTensorDialect.cpp.

References mlir::getAffineDimExpr().

Referenced by toStoredDim().

◆ translateIndicesArray()

void mlir::sparse_tensor::translateIndicesArray ( OpBuilder builder,
Location  loc,
ArrayRef< ReassociationIndices reassociation,
ValueRange  srcIndices,
ArrayRef< Value srcShape,
ArrayRef< Value dstShape,
SmallVectorImpl< Value > &  dstIndices 
)

Translate indices during a reshaping operation.

Definition at line 293 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().

◆ writeExtFROSTT()

template<typename V >
void mlir::sparse_tensor::writeExtFROSTT ( const SparseTensorCOO< V > &  coo,
const char *  filename 
)
inline

Writes the sparse tensor to filename in extended FROSTT format.

Definition at line 359 of file File.h.

References mlir::sparse_tensor::SparseTensorCOO< V >::getDimSizes(), mlir::sparse_tensor::SparseTensorCOO< V >::getElements(), and mlir::sparse_tensor::SparseTensorCOO< V >::getRank().