MLIR  19.0.0git
Namespaces | Classes | Typedefs | Enumerations | Functions
mlir::sparse_tensor Namespace Reference

Namespaces

 detail
 
 ir_detail
 

Classes

struct  LevelType
 This enum defines all the sparse representations supportable by the SparseTensor dialect. More...
 
struct  COOSegment
 A simple structure that encodes a range of levels in the sparse tensors that forms a COO segment. More...
 
class  StorageLayout
 Provides methods to access fields of a sparse tensor with the given encoding. More...
 
class  SparseTensorType
 A wrapper around RankedTensorType, which has three goals: More...
 
struct  SparsifierOptions
 Options for the "sparsifier" pipeline. More...
 
struct  TensorExp
 Tensor expression. Represents an MLIR expression in tensor index notation. More...
 
struct  LatPoint
 Lattice point. More...
 
class  Merger
 A class to handle all iteration lattice operations. More...
 
struct  Element
 An element of a sparse tensor in coordinate-scheme representation (i.e., a pair of coordinates and value). More...
 
struct  ElementLT
 Closure object for operator< on Element with a given rank. More...
 
class  SparseTensorCOO
 A memory-resident sparse tensor in coordinate-scheme representation (a collection of Elements). More...
 
class  SparseTensorReader
 This class abstracts over the information stored in file headers, as well as providing the buffers and methods for parsing those headers. More...
 
class  MapRef
 A class for capturing the sparse tensor type map with a compact encoding. More...
 
class  SparseTensorStorageBase
 Abstract base class for SparseTensorStorage<P,C,V>. More...
 
class  SparseTensorStorage
 A memory-resident sparse tensor using a storage scheme based on per-level sparse/dense annotations. More...
 
class  SparsificationAndBufferizationPass
 A pass that lowers tensor ops to memref ops, regardless of whether they are dense or sparse. More...
 
class  CodegenEnv
 The code generation environment class aggregates a number of data structures that are needed during the code generation phase of sparsification. More...
 
class  FuncCallOrInlineGenerator
 A helper class to simplify lowering operations with/without function calls. More...
 
class  IterationGraphSorter
 
class  LoopEmitter
 
class  SparseTensorSpecifier
 
class  SparseTensorDescriptorImpl
 A helper class around an array of values that corresponds to a sparse tensor. More...
 
class  SparseTensorDescriptor
 Uses ValueRange for immutable descriptors. More...
 
class  MutSparseTensorDescriptor
 Using SmallVector for mutable descriptor allows users to reuse it as a tmp buffers to append value for some special cases, though users should be responsible to restore the buffer to legal states after their use. More...
 
class  SparseTensorLevel
 The base class for all types of sparse tensor levels. More...
 
class  SparseIterator
 Helper class that generates loop conditions, etc, to traverse a sparse tensor level. More...
 

Typedefs

using index_type = uint64_t
 This type is used in the public API at all places where MLIR expects values with the built-in type "index". More...
 
using complex64 = std::complex< double >
 
using complex32 = std::complex< float >
 
using Dimension = uint64_t
 The type of dimension identifiers and dimension-ranks. More...
 
using Level = uint64_t
 The type of level identifiers and level-ranks. More...
 
using Size = int64_t
 The type for individual components of a compile-time shape, including the value ShapedType::kDynamic (for shapes). More...
 
using FieldIndex = unsigned
 The type of field indices. More...
 
using TensorId = unsigned
 Tensor identifiers, chosen to be the BlockArgument::getArgNumber of the value passed to Merger::buildTensorExp. More...
 
using LoopId = unsigned
 Loop identifiers. More...
 
using TensorLoopId = unsigned
 A compressed representation of std::pair<TensorId, LoopId>. More...
 
using ExprId = unsigned
 TensorExp identifiers. More...
 
using LatPointId = unsigned
 LatPoint identifiers. More...
 
using LatSetId = unsigned
 LatSet identifiers. More...
 
using LvlLTPair = std::pair< Level, LevelType >
 A pair of level and its corresponding LevelType of a tensor. More...
 
using LoopCoeffPair = std::pair< LoopId, unsigned >
 A pair of loop id and its coefficients. More...
 
using TensorLevel = unsigned
 

Enumerations

enum class  OverheadType : uint32_t {
  kIndex = 0 , kU64 = 1 , kU32 = 2 , kU16 = 3 ,
  kU8 = 4
}
 Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor. More...
 
enum class  PrimaryType : uint32_t {
  kF64 = 1 , kF32 = 2 , kF16 = 3 , kBF16 = 4 ,
  kI64 = 5 , kI32 = 6 , kI16 = 7 , kI8 = 8 ,
  kC64 = 9 , kC32 = 10
}
 Encoding of the elemental type, for "overloading" @newSparseTensor. More...
 
enum class  Action : uint32_t { kEmpty = 0 , kFromReader = 1 , kPack = 2 , kSortCOOInPlace = 3 }
 The actions performed by @newSparseTensor. More...
 
enum class  LevelFormat : uint64_t {
  Undef = 0x00000000 , Dense = 0x00010000 , Batch = 0x00020000 , Compressed = 0x00040000 ,
  Singleton = 0x00080000 , LooseCompressed = 0x00100000 , NOutOfM = 0x00200000
}
 This enum defines all supported storage format without the level properties. More...
 
enum class  LevelPropNonDefault : uint64_t { Nonunique = 0x0001 , Nonordered = 0x0002 , SoA = 0x0004 }
 This enum defines all the nondefault properties for storage formats. More...
 
enum class  SparseTensorFieldKind : uint32_t { StorageSpec = 0 , PosMemRef = static_cast<uint32_t>(StorageSpecifierKind::PosMemSize) , CrdMemRef = static_cast<uint32_t>(StorageSpecifierKind::CrdMemSize) , ValMemRef = static_cast<uint32_t>(StorageSpecifierKind::ValMemSize) }
 ===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields. More...
 
enum class  EmitCInterface : bool { Off = false , On = true }
 Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall(). More...
 
enum class  SortMask : unsigned {
  kIncludeDenseOutput = 0x1 , kIncludeDenseInput = 0x2 , kIncludeAll = 0x7 , kIncludeDense = 0x3 ,
  kSparseOnly = 0x0
}
 Iteration graph sorting mask,. More...
 
enum class  IterKind : uint8_t {
  kTrivial , kDedup , kSubSect , kNonEmptySubSect ,
  kFilter , kPad
}
 
enum class  ExpArity { kNullary , kUnary , kBinary }
 

Functions

constexpr bool isFloatingPrimaryType (PrimaryType valTy)
 
constexpr bool isIntegralPrimaryType (PrimaryType valTy)
 
constexpr bool isRealPrimaryType (PrimaryType valTy)
 
constexpr bool isComplexPrimaryType (PrimaryType valTy)
 
constexpr bool encPowOfTwo (LevelFormat fmt)
 
template<LevelFormat... targets>
constexpr bool isAnyOfFmt (LevelFormat fmt)
 
constexpr const char * toFormatString (LevelFormat lvlFmt)
 Returns string representation of the given level format. More...
 
constexpr const char * toPropString (LevelPropNonDefault lvlProp)
 Returns string representation of the given level properties. More...
 
constexpr uint64_t nToBits (uint64_t n)
 
constexpr uint64_t mToBits (uint64_t m)
 
std::optional< LevelTypebuildLevelType (LevelFormat lf, const std::vector< LevelPropNonDefault > &properties, uint64_t n=0, uint64_t m=0)
 
std::optional< LevelTypebuildLevelType (LevelFormat lf, bool ordered, bool unique, uint64_t n=0, uint64_t m=0)
 
bool isUndefLT (LevelType lt)
 
bool isDenseLT (LevelType lt)
 
bool isBatchLT (LevelType lt)
 
bool isCompressedLT (LevelType lt)
 
bool isLooseCompressedLT (LevelType lt)
 
bool isSingletonLT (LevelType lt)
 
bool isNOutOfMLT (LevelType lt)
 
bool isOrderedLT (LevelType lt)
 
bool isUniqueLT (LevelType lt)
 
bool isWithCrdLT (LevelType lt)
 
bool isWithPosLT (LevelType lt)
 
bool isValidLT (LevelType lt)
 
std::optional< LevelFormatgetLevelFormat (LevelType lt)
 
uint64_t getN (LevelType lt)
 
uint64_t getM (LevelType lt)
 
bool isValidNOutOfMLT (LevelType lt, uint64_t n, uint64_t m)
 
std::string toMLIRString (LevelType lt)
 
constexpr uint64_t encodeDim (uint64_t i, uint64_t cf, uint64_t cm)
 Bit manipulations for affine encoding. More...
 
constexpr uint64_t encodeLvl (uint64_t i, uint64_t c, uint64_t ii)
 
constexpr bool isEncodedFloor (uint64_t v)
 
constexpr bool isEncodedMod (uint64_t v)
 
constexpr bool isEncodedMul (uint64_t v)
 
constexpr uint64_t decodeIndex (uint64_t v)
 
constexpr uint64_t decodeConst (uint64_t v)
 
constexpr uint64_t decodeMulc (uint64_t v)
 
constexpr uint64_t decodeMuli (uint64_t v)
 
template<typename T >
RankedTensorType getRankedTensorType (T &&t)
 Convenience method to abbreviate casting getType(). More...
 
template<typename T >
MemRefType getMemRefType (T &&t)
 Convenience method to abbreviate casting getType(). More...
 
SparseTensorEncodingAttr getSparseTensorEncoding (Type type)
 Convenience method to get a sparse encoding attribute from a type. More...
 
bool hasAnySparseType (TypeRange types)
 Returns true iff the type range has any sparse tensor type. More...
 
bool hasAnySparseOperand (Operation *op)
 Returns true iff MLIR operand has any sparse operand. More...
 
bool hasAnySparseResult (Operation *op)
 Returns true iff MLIR operand has any sparse result. More...
 
bool hasAnySparseOperandOrResult (Operation *op)
 Returns true iff MLIR operand has any sparse operand or result. More...
 
bool hasAnyNonIdentityOperandsOrResults (Operation *op)
 Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps. More...
 
AffineMap inferLvlToDim (AffineMap dimToLvl, MLIRContext *context)
 Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails. More...
 
AffineMap inverseBlockSparsity (AffineMap dimToLvl, MLIRContext *context)
 Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases. More...
 
SmallVector< unsigned > getBlockSize (AffineMap dimToLvl)
 Given the dimToLvl map, returns the block sizes in a vector. More...
 
bool isBlockSparsity (AffineMap dimToLvl)
 Given the dimToLvl map, returns if it's block sparsity. More...
 
Dimension toDim (SparseTensorEncodingAttr enc, Level l)
 Convenience method to translate the given level to the corresponding dimension. More...
 
Level toLvl (SparseTensorEncodingAttr enc, Dimension d)
 Convenience method to translate the given dimension to the corresponding level. More...
 
StorageSpecifierKind toSpecifierKind (SparseTensorFieldKind kind)
 
SparseTensorFieldKind toFieldKind (StorageSpecifierKind kind)
 
unsigned getNumFieldsFromEncoding (SparseTensorEncodingAttr enc)
 
unsigned getNumDataFieldsFromEncoding (SparseTensorEncodingAttr enc)
 
void foreachFieldInSparseTensor (SparseTensorEncodingAttr enc, llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)> callback)
 
void foreachFieldAndTypeInSparseTensor (SparseTensorType, llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>)
 
SparseTensorType getSparseTensorType (Value val)
 Convenience methods to obtain a SparseTensorType from a Value. More...
 
std::optional< SparseTensorTypetryGetSparseTensorType (Value val)
 
void buildSparsifier (OpPassManager &pm, const SparsifierOptions &options)
 Adds the "sparsifier" pipeline to the OpPassManager. More...
 
void registerSparseTensorPipelines ()
 Registers all pipelines for the sparse_tensor dialect. More...
 
void registerTransformDialectExtension (DialectRegistry &registry)
 
void registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)
 
llvm::hash_code hash_value (LevelType lt)
 
static bool containsSparseTensor (TypeRange types)
 Return true if one of the given types is a sparse tensor type. More...
 
OverheadType overheadTypeEncoding (unsigned width)
 Converts an overhead storage bitwidth to its internal type-encoding. More...
 
OverheadType overheadTypeEncoding (Type tp)
 Converts an overhead storage type to its internal type-encoding. More...
 
Type getOverheadType (Builder &builder, OverheadType ot)
 Converts the internal type-encoding for overhead storage to an mlir::Type. More...
 
OverheadType posTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for position overhead storage. More...
 
OverheadType crdTypeEncoding (SparseTensorEncodingAttr enc)
 Returns the OverheadType for coordinate overhead storage. More...
 
StringRef overheadTypeFunctionSuffix (OverheadType ot)
 Convert OverheadType to its function-name suffix. More...
 
StringRef overheadTypeFunctionSuffix (Type overheadTp)
 Converts an overhead storage type to its function-name suffix. More...
 
PrimaryType primaryTypeEncoding (Type elemTp)
 Converts a primary storage type to its internal type-encoding. More...
 
StringRef primaryTypeFunctionSuffix (PrimaryType pt)
 Convert PrimaryType to its function-name suffix. More...
 
StringRef primaryTypeFunctionSuffix (Type elemTp)
 Converts a primary storage type to its function-name suffix. More...
 
Value genCast (OpBuilder &builder, Location loc, Value value, Type dstTy)
 Add type casting between arith and index types when needed. More...
 
Value genScalarToTensor (OpBuilder &builder, Location loc, Value elem, Type dstTp)
 Add conversion from scalar to given type (possibly a 0-rank tensor). More...
 
Value genIndexLoad (OpBuilder &builder, Location loc, Value mem, ValueRange s)
 Generates a pointer/index load from the sparse storage scheme. More...
 
TypedAttr getOneAttr (Builder &builder, Type tp)
 Generates a 1-valued attribute of the given type. More...
 
Value genIsNonzero (OpBuilder &builder, Location loc, Value v)
 Generates the comparison v != 0 where v is of numeric type. More...
 
void genReshapeDstShape (OpBuilder &builder, Location loc, SmallVectorImpl< Value > &dstShape, ArrayRef< Value > srcShape, ArrayRef< Size > staticDstShape, ArrayRef< ReassociationIndices > reassociation)
 Computes the shape of destination tensor of a reshape operator. More...
 
void reshapeCvs (OpBuilder &builder, Location loc, ArrayRef< ReassociationIndices > reassociation, ValueRange srcSizes, ValueRange srcCvs, ValueRange dstSizes, SmallVectorImpl< Value > &dstCvs)
 Reshape coordinates during a reshaping operation. More...
 
FlatSymbolRefAttr getFunc (ModuleOp module, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Returns a function reference (first hit also inserts into module). More...
 
func::CallOp createFuncCall (OpBuilder &builder, Location loc, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
 Creates a CallOp to the function reference returned by getFunc() in the builder's module. More...
 
Type getOpaquePointerType (MLIRContext *ctx)
 Returns the equivalent of void* for opaque arguments to the execution engine. More...
 
Type getOpaquePointerType (Builder &builder)
 
Value genAlloca (OpBuilder &builder, Location loc, Value sz, Type tp)
 Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>). More...
 
Value genAlloca (OpBuilder &builder, Location loc, unsigned sz, Type tp, bool staticShape=false)
 Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true). More...
 
Value genAllocaScalar (OpBuilder &builder, Location loc, Type tp)
 Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>. More...
 
Value allocaBuffer (OpBuilder &builder, Location loc, ValueRange values)
 Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer). More...
 
Value allocDenseTensor (OpBuilder &builder, Location loc, RankedTensorType tensorTp, ValueRange sizes)
 Generates code to allocate a buffer of the given type, and zero initialize it. More...
 
void deallocDenseTensor (OpBuilder &builder, Location loc, Value buffer)
 Generates code to deallocate a dense buffer. More...
 
void sizesFromSrc (OpBuilder &builder, SmallVectorImpl< Value > &sizes, Location loc, Value src)
 Populates given sizes array from dense tensor or sparse tensor constant. More...
 
OperationgetTop (Operation *op)
 Scans to top of generated loop. More...
 
void foreachInSparseConstant (OpBuilder &builder, Location loc, SparseElementsAttr attr, AffineMap order, function_ref< void(ArrayRef< Value >, Value)> callback)
 Iterate over a sparse constant, generates constantOp for value and coordinates. More...
 
SmallVector< ValueloadAll (OpBuilder &builder, Location loc, size_t size, Value mem, size_t offsetIdx=0, Value offsetVal=Value())
 Loads size-many values from the memref, which must have rank-1 and size greater-or-equal to size. More...
 
void storeAll (OpBuilder &builder, Location loc, Value mem, ValueRange vs, size_t offsetIdx=0, Value offsetVal=Value())
 Stores all the values of vs into the memref mem, which must have rank-1 and size greater-or-equal to vs.size(). More...
 
TypedValue< BaseMemRefTypegenToMemref (OpBuilder &builder, Location loc, Value tensor)
 
Value genValMemSize (OpBuilder &builder, Location loc, Value tensor)
 Generates code to retrieve the values size for the sparse tensor. More...
 
Value createOrFoldSliceOffsetOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
 Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known. More...
 
Value createOrFoldSliceStrideOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
 Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known. More...
 
Value genReader (OpBuilder &builder, Location loc, SparseTensorType stt, Value tensor, SmallVectorImpl< Value > &dimSizesValues, Value &dimSizesBuffer)
 Generates code that opens a reader and sets the dimension sizes. More...
 
Value genMapBuffers (OpBuilder &builder, Location loc, SparseTensorType stt, ArrayRef< Value > dimSizesValues, Value dimSizesBuffer, SmallVectorImpl< Value > &lvlSizesValues, Value &dim2lvlBuffer, Value &lvl2dimBuffer)
 Generates code to set up the buffer parameters for a map. More...
 
Value constantZero (OpBuilder &builder, Location loc, Type tp)
 Generates a 0-valued constant of the given type. More...
 
Value constantOne (OpBuilder &builder, Location loc, Type tp)
 Generates a 1-valued constant of the given type. More...
 
Value constantIndex (OpBuilder &builder, Location loc, int64_t i)
 Generates a constant of index type. More...
 
Value constantI64 (OpBuilder &builder, Location loc, int64_t i)
 Generates a constant of i64 type. More...
 
Value constantI32 (OpBuilder &builder, Location loc, int32_t i)
 Generates a constant of i32 type. More...
 
Value constantI16 (OpBuilder &builder, Location loc, int16_t i)
 Generates a constant of i16 type. More...
 
Value constantI8 (OpBuilder &builder, Location loc, int8_t i)
 Generates a constant of i8 type. More...
 
Value constantI1 (OpBuilder &builder, Location loc, bool b)
 Generates a constant of i1 type. More...
 
Value constantAction (OpBuilder &builder, Location loc, Action action)
 Generates a constant of the given Action. More...
 
Value constantOverheadTypeEncoding (OpBuilder &builder, Location loc, unsigned width)
 Generates a constant of the internal type-encoding for overhead storage. More...
 
Value constantPosTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for position overhead storage. More...
 
Value constantCrdTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
 Generates a constant of the internal type-encoding for coordinate overhead storage. More...
 
Value constantPrimaryTypeEncoding (OpBuilder &builder, Location loc, Type elemTp)
 Generates a constant of the internal type-encoding for primary storage. More...
 
Value constantLevelTypeEncoding (OpBuilder &builder, Location loc, LevelType lt)
 Generates a constant of the internal dimension level type encoding. More...
 
Value genValFromAttr (OpBuilder &builder, Location loc, Attribute attr)
 
bool isZeroRankedTensorOrScalar (Type type)
 
UnrealizedConversionCastOp getTuple (Value tensor)
 Returns the "tuple" value of the adapted tensor. More...
 
Value genTuple (OpBuilder &builder, Location loc, Type tp, ValueRange values)
 Packs the given values as a "tuple" value. More...
 
Value genTuple (OpBuilder &builder, Location loc, SparseTensorDescriptor desc)
 
SparseTensorDescriptor getDescriptorFromTensorTuple (Value tensor)
 
MutSparseTensorDescriptor getMutDescriptorFromTensorTuple (Value tensor, SmallVectorImpl< Value > &fields)
 
std::unique_ptr< SparseTensorLevelmakeSparseTensorLevel (OpBuilder &b, Location l, Value t, unsigned tid, Level lvl)
 Helper function to create a TensorLevel object from given tensor. More...
 
std::unique_ptr< SparseIteratormakeSimpleIterator (const SparseTensorLevel &stl, SparseEmitStrategy strategy)
 Helper function to create a simple SparseIterator object that iterate over the SparseTensorLevel. More...
 
std::pair< std::unique_ptr< SparseTensorLevel >, std::unique_ptr< SparseIterator > > makeSynLevelAndIterator (Value sz, unsigned tid, unsigned lvl, SparseEmitStrategy strategy)
 Helper function to create a synthetic SparseIterator object that iterates over a dense space specified by [0,sz). More...
 
std::unique_ptr< SparseIteratormakeSlicedLevelIterator (std::unique_ptr< SparseIterator > &&sit, Value offset, Value stride, Value size, SparseEmitStrategy strategy)
 Helper function to create a SparseIterator object that iterates over a sliced space, the orignal space (before slicing) is traversed by sit. More...
 
std::unique_ptr< SparseIteratormakePaddedIterator (std::unique_ptr< SparseIterator > &&sit, Value padLow, Value padHigh, SparseEmitStrategy strategy)
 Helper function to create a SparseIterator object that iterates over a padded sparse level (the padded value must be zero). More...
 
std::unique_ptr< SparseIteratormakeNonEmptySubSectIterator (OpBuilder &b, Location l, const SparseIterator *parent, Value loopBound, std::unique_ptr< SparseIterator > &&delegate, Value size, unsigned stride, SparseEmitStrategy strategy)
 Helper function to create a SparseIterator object that iterate over the non-empty subsections set. More...
 
std::unique_ptr< SparseIteratormakeTraverseSubSectIterator (OpBuilder &b, Location l, const SparseIterator &subsectIter, const SparseIterator &parent, std::unique_ptr< SparseIterator > &&wrap, Value loopBound, unsigned stride, SparseEmitStrategy strategy)
 Helper function to create a SparseIterator object that iterates over a non-empty subsection created by NonEmptySubSectIterator. More...
 
static ExpArity getExpArity (TensorExp::Kind k)
 
static const char * kindToOpSymbol (TensorExp::Kind kind)
 
static bool isCertainZero (Value val)
 Only returns true if we are certain this is a zero. More...
 
static bool isAdmissibleBranchExp (Operation *op, Block *block, Value v)
 Ensures that the sparsifier can generate code for expression. More...
 
static bool isAdmissibleBranch (Operation *op, Region &region)
 Ensures that the sparsifier can generate code for branch. More...
 
static bool isGreater (TensorExp::Kind kind, Attribute attr)
 
static Value insertYieldOp (RewriterBase &rewriter, Location loc, Region &region, ValueRange vals)
 
static Value buildUnaryPresent (RewriterBase &rewriter, Location loc, Operation *op, Value v0)
 
static Value buildBinaryOverlap (RewriterBase &rewriter, Location loc, Operation *op, Value v0, Value v1)
 
static Value buildRelu (RewriterBase &rewriter, Location loc, Value v0, Attribute attr)
 

Typedef Documentation

◆ complex32

using mlir::sparse_tensor::complex32 = typedef std::complex<float>

Definition at line 79 of file Enums.h.

◆ complex64

using mlir::sparse_tensor::complex64 = typedef std::complex<double>

Definition at line 78 of file Enums.h.

◆ Dimension

using mlir::sparse_tensor::Dimension = typedef uint64_t

The type of dimension identifiers and dimension-ranks.

Definition at line 35 of file SparseTensor.h.

◆ ExprId

using mlir::sparse_tensor::ExprId = typedef unsigned

TensorExp identifiers.

These are allocated by Merger::addExp, and serve as unique identifiers for the corresponding TensorExp object.

Definition at line 48 of file Merger.h.

◆ FieldIndex

using mlir::sparse_tensor::FieldIndex = typedef unsigned

The type of field indices.

This alias is to help code be more self-documenting; unfortunately it is not type-checked, so it only provides documentation rather than doing anything to prevent mixups.

Definition at line 110 of file SparseTensorStorageLayout.h.

◆ index_type

using mlir::sparse_tensor::index_type = typedef uint64_t

This type is used in the public API at all places where MLIR expects values with the built-in type "index".

For now, we simply assume that type is 64-bit, but targets with different "index" bitwidths should link with an alternatively built runtime support library.

Definition at line 47 of file Enums.h.

◆ LatPointId

using mlir::sparse_tensor::LatPointId = typedef unsigned

LatPoint identifiers.

These are allocated by Merger::addLat, and serve as unique identifiers for the corresponding LatPoint object.

Definition at line 52 of file Merger.h.

◆ LatSetId

using mlir::sparse_tensor::LatSetId = typedef unsigned

LatSet identifiers.

These are allocated by Merger::addSet (and by other methods calling that one), and serve as unique identifiers for the corresponding SmallVector<LatPointId> object.

Definition at line 57 of file Merger.h.

◆ Level

using mlir::sparse_tensor::Level = typedef uint64_t

The type of level identifiers and level-ranks.

Definition at line 38 of file SparseTensor.h.

◆ LoopCoeffPair

using mlir::sparse_tensor::LoopCoeffPair = typedef std::pair<LoopId, unsigned>

A pair of loop id and its coefficients.

E.g., for affine expression in the affine map 2 * d0, loop id = 0, coefficient = 2.

Definition at line 64 of file Merger.h.

◆ LoopId

using mlir::sparse_tensor::LoopId = typedef unsigned

Loop identifiers.

Definition at line 38 of file Merger.h.

◆ LvlLTPair

using mlir::sparse_tensor::LvlLTPair = typedef std::pair<Level, LevelType>

A pair of level and its corresponding LevelType of a tensor.

Definition at line 60 of file Merger.h.

◆ Size

using mlir::sparse_tensor::Size = typedef int64_t

The type for individual components of a compile-time shape, including the value ShapedType::kDynamic (for shapes).

Definition at line 42 of file SparseTensor.h.

◆ TensorId

using mlir::sparse_tensor::TensorId = typedef unsigned

Tensor identifiers, chosen to be the BlockArgument::getArgNumber of the value passed to Merger::buildTensorExp.

Definition at line 35 of file Merger.h.

◆ TensorLevel

using mlir::sparse_tensor::TensorLevel = typedef unsigned

Definition at line 26 of file LoopEmitter.h.

◆ TensorLoopId

using mlir::sparse_tensor::TensorLoopId = typedef unsigned

A compressed representation of std::pair<TensorId, LoopId>.

The compression scheme is such that this also serves as an index into the bitvector stored in LatPoint (since that bitvector is just the implementation for a set of TensorLoopId values).

Definition at line 44 of file Merger.h.

Enumeration Type Documentation

◆ Action

enum mlir::sparse_tensor::Action : uint32_t
strong

The actions performed by @newSparseTensor.

Enumerator
kEmpty 
kFromReader 
kPack 
kSortCOOInPlace 

Definition at line 146 of file Enums.h.

◆ EmitCInterface

Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall().

Enumerator
Off 
On 

Definition at line 36 of file CodegenUtils.h.

◆ ExpArity

Enumerator
kNullary 
kUnary 
kBinary 

Definition at line 22 of file Merger.cpp.

◆ IterKind

enum mlir::sparse_tensor::IterKind : uint8_t
strong
Enumerator
kTrivial 
kDedup 
kSubSect 
kNonEmptySubSect 
kFilter 
kPad 

Definition at line 73 of file SparseTensorIterator.h.

◆ LevelFormat

enum mlir::sparse_tensor::LevelFormat : uint64_t
strong

This enum defines all supported storage format without the level properties.

Enumerator
Undef 
Dense 
Batch 
Compressed 
Singleton 
LooseCompressed 
NOutOfM 

Definition at line 154 of file Enums.h.

◆ LevelPropNonDefault

This enum defines all the nondefault properties for storage formats.

Enumerator
Nonunique 
Nonordered 
SoA 

Definition at line 204 of file Enums.h.

◆ OverheadType

enum mlir::sparse_tensor::OverheadType : uint32_t
strong

Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor.

Enumerator
kIndex 
kU64 
kU32 
kU16 
kU8 

Definition at line 51 of file Enums.h.

◆ PrimaryType

enum mlir::sparse_tensor::PrimaryType : uint32_t
strong

Encoding of the elemental type, for "overloading" @newSparseTensor.

Enumerator
kF64 
kF32 
kF16 
kBF16 
kI64 
kI32 
kI16 
kI8 
kC64 
kC32 

Definition at line 82 of file Enums.h.

◆ SortMask

enum mlir::sparse_tensor::SortMask : unsigned
strong

Iteration graph sorting mask,.

Enumerator
kIncludeDenseOutput 
kIncludeDenseInput 
kIncludeAll 
kIncludeDense 
kSparseOnly 

Definition at line 32 of file IterationGraphSorter.h.

◆ SparseTensorFieldKind

===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields.

Note that every memref with ? size actually behaves as a "vector", i.e. the stored size is the capacity and the used size resides in the storage_specifier struct.

struct { ; per-level l: ; if dense: <nothing> ; if compressed: memref<[batch] x ? x pos> positions ; positions for level l memref<[batch] x ? x crd> coordinates ; coordinates for level l ; if loose-[batch] x compressed: memref<[batch] x ? x pos> positions ; lo/hi pos pairs for level l memref<[batch] x ? x crd> coordinates ; coordinates for level l ; if singleton/2-out-of-4: memref<[batch] x ? x crd> coordinates ; coordinates for level l

memref<[batch] x ? x eltType> values ; values

struct sparse_tensor.storage_specifier { array<rank x int> lvlSizes ; sizes/cardinalities for each level // TODO: memSizes need to be expanded to array<[batch] x n x int> to // support different sizes for different batches. At the moment, we // assume that every batch occupies the same memory size. array<n x int> memSizes ; sizes/lengths for each data memref } };

In addition, for a "trailing COO region", defined as a compressed level followed by one or more singleton levels, the default SOA storage that is inherent to the TACO format is optimized into an AOS storage where all coordinates of a stored element appear consecutively. In such cases, a special operation (sparse_tensor.coordinates_buffer) must be used to access the AOS coordinates array. In the code below, the method getCOOStart is used to find the start of the "trailing COO region".

If the sparse tensor is a slice (produced by tensor.extract_slice operation), instead of allocating a new sparse tensor for it, it reuses the same sets of MemRefs but attaching a additional set of slicing-metadata for per-dimension slice offset and stride.

Examples.

#CSR storage of 2-dim matrix yields memref<?xindex> ; positions-1 memref<?xindex> ; coordinates-1 memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

#COO storage of 2-dim matrix yields memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

Slice on #COO storage of 2-dim matrix yields ;; Inherited from the original sparse tensors memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>, ; lvl0, lvl1, 3xsizes ;; Extra slicing-metadata array<2 x i64>, array<2 x i64>)>) ; dim offset, dim stride.

===-------------------------------------------------------------------—===//

Enumerator
StorageSpec 
PosMemRef 
CrdMemRef 
ValMemRef 

Definition at line 90 of file SparseTensorStorageLayout.h.

Function Documentation

◆ allocaBuffer()

Value mlir::sparse_tensor::allocaBuffer ( OpBuilder builder,
Location  loc,
ValueRange  values 
)

Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer).

Definition at line 380 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and genAlloca().

Referenced by genMapBuffers(), and genReader().

◆ allocDenseTensor()

Value mlir::sparse_tensor::allocDenseTensor ( OpBuilder builder,
Location  loc,
RankedTensorType  tensorTp,
ValueRange  sizes 
)

Generates code to allocate a buffer of the given type, and zero initialize it.

If the buffer type has any dynamic sizes, then the sizes parameter should be as filled by sizesFromPtr(); that way we can reuse the genDimSizeCall() results generated by sizesFromPtr().

Definition at line 392 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), and mlir::get().

◆ buildBinaryOverlap()

static Value mlir::sparse_tensor::buildBinaryOverlap ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0,
Value  v1 
)
static

Definition at line 1588 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ buildLevelType() [1/2]

std::optional<LevelType> mlir::sparse_tensor::buildLevelType ( LevelFormat  lf,
bool  ordered,
bool  unique,
uint64_t  n = 0,
uint64_t  m = 0 
)
inline

Definition at line 403 of file Enums.h.

References mlir::sparse_tensor::LevelType::buildLvlType().

◆ buildLevelType() [2/2]

std::optional<LevelType> mlir::sparse_tensor::buildLevelType ( LevelFormat  lf,
const std::vector< LevelPropNonDefault > &  properties,
uint64_t  n = 0,
uint64_t  m = 0 
)
inline

◆ buildRelu()

static Value mlir::sparse_tensor::buildRelu ( RewriterBase rewriter,
Location  loc,
Value  v0,
Attribute  attr 
)
static

◆ buildSparsifier()

void mlir::sparse_tensor::buildSparsifier ( OpPassManager pm,
const SparsifierOptions options 
)

◆ buildUnaryPresent()

static Value mlir::sparse_tensor::buildUnaryPresent ( RewriterBase rewriter,
Location  loc,
Operation op,
Value  v0 
)
static

Definition at line 1574 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ constantAction()

Value mlir::sparse_tensor::constantAction ( OpBuilder builder,
Location  loc,
Action  action 
)
inline

Generates a constant of the given Action.

Definition at line 364 of file CodegenUtils.h.

References constantI32().

◆ constantCrdTypeEncoding()

Value mlir::sparse_tensor::constantCrdTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for coordinate overhead storage.

Definition at line 384 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantI1()

Value mlir::sparse_tensor::constantI1 ( OpBuilder builder,
Location  loc,
bool  b 
)
inline

Generates a constant of i1 type.

Definition at line 359 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by createEqCompare(), createPartitionFunc(), endIf(), endLoop(), genCompressed(), and genIf().

◆ constantI16()

Value mlir::sparse_tensor::constantI16 ( OpBuilder builder,
Location  loc,
int16_t  i 
)
inline

Generates a constant of i16 type.

Definition at line 349 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantI32()

Value mlir::sparse_tensor::constantI32 ( OpBuilder builder,
Location  loc,
int32_t  i 
)
inline

Generates a constant of i32 type.

Definition at line 344 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantAction(), constantOverheadTypeEncoding(), and constantPrimaryTypeEncoding().

◆ constantI64()

Value mlir::sparse_tensor::constantI64 ( OpBuilder builder,
Location  loc,
int64_t  i 
)
inline

Generates a constant of i64 type.

Definition at line 339 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantLevelTypeEncoding(), and createQuickSortFunc().

◆ constantI8()

Value mlir::sparse_tensor::constantI8 ( OpBuilder builder,
Location  loc,
int8_t  i 
)
inline

Generates a constant of i8 type.

Definition at line 354 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantIndex()

Value mlir::sparse_tensor::constantIndex ( OpBuilder builder,
Location  loc,
int64_t  i 
)
inline

◆ constantLevelTypeEncoding()

Value mlir::sparse_tensor::constantLevelTypeEncoding ( OpBuilder builder,
Location  loc,
LevelType  lt 
)
inline

Generates a constant of the internal dimension level type encoding.

Definition at line 397 of file CodegenUtils.h.

References constantI64().

◆ constantOne()

Value mlir::sparse_tensor::constantOne ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 1-valued constant of the given type.

This supports all the same types as constantZero.

Definition at line 323 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), getOneAttr(), and mlir::Builder::getZeroAttr().

Referenced by createFor(), and createPartitionFunc().

◆ constantOverheadTypeEncoding()

Value mlir::sparse_tensor::constantOverheadTypeEncoding ( OpBuilder builder,
Location  loc,
unsigned  width 
)
inline

Generates a constant of the internal type-encoding for overhead storage.

Definition at line 369 of file CodegenUtils.h.

References constantI32(), and overheadTypeEncoding().

Referenced by constantCrdTypeEncoding(), and constantPosTypeEncoding().

◆ constantPosTypeEncoding()

Value mlir::sparse_tensor::constantPosTypeEncoding ( OpBuilder builder,
Location  loc,
SparseTensorEncodingAttr  enc 
)
inline

Generates a constant of the internal type-encoding for position overhead storage.

Definition at line 377 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantPrimaryTypeEncoding()

Value mlir::sparse_tensor::constantPrimaryTypeEncoding ( OpBuilder builder,
Location  loc,
Type  elemTp 
)
inline

Generates a constant of the internal type-encoding for primary storage.

Definition at line 390 of file CodegenUtils.h.

References constantI32(), and primaryTypeEncoding().

Referenced by genReader().

◆ constantZero()

Value mlir::sparse_tensor::constantZero ( OpBuilder builder,
Location  loc,
Type  tp 
)
inline

Generates a 0-valued constant of the given type.

In addition to the scalar types (ComplexType, FloatType, IndexType, IntegerType), this also works for RankedTensorType and VectorType (for which it generates a constant DenseElementsAttr of zeros).

Definition at line 312 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), and mlir::Builder::getZeroAttr().

Referenced by allocDenseTensor(), allocSchemeForRank(), createAllocation(), createAllocFields(), createFor(), genEndInsert(), and genIsNonzero().

◆ containsSparseTensor()

static bool mlir::sparse_tensor::containsSparseTensor ( TypeRange  types)
static

Return true if one of the given types is a sparse tensor type.

Definition at line 41 of file SparsificationAndBufferizationPass.cpp.

References getSparseTensorEncoding().

◆ crdTypeEncoding()

OverheadType mlir::sparse_tensor::crdTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for coordinate overhead storage.

Definition at line 76 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ createFuncCall()

func::CallOp mlir::sparse_tensor::createFuncCall ( OpBuilder builder,
Location  loc,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Creates a CallOp to the function reference returned by getFunc() in the builder's module.

Definition at line 343 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::OpBuilder::getBlock(), getFunc(), mlir::Operation::getParentOfType(), and mlir::Block::getParentOp().

Referenced by dumpIndexMemRef(), and genReader().

◆ createOrFoldSliceOffsetOp()

Value mlir::sparse_tensor::createOrFoldSliceOffsetOp ( OpBuilder builder,
Location  loc,
Value  tensor,
Dimension  dim 
)

Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 562 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceOffset().

◆ createOrFoldSliceStrideOp()

Value mlir::sparse_tensor::createOrFoldSliceStrideOp ( OpBuilder builder,
Location  loc,
Value  tensor,
Dimension  dim 
)

Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 572 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceStride().

◆ deallocDenseTensor()

void mlir::sparse_tensor::deallocDenseTensor ( OpBuilder builder,
Location  loc,
Value  buffer 
)

Generates code to deallocate a dense buffer.

Definition at line 409 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

◆ decodeConst()

constexpr uint64_t mlir::sparse_tensor::decodeConst ( uint64_t  v)
constexpr

Definition at line 491 of file Enums.h.

◆ decodeIndex()

constexpr uint64_t mlir::sparse_tensor::decodeIndex ( uint64_t  v)
constexpr

Definition at line 490 of file Enums.h.

◆ decodeMulc()

constexpr uint64_t mlir::sparse_tensor::decodeMulc ( uint64_t  v)
constexpr

Definition at line 492 of file Enums.h.

◆ decodeMuli()

constexpr uint64_t mlir::sparse_tensor::decodeMuli ( uint64_t  v)
constexpr

Definition at line 493 of file Enums.h.

◆ encodeDim()

constexpr uint64_t mlir::sparse_tensor::encodeDim ( uint64_t  i,
uint64_t  cf,
uint64_t  cm 
)
constexpr

Bit manipulations for affine encoding.

Note that because the indices in the mappings refer to dimensions and levels (and not the sizes of these dimensions and levels), the 64-bit encoding gives ample room for a compact encoding of affine operations in the higher bits. Pure permutations still allow for 60-bit indices. But non-permutations reserve 20-bits for the potential three components (index i, constant, index ii).

The compact encoding is as follows:

0xffffffffffffffff |0000 | 60-bit idx| e.g. i |0001 floor| 20-bit const|20-bit idx| e.g. i floor c |0010 mod | 20-bit const|20-bit idx| e.g. i mod c |0011 mul |20-bit idx|20-bit const|20-bit idx| e.g. i + c * ii

This encoding provides sufficient generality for currently supported sparse tensor types. To generalize this more, we will need to provide a broader encoding scheme for affine functions. Also, the library encoding may be replaced with pure "direct-IR" code in the future.

Definition at line 467 of file Enums.h.

Referenced by genMapBuffers().

◆ encodeLvl()

constexpr uint64_t mlir::sparse_tensor::encodeLvl ( uint64_t  i,
uint64_t  c,
uint64_t  ii 
)
constexpr

Definition at line 479 of file Enums.h.

Referenced by genMapBuffers().

◆ encPowOfTwo()

constexpr bool mlir::sparse_tensor::encPowOfTwo ( LevelFormat  fmt)
constexpr

Definition at line 164 of file Enums.h.

◆ foreachFieldAndTypeInSparseTensor()

void mlir::sparse_tensor::foreachFieldAndTypeInSparseTensor ( SparseTensorType  stt,
llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>  callback 
)

◆ foreachFieldInSparseTensor()

void mlir::sparse_tensor::foreachFieldInSparseTensor ( SparseTensorEncodingAttr  enc,
llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)>  callback 
)
inline

◆ foreachInSparseConstant()

void mlir::sparse_tensor::foreachInSparseConstant ( OpBuilder builder,
Location  loc,
SparseElementsAttr  attr,
AffineMap  order,
function_ref< void(ArrayRef< Value >, Value)>  callback 
)

Iterate over a sparse constant, generates constantOp for value and coordinates.

E.g., sparse<[ [0], [28], [31] ], [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] > => c1 = arith.constant 0 v1 = complex.constant (5.13, 2.0) callback({c1}, v1)

c2 = arith.constant 28 v2 = complex.constant (3.0, 4.0) callback({c2}, v2)

c3 = arith.constant 31 v3 = complex.constant (5.0, 6.0) callback({c3}, v3)

Definition at line 432 of file CodegenUtils.cpp.

References mlir::AffineMap::compose(), mlir::Builder::getMultiDimIdentityMap(), mlir::AffineMap::getNumResults(), and getRankedTensorType().

◆ genAlloca() [1/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
unsigned  sz,
Type  tp,
bool  staticShape = false 
)

Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true).

Definition at line 360 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), genAlloca(), and mlir::get().

◆ genAlloca() [2/2]

Value mlir::sparse_tensor::genAlloca ( OpBuilder builder,
Location  loc,
Value  sz,
Type  tp 
)

Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>).

Definition at line 369 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

Referenced by allocaBuffer(), and genAlloca().

◆ genAllocaScalar()

Value mlir::sparse_tensor::genAllocaScalar ( OpBuilder builder,
Location  loc,
Type  tp 
)

Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>.

Definition at line 375 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

◆ genCast()

Value mlir::sparse_tensor::genCast ( OpBuilder builder,
Location  loc,
Value  value,
Type  dstTy 
)

Add type casting between arith and index types when needed.

Definition at line 151 of file CodegenUtils.cpp.

References mlir::convertScalarToDtype(), mlir::OpBuilder::create(), and mlir::Value::getType().

Referenced by genCompressed(), genLoad(), genScalarToTensor(), and genStore().

◆ genIndexLoad()

Value mlir::sparse_tensor::genIndexLoad ( OpBuilder builder,
Location  loc,
Value  mem,
ValueRange  s 
)

Generates a pointer/index load from the sparse storage scheme.

Narrower data types need to be zero extended before casting the value into the index type used for looping and indexing.

Definition at line 177 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::Builder::getI64Type(), mlir::Builder::getIndexType(), mlir::Type::getIntOrFloatBitWidth(), and mlir::Value::getType().

◆ genIsNonzero()

Value mlir::sparse_tensor::genIsNonzero ( OpBuilder builder,
mlir::Location  loc,
Value  v 
)

Generates the comparison v != 0 where v is of numeric type.

For floating types, we use the "unordered" comparator (i.e., returns true if v is NaN).

Definition at line 204 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), mlir::Value::getType(), and mlir::Type::isIntOrIndex().

◆ genMapBuffers()

Value mlir::sparse_tensor::genMapBuffers ( OpBuilder builder,
Location  loc,
SparseTensorType  stt,
ArrayRef< Value dimSizesValues,
Value  dimSizesBuffer,
SmallVectorImpl< Value > &  lvlSizesValues,
Value dim2lvlBuffer,
Value lvl2dimBuffer 
)

◆ genReader()

Value mlir::sparse_tensor::genReader ( OpBuilder builder,
Location  loc,
SparseTensorType  stt,
Value  tensor,
SmallVectorImpl< Value > &  dimSizesValues,
Value dimSizesBuffer 
)

◆ genReshapeDstShape()

void mlir::sparse_tensor::genReshapeDstShape ( OpBuilder builder,
Location  loc,
SmallVectorImpl< Value > &  dstShape,
ArrayRef< Value srcShape,
ArrayRef< Size staticDstShape,
ArrayRef< ReassociationIndices reassociation 
)

Computes the shape of destination tensor of a reshape operator.

This is only used when operands have dynamic shape. The shape of the destination is stored into dstShape.

Definition at line 219 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and product().

◆ genScalarToTensor()

Value mlir::sparse_tensor::genScalarToTensor ( OpBuilder builder,
Location  loc,
Value  elem,
Type  dstTp 
)

Add conversion from scalar to given type (possibly a 0-rank tensor).

Definition at line 166 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and genCast().

◆ genToMemref()

TypedValue< BaseMemRefType > mlir::sparse_tensor::genToMemref ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Definition at line 550 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), and mlir::Value::getType().

◆ genTuple() [1/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
SparseTensorDescriptor  desc 
)
inline

◆ genTuple() [2/2]

Value mlir::sparse_tensor::genTuple ( OpBuilder builder,
Location  loc,
Type  tp,
ValueRange  values 
)
inline

Packs the given values as a "tuple" value.

Definition at line 237 of file SparseTensorDescriptor.h.

References mlir::OpBuilder::create().

Referenced by genTuple(), and mlir::SparseTensorTypeToBufferConverter::SparseTensorTypeToBufferConverter().

◆ genValFromAttr()

Value mlir::sparse_tensor::genValFromAttr ( OpBuilder builder,
Location  loc,
Attribute  attr 
)
inline

◆ genValMemSize()

Value mlir::sparse_tensor::genValMemSize ( OpBuilder builder,
Location  loc,
Value  tensor 
)

Generates code to retrieve the values size for the sparse tensor.

Definition at line 557 of file CodegenUtils.cpp.

References getDescriptorFromTensorTuple(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getValMemSize().

◆ getBlockSize()

SmallVector< unsigned > mlir::sparse_tensor::getBlockSize ( AffineMap  dimToLvl)

Given the dimToLvl map, returns the block sizes in a vector.

For instance, a 2x3 block will return [2, 3]. Unblocked dimension i will return 0, and i floordiv 1, i mod 1 will return 1. Therefore, the example below will return [0, 1]. map = ( i, j ) -> ( i : dense, j floordiv 1 : compressed, j mod 1 : dense ) Only valid block sparsity will be accepted.

Definition at line 1111 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getResults(), isBlockSparsity(), and mlir::Mod.

◆ getDescriptorFromTensorTuple()

SparseTensorDescriptor mlir::sparse_tensor::getDescriptorFromTensorTuple ( Value  tensor)
inline

Definition at line 248 of file SparseTensorDescriptor.h.

References getTuple().

Referenced by genValMemSize().

◆ getExpArity()

static ExpArity mlir::sparse_tensor::getExpArity ( TensorExp::Kind  k)
static

Definition at line 28 of file Merger.cpp.

References mlir::sparse_tensor::TensorExp::kAbsC, mlir::sparse_tensor::TensorExp::kAbsF, mlir::sparse_tensor::TensorExp::kAbsI, mlir::sparse_tensor::TensorExp::kAddC, mlir::sparse_tensor::TensorExp::kAddF, mlir::sparse_tensor::TensorExp::kAddI, mlir::sparse_tensor::TensorExp::kAndI, mlir::sparse_tensor::TensorExp::kBinary, kBinary, mlir::sparse_tensor::TensorExp::kBinaryBranch, mlir::sparse_tensor::TensorExp::kBitCast, mlir::sparse_tensor::TensorExp::kCastFS, mlir::sparse_tensor::TensorExp::kCastFU, mlir::sparse_tensor::TensorExp::kCastIdx, mlir::sparse_tensor::TensorExp::kCastS, mlir::sparse_tensor::TensorExp::kCastSF, mlir::sparse_tensor::TensorExp::kCastU, mlir::sparse_tensor::TensorExp::kCastUF, mlir::sparse_tensor::TensorExp::kCeilF, mlir::sparse_tensor::TensorExp::kCIm, mlir::sparse_tensor::TensorExp::kCmpF, mlir::sparse_tensor::TensorExp::kCmpI, mlir::sparse_tensor::TensorExp::kCRe, mlir::sparse_tensor::TensorExp::kDenseOp, mlir::sparse_tensor::TensorExp::kDivC, mlir::sparse_tensor::TensorExp::kDivF, mlir::sparse_tensor::TensorExp::kDivS, mlir::sparse_tensor::TensorExp::kDivU, mlir::sparse_tensor::TensorExp::kExpm1C, mlir::sparse_tensor::TensorExp::kExpm1F, mlir::sparse_tensor::TensorExp::kExtF, mlir::sparse_tensor::TensorExp::kFloorF, mlir::sparse_tensor::TensorExp::kInvariant, mlir::sparse_tensor::TensorExp::kLog1pC, mlir::sparse_tensor::TensorExp::kLog1pF, mlir::sparse_tensor::TensorExp::kLoopVar, mlir::sparse_tensor::TensorExp::kMulC, mlir::sparse_tensor::TensorExp::kMulF, mlir::sparse_tensor::TensorExp::kMulI, mlir::sparse_tensor::TensorExp::kNegC, mlir::sparse_tensor::TensorExp::kNegF, mlir::sparse_tensor::TensorExp::kNegI, kNullary, mlir::sparse_tensor::TensorExp::kOrI, mlir::sparse_tensor::TensorExp::kReduce, mlir::sparse_tensor::TensorExp::kRelu, mlir::sparse_tensor::TensorExp::kSelect, mlir::sparse_tensor::TensorExp::kShlI, mlir::sparse_tensor::TensorExp::kShrS, mlir::sparse_tensor::TensorExp::kShrU, mlir::sparse_tensor::TensorExp::kSinC, mlir::sparse_tensor::TensorExp::kSinF, mlir::sparse_tensor::TensorExp::kSqrtC, mlir::sparse_tensor::TensorExp::kSqrtF, mlir::sparse_tensor::TensorExp::kSubC, mlir::sparse_tensor::TensorExp::kSubF, mlir::sparse_tensor::TensorExp::kSubI, mlir::sparse_tensor::TensorExp::kSynZero, mlir::sparse_tensor::TensorExp::kTanhC, mlir::sparse_tensor::TensorExp::kTanhF, mlir::sparse_tensor::TensorExp::kTensor, mlir::sparse_tensor::TensorExp::kTruncF, mlir::sparse_tensor::TensorExp::kTruncI, mlir::sparse_tensor::TensorExp::kUnary, kUnary, and mlir::sparse_tensor::TensorExp::kXorI.

Referenced by mlir::sparse_tensor::Merger::expContainsTensor(), and mlir::sparse_tensor::Merger::hasNegateOnOut().

◆ getFunc()

FlatSymbolRefAttr mlir::sparse_tensor::getFunc ( ModuleOp  module,
StringRef  name,
TypeRange  resultType,
ValueRange  operands,
EmitCInterface  emitCInterface 
)

Returns a function reference (first hit also inserts into module).

Sets the "_emit_c_interface" on the function declaration when requested, so that LLVM lowering generates a wrapper function that takes care of ABI complications with passing in and returning MemRefs to C functions.

Definition at line 323 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), mlir::ValueRange::getTypes(), and mlir::Operation::setAttr().

Referenced by createFuncCall().

◆ getLevelFormat()

std::optional<LevelFormat> mlir::sparse_tensor::getLevelFormat ( LevelType  lt)
inline

Definition at line 432 of file Enums.h.

References mlir::sparse_tensor::LevelType::getLvlFmt(), and Undef.

◆ getM()

uint64_t mlir::sparse_tensor::getM ( LevelType  lt)
inline

◆ getMemRefType()

template<typename T >
MemRefType mlir::sparse_tensor::getMemRefType ( T &&  t)
inline

Convenience method to abbreviate casting getType().

Definition at line 95 of file SparseTensor.h.

Referenced by getMangledSortHelperFuncName(), and isMatchingWidth().

◆ getMutDescriptorFromTensorTuple()

MutSparseTensorDescriptor mlir::sparse_tensor::getMutDescriptorFromTensorTuple ( Value  tensor,
SmallVectorImpl< Value > &  fields 
)
inline

Definition at line 255 of file SparseTensorDescriptor.h.

References getTuple().

◆ getN()

uint64_t mlir::sparse_tensor::getN ( LevelType  lt)
inline

◆ getNumDataFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumDataFieldsFromEncoding ( SparseTensorEncodingAttr  enc)
inline

◆ getNumFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumFieldsFromEncoding ( SparseTensorEncodingAttr  enc)
inline

◆ getOneAttr()

mlir::TypedAttr mlir::sparse_tensor::getOneAttr ( Builder builder,
Type  tp 
)

Generates a 1-valued attribute of the given type.

This supports all the same types as getZeroAttr; however, unlike getZeroAttr, for unsupported types we raise llvm_unreachable rather than returning a null attribute.

Definition at line 189 of file CodegenUtils.cpp.

References mlir::DenseElementsAttr::get(), mlir::Builder::getFloatAttr(), mlir::Builder::getIndexAttr(), and mlir::Builder::getIntegerAttr().

Referenced by constantOne().

◆ getOpaquePointerType() [1/2]

Type mlir::sparse_tensor::getOpaquePointerType ( Builder builder)

Definition at line 356 of file CodegenUtils.cpp.

References mlir::Builder::getContext(), and getOpaquePointerType().

◆ getOpaquePointerType() [2/2]

Type mlir::sparse_tensor::getOpaquePointerType ( MLIRContext ctx)

Returns the equivalent of void* for opaque arguments to the execution engine.

Definition at line 352 of file CodegenUtils.cpp.

References mlir::get().

Referenced by genReader(), and getOpaquePointerType().

◆ getOverheadType()

Type mlir::sparse_tensor::getOverheadType ( Builder builder,
OverheadType  ot 
)

Converts the internal type-encoding for overhead storage to an mlir::Type.

Definition at line 54 of file CodegenUtils.cpp.

References mlir::Builder::getIndexType(), mlir::Builder::getIntegerType(), kIndex, kU16, kU32, kU64, and kU8.

◆ getRankedTensorType()

template<typename T >
RankedTensorType mlir::sparse_tensor::getRankedTensorType ( T &&  t)
inline

Convenience method to abbreviate casting getType().

Definition at line 87 of file SparseTensor.h.

Referenced by foreachInSparseConstant(), and mlir::sparse_tensor::LoopEmitter::initialize().

◆ getSparseTensorEncoding()

SparseTensorEncodingAttr mlir::sparse_tensor::getSparseTensorEncoding ( Type  type)

◆ getSparseTensorType()

SparseTensorType mlir::sparse_tensor::getSparseTensorType ( Value  val)
inline

◆ getTop()

Operation * mlir::sparse_tensor::getTop ( Operation op)

Scans to top of generated loop.

Definition at line 422 of file CodegenUtils.cpp.

◆ getTuple()

UnrealizedConversionCastOp mlir::sparse_tensor::getTuple ( Value  tensor)
inline

Returns the "tuple" value of the adapted tensor.

Definition at line 232 of file SparseTensorDescriptor.h.

References mlir::Value::getDefiningOp().

Referenced by flattenOperands(), getDescriptorFromTensorTuple(), and getMutDescriptorFromTensorTuple().

◆ hasAnyNonIdentityOperandsOrResults()

bool mlir::sparse_tensor::hasAnyNonIdentityOperandsOrResults ( Operation op)

Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps.

Definition at line 1172 of file SparseTensorDialect.cpp.

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseOperand()

bool mlir::sparse_tensor::hasAnySparseOperand ( Operation op)
inline

Returns true iff MLIR operand has any sparse operand.

Definition at line 113 of file SparseTensor.h.

◆ hasAnySparseOperandOrResult()

bool mlir::sparse_tensor::hasAnySparseOperandOrResult ( Operation op)
inline

Returns true iff MLIR operand has any sparse operand or result.

Definition at line 123 of file SparseTensor.h.

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseResult()

bool mlir::sparse_tensor::hasAnySparseResult ( Operation op)
inline

Returns true iff MLIR operand has any sparse result.

Definition at line 118 of file SparseTensor.h.

◆ hasAnySparseType()

bool mlir::sparse_tensor::hasAnySparseType ( TypeRange  types)
inline

Returns true iff the type range has any sparse tensor type.

Definition at line 106 of file SparseTensor.h.

References getSparseTensorEncoding().

◆ hash_value()

llvm::hash_code mlir::sparse_tensor::hash_value ( LevelType  lt)

Definition at line 51 of file SparseTensorDialect.cpp.

◆ inferLvlToDim()

AffineMap mlir::sparse_tensor::inferLvlToDim ( AffineMap  dimToLvl,
MLIRContext context 
)

Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails.

Definition at line 1043 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getNumSymbols(), inverseBlockSparsity(), mlir::inversePermutation(), isBlockSparsity(), and mlir::AffineMap::isPermutation().

Referenced by genReplaceDimToLvlMap().

◆ insertYieldOp()

static Value mlir::sparse_tensor::insertYieldOp ( RewriterBase rewriter,
Location  loc,
Region region,
ValueRange  vals 
)
static

◆ inverseBlockSparsity()

AffineMap mlir::sparse_tensor::inverseBlockSparsity ( AffineMap  dimToLvl,
MLIRContext context 
)

Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases.

Asserts on failure (so only use when known to succeed).

Definition at line 1058 of file SparseTensorDialect.cpp.

References mlir::Add, mlir::FloorDiv, mlir::AffineMap::get(), mlir::getAffineBinaryOpExpr(), mlir::getAffineDimExpr(), mlir::AffineMap::getNumResults(), mlir::AffineMap::getResult(), mlir::Mod, and mlir::Mul.

Referenced by inferLvlToDim().

◆ isAdmissibleBranch()

static bool mlir::sparse_tensor::isAdmissibleBranch ( Operation op,
Region region 
)
static

Ensures that the sparsifier can generate code for branch.

Definition at line 1264 of file Merger.cpp.

◆ isAdmissibleBranchExp()

static bool mlir::sparse_tensor::isAdmissibleBranchExp ( Operation op,
Block block,
Value  v 
)
static

Ensures that the sparsifier can generate code for expression.

Definition at line 1244 of file Merger.cpp.

◆ isAnyOfFmt()

template<LevelFormat... targets>
constexpr bool mlir::sparse_tensor::isAnyOfFmt ( LevelFormat  fmt)
constexpr

Definition at line 178 of file Enums.h.

Referenced by mlir::sparse_tensor::LevelType::isValidLvlBits().

◆ isBatchLT()

bool mlir::sparse_tensor::isBatchLT ( LevelType  lt)
inline

Definition at line 410 of file Enums.h.

References Batch, and mlir::sparse_tensor::LevelType::isa().

◆ isBlockSparsity()

bool mlir::sparse_tensor::isBlockSparsity ( AffineMap  dimToLvl)

Given the dimToLvl map, returns if it's block sparsity.

Definition at line 1128 of file SparseTensorDialect.cpp.

References mlir::FloorDiv, mlir::AffineMap::getResults(), and mlir::Mod.

Referenced by getBlockSize(), and inferLvlToDim().

◆ isCertainZero()

static bool mlir::sparse_tensor::isCertainZero ( Value  val)
static

Only returns true if we are certain this is a zero.

Definition at line 1201 of file Merger.cpp.

References mlir::Value::getDefiningOp().

◆ isComplexPrimaryType()

constexpr bool mlir::sparse_tensor::isComplexPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 141 of file Enums.h.

References kC32, and kC64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isCompressedLT()

bool mlir::sparse_tensor::isCompressedLT ( LevelType  lt)
inline

◆ isDenseLT()

bool mlir::sparse_tensor::isDenseLT ( LevelType  lt)
inline

◆ isEncodedFloor()

constexpr bool mlir::sparse_tensor::isEncodedFloor ( uint64_t  v)
constexpr

Definition at line 487 of file Enums.h.

◆ isEncodedMod()

constexpr bool mlir::sparse_tensor::isEncodedMod ( uint64_t  v)
constexpr

Definition at line 488 of file Enums.h.

◆ isEncodedMul()

constexpr bool mlir::sparse_tensor::isEncodedMul ( uint64_t  v)
constexpr

Definition at line 489 of file Enums.h.

◆ isFloatingPrimaryType()

constexpr bool mlir::sparse_tensor::isFloatingPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 129 of file Enums.h.

References kBF16, and kF64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isGreater()

static bool mlir::sparse_tensor::isGreater ( TensorExp::Kind  kind,
Attribute  attr 
)
static

Definition at line 1274 of file Merger.cpp.

◆ isIntegralPrimaryType()

constexpr bool mlir::sparse_tensor::isIntegralPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 133 of file Enums.h.

References kI64, and kI8.

◆ isLooseCompressedLT()

bool mlir::sparse_tensor::isLooseCompressedLT ( LevelType  lt)
inline

◆ isNOutOfMLT()

bool mlir::sparse_tensor::isNOutOfMLT ( LevelType  lt)
inline

◆ isOrderedLT()

bool mlir::sparse_tensor::isOrderedLT ( LevelType  lt)
inline

◆ isRealPrimaryType()

constexpr bool mlir::sparse_tensor::isRealPrimaryType ( PrimaryType  valTy)
constexpr

Definition at line 137 of file Enums.h.

References kF64, and kI8.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isSingletonLT()

bool mlir::sparse_tensor::isSingletonLT ( LevelType  lt)
inline

◆ isUndefLT()

bool mlir::sparse_tensor::isUndefLT ( LevelType  lt)
inline

◆ isUniqueLT()

bool mlir::sparse_tensor::isUniqueLT ( LevelType  lt)
inline

◆ isValidLT()

bool mlir::sparse_tensor::isValidLT ( LevelType  lt)
inline

◆ isValidNOutOfMLT()

bool mlir::sparse_tensor::isValidNOutOfMLT ( LevelType  lt,
uint64_t  n,
uint64_t  m 
)
inline

◆ isWithCrdLT()

bool mlir::sparse_tensor::isWithCrdLT ( LevelType  lt)
inline

◆ isWithPosLT()

bool mlir::sparse_tensor::isWithPosLT ( LevelType  lt)
inline

◆ isZeroRankedTensorOrScalar()

bool mlir::sparse_tensor::isZeroRankedTensorOrScalar ( Type  type)
inline

Definition at line 415 of file CodegenUtils.h.

Referenced by mlir::sparse_tensor::LoopEmitter::initialize().

◆ kindToOpSymbol()

static const char* mlir::sparse_tensor::kindToOpSymbol ( TensorExp::Kind  kind)
static

Definition at line 693 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::dumpExp().

◆ loadAll()

SmallVector< Value > mlir::sparse_tensor::loadAll ( OpBuilder builder,
Location  loc,
size_t  size,
Value  mem,
size_t  offsetIdx = 0,
Value  offsetVal = Value() 
)

Loads size-many values from the memref, which must have rank-1 and size greater-or-equal to size.

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value after loading.

Definition at line 507 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::Value::getType().

◆ makeNonEmptySubSectIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeNonEmptySubSectIterator ( OpBuilder b,
Location  l,
const SparseIterator parent,
Value  loopBound,
std::unique_ptr< SparseIterator > &&  delegate,
Value  size,
unsigned  stride,
SparseEmitStrategy  strategy 
)

Helper function to create a SparseIterator object that iterate over the non-empty subsections set.

Definition at line 1570 of file SparseTensorIterator.cpp.

References C_IDX, and tryUnwrapFilter().

◆ makePaddedIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makePaddedIterator ( std::unique_ptr< SparseIterator > &&  sit,
Value  padLow,
Value  padHigh,
SparseEmitStrategy  strategy 
)

Helper function to create a SparseIterator object that iterates over a padded sparse level (the padded value must be zero).

Definition at line 1555 of file SparseTensorIterator.cpp.

◆ makeSimpleIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeSimpleIterator ( const SparseTensorLevel stl,
SparseEmitStrategy  strategy 
)

Helper function to create a simple SparseIterator object that iterate over the SparseTensorLevel.

Definition at line 1529 of file SparseTensorIterator.cpp.

References mlir::sparse_tensor::SparseTensorLevel::getLT(), and isUniqueLT().

◆ makeSlicedLevelIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeSlicedLevelIterator ( std::unique_ptr< SparseIterator > &&  sit,
Value  offset,
Value  stride,
Value  size,
SparseEmitStrategy  strategy 
)

Helper function to create a SparseIterator object that iterates over a sliced space, the orignal space (before slicing) is traversed by sit.

Definition at line 1544 of file SparseTensorIterator.cpp.

◆ makeSparseTensorLevel()

std::unique_ptr< SparseTensorLevel > mlir::sparse_tensor::makeSparseTensorLevel ( OpBuilder b,
Location  l,
Value  t,
unsigned  tid,
Level  lvl 
)

Helper function to create a TensorLevel object from given tensor.

Definition at line 1482 of file SparseTensorIterator.cpp.

References mlir::OpBuilder::create(), mlir::sparse_tensor::LevelType::getLvlFmt(), and getSparseTensorType().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ makeSynLevelAndIterator()

std::pair< std::unique_ptr< SparseTensorLevel >, std::unique_ptr< SparseIterator > > mlir::sparse_tensor::makeSynLevelAndIterator ( Value  sz,
unsigned  tid,
unsigned  lvl,
SparseEmitStrategy  strategy 
)

Helper function to create a synthetic SparseIterator object that iterates over a dense space specified by [0,sz).

Definition at line 1520 of file SparseTensorIterator.cpp.

References mlir::sparse_tensor::SparseIterator::setSparseEmitStrategy().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ makeTraverseSubSectIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeTraverseSubSectIterator ( OpBuilder b,
Location  l,
const SparseIterator subsectIter,
const SparseIterator parent,
std::unique_ptr< SparseIterator > &&  wrap,
Value  loopBound,
unsigned  stride,
SparseEmitStrategy  strategy 
)

Helper function to create a SparseIterator object that iterates over a non-empty subsection created by NonEmptySubSectIterator.

Definition at line 1591 of file SparseTensorIterator.cpp.

References C_IDX, tryUnwrapFilter(), and wrap().

◆ mToBits()

constexpr uint64_t mlir::sparse_tensor::mToBits ( uint64_t  m)
constexpr

◆ nToBits()

constexpr uint64_t mlir::sparse_tensor::nToBits ( uint64_t  n)
constexpr

◆ overheadTypeEncoding() [1/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( Type  tp)

Converts an overhead storage type to its internal type-encoding.

Definition at line 46 of file CodegenUtils.cpp.

References mlir::Type::isIndex(), kIndex, and overheadTypeEncoding().

◆ overheadTypeEncoding() [2/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( unsigned  width)

Converts an overhead storage bitwidth to its internal type-encoding.

Definition at line 30 of file CodegenUtils.cpp.

References kIndex, kU16, kU32, kU64, and kU8.

Referenced by constantOverheadTypeEncoding(), crdTypeEncoding(), overheadTypeEncoding(), overheadTypeFunctionSuffix(), and posTypeEncoding().

◆ overheadTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( OverheadType  ot)

Convert OverheadType to its function-name suffix.

Definition at line 88 of file CodegenUtils.cpp.

References CASE, kIndex, and MLIR_SPARSETENSOR_FOREVERY_FIXED_O.

Referenced by overheadTypeFunctionSuffix().

◆ overheadTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( Type  overheadTp)

Converts an overhead storage type to its function-name suffix.

Definition at line 101 of file CodegenUtils.cpp.

References overheadTypeEncoding(), and overheadTypeFunctionSuffix().

◆ posTypeEncoding()

OverheadType mlir::sparse_tensor::posTypeEncoding ( SparseTensorEncodingAttr  enc)

Returns the OverheadType for position overhead storage.

Definition at line 71 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ primaryTypeEncoding()

PrimaryType mlir::sparse_tensor::primaryTypeEncoding ( Type  elemTp)

Converts a primary storage type to its internal type-encoding.

Definition at line 105 of file CodegenUtils.cpp.

References mlir::Type::isBF16(), mlir::Type::isF16(), mlir::Type::isF32(), mlir::Type::isF64(), mlir::Type::isInteger(), kBF16, kC32, kC64, kF16, kF32, kF64, kI16, kI32, kI64, and kI8.

Referenced by constantPrimaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( PrimaryType  pt)

Convert PrimaryType to its function-name suffix.

Definition at line 132 of file CodegenUtils.cpp.

References CASE, and MLIR_SPARSETENSOR_FOREVERY_V.

Referenced by primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( Type  elemTp)

Converts a primary storage type to its function-name suffix.

Definition at line 143 of file CodegenUtils.cpp.

References primaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels ( DialectRegistry registry)

◆ registerSparseTensorPipelines()

void mlir::sparse_tensor::registerSparseTensorPipelines ( )

Registers all pipelines for the sparse_tensor dialect.

At present, this includes only "sparsifier".

Definition at line 111 of file SparseTensorPipelines.cpp.

References buildSparsifier().

Referenced by mlir::registerAllPasses().

◆ registerTransformDialectExtension()

void mlir::sparse_tensor::registerTransformDialectExtension ( DialectRegistry registry)

◆ reshapeCvs()

void mlir::sparse_tensor::reshapeCvs ( OpBuilder builder,
Location  loc,
ArrayRef< ReassociationIndices reassociation,
ValueRange  srcSizes,
ValueRange  srcCvs,
ValueRange  dstSizes,
SmallVectorImpl< Value > &  dstCvs 
)

Reshape coordinates during a reshaping operation.

Definition at line 275 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().

◆ sizesFromSrc()

void mlir::sparse_tensor::sizesFromSrc ( OpBuilder builder,
SmallVectorImpl< Value > &  sizes,
Location  loc,
Value  src 
)

Populates given sizes array from dense tensor or sparse tensor constant.

Definition at line 414 of file CodegenUtils.cpp.

References mlir::linalg::createOrFoldDimOp(), mlir::sparse_tensor::SparseTensorType::getDimRank(), and getSparseTensorType().

Referenced by concatSizesFromInputs().

◆ storeAll()

void mlir::sparse_tensor::storeAll ( OpBuilder builder,
Location  loc,
Value  mem,
ValueRange  vs,
size_t  offsetIdx = 0,
Value  offsetVal = Value() 
)

Stores all the values of vs into the memref mem, which must have rank-1 and size greater-or-equal to vs.size().

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value before storing.

Definition at line 529 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and mlir::Value::getType().

◆ toDim()

Dimension mlir::sparse_tensor::toDim ( SparseTensorEncodingAttr  enc,
Level  l 
)

Convenience method to translate the given level to the corresponding dimension.

Requires: enc has a permuted dim2lvl map and 0 <= l < lvlRank.

Definition at line 1182 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

Referenced by genSliceOffset(), and genSliceStride().

◆ toFieldKind()

SparseTensorFieldKind mlir::sparse_tensor::toFieldKind ( StorageSpecifierKind  kind)
inline

Definition at line 102 of file SparseTensorStorageLayout.h.

◆ toFormatString()

constexpr const char* mlir::sparse_tensor::toFormatString ( LevelFormat  lvlFmt)
constexpr

Returns string representation of the given level format.

Definition at line 183 of file Enums.h.

References Batch, Compressed, Dense, LooseCompressed, NOutOfM, Singleton, and Undef.

Referenced by mlir::sparse_tensor::LevelType::toMLIRString().

◆ toLvl()

Level mlir::sparse_tensor::toLvl ( SparseTensorEncodingAttr  enc,
Dimension  d 
)

Convenience method to translate the given dimension to the corresponding level.

Requires: enc has a permuted dim2lvl map and 0 <= d < dimRank.

Definition at line 1191 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

◆ toMLIRString()

std::string mlir::sparse_tensor::toMLIRString ( LevelType  lt)
inline

◆ toPropString()

constexpr const char* mlir::sparse_tensor::toPropString ( LevelPropNonDefault  lvlProp)
constexpr

Returns string representation of the given level properties.

Definition at line 211 of file Enums.h.

References Nonordered, Nonunique, and SoA.

Referenced by mlir::sparse_tensor::LevelType::toMLIRString().

◆ toSpecifierKind()

StorageSpecifierKind mlir::sparse_tensor::toSpecifierKind ( SparseTensorFieldKind  kind)
inline

Definition at line 97 of file SparseTensorStorageLayout.h.

◆ tryGetSparseTensorType()

std::optional<SparseTensorType> mlir::sparse_tensor::tryGetSparseTensorType ( Value  val)
inline

Definition at line 377 of file SparseTensorType.h.

References mlir::Value::getType().

Referenced by tryFoldTensors().