Namespaces
	detail

	ir_detail

Classes
struct	LevelType
	This enum defines all the sparse representations supportable by the SparseTensor dialect. More...

struct	COOSegment
	A simple structure that encodes a range of levels in the sparse tensors that forms a COO segment. More...

class	I64BitSet
	A simple wrapper to encode a bitset of (at most 64) levels, currently used by `sparse_tensor.iterate` operation for the set of levels on which the coordinates should be loaded. More...

class	StorageLayout
	Provides methods to access fields of a sparse tensor with the given encoding. More...

class	SparseTensorType
	A wrapper around `RankedTensorType`, which has three goals: More...

struct	SparsifierOptions
	Options for the "sparsifier" pipeline. More...

struct	TensorExp
	Tensor expression. Represents an MLIR expression in tensor index notation. More...

struct	LatPoint
	Lattice point. More...

class	Merger
	A class to handle all iteration lattice operations. More...

struct	Element
	An element of a sparse tensor in coordinate-scheme representation (i.e., a pair of coordinates and value). More...

struct	ElementLT
	Closure object for `operator<` on `Element` with a given rank. More...

class	SparseTensorCOO
	A memory-resident sparse tensor in coordinate-scheme representation (a collection of `Element`s). More...

class	SparseTensorReader
	This class abstracts over the information stored in file headers, as well as providing the buffers and methods for parsing those headers. More...

class	MapRef
	A class for capturing the sparse tensor type map with a compact encoding. More...

class	SparseTensorStorageBase
	Abstract base class for `SparseTensorStorage<P,C,V>`. More...

class	SparseTensorStorage
	A memory-resident sparse tensor using a storage scheme based on per-level sparse/dense annotations. More...

class	SparsificationAndBufferizationPass
	A pass that lowers tensor ops to memref ops, regardless of whether they are dense or sparse. More...

class	CodegenEnv
	The code generation environment class aggregates a number of data structures that are needed during the code generation phase of sparsification. More...

class	FuncCallOrInlineGenerator
	A helper class to simplify lowering operations with/without function calls. More...

class	IterationGraphSorter

class	LoopEmitter

class	SparseTensorSpecifier

class	SparseTensorDescriptorImpl
	A helper class around an array of values that corresponds to a sparse tensor. More...

class	SparseTensorDescriptor
	Uses ValueRange for immutable descriptors. More...

class	MutSparseTensorDescriptor
	Using SmallVector for mutable descriptor allows users to reuse it as a tmp buffers to append value for some special cases, though users should be responsible to restore the buffer to legal states after their use. More...

class	SparseTensorLevel
	The base class for all types of sparse tensor levels. More...

class	SparseIterationSpace
	A `SparseIterationSpace` represents a sparse set of coordinates defined by (possibly multiple) levels of a specific sparse tensor. More...

class	SparseIterator
	Helper class that generates loop conditions, etc, to traverse a sparse tensor level. More...

Typedefs
using	index_type = uint64_t
	This type is used in the public API at all places where MLIR expects values with the built-in type "index". More...

using	complex64 = std::complex< double >

using	complex32 = std::complex< float >

using	Dimension = uint64_t
	The type of dimension identifiers and dimension-ranks. More...

using	Level = uint64_t
	The type of level identifiers and level-ranks. More...

using	Size = int64_t
	The type for individual components of a compile-time shape, including the value `ShapedType::kDynamic` (for shapes). More...

using	FieldIndex = unsigned
	The type of field indices. More...

using	TensorId = unsigned
	Tensor identifiers, chosen to be the `BlockArgument::getArgNumber` of the value passed to `Merger::buildTensorExp`. More...

using	LoopId = unsigned
	Loop identifiers. More...

using	TensorLoopId = unsigned
	A compressed representation of `std::pair<TensorId, LoopId>`. More...

using	ExprId = unsigned
	`TensorExp` identifiers. More...

using	LatPointId = unsigned
	`LatPoint` identifiers. More...

using	LatSetId = unsigned
	`LatSet` identifiers. More...

using	LvlLTPair = std::pair< Level, LevelType >
	A pair of level and its corresponding LevelType of a tensor. More...

using	LoopCoeffPair = std::pair< LoopId, unsigned >
	A pair of loop id and its coefficients. More...

using	TensorLevel = unsigned

Enumerations
enum class	OverheadType : uint32_t { kIndex = 0 , kU64 = 1 , kU32 = 2 , kU16 = 3 , kU8 = 4 }
	Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor. More...

enum class	PrimaryType : uint32_t { kF64 = 1 , kF32 = 2 , kF16 = 3 , kBF16 = 4 , kI64 = 5 , kI32 = 6 , kI16 = 7 , kI8 = 8 , kC64 = 9 , kC32 = 10 }
	Encoding of the elemental type, for "overloading" @newSparseTensor. More...

enum class	Action : uint32_t { kEmpty = 0 , kFromReader = 1 , kPack = 2 , kSortCOOInPlace = 3 }
	The actions performed by @newSparseTensor. More...

enum class	LevelFormat : uint64_t { Undef = 0x00000000 , Dense = 0x00010000 , Batch = 0x00020000 , Compressed = 0x00040000 , Singleton = 0x00080000 , LooseCompressed = 0x00100000 , NOutOfM = 0x00200000 }
	This enum defines all supported storage format without the level properties. More...

enum class	LevelPropNonDefault : uint64_t { Nonunique = 0x0001 , Nonordered = 0x0002 , SoA = 0x0004 }
	This enum defines all the nondefault properties for storage formats. More...

enum class	SparseTensorFieldKind : uint32_t { StorageSpec = 0 , PosMemRef = static_cast<uint32_t>(StorageSpecifierKind::PosMemSize) , CrdMemRef = static_cast<uint32_t>(StorageSpecifierKind::CrdMemSize) , ValMemRef = static_cast<uint32_t>(StorageSpecifierKind::ValMemSize) }
	===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields. More...

enum class	EmitCInterface : bool { Off = false , On = true }
	Shorthand aliases for the `emitCInterface` argument to `getFunc()`, `createFuncCall()`, and `replaceOpWithFuncCall()`. More...

enum class	SortMask : unsigned { kIncludeDenseOutput = 0x1 , kIncludeDenseInput = 0x2 , kIncludeAll = 0x7 , kIncludeDense = 0x3 , kSparseOnly = 0x0 }
	Iteration graph sorting mask,. More...

enum class	IterKind : uint8_t { kTrivial , kDedup , kSubSect , kNonEmptySubSect , kFilter , kPad }

enum class	ExpArity { kNullary , kUnary , kBinary }

Functions
constexpr bool	isFloatingPrimaryType (PrimaryType valTy)

constexpr bool	isIntegralPrimaryType (PrimaryType valTy)

constexpr bool	isRealPrimaryType (PrimaryType valTy)

constexpr bool	isComplexPrimaryType (PrimaryType valTy)

constexpr bool	encPowOfTwo (LevelFormat fmt)

template<LevelFormat... targets>
constexpr bool	isAnyOfFmt (LevelFormat fmt)

constexpr const char *	toFormatString (LevelFormat lvlFmt)
	Returns string representation of the given level format. More...

constexpr const char *	toPropString (LevelPropNonDefault lvlProp)
	Returns string representation of the given level properties. More...

constexpr uint64_t	nToBits (uint64_t n)

constexpr uint64_t	mToBits (uint64_t m)

std::optional< LevelType >	buildLevelType (LevelFormat lf, const std::vector< LevelPropNonDefault > &properties, uint64_t n=0, uint64_t m=0)

std::optional< LevelType >	buildLevelType (LevelFormat lf, bool ordered, bool unique, uint64_t n=0, uint64_t m=0)

bool	isUndefLT (LevelType lt)

bool	isDenseLT (LevelType lt)

bool	isBatchLT (LevelType lt)

bool	isCompressedLT (LevelType lt)

bool	isLooseCompressedLT (LevelType lt)

bool	isSingletonLT (LevelType lt)

bool	isNOutOfMLT (LevelType lt)

bool	isOrderedLT (LevelType lt)

bool	isUniqueLT (LevelType lt)

bool	isWithCrdLT (LevelType lt)

bool	isWithPosLT (LevelType lt)

bool	isValidLT (LevelType lt)

std::optional< LevelFormat >	getLevelFormat (LevelType lt)

uint64_t	getN (LevelType lt)

uint64_t	getM (LevelType lt)

bool	isValidNOutOfMLT (LevelType lt, uint64_t n, uint64_t m)

std::string	toMLIRString (LevelType lt)

constexpr uint64_t	encodeDim (uint64_t i, uint64_t cf, uint64_t cm)
	Bit manipulations for affine encoding. More...

constexpr uint64_t	encodeLvl (uint64_t i, uint64_t c, uint64_t ii)

constexpr bool	isEncodedFloor (uint64_t v)

constexpr bool	isEncodedMod (uint64_t v)

constexpr bool	isEncodedMul (uint64_t v)

constexpr uint64_t	decodeIndex (uint64_t v)

constexpr uint64_t	decodeConst (uint64_t v)

constexpr uint64_t	decodeMulc (uint64_t v)

constexpr uint64_t	decodeMuli (uint64_t v)

template<typename T >
RankedTensorType	getRankedTensorType (T &&t)
	Convenience method to abbreviate casting `getType()`. More...

template<typename T >
MemRefType	getMemRefType (T &&t)
	Convenience method to abbreviate casting `getType()`. More...

SparseTensorEncodingAttr	getSparseTensorEncoding (Type type)
	Convenience method to get a sparse encoding attribute from a type. More...

bool	hasAnySparseType (TypeRange types)
	Returns true iff the type range has any sparse tensor type. More...

bool	hasAnySparseOperand (Operation *op)
	Returns true iff MLIR operand has any sparse operand. More...

bool	hasAnySparseResult (Operation *op)
	Returns true iff MLIR operand has any sparse result. More...

bool	hasAnySparseOperandOrResult (Operation *op)
	Returns true iff MLIR operand has any sparse operand or result. More...

bool	hasAnyNonIdentityOperandsOrResults (Operation *op)
	Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps. More...

AffineMap	inferLvlToDim (AffineMap dimToLvl, MLIRContext *context)
	Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails. More...

AffineMap	inverseBlockSparsity (AffineMap dimToLvl, MLIRContext *context)
	Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases. More...

SmallVector< unsigned >	getBlockSize (AffineMap dimToLvl)
	Given the dimToLvl map, returns the block sizes in a vector. More...

bool	isBlockSparsity (AffineMap dimToLvl)
	Given the dimToLvl map, returns if it's block sparsity. More...

Dimension	toDim (SparseTensorEncodingAttr enc, Level l)
	Convenience method to translate the given level to the corresponding dimension. More...

Level	toLvl (SparseTensorEncodingAttr enc, Dimension d)
	Convenience method to translate the given dimension to the corresponding level. More...

StorageSpecifierKind	toSpecifierKind (SparseTensorFieldKind kind)

SparseTensorFieldKind	toFieldKind (StorageSpecifierKind kind)

unsigned	getNumFieldsFromEncoding (SparseTensorEncodingAttr enc)

unsigned	getNumDataFieldsFromEncoding (SparseTensorEncodingAttr enc)

void	foreachFieldInSparseTensor (SparseTensorEncodingAttr enc, llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)> callback)

void	foreachFieldAndTypeInSparseTensor (SparseTensorType, llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>)

SparseTensorType	getSparseTensorType (Value val)
	Convenience methods to obtain a SparseTensorType from a Value. More...

std::optional< SparseTensorType >	tryGetSparseTensorType (Value val)

void	buildSparsifier (OpPassManager &pm, const SparsifierOptions &options)
	Adds the "sparsifier" pipeline to the `OpPassManager`. More...

void	registerSparseTensorPipelines ()
	Registers all pipelines for the `sparse_tensor` dialect. More...

void	registerTransformDialectExtension (DialectRegistry &registry)

void	registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)

llvm::hash_code	hash_value (LevelType lt)

static bool	containsSparseTensor (TypeRange types)
	Return `true` if one of the given types is a sparse tensor type. More...

OverheadType	overheadTypeEncoding (unsigned width)
	Converts an overhead storage bitwidth to its internal type-encoding. More...

OverheadType	overheadTypeEncoding (Type tp)
	Converts an overhead storage type to its internal type-encoding. More...

Type	getOverheadType (Builder &builder, OverheadType ot)
	Converts the internal type-encoding for overhead storage to an mlir::Type. More...

OverheadType	posTypeEncoding (SparseTensorEncodingAttr enc)
	Returns the OverheadType for position overhead storage. More...

OverheadType	crdTypeEncoding (SparseTensorEncodingAttr enc)
	Returns the OverheadType for coordinate overhead storage. More...

StringRef	overheadTypeFunctionSuffix (OverheadType ot)
	Convert OverheadType to its function-name suffix. More...

StringRef	overheadTypeFunctionSuffix (Type overheadTp)
	Converts an overhead storage type to its function-name suffix. More...

PrimaryType	primaryTypeEncoding (Type elemTp)
	Converts a primary storage type to its internal type-encoding. More...

StringRef	primaryTypeFunctionSuffix (PrimaryType pt)
	Convert PrimaryType to its function-name suffix. More...

StringRef	primaryTypeFunctionSuffix (Type elemTp)
	Converts a primary storage type to its function-name suffix. More...

Value	genCast (OpBuilder &builder, Location loc, Value value, Type dstTy)
	Add type casting between arith and index types when needed. More...

Value	genScalarToTensor (OpBuilder &builder, Location loc, Value elem, Type dstTp)
	Add conversion from scalar to given type (possibly a 0-rank tensor). More...

Value	genIndexLoad (OpBuilder &builder, Location loc, Value mem, ValueRange s)
	Generates a pointer/index load from the sparse storage scheme. More...

TypedAttr	getOneAttr (Builder &builder, Type tp)
	Generates a 1-valued attribute of the given type. More...

Value	genIsNonzero (OpBuilder &builder, Location loc, Value v)
	Generates the comparison `v != 0` where `v` is of numeric type. More...

void	genReshapeDstShape (OpBuilder &builder, Location loc, SmallVectorImpl< Value > &dstShape, ArrayRef< Value > srcShape, ArrayRef< Size > staticDstShape, ArrayRef< ReassociationIndices > reassociation)
	Computes the shape of destination tensor of a reshape operator. More...

void	reshapeCvs (OpBuilder &builder, Location loc, ArrayRef< ReassociationIndices > reassociation, ValueRange srcSizes, ValueRange srcCvs, ValueRange dstSizes, SmallVectorImpl< Value > &dstCvs)
	Reshape coordinates during a reshaping operation. More...

FlatSymbolRefAttr	getFunc (ModuleOp module, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
	Returns a function reference (first hit also inserts into module). More...

func::CallOp	createFuncCall (OpBuilder &builder, Location loc, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface)
	Creates a `CallOp` to the function reference returned by `getFunc()` in the builder's module. More...

Type	getOpaquePointerType (MLIRContext *ctx)
	Returns the equivalent of `void*` for opaque arguments to the execution engine. More...

Type	getOpaquePointerType (Builder &builder)

Value	genAlloca (OpBuilder &builder, Location loc, Value sz, Type tp)
	Generates an uninitialized temporary buffer of the given size and type, but returns it as type `memref<? x $tp>` (rather than as type `memref<$sz x $tp>`). More...

Value	genAlloca (OpBuilder &builder, Location loc, unsigned sz, Type tp, bool staticShape=false)
	Generates an uninitialized temporary buffer of the given size and type, and returns it as type `memref<? x $tp>` (staticShape=false) or `memref<$sz x $tp>` (staticShape=true). More...

Value	genAllocaScalar (OpBuilder &builder, Location loc, Type tp)
	Generates an uninitialized temporary buffer with room for one value of the given type, and returns the `memref<$tp>`. More...

Value	allocaBuffer (OpBuilder &builder, Location loc, ValueRange values)
	Generates a temporary buffer, initializes it with the given contents, and returns it as type `memref<? x $tp>` (rather than specifying the size of the buffer). More...

Value	allocDenseTensor (OpBuilder &builder, Location loc, RankedTensorType tensorTp, ValueRange sizes)
	Generates code to allocate a buffer of the given type, and zero initialize it. More...

void	deallocDenseTensor (OpBuilder &builder, Location loc, Value buffer)
	Generates code to deallocate a dense buffer. More...

void	sizesFromSrc (OpBuilder &builder, SmallVectorImpl< Value > &sizes, Location loc, Value src)
	Populates given sizes array from dense tensor or sparse tensor constant. More...

Operation *	getTop (Operation *op)
	Scans to top of generated loop. More...

void	foreachInSparseConstant (OpBuilder &builder, Location loc, SparseElementsAttr attr, AffineMap order, function_ref< void(ArrayRef< Value >, Value)> callback)
	Iterate over a sparse constant, generates constantOp for value and coordinates. More...

SmallVector< Value >	loadAll (OpBuilder &builder, Location loc, size_t size, Value mem, size_t offsetIdx=0, Value offsetVal=Value())
	Loads `size`-many values from the memref, which must have rank-1 and size greater-or-equal to `size`. More...

void	storeAll (OpBuilder &builder, Location loc, Value mem, ValueRange vs, size_t offsetIdx=0, Value offsetVal=Value())
	Stores all the values of `vs` into the memref `mem`, which must have rank-1 and size greater-or-equal to `vs.size()`. More...

TypedValue< BaseMemRefType >	genToMemref (OpBuilder &builder, Location loc, Value tensor)

Value	createOrFoldSliceOffsetOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
	Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known. More...

Value	createOrFoldSliceStrideOp (OpBuilder &builder, Location loc, Value tensor, Dimension dim)
	Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known. More...

Value	genReader (OpBuilder &builder, Location loc, SparseTensorType stt, Value tensor, SmallVectorImpl< Value > &dimSizesValues, Value &dimSizesBuffer)
	Generates code that opens a reader and sets the dimension sizes. More...

Value	genMapBuffers (OpBuilder &builder, Location loc, SparseTensorType stt, ArrayRef< Value > dimSizesValues, Value dimSizesBuffer, SmallVectorImpl< Value > &lvlSizesValues, Value &dim2lvlBuffer, Value &lvl2dimBuffer)
	Generates code to set up the buffer parameters for a map. More...

Value	constantZero (OpBuilder &builder, Location loc, Type tp)
	Generates a 0-valued constant of the given type. More...

Value	constantOne (OpBuilder &builder, Location loc, Type tp)
	Generates a 1-valued constant of the given type. More...

Value	constantIndex (OpBuilder &builder, Location loc, int64_t i)
	Generates a constant of `index` type. More...

Value	constantI64 (OpBuilder &builder, Location loc, int64_t i)
	Generates a constant of `i64` type. More...

Value	constantI32 (OpBuilder &builder, Location loc, int32_t i)
	Generates a constant of `i32` type. More...

Value	constantI16 (OpBuilder &builder, Location loc, int16_t i)
	Generates a constant of `i16` type. More...

Value	constantI8 (OpBuilder &builder, Location loc, int8_t i)
	Generates a constant of `i8` type. More...

Value	constantI1 (OpBuilder &builder, Location loc, bool b)
	Generates a constant of `i1` type. More...

Value	constantAction (OpBuilder &builder, Location loc, Action action)
	Generates a constant of the given `Action`. More...

Value	constantOverheadTypeEncoding (OpBuilder &builder, Location loc, unsigned width)
	Generates a constant of the internal type-encoding for overhead storage. More...

Value	constantPosTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
	Generates a constant of the internal type-encoding for position overhead storage. More...

Value	constantCrdTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc)
	Generates a constant of the internal type-encoding for coordinate overhead storage. More...

Value	constantPrimaryTypeEncoding (OpBuilder &builder, Location loc, Type elemTp)
	Generates a constant of the internal type-encoding for primary storage. More...

Value	constantLevelTypeEncoding (OpBuilder &builder, Location loc, LevelType lt)
	Generates a constant of the internal dimension level type encoding. More...

Value	genValFromAttr (OpBuilder &builder, Location loc, Attribute attr)

bool	isZeroRankedTensorOrScalar (Type type)

std::pair< Operation *, Value >	genCoIteration (OpBuilder &builder, Location loc, ArrayRef< SparseIterator * > iters, MutableArrayRef< Value > reduc, Value uniIdx, bool userReducFirst=false)

Value	genTuple (OpBuilder &builder, Location loc, Type tp, ValueRange values)
	Packs the given values as a "tuple" value. More...

Value	genTuple (OpBuilder &builder, Location loc, SparseTensorDescriptor desc)

SparseTensorDescriptor	getDescriptorFromTensorTuple (ValueRange adaptorValues, RankedTensorType type)

MutSparseTensorDescriptor	getMutDescriptorFromTensorTuple (ValueRange adaptorValues, SmallVectorImpl< Value > &fields, RankedTensorType type)

std::unique_ptr< SparseTensorLevel >	makeSparseTensorLevel (OpBuilder &b, Location l, Value t, unsigned tid, Level lvl)
	Helper function to create a TensorLevel object from given `tensor`. More...

std::unique_ptr< SparseTensorLevel >	makeSparseTensorLevel (LevelType lt, Value sz, ValueRange buffers, unsigned tid, Level l)
	Helper function to create a TensorLevel object from given ValueRange. More...

std::unique_ptr< SparseIterator >	makeSimpleIterator (OpBuilder &b, Location l, const SparseIterationSpace &iterSpace)
	Helper function to create a simple SparseIterator object that iterate over the entire iteration space. More...

std::unique_ptr< SparseIterator >	makeSimpleIterator (const SparseTensorLevel &stl, SparseEmitStrategy strategy=SparseEmitStrategy::kFunctional)
	Helper function to create a simple SparseIterator object that iterate over the sparse tensor level. More...

std::pair< std::unique_ptr< SparseTensorLevel >, std::unique_ptr< SparseIterator > >	makeSynLevelAndIterator (Value sz, unsigned tid, unsigned lvl, SparseEmitStrategy strategy)
	Helper function to create a synthetic SparseIterator object that iterates over a dense space specified by [0,`sz`). More...

std::unique_ptr< SparseIterator >	makeSlicedLevelIterator (std::unique_ptr< SparseIterator > &&sit, Value offset, Value stride, Value size, SparseEmitStrategy strategy)
	Helper function to create a SparseIterator object that iterates over a sliced space, the orignal space (before slicing) is traversed by `sit`. More...

std::unique_ptr< SparseIterator >	makePaddedIterator (std::unique_ptr< SparseIterator > &&sit, Value padLow, Value padHigh, SparseEmitStrategy strategy)
	Helper function to create a SparseIterator object that iterates over a padded sparse level (the padded value must be zero). More...

std::unique_ptr< SparseIterator >	makeNonEmptySubSectIterator (OpBuilder &b, Location l, const SparseIterator *parent, Value loopBound, std::unique_ptr< SparseIterator > &&delegate, Value size, unsigned stride, SparseEmitStrategy strategy)
	Helper function to create a SparseIterator object that iterate over the non-empty subsections set. More...

std::unique_ptr< SparseIterator >	makeTraverseSubSectIterator (OpBuilder &b, Location l, const SparseIterator &subsectIter, const SparseIterator &parent, std::unique_ptr< SparseIterator > &&wrap, Value loopBound, unsigned stride, SparseEmitStrategy strategy)
	Helper function to create a SparseIterator object that iterates over a non-empty subsection created by NonEmptySubSectIterator. More...

static ExpArity	getExpArity (TensorExp::Kind k)

static const char *	kindToOpSymbol (TensorExp::Kind kind)

static bool	isCertainZero (Value val)
	Only returns true if we are certain this is a zero. More...

static bool	isAdmissibleBranchExp (Operation op, Block block, Value v)
	Ensures that the sparsifier can generate code for expression. More...

static bool	isAdmissibleBranch (Operation *op, Region &region)
	Ensures that the sparsifier can generate code for branch. More...

static bool	isGreater (TensorExp::Kind kind, Attribute attr)

static Value	insertYieldOp (RewriterBase &rewriter, Location loc, Region &region, ValueRange vals)

static Value	buildUnaryPresent (RewriterBase &rewriter, Location loc, Operation *op, Value v0)

static Value	buildBinaryOverlap (RewriterBase &rewriter, Location loc, Operation *op, Value v0, Value v1)

static Value	buildRelu (RewriterBase &rewriter, Location loc, Value v0, Attribute attr)

Typedef Documentation

◆ complex32

using mlir::sparse_tensor::complex32 = typedef std::complex<float>

Definition at line 79 of file Enums.h.

◆ complex64

using mlir::sparse_tensor::complex64 = typedef std::complex<double>

Definition at line 78 of file Enums.h.

◆ Dimension

using mlir::sparse_tensor::Dimension = typedef uint64_t

The type of dimension identifiers and dimension-ranks.

Definition at line 39 of file SparseTensor.h.

◆ ExprId

using mlir::sparse_tensor::ExprId = typedef unsigned

TensorExp identifiers.

These are allocated by Merger::addExp, and serve as unique identifiers for the corresponding TensorExp object.

Definition at line 48 of file Merger.h.

◆ FieldIndex

using mlir::sparse_tensor::FieldIndex = typedef unsigned

The type of field indices.

This alias is to help code be more self-documenting; unfortunately it is not type-checked, so it only provides documentation rather than doing anything to prevent mixups.

Definition at line 110 of file SparseTensorStorageLayout.h.

◆ index_type

using mlir::sparse_tensor::index_type = typedef uint64_t

This type is used in the public API at all places where MLIR expects values with the built-in type "index".

For now, we simply assume that type is 64-bit, but targets with different "index" bitwidths should link with an alternatively built runtime support library.

Definition at line 47 of file Enums.h.

◆ LatPointId

using mlir::sparse_tensor::LatPointId = typedef unsigned

LatPoint identifiers.

These are allocated by Merger::addLat, and serve as unique identifiers for the corresponding LatPoint object.

Definition at line 52 of file Merger.h.

◆ LatSetId

using mlir::sparse_tensor::LatSetId = typedef unsigned

LatSet identifiers.

These are allocated by Merger::addSet (and by other methods calling that one), and serve as unique identifiers for the corresponding SmallVector<LatPointId> object.

Definition at line 57 of file Merger.h.

◆ Level

using mlir::sparse_tensor::Level = typedef uint64_t

The type of level identifiers and level-ranks.

Definition at line 42 of file SparseTensor.h.

◆ LoopCoeffPair

using mlir::sparse_tensor::LoopCoeffPair = typedef std::pair<LoopId, unsigned>

A pair of loop id and its coefficients.

E.g., for affine expression in the affine map 2 * d0, loop id = 0, coefficient = 2.

Definition at line 64 of file Merger.h.

◆ LoopId

using mlir::sparse_tensor::LoopId = typedef unsigned

Loop identifiers.

Definition at line 38 of file Merger.h.

◆ LvlLTPair

using mlir::sparse_tensor::LvlLTPair = typedef std::pair<Level, LevelType>

A pair of level and its corresponding LevelType of a tensor.

Definition at line 60 of file Merger.h.

◆ Size

using mlir::sparse_tensor::Size = typedef int64_t

The type for individual components of a compile-time shape, including the value ShapedType::kDynamic (for shapes).

Definition at line 46 of file SparseTensor.h.

◆ TensorId

using mlir::sparse_tensor::TensorId = typedef unsigned

Tensor identifiers, chosen to be the BlockArgument::getArgNumber of the value passed to Merger::buildTensorExp.

Definition at line 35 of file Merger.h.

◆ TensorLevel

using mlir::sparse_tensor::TensorLevel = typedef unsigned

Definition at line 26 of file LoopEmitter.h.

◆ TensorLoopId

using mlir::sparse_tensor::TensorLoopId = typedef unsigned

A compressed representation of std::pair<TensorId, LoopId>.

The compression scheme is such that this also serves as an index into the bitvector stored in LatPoint (since that bitvector is just the implementation for a set of TensorLoopId values).

Definition at line 44 of file Merger.h.

Enumeration Type Documentation

◆ Action

enum mlir::sparse_tensor::Action : uint32_t

strong

The actions performed by @newSparseTensor.

Enumerator
kEmpty
kFromReader
kPack
kSortCOOInPlace

Definition at line 146 of file Enums.h.

◆ EmitCInterface

enum mlir::sparse_tensor::EmitCInterface : bool

strong

Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall().

Enumerator
Off
On

Definition at line 36 of file CodegenUtils.h.

◆ ExpArity

enum mlir::sparse_tensor::ExpArity

strong

Enumerator
kNullary
kUnary
kBinary

Definition at line 22 of file Merger.cpp.

◆ IterKind

enum mlir::sparse_tensor::IterKind : uint8_t

strong

Enumerator
kTrivial
kDedup
kSubSect
kNonEmptySubSect
kFilter
kPad

Definition at line 82 of file SparseTensorIterator.h.

◆ LevelFormat

enum mlir::sparse_tensor::LevelFormat : uint64_t

strong

This enum defines all supported storage format without the level properties.

Enumerator
Undef
Dense
Batch
Compressed
Singleton
LooseCompressed
NOutOfM

Definition at line 154 of file Enums.h.

◆ LevelPropNonDefault

enum mlir::sparse_tensor::LevelPropNonDefault : uint64_t

strong

This enum defines all the nondefault properties for storage formats.

Enumerator
Nonunique
Nonordered
SoA

Definition at line 204 of file Enums.h.

◆ OverheadType

enum mlir::sparse_tensor::OverheadType : uint32_t

strong

Encoding of overhead types (both position overhead and coordinate overhead), for "overloading" @newSparseTensor.

Enumerator
kIndex
kU64
kU32
kU16
kU8

Definition at line 51 of file Enums.h.

◆ PrimaryType

enum mlir::sparse_tensor::PrimaryType : uint32_t

strong

Encoding of the elemental type, for "overloading" @newSparseTensor.

Enumerator
kF64
kF32
kF16
kBF16
kI64
kI32
kI16
kI8
kC64
kC32

Definition at line 82 of file Enums.h.

◆ SortMask

enum mlir::sparse_tensor::SortMask : unsigned

strong

Iteration graph sorting mask,.

Enumerator
kIncludeDenseOutput
kIncludeDenseInput
kIncludeAll
kIncludeDense
kSparseOnly

Definition at line 32 of file IterationGraphSorter.h.

◆ SparseTensorFieldKind

enum mlir::sparse_tensor::SparseTensorFieldKind : uint32_t

strong

===-------------------------------------------------------------------—===// The sparse tensor storage scheme for a tensor is organized as a single compound type with the following fields.

Note that every memref with ? size actually behaves as a "vector", i.e. the stored size is the capacity and the used size resides in the storage_specifier struct.

struct { ; per-level l: ; if dense: <nothing> ; if compressed: memref<[batch] x ? x pos> positions ; positions for level l memref<[batch] x ? x crd> coordinates ; coordinates for level l ; if loose-[batch] x compressed: memref<[batch] x ? x pos> positions ; lo/hi pos pairs for level l memref<[batch] x ? x crd> coordinates ; coordinates for level l ; if singleton/2-out-of-4: memref<[batch] x ? x crd> coordinates ; coordinates for level l

memref<[batch] x ? x eltType> values ; values

struct sparse_tensor.storage_specifier { array<rank x int> lvlSizes ; sizes/cardinalities for each level // TODO: memSizes need to be expanded to array<[batch] x n x int> to // support different sizes for different batches. At the moment, we // assume that every batch occupies the same memory size. array<n x int> memSizes ; sizes/lengths for each data memref } };

In addition, for a "trailing COO region", defined as a compressed level followed by one or more singleton levels, the default SOA storage that is inherent to the TACO format is optimized into an AOS storage where all coordinates of a stored element appear consecutively. In such cases, a special operation (sparse_tensor.coordinates_buffer) must be used to access the AOS coordinates array. In the code below, the method getCOOStart is used to find the start of the "trailing COO region".

If the sparse tensor is a slice (produced by tensor.extract_slice operation), instead of allocating a new sparse tensor for it, it reuses the same sets of MemRefs but attaching a additional set of slicing-metadata for per-dimension slice offset and stride.

Examples.

#CSR storage of 2-dim matrix yields memref<?xindex> ; positions-1 memref<?xindex> ; coordinates-1 memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

#COO storage of 2-dim matrix yields memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>)>) ; lvl0, lvl1, 3xsizes

Slice on #COO storage of 2-dim matrix yields ;; Inherited from the original sparse tensors memref<?xindex>, ; positions-0, essentially [0,sz] memref<?xindex> ; AOS coordinates storage memref<?xf64> ; values struct<(array<2 x i64>, array<3 x i64>, ; lvl0, lvl1, 3xsizes ;; Extra slicing-metadata array<2 x i64>, array<2 x i64>)>) ; dim offset, dim stride.

===-------------------------------------------------------------------—===//

Enumerator
StorageSpec
PosMemRef
CrdMemRef
ValMemRef

Definition at line 90 of file SparseTensorStorageLayout.h.

Function Documentation

◆ allocaBuffer()

Value mlir::sparse_tensor::allocaBuffer	(	OpBuilder &	builder,
		Location	loc,
		ValueRange	values
	)

Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer).

Definition at line 377 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), genAlloca(), and mlir::getType().

Referenced by genMapBuffers(), and genReader().

◆ allocDenseTensor()

Value mlir::sparse_tensor::allocDenseTensor	(	OpBuilder &	builder,
		Location	loc,
		RankedTensorType	tensorTp,
		ValueRange	sizes
	)

Generates code to allocate a buffer of the given type, and zero initialize it.

If the buffer type has any dynamic sizes, then the sizes parameter should be as filled by sizesFromPtr(); that way we can reuse the genDimSizeCall() results generated by sizesFromPtr().

Definition at line 389 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), and mlir::get().

◆ buildBinaryOverlap()

static Value mlir::sparse_tensor::buildBinaryOverlap	(	RewriterBase &	rewriter,
		Location	loc,
		Operation *	op,
		Value	v0,
		Value	v1
	)

static

Definition at line 1588 of file Merger.cpp.

References mlir::Region::empty(), and insertYieldOp().

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ buildLevelType() [1/2]

std::optional<LevelType> mlir::sparse_tensor::buildLevelType	(	LevelFormat	lf,
		bool	ordered,
		bool	unique,
		uint64_t	n = `0`,
		uint64_t	m = `0`
	)

inline

Definition at line 407 of file Enums.h.

References mlir::sparse_tensor::LevelType::buildLvlType().

◆ buildLevelType() [2/2]

std::optional<LevelType> mlir::sparse_tensor::buildLevelType	(	LevelFormat	lf,
		const std::vector< LevelPropNonDefault > &	properties,
		uint64_t	n = `0`,
		uint64_t	m = `0`
	)

inline

Definition at line 402 of file Enums.h.

References mlir::sparse_tensor::LevelType::buildLvlType().

Referenced by mlir::sparse_tensor::SparseTensorType::getCOOType(), and mlirSparseTensorEncodingAttrBuildLvlType().

◆ buildRelu()

static Value mlir::sparse_tensor::buildRelu	(	RewriterBase &	rewriter,
		Location	loc,
		Value	v0,
		Attribute	attr
	)

static

Definition at line 1602 of file Merger.cpp.

References mlir::OpBuilder::create(), mlir::Value::getType(), and mlir::Builder::getZeroAttr().

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ buildSparsifier()

void mlir::sparse_tensor::buildSparsifier	(	OpPassManager &	pm,
		const SparsifierOptions &	options
	)

Adds the "sparsifier" pipeline to the OpPassManager.

This is the standard pipeline for taking sparsity-agnostic IR using the sparse-tensor type and lowering it to LLVM IR with concrete representations and algorithms for sparse tensors.

Definition at line 26 of file SparseTensorPipelines.cpp.

References mlir::OpPassManager::addNestedPass(), mlir::OpPassManager::addPass(), mlir::createCanonicalizerPass(), mlir::createConvertVectorToSCFPass(), mlir::createSparseGPUCodegenPass(), mlir::createSparsificationAndBufferizationPass(), mlir::createStorageSpecifierToLLVMPass(), mlir::createStripDebugInfoPass(), mlir::getBufferizationOptionsForSparsification(), and options.

Referenced by registerSparseTensorPipelines().

◆ buildUnaryPresent()

static Value mlir::sparse_tensor::buildUnaryPresent	(	RewriterBase &	rewriter,
		Location	loc,
		Operation *	op,
		Value	v0
	)

static

Definition at line 1574 of file Merger.cpp.

References mlir::Region::empty(), and insertYieldOp().

Referenced by mlir::sparse_tensor::Merger::buildExp().

◆ constantAction()

Value mlir::sparse_tensor::constantAction	(	OpBuilder &	builder,
		Location	loc,
		Action	action
	)

inline

Generates a constant of the given Action.

Definition at line 361 of file CodegenUtils.h.

References constantI32().

◆ constantCrdTypeEncoding()

Value mlir::sparse_tensor::constantCrdTypeEncoding	(	OpBuilder &	builder,
		Location	loc,
		SparseTensorEncodingAttr	enc
	)

inline

Generates a constant of the internal type-encoding for coordinate overhead storage.

Definition at line 381 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantI1()

Value mlir::sparse_tensor::constantI1	(	OpBuilder &	builder,
		Location	loc,
		bool	b
	)

inline

Generates a constant of i1 type.

Definition at line 356 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by createEqCompare(), createPartitionFunc(), endIf(), endLoop(), genCoIterateBranchNest(), genCompressed(), genIf(), genInsertionStore(), and genInvariants().

◆ constantI16()

Value mlir::sparse_tensor::constantI16	(	OpBuilder &	builder,
		Location	loc,
		int16_t	i
	)

inline

Generates a constant of i16 type.

Definition at line 346 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantI32()

Value mlir::sparse_tensor::constantI32	(	OpBuilder &	builder,
		Location	loc,
		int32_t	i
	)

inline

Generates a constant of i32 type.

Definition at line 341 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantAction(), constantOverheadTypeEncoding(), and constantPrimaryTypeEncoding().

◆ constantI64()

Value mlir::sparse_tensor::constantI64	(	OpBuilder &	builder,
		Location	loc,
		int64_t	i
	)

inline

Generates a constant of i64 type.

Definition at line 336 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by constantLevelTypeEncoding(), createQuickSortFunc(), and matchAndRewriteSortOp().

◆ constantI8()

Value mlir::sparse_tensor::constantI8	(	OpBuilder &	builder,
		Location	loc,
		int8_t	i
	)

inline

Generates a constant of i8 type.

Definition at line 351 of file CodegenUtils.h.

References mlir::OpBuilder::create().

◆ constantIndex()

Value mlir::sparse_tensor::constantIndex	(	OpBuilder &	builder,
		Location	loc,
		int64_t	i
	)

inline

Generates a constant of index type.

Definition at line 331 of file CodegenUtils.h.

References mlir::OpBuilder::create().

Referenced by allocaBuffer(), allocSchemeForRank(), concatSizesFromInputs(), createAllocFields(), createBinarySearchFunc(), createChoosePivot(), createDimSizes(), createHeapSortFunc(), createOrFoldSliceOffsetOp(), createOrFoldSliceStrideOp(), createPartitionFunc(), createQuickSort(), createQuickSortFunc(), createScanLoop(), createShiftDownFunc(), createSortStableFunc(), createSubTwoDividedByTwo(), forEachIJPairInXs(), genAlloca(), genCompressed(), genEndInsert(), genInsertionStore(), genLoopWithIterator(), genMapBuffers(), genReader(), genReshapeDstShape(), mlir::sparse_tensor::SparseTensorDescriptor::getCrdMemRefOrView(), loadAll(), matchAndRewriteSortOp(), parseGEPIndices(), reshapeCvs(), sizesForTensor(), specializeForLoopForUnrolling(), specializeParallelLoopForUnrolling(), storeAll(), and unFoldOpIntResult().

◆ constantLevelTypeEncoding()

Value mlir::sparse_tensor::constantLevelTypeEncoding	(	OpBuilder &	builder,
		Location	loc,
		LevelType	lt
	)

inline

Generates a constant of the internal dimension level type encoding.

Definition at line 394 of file CodegenUtils.h.

References constantI64().

◆ constantOne()

Value mlir::sparse_tensor::constantOne	(	OpBuilder &	builder,
		Location	loc,
		Type	tp
	)

inline

Generates a 1-valued constant of the given type.

This supports all the same types as constantZero.

Definition at line 320 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), getOneAttr(), and mlir::Builder::getZeroAttr().

Referenced by createFor(), and createPartitionFunc().

◆ constantOverheadTypeEncoding()

Value mlir::sparse_tensor::constantOverheadTypeEncoding	(	OpBuilder &	builder,
		Location	loc,
		unsigned	width
	)

inline

Generates a constant of the internal type-encoding for overhead storage.

Definition at line 366 of file CodegenUtils.h.

References constantI32(), and overheadTypeEncoding().

Referenced by constantCrdTypeEncoding(), and constantPosTypeEncoding().

◆ constantPosTypeEncoding()

Value mlir::sparse_tensor::constantPosTypeEncoding	(	OpBuilder &	builder,
		Location	loc,
		SparseTensorEncodingAttr	enc
	)

inline

Generates a constant of the internal type-encoding for position overhead storage.

Definition at line 374 of file CodegenUtils.h.

References constantOverheadTypeEncoding().

◆ constantPrimaryTypeEncoding()

Value mlir::sparse_tensor::constantPrimaryTypeEncoding	(	OpBuilder &	builder,
		Location	loc,
		Type	elemTp
	)

inline

Generates a constant of the internal type-encoding for primary storage.

Definition at line 387 of file CodegenUtils.h.

References constantI32(), and primaryTypeEncoding().

Referenced by genReader().

◆ constantZero()

Value mlir::sparse_tensor::constantZero	(	OpBuilder &	builder,
		Location	loc,
		Type	tp
	)

inline

Generates a 0-valued constant of the given type.

In addition to the scalar types (ComplexType, FloatType, IndexType, IntegerType), this also works for RankedTensorType and VectorType (for which it generates a constant DenseElementsAttr of zeros).

Definition at line 309 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::Builder::getArrayAttr(), and mlir::Builder::getZeroAttr().

Referenced by allocDenseTensor(), allocSchemeForRank(), createAllocation(), createAllocFields(), createFor(), genBuffers(), genEndInsert(), genExp(), genInsertionLoad(), and genIsNonzero().

◆ containsSparseTensor()

static bool mlir::sparse_tensor::containsSparseTensor ( TypeRange types )

static

Return true if one of the given types is a sparse tensor type.

Definition at line 40 of file SparsificationAndBufferizationPass.cpp.

References getSparseTensorEncoding().

Referenced by mlir::sparse_tensor::SparsificationAndBufferizationPass::runDenseBufferization().

◆ crdTypeEncoding()

OverheadType mlir::sparse_tensor::crdTypeEncoding ( SparseTensorEncodingAttr enc )

Returns the OverheadType for coordinate overhead storage.

Definition at line 73 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ createFuncCall()

func::CallOp mlir::sparse_tensor::createFuncCall	(	OpBuilder &	builder,
		Location	loc,
		StringRef	name,
		TypeRange	resultType,
		ValueRange	operands,
		EmitCInterface	emitCInterface
	)

Creates a CallOp to the function reference returned by getFunc() in the builder's module.

Definition at line 340 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::OpBuilder::getBlock(), getFunc(), mlir::Operation::getParentOfType(), and mlir::Block::getParentOp().

Referenced by dumpIndexMemRef(), and genReader().

◆ createOrFoldSliceOffsetOp()

Value mlir::sparse_tensor::createOrFoldSliceOffsetOp	(	OpBuilder &	builder,
		Location	loc,
		Value	tensor,
		Dimension	dim
	)

Generates code to retrieve the slice offset for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 553 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceOffset().

◆ createOrFoldSliceStrideOp()

Value mlir::sparse_tensor::createOrFoldSliceStrideOp	(	OpBuilder &	builder,
		Location	loc,
		Value	tensor,
		Dimension	dim
	)

Generates code to retrieve the slice slice for the sparse tensor slice, return a constant if the offset is statically known.

Definition at line 563 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), getSparseTensorEncoding(), and mlir::Value::getType().

Referenced by genSliceStride().

◆ deallocDenseTensor()

void mlir::sparse_tensor::deallocDenseTensor	(	OpBuilder &	builder,
		Location	loc,
		Value	buffer
	)

Generates code to deallocate a dense buffer.

Definition at line 406 of file CodegenUtils.cpp.

References mlir::OpBuilder::create().

◆ decodeConst()

constexpr uint64_t mlir::sparse_tensor::decodeConst ( uint64_t v )

constexpr

Definition at line 495 of file Enums.h.

◆ decodeIndex()

constexpr uint64_t mlir::sparse_tensor::decodeIndex ( uint64_t v )

constexpr

Definition at line 494 of file Enums.h.

◆ decodeMulc()

constexpr uint64_t mlir::sparse_tensor::decodeMulc ( uint64_t v )

constexpr

Definition at line 496 of file Enums.h.

◆ decodeMuli()

constexpr uint64_t mlir::sparse_tensor::decodeMuli ( uint64_t v )

constexpr

Definition at line 497 of file Enums.h.

◆ encodeDim()

constexpr uint64_t mlir::sparse_tensor::encodeDim	(	uint64_t	i,
		uint64_t	cf,
		uint64_t	cm
	)

constexpr

Bit manipulations for affine encoding.

Note that because the indices in the mappings refer to dimensions and levels (and not the sizes of these dimensions and levels), the 64-bit encoding gives ample room for a compact encoding of affine operations in the higher bits. Pure permutations still allow for 60-bit indices. But non-permutations reserve 20-bits for the potential three components (index i, constant, index ii).

The compact encoding is as follows:

This encoding provides sufficient generality for currently supported sparse tensor types. To generalize this more, we will need to provide a broader encoding scheme for affine functions. Also, the library encoding may be replaced with pure "direct-IR" code in the future.

Definition at line 471 of file Enums.h.

Referenced by genMapBuffers().

◆ encodeLvl()

constexpr uint64_t mlir::sparse_tensor::encodeLvl	(	uint64_t	i,
		uint64_t	c,
		uint64_t	ii
	)

constexpr

Definition at line 483 of file Enums.h.

Referenced by genMapBuffers().

◆ encPowOfTwo()

constexpr bool mlir::sparse_tensor::encPowOfTwo ( LevelFormat fmt )

constexpr

Definition at line 164 of file Enums.h.

◆ foreachFieldAndTypeInSparseTensor()

void mlir::sparse_tensor::foreachFieldAndTypeInSparseTensor	(	SparseTensorType	stt,
		llvm::function_ref< bool(Type, FieldIndex, SparseTensorFieldKind, Level, LevelType)>	callback
	)

Definition at line 147 of file SparseTensorDialect.cpp.

References CrdMemRef, mlir::sparse_tensor::StorageLayout::foreachField(), mlir::get(), mlir::sparse_tensor::SparseTensorType::getCrdType(), mlir::sparse_tensor::SparseTensorType::getDimShape(), mlir::sparse_tensor::SparseTensorType::getElementType(), mlir::sparse_tensor::SparseTensorType::getEncoding(), mlir::sparse_tensor::SparseTensorType::getPosType(), getSparseFieldShape(), mlir::sparse_tensor::SparseTensorType::hasEncoding(), PosMemRef, StorageSpec, and ValMemRef.

Referenced by convertSparseTensorType(), convTypes(), convVals(), and createAllocFields().

◆ foreachFieldInSparseTensor()

void mlir::sparse_tensor::foreachFieldInSparseTensor	(	SparseTensorEncodingAttr	enc,
		llvm::function_ref< bool(FieldIndex, SparseTensorFieldKind, Level, LevelType)>	callback
	)

inline

Definition at line 168 of file SparseTensorStorageLayout.h.

References mlir::sparse_tensor::StorageLayout::foreachField().

◆ foreachInSparseConstant()

void mlir::sparse_tensor::foreachInSparseConstant	(	OpBuilder &	builder,
		Location	loc,
		SparseElementsAttr	attr,
		AffineMap	order,
		function_ref< void(ArrayRef< Value >, Value)>	callback
	)

Iterate over a sparse constant, generates constantOp for value and coordinates.

E.g., sparse<[ [0], [28], [31] ], [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] > => c1 = arith.constant 0 v1 = complex.constant (5.13, 2.0) callback({c1}, v1)

c2 = arith.constant 28 v2 = complex.constant (3.0, 4.0) callback({c2}, v2)

c3 = arith.constant 31 v3 = complex.constant (5.0, 6.0) callback({c3}, v3)

Definition at line 429 of file CodegenUtils.cpp.

References mlir::AffineMap::compose(), mlir::OpBuilder::create(), mlir::Builder::getMultiDimIdentityMap(), mlir::AffineMap::getNumResults(), and getRankedTensorType().

Referenced by genForeachOnSparseConstant().

◆ genAlloca() [1/2]

Value mlir::sparse_tensor::genAlloca	(	OpBuilder &	builder,
		Location	loc,
		unsigned	sz,
		Type	tp,
		bool	staticShape = `false`
	)

Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true).

Definition at line 357 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), genAlloca(), and mlir::get().

◆ genAlloca() [2/2]

Value mlir::sparse_tensor::genAlloca	(	OpBuilder &	builder,
		Location	loc,
		Value	sz,
		Type	tp
	)

Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>).

Definition at line 366 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

Referenced by allocaBuffer(), and genAlloca().

◆ genAllocaScalar()

Value mlir::sparse_tensor::genAllocaScalar	(	OpBuilder &	builder,
		Location	loc,
		Type	tp
	)

Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>.

Definition at line 372 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and mlir::get().

◆ genCast()

Value mlir::sparse_tensor::genCast	(	OpBuilder &	builder,
		Location	loc,
		Value	value,
		Type	dstTy
	)

Add type casting between arith and index types when needed.

Definition at line 148 of file CodegenUtils.cpp.

References mlir::convertScalarToDtype(), mlir::OpBuilder::create(), and mlir::Value::getType().

Referenced by createPushback(), genCompressed(), genLoad(), genScalarToTensor(), and genStore().

◆ genCoIteration()

std::pair< Operation *, Value > mlir::sparse_tensor::genCoIteration	(	OpBuilder &	builder,
		Location	loc,
		ArrayRef< SparseIterator * >	iters,
		MutableArrayRef< Value >	reduc,
		Value	uniIdx,
		bool	userReducFirst = `false`
	)

Definition at line 900 of file LoopEmitter.cpp.

References ANDI, CMPI, mlir::OpBuilder::create(), mlir::OpBuilder::createBlock(), mlir::sparse_tensor::SparseIterator::deref(), mlir::sparse_tensor::SparseIterator::genWhileCond(), mlir::Block::getArguments(), mlir::sparse_tensor::SparseIterator::getCrd(), mlir::sparse_tensor::SparseIterator::getCursor(), mlir::ValueRange::getTypes(), mlir::sparse_tensor::SparseIterator::linkNewScope(), min(), SELECT, and mlir::OpBuilder::setInsertionPointToStart().

Referenced by genLoop().

◆ genIndexLoad()

Value mlir::sparse_tensor::genIndexLoad	(	OpBuilder &	builder,
		Location	loc,
		Value	mem,
		ValueRange	s
	)

Generates a pointer/index load from the sparse storage scheme.

Narrower data types need to be zero extended before casting the value into the index type used for looping and indexing.

Definition at line 174 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::Builder::getI64Type(), mlir::Builder::getIndexType(), mlir::Type::getIntOrFloatBitWidth(), and mlir::Value::getType().

◆ genIsNonzero()

Value mlir::sparse_tensor::genIsNonzero	(	OpBuilder &	builder,
		mlir::Location	loc,
		Value	v
	)

Generates the comparison v != 0 where v is of numeric type.

For floating types, we use the "unordered" comparator (i.e., returns true if v is NaN).

Definition at line 201 of file CodegenUtils.cpp.

References constantZero(), mlir::OpBuilder::create(), mlir::Value::getType(), and mlir::Type::isIntOrIndex().

Referenced by genInsertionStore().

◆ genMapBuffers()

Value mlir::sparse_tensor::genMapBuffers	(	OpBuilder &	builder,
		Location	loc,
		SparseTensorType	stt,
		ArrayRef< Value >	dimSizesValues,
		Value	dimSizesBuffer,
		SmallVectorImpl< Value > &	lvlSizesValues,
		Value &	dim2lvlBuffer,
		Value &	lvl2dimBuffer
	)

Generates code to set up the buffer parameters for a map.

Definition at line 619 of file CodegenUtils.cpp.

References mlir::Add, allocaBuffer(), constantIndex(), mlir::OpBuilder::create(), mlir::DimId, encodeDim(), encodeLvl(), mlir::presburger::floor(), mlir::FloorDiv, mlir::sparse_tensor::SparseTensorType::getDimRank(), mlir::sparse_tensor::SparseTensorType::getDimToLvl(), mlir::AffineExpr::getKind(), mlir::sparse_tensor::SparseTensorType::getLvlRank(), mlir::sparse_tensor::SparseTensorType::getLvlToDim(), mlir::sparse_tensor::SparseTensorType::isIdentity(), mlir::Mod, and mlir::Mul.

◆ genReader()

Value mlir::sparse_tensor::genReader	(	OpBuilder &	builder,
		Location	loc,
		SparseTensorType	stt,
		Value	tensor,
		SmallVectorImpl< Value > &	dimSizesValues,
		Value &	dimSizesBuffer
	)

Generates code that opens a reader and sets the dimension sizes.

Definition at line 573 of file CodegenUtils.cpp.

References allocaBuffer(), constantIndex(), constantPrimaryTypeEncoding(), mlir::OpBuilder::create(), createFuncCall(), mlir::get(), mlir::sparse_tensor::SparseTensorType::getDimRank(), mlir::sparse_tensor::SparseTensorType::getDimShape(), mlir::sparse_tensor::SparseTensorType::getElementType(), mlir::Builder::getIndexType(), getOpaquePointerType(), mlir::sparse_tensor::SparseTensorType::hasDynamicDimShape(), mlir::sparse_tensor::SparseTensorType::isDynamicDim(), and On.

◆ genReshapeDstShape()

void mlir::sparse_tensor::genReshapeDstShape	(	OpBuilder &	builder,
		Location	loc,
		SmallVectorImpl< Value > &	dstShape,
		ArrayRef< Value >	srcShape,
		ArrayRef< Size >	staticDstShape,
		ArrayRef< ReassociationIndices >	reassociation
	)

Computes the shape of destination tensor of a reshape operator.

This is only used when operands have dynamic shape. The shape of the destination is stored into dstShape.

Definition at line 216 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and product().

◆ genScalarToTensor()

Value mlir::sparse_tensor::genScalarToTensor	(	OpBuilder &	builder,
		Location	loc,
		Value	elem,
		Type	dstTp
	)

Add conversion from scalar to given type (possibly a 0-rank tensor).

Definition at line 163 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), and genCast().

◆ genToMemref()

TypedValue< BaseMemRefType > mlir::sparse_tensor::genToMemref	(	OpBuilder &	builder,
		Location	loc,
		Value	tensor
	)

Definition at line 546 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), and mlir::Value::getType().

◆ genTuple() [1/2]

Value mlir::sparse_tensor::genTuple	(	OpBuilder &	builder,
		Location	loc,
		SparseTensorDescriptor	desc
	)

inline

Definition at line 238 of file SparseTensorDescriptor.h.

References genTuple(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getFields(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getRankedTensorType().

◆ genTuple() [2/2]

Value mlir::sparse_tensor::genTuple	(	OpBuilder &	builder,
		Location	loc,
		Type	tp,
		ValueRange	values
	)

inline

Packs the given values as a "tuple" value.

Definition at line 232 of file SparseTensorDescriptor.h.

References mlir::OpBuilder::create().

Referenced by genTuple(), and materializeTuple().

◆ genValFromAttr()

Value mlir::sparse_tensor::genValFromAttr	(	OpBuilder &	builder,
		Location	loc,
		Attribute	attr
	)

inline

Definition at line 400 of file CodegenUtils.h.

References mlir::OpBuilder::create(), mlir::get(), and mlir::Builder::getArrayAttr().

Referenced by genTensorLoad().

◆ getBlockSize()

SmallVector< unsigned > mlir::sparse_tensor::getBlockSize ( AffineMap dimToLvl )

Given the dimToLvl map, returns the block sizes in a vector.

For instance, a 2x3 block will return [2, 3]. Unblocked dimension i will return 0, and i floordiv 1, i mod 1 will return 1. Therefore, the example below will return [0, 1]. map = ( i, j ) -> ( i : dense, j floordiv 1 : compressed, j mod 1 : dense ) Only valid block sparsity will be accepted.

Definition at line 1108 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getResults(), isBlockSparsity(), and mlir::Mod.

◆ getDescriptorFromTensorTuple()

SparseTensorDescriptor mlir::sparse_tensor::getDescriptorFromTensorTuple	(	ValueRange	adaptorValues,
		RankedTensorType	type
	)

inline

Definition at line 244 of file SparseTensorDescriptor.h.

◆ getExpArity()

static ExpArity mlir::sparse_tensor::getExpArity ( TensorExp::Kind k )

static

Definition at line 28 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::expContainsTensor(), and mlir::sparse_tensor::Merger::hasNegateOnOut().

◆ getFunc()

FlatSymbolRefAttr mlir::sparse_tensor::getFunc	(	ModuleOp	module,
		StringRef	name,
		TypeRange	resultType,
		ValueRange	operands,
		EmitCInterface	emitCInterface
	)

Returns a function reference (first hit also inserts into module).

Sets the "_emit_c_interface" on the function declaration when requested, so that LLVM lowering generates a wrapper function that takes care of ABI complications with passing in and returning MemRefs to C functions.

Definition at line 320 of file CodegenUtils.cpp.

References mlir::OpBuilder::create(), mlir::get(), mlir::ValueRange::getTypes(), and mlir::Operation::setAttr().

Referenced by createFuncCall().

◆ getLevelFormat()

std::optional<LevelFormat> mlir::sparse_tensor::getLevelFormat ( LevelType lt )

inline

Definition at line 436 of file Enums.h.

References mlir::sparse_tensor::LevelType::getLvlFmt(), and Undef.

◆ getM()

uint64_t mlir::sparse_tensor::getM ( LevelType lt )

inline

Definition at line 443 of file Enums.h.

References mlir::sparse_tensor::LevelType::getM().

Referenced by mlirSparseTensorEncodingAttrGetStructuredM().

◆ getMemRefType()

template<typename T >

MemRefType mlir::sparse_tensor::getMemRefType ( T && t )

inline

Convenience method to abbreviate casting getType().

Definition at line 168 of file SparseTensor.h.

References mlir::getType().

Referenced by getMangledSortHelperFuncName(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getMemRefElementType(), isMatchingWidth(), and matchAndRewriteSortOp().

◆ getMutDescriptorFromTensorTuple()

MutSparseTensorDescriptor mlir::sparse_tensor::getMutDescriptorFromTensorTuple	(	ValueRange	adaptorValues,
		SmallVectorImpl< Value > &	fields,
		RankedTensorType	type
	)

inline

Definition at line 249 of file SparseTensorDescriptor.h.

◆ getN()

uint64_t mlir::sparse_tensor::getN ( LevelType lt )

inline

Definition at line 442 of file Enums.h.

References mlir::sparse_tensor::LevelType::getN().

Referenced by mlirSparseTensorEncodingAttrGetStructuredN().

◆ getNumDataFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumDataFieldsFromEncoding ( SparseTensorEncodingAttr enc )

inline

Definition at line 164 of file SparseTensorStorageLayout.h.

References mlir::sparse_tensor::StorageLayout::getNumDataFields().

◆ getNumFieldsFromEncoding()

unsigned mlir::sparse_tensor::getNumFieldsFromEncoding ( SparseTensorEncodingAttr enc )

inline

Definition at line 160 of file SparseTensorStorageLayout.h.

References mlir::sparse_tensor::StorageLayout::getNumFields().

◆ getOneAttr()

mlir::TypedAttr mlir::sparse_tensor::getOneAttr	(	Builder &	builder,
		Type	tp
	)

Generates a 1-valued attribute of the given type.

This supports all the same types as getZeroAttr; however, unlike getZeroAttr, for unsupported types we raise llvm_unreachable rather than returning a null attribute.

Definition at line 186 of file CodegenUtils.cpp.

References mlir::DenseElementsAttr::get(), mlir::Builder::getFloatAttr(), mlir::Builder::getIndexAttr(), and mlir::Builder::getIntegerAttr().

Referenced by constantOne().

◆ getOpaquePointerType() [1/2]

Type mlir::sparse_tensor::getOpaquePointerType ( Builder & builder )

Definition at line 353 of file CodegenUtils.cpp.

References mlir::Builder::getContext(), and getOpaquePointerType().

◆ getOpaquePointerType() [2/2]

Type mlir::sparse_tensor::getOpaquePointerType ( MLIRContext * ctx )

Returns the equivalent of void* for opaque arguments to the execution engine.

Definition at line 349 of file CodegenUtils.cpp.

References mlir::get().

Referenced by genReader(), and getOpaquePointerType().

◆ getOverheadType()

Type mlir::sparse_tensor::getOverheadType	(	Builder &	builder,
		OverheadType	ot
	)

Converts the internal type-encoding for overhead storage to an mlir::Type.

Definition at line 51 of file CodegenUtils.cpp.

References mlir::Builder::getIndexType(), mlir::Builder::getIntegerType(), kIndex, kU16, kU32, kU64, and kU8.

◆ getRankedTensorType()

template<typename T >

RankedTensorType mlir::sparse_tensor::getRankedTensorType ( T && t )

inline

Convenience method to abbreviate casting getType().

Definition at line 160 of file SparseTensor.h.

References mlir::getType().

Referenced by foreachInSparseConstant(), and mlir::sparse_tensor::LoopEmitter::initialize().

◆ getSparseTensorEncoding()

SparseTensorEncodingAttr mlir::sparse_tensor::getSparseTensorEncoding ( Type type )

Convenience method to get a sparse encoding attribute from a type.

Returns null-attribute for any type without an encoding.

Definition at line 1032 of file SparseTensorDialect.cpp.

Referenced by containsSparseTensor(), convTypes(), convVals(), createOrFoldSliceOffsetOp(), createOrFoldSliceStrideOp(), findSparseAnnotations(), genBuffers(), genConstantDenseAddressFromLevel(), genResult(), genSliceOffset(), genSliceStride(), hasAnySparseType(), isSparseTensor(), materializeTuple(), mlir::sparse_tensor::IterationGraphSorter::sort(), and mlir::sparse_tensor::CodegenEnv::startEmit().

◆ getSparseTensorType()

SparseTensorType mlir::sparse_tensor::getSparseTensorType ( Value val )

inline

Convenience methods to obtain a SparseTensorType from a Value.

Definition at line 374 of file SparseTensorType.h.

References mlir::Value::getType().

Referenced by genIf(), genIndex(), genInvariants(), genSubscript(), genTensorLoad(), getAllTidLvlsInLatPoints(), hasNonTrivialAffineOnSparseOut(), inferSparseBufferType(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), mlir::sparse_tensor::CodegenEnv::isAdmissibleTensorExp(), lvlIsInBounds(), makeSparseTensorLevel(), relinkBranch(), and sizesFromSrc().

◆ getTop()

Operation * mlir::sparse_tensor::getTop ( Operation * op )

Scans to top of generated loop.

Definition at line 419 of file CodegenUtils.cpp.

References mlir::Operation::getParentOp().

◆ hasAnyNonIdentityOperandsOrResults()

bool mlir::sparse_tensor::hasAnyNonIdentityOperandsOrResults ( Operation * op )

Returns true iff MLIR operation has any sparse tensor with non-identity dim2lvl maps.

Definition at line 1170 of file SparseTensorDialect.cpp.

References mlir::Operation::getOperands(), mlir::Operation::getResults(), mlir::sparse_tensor::SparseTensorType::isIdentity(), and tryGetSparseTensorType().

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseOperand()

bool mlir::sparse_tensor::hasAnySparseOperand ( Operation * op )

inline

Returns true iff MLIR operand has any sparse operand.

Definition at line 186 of file SparseTensor.h.

References mlir::Operation::getOperands(), mlir::OperandRange::getTypes(), and hasAnySparseType().

Referenced by hasAnySparseOperandOrResult().

◆ hasAnySparseOperandOrResult()

bool mlir::sparse_tensor::hasAnySparseOperandOrResult ( Operation * op )

inline

Returns true iff MLIR operand has any sparse operand or result.

Definition at line 196 of file SparseTensor.h.

References hasAnySparseOperand(), and hasAnySparseResult().

Referenced by mlir::sparse_tensor::IterationGraphSorter::fromGenericOp().

◆ hasAnySparseResult()

bool mlir::sparse_tensor::hasAnySparseResult ( Operation * op )

inline

Returns true iff MLIR operand has any sparse result.

Definition at line 191 of file SparseTensor.h.

References mlir::Operation::getResults(), mlir::ResultRange::getTypes(), and hasAnySparseType().

Referenced by hasAnySparseOperandOrResult().

◆ hasAnySparseType()

bool mlir::sparse_tensor::hasAnySparseType ( TypeRange types )

inline

Returns true iff the type range has any sparse tensor type.

Definition at line 179 of file SparseTensor.h.

References getSparseTensorEncoding().

Referenced by genInsertionStore(), hasAnySparseOperand(), and hasAnySparseResult().

◆ hash_value()

llvm::hash_code mlir::sparse_tensor::hash_value ( LevelType lt )

Definition at line 50 of file SparseTensorDialect.cpp.

◆ inferLvlToDim()

AffineMap mlir::sparse_tensor::inferLvlToDim	(	AffineMap	dimToLvl,
		MLIRContext *	context
	)

Given the dimToLvl map, infers the lvlToDim map, or returns empty Affine map when inference fails.

Definition at line 1040 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getNumSymbols(), inverseBlockSparsity(), mlir::inversePermutation(), isBlockSparsity(), and mlir::AffineMap::isPermutation().

Referenced by genReplaceDimToLvlMap().

◆ insertYieldOp()

static Value mlir::sparse_tensor::insertYieldOp	(	RewriterBase &	rewriter,
		Location	loc,
		Region &	region,
		ValueRange	vals
	)

static

Definition at line 1557 of file Merger.cpp.

References mlir::Region::begin(), mlir::Region::cloneInto(), mlir::OpBuilder::create(), mlir::RewriterBase::eraseOp(), mlir::Region::front(), mlir::Block::getTerminator(), and mlir::RewriterBase::inlineBlockBefore().

Referenced by buildBinaryOverlap(), mlir::sparse_tensor::Merger::buildExp(), and buildUnaryPresent().

◆ inverseBlockSparsity()

AffineMap mlir::sparse_tensor::inverseBlockSparsity	(	AffineMap	dimToLvl,
		MLIRContext *	context
	)

Returns the lvlToDim map for the given dimToLvl map specific to the block sparse cases.

Asserts on failure (so only use when known to succeed).

Definition at line 1055 of file SparseTensorDialect.cpp.

References mlir::Add, mlir::FloorDiv, mlir::AffineMap::get(), mlir::getAffineBinaryOpExpr(), mlir::getAffineDimExpr(), mlir::AffineMap::getNumResults(), mlir::AffineMap::getResult(), mlir::Mod, and mlir::Mul.

Referenced by inferLvlToDim().

◆ isAdmissibleBranch()

static bool mlir::sparse_tensor::isAdmissibleBranch	(	Operation *	op,
		Region &	region
	)

static

Ensures that the sparsifier can generate code for branch.

Definition at line 1264 of file Merger.cpp.

References mlir::Region::empty(), mlir::Region::front(), mlir::Operation::getOperand(), mlir::Block::getTerminator(), and isAdmissibleBranchExp().

◆ isAdmissibleBranchExp()

static bool mlir::sparse_tensor::isAdmissibleBranchExp	(	Operation *	op,
		Block *	block,
		Value	v
	)

static

Ensures that the sparsifier can generate code for expression.

Definition at line 1244 of file Merger.cpp.

References mlir::Operation::getBlock(), mlir::Value::getDefiningOp(), mlir::Operation::getNumOperands(), and mlir::Operation::getOperand().

Referenced by isAdmissibleBranch().

◆ isAnyOfFmt()

template<LevelFormat... targets>

constexpr bool mlir::sparse_tensor::isAnyOfFmt ( LevelFormat fmt )

constexpr

Definition at line 178 of file Enums.h.

Referenced by mlir::sparse_tensor::LevelType::isValidLvlBits().

◆ isBatchLT()

bool mlir::sparse_tensor::isBatchLT ( LevelType lt )

inline

Definition at line 414 of file Enums.h.

References Batch, and mlir::sparse_tensor::LevelType::isa().

◆ isBlockSparsity()

bool mlir::sparse_tensor::isBlockSparsity ( AffineMap dimToLvl )

Given the dimToLvl map, returns if it's block sparsity.

Definition at line 1125 of file SparseTensorDialect.cpp.

References mlir::FloorDiv, mlir::AffineMap::getResults(), and mlir::Mod.

Referenced by getBlockSize(), and inferLvlToDim().

◆ isCertainZero()

static bool mlir::sparse_tensor::isCertainZero ( Value val )

static

Only returns true if we are certain this is a zero.

Definition at line 1201 of file Merger.cpp.

References mlir::Value::getDefiningOp().

◆ isComplexPrimaryType()

constexpr bool mlir::sparse_tensor::isComplexPrimaryType ( PrimaryType valTy )

constexpr

Definition at line 141 of file Enums.h.

References kC32, and kC64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isCompressedLT()

bool mlir::sparse_tensor::isCompressedLT ( LevelType lt )

inline

Definition at line 415 of file Enums.h.

References Compressed, and mlir::sparse_tensor::LevelType::isa().

Referenced by allocSchemeForRank(), createAllocFields(), genEndInsert(), mlir::sparse_tensor::SparseTensorType::isCompressedLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isCompressedLvl().

◆ isDenseLT()

bool mlir::sparse_tensor::isDenseLT ( LevelType lt )

inline

Definition at line 413 of file Enums.h.

References Dense, and mlir::sparse_tensor::LevelType::isa().

Referenced by allocSchemeForRank(), genEndInsert(), getAllTidLvlsInLatPoints(), isAllDense(), mlir::sparse_tensor::SparseTensorType::isDenseLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isDenseLvl().

◆ isEncodedFloor()

constexpr bool mlir::sparse_tensor::isEncodedFloor ( uint64_t v )

constexpr

Definition at line 491 of file Enums.h.

◆ isEncodedMod()

constexpr bool mlir::sparse_tensor::isEncodedMod ( uint64_t v )

constexpr

Definition at line 492 of file Enums.h.

◆ isEncodedMul()

constexpr bool mlir::sparse_tensor::isEncodedMul ( uint64_t v )

constexpr

Definition at line 493 of file Enums.h.

◆ isFloatingPrimaryType()

constexpr bool mlir::sparse_tensor::isFloatingPrimaryType ( PrimaryType valTy )

constexpr

Definition at line 129 of file Enums.h.

References kBF16, and kF64.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isGreater()

static bool mlir::sparse_tensor::isGreater	(	TensorExp::Kind	kind,
		Attribute	attr
	)

static

Definition at line 1274 of file Merger.cpp.

References mlir::sparse_tensor::TensorExp::kCmpF, mlir::sparse_tensor::TensorExp::kCmpI, and kind.

◆ isIntegralPrimaryType()

constexpr bool mlir::sparse_tensor::isIntegralPrimaryType ( PrimaryType valTy )

constexpr

Definition at line 133 of file Enums.h.

References kI64, and kI8.

◆ isLooseCompressedLT()

bool mlir::sparse_tensor::isLooseCompressedLT ( LevelType lt )

inline

Definition at line 418 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and LooseCompressed.

Referenced by allocSchemeForRank(), createAllocFields(), genEndInsert(), mlir::sparse_tensor::SparseTensorType::isLooseCompressedLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isLooseCompressedLvl().

◆ isNOutOfMLT()

bool mlir::sparse_tensor::isNOutOfMLT ( LevelType lt )

inline

Definition at line 424 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and NOutOfM.

Referenced by allocSchemeForRank(), genEndInsert(), mlir::sparse_tensor::SparseTensorType::isNOutOfMLvl(), mlir::sparse_tensor::SparseTensorStorageBase::isNOutOfMLvl(), and isValidNOutOfMLT().

◆ isOrderedLT()

bool mlir::sparse_tensor::isOrderedLT ( LevelType lt )

inline

Definition at line 425 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and Nonordered.

Referenced by mlir::sparse_tensor::SparseTensorType::isOrderedLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isOrderedLvl().

◆ isRealPrimaryType()

constexpr bool mlir::sparse_tensor::isRealPrimaryType ( PrimaryType valTy )

constexpr

Definition at line 137 of file Enums.h.

References kF64, and kI8.

Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().

◆ isSingletonLT()

bool mlir::sparse_tensor::isSingletonLT ( LevelType lt )

inline

Definition at line 421 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and Singleton.

Referenced by allocSchemeForRank(), genEndInsert(), mlir::sparse_tensor::SparseTensorType::isSingletonLvl(), and mlir::sparse_tensor::SparseTensorStorageBase::isSingletonLvl().

◆ isUndefLT()

bool mlir::sparse_tensor::isUndefLT ( LevelType lt )

inline

Definition at line 412 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and Undef.

Referenced by findAffine(), findDepIdxSet(), genIf(), getAllTidLvlsInLatPoints(), and mlir::sparse_tensor::ir_detail::LvlSpec::LvlSpec().

◆ isUniqueLT()

bool mlir::sparse_tensor::isUniqueLT ( LevelType lt )

inline

Definition at line 428 of file Enums.h.

References mlir::sparse_tensor::LevelType::isa(), and Nonunique.

Referenced by getAllTidLvlsInLatPoints(), mlir::sparse_tensor::SparseTensorLevel::isUnique(), mlir::sparse_tensor::SparseTensorType::isUniqueLvl(), mlir::sparse_tensor::SparseTensorStorageBase::isUniqueLvl(), and makeSimpleIterator().

◆ isValidLT()

bool mlir::sparse_tensor::isValidLT ( LevelType lt )

inline

Definition at line 433 of file Enums.h.

References mlir::sparse_tensor::LevelType::isValidLvlBits().

Referenced by mlir::sparse_tensor::ir_detail::LvlSpec::LvlSpec(), mlir::sparse_tensor::ir_detail::LvlTypeParser::parseLvlType(), and mlir::sparse_tensor::Merger::setLevelAndType().

◆ isValidNOutOfMLT()

bool mlir::sparse_tensor::isValidNOutOfMLT	(	LevelType	lt,
		uint64_t	n,
		uint64_t	m
	)

inline

Definition at line 444 of file Enums.h.

References mlir::sparse_tensor::LevelType::getM(), mlir::sparse_tensor::LevelType::getN(), and isNOutOfMLT().

◆ isWithCrdLT()

bool mlir::sparse_tensor::isWithCrdLT ( LevelType lt )

inline

Definition at line 431 of file Enums.h.

References mlir::sparse_tensor::LevelType::isWithCrdLT().

Referenced by mlir::sparse_tensor::StorageLayout::foreachField(), and mlir::sparse_tensor::SparseTensorType::isWithCrd().

◆ isWithPosLT()

bool mlir::sparse_tensor::isWithPosLT ( LevelType lt )

inline

Definition at line 432 of file Enums.h.

References mlir::sparse_tensor::LevelType::isWithPosLT().

Referenced by mlir::sparse_tensor::StorageLayout::foreachField(), and mlir::sparse_tensor::SparseTensorType::isWithPos().

◆ isZeroRankedTensorOrScalar()

bool mlir::sparse_tensor::isZeroRankedTensorOrScalar ( Type type )

inline

Definition at line 412 of file CodegenUtils.h.

Referenced by mlir::sparse_tensor::LoopEmitter::initialize().

◆ kindToOpSymbol()

static const char* mlir::sparse_tensor::kindToOpSymbol ( TensorExp::Kind kind )

static

Definition at line 693 of file Merger.cpp.

Referenced by mlir::sparse_tensor::Merger::dumpExp().

◆ loadAll()

SmallVector< Value > mlir::sparse_tensor::loadAll	(	OpBuilder &	builder,
		Location	loc,
		size_t	size,
		Value	mem,
		size_t	offsetIdx = `0`,
		Value	offsetVal = `Value()`
	)

Loads size-many values from the memref, which must have rank-1 and size greater-or-equal to size.

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value after loading.

Definition at line 503 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::Value::getType().

◆ makeNonEmptySubSectIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeNonEmptySubSectIterator	(	OpBuilder &	b,
		Location	l,
		const SparseIterator *	parent,
		Value	loopBound,
		std::unique_ptr< SparseIterator > &&	delegate,
		Value	size,
		unsigned	stride,
		SparseEmitStrategy	strategy
	)

Helper function to create a SparseIterator object that iterate over the non-empty subsections set.

Definition at line 1670 of file SparseTensorIterator.cpp.

References C_IDX, and tryUnwrapFilter().

◆ makePaddedIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makePaddedIterator	(	std::unique_ptr< SparseIterator > &&	sit,
		Value	padLow,
		Value	padHigh,
		SparseEmitStrategy	strategy
	)

Helper function to create a SparseIterator object that iterates over a padded sparse level (the padded value must be zero).

Definition at line 1655 of file SparseTensorIterator.cpp.

◆ makeSimpleIterator() [1/2]

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeSimpleIterator	(	const SparseTensorLevel &	stl,
		SparseEmitStrategy	strategy = `SparseEmitStrategy::kFunctional`
	)

Helper function to create a simple SparseIterator object that iterate over the sparse tensor level.

TODO: switch to SparseIterationSpace (which support N-D iterator) when feature complete.

Definition at line 1629 of file SparseTensorIterator.cpp.

References mlir::sparse_tensor::SparseTensorLevel::getLT(), and isUniqueLT().

◆ makeSimpleIterator() [2/2]

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeSimpleIterator	(	OpBuilder &	b,
		Location	l,
		const SparseIterationSpace &	iterSpace
	)

Helper function to create a simple SparseIterator object that iterate over the entire iteration space.

Definition at line 1609 of file SparseTensorIterator.cpp.

References mlir::sparse_tensor::SparseIterationSpace::getBoundHi(), mlir::sparse_tensor::SparseIterationSpace::getBoundLo(), mlir::sparse_tensor::SparseIterationSpace::getLastLvl(), mlir::sparse_tensor::SparseIterationSpace::isUnique(), and mlir::kFunctional.

Referenced by mlir::sparse_tensor::SparseIterationSpace::extractIterator().

◆ makeSlicedLevelIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeSlicedLevelIterator	(	std::unique_ptr< SparseIterator > &&	sit,
		Value	offset,
		Value	stride,
		Value	size,
		SparseEmitStrategy	strategy
	)

Helper function to create a SparseIterator object that iterates over a sliced space, the orignal space (before slicing) is traversed by sit.

Definition at line 1644 of file SparseTensorIterator.cpp.

◆ makeSparseTensorLevel() [1/2]

std::unique_ptr< SparseTensorLevel > mlir::sparse_tensor::makeSparseTensorLevel	(	LevelType	lt,
		Value	sz,
		ValueRange	buffers,
		unsigned	tid,
		Level	l
	)

Helper function to create a TensorLevel object from given ValueRange.

Helper function to create a TensorLevel object from given tensor.

Definition at line 1556 of file SparseTensorIterator.cpp.

References Batch, Compressed, Dense, mlir::sparse_tensor::LevelType::getLvlFmt(), mlir::sparse_tensor::LevelType::getNumBuffer(), LooseCompressed, NOutOfM, Singleton, and Undef.

◆ makeSparseTensorLevel() [2/2]

std::unique_ptr< SparseTensorLevel > mlir::sparse_tensor::makeSparseTensorLevel	(	OpBuilder &	b,
		Location	l,
		Value	t,
		unsigned	tid,
		Level	lvl
	)

Helper function to create a TensorLevel object from given tensor.

Definition at line 1579 of file SparseTensorIterator.cpp.

References mlir::OpBuilder::create(), getSparseTensorType(), mlir::sparse_tensor::LevelType::isWithCrdLT(), and mlir::sparse_tensor::LevelType::isWithPosLT().

Referenced by mlir::sparse_tensor::SparseIterationSpace::fromValues(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), and mlir::sparse_tensor::SparseIterationSpace::SparseIterationSpace().

◆ makeSynLevelAndIterator()

std::pair< std::unique_ptr< SparseTensorLevel >, std::unique_ptr< SparseIterator > > mlir::sparse_tensor::makeSynLevelAndIterator	(	Value	sz,
		unsigned	tid,
		unsigned	lvl,
		SparseEmitStrategy	strategy
	)

Helper function to create a synthetic SparseIterator object that iterates over a dense space specified by [0,sz).

Definition at line 1600 of file SparseTensorIterator.cpp.

References mlir::sparse_tensor::SparseIterator::setSparseEmitStrategy().

Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().

◆ makeTraverseSubSectIterator()

std::unique_ptr< SparseIterator > mlir::sparse_tensor::makeTraverseSubSectIterator	(	OpBuilder &	b,
		Location	l,
		const SparseIterator &	subsectIter,
		const SparseIterator &	parent,
		std::unique_ptr< SparseIterator > &&	wrap,
		Value	loopBound,
		unsigned	stride,
		SparseEmitStrategy	strategy
	)

Helper function to create a SparseIterator object that iterates over a non-empty subsection created by NonEmptySubSectIterator.

Definition at line 1691 of file SparseTensorIterator.cpp.

References C_IDX, tryUnwrapFilter(), and wrap().

◆ mToBits()

constexpr uint64_t mlir::sparse_tensor::mToBits ( uint64_t m )

constexpr

Definition at line 399 of file Enums.h.

Referenced by mlir::sparse_tensor::ir_detail::LvlTypeParser::parseLvlType().

◆ nToBits()

constexpr uint64_t mlir::sparse_tensor::nToBits ( uint64_t n )

constexpr

Definition at line 398 of file Enums.h.

Referenced by mlir::sparse_tensor::ir_detail::LvlTypeParser::parseLvlType().

◆ overheadTypeEncoding() [1/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( Type tp )

Converts an overhead storage type to its internal type-encoding.

Definition at line 43 of file CodegenUtils.cpp.

References mlir::Type::isIndex(), kIndex, and overheadTypeEncoding().

◆ overheadTypeEncoding() [2/2]

OverheadType mlir::sparse_tensor::overheadTypeEncoding ( unsigned width )

Converts an overhead storage bitwidth to its internal type-encoding.

Definition at line 27 of file CodegenUtils.cpp.

References kIndex, kU16, kU32, kU64, and kU8.

Referenced by constantOverheadTypeEncoding(), crdTypeEncoding(), overheadTypeEncoding(), overheadTypeFunctionSuffix(), and posTypeEncoding().

◆ overheadTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( OverheadType ot )

Convert OverheadType to its function-name suffix.

Definition at line 85 of file CodegenUtils.cpp.

References CASE, kIndex, and MLIR_SPARSETENSOR_FOREVERY_FIXED_O.

Referenced by overheadTypeFunctionSuffix().

◆ overheadTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix ( Type overheadTp )

Converts an overhead storage type to its function-name suffix.

Definition at line 98 of file CodegenUtils.cpp.

References overheadTypeEncoding(), and overheadTypeFunctionSuffix().

◆ posTypeEncoding()

OverheadType mlir::sparse_tensor::posTypeEncoding ( SparseTensorEncodingAttr enc )

Returns the OverheadType for position overhead storage.

Definition at line 68 of file CodegenUtils.cpp.

References overheadTypeEncoding().

◆ primaryTypeEncoding()

PrimaryType mlir::sparse_tensor::primaryTypeEncoding ( Type elemTp )

Converts a primary storage type to its internal type-encoding.

Definition at line 102 of file CodegenUtils.cpp.

References mlir::Type::isBF16(), mlir::Type::isF16(), mlir::Type::isF32(), mlir::Type::isF64(), mlir::Type::isInteger(), kBF16, kC32, kC64, kF16, kF32, kF64, kI16, kI32, kI64, and kI8.

Referenced by constantPrimaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [1/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( PrimaryType pt )

Convert PrimaryType to its function-name suffix.

Definition at line 129 of file CodegenUtils.cpp.

References CASE, and MLIR_SPARSETENSOR_FOREVERY_V.

Referenced by primaryTypeFunctionSuffix().

◆ primaryTypeFunctionSuffix() [2/2]

StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix ( Type elemTp )

Converts a primary storage type to its function-name suffix.

Definition at line 140 of file CodegenUtils.cpp.

References primaryTypeEncoding(), and primaryTypeFunctionSuffix().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels ( DialectRegistry & registry )

Definition at line 326 of file BufferizableOpInterfaceImpl.cpp.

References mlir::DialectRegistry::addExtension().

Referenced by mlir::registerAllDialects().

◆ registerSparseTensorPipelines()

void mlir::sparse_tensor::registerSparseTensorPipelines ( )

Registers all pipelines for the sparse_tensor dialect.

At present, this includes only "sparsifier".

Definition at line 113 of file SparseTensorPipelines.cpp.

References buildSparsifier().

Referenced by mlir::registerAllPasses().

◆ registerTransformDialectExtension()

void mlir::sparse_tensor::registerTransformDialectExtension ( DialectRegistry & registry )

Definition at line 57 of file SparseTensorTransformOps.cpp.

References mlir::DialectRegistry::addExtensions().

Referenced by mlir::registerAllExtensions().

◆ reshapeCvs()

void mlir::sparse_tensor::reshapeCvs	(	OpBuilder &	builder,
		Location	loc,
		ArrayRef< ReassociationIndices >	reassociation,
		ValueRange	srcSizes,
		ValueRange	srcCvs,
		ValueRange	dstSizes,
		SmallVectorImpl< Value > &	dstCvs
	)

Reshape coordinates during a reshaping operation.

Definition at line 272 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().

◆ sizesFromSrc()

void mlir::sparse_tensor::sizesFromSrc	(	OpBuilder &	builder,
		SmallVectorImpl< Value > &	sizes,
		Location	loc,
		Value	src
	)

Populates given sizes array from dense tensor or sparse tensor constant.

Definition at line 411 of file CodegenUtils.cpp.

References mlir::linalg::createOrFoldDimOp(), mlir::sparse_tensor::SparseTensorType::getDimRank(), and getSparseTensorType().

Referenced by concatSizesFromInputs().

◆ storeAll()

void mlir::sparse_tensor::storeAll	(	OpBuilder &	builder,
		Location	loc,
		Value	mem,
		ValueRange	vs,
		size_t	offsetIdx = `0`,
		Value	offsetVal = `Value()`
	)

Stores all the values of vs into the memref mem, which must have rank-1 and size greater-or-equal to vs.size().

If the optional (offsetIdx,offsetVal) arguments are provided, then the offsetVal will be added to the offsetIdx-th value before storing.

Definition at line 525 of file CodegenUtils.cpp.

References constantIndex(), mlir::OpBuilder::create(), mlir::detail::enumerate(), and mlir::Value::getType().

◆ toDim()

Dimension mlir::sparse_tensor::toDim	(	SparseTensorEncodingAttr	enc,
		Level	l
	)

Convenience method to translate the given level to the corresponding dimension.

Requires: enc has a permuted dim2lvl map and 0 <= l < lvlRank.

Definition at line 1180 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

Referenced by genSliceOffset(), and genSliceStride().

◆ toFieldKind()

SparseTensorFieldKind mlir::sparse_tensor::toFieldKind ( StorageSpecifierKind kind )

inline

Definition at line 102 of file SparseTensorStorageLayout.h.

References kind.

Referenced by SpecifierGetterSetterOpConverter< Base, SourceOp >::matchAndRewrite().

◆ toFormatString()

constexpr const char* mlir::sparse_tensor::toFormatString ( LevelFormat lvlFmt )

constexpr

Returns string representation of the given level format.

Definition at line 183 of file Enums.h.

References Batch, Compressed, Dense, LooseCompressed, NOutOfM, Singleton, and Undef.

Referenced by mlir::sparse_tensor::LevelType::toMLIRString().

◆ toLvl()

Level mlir::sparse_tensor::toLvl	(	SparseTensorEncodingAttr	enc,
		Dimension	d
	)

Convenience method to translate the given dimension to the corresponding level.

Requires: enc has a permuted dim2lvl map and 0 <= d < dimRank.

Definition at line 1189 of file SparseTensorDialect.cpp.

References mlir::AffineMap::getDimPosition().

◆ toMLIRString()

std::string mlir::sparse_tensor::toMLIRString ( LevelType lt )

inline

Definition at line 447 of file Enums.h.

References mlir::sparse_tensor::LevelType::toMLIRString().

Referenced by mlir::sparse_tensor::Merger::dumpBits(), and mlir::sparse_tensor::SparseTensorLevel::toString().

◆ toPropString()

constexpr const char* mlir::sparse_tensor::toPropString ( LevelPropNonDefault lvlProp )

constexpr

Returns string representation of the given level properties.

Definition at line 211 of file Enums.h.

References Nonordered, Nonunique, and SoA.

Referenced by mlir::sparse_tensor::LevelType::toMLIRString().

◆ toSpecifierKind()

StorageSpecifierKind mlir::sparse_tensor::toSpecifierKind ( SparseTensorFieldKind kind )

inline

Definition at line 97 of file SparseTensorStorageLayout.h.

References kind, and StorageSpec.

Referenced by createPushback().

◆ tryGetSparseTensorType()

std::optional<SparseTensorType> mlir::sparse_tensor::tryGetSparseTensorType ( Value val )

inline

Definition at line 377 of file SparseTensorType.h.

References mlir::Value::getType().

Referenced by hasAnyNonIdentityOperandsOrResults(), translateMap(), and tryFoldTensors().

Namespaces

Classes

Typedefs

Enumerations

Functions

Typedef Documentation

◆ complex32

◆ complex64

◆ Dimension

◆ ExprId

◆ FieldIndex

◆ index_type

◆ LatPointId

◆ LatSetId

◆ Level

◆ LoopCoeffPair

◆ LoopId

◆ LvlLTPair

◆ Size

◆ TensorId

◆ TensorLevel

◆ TensorLoopId

Enumeration Type Documentation

◆ Action

◆ EmitCInterface

◆ ExpArity

◆ IterKind

◆ LevelFormat

◆ LevelPropNonDefault

◆ OverheadType

◆ PrimaryType

◆ SortMask

◆ SparseTensorFieldKind

Function Documentation

◆ allocaBuffer()

◆ allocDenseTensor()

◆ buildBinaryOverlap()

◆ buildLevelType() [1/2]

◆ buildLevelType() [2/2]

◆ buildRelu()

◆ buildSparsifier()

◆ buildUnaryPresent()

◆ constantAction()

◆ constantCrdTypeEncoding()

◆ constantI1()

◆ constantI16()

◆ constantI32()

◆ constantI64()

◆ constantI8()

◆ constantIndex()

◆ constantLevelTypeEncoding()

◆ constantOne()

◆ constantOverheadTypeEncoding()

◆ constantPosTypeEncoding()

◆ constantPrimaryTypeEncoding()

◆ constantZero()

◆ containsSparseTensor()

◆ crdTypeEncoding()

◆ createFuncCall()

◆ createOrFoldSliceOffsetOp()

◆ createOrFoldSliceStrideOp()

◆ deallocDenseTensor()

◆ decodeConst()

◆ decodeIndex()

◆ decodeMulc()

◆ decodeMuli()

◆ encodeDim()

◆ encodeLvl()

◆ encPowOfTwo()

◆ foreachFieldAndTypeInSparseTensor()

◆ foreachFieldInSparseTensor()

◆ foreachInSparseConstant()

◆ genAlloca() [1/2]

◆ genAlloca() [2/2]

◆ genAllocaScalar()

◆ genCast()

◆ genCoIteration()

◆ genIndexLoad()

◆ genIsNonzero()

◆ genMapBuffers()