|
MLIR
17.0.0git
|
Namespaces | |
| detail | |
Classes | |
| struct | SparseCompilerOptions |
| Options for the "sparse-compiler" pipeline. More... | |
| struct | Children |
| Children subexpressions of tensor operations. More... | |
| struct | TensorExp |
| Tensor expression. Represents a MLIR expression in tensor index notation. More... | |
| struct | LatPoint |
| Lattice point. More... | |
| class | Merger |
| A class to handle all iteration lattice operations. More... | |
| struct | Element |
| An element of a sparse tensor in coordinate-scheme representation (i.e., a pair of indices and value). More... | |
| struct | ElementLT |
Closure object for operator< on Element with a given rank. More... | |
| class | SparseTensorCOO |
A memory-resident sparse tensor in coordinate-scheme representation (a collection of Elements). More... | |
| class | SparseTensorReader |
| This class abstracts over the information stored in file headers, as well as providing the buffers and methods for parsing those headers. More... | |
| class | SparseTensorEnumeratorBase |
A (higher-order) function object for enumerating the elements of some SparseTensorStorage under a permutation. More... | |
| class | SparseTensorStorageBase |
Abstract base class for SparseTensorStorage<P,I,V>. More... | |
| class | SparseTensorEnumerator |
| class | SparseTensorStorage |
| A memory-resident sparse tensor using a storage scheme based on per-dimension sparse/dense annotations. More... | |
| class | SparseTensorNNZ |
| Statistics regarding the number of nonzero subtensors in a source tensor, for direct sparse=>sparse conversion a la https://arxiv.org/abs/2001.02609. More... | |
| class | CodegenEnv |
| The code generation environment class aggregates a number of data structures that are needed during the code generation phase of sparsification. More... | |
| class | LoopEmitter |
| class | StorageLayout |
| Provides methods to access fields of a sparse tensor with the given encoding. More... | |
| class | SparseTensorSpecifier |
| class | SparseTensorDescriptorImpl |
| A helper class around an array of values that corresponding to a sparse tensor, provides a set of meaningful APIs to query and update a particular field in a consistent way. More... | |
| class | SparseTensorDescriptor |
| Uses ValueRange for immuatable descriptors;. More... | |
| class | MutSparseTensorDescriptor |
| Uses SmallVectorImpl<Value> & for mutable descriptors. More... | |
| class | SparsificationAndBufferizationPass |
| A pass that lowers tensor ops to memref ops, regardless of whether they are dense or sparse. More... | |
Typedefs | |
| using | index_type = uint64_t |
| This type is used in the public API at all places where MLIR expects values with the built-in type "index". More... | |
| using | complex64 = std::complex< double > |
| using | complex32 = std::complex< float > |
| template<typename V > | |
| using | ElementConsumer = const std::function< void(const std::vector< uint64_t > &, V)> & |
| The type of callback functions which receive an element. More... | |
Enumerations | |
| enum class | OverheadType : uint32_t { kIndex = 0 , kU64 = 1 , kU32 = 2 , kU16 = 3 , kU8 = 4 } |
| Encoding of overhead types (both pointer overhead and indices overhead), for "overloading" @newSparseTensor. More... | |
| enum class | PrimaryType : uint32_t { kF64 = 1 , kF32 = 2 , kF16 = 3 , kBF16 = 4 , kI64 = 5 , kI32 = 6 , kI16 = 7 , kI8 = 8 , kC64 = 9 , kC32 = 10 } |
| Encoding of the elemental type, for "overloading" @newSparseTensor. More... | |
| enum class | Action : uint32_t { kEmpty = 0 , kFromCOO = 2 , kSparseToSparse = 3 , kEmptyCOO = 4 , kToCOO = 5 , kToIterator = 6 } |
| The actions performed by @newSparseTensor. More... | |
| enum class | DimLevelType : uint8_t { Undef = 0 , Dense = 4 , Compressed = 8 , CompressedNu = 9 , CompressedNo = 10 , CompressedNuNo = 11 , Singleton = 16 , SingletonNu = 17 , SingletonNo = 18 , SingletonNuNo = 19 } |
| This enum defines all the sparse representations supportable by the SparseTensor dialect. More... | |
| enum class | LevelFormat : uint8_t { Dense = 4 , Compressed = 8 , Singleton = 16 } |
| This enum defines all the storage formats supported by the sparse compiler, without the level properties. More... | |
| enum | Kind { kTensor = 0 , kInvariant , kIndex , kIndex = 0 , kAbsF , kAbsC , kAbsI , kCeilF , kFloorF , kSqrtF , kSqrtC , kExpm1F , kExpm1C , kLog1pF , kLog1pC , kSinF , kSinC , kTanhF , kTanhC , kNegF , kNegC , kNegI , kTruncF , kExtF , kCastFS , kCastFU , kCastSF , kCastUF , kCastS , kCastU , kCastIdx , kTruncI , kCIm , kCRe , kBitCast , kBinaryBranch , kUnary , kSelect , kMulF , kMulC , kMulI , kDivF , kDivC , kDivS , kDivU , kAddF , kAddC , kAddI , kSubF , kSubC , kSubI , kAndI , kOrI , kXorI , kShrS , kShrU , kShlI , kBinary , kReduce } |
| Tensor expression kind. More... | |
| enum class | EmitCInterface : bool { Off = false , On = true } |
Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall(). More... | |
| enum class | SparseTensorFieldKind : uint32_t { StorageSpec = 0 , PtrMemRef = 1 , IdxMemRef = 2 , ValMemRef = 3 } |
| enum class | ExpArity { kNullary , kUnary , kBinary } |
Functions | |
| constexpr bool | isFloatingPrimaryType (PrimaryType valTy) |
| constexpr bool | isIntegralPrimaryType (PrimaryType valTy) |
| constexpr bool | isRealPrimaryType (PrimaryType valTy) |
| constexpr bool | isComplexPrimaryType (PrimaryType valTy) |
| std::string | toMLIRString (DimLevelType dlt) |
| Returns string representation of the given dimension level type. More... | |
| constexpr bool | isValidDLT (DimLevelType dlt) |
Check that the DimLevelType contains a valid (possibly undefined) value. More... | |
| constexpr bool | isUndefDLT (DimLevelType dlt) |
Check if the DimLevelType is the special undefined value. More... | |
| constexpr bool | isDenseDLT (DimLevelType dlt) |
Check if the DimLevelType is dense. More... | |
| constexpr bool | isCompressedDLT (DimLevelType dlt) |
Check if the DimLevelType is compressed (regardless of properties). More... | |
| constexpr bool | isSingletonDLT (DimLevelType dlt) |
Check if the DimLevelType is singleton (regardless of properties). More... | |
| constexpr bool | isOrderedDLT (DimLevelType dlt) |
Check if the DimLevelType is ordered (regardless of storage format). More... | |
| constexpr bool | isUniqueDLT (DimLevelType dlt) |
Check if the DimLevelType is unique (regardless of storage format). More... | |
| constexpr std::optional< LevelFormat > | getLevelFormat (DimLevelType dlt) |
| Convert a DimLevelType to its corresponding LevelFormat. More... | |
| constexpr std::optional< DimLevelType > | getDimLevelType (LevelFormat lf, bool ordered, bool unique) |
| Convert a LevelFormat to its corresponding DimLevelType with the given properties. More... | |
| template<typename T > | |
| RankedTensorType | getRankedTensorType (T t) |
Convenience method to abbreviate casting getType(). More... | |
| template<typename T > | |
| MemRefType | getMemRefType (T t) |
Convenience method to abbreviate casting getType(). More... | |
| SparseTensorEncodingAttr | getSparseTensorEncoding (Type type) |
| Convenience method to get a sparse encoding attribute from a type. More... | |
| bool | isUniqueCOOType (RankedTensorType tp) |
| Returns true iff the given type is a type for a COO tensor with the last dimension level type being unique. More... | |
| unsigned | getCOOStart (SparseTensorEncodingAttr enc) |
| Returns the starting dimension for a trailing COO region that spans across at least two dimensions. More... | |
| DimLevelType | getDimLevelType (SparseTensorEncodingAttr enc, uint64_t d) |
| DimLevelType | getDimLevelType (RankedTensorType type, uint64_t d) |
| bool | isDenseDim (RankedTensorType type, uint64_t d) |
| Convenience function to test for dense dimension (0 <= d < rank). More... | |
| bool | isCompressedDim (RankedTensorType type, uint64_t d) |
| Convenience function to test for compressed dimension (0 <= d < rank). More... | |
| bool | isSingletonDim (RankedTensorType type, uint64_t d) |
| Convenience function to test for singleton dimension (0 <= d < rank). More... | |
| bool | isDenseDim (SparseTensorEncodingAttr enc, uint64_t d) |
| Convenience function to test for dense dimension (0 <= d < rank). More... | |
| bool | isCompressedDim (SparseTensorEncodingAttr enc, uint64_t d) |
| Convenience function to test for compressed dimension (0 <= d < rank). More... | |
| bool | isSingletonDim (SparseTensorEncodingAttr enc, uint64_t d) |
| Convenience function to test for singleton dimension (0 <= d < rank). More... | |
| bool | isOrderedDim (RankedTensorType type, uint64_t d) |
| Convenience function to test for ordered property in the given dimension (0 <= d < rank). More... | |
| bool | isUniqueDim (RankedTensorType type, uint64_t d) |
| Convenience function to test for unique property in the given dimension (0 <= d < rank). More... | |
| uint64_t | toOrigDim (SparseTensorEncodingAttr enc, uint64_t d) |
| uint64_t | toStoredDim (SparseTensorEncodingAttr enc, uint64_t d) |
| uint64_t | toOrigDim (RankedTensorType type, uint64_t d) |
| Convenience method to translate the given stored dimension to the original dimension (0 <= d < rank). More... | |
| uint64_t | toStoredDim (RankedTensorType type, uint64_t d) |
| Convenience method to translate the given original dimension to the stored dimension (0 <= d < rank). More... | |
| void | buildSparseCompiler (OpPassManager &pm, const SparseCompilerOptions &options) |
Adds the "sparse-compiler" pipeline to the OpPassManager. More... | |
| void | registerSparseTensorPipelines () |
Registers all pipelines for the sparse_tensor dialect. More... | |
| void | registerBufferizableOpInterfaceExternalModels (DialectRegistry ®istry) |
| template<typename V > | |
| void | writeExtFROSTT (const SparseTensorCOO< V > &coo, const char *filename) |
Writes the sparse tensor to filename in extended FROSTT format. More... | |
| OverheadType | overheadTypeEncoding (unsigned width) |
| Converts an overhead storage bitwidth to its internal type-encoding. More... | |
| OverheadType | overheadTypeEncoding (Type tp) |
| Converts an overhead storage type to its internal type-encoding. More... | |
| Type | getOverheadType (Builder &builder, OverheadType ot) |
| Converts the internal type-encoding for overhead storage to an mlir::Type. More... | |
| OverheadType | pointerOverheadTypeEncoding (SparseTensorEncodingAttr enc) |
| Returns the OverheadType for pointer overhead storage. More... | |
| OverheadType | indexOverheadTypeEncoding (SparseTensorEncodingAttr enc) |
| Returns the OverheadType for index overhead storage. More... | |
| Type | getPointerOverheadType (Builder &builder, SparseTensorEncodingAttr enc) |
| Returns the mlir::Type for pointer overhead storage. More... | |
| Type | getIndexOverheadType (Builder &builder, SparseTensorEncodingAttr enc) |
| Returns the mlir::Type for index overhead storage. More... | |
| StringRef | overheadTypeFunctionSuffix (OverheadType ot) |
| Convert OverheadType to its function-name suffix. More... | |
| StringRef | overheadTypeFunctionSuffix (Type overheadTp) |
| Converts an overhead storage type to its function-name suffix. More... | |
| PrimaryType | primaryTypeEncoding (Type elemTp) |
| Converts a primary storage type to its internal type-encoding. More... | |
| StringRef | primaryTypeFunctionSuffix (PrimaryType pt) |
| Convert PrimaryType to its function-name suffix. More... | |
| StringRef | primaryTypeFunctionSuffix (Type elemTp) |
| Converts a primary storage type to its function-name suffix. More... | |
| Attribute | getOneAttr (Builder &builder, Type tp) |
| Generates a 1-valued attribute of the given type. More... | |
| Value | genIsNonzero (OpBuilder &builder, Location loc, Value v) |
Generates the comparison v != 0 where v is of numeric type. More... | |
| void | genReshapeDstShape (Location loc, PatternRewriter &rewriter, SmallVectorImpl< Value > &dstShape, ArrayRef< Value > srcShape, ArrayRef< int64_t > staticDstShape, ArrayRef< ReassociationIndices > reassociation) |
| Computes the shape of destination tensor of a reshape operator. More... | |
| void | translateIndicesArray (OpBuilder &builder, Location loc, ArrayRef< ReassociationIndices > reassociation, ValueRange srcIndices, ArrayRef< Value > srcShape, ArrayRef< Value > dstShape, SmallVectorImpl< Value > &dstIndices) |
| Translate indices during a reshaping operation. More... | |
| FlatSymbolRefAttr | getFunc (ModuleOp module, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface) |
| Returns a function reference (first hit also inserts into module). More... | |
| func::CallOp | createFuncCall (OpBuilder &builder, Location loc, StringRef name, TypeRange resultType, ValueRange operands, EmitCInterface emitCInterface) |
Creates a CallOp to the function reference returned by getFunc() in the builder's module. More... | |
| Type | getOpaquePointerType (OpBuilder &builder) |
Returns the equivalent of void* for opaque arguments to the execution engine. More... | |
| Value | genAlloca (OpBuilder &builder, Location loc, Value sz, Type tp) |
Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>). More... | |
| Value | genAlloca (OpBuilder &builder, Location loc, unsigned sz, Type tp, bool staticShape=false) |
Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true). More... | |
| Value | genAllocaScalar (OpBuilder &builder, Location loc, Type tp) |
Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>. More... | |
| Value | allocaBuffer (OpBuilder &builder, Location loc, ValueRange values) |
Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer). More... | |
| Value | allocDenseTensor (OpBuilder &builder, Location loc, RankedTensorType tensorTp, ValueRange sizes) |
| Generates code to allocate a buffer of the given type, and zero initialize it. More... | |
| void | deallocDenseTensor (OpBuilder &builder, Location loc, Value buffer) |
| Generates code to deallocate a dense buffer. More... | |
| Value | genValueForDense (OpBuilder &builder, Location loc, Value tensor, ValueRange ivs) |
| Generates the code to read the value from tensor[ivs]. More... | |
| void | genDenseTensorOrSparseConstantIterLoop (OpBuilder &builder, Location loc, Value src, unsigned rank, function_ref< void(OpBuilder &, Location, Value, ValueRange)> bodyBuilder) |
| Generates the loop structure to iterate over a dense tensor or a sparse tensor constant to support the lowering of dense-to-sparse convert operator. More... | |
| void | sizesFromSrc (OpBuilder &builder, SmallVectorImpl< Value > &sizes, Location loc, Value src) |
| Populates given sizes array from dense tensor or sparse tensor constant. More... | |
| MemRefType | get1DMemRefType (Type etp, bool withLayout) |
| Generates a 1D MemRefType with a dynamic size. More... | |
| Operation * | getTop (Operation *op) |
| Scans to top of generated loop. More... | |
| void | foreachInSparseConstant (Location loc, RewriterBase &rewriter, SparseElementsAttr attr, function_ref< void(ArrayRef< Value >, Value)> callback) |
| Iterate over a sparse constant, generates constantOp for value and indices. More... | |
| void | storeIndices (OpBuilder &builder, Location loc, unsigned rank, Value ind, ValueRange ivs, unsigned offsetDim=0, Value offset=Value()) |
Converts the vector indices and store it into the memory pointed by ind, apply (optional) offset on offsetDim. More... | |
| Value | reshapeValuesToLevels (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc, const SmallVectorImpl< Value > &dimSizes, Value valuesBuffer, Value idxBuffer) |
| Reshapes the linear values buffer for an annotated all dense sparse tensor to match the shape of the corresponding dense tensor to support direct access of the buffer through indices. More... | |
| Value | constantZero (OpBuilder &builder, Location loc, Type tp) |
| Generates a 0-valued constant of the given type. More... | |
| Value | constantOne (OpBuilder &builder, Location loc, Type tp) |
| Generates a 1-valued constant of the given type. More... | |
| Value | constantIndex (OpBuilder &builder, Location loc, int64_t i) |
Generates a constant of index type. More... | |
| Value | constantI32 (OpBuilder &builder, Location loc, int32_t i) |
Generates a constant of i32 type. More... | |
| Value | constantI16 (OpBuilder &builder, Location loc, int16_t i) |
Generates a constant of i16 type. More... | |
| Value | constantI8 (OpBuilder &builder, Location loc, int8_t i) |
Generates a constant of i8 type. More... | |
| Value | constantI1 (OpBuilder &builder, Location loc, bool b) |
Generates a constant of i1 type. More... | |
| Value | constantAction (OpBuilder &builder, Location loc, Action action) |
Generates a constant of the given Action. More... | |
| Value | constantOverheadTypeEncoding (OpBuilder &builder, Location loc, unsigned width) |
| Generates a constant of the internal type-encoding for overhead storage. More... | |
| Value | constantPointerTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc) |
| Generates a constant of the internal type-encoding for pointer overhead storage. More... | |
| Value | constantIndexTypeEncoding (OpBuilder &builder, Location loc, SparseTensorEncodingAttr enc) |
| Generates a constant of the internal type-encoding for index overhead storage. More... | |
| Value | constantPrimaryTypeEncoding (OpBuilder &builder, Location loc, Type elemTp) |
| Generates a constant of the internal type-encoding for primary storage. More... | |
| Value | constantDimLevelTypeEncoding (OpBuilder &builder, Location loc, DimLevelType dlt) |
| Generates a constant of the internal dimension level type encoding. More... | |
| bool | isZeroRankedTensorOrScalar (Type type) |
| Value | genToPointers (OpBuilder &builder, Location loc, Value tensor, uint64_t d) |
| Infers the result type and generates ToPointersOp. More... | |
| Value | genToIndices (OpBuilder &builder, Location loc, Value tensor, uint64_t d, uint64_t cooStart) |
| Infers the result type and generates ToIndicesOp. More... | |
| Value | genToValues (OpBuilder &builder, Location loc, Value tensor) |
| Infers the result type and generates ToValuesOp. More... | |
| Value | genValMemSize (OpBuilder &builder, Location loc, Value tensor) |
| Generates code to retrieve the values size for the sparse tensor. More... | |
| void | foreachFieldInSparseTensor (SparseTensorEncodingAttr, llvm::function_ref< bool(unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>) |
| For each field that will be allocated for the given sparse tensor encoding, calls the callback with the corresponding field index, field kind, dimension (for sparse tensor level memrefs) and dimlevelType. More... | |
| void | foreachFieldAndTypeInSparseTensor (RankedTensorType, llvm::function_ref< bool(Type, unsigned, SparseTensorFieldKind, unsigned, DimLevelType)>) |
| Same as above, except that it also builds the Type for the corresponding field. More... | |
| unsigned | getNumFieldsFromEncoding (SparseTensorEncodingAttr enc) |
| Gets the total number of fields for the given sparse tensor encoding. More... | |
| unsigned | getNumDataFieldsFromEncoding (SparseTensorEncodingAttr enc) |
| Gets the total number of data fields (index arrays, pointer arrays, and a value array) for the given sparse tensor encoding. More... | |
| StorageSpecifierKind | toSpecifierKind (SparseTensorFieldKind kind) |
| SparseTensorFieldKind | toFieldKind (StorageSpecifierKind kind) |
| UnrealizedConversionCastOp | getTuple (Value tensor) |
| Returns the "tuple" value of the adapted tensor. More... | |
| Value | genTuple (OpBuilder &builder, Location loc, Type tp, ValueRange values) |
| Packs the given values as a "tuple" value. More... | |
| Value | genTuple (OpBuilder &builder, Location loc, SparseTensorDescriptor desc) |
| SparseTensorDescriptor | getDescriptorFromTensorTuple (Value tensor) |
| MutSparseTensorDescriptor | getMutDescriptorFromTensorTuple (Value tensor, SmallVectorImpl< Value > &fields) |
| static bool | containsSparseTensor (TypeRange types) |
Return true if one of the given types is a sparse tensor type. More... | |
| static ExpArity | getExpArity (Kind k) |
| static const char * | kindToOpSymbol (Kind kind) |
| static bool | isAdmissibleBranchExp (Operation *op, Block *block, Value v) |
| Ensures that sparse compiler can generate code for expression. More... | |
| static bool | isAdmissibleBranch (Operation *op, Region ®ion) |
| Ensures that sparse compiler can generate code for branch. More... | |
| static Value | insertYieldOp (RewriterBase &rewriter, Location loc, Region ®ion, ValueRange vals) |
| static Value | buildUnaryPresent (RewriterBase &rewriter, Location loc, Operation *op, Value v0) |
| static Value | buildBinaryOverlap (RewriterBase &rewriter, Location loc, Operation *op, Value v0, Value v1) |
| using mlir::sparse_tensor::complex32 = typedef std::complex<float> |
| using mlir::sparse_tensor::complex64 = typedef std::complex<double> |
| using mlir::sparse_tensor::ElementConsumer = typedef const std::function<void(const std::vector<uint64_t> &, V)> & |
| using mlir::sparse_tensor::index_type = typedef uint64_t |
This type is used in the public API at all places where MLIR expects values with the built-in type "index".
For now, we simply assume that type is 64-bit, but targets with different "index" bit widths should link with an alternatively built runtime support library.
|
strong |
|
strong |
This enum defines all the sparse representations supportable by the SparseTensor dialect.
We use a lightweight encoding to encode both the "format" per se (dense, compressed, singleton) as well as the "properties" (ordered, unique). The encoding is chosen for performance of the runtime library, and thus may change in future versions; consequently, client code should use the predicate functions defined below, rather than relying on knowledge about the particular binary encoding.
The Undef "format" is a special value used internally for cases where we need to store an undefined or indeterminate DimLevelType. It should not be used externally, since it does not indicate an actual/representable format.
| Enumerator | |
|---|---|
| Undef | |
| Dense | |
| Compressed | |
| CompressedNu | |
| CompressedNo | |
| CompressedNuNo | |
| Singleton | |
| SingletonNu | |
| SingletonNo | |
| SingletonNuNo | |
|
strong |
Shorthand aliases for the emitCInterface argument to getFunc(), createFuncCall(), and replaceOpWithFuncCall().
| Enumerator | |
|---|---|
| Off | |
| On | |
Definition at line 35 of file CodegenUtils.h.
|
strong |
| Enumerator | |
|---|---|
| kNullary | |
| kUnary | |
| kBinary | |
Definition at line 22 of file Merger.cpp.
Tensor expression kind.
|
strong |
|
strong |
|
strong |
|
strong |
| Enumerator | |
|---|---|
| StorageSpec | |
| PtrMemRef | |
| IdxMemRef | |
| ValMemRef | |
Definition at line 54 of file SparseTensorStorageLayout.h.
| Value mlir::sparse_tensor::allocaBuffer | ( | OpBuilder & | builder, |
| Location | loc, | ||
| ValueRange | values | ||
| ) |
Generates a temporary buffer, initializes it with the given contents, and returns it as type memref<? x $tp> (rather than specifying the size of the buffer).
Definition at line 395 of file CodegenUtils.cpp.
References constantIndex(), mlir::OpBuilder::create(), and genAlloca().
| Value mlir::sparse_tensor::allocDenseTensor | ( | OpBuilder & | builder, |
| Location | loc, | ||
| RankedTensorType | tensorTp, | ||
| ValueRange | sizes | ||
| ) |
Generates code to allocate a buffer of the given type, and zero initialize it.
If the buffer type has any dynamic sizes, then the sizes parameter should be as filled by sizesFromPtr(); that way we can reuse the genDimSizeCall() results generated by sizesFromPtr().
Definition at line 407 of file CodegenUtils.cpp.
References constantZero(), and mlir::OpBuilder::create().
|
static |
Definition at line 1235 of file Merger.cpp.
References mlir::Region::empty(), and insertYieldOp().
Referenced by mlir::sparse_tensor::Merger::buildExp().
| void mlir::sparse_tensor::buildSparseCompiler | ( | OpPassManager & | pm, |
| const SparseCompilerOptions & | options | ||
| ) |
Adds the "sparse-compiler" pipeline to the OpPassManager.
This is the standard pipeline for taking sparsity-agnostic IR using the sparse-tensor type and lowering it to LLVM IR with concrete representations and algorithms for sparse tensors.
Definition at line 53 of file SparseTensorPipelines.cpp.
References mlir::OpPassManager::addNestedPass(), mlir::OpPassManager::addPass(), mlir::arith::createArithExpandOpsPass(), mlir::createCanonicalizerPass(), mlir::createConvertComplexToLibmPass(), mlir::createConvertComplexToLLVMPass(), mlir::createConvertComplexToStandardPass(), mlir::createConvertFuncToLLVMPass(), mlir::createConvertLinalgToLoopsPass(), mlir::createConvertMathToLibmPass(), mlir::createConvertMathToLLVMPass(), mlir::createConvertSCFToCFPass(), mlir::createConvertVectorToLLVMPass(), mlir::createConvertVectorToSCFPass(), mlir::memref::createExpandStridedMetadataPass(), mlir::bufferization::createFinalizingBufferizePass(), mlir::createLinalgGeneralizationPass(), mlir::createLowerAffinePass(), mlir::createReconcileUnrealizedCastsPass(), mlir::createSparsificationAndBufferizationPass(), getBufferizationOptions(), and options.
Referenced by registerSparseTensorPipelines().
|
static |
Definition at line 1221 of file Merger.cpp.
References mlir::Region::empty(), and insertYieldOp().
Referenced by mlir::sparse_tensor::Merger::buildExp().
|
inline |
Generates a constant of the given Action.
Definition at line 299 of file CodegenUtils.h.
References constantI32().
|
inline |
Generates a constant of the internal dimension level type encoding.
Definition at line 332 of file CodegenUtils.h.
References constantI8().
Generates a constant of i1 type.
Definition at line 294 of file CodegenUtils.h.
References mlir::OpBuilder::create().
Referenced by createEqCompare(), createLessThanCompare(), genIf(), and genInsertionStore().
Generates a constant of i16 type.
Definition at line 284 of file CodegenUtils.h.
References mlir::OpBuilder::create().
Generates a constant of i32 type.
Definition at line 279 of file CodegenUtils.h.
References mlir::OpBuilder::create().
Referenced by constantAction(), constantOverheadTypeEncoding(), and constantPrimaryTypeEncoding().
Generates a constant of i8 type.
Definition at line 289 of file CodegenUtils.h.
References mlir::OpBuilder::create().
Referenced by constantDimLevelTypeEncoding().
Generates a constant of index type.
Definition at line 274 of file CodegenUtils.h.
References mlir::OpBuilder::create().
Referenced by allocaBuffer(), concatSizesFromInputs(), createBinarySearchFunc(), createPartitionFunc(), createScanLoop(), createSortNonstableFunc(), createSortStableFunc(), mlir::sparse_tensor::LoopEmitter::enterFilterLoopOverTensorAtDim(), mlir::sparse_tensor::LoopEmitter::enterLoopOverTensorAtDim(), mlir::sparse_tensor::LoopEmitter::enterNewLoopSeq(), forEachIJPairInXs(), mlir::sparse_tensor::LoopEmitter::genAffine(), genAlloca(), genDenseTensorOrSparseConstantIterLoop(), genIndexAndValueForSparse(), genInsertionStore(), genReshapeDstShape(), mlir::sparse_tensor::SparseTensorDescriptor::getIdxMemRefOrView(), matchAndRewriteSortOp(), parseGEPIndices(), sizesForTensor(), specializeForLoopForUnrolling(), specializeParallelLoopForUnrolling(), storeIndices(), and translateIndicesArray().
|
inline |
Generates a constant of the internal type-encoding for index overhead storage.
Definition at line 319 of file CodegenUtils.h.
References constantOverheadTypeEncoding().
Generates a 1-valued constant of the given type.
This supports all the same types as constantZero.
Definition at line 263 of file CodegenUtils.h.
References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Builder::getArrayAttr(), getOneAttr(), and mlir::Builder::getZeroAttr().
|
inline |
Generates a constant of the internal type-encoding for overhead storage.
Definition at line 304 of file CodegenUtils.h.
References constantI32(), and overheadTypeEncoding().
Referenced by constantIndexTypeEncoding(), and constantPointerTypeEncoding().
|
inline |
Generates a constant of the internal type-encoding for pointer overhead storage.
Definition at line 312 of file CodegenUtils.h.
References constantOverheadTypeEncoding().
|
inline |
Generates a constant of the internal type-encoding for primary storage.
Definition at line 325 of file CodegenUtils.h.
References constantI32(), and primaryTypeEncoding().
Generates a 0-valued constant of the given type.
In addition to the scalar types (ComplexType, `FloatType, IndexType, IntegerType), this also works for RankedTensorType and VectorType (for which it generates a constant DenseElementsAttr of zeros).
Definition at line 252 of file CodegenUtils.h.
References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Builder::getArrayAttr(), and mlir::Builder::getZeroAttr().
Referenced by allocDenseTensor(), genBuffers(), genInsertionLoad(), and genIsNonzero().
|
static |
Return true if one of the given types is a sparse tensor type.
Definition at line 32 of file SparsificationAndBufferizationPass.cpp.
References getSparseTensorEncoding().
Referenced by mlir::sparse_tensor::SparsificationAndBufferizationPass::runDenseBufferization().
| func::CallOp mlir::sparse_tensor::createFuncCall | ( | OpBuilder & | builder, |
| Location | loc, | ||
| StringRef | name, | ||
| TypeRange | resultType, | ||
| ValueRange | operands, | ||
| EmitCInterface | emitCInterface | ||
| ) |
Creates a CallOp to the function reference returned by getFunc() in the builder's module.
Definition at line 362 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), mlir::OpBuilder::getBlock(), getFunc(), mlir::Operation::getParentOfType(), and mlir::Block::getParentOp().
Generates code to deallocate a dense buffer.
Definition at line 424 of file CodegenUtils.cpp.
References mlir::OpBuilder::create().
| void mlir::sparse_tensor::foreachFieldAndTypeInSparseTensor | ( | RankedTensorType | rType, |
| llvm::function_ref< bool(Type, unsigned, SparseTensorFieldKind, unsigned, DimLevelType)> | callback | ||
| ) |
Same as above, except that it also builds the Type for the corresponding field.
Definition at line 181 of file SparseTensorStorageLayout.cpp.
References foreachFieldInSparseTensor(), getSparseTensorEncoding(), IdxMemRef, PtrMemRef, StorageSpec, and ValMemRef.
Referenced by convertSparseTensorType().
| void mlir::sparse_tensor::foreachFieldInSparseTensor | ( | SparseTensorEncodingAttr | enc, |
| llvm::function_ref< bool(unsigned, SparseTensorFieldKind, unsigned, DimLevelType)> | callback | ||
| ) |
For each field that will be allocated for the given sparse tensor encoding, calls the callback with the corresponding field index, field kind, dimension (for sparse tensor level memrefs) and dimlevelType.
The field index always starts with zero and increments by one between two callback invocations. Ideally, all other methods should rely on this function to query a sparse tensor fields instead of relying on ad-hoc index computation.
Definition at line 138 of file SparseTensorStorageLayout.cpp.
References getCOOStart(), getDimLevelType(), IdxMemRef, isCompressedDLT(), isDenseDLT(), isSingletonDLT(), kDataFieldStartingIdx, PtrMemRef, RETURN_ON_FALSE, StorageSpec, Undef, and ValMemRef.
Referenced by foreachFieldAndTypeInSparseTensor(), mlir::sparse_tensor::StorageLayout::foreachFieldInSparseTensor(), getNumDataFieldsFromEncoding(), and getNumFieldsFromEncoding().
| void mlir::sparse_tensor::foreachInSparseConstant | ( | Location | loc, |
| RewriterBase & | rewriter, | ||
| SparseElementsAttr | attr, | ||
| function_ref< void(ArrayRef< Value >, Value)> | callback | ||
| ) |
Iterate over a sparse constant, generates constantOp for value and indices.
E.g., sparse<[ [0], [28], [31] ], [ (-5.13, 2.0), (3.0, 4.0), (5.0, 6.0) ] > => c1 = arith.constant 0 v1 = complex.constant (5.13, 2.0) callback({c1}, v1)
c2 = arith.constant 28 v2 = complex.constant (3.0, 4.0) callback({c2}, v2)
c3 = arith.constant 31 v3 = complex.constant (5.0, 6.0) callback({c3}, v3)
Definition at line 501 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), mlir::DenseElementsAttr::getValues(), and mlir::DenseElementsAttr::size().
Referenced by genForeachOnSparseConstant().
| Value mlir::sparse_tensor::genAlloca | ( | OpBuilder & | builder, |
| Location | loc, | ||
| unsigned | sz, | ||
| Type | tp, | ||
| bool | staticShape = false |
||
| ) |
Generates an uninitialized temporary buffer of the given size and type, and returns it as type memref<? x $tp> (staticShape=false) or memref<$sz x $tp> (staticShape=true).
Definition at line 375 of file CodegenUtils.cpp.
References constantIndex(), mlir::OpBuilder::create(), and genAlloca().
Generates an uninitialized temporary buffer of the given size and type, but returns it as type memref<? x $tp> (rather than as type memref<$sz x $tp>).
Definition at line 384 of file CodegenUtils.cpp.
References mlir::OpBuilder::create().
Referenced by allocaBuffer(), and genAlloca().
Generates an uninitialized temporary buffer with room for one value of the given type, and returns the memref<$tp>.
Definition at line 390 of file CodegenUtils.cpp.
References mlir::OpBuilder::create().
| void mlir::sparse_tensor::genDenseTensorOrSparseConstantIterLoop | ( | OpBuilder & | builder, |
| Location | loc, | ||
| Value | src, | ||
| unsigned | rank, | ||
| function_ref< void(OpBuilder &, Location, Value, ValueRange)> | bodyBuilder | ||
| ) |
Generates the loop structure to iterate over a dense tensor or a sparse tensor constant to support the lowering of dense-to-sparse convert operator.
Definition at line 441 of file CodegenUtils.cpp.
References mlir::scf::buildLoopNest(), constantIndex(), mlir::linalg::createOrFoldDimOp(), genIndexAndValueForDense(), genIndexAndValueForSparse(), and genSplitSparseConstant().
| Value mlir::sparse_tensor::genIsNonzero | ( | OpBuilder & | builder, |
| mlir::Location | loc, | ||
| Value | v | ||
| ) |
Generates the comparison v != 0 where v is of numeric type.
For floating types, we use the "unordered" comparator (i.e., returns true if v is NaN).
Definition at line 222 of file CodegenUtils.cpp.
References constantZero(), mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::Value::getType(), mlir::Type::isa(), and mlir::Type::isIntOrIndex().
Referenced by genValueForDense().
| void mlir::sparse_tensor::genReshapeDstShape | ( | Location | loc, |
| PatternRewriter & | rewriter, | ||
| SmallVectorImpl< Value > & | dstShape, | ||
| ArrayRef< Value > | srcShape, | ||
| ArrayRef< int64_t > | staticDstShape, | ||
| ArrayRef< ReassociationIndices > | reassociation | ||
| ) |
Computes the shape of destination tensor of a reshape operator.
This is only used when operands have dynamic shape. The shape of the destination is stored into dstShape.
Definition at line 237 of file CodegenUtils.cpp.
References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().
| Value mlir::sparse_tensor::genToIndices | ( | OpBuilder & | builder, |
| Location | loc, | ||
| Value | tensor, | ||
| uint64_t | d, | ||
| uint64_t | cooStart | ||
| ) |
Infers the result type and generates ToIndicesOp.
If the dim is within a COO region, the result type is a memref with unknown stride and offset. Otherwise, the result type is a memref without any specified layout.
Definition at line 576 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), get1DMemRefType(), mlir::Builder::getIndexAttr(), getIndexOverheadType(), getRankedTensorType(), and getSparseTensorEncoding().
Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().
| Value mlir::sparse_tensor::genToPointers | ( | OpBuilder & | builder, |
| Location | loc, | ||
| Value | tensor, | ||
| uint64_t | d | ||
| ) |
Infers the result type and generates ToPointersOp.
Definition at line 566 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), get1DMemRefType(), mlir::Builder::getIndexAttr(), getPointerOverheadType(), getRankedTensorType(), and getSparseTensorEncoding().
Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().
Infers the result type and generates ToValuesOp.
Definition at line 586 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), get1DMemRefType(), and getRankedTensorType().
Referenced by mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().
|
inline |
Definition at line 392 of file SparseTensorStorageLayout.h.
References genTuple(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getFields(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getTensorType().
|
inline |
Packs the given values as a "tuple" value.
Definition at line 386 of file SparseTensorStorageLayout.h.
References mlir::OpBuilder::create().
Referenced by genTuple(), and mlir::SparseTensorTypeToBufferConverter::SparseTensorTypeToBufferConverter().
Generates code to retrieve the values size for the sparse tensor.
Definition at line 594 of file CodegenUtils.cpp.
References getDescriptorFromTensorTuple(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getValMemSize().
| Value mlir::sparse_tensor::genValueForDense | ( | OpBuilder & | builder, |
| Location | loc, | ||
| Value | tensor, | ||
| ValueRange | ivs | ||
| ) |
Generates the code to read the value from tensor[ivs].
The generated code looks like the following and the insertion point after this routine is inside the if-then branch behind the assignment to ind. if (tensor[ivs] != 0) insert_point
Definition at line 429 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), genIsNonzero(), and mlir::OpBuilder::setInsertionPointToStart().
Referenced by genIndexAndValueForDense().
|
inline |
Generates a 1D MemRefType with a dynamic size.
When withLayout is set, the returned memref has a layout has unknown strides and offsets. Otherwise, a memref with a standard unit stride zero offset layout is returned.
Definition at line 190 of file CodegenUtils.h.
References mlir::Type::getContext().
Referenced by genToIndices(), genToPointers(), and genToValues().
| unsigned mlir::sparse_tensor::getCOOStart | ( | SparseTensorEncodingAttr | enc | ) |
Returns the starting dimension for a trailing COO region that spans across at least two dimensions.
If no such COO region is found, returns the rank of the tensor.
Definition at line 431 of file SparseTensorDialect.cpp.
References isCOOType().
Referenced by foreachFieldInSparseTensor(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getAOSMemRef(), mlir::sparse_tensor::StorageLayout::getFieldIndexAndStride(), mlir::sparse_tensor::SparseTensorDescriptor::getIdxMemRefOrView(), and mlir::sparse_tensor::LoopEmitter::initializeLoopEmit().
|
inline |
Definition at line 397 of file SparseTensorStorageLayout.h.
References getTuple().
Referenced by genValMemSize().
|
constexpr |
Convert a LevelFormat to its corresponding DimLevelType with the given properties.
Returns std::nullopt when the properties are not applicable for the input level format. TODO: factor out a new LevelProperties type so we can add new properties without changing this function's signature
Definition at line 257 of file Enums.h.
References isValidDLT().
Referenced by computeIterationGraph(), findSparseAnnotations(), foreachFieldInSparseTensor(), genConstantDenseAddressFromLevel(), getDimLevelType(), getNormalizedEncodingForSpecifier(), getNumCompoundAffineOnSparseDims(), isCompressedDim(), isCOOType(), isDenseDim(), isOrderedDim(), isSingletonDim(), isUniqueDim(), mlirSparseTensorEncodingAttrGetDimLevelType(), mlirSparseTensorEncodingGetNumDimLevelTypes(), and translateBitsToTidDimPairs().
|
inline |
Definition at line 84 of file SparseTensor.h.
References getDimLevelType(), and getSparseTensorEncoding().
|
inline |
Definition at line 75 of file SparseTensor.h.
References Dense.
Definition at line 28 of file Merger.cpp.
References kAbsC, kAbsF, kAbsI, kAddC, kAddF, kAddI, kAndI, kBinary, kBinaryBranch, kBitCast, kCastFS, kCastFU, kCastIdx, kCastS, kCastSF, kCastU, kCastUF, kCeilF, kCIm, kCRe, kDivC, kDivF, kDivS, kDivU, kExpm1C, kExpm1F, kExtF, kFloorF, kIndex, kInvariant, kLog1pC, kLog1pF, kMulC, kMulF, kMulI, kNegC, kNegF, kNegI, kNullary, kOrI, kReduce, kSelect, kShlI, kShrS, kShrU, kSinC, kSinF, kSqrtC, kSqrtF, kSubC, kSubF, kSubI, kTanhC, kTanhF, kTensor, kTruncF, kTruncI, kUnary, and kXorI.
Referenced by mlir::sparse_tensor::Merger::expContainsTensor(), and mlir::sparse_tensor::Merger::hasNegateOnOut().
| FlatSymbolRefAttr mlir::sparse_tensor::getFunc | ( | ModuleOp | module, |
| StringRef | name, | ||
| TypeRange | resultType, | ||
| ValueRange | operands, | ||
| EmitCInterface | emitCInterface | ||
| ) |
Returns a function reference (first hit also inserts into module).
Sets the "_emit_c_interface" on the function declaration when requested, so that LLVM lowering generates a wrapper function that takes care of ABI complications with passing in and returning MemRefs to C functions.
Definition at line 342 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), mlir::ValueRange::getTypes(), and mlir::Operation::setAttr().
Referenced by createFuncCall().
Returns the mlir::Type for index overhead storage.
Definition at line 135 of file CodegenUtils.cpp.
References getOverheadType(), and indexOverheadTypeEncoding().
Referenced by genToIndices().
|
constexpr |
Convert a DimLevelType to its corresponding LevelFormat.
Returns std::nullopt when input dlt is Undef.
Definition at line 245 of file Enums.h.
References Undef.
Referenced by getNormalizedEncodingForSpecifier().
|
inline |
Convenience method to abbreviate casting getType().
Definition at line 49 of file SparseTensor.h.
Referenced by getMangledSortHelperFuncName(), isMatchingWidth(), matchAndRewriteSortOp(), and reshapeValuesToLevels().
|
inline |
Definition at line 403 of file SparseTensorStorageLayout.h.
References getTuple().
| unsigned mlir::sparse_tensor::getNumDataFieldsFromEncoding | ( | SparseTensorEncodingAttr | enc | ) |
Gets the total number of data fields (index arrays, pointer arrays, and a value array) for the given sparse tensor encoding.
Definition at line 232 of file SparseTensorStorageLayout.cpp.
References foreachFieldInSparseTensor(), getNumFieldsFromEncoding(), and kDataFieldStartingIdx.
| unsigned mlir::sparse_tensor::getNumFieldsFromEncoding | ( | SparseTensorEncodingAttr | enc | ) |
Gets the total number of fields for the given sparse tensor encoding.
Definition at line 220 of file SparseTensorStorageLayout.cpp.
References foreachFieldInSparseTensor().
Referenced by getNumDataFieldsFromEncoding(), mlir::sparse_tensor::StorageLayout::getNumFieldsFromEncoding(), and mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::SparseTensorDescriptorImpl().
| mlir::Attribute mlir::sparse_tensor::getOneAttr | ( | Builder & | builder, |
| Type | tp | ||
| ) |
Generates a 1-valued attribute of the given type.
This supports all the same types as getZeroAttr; however, unlike getZeroAttr, for unsupported types we raise llvm_unreachable rather than returning a null attribute.
Definition at line 207 of file CodegenUtils.cpp.
References mlir::Type::cast(), mlir::Type::dyn_cast(), mlir::DenseElementsAttr::get(), mlir::Builder::getFloatAttr(), mlir::Builder::getIndexAttr(), mlir::Builder::getIntegerAttr(), and mlir::Type::isa().
Referenced by constantOne().
Returns the equivalent of void* for opaque arguments to the execution engine.
Definition at line 371 of file CodegenUtils.cpp.
References mlir::Builder::getI8Type().
| Type mlir::sparse_tensor::getOverheadType | ( | Builder & | builder, |
| OverheadType | ot | ||
| ) |
Converts the internal type-encoding for overhead storage to an mlir::Type.
Definition at line 104 of file CodegenUtils.cpp.
References mlir::Builder::getIndexType(), mlir::Builder::getIntegerType(), kIndex, kU16, kU32, kU64, and kU8.
Referenced by getIndexOverheadType(), and getPointerOverheadType().
| Type mlir::sparse_tensor::getPointerOverheadType | ( | Builder & | builder, |
| SparseTensorEncodingAttr | enc | ||
| ) |
Returns the mlir::Type for pointer overhead storage.
Definition at line 130 of file CodegenUtils.cpp.
References getOverheadType(), and pointerOverheadTypeEncoding().
Referenced by genToPointers().
|
inline |
Convenience method to abbreviate casting getType().
Definition at line 43 of file SparseTensor.h.
Referenced by genToIndices(), genToPointers(), genToValues(), getTypeRank(), and mlir::sparse_tensor::LoopEmitter::initialize().
| SparseTensorEncodingAttr mlir::sparse_tensor::getSparseTensorEncoding | ( | Type | type | ) |
Convenience method to get a sparse encoding attribute from a type.
Returns null-attribute for any type without an encoding.
Definition at line 400 of file SparseTensorDialect.cpp.
References mlir::Type::dyn_cast().
Referenced by computeIterationGraph(), containsSparseTensor(), convertSparseTensorType(), findSparseAnnotations(), foreachFieldAndTypeInSparseTensor(), genBuffers(), genConstantDenseAddressFromLevel(), genIndex(), genInvariants(), genResult(), genSubscript(), genToIndices(), genToPointers(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getAOSMemRef(), getDimLevelType(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getIdxMemRefIndexAndStride(), mlir::sparse_tensor::SparseTensorDescriptor::getIdxMemRefOrView(), mlir::sparse_tensor::SparseTensorSpecifier::getInitValue(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::getMemRefFieldIndex(), getNumCompoundAffineOnSparseDims(), getUnorderedCOOFromTypeWithOrdering(), hasSameDimOrdering(), mlir::sparse_tensor::LoopEmitter::initialize(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), isAdmissibleTensorExp(), isAllDimOrdered(), isSparseTensor(), isUniqueCOOType(), mlir::sparse_tensor::SparseTensorDescriptorImpl< ValueArrayRef >::SparseTensorDescriptorImpl(), mlir::SparseTensorTypeToBufferConverter::SparseTensorTypeToBufferConverter(), toOrigDim(), toStoredDim(), and translateBitsToTidDimPairs().
Scans to top of generated loop.
Definition at line 491 of file CodegenUtils.cpp.
References mlir::Operation::getParentOp().
|
inline |
Returns the "tuple" value of the adapted tensor.
Definition at line 381 of file SparseTensorStorageLayout.h.
References mlir::Value::getDefiningOp().
Referenced by getDescriptorFromTensorTuple(), and getMutDescriptorFromTensorTuple().
| OverheadType mlir::sparse_tensor::indexOverheadTypeEncoding | ( | SparseTensorEncodingAttr | enc | ) |
Returns the OverheadType for index overhead storage.
Definition at line 126 of file CodegenUtils.cpp.
References overheadTypeEncoding().
Referenced by getIndexOverheadType().
|
static |
Definition at line 1204 of file Merger.cpp.
References mlir::Region::begin(), mlir::Region::cloneInto(), mlir::OpBuilder::create(), mlir::RewriterBase::eraseOp(), mlir::Region::front(), mlir::Block::getTerminator(), and mlir::RewriterBase::mergeBlockBefore().
Referenced by buildBinaryOverlap(), mlir::sparse_tensor::Merger::buildExp(), and buildUnaryPresent().
Ensures that sparse compiler can generate code for branch.
Definition at line 1019 of file Merger.cpp.
References mlir::Region::empty(), mlir::Region::front(), mlir::Operation::getOperand(), mlir::Block::getTerminator(), and isAdmissibleBranchExp().
|
static |
Ensures that sparse compiler can generate code for expression.
Definition at line 999 of file Merger.cpp.
References mlir::Value::dyn_cast(), mlir::Operation::getBlock(), mlir::Value::getDefiningOp(), mlir::Operation::getNumOperands(), and mlir::Operation::getOperand().
Referenced by isAdmissibleBranch().
|
constexpr |
Definition at line 119 of file Enums.h.
Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().
|
inline |
Convenience function to test for compressed dimension (0 <= d < rank).
Definition at line 94 of file SparseTensor.h.
References getDimLevelType(), and isCompressedDLT().
Referenced by isCOOType().
|
inline |
Convenience function to test for compressed dimension (0 <= d < rank).
Definition at line 109 of file SparseTensor.h.
References getDimLevelType(), and isCompressedDLT().
|
constexpr |
Check if the DimLevelType is compressed (regardless of properties).
Definition at line 222 of file Enums.h.
References Compressed.
Referenced by computeIterationGraph(), mlir::sparse_tensor::LoopEmitter::enterCoIterationOverTensorsAtDims(), mlir::sparse_tensor::LoopEmitter::enterLoopOverTensorAtDim(), mlir::sparse_tensor::SparseTensorNNZ::forallIndices(), foreachFieldInSparseTensor(), genFor(), genIf(), mlir::sparse_tensor::Merger::hasAnySparse(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), isAdmissibleTensorExp(), isCompressedDim(), mlir::sparse_tensor::SparseTensorStorageBase::isCompressedLvl(), mlir::sparse_tensor::Merger::simplifyCond(), mlir::sparse_tensor::SparseTensorNNZ::SparseTensorNNZ(), mlir::sparse_tensor::SparseTensorStorage< P, I, V >::SparseTensorStorage(), and mlir::sparse_tensor::SparseTensorStorageBase::SparseTensorStorageBase().
|
inline |
Convenience function to test for dense dimension (0 <= d < rank).
Definition at line 89 of file SparseTensor.h.
References getDimLevelType(), and isDenseDLT().
|
inline |
Convenience function to test for dense dimension (0 <= d < rank).
Definition at line 104 of file SparseTensor.h.
References getDimLevelType(), and isDenseDLT().
|
constexpr |
Check if the DimLevelType is dense.
Definition at line 213 of file Enums.h.
References Dense.
Referenced by computeIterationGraph(), mlir::sparse_tensor::LoopEmitter::enterFilterLoopOverTensorAtDim(), mlir::sparse_tensor::LoopEmitter::enterLoopOverTensorAtDim(), findAffine(), foreachFieldInSparseTensor(), genConstantDenseAddressFromLevel(), genIf(), getNumCompoundAffineOnSparseDims(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), isAdmissibleTensorExp(), isDenseDim(), mlir::sparse_tensor::SparseTensorStorageBase::isDenseLvl(), mlir::sparse_tensor::Merger::simplifyCond(), mlir::sparse_tensor::SparseTensorNNZ::SparseTensorNNZ(), mlir::sparse_tensor::SparseTensorStorage< P, I, V >::SparseTensorStorage(), mlir::sparse_tensor::SparseTensorStorageBase::SparseTensorStorageBase(), and translateBitsToTidDimPairs().
|
constexpr |
Definition at line 107 of file Enums.h.
Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().
|
constexpr |
|
inline |
Convenience function to test for ordered property in the given dimension (0 <= d < rank).
Definition at line 124 of file SparseTensor.h.
References getDimLevelType(), and isOrderedDLT().
|
constexpr |
Check if the DimLevelType is ordered (regardless of storage format).
Definition at line 234 of file Enums.h.
Referenced by isAllDimOrdered(), isOrderedDim(), and mlir::sparse_tensor::SparseTensorStorageBase::isOrderedLvl().
|
constexpr |
Definition at line 115 of file Enums.h.
Referenced by mlir::sparse_tensor::SparseTensorReader::canReadAs().
|
inline |
Convenience function to test for singleton dimension (0 <= d < rank).
Definition at line 99 of file SparseTensor.h.
References getDimLevelType(), and isSingletonDLT().
Referenced by isCOOType().
|
inline |
Convenience function to test for singleton dimension (0 <= d < rank).
Definition at line 114 of file SparseTensor.h.
References getDimLevelType(), and isSingletonDLT().
|
constexpr |
Check if the DimLevelType is singleton (regardless of properties).
Definition at line 228 of file Enums.h.
References Singleton.
Referenced by computeIterationGraph(), mlir::sparse_tensor::LoopEmitter::enterCoIterationOverTensorsAtDims(), mlir::sparse_tensor::LoopEmitter::enterLoopOverTensorAtDim(), foreachFieldInSparseTensor(), genFor(), genIf(), mlir::sparse_tensor::Merger::hasAnySparse(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), isAdmissibleTensorExp(), isSingletonDim(), mlir::sparse_tensor::SparseTensorStorageBase::isSingletonLvl(), mlir::sparse_tensor::Merger::simplifyCond(), mlir::sparse_tensor::SparseTensorNNZ::SparseTensorNNZ(), mlir::sparse_tensor::SparseTensorStorage< P, I, V >::SparseTensorStorage(), mlir::sparse_tensor::SparseTensorStorageBase::SparseTensorStorageBase(), and verifySparsifierGetterSetter().
|
constexpr |
Check if the DimLevelType is the special undefined value.
Definition at line 208 of file Enums.h.
References Undef.
Referenced by computeIterationGraph(), findAffine(), genIf(), isAdmissibleTensorExp(), and translateBitsToTidDimPairs().
| bool mlir::sparse_tensor::isUniqueCOOType | ( | RankedTensorType | tp | ) |
Returns true iff the given type is a type for a COO tensor with the last dimension level type being unique.
Definition at line 426 of file SparseTensorDialect.cpp.
References getSparseTensorEncoding(), and isCOOType().
|
inline |
Convenience function to test for unique property in the given dimension (0 <= d < rank).
Definition at line 130 of file SparseTensor.h.
References getDimLevelType(), and isUniqueDLT().
|
constexpr |
Check if the DimLevelType is unique (regardless of storage format).
Definition at line 239 of file Enums.h.
Referenced by isCOOType(), isUniqueDim(), and mlir::sparse_tensor::SparseTensorStorageBase::isUniqueLvl().
|
constexpr |
Check that the DimLevelType contains a valid (possibly undefined) value.
Definition at line 198 of file Enums.h.
Referenced by getDimLevelType(), and mlir::sparse_tensor::Merger::setDimAndDimLevelType().
|
inline |
Definition at line 337 of file CodegenUtils.h.
References mlir::Type::dyn_cast().
Referenced by mlir::sparse_tensor::LoopEmitter::initialize().
|
static |
Definition at line 548 of file Merger.cpp.
References kAbsC, kAbsF, kAbsI, kAddC, kAddF, kAddI, kAndI, kBinary, kBinaryBranch, kBitCast, kCastFS, kCastFU, kCastIdx, kCastS, kCastSF, kCastU, kCastUF, kCeilF, kCIm, kCRe, kDivC, kDivF, kDivS, kDivU, kExpm1C, kExpm1F, kExtF, kFloorF, kIndex, kInvariant, kLog1pC, kLog1pF, kMulC, kMulF, kMulI, kNegC, kNegF, kNegI, kOrI, kReduce, kSelect, kShlI, kShrS, kShrU, kSinC, kSinF, kSqrtC, kSqrtF, kSubC, kSubF, kSubI, kTanhC, kTanhF, kTensor, kTruncF, kTruncI, kUnary, and kXorI.
Referenced by mlir::sparse_tensor::Merger::dumpExp().
| OverheadType mlir::sparse_tensor::overheadTypeEncoding | ( | Type | tp | ) |
Converts an overhead storage type to its internal type-encoding.
Definition at line 96 of file CodegenUtils.cpp.
References mlir::Type::dyn_cast(), mlir::Type::isIndex(), kIndex, and overheadTypeEncoding().
| OverheadType mlir::sparse_tensor::overheadTypeEncoding | ( | unsigned | width | ) |
Converts an overhead storage bitwidth to its internal type-encoding.
Definition at line 80 of file CodegenUtils.cpp.
References kIndex, kU16, kU32, kU64, and kU8.
Referenced by constantOverheadTypeEncoding(), indexOverheadTypeEncoding(), overheadTypeEncoding(), overheadTypeFunctionSuffix(), and pointerOverheadTypeEncoding().
| StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix | ( | OverheadType | ot | ) |
Convert OverheadType to its function-name suffix.
Definition at line 144 of file CodegenUtils.cpp.
References CASE, kIndex, and MLIR_SPARSETENSOR_FOREVERY_FIXED_O.
Referenced by overheadTypeFunctionSuffix().
| StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix | ( | Type | overheadTp | ) |
Converts an overhead storage type to its function-name suffix.
Definition at line 157 of file CodegenUtils.cpp.
References overheadTypeEncoding(), and overheadTypeFunctionSuffix().
| OverheadType mlir::sparse_tensor::pointerOverheadTypeEncoding | ( | SparseTensorEncodingAttr | enc | ) |
Returns the OverheadType for pointer overhead storage.
Definition at line 121 of file CodegenUtils.cpp.
References overheadTypeEncoding().
Referenced by getPointerOverheadType().
| PrimaryType mlir::sparse_tensor::primaryTypeEncoding | ( | Type | elemTp | ) |
Converts a primary storage type to its internal type-encoding.
Definition at line 161 of file CodegenUtils.cpp.
References mlir::Type::dyn_cast(), mlir::Type::isBF16(), mlir::Type::isF16(), mlir::Type::isF32(), mlir::Type::isF64(), mlir::Type::isInteger(), kBF16, kC32, kC64, kF16, kF32, kF64, kI16, kI32, kI64, and kI8.
Referenced by constantPrimaryTypeEncoding(), and primaryTypeFunctionSuffix().
| StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix | ( | PrimaryType | pt | ) |
Convert PrimaryType to its function-name suffix.
Definition at line 188 of file CodegenUtils.cpp.
References CASE, and MLIR_SPARSETENSOR_FOREVERY_V.
Referenced by primaryTypeFunctionSuffix().
| StringRef mlir::sparse_tensor::primaryTypeFunctionSuffix | ( | Type | elemTp | ) |
Converts a primary storage type to its function-name suffix.
Definition at line 199 of file CodegenUtils.cpp.
References primaryTypeEncoding(), and primaryTypeFunctionSuffix().
| void mlir::sparse_tensor::registerBufferizableOpInterfaceExternalModels | ( | DialectRegistry & | registry | ) |
Definition at line 130 of file BufferizableOpInterfaceImpl.cpp.
References mlir::DialectRegistry::addExtension().
Referenced by mlir::registerAllDialects().
| void mlir::sparse_tensor::registerSparseTensorPipelines | ( | ) |
Registers all pipelines for the sparse_tensor dialect.
At present, this includes only "sparse-compiler".
Definition at line 94 of file SparseTensorPipelines.cpp.
References buildSparseCompiler().
Referenced by mlir::registerAllPasses().
| Value mlir::sparse_tensor::reshapeValuesToLevels | ( | OpBuilder & | builder, |
| Location | loc, | ||
| SparseTensorEncodingAttr | enc, | ||
| const SmallVectorImpl< Value > & | dimSizes, | ||
| Value | valuesBuffer, | ||
| Value | idxBuffer | ||
| ) |
Reshapes the linear values buffer for an annotated all dense sparse tensor to match the shape of the corresponding dense tensor to support direct access of the buffer through indices.
Definition at line 546 of file CodegenUtils.cpp.
References mlir::OpBuilder::create(), mlir::Builder::getIndexType(), getMemRefType(), storeIndices(), and toOrigDim().
| void mlir::sparse_tensor::sizesFromSrc | ( | OpBuilder & | builder, |
| SmallVectorImpl< Value > & | sizes, | ||
| Location | loc, | ||
| Value | src | ||
| ) |
Populates given sizes array from dense tensor or sparse tensor constant.
Definition at line 483 of file CodegenUtils.cpp.
References mlir::Type::cast(), mlir::linalg::createOrFoldDimOp(), and mlir::Value::getType().
Referenced by concatSizesFromInputs().
| void mlir::sparse_tensor::storeIndices | ( | OpBuilder & | builder, |
| Location | loc, | ||
| unsigned | rank, | ||
| Value | ind, | ||
| ValueRange | ivs, | ||
| unsigned | offsetDim = 0, |
||
| Value | offset = Value() |
||
| ) |
Converts the vector indices and store it into the memory pointed by ind, apply (optional) offset on offsetDim.
Definition at line 534 of file CodegenUtils.cpp.
References constantIndex(), and mlir::OpBuilder::create().
Referenced by haveNoReadsAfterWriteExceptSameIndex(), and reshapeValuesToLevels().
|
inline |
Definition at line 103 of file SparseTensorStorageLayout.h.
Referenced by mlir::sparse_tensor::StorageLayout::getMemRefFieldIndex().
|
inline |
Returns string representation of the given dimension level type.
Definition at line 170 of file Enums.h.
References Compressed, CompressedNo, CompressedNu, CompressedNuNo, Dense, Singleton, SingletonNo, SingletonNu, SingletonNuNo, and Undef.
Referenced by mlir::sparse_tensor::Merger::dumpBits(), and parseDLT().
| uint64_t mlir::sparse_tensor::toOrigDim | ( | RankedTensorType | type, |
| uint64_t | d | ||
| ) |
Convenience method to translate the given stored dimension to the original dimension (0 <= d < rank).
Definition at line 470 of file SparseTensorDialect.cpp.
References getSparseTensorEncoding(), and toOrigDim().
| uint64_t mlir::sparse_tensor::toOrigDim | ( | SparseTensorEncodingAttr | enc, |
| uint64_t | d | ||
| ) |
Definition at line 443 of file SparseTensorDialect.cpp.
Referenced by computeIterationGraph(), findSparseAnnotations(), genConstantDenseAddressFromLevel(), genIndex(), genInvariants(), getNumCompoundAffineOnSparseDims(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), reshapeValuesToLevels(), toOrigDim(), and translateBitsToTidDimPairs().
|
inline |
Definition at line 98 of file SparseTensorStorageLayout.h.
References StorageSpec.
| uint64_t mlir::sparse_tensor::toStoredDim | ( | RankedTensorType | type, |
| uint64_t | d | ||
| ) |
Convenience method to translate the given original dimension to the stored dimension (0 <= d < rank).
Definition at line 475 of file SparseTensorDialect.cpp.
References getSparseTensorEncoding(), and toStoredDim().
| uint64_t mlir::sparse_tensor::toStoredDim | ( | SparseTensorEncodingAttr | enc, |
| uint64_t | d | ||
| ) |
Definition at line 455 of file SparseTensorDialect.cpp.
References mlir::getAffineDimExpr().
Referenced by toStoredDim().
| void mlir::sparse_tensor::translateIndicesArray | ( | OpBuilder & | builder, |
| Location | loc, | ||
| ArrayRef< ReassociationIndices > | reassociation, | ||
| ValueRange | srcIndices, | ||
| ArrayRef< Value > | srcShape, | ||
| ArrayRef< Value > | dstShape, | ||
| SmallVectorImpl< Value > & | dstIndices | ||
| ) |
Translate indices during a reshaping operation.
Definition at line 293 of file CodegenUtils.cpp.
References constantIndex(), mlir::OpBuilder::create(), and mlir::detail::enumerate().
|
inline |
Writes the sparse tensor to filename in extended FROSTT format.
Definition at line 359 of file File.h.
References mlir::sparse_tensor::SparseTensorCOO< V >::getDimSizes(), mlir::sparse_tensor::SparseTensorCOO< V >::getElements(), and mlir::sparse_tensor::SparseTensorCOO< V >::getRank().
1.9.1