MLIR
16.0.0git
|
Namespaces | |
detail | |
Classes | |
class | CombiningKindAttr |
An attribute that specifies the combining function for vector.contract , and vector.reduction . More... | |
class | ContractionOpLowering |
Progressive lowering of ContractionOp. More... | |
class | ContractionOpToDotLowering |
Progressive lowering of a vector.contract a, b, c with row-major matmul semantics to an output-size-unrolled sequence: ``` out = arith.constant ... More... | |
class | ContractionOpToMatmulOpLowering |
Progressive lowering of a vector.contract a, b, c with row-major matmul semantics to: %flattened_a = vector.shape_cast %a %flattened_b = vector.shape_cast %b %flattened_d = vector.matmul %flattened_a, %flattened_b %d = vector.shape_cast %%flattened_d %e = add %c, %d | |
class | ContractionOpToOuterProductOpLowering |
Progressive lowering of a vector.contract a, b, c with row-major matmul semantics to a reduction_size-unrolled sequence: ``` at = vector.transpose a, [1, 0] bRow0 = vector.extract b[0] atRow0 = vector.extract at[0] c0 = vector.outerproduct atRow0, bRow0, c ... More... | |
struct | DistributeOps |
struct | UnrollVectorOptions |
Options that control the vector unrolling. More... | |
struct | VectorTransferFullPartialRewriter |
Apply splitFullAndPartialTransfer selectively via a pattern. More... | |
struct | VectorTransformsOptions |
Structure to control the behavior of vector transform patterns. More... | |
struct | WarpExecuteOnLane0LoweringOptions |
Typedefs | |
using | DistributionMapFn = std::function< AffineMap(vector::TransferWriteOp)> |
Functions | |
BroadcastableToResult | isBroadcastableTo (Type srcType, VectorType dstVectorType, std::pair< int, int > *mismatchingDims=nullptr) |
void | populateVectorToVectorCanonicalizationPatterns (RewritePatternSet &patterns) |
Collect a set of vector-to-vector canonicalization patterns. More... | |
void | populateShapeCastFoldingPatterns (RewritePatternSet &patterns) |
Collect a set of vector.shape_cast folding patterns. More... | |
void | populateCastAwayVectorLeadingOneDimPatterns (RewritePatternSet &patterns) |
Collect a set of leading one dimension removal patterns. More... | |
void | populateVectorTransferDropUnitDimsPatterns (RewritePatternSet &patterns) |
Collect a set of one dimension removal patterns. More... | |
void | populateFlattenVectorTransferPatterns (RewritePatternSet &patterns) |
Collect a set of patterns to flatten n-D vector transfers on contiguous memref. More... | |
void | populateBubbleVectorBitCastOpPatterns (RewritePatternSet &patterns) |
Collect a set of patterns that bubble up/down bitcast ops. More... | |
void | populateVectorTransferLoweringPatterns (RewritePatternSet &patterns, llvm::Optional< unsigned > maxTransferRank=llvm::None) |
Collect a set of transfer read/write lowering patterns. More... | |
void | populateVectorMaskMaterializationPatterns (RewritePatternSet &patterns, bool force32BitVectorIndices) |
These patterns materialize masks for various vector ops such as transfers. More... | |
void | populatePropagateVectorDistributionPatterns (RewritePatternSet &patterns) |
Collect a set of patterns to propagate insert_map/extract_map in the ssa chain. More... | |
void | populateVectorBroadcastLoweringPatterns (RewritePatternSet &patterns) |
Collects patterns to progressively lower vector.broadcast ops on high-D vectors to low-D vector ops. More... | |
void | populateVectorMaskOpLoweringPatterns (RewritePatternSet &patterns) |
Collects patterns to progressively lower vector mask ops into elementary selection and insertion ops. More... | |
void | populateVectorShapeCastLoweringPatterns (RewritePatternSet &patterns) |
Collects patterns to progressively lower vector.shape_cast ops on high-D vectors into 1-D/2-D vector ops by generating data movement extract/insert ops. More... | |
IntegerType | getVectorSubscriptType (Builder &builder) |
Returns the integer type required for subscripts in the vector dialect. More... | |
ArrayAttr | getVectorSubscriptAttr (Builder &b, ArrayRef< int64_t > values) |
Returns an integer array attribute containing the given values using the integer type required for subscripts in the vector dialect. More... | |
Value | getVectorReductionOp (arith::AtomicRMWKind op, OpBuilder &builder, Location loc, Value vector) |
Returns the value obtained by reducing the vector into a scalar using the operation kind associated with a binary AtomicRMWKind op. More... | |
bool | isLastMemrefDimUnitStride (MemRefType type) |
Return true if the last dimension of the MemRefType has unit stride. More... | |
AffineMap | getTransferMinorIdentityMap (ShapedType shapedType, VectorType vectorType) |
Build the default minor identity map suitable for a vector transfer. More... | |
bool | checkSameValueRAW (TransferWriteOp defWrite, TransferReadOp read) |
Return true if the transfer_write fully writes the data accessed by the transfer_read. More... | |
bool | checkSameValueWAW (TransferWriteOp write, TransferWriteOp priorWrite) |
Return true if the write op fully over-write the priorWrite transfer_write op. More... | |
bool | isDisjointTransferIndices (VectorTransferOpInterface transferA, VectorTransferOpInterface transferB) |
Same behavior as isDisjointTransferSet but doesn't require the operations to have the same tensor/memref. More... | |
bool | isDisjointTransferSet (VectorTransferOpInterface transferA, VectorTransferOpInterface transferB) |
Return true if we can prove that the transfer operations access disjoint memory. More... | |
Value | makeArithReduction (OpBuilder &b, Location loc, CombiningKind kind, Value v1, Value v2) |
Return the result value of reducing two scalar/vector values with the corresponding arith operation. More... | |
void | registerBufferizableOpInterfaceExternalModels (DialectRegistry ®istry) |
std::unique_ptr< Pass > | createVectorBufferizePass () |
Creates an instance of the vector dialect bufferization pass. More... | |
void | populateWarpExecuteOnLane0OpToScfForPattern (RewritePatternSet &patterns, const WarpExecuteOnLane0LoweringOptions &options) |
void | populateVectorTransposeLoweringPatterns (RewritePatternSet &patterns, VectorTransformsOptions options=VectorTransformsOptions()) |
Insert TransposeLowering patterns into extraction/insertion. More... | |
void | populateVectorMultiReductionLoweringPatterns (RewritePatternSet &patterns, VectorMultiReductionLowering options) |
Collect a set of patterns to convert vector.multi_reduction op into a sequence of vector.reduction ops. More... | |
void | populateVectorContractLoweringPatterns (RewritePatternSet &patterns, VectorTransformsOptions options=VectorTransformsOptions()) |
Collects patterns to progressively lower vector contraction ops on high-D into low-D reduction and product ops. More... | |
void | populateVectorReductionToContractPatterns (RewritePatternSet &patterns) |
Collect patterns to convert reduction op to vector.contract and fold transpose/broadcast ops into the contract. More... | |
void | populateVectorScanLoweringPatterns (RewritePatternSet &patterns) |
Collect patterns to convert scan op. More... | |
void | populateVectorTransferPermutationMapLoweringPatterns (RewritePatternSet &patterns) |
Collect a set of transfer read/write lowering patterns that simplify the permutation map (e.g., converting it to a minor identity map) by inserting broadcasts and transposes. More... | |
void | populateVectorTransferCollapseInnerMostContiguousDimsPatterns (RewritePatternSet &patterns) |
Collect a set of patterns to reduce the rank of the operands of vector transfer ops to operate on the largest contigious vector. More... | |
void | populateVectorInsertExtractStridedSliceDecompositionPatterns (RewritePatternSet &patterns) |
Populate patterns with the following patterns. More... | |
void | populateVectorInsertExtractStridedSliceTransforms (RewritePatternSet &patterns) |
Populate patterns with the following patterns. More... | |
void | populateVectorUnrollPatterns (RewritePatternSet &patterns, const UnrollVectorOptions &options) |
Collect a set of pattern to unroll vector operations to a smaller shapes. More... | |
LogicalResult | splitFullAndPartialTransfer (RewriterBase &b, VectorTransferOpInterface xferOp, VectorTransformsOptions options=VectorTransformsOptions(), scf::IfOp *ifOp=nullptr) |
Split a vector.transfer operation into an in-bounds (i.e., no out-of-bounds masking) fastpath and a slowpath. More... | |
Optional< DistributeOps > | distributPointwiseVectorOp (OpBuilder &builder, Operation *op, ArrayRef< Value > id, ArrayRef< int64_t > multiplicity, const AffineMap &map) |
Distribute a N-D vector pointwise operation over a range of given ids taking all values in [0 . More... | |
void | transferOpflowOpt (Operation *rootOp) |
Implements transfer op write to read forwarding and dead transfer write optimizations. More... | |
Value | createOrFoldDimOp (OpBuilder &b, Location loc, Value source, int64_t dim) |
Helper function that creates a memref::DimOp or tensor::DimOp depending on the type of source . More... | |
using mlir::vector::DistributionMapFn = typedef std::function<AffineMap(vector::TransferWriteOp)> |
Definition at line 42 of file VectorDistribution.h.
|
strong |
Return whether srcType
can be broadcast to dstVectorType
under the semantics of the vector.broadcast
op.
Enumerator | |
---|---|
Success | |
SourceRankHigher | |
DimensionMismatch | |
SourceTypeNotAVector |
Definition at line 51 of file VectorOps.h.
|
strong |
Enum to control the lowering of vector.contract
operations.
Enumerator | |
---|---|
Dot | Progressively lower to finer grained |
Matmul | Lower to |
OuterProduct | Lower to |
ParallelArith | Lower contract with all reduction dimensions unrolled to 1 to a vector elementwise operations. |
Definition at line 45 of file VectorRewritePatterns.h.
|
strong |
Enum to control the lowering of vector.multi_reduction
operations.
Enumerator | |
---|---|
InnerParallel | Lower multi_reduction into outer-reduction and inner-parallel ops. |
InnerReduction | Lower multi_reduction into outer-parallel and inner-reduction ops. |
Definition at line 38 of file VectorRewritePatterns.h.
|
strong |
Enum to control the splitting of vector.transfer
operations into in-bounds and out-of-bounds variants.
Definition at line 58 of file VectorRewritePatterns.h.
|
strong |
Enum to control the lowering of vector.transpose
operations.
Enumerator | |
---|---|
EltWise | Lower transpose into element-wise extract and inserts. |
Flat | Lower 2-D transpose to |
Shuffle | Lower 2-D transpose to |
Definition at line 28 of file VectorRewritePatterns.h.
bool mlir::vector::checkSameValueRAW | ( | TransferWriteOp | defWrite, |
TransferReadOp | read | ||
) |
Return true if the transfer_write fully writes the data accessed by the transfer_read.
Referenced by findAncestorOpInRegion(), foldTransferInBoundsAttribute(), and getTransferMinorIdentityMap().
bool mlir::vector::checkSameValueWAW | ( | TransferWriteOp | write, |
TransferWriteOp | priorWrite | ||
) |
Return true if the write op fully over-write the priorWrite transfer_write op.
Referenced by findAncestorOpInRegion(), foldTransferInBoundsAttribute(), and getTransferMinorIdentityMap().
Helper function that creates a memref::DimOp or tensor::DimOp depending on the type of source
.
Definition at line 37 of file VectorUtils.cpp.
References mlir::OpBuilder::createOrFold(), mlir::Value::getType(), and mlir::Type::isa().
Referenced by createFoldedDimOp(), createInBoundsCond(), createSubViewIntersection(), genBuffers(), getResultsPositionInLoopsToShapeMap(), and MaterializeTransferMask< ConcreteOp >::matchAndRewrite().
std::unique_ptr< Pass > mlir::vector::createVectorBufferizePass | ( | ) |
Creates an instance of the vector
dialect bufferization pass.
Definition at line 44 of file Bufferize.cpp.
Optional< mlir::vector::DistributeOps > mlir::vector::distributPointwiseVectorOp | ( | OpBuilder & | builder, |
Operation * | op, | ||
ArrayRef< Value > | id, | ||
ArrayRef< int64_t > | multiplicity, | ||
const AffineMap & | map | ||
) |
Distribute a N-D vector pointwise operation over a range of given ids taking all values in [0 .
. multiplicity - 1] (e.g. loop induction variable or SPMD id). This transformation only inserts vector.extract_map/vector.insert_map. It is meant to be used with canonicalizations pattern to propagate and fold the vector insert_map/extract_map operations. Transforms: to: v = arith.addf a, b : vector<32xf32> ev = vector.extract_map v, id, 32 : vector<32xf32> into vector<1xf32> nv = vector.insert_map ev, id, 32 : vector<1xf32> into vector<32xf32>
Definition at line 1997 of file VectorTransforms.cpp.
References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::vector::DistributeOps::extract, mlir::Operation::getLoc(), mlir::AffineMap::getNumResults(), mlir::Operation::getNumResults(), mlir::Operation::getResult(), mlir::AffineMap::getResults(), mlir::Value::getType(), mlir::vector::DistributeOps::insert, and mlir::OpBuilder::setInsertionPointAfter().
AffineMap mlir::vector::getTransferMinorIdentityMap | ( | ShapedType | shapedType, |
VectorType | vectorType | ||
) |
Build the default minor identity map suitable for a vector transfer.
This also handles the case memref<... x vector<...>> -> vector<...> in which the rank of the identity map must take the vector element type into account.
Definition at line 122 of file VectorOps.cpp.
References checkSameValueRAW(), checkSameValueWAW(), mlir::AffineMap::get(), mlir::getAffineConstantExpr(), and mlir::AffineMap::getMinorIdentityMap().
Referenced by foldExtractStridedOpFromInsertChain(), foldTransferInBoundsAttribute(), and printTransferAttrs().
Value mlir::vector::getVectorReductionOp | ( | arith::AtomicRMWKind | op, |
OpBuilder & | builder, | ||
Location | loc, | ||
Value | vector | ||
) |
Returns the value obtained by reducing the vector into a scalar using the operation kind associated with a binary AtomicRMWKind op.
Definition at line 442 of file VectorOps.cpp.
References mlir::RewritePatternSet::add(), mlir::OpBuilder::create(), mlir::emitOptionalError(), mlir::failure(), mlir::Builder::getI64ArrayAttr(), mlir::Value::getLoc(), getVectorType(), makeArithReduction(), mlir::OpRewritePattern< SourceOp >::OpRewritePattern(), mlir::RewriterBase::replaceOp(), and mlir::success().
Returns an integer array attribute containing the given values using the integer type required for subscripts in the vector dialect.
Definition at line 326 of file VectorOps.cpp.
References mlir::Builder::getI64ArrayAttr().
Referenced by foldExtractStridedOpFromInsertChain(), inferStridedSliceOpResultType(), isBroadcastableTo(), and CanonicalizeContractAdd< AddOpType >::matchAndRewrite().
IntegerType mlir::vector::getVectorSubscriptType | ( | Builder & | builder | ) |
Returns the integer type required for subscripts in the vector dialect.
Definition at line 322 of file VectorOps.cpp.
References mlir::Builder::getIntegerType().
BroadcastableToResult mlir::vector::isBroadcastableTo | ( | Type | srcType, |
VectorType | dstVectorType, | ||
std::pair< int, int > * | mismatchingDims = nullptr |
||
) |
Definition at line 1685 of file VectorOps.cpp.
References mlir::RewritePatternSet::add(), DimensionMismatch, mlir::Attribute::dyn_cast(), mlir::Type::dyn_cast(), mlir::detail::enumerate(), mlir::failure(), mlir::DenseElementsAttr::get(), mlir::getElementTypeOrSelf(), getVectorSubscriptAttr(), getVectorType(), mlir::Type::isIntOrIndexOrFloat(), mlir::RewriterBase::replaceOpWithNewOp(), SourceRankHigher, SourceTypeNotAVector, Success, mlir::success(), vectorType(), and mlir::verify().
Referenced by broadcastIfNeeded().
bool mlir::vector::isDisjointTransferIndices | ( | VectorTransferOpInterface | transferA, |
VectorTransferOpInterface | transferB | ||
) |
Same behavior as isDisjointTransferSet
but doesn't require the operations to have the same tensor/memref.
This allows comparing operations accessing different tensors.
Definition at line 157 of file VectorOps.cpp.
References mlir::presburger::abs().
Referenced by foldTransferInBoundsAttribute(), isDisjointTransferSet(), and tensorChunkAccessedByUnknownOp().
bool mlir::vector::isDisjointTransferSet | ( | VectorTransferOpInterface | transferA, |
VectorTransferOpInterface | transferB | ||
) |
Return true if we can prove that the transfer operations access disjoint memory.
Definition at line 189 of file VectorOps.cpp.
References isDisjointTransferIndices().
Referenced by findAncestorOpInRegion(), and mlir::linalg::hoistRedundantVectorTransfers().
bool mlir::vector::isLastMemrefDimUnitStride | ( | MemRefType | type | ) |
Return true if the last dimension of the MemRefType has unit stride.
Also return true for memrefs with no strides.
Definition at line 115 of file VectorOps.cpp.
References mlir::getStridesAndOffset(), and mlir::succeeded().
Referenced by foldTransferInBoundsAttribute(), TransferReadToVectorLoadLowering::matchAndRewrite(), and TransferWriteToVectorStoreLowering::matchAndRewrite().
Value mlir::vector::makeArithReduction | ( | OpBuilder & | b, |
Location | loc, | ||
CombiningKind | kind, | ||
Value | v1, | ||
Value | v2 | ||
) |
Return the result value of reducing two scalar/vector values with the corresponding arith operation.
Referenced by getVectorReductionOp(), and TwoDimMultiReductionToElementWise::matchAndRewrite().
void mlir::vector::populateBubbleVectorBitCastOpPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of patterns that bubble up/down bitcast ops.
These patterns move vector.bitcast ops to be before insert ops or after extract ops where suitable. With them, bitcast will happen on smaller vectors and there are more chances to share extract/insert ops.
Definition at line 2897 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateCastAwayVectorLeadingOneDimPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of leading one dimension removal patterns.
These patterns insert vector.shape_cast to remove leading one dimensions to expose more canonical forms of read/write/insert/extract operations. With them, there are more chances that we can cancel out extract-insert pairs or forward write-read pairs.
Definition at line 438 of file VectorDropLeadUnitDim.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().
void mlir::vector::populateFlattenVectorTransferPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of patterns to flatten n-D vector transfers on contiguous memref.
These patterns insert memref.collapse_shape + vector.shape_cast patterns to transform multiple small n-D transfers into a larger 1-D transfer where the memref contiguity properties allow it.
Definition at line 553 of file VectorTransferOpTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().
void mlir::vector::populatePropagateVectorDistributionPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of patterns to propagate insert_map/extract_map in the ssa chain.
Definition at line 831 of file VectorUnrollDistribute.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateShapeCastFoldingPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of vector.shape_cast folding patterns.
Definition at line 2892 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by populateCastAwayVectorLeadingOneDimPatterns(), populateFlattenVectorTransferPatterns(), and populateVectorTransferDropUnitDimsPatterns().
void mlir::vector::populateVectorBroadcastLoweringPatterns | ( | RewritePatternSet & | patterns | ) |
Collects patterns to progressively lower vector.broadcast ops on high-D vectors to low-D vector ops.
Definition at line 2904 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateVectorContractLoweringPatterns | ( | RewritePatternSet & | patterns, |
VectorTransformsOptions | options = VectorTransformsOptions() |
||
) |
Collects patterns to progressively lower vector contraction ops on high-D into low-D reduction and product ops.
Definition at line 2922 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorInsertExtractStridedSliceDecompositionPatterns | ( | RewritePatternSet & | patterns | ) |
Populate patterns
with the following patterns.
RewritePattern for InsertStridedSliceOp where source and destination vectors have different ranks.
When ranks are different, InsertStridedSlice needs to extract a properly ranked vector from the destination vector into which to insert. This pattern only takes care of this extraction part and forwards the rest to [VectorInsertStridedSliceOpSameRankRewritePattern].
For a k-D source and n-D destination vector (k < n), we emit:
For such cases, we can rewrite it to ExtractOp/ExtractElementOp + lower rank ExtractStridedSliceOp + InsertOp/InsertElementOp for the n-D case.
Definition at line 288 of file VectorInsertExtractStridedSliceRewritePatterns.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by populateVectorInsertExtractStridedSliceTransforms(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorInsertExtractStridedSliceTransforms | ( | RewritePatternSet & | patterns | ) |
Populate patterns
with the following patterns.
Populate the given list with patterns that convert from Vector to LLVM.
Patterns in populateVectorInsertExtractStridedSliceDecompositionPatterns();
RewritePattern for InsertStridedSliceOp where source and destination vectors have the same rank. For each outermost index in the slice: begin end stride [offset : offset+size*stride : stride]
For such cases, we can lower it to a ShuffleOp.
Definition at line 295 of file VectorInsertExtractStridedSliceRewritePatterns.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateVectorInsertExtractStridedSliceDecompositionPatterns().
Referenced by mlir::populateVectorToLLVMConversionPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorMaskMaterializationPatterns | ( | RewritePatternSet & | patterns, |
bool | force32BitVectorIndices | ||
) |
These patterns materialize masks for various vector ops such as transfers.
Definition at line 2884 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateVectorMaskOpLoweringPatterns | ( | RewritePatternSet & | patterns | ) |
Collects patterns to progressively lower vector mask ops into elementary selection and insertion ops.
Definition at line 2909 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateVectorMultiReductionLoweringPatterns | ( | RewritePatternSet & | patterns, |
VectorMultiReductionLowering | options | ||
) |
Collect a set of patterns to convert vector.multi_reduction op into a sequence of vector.reduction ops.
The patterns comprise:
Definition at line 340 of file VectorMultiDimReductionTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorReductionToContractPatterns | ( | RewritePatternSet & | patterns | ) |
Collect patterns to convert reduction op to vector.contract and fold transpose/broadcast ops into the contract.
Definition at line 2936 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateVectorTransferCollapseInnerMostContiguousDimsPatterns().
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorScanLoweringPatterns | ( | RewritePatternSet & | patterns | ) |
Collect patterns to convert scan op.
Definition at line 2959 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorShapeCastLoweringPatterns | ( | RewritePatternSet & | patterns | ) |
Collects patterns to progressively lower vector.shape_cast ops on high-D vectors into 1-D/2-D vector ops by generating data movement extract/insert ops.
Definition at line 2915 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::vector::populateVectorToVectorCanonicalizationPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of vector-to-vector canonicalization patterns.
void mlir::vector::populateVectorTransferCollapseInnerMostContiguousDimsPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of patterns to reduce the rank of the operands of vector transfer ops to operate on the largest contigious vector.
These patterns are useful when lowering to dialects with 1d vector type such as llvm and it will result fewer memory reads.
Definition at line 2944 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by populateVectorReductionToContractPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorTransferDropUnitDimsPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of one dimension removal patterns.
These patterns insert rank-reducing memref.subview ops to remove one dimensions. With them, there are more chances that we can avoid potentially exensive vector.shape_cast operations.
Definition at line 545 of file VectorTransferOpTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().
void mlir::vector::populateVectorTransferLoweringPatterns | ( | RewritePatternSet & | patterns, |
llvm::Optional< unsigned > | maxTransferRank = llvm::None |
||
) |
Collect a set of transfer read/write lowering patterns.
These patterns lower transfer ops to simpler ops like vector.load
, vector.store
and vector.broadcast
. Only transfers with a transfer rank of a most maxTransferRank
are lowered. This is useful when combined with VectorToSCF, which reduces the rank of vector transfer ops.
Definition at line 2949 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by mlir::populateVectorToLLVMConversionPatterns().
void mlir::vector::populateVectorTransferPermutationMapLoweringPatterns | ( | RewritePatternSet & | patterns | ) |
Collect a set of transfer read/write lowering patterns that simplify the permutation map (e.g., converting it to a minor identity map) by inserting broadcasts and transposes.
More specifically:
[TransferReadPermutationLowering] Lower transfer_read op with permutation into a transfer_read with a permutation map composed of leading zeros followed by a minor identity + vector.transpose op. Ex: vector.transfer_read ... permutation_map: (d0, d1, d2) -> (0, d1) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2) -> (d1, 0) vector.transpose v, [1, 0]
vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, 0, 0, d1, d3) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, 0, d1, 0, d3) vector.transpose v, [0, 1, 3, 2, 4] Note that an alternative is to transform it to linalg.transpose + vector.transfer_read to do the transpose in memory instead.
[TransferWritePermutationLowering] Lower transfer_write op with permutation into a transfer_write with a minor identity permutation map. (transfer_write ops cannot have broadcasts.) Ex: vector.transfer_write v ... permutation_map: (d0, d1, d2) -> (d2, d0, d1) into: tmp = vector.transpose v, [2, 0, 1] vector.transfer_write tmp ... permutation_map: (d0, d1, d2) -> (d0, d1, d2)
vector.transfer_write v ... permutation_map: (d0, d1, d2, d3) -> (d3, d2) into: tmp = vector.transpose v, [1, 0] v = vector.transfer_write tmp ... permutation_map: (d0, d1, d2, d3) -> (d2, d3)
[TransferOpReduceRank] Lower transfer_read op with broadcast in the leading dimensions into transfer_read of lower rank + vector.broadcast. Ex: vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, d1, 0, d3) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (d1, 0, d3) vector.broadcast v
Definition at line 273 of file VectorTransferPermutationMapRewritePatterns.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by mlir::populateVectorToSCFConversionPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorTransposeLoweringPatterns | ( | RewritePatternSet & | patterns, |
VectorTransformsOptions | options = VectorTransformsOptions() |
||
) |
Insert TransposeLowering patterns into extraction/insertion.
Definition at line 2930 of file VectorTransforms.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateVectorUnrollPatterns | ( | RewritePatternSet & | patterns, |
const UnrollVectorOptions & | options | ||
) |
Collect a set of pattern to unroll vector operations to a smaller shapes.
options
structure controls which operations are unrolled and the target shape. op
is unrolled to the targetShape
as follows, for each of its operands:
unrolledVectorType
and number of unrolled instances numUnrolledInstances
are computed from the targetShape
. For now it is assumed the unrolling factors divide the vector sizes.numUnrolledInstances
times, once for each result.Example:
opA(operand0, operand1) // numUnrolledInstances = 3
operand0 operand1 | | fork fork <----------gather all fork ops ---------> /|\ /|\ f00 f01 f02 f10 f11 f12 <---------- clone op 3 times ---------> opA0(f00, f10), opA1(f01, f11), opA2(f02, f12) \ | /
<-----------------— join ----------------------—>
Other local patterns then kick in iteratively (including DCE) and compose to combine the ExtractStridedSlice/InsertStridedSlice.
Definition at line 823 of file VectorUnrollDistribute.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.
Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().
void mlir::vector::populateWarpExecuteOnLane0OpToScfForPattern | ( | RewritePatternSet & | patterns, |
const WarpExecuteOnLane0LoweringOptions & | options | ||
) |
Definition at line 931 of file VectorDistribute.cpp.
References mlir::RewritePatternSet::add(), canBeHoisted(), mlir::RewritePatternSet::getContext(), mlir::Operation::getResults(), mlir::ResultRange::getType(), mlir::Operation::moveBefore(), options, value, and mlir::Block::without_terminator().
void mlir::vector::registerBufferizableOpInterfaceExternalModels | ( | DialectRegistry & | registry | ) |
Definition at line 160 of file BufferizableOpInterfaceImpl.cpp.
References mlir::DialectRegistry::addExtension().
Referenced by mlir::registerAllDialects().
LogicalResult mlir::vector::splitFullAndPartialTransfer | ( | RewriterBase & | b, |
VectorTransferOpInterface | xferOp, | ||
VectorTransformsOptions | options = VectorTransformsOptions() , |
||
scf::IfOp * | ifOp = nullptr |
||
) |
Split a vector.transfer operation into an in-bounds (i.e., no out-of-bounds masking) fastpath and a slowpath.
If ifOp
is not null and the result is success, the
ifOp` points to the newly created conditional upon function return. To accomodate for the fact that the original vector.transfer indexing may be arbitrary and the slow path indexes @[0...0] in the temporary buffer, the scf.if op returns a view and values of type index. At this time, only vector.transfer_read case is implemented.
Example (a 2-D vector.transfer_read): ``` %1 = vector.transfer_read %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { // fastpath, direct cast memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { // slowpath, not in-bounds vector.transfer or linalg.copy. memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index */ // } /** %0 = vector.transfer_read %1#0[%1#1, %1#2] {in_bounds = [true ... true]} `` where
alloc` is a top of the function alloca'ed buffer of one vector.
Preconditions:
xferOp.permutation_map()
must be a minor identity mapxferOp.memref()
and the rank of the xferOp.vector()
must be equal. This will be relaxed in the future but requires rank-reducing subviews.For vector.transfer_read: If ifOp
is not null and the result is success, the
ifOp` points to the newly created conditional upon function return. To accomodate for the fact that the original vector.transfer indexing may be arbitrary and the slow path indexes @[0...0] in the temporary buffer, the scf.if op returns a view and values of type index.
Example (a 2-D vector.transfer_read): ``` %1 = vector.transfer_read %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { // fastpath, direct cast memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { // slowpath, not in-bounds vector.transfer or linalg.copy. memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index */ // } /** %0 = vector.transfer_read %1#0[%1#1, %1#2] {in_bounds = [true ... true]} `` where
alloc` is a top of the function alloca'ed buffer of one vector.
For vector.transfer_write: There are 2 conditional blocks. First a block to decide which memref and indices to use for an unmasked, inbounds write. Then a conditional block to further copy a partial buffer into the final result in the slow path case.
Example (a 2-D vector.transfer_write): ``` vector.transfer_write arg, %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index } %0 = vector.transfer_write arg, %1#0[%1#1, %1#2] {in_bounds = [true ... true]} scf.if (notInBounds) { // slowpath: not in-bounds vector.transfer or linalg.copy. } `` where
alloc` is a top of the function alloca'ed buffer of one vector.
Preconditions:
xferOp.permutation_map()
must be a minor identity mapxferOp.source()
and the rank of the xferOp.vector()
must be equal. This will be relaxed in the future but requires rank-reducing subviews. Definition at line 519 of file VectorTransferSplitRewritePatterns.cpp.
References mlir::Type::cast(), mlir::OpBuilder::clone(), mlir::OpBuilder::create(), createFullPartialLinalgCopy(), createFullPartialVectorTransferRead(), createFullPartialVectorTransferWrite(), createInBoundsCond(), mlir::failure(), ForceInBounds, mlir::Region::front(), getAutomaticAllocationScope(), mlir::Builder::getBoolArrayAttr(), getCastCompatibleMemRefType(), mlir::Builder::getI64IntegerAttr(), mlir::Builder::getIndexType(), mlir::Operation::getLoc(), getLocationToWriteFullVec(), mlir::Operation::getNumRegions(), mlir::Operation::getRegion(), mlir::Value::getType(), mlir::BlockAndValueMapping::map(), None, mlir::Operation::setAttr(), mlir::OpBuilder::setInsertionPoint(), mlir::OpBuilder::setInsertionPointToStart(), splitFullAndPartialTransferPrecondition(), mlir::succeeded(), mlir::success(), VectorTransfer, and mlir::vector::VectorTransformsOptions::vectorTransferSplit.
Referenced by mlir::vector::VectorTransferFullPartialRewriter::matchAndRewrite().
Implements transfer op write to read forwarding and dead transfer write optimizations.
Definition at line 529 of file VectorTransferOpTransforms.cpp.
References mlir::Operation::walk().