Namespaces
	detail

Classes
class	CombiningKindAttr
	An attribute that specifies the combining function for `vector.contract`, and `vector.reduction`. More...

class	ContractionOpLowering
	Progressive lowering of ContractionOp. More...

class	ContractionOpToDotLowering
	Progressive lowering of a `vector.contract a, b, c` with row-major matmul semantics to an output-size-unrolled sequence: ``` out = arith.constant ... More...

class	ContractionOpToMatmulOpLowering
	Progressive lowering of a `vector.contract a, b, c` with row-major matmul semantics to: %flattened_a = vector.shape_cast %a %flattened_b = vector.shape_cast %b %flattened_d = vector.matmul %flattened_a, %flattened_b %d = vector.shape_cast %%flattened_d %e = add %c, %d `vector.matmul` later lowers to `llvm.matrix.multiply`. More...

class	ContractionOpToOuterProductOpLowering
	Progressive lowering of a `vector.contract a, b, c` with row-major matmul semantics to a reduction_size-unrolled sequence: ``` at = vector.transpose a, [1, 0] bRow0 = vector.extract b[0] atRow0 = vector.extract at[0] c0 = vector.outerproduct atRow0, bRow0, c ... More...

struct	DistributeOps

struct	UnrollVectorOptions
	Options that control the vector unrolling. More...

struct	VectorTransferFullPartialRewriter
	Apply `splitFullAndPartialTransfer` selectively via a pattern. More...

struct	VectorTransformsOptions
	Structure to control the behavior of vector transform patterns. More...

struct	WarpExecuteOnLane0LoweringOptions

Typedefs
using	DistributionMapFn = std::function< AffineMap(vector::TransferWriteOp)>

Enumerations
enum	BroadcastableToResult { BroadcastableToResult::Success = 0, BroadcastableToResult::SourceRankHigher = 1, BroadcastableToResult::DimensionMismatch = 2, BroadcastableToResult::SourceTypeNotAVector = 3 }
	Return whether `srcType` can be broadcast to `dstVectorType` under the semantics of the `vector.broadcast` op. More...

enum	VectorTransposeLowering { VectorTransposeLowering::EltWise = 0, VectorTransposeLowering::Flat = 1, VectorTransposeLowering::Shuffle = 2 }
	Enum to control the lowering of `vector.transpose` operations. More...

enum	VectorMultiReductionLowering { VectorMultiReductionLowering::InnerParallel = 0, VectorMultiReductionLowering::InnerReduction = 1 }
	Enum to control the lowering of `vector.multi_reduction` operations. More...

enum	VectorContractLowering { VectorContractLowering::Dot = 0, VectorContractLowering::Matmul = 1, VectorContractLowering::OuterProduct = 2, VectorContractLowering::ParallelArith = 3 }
	Enum to control the lowering of `vector.contract` operations. More...

enum	VectorTransferSplit { VectorTransferSplit::None = 0, VectorTransferSplit::VectorTransfer = 1, VectorTransferSplit::LinalgCopy = 2, VectorTransferSplit::ForceInBounds = 3 }
	Enum to control the splitting of `vector.transfer` operations into in-bounds and out-of-bounds variants. More...

Functions
BroadcastableToResult	isBroadcastableTo (Type srcType, VectorType dstVectorType, std::pair< int, int > *mismatchingDims=nullptr)

void	populateVectorToVectorCanonicalizationPatterns (RewritePatternSet &patterns)
	Collect a set of vector-to-vector canonicalization patterns. More...

void	populateShapeCastFoldingPatterns (RewritePatternSet &patterns)
	Collect a set of vector.shape_cast folding patterns. More...

void	populateCastAwayVectorLeadingOneDimPatterns (RewritePatternSet &patterns)
	Collect a set of leading one dimension removal patterns. More...

void	populateVectorTransferDropUnitDimsPatterns (RewritePatternSet &patterns)
	Collect a set of one dimension removal patterns. More...

void	populateFlattenVectorTransferPatterns (RewritePatternSet &patterns)
	Collect a set of patterns to flatten n-D vector transfers on contiguous memref. More...

void	populateBubbleVectorBitCastOpPatterns (RewritePatternSet &patterns)
	Collect a set of patterns that bubble up/down bitcast ops. More...

void	populateVectorTransferLoweringPatterns (RewritePatternSet &patterns, llvm::Optional< unsigned > maxTransferRank=llvm::None)
	Collect a set of transfer read/write lowering patterns. More...

void	populateVectorMaskMaterializationPatterns (RewritePatternSet &patterns, bool force32BitVectorIndices)
	These patterns materialize masks for various vector ops such as transfers. More...

void	populatePropagateVectorDistributionPatterns (RewritePatternSet &patterns)
	Collect a set of patterns to propagate insert_map/extract_map in the ssa chain. More...

void	populateVectorBroadcastLoweringPatterns (RewritePatternSet &patterns)
	Collects patterns to progressively lower vector.broadcast ops on high-D vectors to low-D vector ops. More...

void	populateVectorMaskOpLoweringPatterns (RewritePatternSet &patterns)
	Collects patterns to progressively lower vector mask ops into elementary selection and insertion ops. More...

void	populateVectorShapeCastLoweringPatterns (RewritePatternSet &patterns)
	Collects patterns to progressively lower vector.shape_cast ops on high-D vectors into 1-D/2-D vector ops by generating data movement extract/insert ops. More...

IntegerType	getVectorSubscriptType (Builder &builder)
	Returns the integer type required for subscripts in the vector dialect. More...

ArrayAttr	getVectorSubscriptAttr (Builder &b, ArrayRef< int64_t > values)
	Returns an integer array attribute containing the given values using the integer type required for subscripts in the vector dialect. More...

Value	getVectorReductionOp (arith::AtomicRMWKind op, OpBuilder &builder, Location loc, Value vector)
	Returns the value obtained by reducing the vector into a scalar using the operation kind associated with a binary AtomicRMWKind op. More...

bool	isLastMemrefDimUnitStride (MemRefType type)
	Return true if the last dimension of the MemRefType has unit stride. More...

AffineMap	getTransferMinorIdentityMap (ShapedType shapedType, VectorType vectorType)
	Build the default minor identity map suitable for a vector transfer. More...

bool	checkSameValueRAW (TransferWriteOp defWrite, TransferReadOp read)
	Return true if the transfer_write fully writes the data accessed by the transfer_read. More...

bool	checkSameValueWAW (TransferWriteOp write, TransferWriteOp priorWrite)
	Return true if the write op fully over-write the priorWrite transfer_write op. More...

bool	isDisjointTransferIndices (VectorTransferOpInterface transferA, VectorTransferOpInterface transferB)
	Same behavior as `isDisjointTransferSet` but doesn't require the operations to have the same tensor/memref. More...

bool	isDisjointTransferSet (VectorTransferOpInterface transferA, VectorTransferOpInterface transferB)
	Return true if we can prove that the transfer operations access disjoint memory. More...

Value	makeArithReduction (OpBuilder &b, Location loc, CombiningKind kind, Value v1, Value v2)
	Return the result value of reducing two scalar/vector values with the corresponding arith operation. More...

void	registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)

std::unique_ptr< Pass >	createVectorBufferizePass ()
	Creates an instance of the `vector` dialect bufferization pass. More...

void	populateWarpExecuteOnLane0OpToScfForPattern (RewritePatternSet &patterns, const WarpExecuteOnLane0LoweringOptions &options)

void	populateVectorTransposeLoweringPatterns (RewritePatternSet &patterns, VectorTransformsOptions options=VectorTransformsOptions())
	Insert TransposeLowering patterns into extraction/insertion. More...

void	populateVectorMultiReductionLoweringPatterns (RewritePatternSet &patterns, VectorMultiReductionLowering options)
	Collect a set of patterns to convert vector.multi_reduction op into a sequence of vector.reduction ops. More...

void	populateVectorContractLoweringPatterns (RewritePatternSet &patterns, VectorTransformsOptions options=VectorTransformsOptions())
	Collects patterns to progressively lower vector contraction ops on high-D into low-D reduction and product ops. More...

void	populateVectorReductionToContractPatterns (RewritePatternSet &patterns)
	Collect patterns to convert reduction op to vector.contract and fold transpose/broadcast ops into the contract. More...

void	populateVectorScanLoweringPatterns (RewritePatternSet &patterns)
	Collect patterns to convert scan op. More...

void	populateVectorTransferPermutationMapLoweringPatterns (RewritePatternSet &patterns)
	Collect a set of transfer read/write lowering patterns that simplify the permutation map (e.g., converting it to a minor identity map) by inserting broadcasts and transposes. More...

void	populateVectorTransferCollapseInnerMostContiguousDimsPatterns (RewritePatternSet &patterns)
	Collect a set of patterns to reduce the rank of the operands of vector transfer ops to operate on the largest contigious vector. More...

void	populateVectorInsertExtractStridedSliceDecompositionPatterns (RewritePatternSet &patterns)
	Populate `patterns` with the following patterns. More...

void	populateVectorInsertExtractStridedSliceTransforms (RewritePatternSet &patterns)
	Populate `patterns` with the following patterns. More...

void	populateVectorUnrollPatterns (RewritePatternSet &patterns, const UnrollVectorOptions &options)
	Collect a set of pattern to unroll vector operations to a smaller shapes. More...

LogicalResult	splitFullAndPartialTransfer (RewriterBase &b, VectorTransferOpInterface xferOp, VectorTransformsOptions options=VectorTransformsOptions(), scf::IfOp *ifOp=nullptr)
	Split a vector.transfer operation into an in-bounds (i.e., no out-of-bounds masking) fastpath and a slowpath. More...

Optional< DistributeOps >	distributPointwiseVectorOp (OpBuilder &builder, Operation *op, ArrayRef< Value > id, ArrayRef< int64_t > multiplicity, const AffineMap &map)
	Distribute a N-D vector pointwise operation over a range of given ids taking all values in [0 . More...

void	transferOpflowOpt (Operation *rootOp)
	Implements transfer op write to read forwarding and dead transfer write optimizations. More...

Value	createOrFoldDimOp (OpBuilder &b, Location loc, Value source, int64_t dim)
	Helper function that creates a memref::DimOp or tensor::DimOp depending on the type of `source`. More...

Typedef Documentation

◆ DistributionMapFn

using mlir::vector::DistributionMapFn = typedef std::function<AffineMap(vector::TransferWriteOp)>

Definition at line 42 of file VectorDistribution.h.

Enumeration Type Documentation

◆ BroadcastableToResult

enum mlir::vector::BroadcastableToResult

strong

Return whether srcType can be broadcast to dstVectorType under the semantics of the vector.broadcast op.

Enumerator
Success
SourceRankHigher
DimensionMismatch
SourceTypeNotAVector

Definition at line 51 of file VectorOps.h.

◆ VectorContractLowering

enum mlir::vector::VectorContractLowering

strong

Enum to control the lowering of vector.contract operations.

Enumerator
Dot	Progressively lower to finer grained `vector.contract` and dot-products.
Matmul	Lower to `vector.matrix_multiply`, maps 1-1 to LLVM matrix intrinsics.
OuterProduct	Lower to `vector.outerproduct`.
ParallelArith	Lower contract with all reduction dimensions unrolled to 1 to a vector elementwise operations.

Definition at line 45 of file VectorRewritePatterns.h.

◆ VectorMultiReductionLowering

enum mlir::vector::VectorMultiReductionLowering

strong

Enum to control the lowering of vector.multi_reduction operations.

Enumerator
InnerParallel	Lower multi_reduction into outer-reduction and inner-parallel ops.
InnerReduction	Lower multi_reduction into outer-parallel and inner-reduction ops.

Definition at line 38 of file VectorRewritePatterns.h.

◆ VectorTransferSplit

enum mlir::vector::VectorTransferSplit

strong

Enum to control the splitting of vector.transfer operations into in-bounds and out-of-bounds variants.

Enumerator
None	Do not split vector transfer operations.
VectorTransfer	Split using in-bounds + out-of-bounds vector.transfer operations.
LinalgCopy	Split using an in-bounds vector.transfer + linalg.fill + linalg.copy operations.
ForceInBounds	Do not split vector transfer operation but instead mark it as "in-bounds".

Definition at line 58 of file VectorRewritePatterns.h.

◆ VectorTransposeLowering

enum mlir::vector::VectorTransposeLowering

strong

Enum to control the lowering of vector.transpose operations.

Enumerator
EltWise	Lower transpose into element-wise extract and inserts.
Flat	Lower 2-D transpose to `vector.flat_transpose`, maps 1-1 to LLVM matrix intrinsics.
Shuffle	Lower 2-D transpose to `vector.shuffle`.

Definition at line 28 of file VectorRewritePatterns.h.

Function Documentation

◆ checkSameValueRAW()

bool mlir::vector::checkSameValueRAW	(	TransferWriteOp	defWrite,
		TransferReadOp	read
	)

Return true if the transfer_write fully writes the data accessed by the transfer_read.

Referenced by findAncestorOpInRegion(), foldTransferInBoundsAttribute(), and getTransferMinorIdentityMap().

◆ checkSameValueWAW()

bool mlir::vector::checkSameValueWAW	(	TransferWriteOp	write,
		TransferWriteOp	priorWrite
	)

Return true if the write op fully over-write the priorWrite transfer_write op.

Referenced by findAncestorOpInRegion(), foldTransferInBoundsAttribute(), and getTransferMinorIdentityMap().

◆ createOrFoldDimOp()

Value mlir::vector::createOrFoldDimOp	(	OpBuilder &	b,
		Location	loc,
		Value	source,
		int64_t	dim
	)

Helper function that creates a memref::DimOp or tensor::DimOp depending on the type of source.

Definition at line 37 of file VectorUtils.cpp.

References mlir::OpBuilder::createOrFold(), mlir::Value::getType(), and mlir::Type::isa().

Referenced by createFoldedDimOp(), createInBoundsCond(), createSubViewIntersection(), genBuffers(), getResultsPositionInLoopsToShapeMap(), and MaterializeTransferMask< ConcreteOp >::matchAndRewrite().

◆ createVectorBufferizePass()

std::unique_ptr< Pass > mlir::vector::createVectorBufferizePass ( )

Creates an instance of the vector dialect bufferization pass.

Definition at line 44 of file Bufferize.cpp.

◆ distributPointwiseVectorOp()

Optional< mlir::vector::DistributeOps > mlir::vector::distributPointwiseVectorOp	(	OpBuilder &	builder,
		Operation *	op,
		ArrayRef< Value >	id,
		ArrayRef< int64_t >	multiplicity,
		const AffineMap &	map
	)

Distribute a N-D vector pointwise operation over a range of given ids taking all values in [0 .

. multiplicity - 1] (e.g. loop induction variable or SPMD id). This transformation only inserts vector.extract_map/vector.insert_map. It is meant to be used with canonicalizations pattern to propagate and fold the vector insert_map/extract_map operations. Transforms: to: v = arith.addf a, b : vector<32xf32> ev = vector.extract_map v, id, 32 : vector<32xf32> into vector<1xf32> nv = vector.insert_map ev, id, 32 : vector<1xf32> into vector<32xf32>

Definition at line 1997 of file VectorTransforms.cpp.

References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), mlir::vector::DistributeOps::extract, mlir::Operation::getLoc(), mlir::AffineMap::getNumResults(), mlir::Operation::getNumResults(), mlir::Operation::getResult(), mlir::AffineMap::getResults(), mlir::Value::getType(), mlir::vector::DistributeOps::insert, and mlir::OpBuilder::setInsertionPointAfter().

◆ getTransferMinorIdentityMap()

AffineMap mlir::vector::getTransferMinorIdentityMap	(	ShapedType	shapedType,
		VectorType	vectorType
	)

Build the default minor identity map suitable for a vector transfer.

This also handles the case memref<... x vector<...>> -> vector<...> in which the rank of the identity map must take the vector element type into account.

Definition at line 122 of file VectorOps.cpp.

References checkSameValueRAW(), checkSameValueWAW(), mlir::AffineMap::get(), mlir::getAffineConstantExpr(), and mlir::AffineMap::getMinorIdentityMap().

Referenced by foldExtractStridedOpFromInsertChain(), foldTransferInBoundsAttribute(), and printTransferAttrs().

◆ getVectorReductionOp()

Value mlir::vector::getVectorReductionOp	(	arith::AtomicRMWKind	op,
		OpBuilder &	builder,
		Location	loc,
		Value	vector
	)

Returns the value obtained by reducing the vector into a scalar using the operation kind associated with a binary AtomicRMWKind op.

Definition at line 442 of file VectorOps.cpp.

References mlir::RewritePatternSet::add(), mlir::OpBuilder::create(), mlir::emitOptionalError(), mlir::failure(), mlir::Builder::getI64ArrayAttr(), mlir::Value::getLoc(), getVectorType(), makeArithReduction(), mlir::OpRewritePattern< SourceOp >::OpRewritePattern(), mlir::RewriterBase::replaceOp(), and mlir::success().

◆ getVectorSubscriptAttr()

ArrayAttr mlir::vector::getVectorSubscriptAttr	(	Builder &	b,
		ArrayRef< int64_t >	values
	)

Returns an integer array attribute containing the given values using the integer type required for subscripts in the vector dialect.

Definition at line 326 of file VectorOps.cpp.

References mlir::Builder::getI64ArrayAttr().

Referenced by foldExtractStridedOpFromInsertChain(), inferStridedSliceOpResultType(), isBroadcastableTo(), and CanonicalizeContractAdd< AddOpType >::matchAndRewrite().

◆ getVectorSubscriptType()

IntegerType mlir::vector::getVectorSubscriptType ( Builder & builder )

Returns the integer type required for subscripts in the vector dialect.

Definition at line 322 of file VectorOps.cpp.

References mlir::Builder::getIntegerType().

◆ isBroadcastableTo()

BroadcastableToResult mlir::vector::isBroadcastableTo	(	Type	srcType,
		VectorType	dstVectorType,
		std::pair< int, int > *	mismatchingDims = `nullptr`
	)

◆ isDisjointTransferIndices()

bool mlir::vector::isDisjointTransferIndices	(	VectorTransferOpInterface	transferA,
		VectorTransferOpInterface	transferB
	)

Same behavior as isDisjointTransferSet but doesn't require the operations to have the same tensor/memref.

This allows comparing operations accessing different tensors.

Definition at line 157 of file VectorOps.cpp.

References mlir::presburger::abs().

Referenced by foldTransferInBoundsAttribute(), isDisjointTransferSet(), and tensorChunkAccessedByUnknownOp().

◆ isDisjointTransferSet()

bool mlir::vector::isDisjointTransferSet	(	VectorTransferOpInterface	transferA,
		VectorTransferOpInterface	transferB
	)

Return true if we can prove that the transfer operations access disjoint memory.

Definition at line 189 of file VectorOps.cpp.

References isDisjointTransferIndices().

Referenced by findAncestorOpInRegion(), and mlir::linalg::hoistRedundantVectorTransfers().

◆ isLastMemrefDimUnitStride()

bool mlir::vector::isLastMemrefDimUnitStride ( MemRefType type )

Return true if the last dimension of the MemRefType has unit stride.

Also return true for memrefs with no strides.

Definition at line 115 of file VectorOps.cpp.

References mlir::getStridesAndOffset(), and mlir::succeeded().

Referenced by foldTransferInBoundsAttribute(), TransferReadToVectorLoadLowering::matchAndRewrite(), and TransferWriteToVectorStoreLowering::matchAndRewrite().

◆ makeArithReduction()

Value mlir::vector::makeArithReduction	(	OpBuilder &	b,
		Location	loc,
		CombiningKind	kind,
		Value	v1,
		Value	v2
	)

Return the result value of reducing two scalar/vector values with the corresponding arith operation.

Referenced by getVectorReductionOp(), and TwoDimMultiReductionToElementWise::matchAndRewrite().

◆ populateBubbleVectorBitCastOpPatterns()

void mlir::vector::populateBubbleVectorBitCastOpPatterns ( RewritePatternSet & patterns )

Collect a set of patterns that bubble up/down bitcast ops.

These patterns move vector.bitcast ops to be before insert ops or after extract ops where suitable. With them, bitcast will happen on smaller vectors and there are more chances to share extract/insert ops.

Definition at line 2897 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateCastAwayVectorLeadingOneDimPatterns()

void mlir::vector::populateCastAwayVectorLeadingOneDimPatterns ( RewritePatternSet & patterns )

Collect a set of leading one dimension removal patterns.

These patterns insert vector.shape_cast to remove leading one dimensions to expose more canonical forms of read/write/insert/extract operations. With them, there are more chances that we can cancel out extract-insert pairs or forward write-read pairs.

Definition at line 438 of file VectorDropLeadUnitDim.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().

◆ populateFlattenVectorTransferPatterns()

void mlir::vector::populateFlattenVectorTransferPatterns ( RewritePatternSet & patterns )

Collect a set of patterns to flatten n-D vector transfers on contiguous memref.

These patterns insert memref.collapse_shape + vector.shape_cast patterns to transform multiple small n-D transfers into a larger 1-D transfer where the memref contiguity properties allow it.

Definition at line 553 of file VectorTransferOpTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().

◆ populatePropagateVectorDistributionPatterns()

void mlir::vector::populatePropagateVectorDistributionPatterns ( RewritePatternSet & patterns )

Collect a set of patterns to propagate insert_map/extract_map in the ssa chain.

Definition at line 831 of file VectorUnrollDistribute.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateShapeCastFoldingPatterns()

void mlir::vector::populateShapeCastFoldingPatterns ( RewritePatternSet & patterns )

Collect a set of vector.shape_cast folding patterns.

Definition at line 2892 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by populateCastAwayVectorLeadingOneDimPatterns(), populateFlattenVectorTransferPatterns(), and populateVectorTransferDropUnitDimsPatterns().

◆ populateVectorBroadcastLoweringPatterns()

void mlir::vector::populateVectorBroadcastLoweringPatterns ( RewritePatternSet & patterns )

Collects patterns to progressively lower vector.broadcast ops on high-D vectors to low-D vector ops.

Definition at line 2904 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateVectorContractLoweringPatterns()

void mlir::vector::populateVectorContractLoweringPatterns	(	RewritePatternSet &	patterns,
		VectorTransformsOptions	options = `VectorTransformsOptions()`
	)

Collects patterns to progressively lower vector contraction ops on high-D into low-D reduction and product ops.

Definition at line 2922 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorInsertExtractStridedSliceDecompositionPatterns()

void mlir::vector::populateVectorInsertExtractStridedSliceDecompositionPatterns ( RewritePatternSet & patterns )

Populate patterns with the following patterns.

[DecomposeDifferentRankInsertStridedSlice]

RewritePattern for InsertStridedSliceOp where source and destination vectors have different ranks.

When ranks are different, InsertStridedSlice needs to extract a properly ranked vector from the destination vector into which to insert. This pattern only takes care of this extraction part and forwards the rest to [VectorInsertStridedSliceOpSameRankRewritePattern].

For a k-D source and n-D destination vector (k < n), we emit:

ExtractOp to extract the (unique) (n-1)-D subvector into which to insert the k-D source.
k-D -> (n-1)-D InsertStridedSlice op
InsertOp that is the reverse of 1.

[DecomposeNDExtractStridedSlice]

For such cases, we can rewrite it to ExtractOp/ExtractElementOp + lower rank ExtractStridedSliceOp + InsertOp/InsertElementOp for the n-D case.

Definition at line 288 of file VectorInsertExtractStridedSliceRewritePatterns.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by populateVectorInsertExtractStridedSliceTransforms(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorInsertExtractStridedSliceTransforms()

void mlir::vector::populateVectorInsertExtractStridedSliceTransforms ( RewritePatternSet & patterns )

Populate patterns with the following patterns.

Populate the given list with patterns that convert from Vector to LLVM.

Patterns in populateVectorInsertExtractStridedSliceDecompositionPatterns();

[ConvertSameRankInsertStridedSliceIntoShuffle]

RewritePattern for InsertStridedSliceOp where source and destination vectors have the same rank. For each outermost index in the slice: begin end stride [offset : offset+size*stride : stride]

ExtractOp one (k-1)-D source subvector and one (n-1)-D dest subvector.
InsertStridedSlice (k-1)-D into (n-1)-D
the destination subvector is inserted back in the proper place

InsertOp that is the reverse of 1.

[Convert1DExtractStridedSliceIntoShuffle]

For such cases, we can lower it to a ShuffleOp.

Definition at line 295 of file VectorInsertExtractStridedSliceRewritePatterns.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateVectorInsertExtractStridedSliceDecompositionPatterns().

Referenced by mlir::populateVectorToLLVMConversionPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorMaskMaterializationPatterns()

void mlir::vector::populateVectorMaskMaterializationPatterns	(	RewritePatternSet &	patterns,
		bool	force32BitVectorIndices
	)

These patterns materialize masks for various vector ops such as transfers.

Definition at line 2884 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateVectorMaskOpLoweringPatterns()

void mlir::vector::populateVectorMaskOpLoweringPatterns ( RewritePatternSet & patterns )

Collects patterns to progressively lower vector mask ops into elementary selection and insertion ops.

Definition at line 2909 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateVectorMultiReductionLoweringPatterns()

void mlir::vector::populateVectorMultiReductionLoweringPatterns	(	RewritePatternSet &	patterns,
		VectorMultiReductionLowering	options
	)

Collect a set of patterns to convert vector.multi_reduction op into a sequence of vector.reduction ops.

The patterns comprise:

InnerOuterDimReductionConversion: rewrites vector.multi_reduction such that all reduction dimensions are either innermost or outermost, by adding the proper vector.transpose operations.
ReduceMultiDimReductionRank: once in innermost or outermost reduction form, rewrites n-D vector.multi_reduction into 2-D vector.multi_reduction, by introducing vector.shape_cast ops to collapse + multi-reduce + expand back.
TwoDimMultiReductionToElementWise: once in 2-D vector.multi_reduction form, with an outermost reduction dimension, unroll the outer dimension to obtain a sequence of 1-D vector ops. This also has an opportunity for tree-reduction (in the future).
TwoDimMultiReductionToReduction: once in 2-D vector.multi_reduction form, with an innermost reduction dimension, unroll the outer dimension to obtain a sequence of extract + vector.reduction + insert. This can further lower to horizontal reduction ops.
OneDimMultiReductionToTwoDim: for cases that reduce to 1-D vector<k> reduction (and are thus missing either a parallel or a reduction), we lift them back up to 2-D with a simple vector.shape_cast to vector<1xk> so that the other patterns can kick in, thus fully exiting out of the vector.multi_reduction abstraction.

Definition at line 340 of file VectorMultiDimReductionTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorReductionToContractPatterns()

void mlir::vector::populateVectorReductionToContractPatterns ( RewritePatternSet & patterns )

Collect patterns to convert reduction op to vector.contract and fold transpose/broadcast ops into the contract.

Definition at line 2936 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateVectorTransferCollapseInnerMostContiguousDimsPatterns().

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorScanLoweringPatterns()

void mlir::vector::populateVectorScanLoweringPatterns ( RewritePatternSet & patterns )

Collect patterns to convert scan op.

Definition at line 2959 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorShapeCastLoweringPatterns()

void mlir::vector::populateVectorShapeCastLoweringPatterns ( RewritePatternSet & patterns )

Collects patterns to progressively lower vector.shape_cast ops on high-D vectors into 1-D/2-D vector ops by generating data movement extract/insert ops.

Definition at line 2915 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateVectorToVectorCanonicalizationPatterns()

void mlir::vector::populateVectorToVectorCanonicalizationPatterns ( RewritePatternSet & patterns )

Collect a set of vector-to-vector canonicalization patterns.

◆ populateVectorTransferCollapseInnerMostContiguousDimsPatterns()

void mlir::vector::populateVectorTransferCollapseInnerMostContiguousDimsPatterns ( RewritePatternSet & patterns )

Collect a set of patterns to reduce the rank of the operands of vector transfer ops to operate on the largest contigious vector.

These patterns are useful when lowering to dialects with 1d vector type such as llvm and it will result fewer memory reads.

Definition at line 2944 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by populateVectorReductionToContractPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorTransferDropUnitDimsPatterns()

void mlir::vector::populateVectorTransferDropUnitDimsPatterns ( RewritePatternSet & patterns )

Collect a set of one dimension removal patterns.

These patterns insert rank-reducing memref.subview ops to remove one dimensions. With them, there are more chances that we can avoid potentially exensive vector.shape_cast operations.

Definition at line 545 of file VectorTransferOpTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and populateShapeCastFoldingPatterns().

◆ populateVectorTransferLoweringPatterns()

void mlir::vector::populateVectorTransferLoweringPatterns	(	RewritePatternSet &	patterns,
		llvm::Optional< unsigned >	maxTransferRank = `llvm::None`
	)

Collect a set of transfer read/write lowering patterns.

These patterns lower transfer ops to simpler ops like vector.load, vector.store and vector.broadcast. Only transfers with a transfer rank of a most maxTransferRank are lowered. This is useful when combined with VectorToSCF, which reduces the rank of vector transfer ops.

Definition at line 2949 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by mlir::populateVectorToLLVMConversionPatterns().

◆ populateVectorTransferPermutationMapLoweringPatterns()

void mlir::vector::populateVectorTransferPermutationMapLoweringPatterns ( RewritePatternSet & patterns )

Collect a set of transfer read/write lowering patterns that simplify the permutation map (e.g., converting it to a minor identity map) by inserting broadcasts and transposes.

More specifically:

[TransferReadPermutationLowering] Lower transfer_read op with permutation into a transfer_read with a permutation map composed of leading zeros followed by a minor identity + vector.transpose op. Ex: vector.transfer_read ... permutation_map: (d0, d1, d2) -> (0, d1) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2) -> (d1, 0) vector.transpose v, [1, 0]

vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, 0, 0, d1, d3) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, 0, d1, 0, d3) vector.transpose v, [0, 1, 3, 2, 4] Note that an alternative is to transform it to linalg.transpose + vector.transfer_read to do the transpose in memory instead.

[TransferWritePermutationLowering] Lower transfer_write op with permutation into a transfer_write with a minor identity permutation map. (transfer_write ops cannot have broadcasts.) Ex: vector.transfer_write v ... permutation_map: (d0, d1, d2) -> (d2, d0, d1) into: tmp = vector.transpose v, [2, 0, 1] vector.transfer_write tmp ... permutation_map: (d0, d1, d2) -> (d0, d1, d2)

vector.transfer_write v ... permutation_map: (d0, d1, d2, d3) -> (d3, d2) into: tmp = vector.transpose v, [1, 0] v = vector.transfer_write tmp ... permutation_map: (d0, d1, d2, d3) -> (d2, d3)

[TransferOpReduceRank] Lower transfer_read op with broadcast in the leading dimensions into transfer_read of lower rank + vector.broadcast. Ex: vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (0, d1, 0, d3) into: v = vector.transfer_read ... permutation_map: (d0, d1, d2, d3) -> (d1, 0, d3) vector.broadcast v

Definition at line 273 of file VectorTransferPermutationMapRewritePatterns.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by mlir::populateVectorToSCFConversionPatterns(), and mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorTransposeLoweringPatterns()

void mlir::vector::populateVectorTransposeLoweringPatterns	(	RewritePatternSet &	patterns,
		VectorTransformsOptions	options = `VectorTransformsOptions()`
	)

Insert TransposeLowering patterns into extraction/insertion.

Definition at line 2930 of file VectorTransforms.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateVectorUnrollPatterns()

void mlir::vector::populateVectorUnrollPatterns	(	RewritePatternSet &	patterns,
		const UnrollVectorOptions &	options
	)

Collect a set of pattern to unroll vector operations to a smaller shapes.

options structure controls which operations are unrolled and the target shape. op is unrolled to the targetShape as follows, for each of its operands:

the unrolled type unrolledVectorType and number of unrolled instances numUnrolledInstances are computed from the targetShape. For now it is assumed the unrolling factors divide the vector sizes.
ExtractStridedSlice are created to break-up the vector operands.
the original op is cloned numUnrolledInstances times, once for each result.
InsertStridedSlice are inserted to re-assemble the slices into the original vectore shape.

Example:

opA(operand0, operand1) // numUnrolledInstances = 3

    operand0                   operand1
       |                          |
     fork                       fork
<----------gather all fork ops --------->
      /|\                        /|\
  f00 f01 f02                f10 f11 f12
<---------- clone op 3 times --------->
  opA0(f00, f10), opA1(f01, f11), opA2(f02, f12)
         \            |            /

<-----------------— join ----------------------—>

Other local patterns then kick in iteratively (including DCE) and compose to combine the ExtractStridedSlice/InsertStridedSlice.

Definition at line 823 of file VectorUnrollDistribute.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.

Referenced by mlir::vector::UnrollVectorOptions::setUnrollTraversalOrderFn().

◆ populateWarpExecuteOnLane0OpToScfForPattern()

void mlir::vector::populateWarpExecuteOnLane0OpToScfForPattern	(	RewritePatternSet &	patterns,
		const WarpExecuteOnLane0LoweringOptions &	options
	)

Definition at line 931 of file VectorDistribute.cpp.

References mlir::RewritePatternSet::add(), canBeHoisted(), mlir::RewritePatternSet::getContext(), mlir::Operation::getResults(), mlir::ResultRange::getType(), mlir::Operation::moveBefore(), options, value, and mlir::Block::without_terminator().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::vector::registerBufferizableOpInterfaceExternalModels ( DialectRegistry & registry )

Definition at line 160 of file BufferizableOpInterfaceImpl.cpp.

References mlir::DialectRegistry::addExtension().

Referenced by mlir::registerAllDialects().

◆ splitFullAndPartialTransfer()

LogicalResult mlir::vector::splitFullAndPartialTransfer	(	RewriterBase &	b,
		VectorTransferOpInterface	xferOp,
		VectorTransformsOptions	options = `VectorTransformsOptions()`,
		scf::IfOp *	ifOp = `nullptr`
	)

Split a vector.transfer operation into an in-bounds (i.e., no out-of-bounds masking) fastpath and a slowpath.

If ifOp is not null and the result is success, theifOp` points to the newly created conditional upon function return. To accomodate for the fact that the original vector.transfer indexing may be arbitrary and the slow path indexes @[0...0] in the temporary buffer, the scf.if op returns a view and values of type index. At this time, only vector.transfer_read case is implemented.

Example (a 2-D vector.transfer_read): ``` %1 = vector.transfer_read %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { // fastpath, direct cast memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { // slowpath, not in-bounds vector.transfer or linalg.copy. memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index */ // } /** %0 = vector.transfer_read %1#0[%1#1, %1#2] {in_bounds = [true ... true]} `` wherealloc` is a top of the function alloca'ed buffer of one vector.

Preconditions:

xferOp.permutation_map() must be a minor identity map
the rank of the xferOp.memref() and the rank of the xferOp.vector() must be equal. This will be relaxed in the future but requires rank-reducing subviews.

For vector.transfer_read: If ifOp is not null and the result is success, theifOp` points to the newly created conditional upon function return. To accomodate for the fact that the original vector.transfer indexing may be arbitrary and the slow path indexes @[0...0] in the temporary buffer, the scf.if op returns a view and values of type index.

Example (a 2-D vector.transfer_read): ``` %1 = vector.transfer_read %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { // fastpath, direct cast memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { // slowpath, not in-bounds vector.transfer or linalg.copy. memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index */ // } /** %0 = vector.transfer_read %1#0[%1#1, %1#2] {in_bounds = [true ... true]} `` wherealloc` is a top of the function alloca'ed buffer of one vector.

For vector.transfer_write: There are 2 conditional blocks. First a block to decide which memref and indices to use for an unmasked, inbounds write. Then a conditional block to further copy a partial buffer into the final result in the slow path case.

Example (a 2-D vector.transfer_write): ``` vector.transfer_write arg, %0[...], pad : memref<A...>, vector<...> ``` is transformed into: ``` %1:3 = scf.if (inBounds) { memref.cast A: memref<A...> to compatibleMemRefType scf.yield view : compatibleMemRefType, index, index } else { memref.cast alloc: memref<B...> to compatibleMemRefType scf.yield %4 : compatibleMemRefType, index, index } %0 = vector.transfer_write arg, %1#0[%1#1, %1#2] {in_bounds = [true ... true]} scf.if (notInBounds) { // slowpath: not in-bounds vector.transfer or linalg.copy. } `` wherealloc` is a top of the function alloca'ed buffer of one vector.

Preconditions:

xferOp.permutation_map() must be a minor identity map
the rank of the xferOp.source() and the rank of the xferOp.vector() must be equal. This will be relaxed in the future but requires rank-reducing subviews.

Definition at line 519 of file VectorTransferSplitRewritePatterns.cpp.

References mlir::Type::cast(), mlir::OpBuilder::clone(), mlir::OpBuilder::create(), createFullPartialLinalgCopy(), createFullPartialVectorTransferRead(), createFullPartialVectorTransferWrite(), createInBoundsCond(), mlir::failure(), ForceInBounds, mlir::Region::front(), getAutomaticAllocationScope(), mlir::Builder::getBoolArrayAttr(), getCastCompatibleMemRefType(), mlir::Builder::getI64IntegerAttr(), mlir::Builder::getIndexType(), mlir::Operation::getLoc(), getLocationToWriteFullVec(), mlir::Operation::getNumRegions(), mlir::Operation::getRegion(), mlir::Value::getType(), mlir::BlockAndValueMapping::map(), None, mlir::Operation::setAttr(), mlir::OpBuilder::setInsertionPoint(), mlir::OpBuilder::setInsertionPointToStart(), splitFullAndPartialTransferPrecondition(), mlir::succeeded(), mlir::success(), VectorTransfer, and mlir::vector::VectorTransformsOptions::vectorTransferSplit.

Referenced by mlir::vector::VectorTransferFullPartialRewriter::matchAndRewrite().

◆ transferOpflowOpt()

void mlir::vector::transferOpflowOpt ( Operation * rootOp )

Implements transfer op write to read forwarding and dead transfer write optimizations.

Definition at line 529 of file VectorTransferOpTransforms.cpp.

References mlir::Operation::walk().

Namespaces

Classes

Typedefs

Enumerations

Functions

Typedef Documentation

◆ DistributionMapFn

Enumeration Type Documentation

◆ BroadcastableToResult

◆ VectorContractLowering

◆ VectorMultiReductionLowering

◆ VectorTransferSplit

◆ VectorTransposeLowering

Function Documentation

◆ checkSameValueRAW()

◆ checkSameValueWAW()

◆ createOrFoldDimOp()

◆ createVectorBufferizePass()

◆ distributPointwiseVectorOp()

◆ getTransferMinorIdentityMap()

◆ getVectorReductionOp()

◆ getVectorSubscriptAttr()

◆ getVectorSubscriptType()

◆ isBroadcastableTo()

◆ isDisjointTransferIndices()

◆ isDisjointTransferSet()

◆ isLastMemrefDimUnitStride()

◆ makeArithReduction()

◆ populateBubbleVectorBitCastOpPatterns()

◆ populateCastAwayVectorLeadingOneDimPatterns()

◆ populateFlattenVectorTransferPatterns()

◆ populatePropagateVectorDistributionPatterns()

◆ populateShapeCastFoldingPatterns()

◆ populateVectorBroadcastLoweringPatterns()

◆ populateVectorContractLoweringPatterns()

◆ populateVectorInsertExtractStridedSliceDecompositionPatterns()

[DecomposeDifferentRankInsertStridedSlice]

[DecomposeNDExtractStridedSlice]

◆ populateVectorInsertExtractStridedSliceTransforms()

[ConvertSameRankInsertStridedSliceIntoShuffle]

[Convert1DExtractStridedSliceIntoShuffle]

◆ populateVectorMaskMaterializationPatterns()

◆ populateVectorMaskOpLoweringPatterns()

◆ populateVectorMultiReductionLoweringPatterns()

◆ populateVectorReductionToContractPatterns()

◆ populateVectorScanLoweringPatterns()

◆ populateVectorShapeCastLoweringPatterns()

◆ populateVectorToVectorCanonicalizationPatterns()

◆ populateVectorTransferCollapseInnerMostContiguousDimsPatterns()

◆ populateVectorTransferDropUnitDimsPatterns()

◆ populateVectorTransferLoweringPatterns()

◆ populateVectorTransferPermutationMapLoweringPatterns()

◆ populateVectorTransposeLoweringPatterns()

◆ populateVectorUnrollPatterns()

◆ populateWarpExecuteOnLane0OpToScfForPattern()

◆ registerBufferizableOpInterfaceExternalModels()

◆ splitFullAndPartialTransfer()

◆ transferOpflowOpt()