Classes
class	ForLoopPipeliningPattern
struct	LoopNest
struct	PipeliningOption
	Options to dictate how loops should be pipelined. More...

Typedefs
using	ValueVector = SmallVector<Value>
	An owning vector of values, handy to return from functions.
using	LoopVector = SmallVector<scf::ForOp>
using	ValueTypeCastFnTy
	Perform a replacement of one iter OpOperand of an scf.for to the replacement value with a different type.
using	SCFTileSizeComputationFunction
using	LoopMatcherFn
	Match "for loop"-like operations: If the first parameter is an iteration variable, return lower/upper bounds via the second/third parameter and the step size via the last parameter.

Functions
void	buildTerminatedBody (OpBuilder &builder, Location loc)
	Default callback for IfOp builders. Inserts a yield without arguments.
ForOp	getForInductionVarOwner (Value val)
	Returns the loop parent of an induction variable.
ParallelOp	getParallelForInductionVarOwner (Value val)
	Returns the parallel loop parent of an induction variable.
ForallOp	getForallOpThreadIndexOwner (Value val)
	Returns the ForallOp parent of an thread index variable.
bool	insideMutuallyExclusiveBranches (Operation a, Operation b)
	Return true if ops a and b (or their ancestors) are in mutually exclusive regions/blocks of an IfOp.
void	promote (RewriterBase &rewriter, scf::ForallOp forallOp)
	Promotes the loop body of a scf::ForallOp to its containing block.
LoopNest	buildLoopNest (OpBuilder &builder, Location loc, ValueRange lbs, ValueRange ubs, ValueRange steps, ValueRange iterArgs, function_ref< ValueVector(OpBuilder &, Location, ValueRange, ValueRange)> bodyBuilder=nullptr)
	Creates a perfect nest of "for" loops, i.e.
LoopNest	buildLoopNest (OpBuilder &builder, Location loc, ValueRange lbs, ValueRange ubs, ValueRange steps, function_ref< void(OpBuilder &, Location, ValueRange)> bodyBuilder=nullptr)
	A convenience version for building loop nests without iteration arguments (like for reductions).
SmallVector< Value >	replaceAndCastForOpIterArg (RewriterBase &rewriter, scf::ForOp forOp, OpOperand &operand, Value replacement, const ValueTypeCastFnTy &castFn)
std::optional< llvm::APSInt >	computeUbMinusLb (Value lb, Value ub, bool isSigned)
	Helper function to compute the difference between two values.
void	registerValueBoundsOpInterfaceExternalModels (DialectRegistry &registry)
void	registerTransformDialectExtension (DialectRegistry &registry)
void	registerBufferDeallocationOpInterfaceExternalModels (DialectRegistry &registry)
void	registerBufferizableOpInterfaceExternalModels (DialectRegistry &registry)
void	populateSCFStructuralTypeConversionsAndLegality (const TypeConverter &typeConverter, RewritePatternSet &patterns, ConversionTarget &target, PatternBenefit benefit=1)
	Populates patterns for SCF structural type conversions and sets up the provided ConversionTarget with the appropriate legality configuration for the ops to get converted properly.
void	populateSCFStructuralTypeConversions (const TypeConverter &typeConverter, RewritePatternSet &patterns, PatternBenefit benefit=1)
	Similar to populateSCFStructuralTypeConversionsAndLegality but does not populate the conversion target.
void	populateSCFStructuralTypeConversionTarget (const TypeConverter &typeConverter, ConversionTarget &target)
	Updates the ConversionTarget with dynamic legality of SCF operations based on the provided type converter.
void	populateSCFLoopPipeliningPatterns (RewritePatternSet &patterns, const PipeliningOption &options)
	Populate patterns for SCF software pipelining transformation.
void	populateSCFForLoopCanonicalizationPatterns (RewritePatternSet &patterns)
	Populate patterns for canonicalizing operations inside SCF loop bodies.
void	populateUpliftWhileToForPatterns (RewritePatternSet &patterns)
	Populate patterns to uplift scf.while ops to scf.for.
void	populateSCFRotateWhileLoopPatterns (RewritePatternSet &patterns)
	Populate patterns to rotate scf.while ops, constructing do-while loops from while loops.
LogicalResult	forallToForLoop (RewriterBase &rewriter, ForallOp forallOp, SmallVectorImpl< Operation * > *results=nullptr)
	Try converting scf.forall into a set of nested scf.for loops.
LogicalResult	forallToParallelLoop (RewriterBase &rewriter, ForallOp forallOp, ParallelOp *result=nullptr)
	Try converting scf.forall into an scf.parallel loop.
FailureOr< scf::LoopNest >	parallelForToNestedFors (RewriterBase &rewriter, ParallelOp parallelOp)
	Try converting scf.forall into an scf.parallel loop.
void	naivelyFuseParallelOps (Region &region, llvm::function_ref< bool(Value, Value)> mayAlias)
	Fuses all adjacent scf.parallel operations with identical bounds and step into one scf.parallel operations.
LogicalResult	peelForLoopAndSimplifyBounds (RewriterBase &rewriter, ForOp forOp, scf::ForOp &partialIteration)
	Rewrite a for loop with bounds/step that potentially do not divide evenly into a for loop where the step divides the iteration space evenly, followed by another scf.for for the last (partial) iteration (if any; returned via partialIteration).
LogicalResult	peelForLoopFirstIteration (RewriterBase &rewriter, ForOp forOp, scf::ForOp &partialIteration)
	Peel the first iteration out of the scf.for loop.
std::pair< ParallelOp, ParallelOp >	tileParallelLoop (ParallelOp op, llvm::ArrayRef< int64_t > tileSizes, bool noMinMaxBounds)
	Tile a parallel loop of the form scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4, arg5)
FailureOr< ForOp >	pipelineForLoop (RewriterBase &rewriter, ForOp forOp, const PipeliningOption &options, bool *modifiedIR=nullptr)
	Generate a pipelined version of the scf.for loop based on the schedule given as option.
FailureOr< WhileOp >	wrapWhileLoopInZeroTripCheck (WhileOp whileOp, RewriterBase &rewriter, bool forceCreateCheck=false)
	Create zero-trip-check around a while op and return the new loop op in the check.
FailureOr< ForOp >	upliftWhileToForLoop (RewriterBase &rewriter, WhileOp loop)
	Try to uplift scf.while op to scf.for.
LogicalResult	matchForLikeLoop (Value iv, OpFoldResult &lb, OpFoldResult &ub, OpFoldResult &step)
	Match "for loop"-like operations from the SCF dialect.
LogicalResult	addLoopRangeConstraints (affine::FlatAffineValueConstraints &cstr, Value iv, OpFoldResult lb, OpFoldResult ub, OpFoldResult step)
	Populate the given constraint set with induction variable constraints of a "for" loop with the given range and step.
LogicalResult	canonicalizeMinMaxOpInLoop (RewriterBase &rewriter, Operation *op, LoopMatcherFn loopMatcher)
	Try to canonicalize the given affine.min/max operation in the context of for loops with a known range.
LogicalResult	rewritePeeledMinMaxOp (RewriterBase &rewriter, Operation *op, Value iv, Value ub, Value step, bool insideLoop)
	Try to simplify the given affine.min/max operation op after loop peeling.

Variables
SmallVector< Operation * >	mergeOps
	Computation function returning, for the op currently being tiled or fused, the per-iteration-domain-dimension InnerTileAlignment array.

Typedef Documentation

◆ LoopMatcherFn

using mlir::scf::LoopMatcherFn

Initial value:

function_ref<LogicalResult(

Value, OpFoldResult &, OpFoldResult &, OpFoldResult &)>

mlir::OpFoldResult

This class represents a single result from folding an operation.

Definition OpDefinition.h:273

mlir::Value

This class represents an instance of an SSA value in the MLIR system, representing a computable value...

Definition Value.h:96

mlir::function_ref

llvm::function_ref< Fn > function_ref

Definition LLVM.h:147

Match "for loop"-like operations: If the first parameter is an iteration variable, return lower/upper bounds via the second/third parameter and the step size via the last parameter.

The function should return success in that case. If the first parameter is not an iteration variable, return failure.

Definition at line 39 of file AffineCanonicalizationUtils.h.

◆ LoopVector

using mlir::scf::LoopVector = SmallVector<scf::ForOp>

Definition at line 66 of file SCF.h.

◆ SCFTileSizeComputationFunction

using mlir::scf::SCFTileSizeComputationFunction

Initial value:

std::function<SmallVector<OpFoldResult>(OpBuilder &, Operation *)>

mlir::OpBuilder

This class helps build Operations.

Definition Builders.h:209

mlir::Operation

Operation is the basic unit of execution within MLIR.

Definition Operation.h:87

Definition at line 31 of file TileUsingInterface.h.

◆ ValueTypeCastFnTy

using mlir::scf::ValueTypeCastFnTy

Initial value:

llvm::function_ref<Value(OpBuilder &, Location loc, Type, Value)>

llvm::function_ref

Definition LLVM.h:84

mlir::Location

This class defines the main interface for locations in MLIR and acts as a non-nullable wrapper around...

Definition Location.h:76

mlir::Type

Instances of the Type class are uniqued, have an immutable identifier and an optional mutable compone...

Definition Types.h:74

Perform a replacement of one iter OpOperand of an scf.for to the replacement value with a different type.

A callback is used to insert cast ops inside the block to account for type differences.

Definition at line 108 of file SCF.h.

◆ ValueVector

using mlir::scf::ValueVector = SmallVector<Value>

An owning vector of values, handy to return from functions.

Definition at line 65 of file SCF.h.

Function Documentation

◆ addLoopRangeConstraints()

LogicalResult mlir::scf::addLoopRangeConstraints	(	affine::FlatAffineValueConstraints &	cstr,
		Value	iv,
		OpFoldResult	lb,
		OpFoldResult	ub,
		OpFoldResult	step )

Populate the given constraint set with induction variable constraints of a "for" loop with the given range and step.

Definition at line 77 of file AffineCanonicalizationUtils.cpp.

References mlir::affine::FlatAffineValueConstraints::addBound(), mlir::presburger::IntegerRelation::addInequality(), mlir::FlatLinearValueConstraints::appendDimVar(), mlir::FlatLinearValueConstraints::appendSymbolVar(), b, mlir::AffineMap::get(), mlir::getConstantIntValue(), mlir::Value::getContext(), mlir::presburger::IntegerRelation::getNumCols(), mlir::presburger::IntegerRelation::getNumDimVars(), and mlir::presburger::IntegerRelation::getNumSymbolVars().

Referenced by canonicalizeMinMaxOpInLoop().

◆ buildLoopNest() [1/2]

LoopNest mlir::scf::buildLoopNest	(	OpBuilder &	builder,
		Location	loc,
		ValueRange	lbs,
		ValueRange	ubs,
		ValueRange	steps,
		function_ref< void(OpBuilder &, Location, ValueRange)>	bodyBuilder = nullptr )

A convenience version for building loop nests without iteration arguments (like for reductions).

Does not take the initial value of reductions or expect the body building functions to return their current value. The built nested scf::For are captured in capturedLoops when non-null.

Definition at line 866 of file SCF.cpp.

References buildLoopNest().

◆ buildLoopNest() [2/2]

LoopNest mlir::scf::buildLoopNest	(	OpBuilder &	builder,
		Location	loc,
		ValueRange	lbs,
		ValueRange	ubs,
		ValueRange	steps,
		ValueRange	iterArgs,
		function_ref< ValueVector(OpBuilder &, Location, ValueRange, ValueRange)>	bodyBuilder = nullptr )

Creates a perfect nest of "for" loops, i.e.

all loops but the innermost contain only another loop and a terminator. The lower, upper bounds and steps are provided as lbs, ubs and steps, which are expected to be of the same size. iterArgs points to the initial values of the loop iteration arguments, which will be forwarded through the nest to the innermost loop. The body of the loop is populated using bodyBuilder, which accepts an ordered list of induction variables of all loops, followed by a list of iteration arguments of the innermost loop, in the same order as provided to iterArgs. This function is expected to return as many values as iterArgs, of the same type and in the same order, that will be treated as yielded from the loop body and forwarded back through the loop nest. If the function is not provided, the loop nest is not expected to have iteration arguments, the body of the innermost loop will be left empty, containing only the zero-operand terminator. Returns the LoopNest containing the list of perfectly nest scf::ForOp build during the call. If bound arrays are empty, the body builder will be called once to construct the IR outside of the loop with an empty list of induction variables.

Definition at line 793 of file SCF.cpp.

References mlir::OpBuilder::setInsertionPointToEnd(), mlir::OpBuilder::setInsertionPointToStart(), and ValueRange.

Referenced by buildLoopNest(), mlir::linalg::GenerateLoopNest< LoopTy >::doit(), mlir::linalg::generateParallelLoopNest(), and insertCopyLoops().

◆ buildTerminatedBody()

void mlir::scf::buildTerminatedBody	(	OpBuilder &	builder,
		Location	loc )

Default callback for IfOp builders. Inserts a yield without arguments.

Definition at line 94 of file SCF.cpp.

◆ canonicalizeMinMaxOpInLoop()

LogicalResult mlir::scf::canonicalizeMinMaxOpInLoop	(	RewriterBase &	rewriter,
		Operation *	op,
		LoopMatcherFn	loopMatcher )

Try to canonicalize the given affine.min/max operation in the context of for loops with a known range.

Canonicalize min/max operations in the context of for loops with a known range.

loopMatcher is used to retrieve loop bounds and the step size for a given iteration variable.

Note: loopMatcher allows this function to be used with any "for loop"-like operation (scf.for, scf.parallel and even ops defined in other dialects).

Call canonicalizeMinMaxOp and add the following constraints to the constraint system (along with the missing dimensions):

iv >= lb
iv < lb + step * ((ub - lb - 1) floorDiv step) + 1

Note: Due to limitations of IntegerPolyhedron, only constant step sizes are currently supported.

Definition at line 145 of file AffineCanonicalizationUtils.cpp.

References addLoopRangeConstraints(), canonicalizeMinMaxOp(), and mlir::Operation::getOperands().

◆ computeUbMinusLb()

std::optional< llvm::APSInt > mlir::scf::computeUbMinusLb	(	Value	lb,
		Value	ub,
		bool	isSigned )

Helper function to compute the difference between two values.

This is used by the loop implementations to compute the trip count.

Definition at line 115 of file SCF.cpp.

References mlir::m_ConstantInt(), and mlir::matchPattern().

Referenced by mlir::getConstLoopTripCounts().

◆ forallToForLoop()

LogicalResult mlir::scf::forallToForLoop	(	RewriterBase &	rewriter,
		ForallOp	forallOp,
		SmallVectorImpl< Operation * > *	results = nullptr )

Try converting scf.forall into a set of nested scf.for loops.

The newly created scf.for ops will be returned through the results vector if provided.

◆ forallToParallelLoop()

LogicalResult mlir::scf::forallToParallelLoop	(	RewriterBase &	rewriter,
		ForallOp	forallOp,
		ParallelOp *	result = nullptr )

Try converting scf.forall into an scf.parallel loop.

The conversion is only supported for forall operations with no results.

References result.

◆ getForallOpThreadIndexOwner()

ForallOp mlir::scf::getForallOpThreadIndexOwner ( Value val )

Returns the ForallOp parent of an thread index variable.

If the provided value is not a thread index variable, then return nullptr.

Definition at line 1388 of file SCF.cpp.

Referenced by matchForLikeLoop().

◆ getForInductionVarOwner()

ForOp mlir::scf::getForInductionVarOwner ( Value val )

Returns the loop parent of an induction variable.

If the provided value is not an induction variable, then return nullptr.

Definition at line 675 of file SCF.cpp.

Referenced by buildPackingLoopNestImpl(), matchForLikeLoop(), and replaceByPackingResult().

◆ getParallelForInductionVarOwner()

ParallelOp mlir::scf::getParallelForInductionVarOwner ( Value val )

Returns the parallel loop parent of an induction variable.

If the provided value is not an induction variable, then return nullptr.

Definition at line 2965 of file SCF.cpp.

Referenced by matchForLikeLoop().

◆ insideMutuallyExclusiveBranches()

bool mlir::scf::insideMutuallyExclusiveBranches	(	Operation *	a,
		Operation *	b )

Return true if ops a and b (or their ancestors) are in mutually exclusive regions/blocks of an IfOp.

Definition at line 1914 of file SCF.cpp.

References b, and mlir::Operation::getParentOfType().

◆ matchForLikeLoop()

LogicalResult mlir::scf::matchForLikeLoop	(	Value	iv,
		OpFoldResult &	lb,
		OpFoldResult &	ub,
		OpFoldResult &	step )

Match "for loop"-like operations from the SCF dialect.

Definition at line 31 of file AffineCanonicalizationUtils.cpp.

References getForallOpThreadIndexOwner(), getForInductionVarOwner(), getParallelForInductionVarOwner(), and success().

◆ naivelyFuseParallelOps()

void mlir::scf::naivelyFuseParallelOps	(	Region &	region,
		llvm::function_ref< bool(Value, Value)>	mayAlias )

Fuses all adjacent scf.parallel operations with identical bounds and step into one scf.parallel operations.

Uses a naive aliasing and dependency analysis. User can additionally customize alias checking with mayAlias hook. mayAlias must return false if 2 values are guaranteed to not alias.

Definition at line 1038 of file ParallelLoopFusion.cpp.

References b, fuseIfLegal(), mlir::isMemoryEffectFree(), and mayAlias().

◆ parallelForToNestedFors()

FailureOr< scf::LoopNest > mlir::scf::parallelForToNestedFors	(	RewriterBase &	rewriter,
		ParallelOp	parallelOp )

Try converting scf.forall into an scf.parallel loop.

The conversion is only supported for parallel operations with no results.

References mayAlias().

◆ peelForLoopAndSimplifyBounds()

LogicalResult mlir::scf::peelForLoopAndSimplifyBounds	(	RewriterBase &	rewriter,
		ForOp	forOp,
		scf::ForOp &	partialIteration )

Rewrite a for loop with bounds/step that potentially do not divide evenly into a for loop where the step divides the iteration space evenly, followed by another scf.for for the last (partial) iteration (if any; returned via partialIteration).

This transformation is called "loop peeling".

This transformation is beneficial for a wide range of transformations such as vectorization or loop tiling: It enables additional canonicalizations inside the peeled loop body such as rewriting masked loads into unmaked loads.

E.g., assuming a lower bound of 0 (for illustration purposes):

scf.for %iv = %c0 to %ub step %c4 {
  (loop body)
}

is rewritten into the following pseudo IR:

%newUb = %ub - (%ub mod %c4)
scf.for %iv = %c0 to %newUb step %c4 {
  (loop body)
}
scf.for %iv2 = %newUb to %ub {
  (loop body)
}

After loop peeling, this function tries to simplify affine.min and affine.max ops in the body of the peeled loop and in the body of the partial iteration loop, taking advantage of the fact that the peeled loop has only "full" iterations. This simplification is expected to enable further canonicalization opportunities through other patterns.

The return value indicates whether the loop was rewritten or not. Loops are not rewritten if:

Loop step size is 1 or
Loop bounds and step size are static, and step already divides the iteration space evenly.

Note: This function rewrites the given scf.for loop in-place and creates a new scf.for operation for the last iteration. It replaces all uses of the unpeeled loop with the results of the newly generated scf.for.

Referenced by mlir::linalg::peelLoop().

◆ peelForLoopFirstIteration()

LogicalResult mlir::scf::peelForLoopFirstIteration	(	RewriterBase &	rewriter,
		ForOp	forOp,
		scf::ForOp &	partialIteration )

Peel the first iteration out of the scf.for loop.

If there is only one iteration, return the original loop.

◆ pipelineForLoop()

FailureOr< ForOp > mlir::scf::pipelineForLoop	(	RewriterBase &	rewriter,
		ForOp	forOp,
		const PipeliningOption &	options,
		bool *	modifiedIR = nullptr )

Generate a pipelined version of the scf.for loop based on the schedule given as option.

This applies the mechanical transformation of changing the loop and generating the prologue/epilogue for the pipelining and doesn't make any decision regarding the schedule. Based on the options the loop is split into several stages. The transformation assumes that the scheduling given by user is valid. For example if we break a loop into 3 stages named S0, S1, S2 we would generate the following code with the number in parenthesis as the iteration index:

S0(0) // Prologue S0(1) S1(0) // Prologue scf.for I = C0 to N - 2 { S0(I+2) S1(I+1) S2(I) // Pipelined kernel } S1(N) S2(N-1) // Epilogue S2(N) // Epilogue

If modifiedIR is provided, it will be set to a value that indicates whether pipelining modified the IR before failing, signaling to the caller whether they can proceed with different transformations.

Definition at line 789 of file LoopPipelining.cpp.

References mlir::RewriterBase::eraseOp(), options, mlir::RewriterBase::replaceOp(), and mlir::OpBuilder::setInsertionPointAfter().

Referenced by mlir::scf::ForLoopPipeliningPattern::returningMatchAndRewrite().

◆ populateSCFForLoopCanonicalizationPatterns()

void mlir::scf::populateSCFForLoopCanonicalizationPatterns ( RewritePatternSet & patterns )

Populate patterns for canonicalizing operations inside SCF loop bodies.

At the moment, only affine.min/max computations with iteration variables, loop bounds and loop steps are canonicalized.

Definition at line 175 of file LoopCanonicalization.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateSCFLoopPipeliningPatterns()

void mlir::scf::populateSCFLoopPipeliningPatterns	(	RewritePatternSet &	patterns,
		const PipeliningOption &	options )

Populate patterns for SCF software pipelining transformation.

See the ForLoopPipeliningPattern for the transformation details.

Definition at line 847 of file LoopPipelining.cpp.

References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.

◆ populateSCFRotateWhileLoopPatterns()

void mlir::scf::populateSCFRotateWhileLoopPatterns ( RewritePatternSet & patterns )

Populate patterns to rotate scf.while ops, constructing do-while loops from while loops.

Definition at line 40 of file RotateWhileLoop.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateSCFStructuralTypeConversions()

void mlir::scf::populateSCFStructuralTypeConversions	(	const TypeConverter &	typeConverter,
		RewritePatternSet &	patterns,
		PatternBenefit	benefit = 1 )

Similar to populateSCFStructuralTypeConversionsAndLegality but does not populate the conversion target.

Definition at line 243 of file StructuralTypeConversions.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

Referenced by populateSCFStructuralTypeConversionsAndLegality().

◆ populateSCFStructuralTypeConversionsAndLegality()

void mlir::scf::populateSCFStructuralTypeConversionsAndLegality	(	const TypeConverter &	typeConverter,
		RewritePatternSet &	patterns,
		ConversionTarget &	target,
		PatternBenefit	benefit = 1 )

Populates patterns for SCF structural type conversions and sets up the provided ConversionTarget with the appropriate legality configuration for the ops to get converted properly.

A "structural" type conversion is one where the underlying ops are completely agnostic to the actual types involved and simply need to update their types. An example of this is scf.if – the scf.if op and the corresponding scf.yield ops need to update their types accordingly to the TypeConverter, but otherwise don't care what type conversions are happening.

Definition at line 267 of file StructuralTypeConversions.cpp.

References populateSCFStructuralTypeConversions(), populateSCFStructuralTypeConversionTarget(), and target.

◆ populateSCFStructuralTypeConversionTarget()

void mlir::scf::populateSCFStructuralTypeConversionTarget	(	const TypeConverter &	typeConverter,
		ConversionTarget &	target )

Updates the ConversionTarget with dynamic legality of SCF operations based on the provided type converter.

Definition at line 252 of file StructuralTypeConversions.cpp.

References target.

Referenced by populateSCFStructuralTypeConversionsAndLegality().

◆ populateUpliftWhileToForPatterns()

void mlir::scf::populateUpliftWhileToForPatterns ( RewritePatternSet & patterns )

Populate patterns to uplift scf.while ops to scf.for.

Uplifitng expects a specific ops pattern:

before block consisting of single arith.cmp op
after block containing arith.addi

Definition at line 269 of file UpliftWhileToFor.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ promote()

void mlir::scf::promote	(	RewriterBase &	rewriter,
		scf::ForallOp	forallOp )

Promotes the loop body of a scf::ForallOp to its containing block.

Definition at line 748 of file SCF.cpp.

References mlir::RewriterBase::eraseOp(), mlir::Value::getType(), mlir::RewriterBase::inlineBlockBefore(), mlir::RewriterBase::replaceAllUsesWith(), and mlir::OpBuilder::setInsertionPointAfter().

◆ registerBufferDeallocationOpInterfaceExternalModels()

void mlir::scf::registerBufferDeallocationOpInterfaceExternalModels ( DialectRegistry & registry )

Definition at line 75 of file BufferDeallocationOpInterfaceImpl.cpp.

References mlir::DialectRegistry::addExtension().

Referenced by mlir::registerAllDialects().

◆ registerBufferizableOpInterfaceExternalModels()

void mlir::scf::registerBufferizableOpInterfaceExternalModels ( DialectRegistry & registry )

Definition at line 1359 of file BufferizableOpInterfaceImpl.cpp.

References mlir::DialectRegistry::addExtension().

Referenced by mlir::registerAllDialects().

◆ registerTransformDialectExtension()

void mlir::scf::registerTransformDialectExtension ( DialectRegistry & registry )

Referenced by mlir::registerAllExtensions().

◆ registerValueBoundsOpInterfaceExternalModels()

void mlir::scf::registerValueBoundsOpInterfaceExternalModels ( DialectRegistry & registry )

Definition at line 237 of file ValueBoundsOpInterfaceImpl.cpp.

References mlir::DialectRegistry::addExtension().

Referenced by mlir::registerAllDialects().

◆ replaceAndCastForOpIterArg()

SmallVector< Value > mlir::scf::replaceAndCastForOpIterArg	(	RewriterBase &	rewriter,
		scf::ForOp	forOp,
		OpOperand &	operand,
		Value	replacement,
		const ValueTypeCastFnTy &	castFn )

Definition at line 882 of file SCF.cpp.

References mlir::RewriterBase::eraseOp(), mlir::Block::front(), mlir::IROperand< DerivedT, IRValueT >::get(), mlir::BlockArgument::getArgNumber(), mlir::Block::getArguments(), mlir::OpOperand::getOperandNumber(), mlir::detail::IROperandBase::getOwner(), mlir::Block::getTerminator(), mlir::Value::getType(), mlir::RewriterBase::mergeBlocks(), replacement(), mlir::OpBuilder::setInsertionPoint(), mlir::OpBuilder::setInsertionPointAfter(), and mlir::OpBuilder::setInsertionPointToStart().

◆ rewritePeeledMinMaxOp()

LogicalResult mlir::scf::rewritePeeledMinMaxOp	(	RewriterBase &	rewriter,
		Operation *	op,
		Value	iv,
		Value	ub,
		Value	step,
		bool	insideLoop )

Try to simplify the given affine.min/max operation op after loop peeling.

This function can simplify min/max operations such as (ub is the previous upper bound of the unpeeled loop):

#map = affine_map<(d0)[s0, s1] -> (s0, -d0 + s1)>

%r = affine.min #map(%iv)[%step, %ub]

mlir::affine

Definition AffineToStandard.h:23

and rewrites them into (in the case the peeled loop):

%r = %step

min/max operations inside the partial iteration are rewritten in a similar way.

This function can simplify min/max operations such as (ub is the previous upper bound of the unpeeled loop):

#map = affine_map<(d0)[s0, s1] -> (s0, -d0 + s1)>

%r = affine.min #affine.min #map(%iv)[%step, %ub]

and rewrites them into (in the case the peeled loop):

%r = %step

min/max operations inside the partial iteration are rewritten in a similar way.

This function builds up a set of constraints, capable of proving that:

Inside the peeled loop: min(step, ub - iv) == step
Inside the partial iteration: min(step, ub - iv) == ub - iv

Returns success if the given operation was replaced by a new operation; failure otherwise.

Note: ub is the previous upper bound of the loop (before peeling). insideLoop must be true for min/max ops inside the loop and false for affine.min ops inside the partial iteration. For an explanation of the other parameters, see comment of canonicalizeMinMaxOpInLoop.

Definition at line 197 of file AffineCanonicalizationUtils.cpp.

References mlir::affine::FlatAffineValueConstraints::addBound(), mlir::presburger::IntegerRelation::addInequality(), mlir::FlatLinearValueConstraints::appendDimVar(), mlir::FlatLinearValueConstraints::appendSymbolVar(), canonicalizeMinMaxOp(), and mlir::getConstantIntValue().

Referenced by rewriteAffineOpAfterPeeling().

◆ tileParallelLoop()

std::pair< ParallelOp, ParallelOp > mlir::scf::tileParallelLoop	(	ParallelOp	op,
		llvm::ArrayRef< int64_t >	tileSizes,
		bool	noMinMaxBounds )

Tile a parallel loop of the form scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4, arg5)

into scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4*tileSize[0], arg5*tileSize[1]) scf.parallel (j0, j1) = (0, 0) to (min(tileSize[0], arg2-j0) min(tileSize[1], arg3-j1)) step (arg4, arg5) The old loop is replaced with the new one.

The function returns the resulting ParallelOps, i.e. {outer_loop_op, inner_loop_op}.

◆ upliftWhileToForLoop()

FailureOr< ForOp > mlir::scf::upliftWhileToForLoop	(	RewriterBase &	rewriter,
		WhileOp	loop )

Try to uplift scf.while op to scf.for.

Uplifitng expects a specific ops pattern:

before block consisting of single arith.cmp op
after block containing arith.addi

◆ wrapWhileLoopInZeroTripCheck()

FailureOr< WhileOp > mlir::scf::wrapWhileLoopInZeroTripCheck	(	WhileOp	whileOp,
		RewriterBase &	rewriter,
		bool	forceCreateCheck = false )

Create zero-trip-check around a while op and return the new loop op in the check.

The while loop is rotated to avoid evaluating the condition twice

By default the check won't be created for do-while loop as it is not required. forceCreateCheck can force the creation.

It turns:

scf.while (arg0 = init) : (i32) -> i64 { val = .., arg0 : i64 cond = arith.cmpi .., arg0 : i32 scf.condition(cond) val : i64 } do { ^bb0(arg1: i64): next = .., arg1 : i32 scf.yield next : i32 }

into:

pre_val = .., init : i64 pre_cond = arith.cmpi .., init : i32 scf.if pre_cond -> i64 { res = scf.while (arg1 = va0) : (i64) -> i64 { next = .., arg1 : i32 val = .., next : i64 cond = arith.cmpi .., next : i32 scf.condition(cond) val : i64 } do { ^bb0(arg2: i64): scf.yield arg2 : i32 } scf.yield res : i64 } else { scf.yield pre_val : i64 }

Failure mechanism is not implemented for this function, so it currently always returns a WhileOp operation: a new one if the transformation took place or the input whileOp if the loop was already in a do-while form and forceCreateCheck is false.

Variable Documentation

◆ mergeOps

SmallVector<Operation *> mlir::scf::mergeOps

Computation function returning, for the op currently being tiled or fused, the per-iteration-domain-dimension InnerTileAlignment array.

tileSizes holds the given tile sizes for pure tiling / and is empty for fusion; slices holds the tensor.extract_slice (producer / fusion) or tensor.insert_slice/tensor.parallel_insert_slices (consumer / fusion) being fused, and is empty for pure tiling. Only pack/unpack / implementations consult the result; every other op ignores it. using InnerTileAlignmentFnTy = std::function<SmallVector<InnerTileAlignment>( TilingInterface op, ArrayRef<OpFoldResult> tileSizes, ArrayRef<Operation *> slices)>;

/ Options to use to control tiling. struct SCFTilingOptions { / Specify which loop construct to use for tile and fuse. enum class LoopType { ForOp, ForallOp, CustomOp }; LoopType loopType = LoopType::ForOp; SCFTilingOptions &setLoopType(LoopType type) { loopType = type; return *this; }

/ Computation function that returns the tile sizes to use for each loop. / Returning a tile size of zero implies no tiling for that loop. If the / size of the returned vector is smaller than the number of loops, the inner / loops are not tiled. If the size of the returned vector is larger, then / the vector is truncated to number of loops. SCFTileSizeComputationFunction tileSizeComputationFunction = nullptr;

SCFTilingOptions & setTileSizeComputationFunction(SCFTileSizeComputationFunction fun) { tileSizeComputationFunction = std::move(fun); return *this; } / Convenience function to set the tileSizeComputationFunction to a / function that computes tile sizes at the point they are needed. Allows / proper interaction with folding. SCFTilingOptions &setTileSizes(ArrayRef<OpFoldResult> tileSizes);

/ The interchange vector to reorder the tiled loops. SmallVector<int64_t> interchangeVector = {}; SCFTilingOptions &setInterchange(ArrayRef<int64_t> interchange) { interchangeVector = llvm::to_vector(interchange); return *this; }

/ Optional control function returning, per tiled/fused op, the inner tile / alignment hints. InnerTileAlignmentFnTy innerTileAlignmentFn = nullptr; SCFTilingOptions &setInnerTileAlignmentFn(InnerTileAlignmentFnTy fn) { innerTileAlignmentFn = std::move(fn); return *this; }

/ Sets a constant innerTileAlignmentFn that returns alignments for every / op, ignoring which op is being tiled or fused. Use when a single fixed / array is correct for the whole tiling/fusion (e.g. tiling a single op). SCFTilingOptions & setInnerTileAlignments(ArrayRef<InnerTileAlignment> alignments) { SmallVector<InnerTileAlignment> fixed = llvm::to_vector(alignments); innerTileAlignmentFn = [fixed = std::move(fixed)](TilingInterface, ArrayRef<OpFoldResult>, ArrayRef<Operation *>) { return fixed; }; return *this; }

----------------------------------------------------------------------—// Options related to tiling using scf.forall. ----------------------------------------------------------------------—//

/ Computation function that returns the number of threads to use for / each loop. Returning a num threads of zero implies no tiling for that / loop. If the size of the returned vector is smaller than the number of / loops, the inner loops are not tiled. If the size of the returned vector / is larger, then the vector is truncated to number of loops. Note: This / option is only supported with loopType set to LoopType::ForallOp. If the / tile size function is not specified while the num threads computation is, / then the tile size is determined automatically to map at most one tile per / thread. SCFTileSizeComputationFunction numThreadsComputationFunction = nullptr;

SCFTilingOptions & setNumThreadsComputationFunction(SCFTileSizeComputationFunction fun) { numThreadsComputationFunction = std::move(fun); return *this; } / Convenience function to set the numThreadsComputationFunction to a / function that computes num threads at the point they are needed. SCFTilingOptions &setNumThreads(ArrayRef<OpFoldResult> numThreads);

/ Specify mapping of loops to devices. This is only respected when the loop / constructs support such a mapping (like scf.forall). Will be ignored / when using loop constructs that dont support such a mapping (like / scf.for) SmallVector<Attribute> mappingVector = {}; SCFTilingOptions &setMapping(ArrayRef<Attribute> mapping) { mappingVector = llvm::to_vector(mapping); return *this; }

----------------------------------------------------------------------—// Options related reduction tiling ----------------------------------------------------------------------—//

/ Specify how reduction dimensions should be tiled. ReductionTilingStrategy reductionStrategy = ReductionTilingStrategy::FullReduction; SCFTilingOptions & setReductionTilingStrategy(ReductionTilingStrategy strategy) { reductionStrategy = strategy; return *this; }

/ Specify the reduction dimensions to be tiled. Note that this needs to be / specified. If left unspecified, then none of the reduction dimensions are / tiled. SetVector<unsigned> reductionDims; SCFTilingOptions &setReductionDims(ArrayRef<unsigned> dims) { reductionDims.clear(); reductionDims.insert(dims.begin(), dims.end()); return *this; }

----------------------------------------------------------------------—// Options related to tiling using custom loop. ----------------------------------------------------------------------—//

For generating the inter-tile loops using a custom loop, two callback functions are needed

That generates the "loop header", i.e. the loop that iterates over the different tiles.
That generates the loop terminator

For scf.forall case the call back to generate loop header would generate

scf.forall (...) = ... {
..
}

and the call back to generate the loop terminator would generate the scf.in_parallel region

scf.forall (...) = ... {
scf.in_parallel {
tensor.parallel_insert_slice ...
}
}

Information that is to be returned by loop header callback needed for the rest of the tiled codegeneration.

loops: The generated loops
tileOffset: The values that represent the offset of the iteration space tile.
tileSizes : The values that represent the size of the iteration space tile.
destinationTensors : The tensors to use as destinations during tiling. struct CustomLoopHeaderInfo { SmallVector<LoopLikeOpInterface> loops; SmallVector<OpFoldResult> tileOffset; SmallVector<OpFoldResult> tileSizes; SmallVector<Value> destinationTensors; };

Type of the callback function that generates the loop headers.

loopRanges : Values that represent the full size of the iteration space being tiled.
givenTileSizes : The tile sizes that are to be used to tile the iteration space.
destinationTensors : The tensors to use as destinations for the results of the tiled loop for loops that implement DestinationStyleOpInterface. Returns the CustomLoopHeaderInfo object (described above). it is expected that this function sets the insertion point of rewriter to the program point where the intra-tile loop computation is to be generated. using GenerateLoopHeaderFn = std::function<FailureOr<CustomLoopHeaderInfo>( RewriterBase &rewriter, Location loc, ArrayRef<Range> loopRanges, ArrayRef<OpFoldResult> givenTileSizes, ValueRange destinationTensors)>;

Type of the callback function that generates the loop terminator.

loops : generated loops from the GenerateLoopHeaderFn callback
tiledResults : Tiles of the result computed for the iteration space tile.
resultOffsets : For each of the tiledResults, the offset at which the result tile is to be "inserted" back into the destination tensor.
resultSizes : For each of the tiledResults, the size of the result tile that is to be "inserted" back into the destination tensor. Returns the CustomLoopHeaderInfo object (described above) using GenerateLoopTerminatorFn = std::function<LogicalResult( RewriterBase &rewriter, Location loc, ArrayRef<LoopLikeOpInterface> loops, ValueRange tiledResults, ArrayRef<SmallVector<OpFoldResult>> resultOffsets, ArrayRef<SmallVector<OpFoldResult>> resultSizes, ValueRange destinationTensors)>;

Callback function to generate the inter-tile loop header. GenerateLoopHeaderFn generateLoopHeaderFn = nullptr; Callback function to generate the inter-tile loop terminator. GenerateLoopTerminatorFn generateLoopTerminatorFn = nullptr; Helper function to set the callbacks for inter-tile loop header and terminator functions when using a custom operation for the loop. SCFTilingOptions & setCustomLoopGenerationFns(GenerateLoopHeaderFn headerFn, GenerateLoopTerminatorFn terminatorFn) { generateLoopHeaderFn = std::move(headerFn); generateLoopTerminatorFn = std::move(terminatorFn); return *this; } };

/ Transformation information returned after tiling. struct SCFTilingResult { / Tiled operations that are generated during tiling. The order does not / matter except the last op. The replacements are expected to be the results / of the last op. SmallVector<Operation *> tiledOps; / The initial destination values passed to the tiled operations. SmallVector<Value> initialValues; / The scf.for operations that iterate over the tiles. SmallVector<LoopLikeOpInterface> loops; / Values to use as replacements for the untiled op. Is the same size as the / number of results of the untiled op. SmallVector<Value> replacements; / Slices generated after tiling that can be used for fusing with the tiled / producer. SmallVector<Operation *> generatedSlices; / In cases where there as an additional merge step after tiling / return the merged ops after tiling. This list is empty when reduction / tiling strategy is / `scf::SCFTilingOptions::ReductionTilingStrategy::FullReduction.

Definition at line 269 of file TileUsingInterface.h.

Classes

Typedefs

Functions

Variables

Typedef Documentation

◆ LoopMatcherFn

◆ LoopVector

◆ SCFTileSizeComputationFunction

◆ ValueTypeCastFnTy

◆ ValueVector

Function Documentation

◆ addLoopRangeConstraints()

◆ buildLoopNest() [1/2]

◆ buildLoopNest() [2/2]

◆ buildTerminatedBody()

◆ canonicalizeMinMaxOpInLoop()

◆ computeUbMinusLb()

◆ forallToForLoop()

◆ forallToParallelLoop()

◆ getForallOpThreadIndexOwner()

◆ getForInductionVarOwner()

◆ getParallelForInductionVarOwner()

◆ insideMutuallyExclusiveBranches()

◆ matchForLikeLoop()

◆ naivelyFuseParallelOps()

◆ parallelForToNestedFors()

◆ peelForLoopAndSimplifyBounds()

◆ peelForLoopFirstIteration()

◆ pipelineForLoop()

◆ populateSCFForLoopCanonicalizationPatterns()

◆ populateSCFLoopPipeliningPatterns()

◆ populateSCFRotateWhileLoopPatterns()

◆ populateSCFStructuralTypeConversions()

◆ populateSCFStructuralTypeConversionsAndLegality()

◆ populateSCFStructuralTypeConversionTarget()

◆ populateUpliftWhileToForPatterns()

◆ promote()

◆ registerBufferDeallocationOpInterfaceExternalModels()

◆ registerBufferizableOpInterfaceExternalModels()

◆ registerTransformDialectExtension()

◆ registerValueBoundsOpInterfaceExternalModels()

◆ replaceAndCastForOpIterArg()

◆ rewritePeeledMinMaxOp()

◆ tileParallelLoop()

◆ upliftWhileToForLoop()

◆ wrapWhileLoopInZeroTripCheck()

Variable Documentation

◆ mergeOps