MLIR
20.0.0git
|
Classes | |
struct | LoopNest |
class | ForLoopPipeliningPattern |
struct | SCFTilingOptions |
Options to use to control tiling. More... | |
struct | SCFTilingResult |
Transformation information returned after tiling. More... | |
struct | SCFTileAndFuseOptions |
Options used to control tile + fuse. More... | |
struct | SCFFuseProducerOfSliceResult |
Fuse the producer of the source of candidateSliceOp by computing the required slice of the producer in-place. More... | |
struct | SCFTileAndFuseResult |
Transformation information returned after tile and fuse. More... | |
struct | SCFFuseConsumerOfSliceResult |
Fuse the consumer of the source of candidateSliceOp by computing the required slice of the consumer in-place. More... | |
struct | SCFReductionTilingResult |
Transformation information returned after reduction tiling. More... | |
struct | PipeliningOption |
Options to dictate how loops should be pipelined. More... | |
Typedefs | |
using | ValueVector = SmallVector< Value > |
An owning vector of values, handy to return from functions. More... | |
using | LoopVector = SmallVector< scf::ForOp > |
using | ValueTypeCastFnTy = llvm::function_ref< Value(OpBuilder &, Location loc, Type, Value)> |
Perform a replacement of one iter OpOperand of an scf.for to the replacement value with a different type. More... | |
using | SCFTileSizeComputationFunction = std::function< SmallVector< OpFoldResult >(OpBuilder &, Operation *)> |
using | LoopMatcherFn = function_ref< LogicalResult(Value, OpFoldResult &, OpFoldResult &, OpFoldResult &)> |
Match "for loop"-like operations: If the first parameter is an iteration variable, return lower/upper bounds via the second/third parameter and the step size via the last parameter. More... | |
Functions | |
void | buildTerminatedBody (OpBuilder &builder, Location loc) |
Default callback for IfOp builders. Inserts a yield without arguments. More... | |
void | ensureLoopTerminator (Region ®ion, Builder &builder, Location loc) |
ForOp | getForInductionVarOwner (Value val) |
Returns the loop parent of an induction variable. More... | |
ParallelOp | getParallelForInductionVarOwner (Value val) |
Returns the parallel loop parent of an induction variable. More... | |
ForallOp | getForallOpThreadIndexOwner (Value val) |
Returns the ForallOp parent of an thread index variable. More... | |
bool | insideMutuallyExclusiveBranches (Operation *a, Operation *b) |
Return true if ops a and b (or their ancestors) are in mutually exclusive regions/blocks of an IfOp. More... | |
void | promote (RewriterBase &rewriter, scf::ForallOp forallOp) |
Promotes the loop body of a scf::ForallOp to its containing block. More... | |
LoopNest | buildLoopNest (OpBuilder &builder, Location loc, ValueRange lbs, ValueRange ubs, ValueRange steps, ValueRange iterArgs, function_ref< ValueVector(OpBuilder &, Location, ValueRange, ValueRange)> bodyBuilder=nullptr) |
Creates a perfect nest of "for" loops, i.e. More... | |
LoopNest | buildLoopNest (OpBuilder &builder, Location loc, ValueRange lbs, ValueRange ubs, ValueRange steps, function_ref< void(OpBuilder &, Location, ValueRange)> bodyBuilder=nullptr) |
A convenience version for building loop nests without iteration arguments (like for reductions). More... | |
SmallVector< Value > | replaceAndCastForOpIterArg (RewriterBase &rewriter, scf::ForOp forOp, OpOperand &operand, Value replacement, const ValueTypeCastFnTy &castFn) |
void | registerValueBoundsOpInterfaceExternalModels (DialectRegistry ®istry) |
void | registerTransformDialectExtension (DialectRegistry ®istry) |
void | registerBufferDeallocationOpInterfaceExternalModels (DialectRegistry ®istry) |
void | registerBufferizableOpInterfaceExternalModels (DialectRegistry ®istry) |
void | populateSCFStructuralTypeConversionsAndLegality (const TypeConverter &typeConverter, RewritePatternSet &patterns, ConversionTarget &target) |
Populates patterns for SCF structural type conversions and sets up the provided ConversionTarget with the appropriate legality configuration for the ops to get converted properly. More... | |
void | populateSCFStructuralTypeConversions (const TypeConverter &typeConverter, RewritePatternSet &patterns) |
Similar to populateSCFStructuralTypeConversionsAndLegality but does not populate the conversion target. More... | |
void | populateSCFStructuralTypeConversionTarget (const TypeConverter &typeConverter, ConversionTarget &target) |
Updates the ConversionTarget with dynamic legality of SCF operations based on the provided type converter. More... | |
void | populateSCFStructuralOneToNTypeConversions (const TypeConverter &typeConverter, RewritePatternSet &patterns) |
Populates the provided pattern set with patterns that do 1:N type conversions on (some) SCF ops. More... | |
void | populateSCFLoopPipeliningPatterns (RewritePatternSet &patterns, const PipeliningOption &options) |
Populate patterns for SCF software pipelining transformation. More... | |
void | populateSCFForLoopCanonicalizationPatterns (RewritePatternSet &patterns) |
Populate patterns for canonicalizing operations inside SCF loop bodies. More... | |
void | populateUpliftWhileToForPatterns (RewritePatternSet &patterns) |
Populate patterns to uplift scf.while ops to scf.for . More... | |
void | populateSCFRotateWhileLoopPatterns (RewritePatternSet &patterns) |
Populate patterns to rotate scf.while ops, constructing do-while loops from while loops. More... | |
FailureOr< SCFTilingResult > | tileUsingSCF (RewriterBase &rewriter, TilingInterface op, const SCFTilingOptions &options) |
Method to tile an op that implements the TilingInterface using scf.for for iterating over the tiles. More... | |
std::optional< SCFFuseProducerOfSliceResult > | tileAndFuseProducerOfSlice (RewriterBase &rewriter, tensor::ExtractSliceOp candidateSliceOp, MutableArrayRef< LoopLikeOpInterface > loops) |
Implementation of fusing producer of a single slice by computing the slice of the producer in-place. More... | |
FailureOr< SmallVector< Operation * > > | yieldReplacementForFusedProducer (RewriterBase &rewriter, tensor::ExtractSliceOp sliceOp, scf::SCFFuseProducerOfSliceResult fusedProducerInfo, MutableArrayRef< LoopLikeOpInterface > loops, ArrayRef< unsigned > yieldResultNumber=ArrayRef< unsigned >{}) |
Reconstruct the fused producer from within the tiled-and-fused code. More... | |
FailureOr< SCFTileAndFuseResult > | tileConsumerAndFuseProducersUsingSCF (RewriterBase &rewriter, TilingInterface consumer, const SCFTileAndFuseOptions &options) |
Method to tile and fuse a sequence of operations, by tiling the consumer and fusing its producers. More... | |
FailureOr< scf::SCFFuseConsumerOfSliceResult > | tileAndFuseConsumerOfSlice (RewriterBase &rewriter, Operation *candidateSliceOp) |
Implementation of fusing consumer of a single slice by computing the slice of the consumer in-place for scf loop. More... | |
FailureOr< SmallVector< scf::ForOp > > | lowerToLoopsUsingSCFForOp (RewriterBase &rewriter, TilingInterface op) |
Method to lower an op that implements the TilingInterface to loops/scalars. More... | |
FailureOr< scf::SCFReductionTilingResult > | tileReductionUsingScf (RewriterBase &b, PartialReductionOpInterface op, ArrayRef< OpFoldResult > tileSize) |
Method to tile a reduction and generate a parallel op within a serial loop. More... | |
LogicalResult | forallToForLoop (RewriterBase &rewriter, ForallOp forallOp, SmallVectorImpl< Operation * > *results=nullptr) |
Try converting scf.forall into a set of nested scf.for loops. More... | |
LogicalResult | forallToParallelLoop (RewriterBase &rewriter, ForallOp forallOp, ParallelOp *result=nullptr) |
Try converting scf.forall into an scf.parallel loop. More... | |
void | naivelyFuseParallelOps (Region ®ion, llvm::function_ref< bool(Value, Value)> mayAlias) |
Fuses all adjacent scf.parallel operations with identical bounds and step into one scf.parallel operations. More... | |
LogicalResult | peelForLoopAndSimplifyBounds (RewriterBase &rewriter, ForOp forOp, scf::ForOp &partialIteration) |
Rewrite a for loop with bounds/step that potentially do not divide evenly into a for loop where the step divides the iteration space evenly, followed by another scf.for for the last (partial) iteration (if any; returned via partialIteration ). More... | |
LogicalResult | peelForLoopFirstIteration (RewriterBase &rewriter, ForOp forOp, scf::ForOp &partialIteration) |
Peel the first iteration out of the scf.for loop. More... | |
std::pair< ParallelOp, ParallelOp > | tileParallelLoop (ParallelOp op, llvm::ArrayRef< int64_t > tileSizes, bool noMinMaxBounds) |
Tile a parallel loop of the form scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4, arg5) More... | |
FailureOr< ForOp > | pipelineForLoop (RewriterBase &rewriter, ForOp forOp, const PipeliningOption &options, bool *modifiedIR=nullptr) |
Generate a pipelined version of the scf.for loop based on the schedule given as option. More... | |
FailureOr< WhileOp > | wrapWhileLoopInZeroTripCheck (WhileOp whileOp, RewriterBase &rewriter, bool forceCreateCheck=false) |
Create zero-trip-check around a while op and return the new loop op in the check. More... | |
FailureOr< ForOp > | upliftWhileToForLoop (RewriterBase &rewriter, WhileOp loop) |
Try to uplift scf.while op to scf.for . More... | |
LogicalResult | matchForLikeLoop (Value iv, OpFoldResult &lb, OpFoldResult &ub, OpFoldResult &step) |
Match "for loop"-like operations from the SCF dialect. More... | |
LogicalResult | addLoopRangeConstraints (affine::FlatAffineValueConstraints &cstr, Value iv, OpFoldResult lb, OpFoldResult ub, OpFoldResult step) |
Populate the given constraint set with induction variable constraints of a "for" loop with the given range and step. More... | |
LogicalResult | canonicalizeMinMaxOpInLoop (RewriterBase &rewriter, Operation *op, LoopMatcherFn loopMatcher) |
Try to canonicalize the given affine.min/max operation in the context of for loops with a known range. More... | |
LogicalResult | rewritePeeledMinMaxOp (RewriterBase &rewriter, Operation *op, Value iv, Value ub, Value step, bool insideLoop) |
Try to simplify the given affine.min/max operation op after loop peeling. More... | |
using mlir::scf::LoopMatcherFn = typedef function_ref<LogicalResult( Value, OpFoldResult &, OpFoldResult &, OpFoldResult &)> |
Match "for loop"-like operations: If the first parameter is an iteration variable, return lower/upper bounds via the second/third parameter and the step size via the last parameter.
The function should return success
in that case. If the first parameter is not an iteration variable, return failure
.
Definition at line 39 of file AffineCanonicalizationUtils.h.
using mlir::scf::LoopVector = typedef SmallVector<scf::ForOp> |
using mlir::scf::SCFTileSizeComputationFunction = typedef std::function<SmallVector<OpFoldResult>(OpBuilder &, Operation *)> |
Definition at line 31 of file TileUsingInterface.h.
using mlir::scf::ValueTypeCastFnTy = typedef llvm::function_ref<Value(OpBuilder &, Location loc, Type, Value)> |
using mlir::scf::ValueVector = typedef SmallVector<Value> |
LogicalResult mlir::scf::addLoopRangeConstraints | ( | affine::FlatAffineValueConstraints & | cstr, |
Value | iv, | ||
OpFoldResult | lb, | ||
OpFoldResult | ub, | ||
OpFoldResult | step | ||
) |
Populate the given constraint set with induction variable constraints of a "for" loop with the given range and step.
Definition at line 79 of file AffineCanonicalizationUtils.cpp.
References mlir::affine::FlatAffineValueConstraints::addBound(), mlir::presburger::IntegerRelation::addInequality(), mlir::FlatLinearValueConstraints::appendDimVar(), mlir::FlatLinearValueConstraints::appendSymbolVar(), mlir::AffineMap::get(), mlir::Builder::getAffineConstantExpr(), mlir::Builder::getAffineSymbolExpr(), mlir::getConstantIntValue(), mlir::Value::getContext(), mlir::presburger::IntegerRelation::getNumCols(), mlir::presburger::IntegerRelation::getNumDimVars(), and mlir::presburger::IntegerRelation::getNumSymbolVars().
LoopNest mlir::scf::buildLoopNest | ( | OpBuilder & | builder, |
Location | loc, | ||
ValueRange | lbs, | ||
ValueRange | ubs, | ||
ValueRange | steps, | ||
function_ref< void(OpBuilder &, Location, ValueRange)> | bodyBuilder = nullptr |
||
) |
A convenience version for building loop nests without iteration arguments (like for reductions).
Does not take the initial value of reductions or expect the body building functions to return their current value. The built nested scf::For are captured in capturedLoops
when non-null.
Definition at line 760 of file SCF.cpp.
References buildLoopNest().
LoopNest mlir::scf::buildLoopNest | ( | OpBuilder & | builder, |
Location | loc, | ||
ValueRange | lbs, | ||
ValueRange | ubs, | ||
ValueRange | steps, | ||
ValueRange | iterArgs, | ||
function_ref< ValueVector(OpBuilder &, Location, ValueRange, ValueRange)> | bodyBuilder = nullptr |
||
) |
Creates a perfect nest of "for" loops, i.e.
all loops but the innermost contain only another loop and a terminator. The lower, upper bounds and steps are provided as lbs
, ubs
and steps
, which are expected to be of the same size. iterArgs
points to the initial values of the loop iteration arguments, which will be forwarded through the nest to the innermost loop. The body of the loop is populated using bodyBuilder
, which accepts an ordered list of induction variables of all loops, followed by a list of iteration arguments of the innermost loop, in the same order as provided to iterArgs
. This function is expected to return as many values as iterArgs
, of the same type and in the same order, that will be treated as yielded from the loop body and forwarded back through the loop nest. If the function is not provided, the loop nest is not expected to have iteration arguments, the body of the innermost loop will be left empty, containing only the zero-operand terminator. Returns the LoopNest containing the list of perfectly nest scf::ForOp build during the call. If bound arrays are empty, the body builder will be called once to construct the IR outside of the loop with an empty list of induction variables.
Definition at line 687 of file SCF.cpp.
References copy(), mlir::OpBuilder::create(), mlir::OpBuilder::setInsertionPointToEnd(), and mlir::OpBuilder::setInsertionPointToStart().
Referenced by buildLoopNest(), mlir::linalg::GenerateLoopNest< LoopTy >::doit(), mlir::linalg::generateParallelLoopNest(), and insertCopyLoops().
Default callback for IfOp builders. Inserts a yield without arguments.
Definition at line 85 of file SCF.cpp.
References mlir::OpBuilder::create().
LogicalResult mlir::scf::canonicalizeMinMaxOpInLoop | ( | RewriterBase & | rewriter, |
Operation * | op, | ||
LoopMatcherFn | loopMatcher | ||
) |
Try to canonicalize the given affine.min/max operation in the context of for loops
with a known range.
Canonicalize min/max operations in the context of for loops with a known range.
loopMatcher
is used to retrieve loop bounds and the step size for a given iteration variable.
Note: loopMatcher
allows this function to be used with any "for loop"-like operation (scf.for, scf.parallel and even ops defined in other dialects).
Call canonicalizeMinMaxOp
and add the following constraints to the constraint system (along with the missing dimensions):
Note: Due to limitations of IntegerPolyhedron, only constant step sizes are currently supported.
Definition at line 147 of file AffineCanonicalizationUtils.cpp.
LogicalResult mlir::scf::forallToForLoop | ( | RewriterBase & | rewriter, |
ForallOp | forallOp, | ||
SmallVectorImpl< Operation * > * | results = nullptr |
||
) |
Try converting scf.forall into a set of nested scf.for loops.
The newly created scf.for ops will be returned through the results
vector if provided.
LogicalResult mlir::scf::forallToParallelLoop | ( | RewriterBase & | rewriter, |
ForallOp | forallOp, | ||
ParallelOp * | result = nullptr |
||
) |
Try converting scf.forall into an scf.parallel loop.
The conversion is only supported for forall operations with no results.
ForallOp mlir::scf::getForallOpThreadIndexOwner | ( | Value | val | ) |
Returns the ForallOp parent of an thread index variable.
If the provided value is not a thread index variable, then return nullptr.
Definition at line 1451 of file SCF.cpp.
Referenced by matchForLikeLoop().
ForOp mlir::scf::getForInductionVarOwner | ( | Value | val | ) |
Returns the loop parent of an induction variable.
If the provided value is not an induction variable, then return nullptr.
Definition at line 597 of file SCF.cpp.
Referenced by matchForLikeLoop().
ParallelOp mlir::scf::getParallelForInductionVarOwner | ( | Value | val | ) |
Returns the parallel loop parent of an induction variable.
If the provided value is not an induction variable, then return nullptr.
Definition at line 3050 of file SCF.cpp.
Referenced by matchForLikeLoop().
Return true if ops a and b (or their ancestors) are in mutually exclusive regions/blocks of an IfOp.
Definition at line 1977 of file SCF.cpp.
References mlir::Operation::getParentOfType().
FailureOr< SmallVector< scf::ForOp > > mlir::scf::lowerToLoopsUsingSCFForOp | ( | RewriterBase & | rewriter, |
TilingInterface | op | ||
) |
Method to lower an op
that implements the TilingInterface
to loops/scalars.
Definition at line 1972 of file TileUsingInterface.cpp.
LogicalResult mlir::scf::matchForLikeLoop | ( | Value | iv, |
OpFoldResult & | lb, | ||
OpFoldResult & | ub, | ||
OpFoldResult & | step | ||
) |
Match "for loop"-like operations from the SCF dialect.
Definition at line 33 of file AffineCanonicalizationUtils.cpp.
References getForallOpThreadIndexOwner(), getForInductionVarOwner(), and getParallelForInductionVarOwner().
void mlir::scf::naivelyFuseParallelOps | ( | Region & | region, |
llvm::function_ref< bool(Value, Value)> | mayAlias | ||
) |
Fuses all adjacent scf.parallel operations with identical bounds and step into one scf.parallel operations.
Uses a naive aliasing and dependency analysis. User can additionally customize alias checking with mayAlias
hook. mayAlias
must return false if 2 values are guaranteed to not alias.
Definition at line 234 of file ParallelLoopFusion.cpp.
LogicalResult mlir::scf::peelForLoopAndSimplifyBounds | ( | RewriterBase & | rewriter, |
ForOp | forOp, | ||
scf::ForOp & | partialIteration | ||
) |
Rewrite a for loop with bounds/step that potentially do not divide evenly into a for loop where the step divides the iteration space evenly, followed by another scf.for for the last (partial) iteration (if any; returned via partialIteration
).
This transformation is called "loop peeling".
This transformation is beneficial for a wide range of transformations such as vectorization or loop tiling: It enables additional canonicalizations inside the peeled loop body such as rewriting masked loads into unmaked loads.
E.g., assuming a lower bound of 0 (for illustration purposes):
is rewritten into the following pseudo IR:
After loop peeling, this function tries to simplify affine.min and affine.max ops in the body of the peeled loop and in the body of the partial iteration loop, taking advantage of the fact that the peeled loop has only "full" iterations. This simplification is expected to enable further canonicalization opportunities through other patterns.
The return value indicates whether the loop was rewritten or not. Loops are not rewritten if:
Note: This function rewrites the given scf.for loop in-place and creates a new scf.for operation for the last iteration. It replaces all uses of the unpeeled loop with the results of the newly generated scf.for.
LogicalResult mlir::scf::peelForLoopFirstIteration | ( | RewriterBase & | rewriter, |
ForOp | forOp, | ||
scf::ForOp & | partialIteration | ||
) |
Peel the first iteration out of the scf.for loop.
If there is only one iteration, return the original loop.
FailureOr< ForOp > mlir::scf::pipelineForLoop | ( | RewriterBase & | rewriter, |
ForOp | forOp, | ||
const PipeliningOption & | options, | ||
bool * | modifiedIR = nullptr |
||
) |
Generate a pipelined version of the scf.for loop based on the schedule given as option.
This applies the mechanical transformation of changing the loop and generating the prologue/epilogue for the pipelining and doesn't make any decision regarding the schedule. Based on the options the loop is split into several stages. The transformation assumes that the scheduling given by user is valid. For example if we break a loop into 3 stages named S0, S1, S2 we would generate the following code with the number in parenthesis as the iteration index:
S0(0) // Prologue S0(1) S1(0) // Prologue scf.for I = C0 to N - 2 { S0(I+2) S1(I+1) S2(I) // Pipelined kernel } S1(N) S2(N-1) // Epilogue S2(N) // Epilogue
If modifiedIR
is provided, it will be set to a value that indicates whether pipelining modified the IR before failing, signaling to the caller whether they can proceed with different transformations.
Definition at line 783 of file LoopPipelining.cpp.
References mlir::RewriterBase::eraseOp(), options, mlir::RewriterBase::replaceOp(), and mlir::OpBuilder::setInsertionPointAfter().
Referenced by mlir::scf::ForLoopPipeliningPattern::returningMatchAndRewrite().
void mlir::scf::populateSCFForLoopCanonicalizationPatterns | ( | RewritePatternSet & | patterns | ) |
Populate patterns for canonicalizing operations inside SCF loop bodies.
At the moment, only affine.min/max computations with iteration variables, loop bounds and loop steps are canonicalized.
Definition at line 176 of file LoopCanonicalization.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::scf::populateSCFLoopPipeliningPatterns | ( | RewritePatternSet & | patterns, |
const PipeliningOption & | options | ||
) |
Populate patterns for SCF software pipelining transformation.
See the ForLoopPipeliningPattern for the transformation details.
Definition at line 836 of file LoopPipelining.cpp.
References mlir::RewritePatternSet::add(), mlir::RewritePatternSet::getContext(), and options.
void mlir::scf::populateSCFRotateWhileLoopPatterns | ( | RewritePatternSet & | patterns | ) |
Populate patterns to rotate scf.while
ops, constructing do-while
loops from while
loops.
Definition at line 40 of file RotateWhileLoop.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::scf::populateSCFStructuralOneToNTypeConversions | ( | const TypeConverter & | typeConverter, |
RewritePatternSet & | patterns | ||
) |
Populates the provided pattern set with patterns that do 1:N type conversions on (some) SCF ops.
This is intended to be used with applyPartialOneToNConversion.
Definition at line 201 of file OneToNTypeConversion.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::scf::populateSCFStructuralTypeConversions | ( | const TypeConverter & | typeConverter, |
RewritePatternSet & | patterns | ||
) |
Similar to populateSCFStructuralTypeConversionsAndLegality
but does not populate the conversion target.
Definition at line 250 of file StructuralTypeConversions.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
Referenced by populateSCFStructuralTypeConversionsAndLegality().
void mlir::scf::populateSCFStructuralTypeConversionsAndLegality | ( | const TypeConverter & | typeConverter, |
RewritePatternSet & | patterns, | ||
ConversionTarget & | target | ||
) |
Populates patterns for SCF structural type conversions and sets up the provided ConversionTarget with the appropriate legality configuration for the ops to get converted properly.
A "structural" type conversion is one where the underlying ops are completely agnostic to the actual types involved and simply need to update their types. An example of this is scf.if – the scf.if op and the corresponding scf.yield ops need to update their types accordingly to the TypeConverter, but otherwise don't care what type conversions are happening.
Definition at line 273 of file StructuralTypeConversions.cpp.
References populateSCFStructuralTypeConversions(), and populateSCFStructuralTypeConversionTarget().
void mlir::scf::populateSCFStructuralTypeConversionTarget | ( | const TypeConverter & | typeConverter, |
ConversionTarget & | target | ||
) |
Updates the ConversionTarget with dynamic legality of SCF operations based on the provided type converter.
Definition at line 257 of file StructuralTypeConversions.cpp.
References mlir::ConversionTarget::addDynamicallyLegalOp().
Referenced by populateSCFStructuralTypeConversionsAndLegality().
void mlir::scf::populateUpliftWhileToForPatterns | ( | RewritePatternSet & | patterns | ) |
Populate patterns to uplift scf.while
ops to scf.for
.
Uplifitng expects a specific ops pattern:
before
block consisting of single arith.cmp opafter
block containing arith.addi Definition at line 212 of file UpliftWhileToFor.cpp.
References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().
void mlir::scf::promote | ( | RewriterBase & | rewriter, |
scf::ForallOp | forallOp | ||
) |
Promotes the loop body of a scf::ForallOp to its containing block.
Definition at line 644 of file SCF.cpp.
References mlir::OpBuilder::create(), mlir::RewriterBase::eraseOp(), mlir::Value::getType(), mlir::RewriterBase::inlineBlockBefore(), mlir::RewriterBase::replaceAllUsesWith(), and mlir::OpBuilder::setInsertionPointAfter().
void mlir::scf::registerBufferDeallocationOpInterfaceExternalModels | ( | DialectRegistry & | registry | ) |
Definition at line 76 of file BufferDeallocationOpInterfaceImpl.cpp.
References mlir::DialectRegistry::addExtension().
Referenced by mlir::registerAllDialects().
void mlir::scf::registerBufferizableOpInterfaceExternalModels | ( | DialectRegistry & | registry | ) |
Definition at line 1355 of file BufferizableOpInterfaceImpl.cpp.
References mlir::DialectRegistry::addExtension().
Referenced by mlir::registerAllDialects().
void mlir::scf::registerTransformDialectExtension | ( | DialectRegistry & | registry | ) |
Referenced by mlir::registerAllExtensions().
void mlir::scf::registerValueBoundsOpInterfaceExternalModels | ( | DialectRegistry & | registry | ) |
Definition at line 160 of file ValueBoundsOpInterfaceImpl.cpp.
References mlir::DialectRegistry::addExtension().
Referenced by mlir::registerAllDialects().
SmallVector< Value > mlir::scf::replaceAndCastForOpIterArg | ( | RewriterBase & | rewriter, |
scf::ForOp | forOp, | ||
OpOperand & | operand, | ||
Value | replacement, | ||
const ValueTypeCastFnTy & | castFn | ||
) |
Definition at line 776 of file SCF.cpp.
References mlir::Block::begin(), mlir::OpBuilder::create(), mlir::RewriterBase::eraseOp(), mlir::Block::front(), mlir::IROperand< DerivedT, IRValueT >::get(), mlir::BlockArgument::getArgNumber(), mlir::Block::getArguments(), mlir::OpOperand::getOperandNumber(), mlir::detail::IROperandBase::getOwner(), mlir::Block::getTerminator(), mlir::Value::getType(), mlir::RewriterBase::mergeBlocks(), mlir::Operation::setAttrs(), mlir::OpBuilder::setInsertionPoint(), and mlir::OpBuilder::setInsertionPointAfter().
LogicalResult mlir::scf::rewritePeeledMinMaxOp | ( | RewriterBase & | rewriter, |
Operation * | op, | ||
Value | iv, | ||
Value | ub, | ||
Value | step, | ||
bool | insideLoop | ||
) |
Try to simplify the given affine.min/max operation op
after loop peeling.
This function can simplify min/max operations such as (ub is the previous upper bound of the unpeeled loop):
and rewrites them into (in the case the peeled loop):
min/max operations inside the partial iteration are rewritten in a similar way.
This function can simplify min/max operations such as (ub is the previous upper bound of the unpeeled loop):
and rewrites them into (in the case the peeled loop):
min/max operations inside the partial iteration are rewritten in a similar way.
This function builds up a set of constraints, capable of proving that:
Returns success
if the given operation was replaced by a new operation; failure
otherwise.
Note: ub
is the previous upper bound of the loop (before peeling). insideLoop
must be true for min/max ops inside the loop and false for affine.min ops inside the partial iteration. For an explanation of the other parameters, see comment of canonicalizeMinMaxOpInLoop
.
Definition at line 199 of file AffineCanonicalizationUtils.cpp.
Referenced by rewriteAffineOpAfterPeeling().
FailureOr< scf::SCFFuseConsumerOfSliceResult > mlir::scf::tileAndFuseConsumerOfSlice | ( | RewriterBase & | rewriter, |
Operation * | candidateSliceOp | ||
) |
Implementation of fusing consumer of a single slice by computing the slice of the consumer in-place for scf loop.
Definition at line 1743 of file TileUsingInterface.cpp.
References addInitOperandsToLoopNest(), checkAssumptionForLoop(), mlir::OpBuilder::clone(), mlir::OpBuilder::create(), mlir::detail::enumerate(), mlir::RewriterBase::eraseOp(), mlir::IROperand< DerivedT, IRValueT >::get(), mlir::Builder::getIndexAttr(), mlir::OpBuilder::getInsertionPoint(), mlir::OpOperand::getOperandNumber(), mlir::detail::IROperandBase::getOwner(), mlir::Operation::getParentOfType(), getPerfectlyNestedLoopsOutsideOf(), mlir::Operation::getResults(), mlir::Block::getTerminator(), getUntiledConsumerFromSlice(), mlir::isConstantIntValue(), mlir::RewriterBase::modifyOpInPlace(), mlir::RewriterBase::moveOpBefore(), mlir::RewriterBase::notifyMatchFailure(), mlir::RewriterBase::replaceAllUsesWith(), mlir::tensor::replaceInsertSliceWithTiledConsumer(), mlir::IROperand< DerivedT, IRValueT >::set(), and mlir::OpBuilder::setInsertionPoint().
std::optional< scf::SCFFuseProducerOfSliceResult > mlir::scf::tileAndFuseProducerOfSlice | ( | RewriterBase & | rewriter, |
tensor::ExtractSliceOp | candidateSliceOp, | ||
MutableArrayRef< LoopLikeOpInterface > | loops | ||
) |
Implementation of fusing producer of a single slice by computing the slice of the producer in-place.
Definition at line 1079 of file TileUsingInterface.cpp.
References mlir::clone(), cloneOpAndUpdateDestinationArgs(), mlir::RewriterBase::eraseOp(), mlir::Operation::getLoc(), mlir::tensor::getOrCreateDestinations(), mlir::Operation::getResult(), getUntiledProducerFromSliceSource(), mlir::RewriterBase::replaceAllUsesWith(), mlir::tensor::replaceExtractSliceWithTiledProducer(), mlir::OpBuilder::setInsertionPoint(), and mlir::scf::SCFFuseProducerOfSliceResult::tiledOps.
Referenced by tileConsumerAndFuseProducersUsingSCF().
FailureOr< scf::SCFTileAndFuseResult > mlir::scf::tileConsumerAndFuseProducersUsingSCF | ( | RewriterBase & | rewriter, |
TilingInterface | consumer, | ||
const SCFTileAndFuseOptions & | options | ||
) |
Method to tile and fuse a sequence of operations, by tiling the consumer and fusing its producers.
Implementation of tile consumer and fuse producer greedily.
Note that this assumes that it is valid to tile+fuse the producer into the innermost tiled loop. Its up to the caller to ensure that the tile sizes provided make this fusion valid.
For example, for the following sequence
it is legal to fuse the fill with the matmul only if the matmul is tiled along the parallel dimensions and not the reduction dimension, i.e. the tile size for the reduction dimension should be 0. The resulting fused transformation is
Definition at line 1420 of file TileUsingInterface.cpp.
References mlir::detail::enumerate(), mlir::Operation::getNumResults(), mlir::Operation::getResults(), getUntiledProducerFromSliceSource(), mlir::RewriterBase::notifyMatchFailure(), options, tileAndFuseProducerOfSlice(), tileUsingSCF(), and yieldReplacementForFusedProducer().
std::pair<ParallelOp, ParallelOp> mlir::scf::tileParallelLoop | ( | ParallelOp | op, |
llvm::ArrayRef< int64_t > | tileSizes, | ||
bool | noMinMaxBounds | ||
) |
Tile a parallel loop of the form scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4, arg5)
into scf.parallel (i0, i1) = (arg0, arg1) to (arg2, arg3) step (arg4*tileSize[0], arg5*tileSize[1]) scf.parallel (j0, j1) = (0, 0) to (min(tileSize[0], arg2-j0) min(tileSize[1], arg3-j1)) step (arg4, arg5) The old loop is replaced with the new one.
The function returns the resulting ParallelOps, i.e. {outer_loop_op, inner_loop_op}.
FailureOr< scf::SCFReductionTilingResult > mlir::scf::tileReductionUsingScf | ( | RewriterBase & | b, |
PartialReductionOpInterface | op, | ||
ArrayRef< OpFoldResult > | tileSize | ||
) |
Method to tile a reduction and generate a parallel op within a serial loop.
Each of the partial reductions are calculated in parallel. Then after the loop all the partial reduction are merged into a final reduction. For example for the following sequence
into:
Definition at line 927 of file TileUsingInterface.cpp.
FailureOr< scf::SCFTilingResult > mlir::scf::tileUsingSCF | ( | RewriterBase & | rewriter, |
TilingInterface | op, | ||
const SCFTilingOptions & | options | ||
) |
Method to tile an op that implements the TilingInterface
using scf.for
for iterating over the tiles.
Implementation of tiling transformation of op
that implements the TilingInterface
using scf.for
to iterate over the tiles.
Definition at line 787 of file TileUsingInterface.cpp.
Referenced by tileConsumerAndFuseProducersUsingSCF().
FailureOr<ForOp> mlir::scf::upliftWhileToForLoop | ( | RewriterBase & | rewriter, |
WhileOp | loop | ||
) |
Try to uplift scf.while
op to scf.for
.
Uplifitng expects a specific ops pattern:
before
block consisting of single arith.cmp opafter
block containing arith.addi FailureOr<WhileOp> mlir::scf::wrapWhileLoopInZeroTripCheck | ( | WhileOp | whileOp, |
RewriterBase & | rewriter, | ||
bool | forceCreateCheck = false |
||
) |
Create zero-trip-check around a while
op and return the new loop op in the check.
The while loop is rotated to avoid evaluating the condition twice
By default the check won't be created for do-while loop as it is not required. forceCreateCheck
can force the creation.
It turns:
scf.while (arg0 = init) : (i32) -> i64 { val = .., arg0 : i64 cond = arith.cmpi .., arg0 : i32 scf.condition(cond) val : i64 } do { ^bb0(arg1: i64): next = .., arg1 : i32 scf.yield next : i32 }
into:
pre_val = .., init : i64 pre_cond = arith.cmpi .., init : i32 scf.if pre_cond -> i64 { res = scf.while (arg1 = va0) : (i64) -> i64 { next = .., arg1 : i32 val = .., next : i64 cond = arith.cmpi .., next : i32 scf.condition(cond) val : i64 } do { ^bb0(arg2: i64): scf.yield arg2 : i32 } scf.yield res : i64 } else { scf.yield pre_val : i64 }
Failure mechanism is not implemented for this function, so it currently always returns a WhileOp
operation: a new one if the transformation took place or the input whileOp
if the loop was already in a do-while
form and forceCreateCheck
is false
.
FailureOr< SmallVector< Operation * > > mlir::scf::yieldReplacementForFusedProducer | ( | RewriterBase & | rewriter, |
tensor::ExtractSliceOp | sliceOp, | ||
scf::SCFFuseProducerOfSliceResult | fusedProducerInfo, | ||
MutableArrayRef< LoopLikeOpInterface > | loops, | ||
ArrayRef< unsigned > | yieldResultNumber = ArrayRef<unsigned>{} |
||
) |
Reconstruct the fused producer from within the tiled-and-fused code.
Based on the slice of the producer computed in place it is possible that within the loop nest same slice of the producer is computed multiple times. It is in general not possible to recompute the value of the fused producer from the tiled loop code in such cases. For the cases where no slice of the producer is computed in a redundant fashion it is possible to reconstruct the value of the original producer from within the tiled loop. It is upto the caller to ensure that the producer is not computed redundantly within the tiled loop nest. For example, consider
If %1
is tiled in a 2D fashion and %0
is fused with it, the resulting IR is,
Here t1_2
is the same for all iterations of the inner scf.for
. Instead if %1
were tiled only along the rows, the resultant code would be
Here there is no intersection in the different slices of t2_1
computed across iterations of the scf.for
. In such cases, the value of the original %0
can be reconstructed from within the loop body. This is useful in cases where %0
had other uses as well. If not reconstructed from within the loop body, uses of %0
could not be replaced, making it still live and the fusion immaterial.
The
<tt>yieldResultNumber</tt> | decides which result would be yield. If not given, yield all opResult of fused producer. |
The method returns the list of new slices added during the process (which can be used to fuse along).
Definition at line 1194 of file TileUsingInterface.cpp.
References addInitOperandsToLoopNest(), mlir::OpBuilder::create(), mlir::detail::enumerate(), mlir::Builder::getIndexAttr(), mlir::OpBuilder::getInsertionPoint(), mlir::Operation::getLoc(), mlir::Operation::getNumResults(), mlir::tensor::getOrCreateDestination(), mlir::OpResult::getOwner(), mlir::Operation::getResult(), mlir::OpResult::getResultNumber(), mlir::Block::getTerminator(), mlir::RewriterBase::modifyOpInPlace(), mlir::scf::SCFFuseProducerOfSliceResult::origProducer, mlir::OpBuilder::setInsertionPoint(), and mlir::scf::SCFFuseProducerOfSliceResult::tiledOps.
Referenced by tileConsumerAndFuseProducersUsingSCF().