MLIR  20.0.0git
Namespaces | Classes | Typedefs | Enumerations | Functions
mlir::bufferization Namespace Reference

Namespaces

 deallocation_impl
 
 detail
 
 func_ext
 

Classes

struct  ValueComparator
 Compare two SSA values in a deterministic manner. More...
 
class  Ownership
 This class is used to track the ownership of values. More...
 
struct  DeallocationOptions
 Options for BufferDeallocationOpInterface-based buffer deallocation. More...
 
class  DeallocationState
 This class collects all the state that we need to perform the buffer deallocation pass with associated helper functions such that we have easy access to it in the BufferDeallocationOpInterface implementations and the BufferDeallocation pass. More...
 
struct  AliasingOpOperand
 A maybe aliasing OpOperand. More...
 
struct  AliasingValue
 A maybe aliasing Value. More...
 
class  AliasList
 
class  OpFilter
 
struct  BufferizationOptions
 Options for BufferizableOpInterface-based bufferization. More...
 
struct  TraversalConfig
 Traversal parameters for findValueInReverseUseDefChain. More...
 
class  AnalysisState
 AnalysisState provides a variety of helper functions for dealing with tensor values. More...
 
struct  DstBufferizableOpInterfaceExternalModel
 Bufferizable ops that implement the DestinationStyleOpInterface can use this external model base class. More...
 
struct  OpWithUnstructuredControlFlowBufferizableOpInterfaceExternalModel
 A template that provides a default implementation of getAliasingOpOperands for ops that support unstructured control flow within their regions. More...
 
struct  BranchOpBufferizableOpInterfaceExternalModel
 A template that provides a default implementation of getAliasingValues for ops that implement the BranchOpInterface. More...
 
struct  BufferDeallocationPipelineOptions
 Options for the buffer deallocation pipeline. More...
 
struct  BufferizationStatistics
 Bufferization statistics for debugging. More...
 
class  BufferizeTypeConverter
 A helper type converter class that automatically populates the relevant materializations and type conversions for bufferization. More...
 
class  BufferPlacementAllocs
 A simple analysis that detects allocation operations. More...
 
class  BufferPlacementTransformationBase
 The base class for all BufferPlacement transformations. More...
 
struct  OneShotBufferizationOptions
 Options for analysis-enabled bufferization. More...
 
class  OneShotAnalysisState
 State for analysis-enabled bufferization. More...
 
struct  BufferResultsToOutParamsOpts
 

Typedefs

using AliasingOpOperandList = AliasList< AliasingOpOperand >
 A list of possible aliasing OpOperands. More...
 
using AliasingValueList = AliasList< AliasingValue >
 A list of possible aliasing Values. More...
 
using RegisterDependenciesFn = std::function< void(ValueRange, ValueRange)>
 
using DeallocHelperMap = llvm::DenseMap< Operation *, func::FuncOp >
 Maps from symbol table to its corresponding dealloc helper function. More...
 

Enumerations

enum class  BufferRelation { Unknown , Equivalent }
 Specifies a fine-grain relationship between buffers to enable more analysis. More...
 

Functions

FailureOr< ValueallocateTensorForShapedValue (OpBuilder &b, Location loc, Value shapedValue, const BufferizationOptions &options, bool copy=true)
 Create an AllocTensorOp for the given shaped value (memref or tensor). More...
 
FailureOr< ValuegetBuffer (RewriterBase &rewriter, Value value, const BufferizationOptions &options)
 Lookup the buffer for the given value. More...
 
FailureOr< BaseMemRefTypegetBufferType (Value value, const BufferizationOptions &options)
 Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR. More...
 
FailureOr< BaseMemRefTypegetBufferType (Value value, const BufferizationOptions &options, SmallVector< Value > &invocationStack)
 Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR. More...
 
bool hasTensorSemantics (Operation *op)
 Return "true" if the given op has tensor semantics and should be bufferized. More...
 
void replaceOpWithBufferizedValues (RewriterBase &rewriter, Operation *op, ValueRange values)
 Replace an op with replacement values. More...
 
template<typename OpTy , typename... Args>
OpTy replaceOpWithNewBufferizedOp (RewriterBase &rewriter, Operation *op, Args &&...args)
 Replace an op with a new op. More...
 
BaseMemRefType getMemRefType (Value value, const BufferizationOptions &options, MemRefLayoutAttrInterface layout={}, Attribute memorySpace=nullptr)
 Return a MemRefType to which the type of the given value can be bufferized. More...
 
BaseMemRefType getMemRefTypeWithFullyDynamicLayout (TensorType tensorType, Attribute memorySpace=nullptr)
 Return a MemRef type with fully dynamic layout. More...
 
BaseMemRefType getMemRefTypeWithStaticIdentityLayout (TensorType tensorType, Attribute memorySpace=nullptr)
 Return a MemRef type with a static identity layout (i.e., no layout map). More...
 
OperationgetOwnerOfValue (Value value)
 Return the owner of the given value. More...
 
RegiongetNextEnclosingRepetitiveRegion (Region *region, const BufferizationOptions &options)
 Assuming that the given region is repetitive, find the next enclosing repetitive region. More...
 
RegiongetParallelRegion (Region *region, const BufferizationOptions &options)
 If region is a parallel region, return region. More...
 
void populateDynamicDimSizes (OpBuilder &b, Location loc, Value shapedValue, SmallVector< Value > &dynamicDims)
 Populate dynamicDims with tensor::DimOp / memref::DimOp results for all dynamic dimensions of the given shaped value. More...
 
FailureOr< ValuecastOrReallocMemRefValue (OpBuilder &b, Value value, MemRefType type, const BufferizationOptions &options)
 Try to cast the given ranked MemRef-typed value to the given ranked MemRef type. More...
 
LogicalResult foldToMemrefToTensorPair (RewriterBase &rewriter, ToMemrefOp toMemref, const BufferizationOptions &options)
 Try to fold to_memref(to_tensor(x)). More...
 
void populateDeallocOpCanonicalizationPatterns (RewritePatternSet &patterns, MLIRContext *context)
 Add the canonicalization patterns for bufferization.dealloc to the given pattern set to make them available to other passes (such as BufferDeallocationSimplification). More...
 
void buildBufferDeallocationPipeline (OpPassManager &pm, const BufferDeallocationPipelineOptions &options)
 Adds the buffer deallocation pipeline to the OpPassManager. More...
 
void registerBufferizationPipelines ()
 Registers all pipelines for the bufferization dialect. More...
 
void registerTransformDialectExtension (DialectRegistry &registry)
 
void populateBufferizeMaterializationLegality (ConversionTarget &target)
 Marks ops used by bufferization for type conversion materializations as "legal" in the given ConversionTarget. More...
 
void populateEliminateBufferizeMaterializationsPatterns (const BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns)
 Populate patterns to eliminate bufferize materializations. More...
 
LogicalResult bufferizeOp (Operation *op, const BufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
 Bufferize op and its nested ops that implement BufferizableOpInterface. More...
 
LogicalResult bufferizeBlockSignature (Block *block, RewriterBase &rewriter, const BufferizationOptions &options)
 Bufferize the signature of block and its callers (i.e., ops that have the given block as a successor). More...
 
BufferizationOptions getPartialBufferizationOptions ()
 Return BufferizationOptions such that the bufferizeOp behaves like the old (deprecated) partial, dialect conversion-based bufferization passes. More...
 
template<typename DominatorT >
BlockfindCommonDominator (Value value, const BufferViewFlowAnalysis::ValueSetT &values, const DominatorT &doms)
 Finds a common dominator for the given value while taking the positions of the values in the value set into account. More...
 
FailureOr< memref::GlobalOp > getGlobalFor (arith::ConstantOp constantOp, uint64_t alignment, Attribute memorySpace={})
 
LogicalResult analyzeOp (Operation *op, OneShotAnalysisState &state, BufferizationStatistics *statistics=nullptr)
 Analyze op and its nested ops. More...
 
LogicalResult runOneShotBufferize (Operation *op, const OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
 Run One-Shot Bufferize on the given op: Analysis + Bufferization. More...
 
llvm::LogicalResult analyzeModuleOp (ModuleOp moduleOp, OneShotAnalysisState &state, BufferizationStatistics *statistics=nullptr)
 Analyze moduleOp and its nested ops. More...
 
llvm::LogicalResult bufferizeModuleOp (ModuleOp moduleOp, const OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
 Bufferize op and its nested ops that implement BufferizableOpInterface. More...
 
void removeBufferizationAttributesInModule (ModuleOp moduleOp)
 Remove bufferization attributes on every FuncOp arguments in the ModuleOp. More...
 
llvm::LogicalResult runOneShotModuleBufferize (ModuleOp moduleOp, const bufferization::OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
 Run One-Shot Module Bufferization on the given module. More...
 
std::unique_ptr< PasscreateBufferDeallocationPass ()
 Creates an instance of the BufferDeallocation pass to free all allocated buffers. More...
 
std::unique_ptr< PasscreateOwnershipBasedBufferDeallocationPass (DeallocationOptions options=DeallocationOptions())
 Creates an instance of the OwnershipBasedBufferDeallocation pass to free all allocated buffers. More...
 
std::unique_ptr< PasscreateOptimizeAllocationLivenessPass ()
 Creates a pass that finds all temporary allocations and attempts to move the deallocation after the last user/dependency of the allocation, thereby optimizing allocation liveness. More...
 
std::unique_ptr< PasscreateBufferDeallocationSimplificationPass ()
 Creates a pass that optimizes bufferization.dealloc operations. More...
 
std::unique_ptr< PasscreateLowerDeallocationsPass ()
 Creates an instance of the LowerDeallocations pass to lower bufferization.dealloc operations to the memref dialect. More...
 
void populateBufferizationDeallocLoweringPattern (RewritePatternSet &patterns, const DeallocHelperMap &deallocHelperFuncMap)
 Adds the conversion pattern of the bufferization.dealloc operation to the given pattern set for use in other transformation passes. More...
 
func::FuncOp buildDeallocationLibraryFunction (OpBuilder &builder, Location loc, SymbolTable &symbolTable)
 Construct the library function needed for the fully generic bufferization.dealloc lowering implemented in the LowerDeallocations pass. More...
 
LogicalResult deallocateBuffers (Operation *op)
 Run buffer deallocation. More...
 
LogicalResult deallocateBuffersOwnershipBased (FunctionOpInterface op, DeallocationOptions options)
 Run the ownership-based buffer deallocation. More...
 
std::unique_ptr< PasscreateBufferHoistingPass ()
 Creates a pass that moves allocations upwards to reduce the number of required copies that are inserted during the BufferDeallocation pass. More...
 
std::unique_ptr< PasscreateBufferLoopHoistingPass ()
 Creates a pass that moves allocations upwards out of loops. More...
 
std::unique_ptr< PasscreateBufferResultsToOutParamsPass (const BufferResultsToOutParamsOpts &options={})
 Creates a pass that converts memref function results to out-params. More...
 
LogicalResult promoteBufferResultsToOutParams (ModuleOp module, const BufferResultsToOutParamsOpts &options)
 Replace buffers that are returned from a function with an out parameter. More...
 
std::unique_ptr< PasscreateDropEquivalentBufferResultsPass ()
 Creates a pass that drops memref function results that are equivalent to a function argument. More...
 
std::unique_ptr< PasscreateEmptyTensorToAllocTensorPass ()
 Create a pass that rewrites tensor.empty to bufferization.alloc_tensor. More...
 
LogicalResult dropEquivalentBufferResults (ModuleOp module)
 Drop all memref function results that are equivalent to a function argument. More...
 
std::unique_ptr< OperationPass< func::FuncOp > > createFinalizingBufferizePass ()
 Creates a pass that finalizes a partial bufferization by removing remaining bufferization.to_tensor and bufferization.to_memref operations. More...
 
std::unique_ptr< PasscreateOneShotBufferizePass ()
 Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize. More...
 
std::unique_ptr< PasscreateOneShotBufferizePass (const OneShotBufferizationOptions &options)
 Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize and the specified bufferization options. More...
 
std::unique_ptr< PasscreatePromoteBuffersToStackPass (unsigned maxAllocSizeInBytes=1024, unsigned maxRankOfAllocatedMemRef=1)
 Creates a pass that promotes heap-based allocations to stack-based ones. More...
 
std::unique_ptr< PasscreatePromoteBuffersToStackPass (std::function< bool(Value)> isSmallAlloc)
 Creates a pass that promotes heap-based allocations to stack-based ones. More...
 
std::unique_ptr< PasscreateEmptyTensorEliminationPass ()
 Create a pass that tries to eliminate tensor.empty ops that are anchored on insert_slice ops. More...
 
LogicalResult eliminateEmptyTensors (RewriterBase &rewriter, Operation *op)
 Try to eliminate "tensor.empty" ops inside op. More...
 
LogicalResult eliminateEmptyTensors (RewriterBase &rewriter, Operation *op, OneShotAnalysisState &state)
 Try to eliminate "tensor.empty" ops inside op. More...
 
void hoistBuffersFromLoops (Operation *op)
 Within the given operation, hoist buffers from loops where possible. More...
 
LogicalResult insertTensorCopies (Operation *op, const OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
 Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops. More...
 
LogicalResult insertTensorCopies (Operation *op, const AnalysisState &state)
 Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops. More...
 
void populateEmptyTensorToAllocTensorPattern (RewritePatternSet &patterns)
 Populate patterns to lower tensor.empty ops to bufferization.alloc_tensor ops. More...
 

Typedef Documentation

◆ AliasingOpOperandList

A list of possible aliasing OpOperands.

This list models the runtime aliasing relationship for a Value.

Definition at line 94 of file BufferizableOpInterface.h.

◆ AliasingValueList

A list of possible aliasing Values.

This list models the runtime aliasing relationship for an OpOperand.

Definition at line 98 of file BufferizableOpInterface.h.

◆ DeallocHelperMap

Maps from symbol table to its corresponding dealloc helper function.

Definition at line 22 of file Passes.h.

◆ RegisterDependenciesFn

using mlir::bufferization::RegisterDependenciesFn = typedef std::function<void(ValueRange, ValueRange)>

Definition at line 20 of file BufferViewFlowOpInterface.h.

Enumeration Type Documentation

◆ BufferRelation

Specifies a fine-grain relationship between buffers to enable more analysis.

Enumerator
Unknown 
Equivalent 

Definition at line 34 of file BufferizableOpInterface.h.

Function Documentation

◆ allocateTensorForShapedValue()

FailureOr< Value > mlir::bufferization::allocateTensorForShapedValue ( OpBuilder b,
Location  loc,
Value  shapedValue,
const BufferizationOptions options,
bool  copy = true 
)

Create an AllocTensorOp for the given shaped value (memref or tensor).

Create an AllocTensorOp for the given shaped value.

If copy is set, the shaped value is copied. Otherwise, a tensor with undefined contents is allocated.

Definition at line 147 of file BufferizableOpInterface.cpp.

References copy(), mlir::OpBuilder::create(), mlir::Operation::emitError(), mlir::detail::enumerate(), getBufferType(), mlir::Value::getDefiningOp(), getOwnerOfValue(), mlir::Operation::getResult(), mlir::Value::getType(), options, populateDynamicDimSizes(), and mlir::reifyResultShapes().

◆ analyzeModuleOp()

LogicalResult mlir::bufferization::analyzeModuleOp ( ModuleOp  moduleOp,
OneShotAnalysisState state,
BufferizationStatistics statistics = nullptr 
)

Analyze moduleOp and its nested ops.

Bufferization decisions are stored in state.

Definition at line 374 of file OneShotModuleBufferize.cpp.

◆ analyzeOp()

LogicalResult mlir::bufferization::analyzeOp ( Operation op,
OneShotAnalysisState state,
BufferizationStatistics statistics = nullptr 
)

Analyze op and its nested ops.

Bufferization decisions are stored in state.

Definition at line 1339 of file OneShotAnalysis.cpp.

◆ bufferizeBlockSignature()

LogicalResult mlir::bufferization::bufferizeBlockSignature ( Block block,
RewriterBase rewriter,
const BufferizationOptions options 
)

Bufferize the signature of block and its callers (i.e., ops that have the given block as a successor).

All block argument types are changed to memref types. All corresponding operands of all callers are wrapped in bufferization.to_memref ops. All uses of bufferized tensor block arguments are wrapped in bufferization.to_tensor ops.

It is expected that all callers implement the BranchOpInterface. Otherwise, this function will fail. The BranchOpInterface is used to query the range of operands that are forwarded to this block.

It is expected that the parent op of this block implements the BufferizableOpInterface. The buffer types of tensor block arguments are computed with BufferizableOpIntercace::getBufferType.

Definition at line 537 of file Bufferize.cpp.

References mlir::OpBuilder::create(), mlir::Block::getArguments(), getBufferType(), mlir::Block::getParentOp(), options, and mlir::OpBuilder::setInsertionPointToStart().

◆ bufferizeModuleOp()

LogicalResult mlir::bufferization::bufferizeModuleOp ( ModuleOp  moduleOp,
const OneShotBufferizationOptions options,
BufferizationStatistics statistics = nullptr 
)

Bufferize op and its nested ops that implement BufferizableOpInterface.

Note: This function does not run One-Shot Analysis. No buffer copies are inserted except two cases:

  • options.copyBeforeWrite is set, in which case buffers are copied before every write.
  • options.copyBeforeWrite is not set and options.noAnalysisFuncFilter is not empty. The FuncOps it contains were not analyzed. Buffer copies will be inserted only to these FuncOps.

Definition at line 425 of file OneShotModuleBufferize.cpp.

References bufferizeOp(), mlir::bufferization::BufferizationOptions::copyBeforeWrite, foldMemRefCasts(), getFuncOpsOrderedByCalls(), and options.

◆ bufferizeOp()

LogicalResult mlir::bufferization::bufferizeOp ( Operation op,
const BufferizationOptions options,
BufferizationStatistics statistics = nullptr 
)

Bufferize op and its nested ops that implement BufferizableOpInterface.

Note: This function does not resolve read-after-write conflicts. Use this function only if it is guaranteed that the input IR can bufferize without additional buffer copies or set "options.copyBeforeWrite = true". The general bufferization entry point is runOneShotBufferize.

Check the result of bufferization. Return an error if an op was not bufferized, unless partial bufferization is allowed.

Definition at line 419 of file Bufferize.cpp.

Referenced by bufferizeModuleOp().

◆ buildBufferDeallocationPipeline()

void mlir::bufferization::buildBufferDeallocationPipeline ( OpPassManager pm,
const BufferDeallocationPipelineOptions options 
)

Adds the buffer deallocation pipeline to the OpPassManager.

This is the standard pipeline for deallocating the MemRefs introduced by the One-Shot bufferization pass.

Definition at line 21 of file BufferizationPipelines.cpp.

References mlir::OpPassManager::addPass(), createBufferDeallocationSimplificationPass(), mlir::createCanonicalizerPass(), mlir::createCSEPass(), mlir::memref::createExpandReallocPass(), createLowerDeallocationsPass(), createOwnershipBasedBufferDeallocationPass(), and options.

Referenced by registerBufferizationPipelines().

◆ buildDeallocationLibraryFunction()

func::FuncOp mlir::bufferization::buildDeallocationLibraryFunction ( OpBuilder builder,
Location  loc,
SymbolTable symbolTable 
)

Construct the library function needed for the fully generic bufferization.dealloc lowering implemented in the LowerDeallocations pass.

The function can then be called at bufferization dealloc sites to determine aliasing and ownership.

The generated function takes two memrefs of indices and three memrefs of booleans as arguments:

  • The first argument A should contain the result of the extract_aligned_pointer_as_index operation applied to the memrefs to be deallocated
  • The second argument B should contain the result of the extract_aligned_pointer_as_index operation applied to the memrefs to be retained
  • The third argument C should contain the conditions as passed directly to the deallocation operation.
  • The fourth argument D is used to pass results to the caller. Those represent the condition under which the memref at the corresponding position in A should be deallocated.
  • The fifth argument E is used to pass results to the caller. It provides the ownership value corresponding the the memref at the same position in B

This helper function is supposed to be called once for each bufferization.dealloc operation to determine the deallocation need and new ownership indicator for the retained values, but does not perform the deallocation itself.

Generated code:

func.func @dealloc_helper(
%dyn_dealloc_base_pointer_list: memref<?xindex>,
%dyn_retain_base_pointer_list: memref<?xindex>,
%dyn_cond_list: memref<?xi1>,
%dyn_dealloc_cond_out: memref<?xi1>,
%dyn_ownership_out: memref<?xi1>) {
%c0 = arith.constant 0 : index
%c1 = arith.constant 1 : index
%true = arith.constant true
%false = arith.constant false
%num_dealloc_memrefs = memref.dim %dyn_dealloc_base_pointer_list, %c0
%num_retain_memrefs = memref.dim %dyn_retain_base_pointer_list, %c0
// Zero initialize result buffer.
scf.for %i = %c0 to %num_retain_memrefs step %c1 {
memref.store %false, %dyn_ownership_out[%i] : memref<?xi1>
}
scf.for %i = %c0 to %num_dealloc_memrefs step %c1 {
%dealloc_bp = memref.load %dyn_dealloc_base_pointer_list[%i]
%cond = memref.load %dyn_cond_list[%i]
// Check for aliasing with retained memrefs.
%does_not_alias_retained = scf.for %j = %c0 to %num_retain_memrefs
step %c1 iter_args(%does_not_alias_aggregated = %true) -> (i1) {
%retain_bp = memref.load %dyn_retain_base_pointer_list[%j]
%does_alias = arith.cmpi eq, %retain_bp, %dealloc_bp : index
scf.if %does_alias {
%curr_ownership = memref.load %dyn_ownership_out[%j]
%updated_ownership = arith.ori %curr_ownership, %cond : i1
memref.store %updated_ownership, %dyn_ownership_out[%j]
}
%does_not_alias = arith.cmpi ne, %retain_bp, %dealloc_bp : index
%updated_aggregate = arith.andi %does_not_alias_aggregated,
%does_not_alias : i1
scf.yield %updated_aggregate : i1
}
// Check for aliasing with dealloc memrefs in the list before the
// current one, i.e.,
// `fix i, forall j < i: check_aliasing(%dyn_dealloc_base_pointer[j],
// %dyn_dealloc_base_pointer[i])`
%does_not_alias_any = scf.for %j = %c0 to %i step %c1
iter_args(%does_not_alias_agg = %does_not_alias_retained) -> (i1) {
%prev_dealloc_bp = memref.load %dyn_dealloc_base_pointer_list[%j]
%does_not_alias = arith.cmpi ne, %prev_dealloc_bp, %dealloc_bp
%updated_alias_agg = arith.andi %does_not_alias_agg, %does_not_alias
scf.yield %updated_alias_agg : i1
}
%dealloc_cond = arith.andi %does_not_alias_any, %cond : i1
memref.store %dealloc_cond, %dyn_dealloc_cond_out[%i] : memref<?xi1>
}
return
}
Eliminates variable at the specified position using Fourier-Motzkin variable elimination.

Definition at line 427 of file LowerDeallocations.cpp.

References mlir::OpBuilder::clearInsertionPoint(), mlir::OpBuilder::create(), mlir::get(), mlir::Builder::getBoolAttr(), mlir::Builder::getFunctionType(), mlir::Builder::getI1Type(), mlir::Builder::getIndexAttr(), mlir::Builder::getIndexType(), mlir::SymbolTable::insert(), mlir::SymbolTable::Private, and mlir::OpBuilder::setInsertionPointToStart().

◆ castOrReallocMemRefValue()

FailureOr< Value > mlir::bufferization::castOrReallocMemRefValue ( OpBuilder b,
Value  value,
MemRefType  type,
const BufferizationOptions options 
)

Try to cast the given ranked MemRef-typed value to the given ranked MemRef type.

Insert a reallocation + copy if it cannot be statically guaranteed that a direct cast would be valid.

E.g., when casting from a ranked MemRef type with dynamic layout to a ranked MemRef type with static layout, it is not statically known whether the cast will succeed or not. Such memref.cast ops may fail at runtime. This function never generates such casts and conservatively inserts a copy.

This function returns failure() in case of unsupported casts. E.g., casts with differing element types or memory spaces.

Definition at line 26 of file BufferizationOps.cpp.

References copy(), mlir::OpBuilder::create(), mlir::Value::getLoc(), mlir::getStridesAndOffset(), mlir::Value::getType(), and options.

Referenced by mlir::bufferization::BufferizeTypeConverter::BufferizeTypeConverter(), and foldToMemrefToTensorPair().

◆ createBufferDeallocationPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferDeallocationPass ( )

Creates an instance of the BufferDeallocation pass to free all allocated buffers.

Definition at line 691 of file BufferDeallocation.cpp.

◆ createBufferDeallocationSimplificationPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferDeallocationSimplificationPass ( )

Creates a pass that optimizes bufferization.dealloc operations.

For example, it reduces the number of alias checks needed at runtime using static alias analysis.

Definition at line 482 of file BufferDeallocationSimplification.cpp.

Referenced by buildBufferDeallocationPipeline().

◆ createBufferHoistingPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferHoistingPass ( )

Creates a pass that moves allocations upwards to reduce the number of required copies that are inserted during the BufferDeallocation pass.

Definition at line 482 of file BufferOptimizations.cpp.

◆ createBufferLoopHoistingPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferLoopHoistingPass ( )

Creates a pass that moves allocations upwards out of loops.

This avoids reallocations inside of loops.

Definition at line 486 of file BufferOptimizations.cpp.

◆ createBufferResultsToOutParamsPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferResultsToOutParamsPass ( const BufferResultsToOutParamsOpts options = {})

Creates a pass that converts memref function results to out-params.

Definition at line 257 of file BufferResultsToOutParams.cpp.

References options.

◆ createDropEquivalentBufferResultsPass()

std::unique_ptr< Pass > mlir::bufferization::createDropEquivalentBufferResultsPass ( )

Creates a pass that drops memref function results that are equivalent to a function argument.

Definition at line 162 of file DropEquivalentBufferResults.cpp.

◆ createEmptyTensorEliminationPass()

std::unique_ptr< Pass > mlir::bufferization::createEmptyTensorEliminationPass ( )

Create a pass that tries to eliminate tensor.empty ops that are anchored on insert_slice ops.

Definition at line 216 of file EmptyTensorElimination.cpp.

◆ createEmptyTensorToAllocTensorPass()

std::unique_ptr< Pass > mlir::bufferization::createEmptyTensorToAllocTensorPass ( )

Create a pass that rewrites tensor.empty to bufferization.alloc_tensor.

Definition at line 68 of file EmptyTensorToAllocTensor.cpp.

Referenced by mlir::sparse_tensor::SparsificationAndBufferizationPass::runOnOperation().

◆ createFinalizingBufferizePass()

std::unique_ptr< OperationPass< func::FuncOp > > mlir::bufferization::createFinalizingBufferizePass ( )

Creates a pass that finalizes a partial bufferization by removing remaining bufferization.to_tensor and bufferization.to_memref operations.

Definition at line 334 of file Bufferize.cpp.

Referenced by mlir::sparse_tensor::buildSparsifier().

◆ createLowerDeallocationsPass()

std::unique_ptr< Pass > mlir::bufferization::createLowerDeallocationsPass ( )

Creates an instance of the LowerDeallocations pass to lower bufferization.dealloc operations to the memref dialect.

Definition at line 550 of file LowerDeallocations.cpp.

Referenced by buildBufferDeallocationPipeline().

◆ createOneShotBufferizePass() [1/2]

std::unique_ptr< Pass > mlir::bufferization::createOneShotBufferizePass ( )

Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize.

Definition at line 324 of file Bufferize.cpp.

◆ createOneShotBufferizePass() [2/2]

std::unique_ptr< Pass > mlir::bufferization::createOneShotBufferizePass ( const OneShotBufferizationOptions options)

Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize and the specified bufferization options.

Definition at line 328 of file Bufferize.cpp.

References options.

◆ createOptimizeAllocationLivenessPass()

std::unique_ptr< Pass > mlir::bufferization::createOptimizeAllocationLivenessPass ( )

Creates a pass that finds all temporary allocations and attempts to move the deallocation after the last user/dependency of the allocation, thereby optimizing allocation liveness.

Definition at line 159 of file OptimizeAllocationLiveness.cpp.

◆ createOwnershipBasedBufferDeallocationPass()

std::unique_ptr< Pass > mlir::bufferization::createOwnershipBasedBufferDeallocationPass ( DeallocationOptions  options = DeallocationOptions())

Creates an instance of the OwnershipBasedBufferDeallocation pass to free all allocated buffers.

Definition at line 1069 of file OwnershipBasedBufferDeallocation.cpp.

References options.

Referenced by buildBufferDeallocationPipeline().

◆ createPromoteBuffersToStackPass() [1/2]

std::unique_ptr< Pass > mlir::bufferization::createPromoteBuffersToStackPass ( std::function< bool(Value)>  isSmallAlloc)

Creates a pass that promotes heap-based allocations to stack-based ones.

Only buffers smaller with isSmallAlloc(alloc) == true are promoted.

Definition at line 496 of file BufferOptimizations.cpp.

◆ createPromoteBuffersToStackPass() [2/2]

std::unique_ptr< Pass > mlir::bufferization::createPromoteBuffersToStackPass ( unsigned  maxAllocSizeInBytes = 1024,
unsigned  maxRankOfAllocatedMemRef = 1 
)

Creates a pass that promotes heap-based allocations to stack-based ones.

Only buffers smaller than the provided size are promoted. Dynamic shaped buffers are promoted up to the given rank.

Definition at line 490 of file BufferOptimizations.cpp.

◆ deallocateBuffers()

LogicalResult mlir::bufferization::deallocateBuffers ( Operation op)

Run buffer deallocation.

Definition at line 651 of file BufferDeallocation.cpp.

◆ deallocateBuffersOwnershipBased()

LogicalResult mlir::bufferization::deallocateBuffersOwnershipBased ( FunctionOpInterface  op,
DeallocationOptions  options 
)

Run the ownership-based buffer deallocation.

Definition at line 1055 of file OwnershipBasedBufferDeallocation.cpp.

◆ dropEquivalentBufferResults()

LogicalResult mlir::bufferization::dropEquivalentBufferResults ( ModuleOp  module)

◆ eliminateEmptyTensors() [1/2]

LogicalResult mlir::bufferization::eliminateEmptyTensors ( RewriterBase rewriter,
Operation op 
)

Try to eliminate "tensor.empty" ops inside op.

This transformation looks for subset ops that insert a tensor that originates from a "tensor.empty" (as per the reverse use-def chain). Such "tensor.empty" ops are replaced with the destination subset.

E.g.: %0 = tensor.empty() : tensor<10xf32> %1 = linalg.fill ... outs(%0 : tensor<10xf32>) %2 = tensor.insert_slice %0 into t ...

In the above example, the subset op is "tensor.insert_slice". When tracing back the reverse use-def chain of a the source, we end up at a "tensor.empty" op.

Definition at line 188 of file EmptyTensorElimination.cpp.

◆ eliminateEmptyTensors() [2/2]

LogicalResult mlir::bufferization::eliminateEmptyTensors ( RewriterBase rewriter,
Operation op,
OneShotAnalysisState state 
)

Try to eliminate "tensor.empty" ops inside op.

This function overload accepts an existing OneShotAnalysisState, which contains in-place bufferization decisions. This overload is useful if an existing analysis should be reused for empty tensor elimination.

Definition at line 103 of file EmptyTensorElimination.cpp.

◆ findCommonDominator()

template<typename DominatorT >
Block* mlir::bufferization::findCommonDominator ( Value  value,
const BufferViewFlowAnalysis::ValueSetT values,
const DominatorT &  doms 
)

Finds a common dominator for the given value while taking the positions of the values in the value set into account.

It supports dominator and post-dominator analyses via template arguments. If no common dominator can be found, this function will return "nullptr".

Definition at line 81 of file BufferUtils.h.

References mlir::Value::getParentBlock(), and mlir::Operation::getUsers().

◆ foldToMemrefToTensorPair()

LogicalResult mlir::bufferization::foldToMemrefToTensorPair ( RewriterBase rewriter,
ToMemrefOp  toMemref,
const BufferizationOptions options 
)

Try to fold to_memref(to_tensor(x)).

If x's type and the result type of the to_memref op are different, a memref.cast is needed.

Definition at line 88 of file BufferizationOps.cpp.

References castOrReallocMemRefValue(), options, mlir::RewriterBase::replaceOp(), and mlir::RewriterBase::replaceOpWithNewOp().

◆ getBuffer()

FailureOr< Value > mlir::bufferization::getBuffer ( RewriterBase rewriter,
Value  value,
const BufferizationOptions options 
)

Lookup the buffer for the given value.

If the value was not bufferized yet, wrap it in a ToMemrefOp. Otherwise, it is the result of a ToTensorOp, from which the memref operand is returned.

Definition at line 639 of file BufferizableOpInterface.cpp.

References mlir::OpBuilder::create(), ensureToMemrefOpIsValid(), getBufferType(), mlir::Value::getDefiningOp(), mlir::Value::getLoc(), mlir::Value::getType(), options, and setInsertionPointAfter().

◆ getBufferType() [1/2]

FailureOr< BaseMemRefType > mlir::bufferization::getBufferType ( Value  value,
const BufferizationOptions options 
)

Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR.

Return the buffer type for a given Value (tensor) after bufferization.

Note: It should be sufficient to call getBuffer()->getType() in most cases. However, when a buffer type should be predicted without modifying any IR, this function can be used.

This function is a wrapper around BufferizableOpInterface::getBufferType.

Definition at line 664 of file BufferizableOpInterface.cpp.

References options.

Referenced by allocateTensorForShapedValue(), bufferizeBlockSignature(), and getBuffer().

◆ getBufferType() [2/2]

FailureOr< BaseMemRefType > mlir::bufferization::getBufferType ( Value  value,
const BufferizationOptions options,
SmallVector< Value > &  invocationStack 
)

Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR.

Return the buffer type for a given Value (tensor) after bufferization.

This function (and not the other overload without invocationStack) can be used from getBufferType implementations of the BufferizableOpInterface.

Note: It should be sufficient to call getBuffer()->getType() in most cases. However, when a buffer type should be predicted without modifying any IR, this function can be used.

This function is a wrapper around BufferizableOpInterface::getBufferType.

Definition at line 671 of file BufferizableOpInterface.cpp.

References mlir::Value::getType().

◆ getGlobalFor()

FailureOr< memref::GlobalOp > mlir::bufferization::getGlobalFor ( arith::ConstantOp  constantOp,
uint64_t  alignment,
Attribute  memorySpace = {} 
)

Definition at line 104 of file BufferUtils.cpp.

◆ getMemRefType()

BaseMemRefType mlir::bufferization::getMemRefType ( Value  value,
const BufferizationOptions options,
MemRefLayoutAttrInterface  layout = {},
Attribute  memorySpace = nullptr 
)

Return a MemRefType to which the type of the given value can be bufferized.

If possible, op bufferization implementations should not use this function and instead infer precise memref types for tensor results by themselves.

Unless a layout map was specified, options.unknownTypeConverterFn determines what kind of layout map will be used. For best composability (without copies), the fully dynamic layout map is used by default.

Note: Canonicalization patterns could clean up layout maps and infer more precise layout maps after bufferization. However, many possible canonicalizations are currently not implemented.

Definition at line 764 of file BufferizableOpInterface.cpp.

References mlir::get(), mlir::Value::getType(), and options.

Referenced by mlir::bufferization::detail::defaultGetBufferType().

◆ getMemRefTypeWithFullyDynamicLayout()

BaseMemRefType mlir::bufferization::getMemRefTypeWithFullyDynamicLayout ( TensorType  tensorType,
Attribute  memorySpace = nullptr 
)

Return a MemRef type with fully dynamic layout.

If the given tensor type is unranked, return an unranked MemRef type.

Definition at line 790 of file BufferizableOpInterface.cpp.

References mlir::get(), and mlir::Type::getContext().

Referenced by createMemcpy(), mlir::sparse_tensor::LoopEmitter::initializeLoopEmit(), and mlir::bufferization::BufferizationOptions::setFunctionBoundaryTypeConversion().

◆ getMemRefTypeWithStaticIdentityLayout()

BaseMemRefType mlir::bufferization::getMemRefTypeWithStaticIdentityLayout ( TensorType  tensorType,
Attribute  memorySpace = nullptr 
)

Return a MemRef type with a static identity layout (i.e., no layout map).

If the given tensor type is unranked, return an unranked MemRef type.

Definition at line 814 of file BufferizableOpInterface.cpp.

References mlir::get().

Referenced by mlir::getBufferizationOptionsForSparsification(), getPartialBufferizationOptions(), and mlir::bufferization::BufferizationOptions::setFunctionBoundaryTypeConversion().

◆ getNextEnclosingRepetitiveRegion()

Region * mlir::bufferization::getNextEnclosingRepetitiveRegion ( Region region,
const BufferizationOptions options 
)

Assuming that the given region is repetitive, find the next enclosing repetitive region.

Definition at line 113 of file BufferizableOpInterface.cpp.

References mlir::Region::getParentRegion(), isRepetitiveRegion(), and options.

◆ getOwnerOfValue()

Operation * mlir::bufferization::getOwnerOfValue ( Value  value)

Return the owner of the given value.

In case of a BlockArgument that is the owner of the block. In case of an OpResult that is the defining op.

Definition at line 138 of file BufferizableOpInterface.cpp.

References mlir::Operation::getParentOp().

Referenced by allocateTensorForShapedValue(), mlir::bufferization::BufferizationOptions::dynCastBufferizableOp(), and mlir::bufferization::OneShotAnalysisState::isWritable().

◆ getParallelRegion()

Region * mlir::bufferization::getParallelRegion ( Region region,
const BufferizationOptions options 
)

If region is a parallel region, return region.

Otherwise, find the first enclosing parallel region of region. If there is no such region, return "nullptr".

Note: Whether a region is parallel or sequential is queried from the BufferizableOpInterface.

Definition at line 123 of file BufferizableOpInterface.cpp.

References mlir::Region::getParentOp(), mlir::Region::getParentRegion(), mlir::Region::getRegionNumber(), isRepetitiveRegion(), and options.

◆ getPartialBufferizationOptions()

BufferizationOptions mlir::bufferization::getPartialBufferizationOptions ( )

Return BufferizationOptions such that the bufferizeOp behaves like the old (deprecated) partial, dialect conversion-based bufferization passes.

A copy will be inserted before every buffer write.

Definition at line 623 of file Bufferize.cpp.

References getMemRefTypeWithStaticIdentityLayout(), mlir::Value::getType(), and options.

◆ hasTensorSemantics()

bool mlir::bufferization::hasTensorSemantics ( Operation op)

Return "true" if the given op has tensor semantics and should be bufferized.

If the op is bufferizable, the BufferizableOpInterface is queried. Otherwise, an op has tensor semantics if it has tensor operands, tensor op results and/or tensor block arguments.

Definition at line 694 of file BufferizableOpInterface.cpp.

Referenced by mlir::vector::getMixedSizesXfer(), mlir::linalg::transposeBatchMatmul(), and mlir::linalg::transposeMatmul().

◆ hoistBuffersFromLoops()

void mlir::bufferization::hoistBuffersFromLoops ( Operation op)

Within the given operation, hoist buffers from loops where possible.

See "BufferLoopHoistingPass" for more information.

Definition at line 477 of file BufferOptimizations.cpp.

◆ insertTensorCopies() [1/2]

LogicalResult mlir::bufferization::insertTensorCopies ( Operation op,
const AnalysisState state 
)

Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops.

After applying this transform, the IR can be bufferized without inserting additional buffer allocations.

Definition at line 51 of file TensorCopyInsertion.cpp.

◆ insertTensorCopies() [2/2]

LogicalResult mlir::bufferization::insertTensorCopies ( Operation op,
const OneShotBufferizationOptions options,
BufferizationStatistics statistics = nullptr 
)

Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops.

After applying this transform, the IR can be bufferized without inserting additional buffer allocations.

Definition at line 29 of file TensorCopyInsertion.cpp.

Referenced by runOneShotModuleBufferize(), and mlir::sparse_tensor::SparsificationAndBufferizationPass::runOnOperation().

◆ populateBufferizationDeallocLoweringPattern()

void mlir::bufferization::populateBufferizationDeallocLoweringPattern ( RewritePatternSet patterns,
const DeallocHelperMap deallocHelperFuncMap 
)

Adds the conversion pattern of the bufferization.dealloc operation to the given pattern set for use in other transformation passes.

Definition at line 543 of file LowerDeallocations.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateBufferizeMaterializationLegality()

void mlir::bufferization::populateBufferizeMaterializationLegality ( ConversionTarget target)

Marks ops used by bufferization for type conversion materializations as "legal" in the given ConversionTarget.

This function should be called by all bufferization passes using BufferizeTypeConverter so that materializations work properly. One exception is bufferization passes doing "full" conversions, where it can be desirable for even the materializations to remain illegal so that they are eliminated, such as via the patterns in populateEliminateBufferizeMaterializationsPatterns.

Definition at line 95 of file Bufferize.cpp.

References mlir::ConversionTarget::addLegalOp().

◆ populateDeallocOpCanonicalizationPatterns()

void mlir::bufferization::populateDeallocOpCanonicalizationPatterns ( RewritePatternSet patterns,
MLIRContext context 
)

Add the canonicalization patterns for bufferization.dealloc to the given pattern set to make them available to other passes (such as BufferDeallocationSimplification).

Definition at line 1175 of file BufferizationOps.cpp.

References mlir::RewritePatternSet::add().

◆ populateDynamicDimSizes()

void mlir::bufferization::populateDynamicDimSizes ( OpBuilder b,
Location  loc,
Value  shapedValue,
SmallVector< Value > &  dynamicDims 
)

Populate dynamicDims with tensor::DimOp / memref::DimOp results for all dynamic dimensions of the given shaped value.

Definition at line 133 of file BufferizationOps.cpp.

References mlir::OpBuilder::create(), and mlir::Value::getType().

Referenced by allocateTensorForShapedValue().

◆ populateEliminateBufferizeMaterializationsPatterns()

void mlir::bufferization::populateEliminateBufferizeMaterializationsPatterns ( const BufferizeTypeConverter typeConverter,
RewritePatternSet patterns 
)

Populate patterns to eliminate bufferize materializations.

In particular, these are the tensor_load/buffer_cast ops.

Definition at line 132 of file Bufferize.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateEmptyTensorToAllocTensorPattern()

void mlir::bufferization::populateEmptyTensorToAllocTensorPattern ( RewritePatternSet patterns)

Populate patterns to lower tensor.empty ops to bufferization.alloc_tensor ops.

Definition at line 54 of file EmptyTensorToAllocTensor.cpp.

References mlir::RewritePatternSet::getContext(), and mlir::RewritePatternSet::insert().

◆ promoteBufferResultsToOutParams()

LogicalResult mlir::bufferization::promoteBufferResultsToOutParams ( ModuleOp  module,
const BufferResultsToOutParamsOpts options 
)

Replace buffers that are returned from a function with an out parameter.

Also update all call sites.

Definition at line 204 of file BufferResultsToOutParams.cpp.

References mlir::OpBuilder::create(), options, updateCalls(), updateFuncOp(), and updateReturnOps().

◆ registerBufferizationPipelines()

void mlir::bufferization::registerBufferizationPipelines ( )

Registers all pipelines for the bufferization dialect.

Currently, this includes only the "buffer-deallocation-pipeline".

Definition at line 37 of file BufferizationPipelines.cpp.

References buildBufferDeallocationPipeline().

Referenced by mlir::registerAllPasses().

◆ registerTransformDialectExtension()

void mlir::bufferization::registerTransformDialectExtension ( DialectRegistry registry)

◆ removeBufferizationAttributesInModule()

void mlir::bufferization::removeBufferizationAttributesInModule ( ModuleOp  moduleOp)

Remove bufferization attributes on every FuncOp arguments in the ModuleOp.

Definition at line 417 of file OneShotModuleBufferize.cpp.

◆ replaceOpWithBufferizedValues()

void mlir::bufferization::replaceOpWithBufferizedValues ( RewriterBase rewriter,
Operation op,
ValueRange  values 
)

Replace an op with replacement values.

The op is deleted. Tensor OpResults must be replaced with memref values.

Definition at line 700 of file BufferizableOpInterface.cpp.

◆ replaceOpWithNewBufferizedOp()

template<typename OpTy , typename... Args>
OpTy mlir::bufferization::replaceOpWithNewBufferizedOp ( RewriterBase rewriter,
Operation op,
Args &&...  args 
)

Replace an op with a new op.

The new op must have the same number of results as the replaced op. The new op may not return any tensor values.

Definition at line 632 of file BufferizableOpInterface.h.

◆ runOneShotBufferize()

LogicalResult mlir::bufferization::runOneShotBufferize ( Operation op,
const OneShotBufferizationOptions options,
BufferizationStatistics statistics = nullptr 
)

Run One-Shot Bufferize on the given op: Analysis + Bufferization.

Definition at line 1381 of file OneShotAnalysis.cpp.

◆ runOneShotModuleBufferize()

LogicalResult mlir::bufferization::runOneShotModuleBufferize ( ModuleOp  moduleOp,
const bufferization::OneShotBufferizationOptions options,
BufferizationStatistics statistics = nullptr 
)

Run One-Shot Module Bufferization on the given module.

Performs a simple function call analysis to determine which function arguments are inplaceable. Then analyzes and bufferizes FuncOps one-by-one with One-Shot Bufferize.

Definition at line 478 of file OneShotModuleBufferize.cpp.

References insertTensorCopies(), and options.