Namespaces
	detail

	func_ext

Classes
class	OpFilter

struct	BufferizationOptions
	Options for BufferizableOpInterface-based bufferization. More...

class	AnalysisState
	AnalysisState provides a variety of helper functions for dealing with tensor values. More...

struct	DstBufferizableOpInterfaceExternalModel
	Bufferizable ops that implement the DestinationStyleOpInterface can use this external model base class. More...

struct	BufferizationStatistics
	Bufferization statistics for debugging. More...

class	BufferizeTypeConverter
	A helper type converter class that automatically populates the relevant materializations and type conversions for bufferization. More...

class	BufferPlacementAllocs
	A simple analysis that detects allocation operations. More...

class	BufferPlacementTransformationBase
	The base class for all BufferPlacement transformations. More...

struct	OneShotBufferizationOptions
	Options for analysis-enabled bufferization. More...

class	BufferizationAliasInfo
	The BufferizationAliasInfo class maintains a list of buffer aliases and equivalence classes to support bufferization. More...

class	OneShotAnalysisState
	State for analysis-enabled bufferization. More...

struct	BufferResultsToOutParamsOptions

Typedefs
using	AnchorMatchFn = std::function< bool(OpOperand &, SmallVector< Value > &)>
	A function that matches anchor OpOperands for tensor::EmptyOp elimination. More...

using	RewriteFn = std::function< Value(OpBuilder &, Location, OpOperand &)>
	A function that rewrites matched anchors. More...

Enumerations
enum class	BufferRelation { None , Equivalent }
	Specify fine-grain relationship between buffers to enable more analysis. More...

Functions
bool	isFunctionArgument (Value value)
	Return `true` if the given value is a BlockArgument of a func::FuncOp. More...

FailureOr< Value >	allocateTensorForShapedValue (OpBuilder &b, Location loc, Value shapedValue, bool escape, const BufferizationOptions &options, bool copy=true)
	Create an AllocTensorOp for the given shaped value (memref or tensor). More...

bool	allocationDoesNotEscape (OpResult opResult)
	Return `true` if the allocation of the given op is guaranteed to not escape the containing block. More...

FailureOr< Value >	getBuffer (RewriterBase &rewriter, Value value, const BufferizationOptions &options)
	Lookup the buffer for the given value. More...

FailureOr< BaseMemRefType >	getBufferType (Value value, const BufferizationOptions &options)
	Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR. More...

FailureOr< BaseMemRefType >	getBufferType (Value value, const BufferizationOptions &options, const DenseMap< Value, BaseMemRefType > &fixedTypes)
	Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR. More...

void	replaceOpWithBufferizedValues (RewriterBase &rewriter, Operation *op, ValueRange values)
	Replace an op with replacement values. More...

template<typename OpTy , typename... Args>
OpTy	replaceOpWithNewBufferizedOp (RewriterBase &rewriter, Operation *op, Args &&...args)
	Replace an op with a new op. More...

bool	shouldDeallocateOpResult (OpResult opResult, const BufferizationOptions &options)
	Return `true` if the buffer of given OpResult should be deallocated. More...

BaseMemRefType	getMemRefType (Value value, const BufferizationOptions &options, MemRefLayoutAttrInterface layout={}, Attribute memorySpace=nullptr)
	Return a MemRefType to which the type of the given value can be bufferized. More...

BaseMemRefType	getMemRefTypeWithFullyDynamicLayout (TensorType tensorType, Attribute memorySpace=nullptr)
	Return a MemRef type with fully dynamic layout. More...

BaseMemRefType	getMemRefTypeWithStaticIdentityLayout (TensorType tensorType, Attribute memorySpace=nullptr)
	Return a MemRef type with a static identity layout (i.e., no layout map). More...

Operation *	getOwnerOfValue (Value value)
	Return the owner of the given value. More...

Region *	getEnclosingRepetitiveRegion (Operation *op, const BufferizationOptions &options)
	Return the closest enclosing repetitive region around the given op. More...

Region *	getEnclosingRepetitiveRegion (Value value, const BufferizationOptions &options)
	Return the closest enclosing repetitive region around the place where the given value is defined. More...

Region *	getEnclosingRepetitiveRegion (Block *block, const BufferizationOptions &options)
	Return the closest enclosing repetitive region around the given block. More...

void	populateDynamicDimSizes (OpBuilder &b, Location loc, Value shapedValue, SmallVector< Value > &dynamicDims)
	Populate `dynamicDims` with tensor::DimOp / memref::DimOp results for all dynamic dimensions of the given shaped value. More...

FailureOr< Value >	castOrReallocMemRefValue (OpBuilder &b, Value value, MemRefType type)
	Try to cast the given ranked MemRef-typed value to the given ranked MemRef type. More...

LogicalResult	foldToMemrefToTensorPair (RewriterBase &rewriter, ToMemrefOp toMemref)
	Try to fold to_memref(to_tensor(x)). More...

void	registerTransformDialectExtension (DialectRegistry &registry)

void	populateBufferizeMaterializationLegality (ConversionTarget &target)
	Marks ops used by bufferization for type conversion materializations as "legal" in the given ConversionTarget. More...

void	populateEliminateBufferizeMaterializationsPatterns (BufferizeTypeConverter &typeConverter, RewritePatternSet &patterns)
	Populate patterns to eliminate bufferize materializations. More...

LogicalResult	bufferizeOp (Operation op, const BufferizationOptions &options, bool copyBeforeWrite=true, const OpFilter opFilter=nullptr, BufferizationStatistics *statistics=nullptr)
	Bufferize `op` and its nested ops that implement `BufferizableOpInterface`. More...

BufferizationOptions	getPartialBufferizationOptions ()

FailureOr< memref::GlobalOp >	getGlobalFor (arith::ConstantOp constantOp, uint64_t alignment)

LogicalResult	analyzeOp (Operation op, OneShotAnalysisState &state, BufferizationStatistics statistics=nullptr)
	Analyze `op` and its nested ops. More...

LogicalResult	runOneShotBufferize (Operation op, const OneShotBufferizationOptions &options, BufferizationStatistics statistics=nullptr)
	Run One-Shot Bufferize on the given op: Analysis + Bufferization. More...

LogicalResult	analyzeModuleOp (ModuleOp moduleOp, OneShotAnalysisState &state, BufferizationStatistics *statistics=nullptr)
	Analyze `moduleOp` and its nested ops. More...

LogicalResult	bufferizeModuleOp (ModuleOp moduleOp, const OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
	Bufferize `op` and its nested ops that implement `BufferizableOpInterface`. More...

void	removeBufferizationAttributesInModule (ModuleOp moduleOp)
	Remove bufferization attributes on every FuncOp arguments in the ModuleOp. More...

LogicalResult	runOneShotModuleBufferize (ModuleOp moduleOp, const bufferization::OneShotBufferizationOptions &options, BufferizationStatistics *statistics=nullptr)
	Run One-Shot Module Bufferization on the given module. More...

std::unique_ptr< Pass >	createBufferDeallocationPass ()
	Creates an instance of the BufferDeallocation pass to free all allocated buffers. More...

LogicalResult	deallocateBuffers (Operation *op)
	Run buffer deallocation. More...

std::unique_ptr< Pass >	createBufferHoistingPass ()
	Creates a pass that moves allocations upwards to reduce the number of required copies that are inserted during the BufferDeallocation pass. More...

std::unique_ptr< Pass >	createBufferLoopHoistingPass ()
	Creates a pass that moves allocations upwards out of loops. More...

std::unique_ptr< Pass >	createBufferResultsToOutParamsPass (const BufferResultsToOutParamsOptions &options={})
	Creates a pass that converts memref function results to out-params. More...

LogicalResult	promoteBufferResultsToOutParams (ModuleOp module, const BufferResultsToOutParamsOptions &options)
	Replace buffers that are returned from a function with an out parameter. More...

std::unique_ptr< Pass >	createDropEquivalentBufferResultsPass ()
	Creates a pass that drops memref function results that are equivalent to a function argument. More...

std::unique_ptr< Pass >	createEmptyTensorToAllocTensorPass ()
	Create a pass that rewrites tensor.empty to bufferization.alloc_tensor. More...

LogicalResult	dropEquivalentBufferResults (ModuleOp module)
	Drop all memref function results that are equivalent to a function argument. More...

std::unique_ptr< OperationPass< func::FuncOp > >	createFinalizingBufferizePass ()
	Creates a pass that finalizes a partial bufferization by removing remaining bufferization.to_tensor and bufferization.to_memref operations. More...

std::unique_ptr< Pass >	createOneShotBufferizePass ()
	Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize. More...

std::unique_ptr< Pass >	createOneShotBufferizePass (const OneShotBufferizationOptions &options)
	Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize and the specified bufferization options. More...

std::unique_ptr< Pass >	createPromoteBuffersToStackPass (unsigned maxAllocSizeInBytes=1024, unsigned maxRankOfAllocatedMemRef=1)
	Creates a pass that promotes heap-based allocations to stack-based ones. More...

std::unique_ptr< Pass >	createPromoteBuffersToStackPass (std::function< bool(Value)> isSmallAlloc)
	Creates a pass that promotes heap-based allocations to stack-based ones. More...

std::unique_ptr< Pass >	createEmptyTensorEliminationPass ()
	Create a pass that tries to eliminate tensor.empty ops that are anchored on insert_slice ops. More...

std::unique_ptr< Pass >	createBufferizationBufferizePass ()
	Create a pass that bufferizes ops from the bufferization dialect. More...

void	registerAllocationOpInterfaceExternalModels (DialectRegistry &registry)
	Register external models for AllocationOpInterface. More...

LogicalResult	eliminateEmptyTensors (RewriterBase &rewriter, Operation *op, bufferization::AnalysisState &state, AnchorMatchFn anchorMatchFunc, RewriteFn rewriteFunc)
	Try to eliminate tensor::EmptyOps inside `op`. More...

LogicalResult	insertSliceAnchoredEmptyTensorEliminationStep (RewriterBase &rewriter, Operation *op, bufferization::AnalysisState &state)
	Try to eliminate tensor::EmptyOps inside `op` that are anchored on an InsertSliceOp, i.e., if it is eventually inserted into another tensor (and some other conditions are met). More...

LogicalResult	insertTensorCopies (Operation op, const OneShotBufferizationOptions &options, BufferizationStatistics statistics=nullptr)
	Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops. More...

LogicalResult	insertTensorCopies (Operation *op, const AnalysisState &state)
	Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops. More...

void	populateEmptyTensorToAllocTensorPattern (RewritePatternSet &patterns)
	Populate patterns to lower tensor.empty ops to bufferization.alloc_tensor ops. More...

Typedef Documentation

◆ AnchorMatchFn

using mlir::bufferization::AnchorMatchFn = typedef std::function<bool(OpOperand &, SmallVector<Value> &)>

A function that matches anchor OpOperands for tensor::EmptyOp elimination.

If an OpOperand is matched, the function should populate the SmallVector with all values that are needed during RewriteFn to produce the replacement value.

Definition at line 25 of file Transforms.h.

◆ RewriteFn

using mlir::bufferization::RewriteFn = typedef std::function<Value(OpBuilder &, Location, OpOperand &)>

A function that rewrites matched anchors.

Definition at line 28 of file Transforms.h.

Enumeration Type Documentation

◆ BufferRelation

enum mlir::bufferization::BufferRelation

strong

Specify fine-grain relationship between buffers to enable more analysis.

Enumerator
None
Equivalent

Definition at line 304 of file BufferizableOpInterface.h.

Function Documentation

◆ allocateTensorForShapedValue()

FailureOr< Value > mlir::bufferization::allocateTensorForShapedValue	(	OpBuilder &	b,
		Location	loc,
		Value	shapedValue,
		bool	escape,
		const BufferizationOptions &	options,
		bool	copy = `true`
	)

Create an AllocTensorOp for the given shaped value (memref or tensor).

Create an AllocTensorOp for the given shaped value.

If copy is set, the shaped value is copied. Otherwise, a tensor with undefined contents is allocated.

Definition at line 102 of file BufferizableOpInterface.cpp.

References mlir::Type::cast(), mlir::Value::cast(), copy(), mlir::OpBuilder::create(), mlir::detail::enumerate(), mlir::failed(), mlir::failure(), mlir::Builder::getBoolArrayAttr(), getBufferType(), mlir::Value::getDefiningOp(), mlir::Builder::getI64IntegerAttr(), mlir::Value::getType(), mlir::Type::isa(), mlir::Value::isa(), options, populateDynamicDimSizes(), mlir::Operation::setAttr(), and mlir::succeeded().

◆ allocationDoesNotEscape()

bool mlir::bufferization::allocationDoesNotEscape ( OpResult opResult )

Return true if the allocation of the given op is guaranteed to not escape the containing block.

Definition at line 83 of file BufferizableOpInterface.cpp.

References mlir::Operation::getAttrOfType(), mlir::Value::getDefiningOp(), mlir::OpResult::getResultNumber(), and mlir::Operation::hasAttr().

◆ analyzeModuleOp()

LogicalResult mlir::bufferization::analyzeModuleOp	(	ModuleOp	moduleOp,
		OneShotAnalysisState &	state,
		BufferizationStatistics *	statistics = `nullptr`
	)

Analyze moduleOp and its nested ops.

Bufferization decisions are stored in state.

Definition at line 365 of file OneShotModuleBufferize.cpp.

References mlir::bufferization::func_ext::FuncAnalysisState::analyzedFuncOps, analyzeOp(), mlir::bufferization::BufferizationOptions::bufferizeFunctionBoundaries, equivalenceAnalysis(), mlir::failed(), mlir::failure(), mlir::bufferization::OneShotAnalysisState::getAliasInfo(), getFuncOpsOrderedByCalls(), mlir::bufferization::OneShotAnalysisState::getOptions(), getOrCreateFuncAnalysisState(), mlir::bufferization::func_ext::FuncAnalysisState::startFunctionAnalysis(), and mlir::success().

Referenced by insertTensorCopies().

◆ analyzeOp()

LogicalResult mlir::bufferization::analyzeOp	(	Operation *	op,
		OneShotAnalysisState &	state,
		BufferizationStatistics *	statistics = `nullptr`
	)

Analyze op and its nested ops.

Bufferization decisions are stored in state.

Definition at line 1136 of file OneShotAnalysis.cpp.

Referenced by analyzeModuleOp(), inPlaceAnalysis(), and insertTensorCopies().

◆ bufferizeModuleOp()

LogicalResult mlir::bufferization::bufferizeModuleOp	(	ModuleOp	moduleOp,
		const OneShotBufferizationOptions &	options,
		BufferizationStatistics *	statistics = `nullptr`
	)

Bufferize op and its nested ops that implement BufferizableOpInterface.

Note: This function does not run One-Shot Analysis. No buffer copies are inserted unless options.copyBeforeWrite is set, in which case buffers are copied before every write.

Definition at line 414 of file OneShotModuleBufferize.cpp.

References bufferizeOp(), mlir::failed(), mlir::failure(), foldMemRefCasts(), getFuncOpsOrderedByCalls(), options, removeBufferizationAttributesInModule(), and mlir::success().

Referenced by runOneShotModuleBufferize().

◆ bufferizeOp()

LogicalResult mlir::bufferization::bufferizeOp	(	Operation *	op,
		const BufferizationOptions &	options,
		bool	copyBeforeWrite = `true`,
		const OpFilter *	opFilter = `nullptr`,
		BufferizationStatistics *	statistics = `nullptr`
	)

Bufferize op and its nested ops that implement BufferizableOpInterface.

If copyBeforeWrite, buffers are duplicated and copied before any tensor use that bufferizes to a memory write.

Note: In the general case, it unsafe to run with copyBeforeWrite = false because read-after-write conflicts may materialize during bufferization. copyBeforeWrite = false is safe only if the input IR is guaranteed to not require any out-of-place bufferization.

Note: This function bufferizes ops without utilizing analysis results. It can be used to implement partial bufferization passes.

Check the result of bufferization. Return an error if an op was not bufferized, unless partial bufferization is allowed.

Definition at line 421 of file Bufferize.cpp.

References mlir::Operation::emitError(), mlir::failed(), mlir::failure(), foldToMemrefToTensorPair(), mlir::Operation::getContext(), mlir::Operation::getName(), mlir::Operation::getUses(), hasTensorSemantics(), insertTensorCopies(), mlir::isMemoryEffectFree(), mlir::bufferization::OpFilter::isOpAllowed(), options, mlir::PostOrder, mlir::success(), and mlir::Operation::walk().

Referenced by bufferizeModuleOp(), mlir::sparse_tensor::SparsificationAndBufferizationPass::runDenseBufferization(), and runOneShotBufferize().

◆ castOrReallocMemRefValue()

FailureOr< Value > mlir::bufferization::castOrReallocMemRefValue	(	OpBuilder &	b,
		Value	value,
		MemRefType	type
	)

Try to cast the given ranked MemRef-typed value to the given ranked MemRef type.

Insert a reallocation + copy if it cannot be statically guaranteed that a direct cast would be valid.

E.g., when casting from a ranked MemRef type with dynamic layout to a ranked MemRef type with static layout, it is not statically known whether the cast will succeed or not. Such memref.cast ops may fail at runtime. This function never generates such casts and conservatively inserts a copy.

This function returns failure() in case of unsupported casts. E.g., casts with differing element types or memory spaces.

Definition at line 28 of file BufferizationOps.cpp.

References mlir::Type::cast(), copy(), mlir::OpBuilder::create(), mlir::OpBuilder::createOrFold(), mlir::failed(), mlir::failure(), mlir::Value::getLoc(), mlir::getStridesAndOffset(), and mlir::Value::getType().

Referenced by mlir::bufferization::BufferizeTypeConverter::BufferizeTypeConverter(), and foldToMemrefToTensorPair().

◆ createBufferDeallocationPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferDeallocationPass ( )

Creates an instance of the BufferDeallocation pass to free all allocated buffers.

Definition at line 713 of file BufferDeallocation.cpp.

◆ createBufferHoistingPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferHoistingPass ( )

Creates a pass that moves allocations upwards to reduce the number of required copies that are inserted during the BufferDeallocation pass.

Definition at line 435 of file BufferOptimizations.cpp.

◆ createBufferizationBufferizePass()

std::unique_ptr< Pass > mlir::bufferization::createBufferizationBufferizePass ( )

Create a pass that bufferizes ops from the bufferization dialect.

Definition at line 303 of file Bufferize.cpp.

◆ createBufferLoopHoistingPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferLoopHoistingPass ( )

Creates a pass that moves allocations upwards out of loops.

This avoids reallocations inside of loops.

Definition at line 439 of file BufferOptimizations.cpp.

◆ createBufferResultsToOutParamsPass()

std::unique_ptr< Pass > mlir::bufferization::createBufferResultsToOutParamsPass ( const BufferResultsToOutParamsOptions & options = {} )

Creates a pass that converts memref function results to out-params.

Definition at line 221 of file BufferResultsToOutParams.cpp.

References options.

◆ createDropEquivalentBufferResultsPass()

std::unique_ptr< Pass > mlir::bufferization::createDropEquivalentBufferResultsPass ( )

Creates a pass that drops memref function results that are equivalent to a function argument.

Definition at line 157 of file DropEquivalentBufferResults.cpp.

◆ createEmptyTensorEliminationPass()

std::unique_ptr< Pass > mlir::bufferization::createEmptyTensorEliminationPass ( )

Create a pass that tries to eliminate tensor.empty ops that are anchored on insert_slice ops.

Definition at line 267 of file EmptyTensorElimination.cpp.

◆ createEmptyTensorToAllocTensorPass()

std::unique_ptr< Pass > mlir::bufferization::createEmptyTensorToAllocTensorPass ( )

Create a pass that rewrites tensor.empty to bufferization.alloc_tensor.

Definition at line 68 of file EmptyTensorToAllocTensor.cpp.

Referenced by mlir::sparse_tensor::SparsificationAndBufferizationPass::runOnOperation().

◆ createFinalizingBufferizePass()

std::unique_ptr< OperationPass< func::FuncOp > > mlir::bufferization::createFinalizingBufferizePass ( )

Creates a pass that finalizes a partial bufferization by removing remaining bufferization.to_tensor and bufferization.to_memref operations.

Definition at line 317 of file Bufferize.cpp.

Referenced by mlir::sparse_tensor::buildSparseCompiler().

◆ createOneShotBufferizePass() [1/2]

std::unique_ptr< Pass > mlir::bufferization::createOneShotBufferizePass ( )

Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize.

Definition at line 307 of file Bufferize.cpp.

◆ createOneShotBufferizePass() [2/2]

std::unique_ptr< Pass > mlir::bufferization::createOneShotBufferizePass ( const OneShotBufferizationOptions & options )

Create a pass that bufferizes all ops that implement BufferizableOpInterface with One-Shot Bufferize and the specified bufferization options.

Definition at line 311 of file Bufferize.cpp.

References options.

◆ createPromoteBuffersToStackPass() [1/2]

std::unique_ptr< Pass > mlir::bufferization::createPromoteBuffersToStackPass ( std::function< bool(Value)> isSmallAlloc )

Creates a pass that promotes heap-based allocations to stack-based ones.

Only buffers smaller with isSmallAlloc(alloc) == true are promoted.

Definition at line 449 of file BufferOptimizations.cpp.

◆ createPromoteBuffersToStackPass() [2/2]

std::unique_ptr< Pass > mlir::bufferization::createPromoteBuffersToStackPass	(	unsigned	maxAllocSizeInBytes = `1024`,
		unsigned	maxRankOfAllocatedMemRef = `1`
	)

Creates a pass that promotes heap-based allocations to stack-based ones.

Only buffers smaller than the provided size are promoted. Dynamic shaped buffers are promoted up to the given rank.

Definition at line 443 of file BufferOptimizations.cpp.

◆ deallocateBuffers()

LogicalResult mlir::bufferization::deallocateBuffers ( Operation * op )

Run buffer deallocation.

Definition at line 666 of file BufferDeallocation.cpp.

References mlir::WalkResult::advance(), mlir::Operation::emitError(), mlir::failed(), mlir::failure(), mlir::WalkResult::interrupt(), mlir::success(), validateSupportedControlFlow(), mlir::Operation::walk(), and mlir::WalkResult::wasInterrupted().

◆ dropEquivalentBufferResults()

LogicalResult mlir::bufferization::dropEquivalentBufferResults ( ModuleOp module )

Drop all memref function results that are equivalent to a function argument.

Definition at line 70 of file DropEquivalentBufferResults.cpp.

References mlir::WalkResult::advance(), mlir::OpBuilder::create(), mlir::detail::enumerate(), getAssumedUniqueReturnOp(), getCalledFunction(), mlir::Value::getDefiningOp(), mlir::Value::getType(), mlir::RewriterBase::replaceOp(), mlir::OpBuilder::setInsertionPoint(), mlir::WalkResult::skip(), and mlir::success().

◆ eliminateEmptyTensors()

LogicalResult mlir::bufferization::eliminateEmptyTensors	(	RewriterBase &	rewriter,
		Operation *	op,
		bufferization::AnalysisState &	state,
		AnchorMatchFn	anchorMatchFunc,
		RewriteFn	rewriteFunc
	)

Try to eliminate tensor::EmptyOps inside op.

rewriteFunc generates the replacement for the tensor::EmptyOp.
Only tensor::EmptyOps that are anchored on a matching OpOperand as per anchorMatchFunc are considered. "Anchored" means that there is a path on the reverse SSA use-def chain, starting from the OpOperand and always following the aliasing OpOperand, that eventually ends at a single tensor::EmptyOp.

A tensor::EmptyOp is replaced with the result of rewriteFunc if it is anchored on a matching OpOperand. "Anchored" means that there is a path on the reverse SSA use-def chain, starting from the OpOperand and always following the aliasing OpOperand, that eventually ends at a single tensor::EmptyOp.

Definition at line 107 of file EmptyTensorElimination.cpp.

References mlir::WalkResult::advance(), findValidInsertionPoint(), mlir::Value::getDefiningOp(), mlir::Value::getLoc(), mlir::Operation::getOpOperands(), mlir::Value::getType(), mlir::RewriterBase::replaceOp(), mlir::OpBuilder::setInsertionPoint(), and mlir::Operation::walk().

Referenced by insertSliceLikeAnchoredEmptyTensorEliminationStep().

◆ foldToMemrefToTensorPair()

LogicalResult mlir::bufferization::foldToMemrefToTensorPair	(	RewriterBase &	rewriter,
		ToMemrefOp	toMemref
	)

Try to fold to_memref(to_tensor(x)).

If x's type and the result type of the to_memref op are different, a memref.cast is needed.

Definition at line 88 of file BufferizationOps.cpp.

References castOrReallocMemRefValue(), mlir::Type::dyn_cast(), mlir::failed(), mlir::failure(), mlir::RewriterBase::replaceOp(), mlir::RewriterBase::replaceOpWithNewOp(), and mlir::success().

Referenced by bufferizeOp().

◆ getBuffer()

FailureOr< Value > mlir::bufferization::getBuffer	(	RewriterBase &	rewriter,
		Value	value,
		const BufferizationOptions &	options
	)

Lookup the buffer for the given value.

If the value was not bufferized yet, wrap it in a ToMemrefOp. Otherwise, it is the result of a ToTensorOp, from which the memref operand is returned.

Definition at line 598 of file BufferizableOpInterface.cpp.

References mlir::OpBuilder::create(), mlir::Type::dyn_cast(), ensureToMemrefOpIsValid(), mlir::failed(), mlir::failure(), getBufferType(), mlir::Value::getDefiningOp(), mlir::Value::getLoc(), mlir::Value::getType(), options, and setInsertionPointAfter().

Referenced by mlir::bufferization::func_ext::CallOpInterface::bufferize().

◆ getBufferType() [1/2]

FailureOr< BaseMemRefType > mlir::bufferization::getBufferType	(	Value	value,
		const BufferizationOptions &	options
	)

Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR.

Return the buffer type for a given Value (tensor) after bufferization.

Note: It should be sufficient to call getBuffer()->getType() in most cases. However, when a buffer type should be predicted without modifying any IR, this function can be used.

This function is a wrapper around BufferizableOpInterface::getBufferType.

Definition at line 656 of file BufferizableOpInterface.cpp.

References options.

Referenced by allocateTensorForShapedValue(), mlir::bufferization::detail::defaultGetBufferType(), and getBuffer().

◆ getBufferType() [2/2]

FailureOr< BaseMemRefType > AllocTensorOp::getBufferType	(	Value	value,
		const BufferizationOptions &	options,
		const DenseMap< Value, BaseMemRefType > &	fixedTypes
	)

Return the buffer type for a given Value (tensor) after bufferization without bufferizing any IR.

Return the buffer type for a given Value (tensor) after bufferization.

If at any point during the type computation, the type of a value in fixedTypes in required, the mapped type is used.

Note: It should be sufficient to call getBuffer()->getType() in most cases. However, when a buffer type should be predicted without modifying any IR, this function can be used.

This function is a wrapper around BufferizableOpInterface::getBufferType.

Definition at line 662 of file BufferizableOpInterface.cpp.

References mlir::Operation::emitError(), getMemRefType(), getOwnerOfValue(), mlir::Value::getType(), mlir::Type::isa(), and options.

◆ getEnclosingRepetitiveRegion() [1/3]

Region * mlir::bufferization::getEnclosingRepetitiveRegion	(	Block *	block,
		const BufferizationOptions &	options
	)

Return the closest enclosing repetitive region around the given block.

Definition at line 64 of file BufferizableOpInterface.cpp.

References mlir::Block::getParent(), mlir::Region::getParentOp(), mlir::Operation::getParentRegion(), mlir::Region::getRegionNumber(), and options.

◆ getEnclosingRepetitiveRegion() [2/3]

Region * mlir::bufferization::getEnclosingRepetitiveRegion	(	Operation *	op,
		const BufferizationOptions &	options
	)

Return the closest enclosing repetitive region around the given op.

Definition at line 44 of file BufferizableOpInterface.cpp.

References mlir::Operation::getBlock(), and options.

◆ getEnclosingRepetitiveRegion() [3/3]

Region * mlir::bufferization::getEnclosingRepetitiveRegion	(	Value	value,
		const BufferizationOptions &	options
	)

Return the closest enclosing repetitive region around the place where the given value is defined.

Definition at line 51 of file BufferizableOpInterface.cpp.

References mlir::Region::getParentOp(), mlir::Operation::getParentRegion(), mlir::Value::getParentRegion(), mlir::Region::getRegionNumber(), and options.

◆ getGlobalFor()

FailureOr< memref::GlobalOp > mlir::bufferization::getGlobalFor	(	arith::ConstantOp	constantOp,
		uint64_t	alignment
	)

Definition at line 150 of file BufferUtils.cpp.

References mlir::TypeConverter::convertType(), mlir::OpBuilder::create(), mlir::failure(), mlir::Builder::getI64Type(), mlir::Region::getOps(), mlir::Operation::getRegion(), mlir::Builder::getStringAttr(), and mlir::SymbolTable::insert().

◆ getMemRefType()

BaseMemRefType mlir::bufferization::getMemRefType	(	Value	value,
		const BufferizationOptions &	options,
		MemRefLayoutAttrInterface	layout = `{}`,
		Attribute	memorySpace = `nullptr`
	)

Return a MemRefType to which the type of the given value can be bufferized.

If possible, op bufferization implementations should not use this function and instead infer precise memref types for tensor results by themselves.

Unless a layout map was specified, options.unknownTypeConverterFn determines what kind of layout map will be used. For best composability (without copies), the fully dynamic layout map is used by default.

Note: Canonicalization patterns could clean up layout maps and infer more precise layout maps after bufferization. However, many possible canonicalizations are currently not implemented.

Definition at line 769 of file BufferizableOpInterface.cpp.

References mlir::Type::cast(), mlir::Value::getType(), and options.

Referenced by mlir::bufferization::detail::defaultGetBufferType(), and getBufferType().

◆ getMemRefTypeWithFullyDynamicLayout()

BaseMemRefType mlir::bufferization::getMemRefTypeWithFullyDynamicLayout	(	TensorType	tensorType,
		Attribute	memorySpace = `nullptr`
	)

Return a MemRef type with fully dynamic layout.

If the given tensor type is unranked, return an unranked MemRef type.

Definition at line 794 of file BufferizableOpInterface.cpp.

References mlir::Type::cast(), mlir::Type::dyn_cast(), and mlir::Type::getContext().

Referenced by mlir::bufferization::func_ext::FuncOpInterface::bufferize(), defaultUnknownTypeConverter(), and mlir::bufferization::func_ext::getBufferizedFunctionArgType().

◆ getMemRefTypeWithStaticIdentityLayout()

BaseMemRefType mlir::bufferization::getMemRefTypeWithStaticIdentityLayout	(	TensorType	tensorType,
		Attribute	memorySpace = `nullptr`
	)

Return a MemRef type with a static identity layout (i.e., no layout map).

If the given tensor type is unranked, return an unranked MemRef type.

Definition at line 817 of file BufferizableOpInterface.cpp.

References mlir::Type::cast(), and mlir::Type::dyn_cast().

Referenced by mlir::bufferization::func_ext::FuncOpInterface::bufferize(), getBufferizationOptions(), mlir::bufferization::func_ext::getBufferizedFunctionArgType(), and getPartialBufferizationOptions().

◆ getOwnerOfValue()

Operation * mlir::bufferization::getOwnerOfValue ( Value value )

Return the owner of the given value.

In case of a BlockArgument that is the owner of the block. In case of an OpResult that is the defining op.

Definition at line 77 of file BufferizableOpInterface.cpp.

References mlir::Value::cast(), and mlir::Value::dyn_cast().

Referenced by mlir::bufferization::detail::defaultGetBufferType(), and getBufferType().

◆ getPartialBufferizationOptions()

BufferizationOptions mlir::bufferization::getPartialBufferizationOptions ( )

Definition at line 529 of file Bufferize.cpp.

References mlir::Type::cast(), getMemRefTypeWithStaticIdentityLayout(), mlir::Value::getType(), and options.

◆ insertSliceAnchoredEmptyTensorEliminationStep()

LogicalResult mlir::bufferization::insertSliceAnchoredEmptyTensorEliminationStep	(	RewriterBase &	rewriter,
		Operation *	op,
		bufferization::AnalysisState &	state
	)

Try to eliminate tensor::EmptyOps inside op that are anchored on an InsertSliceOp, i.e., if it is eventually inserted into another tensor (and some other conditions are met).

Definition at line 226 of file EmptyTensorElimination.cpp.

References mlir::failed(), mlir::failure(), insertSliceLikeAnchoredEmptyTensorEliminationStep(), and mlir::success().

◆ insertTensorCopies() [1/2]

LogicalResult mlir::bufferization::insertTensorCopies	(	Operation *	op,
		const AnalysisState &	state
	)

Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops.

After applying this transform, the IR can be bufferized without inserting additional buffer allocations.

Definition at line 124 of file TensorCopyInsertion.cpp.

References mlir::WalkResult::advance(), mlir::failed(), mlir::Builder::getBoolArrayAttr(), mlir::Operation::getContext(), mlir::bufferization::AnalysisState::getOptions(), mlir::Operation::hasAttr(), mlir::WalkResult::interrupt(), mlir::Operation::setAttr(), mlir::OpBuilder::setInsertionPoint(), mlir::WalkResult::skip(), and mlir::Operation::walk().

◆ insertTensorCopies() [2/2]

LogicalResult mlir::bufferization::insertTensorCopies	(	Operation *	op,
		const OneShotBufferizationOptions &	options,
		BufferizationStatistics *	statistics = `nullptr`
	)

Resolve RaW and other conflicts by inserting bufferization.alloc_tensor ops.

After applying this transform, the IR can be bufferized without inserting additional buffer allocations.

Definition at line 99 of file TensorCopyInsertion.cpp.

References analyzeModuleOp(), analyzeOp(), mlir::failed(), mlir::failure(), options, resolveUsesInRepetitiveRegions(), and mlir::success().

Referenced by bufferizeOp(), runOneShotBufferize(), runOneShotModuleBufferize(), and mlir::sparse_tensor::SparsificationAndBufferizationPass::runOnOperation().

◆ isFunctionArgument()

bool mlir::bufferization::isFunctionArgument ( Value value )

Return true if the given value is a BlockArgument of a func::FuncOp.

Definition at line 762 of file BufferizableOpInterface.cpp.

References mlir::Value::dyn_cast().

◆ populateBufferizeMaterializationLegality()

void mlir::bufferization::populateBufferizeMaterializationLegality ( ConversionTarget & target )

Marks ops used by bufferization for type conversion materializations as "legal" in the given ConversionTarget.

This function should be called by all bufferization passes using BufferizeTypeConverter so that materializations work properly. One exception is bufferization passes doing "full" conversions, where it can be desirable for even the materializations to remain illegal so that they are eliminated, such as via the patterns in populateEliminateBufferizeMaterializationsPatterns.

Definition at line 92 of file Bufferize.cpp.

References mlir::ConversionTarget::addLegalOp().

◆ populateDynamicDimSizes()

void mlir::bufferization::populateDynamicDimSizes	(	OpBuilder &	b,
		Location	loc,
		Value	shapedValue,
		SmallVector< Value > &	dynamicDims
	)

Populate dynamicDims with tensor::DimOp / memref::DimOp results for all dynamic dimensions of the given shaped value.

Definition at line 132 of file BufferizationOps.cpp.

References mlir::Type::cast(), mlir::OpBuilder::create(), and mlir::Value::getType().

Referenced by allocateTensorForShapedValue().

◆ populateEliminateBufferizeMaterializationsPatterns()

void mlir::bufferization::populateEliminateBufferizeMaterializationsPatterns	(	BufferizeTypeConverter &	typeConverter,
		RewritePatternSet &	patterns
	)

Populate patterns to eliminate bufferize materializations.

In particular, these are the tensor_load/buffer_cast ops.

Definition at line 129 of file Bufferize.cpp.

References mlir::RewritePatternSet::add(), and mlir::RewritePatternSet::getContext().

◆ populateEmptyTensorToAllocTensorPattern()

void mlir::bufferization::populateEmptyTensorToAllocTensorPattern ( RewritePatternSet & patterns )

Populate patterns to lower tensor.empty ops to bufferization.alloc_tensor ops.

Definition at line 54 of file EmptyTensorToAllocTensor.cpp.

References mlir::RewritePatternSet::getContext(), and mlir::RewritePatternSet::insert().

◆ promoteBufferResultsToOutParams()

LogicalResult mlir::bufferization::promoteBufferResultsToOutParams	(	ModuleOp	module,
		const BufferResultsToOutParamsOptions &	options
	)

Replace buffers that are returned from a function with an out parameter.

Also update all call sites.

Definition at line 184 of file BufferResultsToOutParams.cpp.

References mlir::failed(), mlir::failure(), options, mlir::success(), updateCalls(), updateFuncOp(), and updateReturnOps().

◆ registerAllocationOpInterfaceExternalModels()

void mlir::bufferization::registerAllocationOpInterfaceExternalModels ( DialectRegistry & registry )

Register external models for AllocationOpInterface.

Definition at line 702 of file BufferDeallocation.cpp.

References mlir::DialectRegistry::addExtension().

◆ registerTransformDialectExtension()

void mlir::bufferization::registerTransformDialectExtension ( DialectRegistry & registry )

Definition at line 120 of file BufferizationTransformOps.cpp.

References mlir::DialectRegistry::addExtensions().

Referenced by mlir::registerAllDialects().

◆ removeBufferizationAttributesInModule()

void mlir::bufferization::removeBufferizationAttributesInModule ( ModuleOp moduleOp )

Remove bufferization attributes on every FuncOp arguments in the ModuleOp.

Definition at line 406 of file OneShotModuleBufferize.cpp.

References removeBufferizationAttributes().

Referenced by bufferizeModuleOp(), and mlir::sparse_tensor::SparsificationAndBufferizationPass::runDenseBufferization().

◆ replaceOpWithBufferizedValues()

void mlir::bufferization::replaceOpWithBufferizedValues	(	RewriterBase &	rewriter,
		Operation *	op,
		ValueRange	values
	)

Replace an op with replacement values.

The op is deleted. Tensor OpResults must be replaced with memref values.

Definition at line 686 of file BufferizableOpInterface.cpp.

References mlir::OpBuilder::create(), mlir::Value::getLoc(), mlir::Operation::getNumResults(), mlir::Operation::getOpResults(), mlir::Value::getType(), mlir::Type::isa(), mlir::RewriterBase::replaceOp(), and mlir::OpBuilder::setInsertionPointAfter().

Referenced by mlir::bufferization::func_ext::CallOpInterface::bufferize(), and replaceOpWithNewBufferizedOp().

◆ replaceOpWithNewBufferizedOp()

template<typename OpTy , typename... Args>

OpTy mlir::bufferization::replaceOpWithNewBufferizedOp	(	RewriterBase &	rewriter,
		Operation *	op,
		Args &&...	args
	)

Replace an op with a new op.

The new op must have the same number of results as the replaced op. The new op may not return any tensor values.

Definition at line 481 of file BufferizableOpInterface.h.

References mlir::OpBuilder::create(), mlir::Operation::getLoc(), mlir::Operation::getResults(), and replaceOpWithBufferizedValues().

◆ runOneShotBufferize()

LogicalResult mlir::bufferization::runOneShotBufferize	(	Operation *	op,
		const OneShotBufferizationOptions &	options,
		BufferizationStatistics *	statistics = `nullptr`
	)

Run One-Shot Bufferize on the given op: Analysis + Bufferization.

Definition at line 1183 of file OneShotAnalysis.cpp.

References bufferizeOp(), mlir::failed(), mlir::failure(), insertTensorCopies(), options, and mlir::success().

◆ runOneShotModuleBufferize()

LogicalResult mlir::bufferization::runOneShotModuleBufferize	(	ModuleOp	moduleOp,
		const bufferization::OneShotBufferizationOptions &	options,
		BufferizationStatistics *	statistics = `nullptr`
	)

Run One-Shot Module Bufferization on the given module.

Performs a simple function call analysis to determine which function arguments are inplaceable. Then analyzes and bufferizes FuncOps one-by-one with One-Shot Bufferize.

Definition at line 449 of file OneShotModuleBufferize.cpp.

References bufferizeModuleOp(), mlir::failed(), mlir::failure(), insertTensorCopies(), options, and mlir::success().

◆ shouldDeallocateOpResult()

bool mlir::bufferization::shouldDeallocateOpResult	(	OpResult	opResult,
		const BufferizationOptions &	options
	)

Return true if the buffer of given OpResult should be deallocated.

This function should be called during BufferizableOpInterface::bufferize implementations that allocate a new buffer for the given OpResult.

Definition at line 250 of file BufferizableOpInterface.cpp.

References mlir::Attribute::cast(), mlir::Operation::getAttr(), mlir::OpResult::getOwner(), mlir::Operation::hasAttr(), mlir::bufferization::AnalysisState::isTensorYielded(), and options.

Namespaces

Classes

Typedefs

Enumerations

Functions

Typedef Documentation

◆ AnchorMatchFn

◆ RewriteFn

Enumeration Type Documentation

◆ BufferRelation

Function Documentation

◆ allocateTensorForShapedValue()

◆ allocationDoesNotEscape()

◆ analyzeModuleOp()

◆ analyzeOp()

◆ bufferizeModuleOp()

◆ bufferizeOp()

◆ castOrReallocMemRefValue()

◆ createBufferDeallocationPass()

◆ createBufferHoistingPass()

◆ createBufferizationBufferizePass()

◆ createBufferLoopHoistingPass()

◆ createBufferResultsToOutParamsPass()

◆ createDropEquivalentBufferResultsPass()

◆ createEmptyTensorEliminationPass()

◆ createEmptyTensorToAllocTensorPass()

◆ createFinalizingBufferizePass()

◆ createOneShotBufferizePass() [1/2]

◆ createOneShotBufferizePass() [2/2]

◆ createPromoteBuffersToStackPass() [1/2]

◆ createPromoteBuffersToStackPass() [2/2]

◆ deallocateBuffers()

◆ dropEquivalentBufferResults()

◆ eliminateEmptyTensors()

◆ foldToMemrefToTensorPair()

◆ getBuffer()

◆ getBufferType() [1/2]

◆ getBufferType() [2/2]

◆ getEnclosingRepetitiveRegion() [1/3]

◆ getEnclosingRepetitiveRegion() [2/3]

◆ getEnclosingRepetitiveRegion() [3/3]

◆ getGlobalFor()

◆ getMemRefType()

◆ getMemRefTypeWithFullyDynamicLayout()

◆ getMemRefTypeWithStaticIdentityLayout()

◆ getOwnerOfValue()

◆ getPartialBufferizationOptions()

◆ insertSliceAnchoredEmptyTensorEliminationStep()

◆ insertTensorCopies() [1/2]

◆ insertTensorCopies() [2/2]

◆ isFunctionArgument()

◆ populateBufferizeMaterializationLegality()

◆ populateDynamicDimSizes()

◆ populateEliminateBufferizeMaterializationsPatterns()

◆ populateEmptyTensorToAllocTensorPattern()

◆ promoteBufferResultsToOutParams()

◆ registerAllocationOpInterfaceExternalModels()

◆ registerTransformDialectExtension()

◆ removeBufferizationAttributesInModule()

◆ replaceOpWithBufferizedValues()

◆ replaceOpWithNewBufferizedOp()

◆ runOneShotBufferize()

◆ runOneShotModuleBufferize()

◆ shouldDeallocateOpResult()