This represents a token in the MLIR syntax. More...

#include "AsmParser/Token.h"

Public Types
enum	Kind

Public Member Functions
	Token (Kind kind, StringRef spelling)

StringRef	getSpelling () const

Kind	getKind () const

bool	is (Kind k) const

bool	isAny (Kind k1, Kind k2) const

template<typename... T>
bool	isAny (Kind k1, Kind k2, Kind k3, T... others) const
	Return true if this token is one of the specified kinds. More...

bool	isNot (Kind k) const

template<typename... T>
bool	isNot (Kind k1, Kind k2, T... others) const
	Return true if this token isn't one of the specified kinds. More...

bool	isKeyword () const
	Return true if this is one of the keyword token kinds (e.g. kw_if). More...

bool	isCodeCompletion () const
	Returns true if the current token represents a code completion. More...

bool	isCodeCompletionFor (Kind kind) const
	Returns true if the current token represents a code completion for the "normal" token type. More...

bool	isOrIsCodeCompletionFor (Kind kind) const
	Returns true if the current token is the given type, or represents a code completion for that type. More...

std::optional< unsigned >	getUnsignedIntegerValue () const
	For an integer token, return its value as an unsigned. More...

std::optional< uint64_t >	getUInt64IntegerValue () const

std::optional< double >	getFloatingPointValue () const
	For a floatliteral token, return its value as a double. More...

std::optional< unsigned >	getIntTypeBitwidth () const
	For an inttype token, return its bitwidth. More...

std::optional< bool >	getIntTypeSignedness () const
	For an inttype token, return its signedness semantics: std::nullopt means no signedness semantics; true means signed integer type; false means unsigned integer type. More...

std::optional< unsigned >	getHashIdentifierNumber () const
	Given a hash_identifier token like #123, try to parse the number out of the identifier, returning std::nullopt if it is a named identifier like #x or if the integer doesn't fit. More...

std::string	getStringValue () const
	Given a token containing a string literal, return its value, including removing the quote characters and unescaping the contents of the string. More...

std::optional< std::string >	getHexStringValue () const
	Given a token containing a hex string literal, return its value or std::nullopt if the token does not contain a valid hex string. More...

std::string	getSymbolReference () const
	Given a token containing a symbol reference, return the unescaped string value. More...

SMLoc	getLoc () const

SMLoc	getEndLoc () const

SMRange	getLocRange () const

Static Public Member Functions
static std::optional< uint64_t >	getUInt64IntegerValue (StringRef spelling)
	For an integer token, return its value as an uint64_t. More...

static StringRef	getTokenSpelling (Kind kind)
	Given a punctuation or keyword token kind, return the spelling of the token as a string. More...

Detailed Description

This represents a token in the MLIR syntax.

Definition at line 20 of file Token.h.

Member Enumeration Documentation

◆ Kind

enum mlir::Token::Kind

Definition at line 22 of file Token.h.

Constructor & Destructor Documentation

◆ Token()

mlir::Token::Token	(	Kind	kind,
		StringRef	spelling
	)

inline

Definition at line 31 of file Token.h.

Member Function Documentation

◆ getEndLoc()

SMLoc Token::getEndLoc ( ) const

Definition at line 26 of file Token.cpp.

Referenced by getLocRange(), and mlir::detail::Parser::parseDialectSymbolBody().

◆ getFloatingPointValue()

std::optional< double > Token::getFloatingPointValue ( ) const

For a floatliteral token, return its value as a double.

For a floatliteral, return its value as a double.

Returns std::nullopt in the case of underflow or overflow.

Return std::nullopt if the value underflows or overflows.

Definition at line 56 of file Token.cpp.

Referenced by mlir::detail::Parser::parseFloatAttr(), and mlir::detail::Parser::parseFloatFromLiteral().

◆ getHashIdentifierNumber()

std::optional< unsigned > Token::getHashIdentifierNumber ( ) const

Given a hash_identifier token like #123, try to parse the number out of the identifier, returning std::nullopt if it is a named identifier like #x or if the integer doesn't fit.

Definition at line 166 of file Token.cpp.

References getKind().

◆ getHexStringValue()

std::optional< std::string > Token::getHexStringValue ( ) const

Given a token containing a hex string literal, return its value or std::nullopt if the token does not contain a valid hex string.

A hex string literal is a string starting with 0x and only containing hex digits.

Definition at line 135 of file Token.cpp.

References getKind(), and getSpelling().

Referenced by parseElementAttrHexValues().

◆ getIntTypeBitwidth()

std::optional< unsigned > Token::getIntTypeBitwidth ( ) const

For an inttype token, return its bitwidth.

Definition at line 64 of file Token.cpp.

References getKind().

Referenced by mlir::detail::Parser::parseNonFunctionType().

◆ getIntTypeSignedness()

std::optional< bool > Token::getIntTypeSignedness ( ) const

For an inttype token, return its signedness semantics: std::nullopt means no signedness semantics; true means signed integer type; false means unsigned integer type.

Definition at line 73 of file Token.cpp.

References getKind().

◆ getKind()

Kind mlir::Token::getKind ( ) const

inline

Definition at line 37 of file Token.h.

Referenced by getHashIdentifierNumber(), getHexStringValue(), getIntTypeBitwidth(), getIntTypeSignedness(), and getStringValue().

◆ getLoc()

SMLoc Token::getLoc ( ) const

Definition at line 24 of file Token.cpp.

◆ getLocRange()

SMRange Token::getLocRange ( ) const

Definition at line 30 of file Token.cpp.

References getEndLoc(), and getLoc().

Referenced by parseExtendedSymbol(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalSymbolName().

◆ getSpelling()

StringRef mlir::Token::getSpelling ( ) const

inline

Definition at line 34 of file Token.h.

Referenced by mlir::detail::AsmParserImpl< BaseT >::codeCompleteExpectedTokens(), getHexStringValue(), getStringValue(), getSymbolReference(), mlir::detail::Parser::getTokenSpelling(), getUInt64IntegerValue(), mlir::detail::Parser::parseDecOrHexAttr(), parseExtendedSymbol(), mlir::detail::Parser::parseFloatFromIntegerLiteral(), mlir::detail::AsmParserImpl< BaseT >::parseKeywordOrCompletion(), mlir::detail::Parser::parseOptionalDecimalInteger(), and mlir::detail::Parser::parseOptionalInteger().

◆ getStringValue()

std::string Token::getStringValue ( ) const

Given a token containing a string literal, return its value, including removing the quote characters and unescaping the contents of the string.

The lexer has already verified that this token is valid.

Definition at line 86 of file Token.cpp.

References getKind(), and getSpelling().

Referenced by getSymbolReference(), mlir::detail::Parser::parseAttribute(), mlir::detail::Parser::parseAttributeDict(), mlir::detail::Parser::parseNameOrFileLineColRange(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalString().

◆ getSymbolReference()

std::string Token::getSymbolReference ( ) const

Given a token containing a symbol reference, return the unescaped string value.

Definition at line 153 of file Token.cpp.

References getSpelling(), getStringValue(), and is().

Referenced by mlir::detail::Parser::parseAttribute(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalSymbolName().

◆ getTokenSpelling()

StringRef Token::getTokenSpelling ( Kind kind )

static

Given a punctuation or keyword token kind, return the spelling of the token as a string.

Warning: This will abort on markers, identifiers and literal tokens since they have no fixed spelling.

Definition at line 177 of file Token.cpp.

Referenced by mlir::detail::Parser::parseCommaSeparatedListUntil().

◆ getUInt64IntegerValue() [1/2]

std::optional<uint64_t> mlir::Token::getUInt64IntegerValue ( ) const

inline

Definition at line 83 of file Token.h.

References getSpelling().

◆ getUInt64IntegerValue() [2/2]

std::optional< uint64_t > Token::getUInt64IntegerValue ( StringRef spelling )

static

For an integer token, return its value as an uint64_t.

For an integer token, return its value as a uint64_t.

If it doesn't fit, return std::nullopt.

Definition at line 45 of file Token.cpp.

Referenced by mlir::detail::Parser::parseDistinctAttr(), mlir::detail::Parser::parseIntegerInDimensionList(), and mlir::detail::Parser::parseStridedLayoutAttr().

◆ getUnsignedIntegerValue()

std::optional< unsigned > Token::getUnsignedIntegerValue ( ) const

For an integer token, return its value as an unsigned.

If it doesn't fit, return std::nullopt.

Definition at line 34 of file Token.cpp.

Referenced by mlir::detail::Parser::parseNameOrFileLineColRange().

◆ is()

bool mlir::Token::is ( Kind k ) const

inline

Definition at line 38 of file Token.h.

Referenced by mlir::detail::Parser::consumeToken(), mlir::detail::Parser::emitWrongTokenError(), getSymbolReference(), isAny(), isCodeCompletion(), isOrIsCodeCompletionFor(), mlir::detail::Parser::parseAttributeDict(), mlir::detail::AsmParserImpl< BaseT >::parseBase64Bytes(), parseExtendedSymbol(), mlir::detail::Parser::parseFloatFromLiteral(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalString().

◆ isAny() [1/2]

bool mlir::Token::isAny	(	Kind	k1,
		Kind	k2
	)		const

inline

Definition at line 40 of file Token.h.

References is().

Referenced by isAny(), mlir::detail::Parser::isCurrentTokenAKeyword(), isIdentifier(), isNot(), and mlir::detail::Parser::parseAttributeDict().

◆ isAny() [2/2]

template<typename... T>

bool mlir::Token::isAny	(	Kind	k1,
		Kind	k2,
		Kind	k3,
		T...	others
	)		const

inline

Return true if this token is one of the specified kinds.

Definition at line 44 of file Token.h.

References is(), and isAny().

◆ isCodeCompletion()

bool mlir::Token::isCodeCompletion ( ) const

inline

Returns true if the current token represents a code completion.

Definition at line 62 of file Token.h.

References is().

Referenced by mlir::detail::AsmParserImpl< BaseT >::codeCompleteExpectedTokens(), isCodeCompletionFor(), mlir::detail::Parser::parseDialectSymbolBody(), parseExtendedSymbol(), mlir::detail::AsmParserImpl< BaseT >::parseKeyword(), mlir::detail::AsmParserImpl< BaseT >::parseKeywordOrCompletion(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalKeyword().

◆ isCodeCompletionFor()

bool Token::isCodeCompletionFor ( Kind kind ) const

Returns true if the current token represents a code completion for the "normal" token type.

Definition at line 203 of file Token.cpp.

References isCodeCompletion().

Referenced by isOrIsCodeCompletionFor().

◆ isKeyword()

bool Token::isKeyword ( ) const

Return true if this is one of the keyword token kinds (e.g. kw_if).

Definition at line 192 of file Token.cpp.

Referenced by mlir::detail::Parser::isCurrentTokenAKeyword(), isIdentifier(), and mlir::detail::Parser::parseAttributeDict().

◆ isNot() [1/2]

bool mlir::Token::isNot ( Kind k ) const

inline

Definition at line 50 of file Token.h.

Referenced by mlir::detail::Parser::consumeIf(), mlir::detail::Parser::consumeToken(), mlir::detail::Parser::emitError(), mlir::detail::AsmParserImpl< BaseT >::parseCustomAttributeWithFallback(), mlir::detail::AsmParserImpl< BaseT >::parseCustomTypeWithFallback(), mlir::detail::Parser::parseDialectSymbolBody(), mlir::detail::AsmParserImpl< BaseT >::parseOptionalAttrDict(), mlir::detail::Parser::parseOptionalDecimalInteger(), mlir::detail::Parser::parseOptionalInteger(), and mlir::detail::AsmParserImpl< BaseT >::parseOptionalSymbolName().

◆ isNot() [2/2]

template<typename... T>

bool mlir::Token::isNot	(	Kind	k1,
		Kind	k2,
		T...	others
	)		const

inline

Return true if this token isn't one of the specified kinds.

Definition at line 54 of file Token.h.

References isAny().

◆ isOrIsCodeCompletionFor()

bool mlir::Token::isOrIsCodeCompletionFor ( Kind kind ) const

inline

Returns true if the current token is the given type, or represents a code completion for that type.

Definition at line 70 of file Token.h.

References is(), and isCodeCompletionFor().

The documentation for this class was generated from the following files:

lib/AsmParser/Token.h
lib/AsmParser/Token.cpp
lib/Tools/PDLL/Parser/Lexer.cpp

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Enumeration Documentation

◆ Kind

Constructor & Destructor Documentation

◆ Token()

Member Function Documentation

◆ getEndLoc()

◆ getFloatingPointValue()

◆ getHashIdentifierNumber()

◆ getHexStringValue()

◆ getIntTypeBitwidth()

◆ getIntTypeSignedness()

◆ getKind()

◆ getLoc()

◆ getLocRange()

◆ getSpelling()

◆ getStringValue()

◆ getSymbolReference()

◆ getTokenSpelling()

◆ getUInt64IntegerValue() [1/2]

◆ getUInt64IntegerValue() [2/2]

◆ getUnsignedIntegerValue()

◆ is()

◆ isAny() [1/2]

◆ isAny() [2/2]

◆ isCodeCompletion()

◆ isCodeCompletionFor()

◆ isKeyword()

◆ isNot() [1/2]

◆ isNot() [2/2]

◆ isOrIsCodeCompletionFor()