Namespace: go.std.go.token

v1.0

Contents

• Summary
• Index
• Constants
• Variables
• Functions, Macros, and Special Forms
• Types

Summary

Provides a low-level interface to the go/token package.

Package token defines constants representing the lexical tokens of the Go
programming language and basic operations on tokens (printing, predicates).

Index

• *File
• *FileSet
• *Pos
• *Position
• *Token
• ADD
• ADD_ASSIGN
• AND
• AND_ASSIGN
• AND_NOT
• AND_NOT_ASSIGN
• ARROW
• ASSIGN
• BREAK
• CASE
• CHAN
• CHAR
• COLON
• COMMA
• COMMENT
• CONST
• CONTINUE
• DEC
• DEFAULT
• DEFER
• DEFINE
• ELLIPSIS
• ELSE
• EOF
• EQL
• FALLTHROUGH
• FLOAT
• FOR
• FUNC
• File
• FileSet
• GEQ
• GO
• GOTO
• GTR
• HighestPrec
• IDENT
• IF
• ILLEGAL
• IMAG
• IMPORT
• INC
• INT
• INTERFACE
• IsExported
• IsIdentifier
• IsKeyword
• LAND
• LBRACE
• LBRACK
• LEQ
• LOR
• LPAREN
• LSS
• Lookup
• LowestPrec
• MAP
• MUL
• MUL_ASSIGN
• NEQ
• NOT
• NewFileSet
• NoPos
• OR
• OR_ASSIGN
• PACKAGE
• PERIOD
• Pos
• Position
• QUO
• QUO_ASSIGN
• RANGE
• RBRACE
• RBRACK
• REM
• REM_ASSIGN
• RETURN
• RPAREN
• SELECT
• SEMICOLON
• SHL
• SHL_ASSIGN
• SHR
• SHR_ASSIGN
• STRING
• STRUCT
• SUB
• SUB_ASSIGN
• SWITCH
• TILDE
• TYPE
• Token
• UnaryPrec
• VAR
• XOR
• XOR_ASSIGN
• arrayOfFile
• arrayOfFileSet
• arrayOfPos
• arrayOfPosition
• arrayOfToken

Legend

Constant Variable Function Macro Special form Type GoVar Receiver/Method

Constants

Constants are variables with :const true in their metadata. Joker currently does not recognize them as special; as such, it allows redefining them or their values.

• HighestPrec

  Int v1.0

  A set of constants for precedence-based expression parsing.
  Non-operators have lowest precedence, followed by operators
  starting with precedence 1 up to unary operators. The highest
  precedence serves as "catch-all" precedence for selector,
  indexing, and other operator and delimiter tokens.
  

• LowestPrec

  Int v1.0

  non-operators
  

• UnaryPrec

  Int v1.0

  A set of constants for precedence-based expression parsing.
  Non-operators have lowest precedence, followed by operators
  starting with precedence 1 up to unary operators. The highest
  precedence serves as "catch-all" precedence for selector,
  indexing, and other operator and delimiter tokens.
  

Variables

• ADD

  GoObject v1.0

  Operators and delimiters
  

• ADD_ASSIGN

  GoObject v1.0

  +=
  

• AND

  GoObject v1.0

  &
  

• AND_ASSIGN

  GoObject v1.0

  &=
  

• AND_NOT

  GoObject v1.0

  &^
  

• AND_NOT_ASSIGN

  GoObject v1.0

  &^=
  

• ARROW

  GoObject v1.0

  <-
  

• ASSIGN

  GoObject v1.0

  =
  

• BREAK

  GoObject v1.0

  Keywords
  

• CASE

  GoObject v1.0

  The list of tokens.
  

• CHAN

  GoObject v1.0

  The list of tokens.
  

• CHAR

  GoObject v1.0

  'a'
  

• COLON

  GoObject v1.0

  :
  

• COMMA

  GoObject v1.0

  ,
  

• COMMENT

  GoObject v1.0

  The list of tokens.
  

• CONST

  GoObject v1.0

  The list of tokens.
  

• CONTINUE

  GoObject v1.0

  The list of tokens.
  

• DEC

  GoObject v1.0

  --
  

• DEFAULT

  GoObject v1.0

  The list of tokens.
  

• DEFER

  GoObject v1.0

  The list of tokens.
  

• DEFINE

  GoObject v1.0

  :=
  

• ELLIPSIS

  GoObject v1.0

  ...
  

• ELSE

  GoObject v1.0

  The list of tokens.
  

• EOF

  GoObject v1.0

  The list of tokens.
  

• EQL

  GoObject v1.0

  ==
  

• FALLTHROUGH

  GoObject v1.0

  The list of tokens.
  

• FLOAT

  GoObject v1.0

  123.45
  

• FOR

  GoObject v1.0

  The list of tokens.
  

• FUNC

  GoObject v1.0

  The list of tokens.
  

• GEQ

  GoObject v1.0

  >=
  

• GO

  GoObject v1.0

  The list of tokens.
  

• GOTO

  GoObject v1.0

  The list of tokens.
  

• GTR

  GoObject v1.0

  >
  

• IDENT

  GoObject v1.0

  Identifiers and basic type literals
  (these tokens stand for classes of literals)
  

• IF

  GoObject v1.0

  The list of tokens.
  

• ILLEGAL

  GoObject v1.0

  Special tokens
  

• IMAG

  GoObject v1.0

  123.45i
  

• IMPORT

  GoObject v1.0

  The list of tokens.
  

• INC

  GoObject v1.0

  ++
  

• INT

  GoObject v1.0

  12345
  

• INTERFACE

  GoObject v1.0

  The list of tokens.
  

• LAND

  GoObject v1.0

  &&
  

• LBRACE

  GoObject v1.0

  {
  

• LBRACK

  GoObject v1.0

  [
  

• LEQ

  GoObject v1.0

  <=
  

• LOR

  GoObject v1.0

  ||
  

• LPAREN

  GoObject v1.0

  (
  

• LSS

  GoObject v1.0

  <
  

• MAP

  GoObject v1.0

  The list of tokens.
  

• MUL

  GoObject v1.0

  *
  

• MUL_ASSIGN

  GoObject v1.0

  *=
  

• NEQ

  GoObject v1.0

  !=
  

• NOT

  GoObject v1.0

  !
  

• NoPos

  GoObject v1.0

  The zero value for Pos is NoPos; there is no file and line information
  associated with it, and NoPos.IsValid() is false. NoPos is always
  smaller than any other Pos value. The corresponding Position value
  for NoPos is the zero value for Position.
  

• OR

  GoObject v1.0

  |
  

• OR_ASSIGN

  GoObject v1.0

  |=
  

• PACKAGE

  GoObject v1.0

  The list of tokens.
  

• PERIOD

  GoObject v1.0

  .
  

• QUO

  GoObject v1.0

  /
  

• QUO_ASSIGN

  GoObject v1.0

  /=
  

• RANGE

  GoObject v1.0

  The list of tokens.
  

• RBRACE

  GoObject v1.0

  }
  

• RBRACK

  GoObject v1.0

  ]
  

• REM

  GoObject v1.0

  %
  

• REM_ASSIGN

  GoObject v1.0

  %=
  

• RETURN

  GoObject v1.0

  The list of tokens.
  

• RPAREN

  GoObject v1.0

  )
  

• SELECT

  GoObject v1.0

  The list of tokens.
  

• SEMICOLON

  GoObject v1.0

  ;
  

• SHL

  GoObject v1.0

  <<
  

• SHL_ASSIGN

  GoObject v1.0

  <<=
  

• SHR

  GoObject v1.0

  >>
  

• SHR_ASSIGN

  GoObject v1.0

  >>=
  

• STRING

  GoObject v1.0

  "abc"
  

• STRUCT

  GoObject v1.0

  The list of tokens.
  

• SUB

  GoObject v1.0

  -
  

• SUB_ASSIGN

  GoObject v1.0

  -=
  

• SWITCH

  GoObject v1.0

  The list of tokens.
  

• TILDE

  GoObject v1.0

  additional tokens, handled in an ad-hoc manner
  

• TYPE

  GoObject v1.0

  The list of tokens.
  

• VAR

  GoObject v1.0

  The list of tokens.
  

• XOR

  GoObject v1.0

  ^
  

• XOR_ASSIGN

  GoObject v1.0

  ^=
  

Functions, Macros, and Special Forms

• IsExported

  Function v1.0

  (IsExported name)
  

  IsExported reports whether name starts with an upper-case letter.
  
  Go input arguments: (name string)
  
  Go returns: bool
  
  Joker input arguments: [^String name]
  
  Joker returns: ^Boolean

• IsIdentifier

  Function v1.0

  (IsIdentifier name)
  

  IsIdentifier reports whether name is a Go identifier, that is, a non-empty
  string made up of letters, digits, and underscores, where the first character
  is not a digit. Keywords are not identifiers.
  
  Go input arguments: (name string)
  
  Go returns: bool
  
  Joker input arguments: [^String name]
  
  Joker returns: ^Boolean

• IsKeyword

  Function v1.0

  (IsKeyword name)
  

  IsKeyword reports whether name is a Go keyword, such as "func" or "return".
  
  Go input arguments: (name string)
  
  Go returns: bool
  
  Joker input arguments: [^String name]
  
  Joker returns: ^Boolean

• Lookup

  Function v1.0

  (Lookup ident)
  

  Lookup maps an identifier to its keyword token or IDENT (if not a keyword).
  
  Go input arguments: (ident string)
  
  Go returns: Token
  
  Joker input arguments: [^String ident]
  
  Joker returns: ^Token

• NewFileSet

  Function v1.0

  (NewFileSet)
  

  NewFileSet creates a new file set.
  
  Go returns: *FileSet
  
  Joker input arguments: []
  
  Joker returns: ^*FileSet

Types

• *File

  Concrete Type v1.0

  A File is a handle for a file belonging to a FileSet.
  A File has a name, size, and line offset table.
  

• AddLine

  Receiver for *File v1.0

  ([offset])
  

  AddLine adds the line offset for a new line.
  The line offset must be larger than the offset for the previous line
  and smaller than the file size; otherwise the line offset is ignored.
  

• AddLineColumnInfo

  Receiver for *File v1.0

  ([offset filename line column])
  

  AddLineColumnInfo adds alternative file, line, and column number
  information for a given file offset. The offset must be larger
  than the offset for the previously added alternative line info
  and smaller than the file size; otherwise the information is
  ignored.
  
  AddLineColumnInfo is typically used to register alternative position
  information for line directives such as //line filename:line:column.
  

• AddLineInfo

  Receiver for *File v1.0

  ([offset filename line])
  

  AddLineInfo is like AddLineColumnInfo with a column = 1 argument.
  It is here for backward-compatibility for code prior to Go 1.11.
  

• Base

  Receiver for *File v1.0

  ([])
  

  Base returns the base offset of file f as registered with AddFile.
  

• Line

  Receiver for *File v1.0

  ([p])
  

  Line returns the line number for the given file position p;
  p must be a Pos value in that file or NoPos.
  

• LineCount

  Receiver for *File v1.0

  ([])
  

  LineCount returns the number of lines in file f.
  

• LineStart

  Receiver for *File v1.0

  ([line])
  

  LineStart returns the Pos value of the start of the specified line.
  It ignores any alternative positions set using AddLineColumnInfo.
  LineStart panics if the 1-based line number is invalid.
  

• MergeLine

  Receiver for *File v1.0

  ([line])
  

  MergeLine merges a line with the following line. It is akin to replacing
  the newline character at the end of the line with a space (to not change the
  remaining offsets). To obtain the line number, consult e.g. Position.Line.
  MergeLine will panic if given an invalid line number.
  

• Name

  Receiver for *File v1.0

  ([])
  

  Name returns the file name of file f as registered with AddFile.
  

• Offset

  Receiver for *File v1.0

  ([p])
  

  Offset returns the offset for the given file position p;
  p must be a valid Pos value in that file.
  f.Offset(f.Pos(offset)) == offset.
  

• Pos

  Receiver for *File v1.0

  ([offset])
  

  Pos returns the Pos value for the given file offset;
  the offset must be <= f.Size().
  f.Pos(f.Offset(p)) == p.
  

• Position

  Receiver for *File v1.0

  ([p])
  

  Position returns the Position value for the given file position p.
  Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).
  

• PositionFor

  Receiver for *File v1.0

  ([p adjusted])
  

  PositionFor returns the Position value for the given file position p.
  If adjusted is set, the position may be adjusted by position-altering
  //line comments; otherwise those comments are ignored.
  p must be a Pos value in f or NoPos.
  

• SetLines

  Receiver for *File v1.0

  ([lines])
  

  SetLines sets the line offsets for a file and reports whether it succeeded.
  The line offsets are the offsets of the first character of each line;
  for instance for the content "ab\nc\n" the line offsets are {0, 3}.
  An empty file has an empty line offset table.
  Each line offset must be larger than the offset for the previous line
  and smaller than the file size; otherwise SetLines fails and returns
  false.
  Callers must not mutate the provided slice after SetLines returns.
  

• SetLinesForContent

  Receiver for *File v1.0

  ([content])
  

  SetLinesForContent sets the line offsets for the given file content.
  It ignores position-altering //line comments.
  

• Size

  Receiver for *File v1.0

  ([])
  

  Size returns the size of file f as registered with AddFile.
  

• *FileSet

  Concrete Type v1.0

  A FileSet represents a set of source files.
  Methods of file sets are synchronized; multiple goroutines
  may invoke them concurrently.
  
  The byte offsets for each file in a file set are mapped into
  distinct (integer) intervals, one interval [base, base+size]
  per file. Base represents the first byte in the file, and size
  is the corresponding file size. A Pos value is a value in such
  an interval. By determining the interval a Pos value belongs
  to, the file, its file base, and thus the byte offset (position)
  the Pos value is representing can be computed.
  
  When adding a new file, a file base must be provided. That can
  be any integer value that is past the end of any interval of any
  file already in the file set. For convenience, FileSet.Base provides
  such a value, which is simply the end of the Pos interval of the most
  recently added file, plus one. Unless there is a need to extend an
  interval later, using the FileSet.Base should be used as argument
  for FileSet.AddFile.
  

• AddFile

  Receiver for *FileSet v1.0

  ([filename base size])
  

  AddFile adds a new file with a given filename, base offset, and file size
  to the file set s and returns the file. Multiple files may have the same
  name. The base offset must not be smaller than the FileSet's Base(), and
  size must not be negative. As a special case, if a negative base is provided,
  the current value of the FileSet's Base() is used instead.
  
  Adding the file will set the file set's Base() value to base + size + 1
  as the minimum base value for the next file. The following relationship
  exists between a Pos value p for a given file offset offs:
  
  int(p) = base + offs
  
  with offs in the range [0, size] and thus p in the range [base, base+size].
  For convenience, File.Pos may be used to create file-specific position
  values from a file offset.
  

• Base

  Receiver for *FileSet v1.0

  ([])
  

  Base returns the minimum base offset that must be provided to
  AddFile when adding the next file.
  

• File

  Receiver for *FileSet v1.0

  ([p])
  

  File returns the file that contains the position p.
  If no such file is found (for instance for p == NoPos),
  the result is nil.
  

• Position

  Receiver for *FileSet v1.0

  ([p])
  

  Position converts a Pos p in the fileset into a Position value.
  Calling s.Position(p) is equivalent to calling s.PositionFor(p, true).
  

• PositionFor

  Receiver for *FileSet v1.0

  ([p adjusted])
  

  PositionFor converts a Pos p in the fileset into a Position value.
  If adjusted is set, the position may be adjusted by position-altering
  //line comments; otherwise those comments are ignored.
  p must be a Pos value in s or NoPos.
  

• *Pos

  Concrete Type v1.0

  Pos is a compact encoding of a source position within a file set.
  It can be converted into a Position for a more convenient, but much
  larger, representation.
  
  The Pos value for a given file is a number in the range [base, base+size],
  where base and size are specified when a file is added to the file set.
  The difference between a Pos value and the corresponding file base
  corresponds to the byte offset of that position (represented by the Pos value)
  from the beginning of the file. Thus, the file base offset is the Pos value
  representing the first byte in the file.
  
  To create the Pos value for a specific source offset (measured in bytes),
  first add the respective file to the current file set using FileSet.AddFile
  and then call File.Pos(offset) for that file. Given a Pos value p
  for a specific file set fset, the corresponding Position value is
  obtained by calling fset.Position(p).
  
  Pos values can be compared directly with the usual comparison operators:
  If two Pos values p and q are in the same file, comparing p and q is
  equivalent to comparing the respective source file offsets. If p and q
  are in different files, p < q is true if the file implied by p was added
  to the respective file set before the file implied by q.
  

• *Position

  Concrete Type v1.0

  Position describes an arbitrary source position
  including the file, line, and column location.
  A Position is valid if the line number is > 0.
  

• IsValid

  Receiver for *Position v1.0

  ([])
  

  IsValid reports whether the position is valid.
  

• *Token

  Concrete Type v1.0

  Token is the set of lexical tokens of the Go programming language.
  

• File

  Concrete Type v1.0

  A File is a handle for a file belonging to a FileSet.
  A File has a name, size, and line offset table.
  

• FileSet

  Concrete Type v1.0

  A FileSet represents a set of source files.
  Methods of file sets are synchronized; multiple goroutines
  may invoke them concurrently.
  
  The byte offsets for each file in a file set are mapped into
  distinct (integer) intervals, one interval [base, base+size]
  per file. Base represents the first byte in the file, and size
  is the corresponding file size. A Pos value is a value in such
  an interval. By determining the interval a Pos value belongs
  to, the file, its file base, and thus the byte offset (position)
  the Pos value is representing can be computed.
  
  When adding a new file, a file base must be provided. That can
  be any integer value that is past the end of any interval of any
  file already in the file set. For convenience, FileSet.Base provides
  such a value, which is simply the end of the Pos interval of the most
  recently added file, plus one. Unless there is a need to extend an
  interval later, using the FileSet.Base should be used as argument
  for FileSet.AddFile.
  

• Pos

  Concrete Type v1.0

  Pos is a compact encoding of a source position within a file set.
  It can be converted into a Position for a more convenient, but much
  larger, representation.
  
  The Pos value for a given file is a number in the range [base, base+size],
  where base and size are specified when a file is added to the file set.
  The difference between a Pos value and the corresponding file base
  corresponds to the byte offset of that position (represented by the Pos value)
  from the beginning of the file. Thus, the file base offset is the Pos value
  representing the first byte in the file.
  
  To create the Pos value for a specific source offset (measured in bytes),
  first add the respective file to the current file set using FileSet.AddFile
  and then call File.Pos(offset) for that file. Given a Pos value p
  for a specific file set fset, the corresponding Position value is
  obtained by calling fset.Position(p).
  
  Pos values can be compared directly with the usual comparison operators:
  If two Pos values p and q are in the same file, comparing p and q is
  equivalent to comparing the respective source file offsets. If p and q
  are in different files, p < q is true if the file implied by p was added
  to the respective file set before the file implied by q.
  

• IsValid

  Receiver for Pos v1.0

  ([])
  

  IsValid reports whether the position is valid.
  

• Position

  Concrete Type v1.0

  Position describes an arbitrary source position
  including the file, line, and column location.
  A Position is valid if the line number is > 0.
  

• String

  Receiver for Position v1.0

  ([])
  

  String returns a string in one of several forms:
  
  file:line:column valid position with file name
  file:line valid position with file name but no column (column == 0)
  line:column valid position without file name
  line valid position without file name and no column (column == 0)
  file invalid position with file name
  - invalid position without file name
  

• Token

  Concrete Type v1.0

  Token is the set of lexical tokens of the Go programming language.
  

• IsKeyword

  Receiver for Token v1.0

  ([])
  

  IsKeyword returns true for tokens corresponding to keywords;
  it returns false otherwise.
  

• IsLiteral

  Receiver for Token v1.0

  ([])
  

  IsLiteral returns true for tokens corresponding to identifiers
  and basic type literals; it returns false otherwise.
  

• IsOperator

  Receiver for Token v1.0

  ([])
  

  IsOperator returns true for tokens corresponding to operators and
  delimiters; it returns false otherwise.
  

• Precedence

  Receiver for Token v1.0

  ([])
  

  Precedence returns the operator precedence of the binary
  operator op. If op is not a binary operator, the result
  is LowestPrecedence.
  

• String

  Receiver for Token v1.0

  ([])
  

  String returns the string corresponding to the token tok.
  For operators, delimiters, and keywords the string is the actual
  token character sequence (e.g., for the token ADD, the string is
  "+"). For all other tokens the string corresponds to the token
  constant name (e.g. for the token IDENT, the string is "IDENT").
  

• arrayOfFile

  Concrete Type v1.0

  A File is a handle for a file belonging to a FileSet.
  A File has a name, size, and line offset table.
  

• arrayOfFileSet

  Concrete Type v1.0

  A FileSet represents a set of source files.
  Methods of file sets are synchronized; multiple goroutines
  may invoke them concurrently.
  
  The byte offsets for each file in a file set are mapped into
  distinct (integer) intervals, one interval [base, base+size]
  per file. Base represents the first byte in the file, and size
  is the corresponding file size. A Pos value is a value in such
  an interval. By determining the interval a Pos value belongs
  to, the file, its file base, and thus the byte offset (position)
  the Pos value is representing can be computed.
  
  When adding a new file, a file base must be provided. That can
  be any integer value that is past the end of any interval of any
  file already in the file set. For convenience, FileSet.Base provides
  such a value, which is simply the end of the Pos interval of the most
  recently added file, plus one. Unless there is a need to extend an
  interval later, using the FileSet.Base should be used as argument
  for FileSet.AddFile.
  

• arrayOfPos

  Concrete Type v1.0

  Pos is a compact encoding of a source position within a file set.
  It can be converted into a Position for a more convenient, but much
  larger, representation.
  
  The Pos value for a given file is a number in the range [base, base+size],
  where base and size are specified when a file is added to the file set.
  The difference between a Pos value and the corresponding file base
  corresponds to the byte offset of that position (represented by the Pos value)
  from the beginning of the file. Thus, the file base offset is the Pos value
  representing the first byte in the file.
  
  To create the Pos value for a specific source offset (measured in bytes),
  first add the respective file to the current file set using FileSet.AddFile
  and then call File.Pos(offset) for that file. Given a Pos value p
  for a specific file set fset, the corresponding Position value is
  obtained by calling fset.Position(p).
  
  Pos values can be compared directly with the usual comparison operators:
  If two Pos values p and q are in the same file, comparing p and q is
  equivalent to comparing the respective source file offsets. If p and q
  are in different files, p < q is true if the file implied by p was added
  to the respective file set before the file implied by q.
  

• arrayOfPosition

  Concrete Type v1.0

  Position describes an arbitrary source position
  including the file, line, and column location.
  A Position is valid if the line number is > 0.
  

• arrayOfToken

  Concrete Type v1.0

  Token is the set of lexical tokens of the Go programming language.