Namespace: go.std.text.scanner

v1.0

Contents

• Summary
• Index
• Constants
• Variables
• Functions, Macros, and Special Forms
• Types

Summary

Provides a low-level interface to the text/scanner package.

Package scanner provides a scanner and tokenizer for UTF-8-encoded text.
It takes an io.Reader providing the source, which then can be tokenized
through repeated calls to the Scan function. For compatibility with
existing tools, the NUL character is not allowed. If the first character
in the source is a UTF-8 encoded byte order mark (BOM), it is discarded.

By default, a Scanner skips white space and Go comments and recognizes all
literals as defined by the Go language specification. It may be
customized to recognize only a subset of those literals and to recognize
different identifier and white space characters.

Index

• *Position
• *Scanner
• Char
• Comment
• EOF
• Float
• GoTokens
• GoWhitespace
• Ident
• Int
• Position
• RawString
• ScanChars
• ScanComments
• ScanFloats
• ScanIdents
• ScanInts
• ScanRawStrings
• ScanStrings
• Scanner
• SkipComments
• String
• TokenString
• arrayOfPosition
• arrayOfScanner

Legend

Constant Variable Function Macro Special form Type GoVar Receiver/Method

Constants

Constants are variables with :const true in their metadata. Joker currently does not recognize them as special; as such, it allows redefining them or their values.

• Char

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• Comment

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• EOF

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• Float

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• GoTokens

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• GoWhitespace

  Int v1.0

  GoWhitespace is the default value for the Scanner's Whitespace field.
  Its value selects Go's white space characters.
  

• Ident

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• Int

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• RawString

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

• ScanChars

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• ScanComments

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• ScanFloats

  Int v1.0

  includes Ints and hexadecimal floats
  

• ScanIdents

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• ScanInts

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• ScanRawStrings

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• ScanStrings

  Int v1.0

  Predefined mode bits to control recognition of tokens. For instance,
  to configure a Scanner such that it only recognizes (Go) identifiers,
  integers, and skips comments, set the Scanner's Mode field to:
  
  ScanIdents | ScanInts | SkipComments
  
  With the exceptions of comments, which are skipped if SkipComments is
  set, unrecognized tokens are not ignored. Instead, the scanner simply
  returns the respective individual characters (or possibly sub-tokens).
  For instance, if the mode is ScanIdents (not ScanStrings), the string
  "foo" is scanned as the token sequence '"' Ident '"'.
  
  Use GoTokens to configure the Scanner such that it accepts all Go
  literal tokens including Go identifiers. Comments will be skipped.
  

• SkipComments

  Int v1.0

  if set with ScanComments, comments become white space
  

• String

  Int v1.0

  The result of Scan is one of these tokens or a Unicode character.
  

Variables

(None.)

Functions, Macros, and Special Forms

• TokenString

  Function v1.0

  (TokenString tok)
  

  TokenString returns a printable string for a token or Unicode character.
  
  Go input arguments: (tok rune)
  
  Go returns: string
  
  Joker input arguments: [^Char tok]
  
  Joker returns: ^String

Types

• *Position

  Concrete Type v1.0

  Position is a value that represents a source position.
  A position is valid if Line > 0.
  

• IsValid

  Receiver for *Position v1.0

  ([])
  

  IsValid reports whether the position is valid.
  

• *Scanner

  Concrete Type v1.0

  A Scanner implements reading of Unicode characters and tokens from an io.Reader.
  

• Init

  Receiver for *Scanner v1.0

  ([src])
  

  Init initializes a Scanner with a new source and returns s.
  Error is set to nil, ErrorCount is set to 0, Mode is set to GoTokens,
  and Whitespace is set to GoWhitespace.
  

• IsValid

  Receiver for *Scanner v1.0

  ([])
  

  IsValid reports whether the position is valid.
  

• Next

  Receiver for *Scanner v1.0

  ([])
  

  Next reads and returns the next Unicode character.
  It returns EOF at the end of the source. It reports
  a read error by calling s.Error, if not nil; otherwise
  it prints an error message to os.Stderr. Next does not
  update the Scanner's Position field; use Pos() to
  get the current position.
  

• Peek

  Receiver for *Scanner v1.0

  ([])
  

  Peek returns the next Unicode character in the source without advancing
  the scanner. It returns EOF if the scanner's position is at the last
  character of the source.
  

• Pos

  Receiver for *Scanner v1.0

  ([])
  

  Pos returns the position of the character immediately after
  the character or token returned by the last call to Next or Scan.
  Use the Scanner's Position field for the start position of the most
  recently scanned token.
  

• Scan

  Receiver for *Scanner v1.0

  ([])
  

  Scan reads the next token or Unicode character from source and returns it.
  It only recognizes tokens t for which the respective Mode bit (1<<-t) is set.
  It returns EOF at the end of the source. It reports scanner errors (read and
  token errors) by calling s.Error, if not nil; otherwise it prints an error
  message to os.Stderr.
  

• String

  Receiver for *Scanner v1.0

  ([])
  

• TokenText

  Receiver for *Scanner v1.0

  ([])
  

  TokenText returns the string corresponding to the most recently scanned token.
  Valid after calling Scan and in calls of Scanner.Error.
  

• Position

  Concrete Type v1.0

  Position is a value that represents a source position.
  A position is valid if Line > 0.
  

• String

  Receiver for Position v1.0

  ([])
  

• Scanner

  Concrete Type v1.0

  A Scanner implements reading of Unicode characters and tokens from an io.Reader.
  

• String

  Receiver for Scanner v1.0

  ([])
  

• arrayOfPosition

  Concrete Type v1.0

  Position is a value that represents a source position.
  A position is valid if Line > 0.
  

• arrayOfScanner

  Concrete Type v1.0

  A Scanner implements reading of Unicode characters and tokens from an io.Reader.