The tokenizer namespace deals with splitting text or files into tokens, typically for parsing data, or data-scraping from data files or websites.

NoReturn tok:Load(String text)
Loads the given text and splits it into tokens that can be used by the tok functions.

String tok:LoadUrl(String text)
Loads text from the given URL into the tokenizer.

Number tok:GetBookmark()
Returns a bookmark to remember the current tokenizer location.

NoReturn tok:SetBookmark(Number obj)
Jumps to the given bookmark location in the tokenizer.

String tok:GetText(Number bookmark1, Number bookmark2)
Retreves the text between the two given bookmarks.

NoReturn tok:LoadEx(String text, Boolean indicateLines, String letterChars)
Loads the given text into the tokenizer, optionally recognizing line returns (indicateLines), and using letterChars to split tokens.

Boolean tok:Closer(String text1, String text2)
Returns true if text1 is closer in the tokenizer than text2.

Boolean tok:EOF()
Returns true if the tokenizer has reached the end-of-file.

Boolean tok:Exists(String text)
Returns true if the given text token is in the tokenizer.

String tok:Where()
Returns a line of text that shows where the current tokenizer cursor is.

Number tok:SkipTo(String text)
Skips the tokenizer cursor to the given text token.

NoReturn tok:SkipBackTo(String text)
Skips the tokenizer cursor back to the given text token.

String tok:Peek()
Retrieves the current token from the tokenizer without moving the cursor.

String tok:Next()
Retrueves the next token from the tokenizer.

NoReturn tok:BackOne()
Moves the tokenizer cursor back one.

Number tok:NextNum()
Retrieves the next token as a number.

DateTime tok:NextDate()
Retrieves the next token as a date.

String tok:NextTo(String text)
Retrieves the text between the cursor position and the next token that matches text.

NoReturn tok:MustBe(String text)
Checks that the next token is text, if not it throws an error, if it is then it moves the cursor on one.

Boolean tok:IsNext(String text)
Returns true if the next token is text.