Title: Theory / Unimportant / CS / Programming / Python / Lib / Pygment
Tags: automatedNewLineAfterCommaSpace constant

Name: Pygments
Definition: a generic syntax highlighter suitable for use in code hosting, forums, wikis or other applications that need to prettify source code
Page: https://pygments.org/

Question: Advantage of Pygments
Answer: a wide range of over 500 languages and other text formats is supported
Page: https://pygments.org/
Answer2: special attention is paid to details that increase highlighting quality
Answer3: support for new languages and formats are added easily; most languages use a simple regex-based lexing mechanism
Answer4: a number of output formats is available, among them HTML, RTF, LaTeX and ANSI sequences
Answer5: it is usable as a command-line tool and as a library

Instruction: pygmentize
Page: https://pygments.org/
Language: sh
Returns: print on stdout the content of stdin with syntax highlighting. Language is guessed automatically, formatted for terminal.

Fla: -f formatter
Effect: use output format formatter
Page: https://pygments.org/
Prefix: pygmentize foo
Default: terminal

Fla: -o output
Effect: put output it file output
Page: https://pygments.org/
Prefix: pygmentize foo
Default: stdout

Fla: -s
Effect: Analyze code one line at a time
Page: https://pygments.org/
Prefix: pygmentize foo

Fla: -L
Page: https://pygments.org/
Prefix: pygmentize
Print: print all lexer, styles formatters and filters

Fla: -l lexer
Effect: uses lexer
Page: https://pygments.org/
Prefix: pygmentize
Default: guessed from context/extension. Same as -g

Fla: -g
Effect: attempt to guess the best lexer depending on extension and input
Page: https://pygments.org/
Prefix: pygmentize

Fla: filename
Effect: use filename as input
Page: man
Prefix: pygmentize
Default: stdin

Fla: -x
Effect: uses custom lexer and formatter. Guess their path from the formatter input
Page: man
Prefix: pygmentize

Fla: -O param_1=val_1,...,param_n=val_n
Effect: Give the lexer and formatter options param_i with values val_i
Page: man
Prefix: pygmentize

Fla: -P param=value
Effect: Give the lexer and formatter options param with value val
Page: man
Prefix: pygmentize

Fla: -F
Effect: add filter to the token stream???
Page: man
Prefix: pygmentize

Fla: -S <style>
Effect: print out style definitions for style <style>  for formatter <formatter>
Page: man
Prefix: pygmentize -f <formatter>

Fla: -N file
Effect: guesses and prints out a lexer name based solely on
the given filename
Page: man
Prefix: pygmentize

Fla: -H <type> <name>
Effect: prints detailed help for the object <name> of type <type>
Page: man
Prefix: pygmentize
Assuming: <type> is one of "lexer", "formatter" or "filter"

Fla: -v
Page: man
Prefix: pygmentize
Print: detailed traceback on unhandled exceptions
Intuition: debugging and bug reports

Fla: -h
Page: man
Prefix: pygmentize
Print: help

Fla: -V
Page: man
Prefix: pygmentize
Print: package version

Typ: Style
Meaning: A description of the way code should be displayed
Page: https://pygments.org/docs/styles/

Variable: default_style
Meaning: Style applied to all elements
Page: https://pygments.org/docs/styles/
Context: Style overriding
Language: python

Variable: styles
Meaning: map from Token to their style
Page: https://pygments.org/docs/styles/
Context: Style overriding
Language: python

Variable: StylenameClass.py
Meaning: the file containing the style Stylename
Page: https://pygments.org/docs/styles/
Language: python

Question: How to add a style
Answer: either register it as a plugin (see the plugin docs)
Page: https://pygments.org/docs/styles/
Context: Pygment
Answer2: or drop it into the styles subpackage of your Pygments distribution one style class per style, where the file name is the style name and the class name is StylenameClass. For example, if your style should be called "mondrian", name the class MondrianStyle, put it into the file mondrian.py and this file into the pygments.styles subpackage directory.

Instruction: bold
Page: https://pygments.org/docs/styles/
Effect: render text as bold
Language: How to add a style
Context: Pygment style rules

Instruction: nobold
Page: https://pygments.org/docs/styles/
Effect: cancel a bold instruction
Language: How to add a style
Context: Pygment style rules

Instruction: noitalic
Page: https://pygments.org/docs/styles/
Effect: cancel an italic instruction
Language: How to add a style
Context: Pygment style rules

Instruction: italic
Page: https://pygments.org/docs/styles/
Effect: render as italic
Language: How to add a style
Context: Pygment style rules

Instruction: underline
Page: https://pygments.org/docs/styles/
Effect: underline the text
Language: How to add a style
Context: Pygment style rules

Instruction: nounderline
Page: https://pygments.org/docs/styles/
Effect: cancel an underline instruction
Language: How to add a style
Context: Pygment style rules

Instruction: border:
Page: https://pygments.org/docs/styles/
Effect: no border
Language: How to add a style
Context: Pygment style rules

Instruction: bg:
Page: https://pygments.org/docs/styles/
Effect: no background
Language: How to add a style
Context: Pygment style rules

Instruction: bg:#hex_color
Page: https://pygments.org/docs/styles/
Effect: background in color hex_color
Language: How to add a style
Context: Pygment style rules

Instruction: border:#hex_color
Page: https://pygments.org/docs/styles/
Effect: border in color hex_color
Language: How to add a style
Context: Pygment style rules

Instruction: #hex_color
Page: https://pygments.org/docs/styles/
Effect: text color hex_color
Language: How to add a style
Context: Pygment style rules

Constant: noinherit
Page: https://pygments.org/docs/styles/
Language: How to add a style
Context: Pygment style rules

Instruction: "style_1 .... style_n"
Page: https://pygments.org/docs/styles/
Effect: apply all of style_1 ... style_n rules
Language: How to add a style
Context: Pygment style rules

Typ: Filter
Function: filter(self, lexer, stream): generate the
Page: https://pygments.org/docs/filters/
Context: Pygments

Instruction: CodeTagFilter(l)
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: All words of l are highlighted as code tag

Instruction: KeywordCaseFilter('lower')
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Convert keywords to lowercase
Abbreviation: KeywordCaseFilter()

Instruction: KeywordCaseFilter('upper')
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Convert keywords to uppercase

Instruction: KeywordCaseFilter('capitalize')
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Capitalize eyword

Instruction: NameHighlightFilter(
  l,
  tokenType
)
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Highlights the name of list l as a tokenType

Instruction: RaiseOnErrorTokenFilter()
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Filter that raise an exception when the lexer generates an error token.

Instruction: VisibleWhitespaceFilter()
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Filter that replace white space by some symbols (depending on options)

Instruction: GobbleFilter(n)
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Remove the n first characters of each line
Intuition: When the source contains n useless symbol on each line

Instruction: TokenMergeFilter()
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: Merges consecutive tokens with the same token type in the output stream of a lexer.

Instruction: SymbolFilter()
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: convert isabelle symbol into unicode characters when possible

Instruction: SymbolFilter('latex')
Page: https://pygments.org/docs/filters/
Language: Python
Context: Pygments filter
Returns: convert latex symbol into unicode characters when possible

Variable: encoding
Meaning: the encoding of input
Page: https://pygments.org/docs/formatters/
Context: Pygments formatters
Language: Python
Initialization: unicode

Variable: outencoding
Meaning: the encoding of output
Page: https://pygments.org/docs/formatters/
Context: Pygments formatters
Language: Python
Initialization: encoding

Variable: formatters
Meaning: todo
Page: https://pygments.org/docs/formatters/
Context: Pygments formatters
Language: Python

Typ: Token
Meaning: A representation of a kind of value. It represents the semantic meaning of the element in the code and is associated with a way to represent the code in style
Page: https://pygments.org/docs/tokens/
Language: Python
Context: Pygments

Question: Why is it better to use default pygments tokens
Answer: Because they are already considered in styles and can be displayed properly
Page: https://pygments.org/docs/tokens/
Context: Pygments

Constant: Token.Text
Meaning: for any type of text data
Page: https://pygments.org/docs/tokens/

Constant: Token.Text.Whitespace
Meaning: for specially highlighted whitespace
Page: https://pygments.org/docs/tokens/

Constant: Token.Error
Meaning: represents lexer errors
Page: https://pygments.org/docs/tokens/

Constant: Token.Other
Meaning: special token for data not matched by a parser (e.g. HTML markup in PHP code)
Page: https://pygments.org/docs/tokens/

Constant: Token.Keyword
Meaning: any kind of keywords
Page: https://pygments.org/docs/tokens/

Constant: Token.Name
Meaning: variable/function names
Page: https://pygments.org/docs/tokens/

Constant: Token.Literal
Meaning: Any literals
Page: https://pygments.org/docs/tokens/

Constant: Token.Literal.String
Meaning: string literals
Page: https://pygments.org/docs/tokens/

Constant: Token.Literal.Number
Meaning: number literals
Page: https://pygments.org/docs/tokens/

Constant: Token.Operator
Meaning: operators (+, not…)
Page: https://pygments.org/docs/tokens/

Constant: Token.Punctuation
Meaning: punctuation ([, (…)
Page: https://pygments.org/docs/tokens/

Constant: Token.Comment
Meaning: any kind of comments
Page: https://pygments.org/docs/tokens/

Constant: Token.Generic
Meaning: doesn’t really highlight a programming language but a patch file.
Page: https://pygments.org/docs/tokens

Instruction: is_token_subtype(
  t1,
  t2
)
Page: https://pygments.org/docs/tokens/
Language: python
Context: Pygments
Returns: Whether t1 is a specific kind of t2
Variables: token t1 t2
Abbreviation: t1 in t2

Instruction: pygments.lex(
  code,
  lexer
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: An iterable of token from code lexed with lexer

Instruction: format(
  tokens,
  formatter
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: A string formatted from tokens with formatter
Library: pygments

Instruction: format(
  code,
  lexer,
  formatter
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: A string, formatted from tokens with formatter applied to tokens from code lexed by lexer
Library: pygments

Instruction: get_lexer_by_name(
  alias,
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: an instance of a Lexer subclass that has alias in its aliases list. The lexer is given the options at its instantiation.
Library: pygments.lexer

Instruction: get_lexer_for_filename(
  fn,
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: a Lexer subclass instance that has a filename pattern matching fn. The lexer is given the options at its instantiation.
Library: pygments.lexer

Instruction: get_lexer_for_mimetype(
  mime,
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: a Lexer subclass instance that has mime in its mimetype list.
Library: pygments.lexer

Instruction: load_lexer_from_file(
  filename,
  lexername='CustomLexer',
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: a Lexer subclass instance loaded from the provided file, relative to the current directory.
Library: pygments.lexer

Instruction: guess_lexer(
  text,
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: a Lexer subclass instance that’s guessed from the text in text.
Library: pygments.lexer

Instruction: guess_lexer_for_filename(
  filename,
  **options
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: a Lexer subclass instance that’s guessed from  filename.
Library: pygments.lexer

Instruction: get_all_lexers()
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: Return an iterable over all registered lexers, yielding tuples in the format:
(longname, tuple of aliases, tuple of filename patterns, tuple of mimetypes)
Library: pygments.lexer

Instruction: find_lexer_class_by_name(alias)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: the Lexer subclass that has alias in its aliases list, without instantiating it.
Library: pygments.lexer

Instruction: find_lexer_class(name)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: the Lexer subclass that with the name attribute as given by the name argument.
Library: pygments.lexer

Instruction: formatter and style api
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments
Returns: todo?
Library: pygments.formatter

Question: Constraint on option for lexer, parser and style
Answer: they must be processable as string, in order to come from command line
Page: https://pygments.org/docs/api/
Context: Pygments

Instruction: lexer.get_tokens(text)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: an iterable of (tokentype, value) pairs from text.
Variables: Lexer lexer

Instruction: lexer.get_tokens_unprocessed(text)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: an iterable of (index, tokentype, value), where index is the starting position of the token within the input text
Variables: Lexer lexer

Instruction: lexer.analyse_text(text) (static)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: Return a float between 0 and 1 to indicate how good this lexer would be for this test
Variables: Lexer lexer

Instruction: pygment_class.name
Page: https://pygments.org/docs/api/
Effect: Full name for the lexer, in human-readable form.
Language: Python
Context: Pygments
Variables: pygment_class lexer, stle or formatter

Instruction: pygment_class.aliases
Page: https://pygments.org/docs/api/
Effect: A list of short, unique identifiers that can be used to lookup the lexer from a list, e.g. using get_lexer_by_name().
Language: Python
Context: Pygments
Variables: pygment_class lexer, stle or formatter

Instruction: lexer.filenames
Page: https://pygments.org/docs/api/
Effect: A list of fnmatch patterns that match filenames which contain content for this lexer.
Language: Python
Context: Pygments
Variables: Lexer lexer
Assuming: The patterns in this list should be unique among all lexers.

Instruction: lexer.alias_filenames
Page: https://pygments.org/docs/api/
Effect: A list of fnmatch patterns that match filenames which may or may not contain content for this lexer. This list is used by the guess_lexer_for_filename() function, to determine which lexers are then included in guessing the correct one. That means that e.g. every lexer for HTML and a template language should include \*.html in this list.
Language: Python
Context: Pygments
Variables: Lexer lexer

Instruction: lexer.mimetypes
Page: https://pygments.org/docs/api/
Effect: A list of MIME types for content that can be lexed with this lexer.
Language: Python
Context: Pygments
Variables: Lexer lexer

Instruction: formatter.get_style_defs(arg='')
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: statements or declarations suitable to define the current style for subsequent highlighted text (e.g. ).
Variables: Formatter formatter
Example: CSS classes in the HTMLFormatter

Instruction: formatter.format(
  tokensource,
  outfile
)
Page: https://pygments.org/docs/api/
Effect: format the tokens from the tokensource iterable and write the formatted version to the file object outfile.
Language: Python
Context: Pygments
Variables: Formatter formatter

Instruction: formatter.filenames
Page: https://pygments.org/docs/api/
Effect: A list of fnmatch patterns that match filenames for which this formatter can produce output. The patterns in this list should be unique among all formatters.
Language: Python
Context: Pygments
Variables: Formatter formatter

Constant: OptionError
Meaning: This exception will be raised by all option processing functions if the type or value of the argument is not correct.
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments

Instruction: get_bool_opt(
  optinos,
  optname,
  default=None
)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: Interpret the key optname from the dictionary options as a boolean and return it. Return default if optname is not in options.
Variables: Formatter formatter
Library: pygments.util

Instruction: get_int_opt(
  options,
  optname,
  default=None
)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: Interpret the key optname from the dictionary options as an int and return it. Return default if optname is not in options.
Variables: Formatter formatter
Library: pygments.util

Instruction: get_list_opt(
  options,
  optname,
  default=None
)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: Return list and tuple as themselve. Return the strings splitted at whitespace
Variables: Formatter formatter
Library: pygments.util
Intuition: allow to parse arguments from command line

Instruction: get_choice_opt(
  options,
  optname,
  allowed,
  default=None
)
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments
Returns: options.get(  optname, default) if it is in allowed
Variables: Formatter formatter
Library: pygments.util
Exception: OptionError if the value associated to optname is not present in allowed

Question: How to create a lexer
Answer: __init__ must pass its options to its parents
Page: https://pygments.org/docs/api/
Context: Pygments
Answer2: override get_tokens_unprocessed
Answer3: optional: override analyse_text to help determine when your lexer is good

Typ: RegexLexer
Meaning: Base type to override to create a simple lexer with state machine
Page: https://pygments.org/docs/api/
Language: Python
Context: Pygments

Variable: tokens
Meaning: map from state to lexing rules that can be applied in this state
Page: https://pygments.org/docs/api/
Context: Pygments
Language: Python
Typ: RegexLexer

Variable: tokens['state_name']
Meaning: List of lexing rules, apply them until one match
Page: https://pygments.org/docs/api/
Context: Pygments
Language: Python
Typ: RegexLexer

Instruction: (
  r'regexp',
  Token
)
Page: https://pygments.org/docs/api/
Effect: if regexp match the remaining of the lexed text, output the lexed text with Token. Stay in the same state
Language: Python
Context: Pygments lexing rule

Instruction: (
  r'regexp',
  Token,
  newstate
)
Page: https://pygments.org/docs/api/
Effect: if regexp match the remaining of the lexed text, output the lexed text with Token. Push newstate on the stack
Language: Python
Context: Pygments lexing rule

Instruction: (
  r'regexp',
  bygroups(Token1, ...,  Token_n)
)
Page: https://pygments.org/docs/api/
Effect: if regexp match the remaining of the lexed text, yeld captured value i with Token_i, unless it's none
Language: Python
Context: Pygments lexing rule
Assuming: as many capturing groups as Tokens

Question: Example of use of states in parsing
Answer: Switching between regular code, multiline comment, multiline string
Page: https://pygments.org/docs/api/
Context: Pygments lexing rule

Page: https://pygments.org/docs/api/
Syntactic sugar: (regex, action, "#pop")
Implementation: (regex, action, "#pop:1")
Language: python
Context: Pygments lexing rule

Instruction: (
  regex,
  action,
  "#pop:n"
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: same as (  regex, action) and pop two states from the stack

Instruction: (
  regex,
  action,
  "#push"
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: same as (  regex, action) and push the current state to the stack again
Intuition: If the rule is included in other state,
  it ensure the other state is pushed.

Instruction: (
  regex,
  action,
  (state1, ..., state_n)
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: same as (  regex, action) but do the actions related to state_1... and state_n

Instruction: include('state')
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: copy all of the rules of 'state' in the currently defined state

Instruction: combine(
  'state_1',
  ...,
  'state_n'
)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: a state where all rules of state_1 can apply, then if none match, all rules of ... state_n

Page: https://pygments.org/docs/api/
Syntactic sugar: default(new_state)
Implementation: ('', Text, new_state)
Language: python
Context: Pygments lexing rule
Intuition: ensure this rule is always taken, useful if all other rules failed

Instruction: pygments.lexer.inherits
Page: https://pygments.org/docs/api/
Effect: indicate to inherits the rules related to the same state in the parent
Language: python
Context: Pygments lexing rule
Where to use it: in a class inheriting from a class inheritic from RegexLexer
tokens['state'] {
...
}

Instruction: using(otherLexer)
Page: https://pygments.org/docs/api/
Language: python
Context: Pygments lexing rule
Returns: Instead of emitting matched text, pass it to otherLexer and yeld what it yelds

Instruction: DelegatingLexer(
  root_lexer,
  language_lexer
)
Page: https://pygments.org/docs/lexerdevelopment/
Language: python
Context: Pygments lexing rule
Returns: Apply root_lexer, and all of its Other token are sent to be lexed by language_lexer

Instruction: (
  regex,
  f
)
Page: https://pygments.org/docs/lexerdevelopment/
Language: python
Context: Pygments lexing rule
Returns: if regex match, yeld from f(  lexer, match)

Typ: ExtendedRegexLexer
Meaning: Extension of RegexLexer where callback can take a context as argument
Page: https://pygments.org/docs/lexerdevelopment/
Language: python
Context: Pygments lexing rule

Instruction: words(("word_1",...,"word_n"))
Page: https://pygments.org/docs/lexerdevelopment/
Language: python
Context: Pygments lexing rule
Returns: a regexp matching exactly the word_i's
Library: pygments.lexer

Question: How to deal with language with so many builtin that you can't just put them all in a regexp
Answer: consider builtin as regular function, then test in a filter/class inheriting the Lexer whether the function is a builtin or not to decide which token to return
Page: https://pygments.org/docs/lexerdevelopment/
Context: Pygments lexing rule

Constant: NullFormatter
Meaning: The formatter outputing each token as-is
Page: https://pygments.org/docs/formatterdevelopment/
Language: Python
Context: Pygments lexing rule

Instruction: formatter.format(
  tokensource,
  outfile
)
Page: https://pygments.org/docs/formatterdevelopment/
Effect: call outfile.write() on the formatted string created from TokenSource
Language: Python
Context: Pygments lexing rule
Variables: Formatter formatter

Instruction: @simplefilter
def fun_name(
  self,
  lexer,
  stream,
  options
):
  ...
Page: https://pygments.org/docs/filterdevelopment/
Meaning: A filter,
  that can be instantiated with fun_name(
  options
)
Language: Python
Context: Pygments
Identifier: fun_name

Typ: Filter
Function: filter(self, lexer, stream), with stream an iterable of token_type, value, generating (token_type, value)
Page: https://pygments.org/docs/filterdevelopment/
Context: Pygments
Extends: Python

Instruction: itemname = module:class
Page: https://pygments.org/docs/filterdevelopment/
Effect: ensure that Pygments know there is a lexer/formatter/style/filter called lexername in module/class
Language: setuptools
Context: Pygments

Instruction: [pygments.formatters/styles/filter/lexers]
Page: https://pygments.org/docs/plugins/
Effect: line below introduce formatters/styles/filter/lexers
Language: setuptools
Context: Pygments

Instruction: .. sourcecode:: language
  code
Page: https://pygments.org/docs/plugins/
Language: ReST
Context: Pygments
Returns: code, with syntax highlighting according to language