Title: Theory / Unimportant / CS / Programming / Python / Lib / urllib
Tags: automatedNewLineAfterCommaSpace
On every note of this deck, unless a note says otherwise:
Context: Python
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse

Name: urllib
Definition: a package that collects several modules for working with URLs
Page: https://docs.python.org/3/library/urllib.html
Context: Python

Typ: URLError
Meaning: The handlers raise this exception (or derived exceptions) when they run into a problem
Page: https://docs.python.org/3/library/urllib.error.html#module-urllib.error
Language: Python
Subtype of: OSError.
Library: urllib.error

Typ: HTTPError
Meaning: non-exceptional file-like return value (the same thing that urlopen() returns). This is useful when handling exotic HTTP errors, such as requests for authentication.
Page: https://docs.python.org/3/library/urllib.error.html#module-urllib.error
Language: Python
Subtype of: URLError
Library: urllib.error

Typ: ContentTooShortError
Meaning: raised when the urlretrieve() function detects that the amount of the downloaded data is less than the expected amount
Page: https://docs.python.org/3/library/urllib.error.html#module-urllib.error
Language: Python
Library: urllib.error

Name: urllib.parse
Definition: Parse URLs into components
Page: https://docs.python.org/3/library/urllib.error.html#module-urllib.error
Context: Python

Instruction: urlparse(
  urlstring,
  scheme='',
  allow_fragments=True
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: scheme, netloc, hostname, port, path, fragment of urlstring
Typ: ParseResult
Library: urllib.parse

Typ: ParseResult
Meaning: Information obtained from a URL
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Context: Python
Library: urllib.parse

Variable: parseResult.scheme
Meaning: URL scheme specifier
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.netloc
Meaning: Network location part
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.path
Meaning: Hierarchical path
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.params
Meaning: always an empty string
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.query
Meaning: Query component
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.fragment
Meaning: Fragment identifier
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.username
Meaning: User name
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.password
Meaning: Password
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.hostname
Meaning: Host name (lower case)
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Variable: parseResult.port
Meaning: Port number as integer, if present
Context: Python
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: parse_qs(
  qs,
  keep_blank_values=False,
  strict_parsing=False,
  encoding='utf-8',
  errors='replace',
  max_num_fields=None,
  separator='&'
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: todo
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: parse_ParseResult
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: todo
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: urlunparse(parts)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: a url from a urlparse's result
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: urlsplit(
  urslstring,
  scheme='',
  allow_fragments=True
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: a named enum describing urlstring for recent URL norm
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: urlunsplit(parts)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: url from the result of urlsplit
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: urljoin(
  base,
  url,
  allow_fragments=True
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: a full (  “absolute”) URL by combining a “base URL” (  base) with another URL (  url).
Variables: ParseResult parseResult
Library: urllib.parse

Instruction: urldefrag(url)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: a tuple with url and fragment
Library: urllib.parse

Instruction: unwrap(url)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: the url that is wrapped, if it the case, othewise url itself
Library: urllib.parse

Name: A wrapped URL
Definition: <URL:scheme://host/path>, <scheme://host/path>, URL:scheme://host/path or scheme://host/path
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Chapter: urllib.parse
Context: Python

Instruction: quote(
  string,
  safe='/',
  encoding=None,
  errors=None
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: string so that it can be used in url

Instruction: quote_plus(
  string,
  safe='/',
  encoding=None,
  errors=None
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: string so that it can be used in url, replacing spaces with plus signr

Instruction: quote_from_bytes(
  bytes,
  safe='/'
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: quote a bytes object

Instruction: unquote_foo(
  string,
  encoding='utf-8',
  errors='replace'
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: reverse quote_foo

Instruction: urlencode(
  [(arg_1, value_1),
    ...,
    (arg_n, value_n)
  ],
  doseq=False,
  safe='',
  encoding=None,
  errors=None,
  quote_via=quote_plus
)
Page: https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
Returns: a percent-encoded ASCII text string for url associating value_i to argument arg_i

Typ: RobotFileParser
Meaning: answers questions about whether or not a particular user agent can fetch a URL on the web site that published the robots.txt file
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Context: Python

Instruction: robotFileParser.set_url(url)
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Effect: Sets the URL referring to a robots.txt file.
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.read()
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Effect: Reads the robots.txt URL and feeds it to the parser.
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.parse(lines)
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Effect: Parses the lines argument.
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.can_fetch(
  useragent,
  url
)
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Returns: whether the useragent is allowed to fetch the url according to the rules contained in the parsed robots.txt file.
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.mtime()
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Returns: the time the robots.txt file was last fetched
Variables: RobotFileParser robotFileParser
Extra: This is useful for long-running web spiders that need to check for new robots.txt files periodically.

Instruction: robotFileParser.modified()
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Effect: Sets the time the robots.txt file was last fetched to the current time.
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.crawl_delay(useragent)
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Returns: the value of the Crawl-delay parameter from robots.txt for the useragent in question or None
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.request_rate(useragent)
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Returns: the contents of the Request-rate parameter from robots.txt
Variables: RobotFileParser robotFileParser

Instruction: robotFileParser.site_maps()
Page: https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
Returns: the contents of the Sitemap parameter from robots.txt in the form of a list()
Variables: RobotFileParser robotFileParser