This is a first approach to make the test suite run again.

I think it's fine not to rely on a pytest-plugin that requires a templating language and run mypy directly instead.

However,

• that mypy run yields errors
• ~~at least locally the package can't be installed for Python 3.6~~ (not an issue w/ the CI platform)
• ~~at least locally~~ pytest doesn't run tests; i leave that to someone acquainted with yaml-based test definitions

Running mypy foo.py with lxml-stubs installed handles import lxml fine, but import lxml.html reports an error since it is not defined in lxml-stubs.

foo.py:2: error: Skipping analyzing 'lxml.html': found module but no type hints or library stubs
foo.py:2: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

foo.py file:

import lxml
import lxml.html

The problem I'm trying to solve is having a method that takes an element or elementtree as argument, and forwards that to a XPath object.

x = etree.XPath("//div")

def method(elem):
  // other stuff
  r = x(elem)
  // other stuff

If I do this with fx pylance, I would get the complaint that elem is Unknown. If I try to fix this by adding the type, I would have to write

def method(elem: typing.Union[etree._Element, etree._ElementTree]):

Which of course triggers that the types are private.

For easy fix I would suggest some simple typing aliases

TElement = _Element or something like that (Don't really care about the syntax - more about that the types would be public)

If I have the following

from lxml import etree
tree: etree._ElementTree = etree.parse(file_obj)

I get a pylint warning: W0212: Access to a protected member _ElementTree of a client class (protected-access)

I see _Element and _ElementTree are the lxml names, I gather a workaround is outside the scope of this project?

This simplified code reproduces an error I am seeing when running mypy over my project.

from lxml import etree


def fcn() -> None:
    namespaces = {"a": "http://www.w3.org/2001/XMLSchema"}
    etree.Element(
        "{{{scl_namespaces['xs']}}}pattern", attrib={"value": "0"}, nsmap=namespaces
    )


if __name__ == "__main__":
    fcn()

The following error is produced.

c:\>mypy --strict -m lxml_mypy
lxml_mypy.py:7: error: Argument "nsmap" to "Element" has incompatible type "Dict[str, str]"; expected "Union[Dict[Optional[bytes], bytes], Dict[Optional[str], str], None]"
Found 1 error in 1 file (checked 1 source file)

I am using:

• python 3.9.5
• lxml 4.6.3
• lxml-stubs 0.2.0

I did some searching and found TypeVar and TypedDict but neither of those seemed to be able to address the issue. Does anyone have suggestions on how to resolve this typing error?

Please let me know if you need more info and thanks in advance.

See https://github.com/lxml/lxml/pull/337

Also created alias _FileSource for valid types for file arguments. I'm not sure how this should be added since it was only added in the newest release of lxml.

You approved my PR #8 a few days ago to add types for CustomElementClassLookup.

The lookup method of that class, according to the docs should return a subclass of ElementBase or None. Because it is returning a class, not an instance of a class, the return type should be Optional[Type[ElementBase]].

This was changed to Optional[ElementBase] in c6120e4997aa59535d434b03f1433fdbd1f2fdc9. I think this change should be reverted as it's incorrect according to the documentation and the required implementation of the function.

Thanks for the help and for approving the original PR.

Although format checks can help users reading the code with more consistency, the way it's done currently is a burden for both contributor and maintainer alike.

1. In particular, it insists on changing function / method layout every now and then, making patch harder to read.
2. Besides, format error in github workflow worker can potentially hide typing error. If format error is found, mypy could stop checking as well, depending on the execution order.

mypy check is much more important, preventing code error to manifest in annotation for too long. Given how lxml-stubs is maintained currently, it's good enough for format check to be performed manually only once, just before tagged release.

Using TypeGuard from python 3.10 via typing-extensions. However, I noticed that lxml-stubs has no dependency on typing-extensions even though it's already used. Should this be added?

from lxml import etree

xml_string = b"""<?xml version="1.0" encoding="UTF-8"?>
<searchResults>
  <resultCount>59</resultCount>
  <Book>
    <bookId>30323794902</bookId>
  </Book>
</searchResults>
"""
root = etree.fromstring(xml_string)
books = root.findall("Book")
for book in books:
    print(book.findtext("bookId"))  # Unresolved attribute reference 'findtext' for class '_Element' 

it seems that there's already a lxml-stubs package on the PyPI.

i'm opening this issue because the situation will lead to confusion and the docs should warn about installing this package.

or could the stubs be integrated w/ the lxml package?

I'm not entirely sure whether always returning HtmlElement from the lxml.html.*fromstring functions is 100% correct – as far as I understand, the actual return type depends on the parser that is used, and therefore can actually be plain _Elements like it was before.

Do you think it makes sense to @overload these functions depending on the passed parser?

by this comment i felt strongly encouraged refactor code in order to use an empty string as key for the default namespace in a namespace mapping, but soon found inconsistencies with the current annotations and implementation.

first, mypy complains about this:

an_element = etree.fromstring("<element/>")
a_new_element = an_element.makeelement("new", nsmap=an_element.nsmap)

with error: Argument "nsmap" to "makeelement" of "_Element" has incompatible type "Dict[Optional[str], str]"; expected "Optional[Mapping[str, str]]".

then, the empty string as key isn't even the default:

In [1]: from lxml import etree
In [2]: t = etree.fromstring("<element xmlns='test'/>")
In[3]: t.nsmap.get(None)
Out[3]: 'test'
In[4]: t.nsmap.get("")
Out[[4]:

i have no proposal how to solve these issues, but i consider the first demo a bug and the latter at least confusing.

Have been feeding on my own dogfood for quite a while, the annotation of XPathObject as XPath evaluation result is one of the spots that I feel constantly irritated. Although the annotation itself is correct per se, its presence turns out to be a nuisance for developers. The problem is, it is a long union of so many types that the selection result can never be used directly. For xpath evluation result to be useful, it has to be narrowed down to specific type(s) with approaches like:

• try-except block
• isinstance() check
• assert

All of them have one thing in common: throw away the type supplied by stub, and perform manual type narrowing afterwards. It is more like a roadblock rather than something that helps. As a supporting example, elementpath package returns Any as evaluation result even when the package is considered fully annotated.

Input argument used by variable inside xpath expression is different though, as that doesn't need extra processing and isn't as complex as output types.

My suggestion is set XPathObject as alias to Any, and cleanup input arguments as another alias, like:

_XPathObject = Any
_XPathVarArg = Union[...]
class XPath:
    def __call__(self,
        _etree_or_element: _ElementOrTree,
        **_variables: _XPathVarArg
    ) -> _XPathObject: ...

According to the documentation it is possible to validate an ElementTree object against an XML schema using etree.XMLSchema.validate() method or using it as a callable. In both cases, mypy traces an error. E.g.:

import sys

from lxml import etree


def main() -> None:
    try:
        schema_file = sys.argv[1]
        file = sys.argv[2]
    except IndexError:
        sys.exit("Missing argument(s)")

    schema = etree.XMLSchema(etree.parse(schema_file))
    doc = etree.parse(file)

    if schema(doc):
        print(f"{file} is valid")
    else:
        print(f"{file} is not valid")

    if schema.validate(doc):
        print(f"{file} is valid")
    else:
        print(f"{file} is not valid")


if __name__ == '__main__':
    main()

% mypy main.py     
main.py:16: error: "XMLSchema" not callable
main.py:21: error: "XMLSchema" has no attribute "validate"