https://www.python.org/dev/peps/pep-0563/#runtime-annotation-resolution-and-class-decorators it can throw NameError on metaclasses

@deathowl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Pull Request Test Coverage Report for Build 742010833

• 10 of 11 (90.91%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.02%) to 94.135%

Totals
Change from base Build 708713380:	-0.02%
Covered Lines:	2568
Relevant Lines:	2728

💛 - Coveralls

What:

Extend the builtin testing to cover function/methods created by Cython 3.0 Exempt builtins from signature validation since its mostly broken for builtins around keyword arguments. Changed inspect.iscoroutinefunction to asyncio.iscoroutinefunction

Why:

at Facebook I am trying to upgrade from Cython 0.29.x to Cython 3.0a6. Many things change under the hood that cause things like inspect module to behave differently. One thing Cython functions and methods are no longer "method_descriptors" they are a new thing called "cython_function_or_method"

Also inspect.signature on methods/functions in Cython 0.29.x would raise ValueError. In Cython 3.0 we get a signature but it believes keyword arguments are all required. I remember my last pull request for cpython there was this allow/deny list for opting out certain builtins for signature validation. I figure if even the stdlib can't rely on testing the signatures of builtins then testslide can make do with ignoring signature validation on builtins.

Also around the iscoroutinefunction. The next Cython 3.0 release "3.0a7" async def functions will return True when used with asyncio.iscoroutinefunction, but will return False when used with inspect.iscoroutinefunction. Since the asyncio variant also calls the inspect variant it seems wise to use it instead.

How:

Extend the previous builtins detection logic to include the new cython_function_or_method. Also use this logic for exempting builtins from _validate_callable_signature. Cleaned up some code copies by extracting the logic into a function.

Risks:

Some builtin using C-api might return valid signatures from a inspect.signature call.

Checklist:

• [ ] Added tests, if you've added code that should be tested
• [ ] Updated the documentation, if you've changed APIs
• [X] Ensured the test suite passes
• [X] Made sure your code lints
• [X] Completed the Contributor License Agreement ("CLA")

With these changes I was able to get pre-written testslide based tests to pass on cython modules built under 3.0a6

KeyError seemed more specific to the modules with different __module__ . Are there any other failures that expose a bug with get_type_hints? I couldn't see the test failures since it's private.

can we do except Exception just to make sure we dont break tests when we fail to get typehints for any funny modules?

@deathowl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Thanks, I will make the changes. On further search seems this is an open issue in CPython where there is a PR open to catch the KeyError and discard it for these type of cases : https://bugs.python.org/issue41515

I guess catching key error and returning empty annotations would work well as a workaround for us as well

Thanks, this seems to have caused an issue in sphinx as well where they catch keyerror and have empty annotations. I don't have the right setup but it will be good to know what was the value of annotations using __annotations__ attribute in earlier versions.

https://github.com/sphinx-doc/sphinx/issues/8084

Hey @EricLiclair build still seems broken to me

This does not play nice with pybind11 based modules

in get_type_hints
    base_globals = sys.modules[base.__module__].__dict__
KeyError: 'pybind11_builtins'

@deathowl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Pull Request Test Coverage Report for Build 726552474

• 3 of 3 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 94.159%

Totals
Change from base Build 708713380:	0.0%
Covered Lines:	2563
Relevant Lines:	2722

💛 - Coveralls

Thanks @deathowl I have fixed the formatting issues. I will wait if someone has issues on using get_type_hints by default to emulate behavior before Python 3.10 and https://www.python.org/dev/peps/pep-0563/#backwards-compatibility

Hey @tirkarthi , Thanks for the PR, please make format, otherwise it looks good, we'll add a Python 3.10 Target to our builds

What:

Fixes #296

Why:

Due to PEP 563 becoming default in Python 3.10 the __annotations__ value doesn't store types and it stores string values. Using typing.get_type_hints ensures the types are evaluated and returned like behavior before Python 3.10 .

How:

<!-- How were these changes implemented? -->

Risks:

<!-- Any possible risks you've likely introduced in this PR? -->

Checklist:

<!-- Have you done all of these things? To check an item, place an "x" in the box like so: "- [x] Tests" Add "N/A" to the end of each line that's irrelevant to your changes -->

• [ ] Added tests, if you've added code that should be tested
• [ ] Updated the documentation, if you've changed APIs
• [ ] Ensured the test suite passes
• [ ] Made sure your code lints
• [ ] Completed the Contributor License Agreement ("CLA")

This is due to https://www.python.org/dev/peps/pep-0563 becoming default in Python 3.10 . So __annotations__ will have string values of type instead of actual types. Like in this test case {"typedattr": "str"} whereas in Python 3.9 the value for __annotations__ will be {'typedattr': <class 'str'>} . Using typing.get_type_hints but might have some backwards incompatible cases like https://www.python.org/dev/peps/pep-0563/#backwards-compatibility

@macisamuele merged this pull request in facebook/TestSlide@f42e0e969776ddb05d55266fc699d790f94df41e.

What: Ensure that we do run all the tests (platforms/python-versions) and that we don't immediately stop if one has failed. This is a follow-up from #292

Why: In case of test failures is extremely interesting to have information about "is this failure a genuine bug or is an issue with a single python version?".

How: Ensure that github workflow is correctly defined. Doc in here

Risks:

n/a

Checklist:

• [x] Added tests, if you've added code that should be tested
• [x] Updated the documentation, if you've changed APIs
• [x] Ensured the test suite passes
• [x] Made sure your code lints
• [x] Completed the Contributor License Agreement ("CLA")

TestSlide 2.6.4 fails with Python 3.10.0a6.

Failures:

  1) patch_attribute(), when target is a StrictMock, with a template, type validation: it fails if new value is of incompatible type
    1) AssertionError: TypeCheckError not raised
      File "testslide/runner.py", line 639, in run
        self._run_example(example)
      File "testslide/runner.py", line 623, in _run_example
        _ExampleRunner(example, self.formatter).run()
      File "testslide/__init__.py", line 540, in run
        self._sync_run_all_hooks_and_example(context_data)
      File "testslide/__init__.py", line 512, in _sync_run_all_hooks_and_example
        aggregated_exceptions.raise_correct_exception()
      File "testslide/__init__.py", line 289, in raise_correct_exception
        raise self.exceptions[0]
      File "testslide/__init__.py", line 273, in catch
        yield
      File "testslide/__init__.py", line 503, in _sync_run_all_hooks_and_example
        self._fail_if_coroutine_function(self.example.code, context_data)
      File "testslide/__init__.py", line 471, in _fail_if_coroutine_function
        return func(*args, **kwargs)
      File "tests/patch_attribute_testslide.py", line 119, in it_fails_if_new_value_is_of_incompatible_type
        self.patch_attribute(self.target, "typedattr", 123)
      File "/usr/lib64/python3.10/unittest/case.py", line 226, in __exit__
        self._raiseFailure("{} not raised".format(exc_name))
      File "/usr/lib64/python3.10/unittest/case.py", line 163, in _raiseFailure
        raise self.test_case.failureException(msg)

See for logs and more details: https://bugzilla.redhat.com/show_bug.cgi?id=1944109

@macisamuele has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Pull Request Test Coverage Report for Build 697672057

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 94.159%

Totals
Change from base Build 687383861:	0.0%
Covered Lines:	2563
Relevant Lines:	2722

💛 - Coveralls

What: Ensure that we do run all the tests (platforms/python-versions) and that we don't immediately stop if one has failed. This is a follow-up from #292

Why: In case of test failures is extremely interesting to have information about "is this failure a genuine bug or is an issue with a single python version?".

How: Ensure that github workflow is correctly defined. Doc in here

Risks:

n/a

Checklist:

• [x] Added tests, if you've added code that should be tested
• [x] Updated the documentation, if you've changed APIs
• [x] Ensured the test suite passes
• [x] Made sure your code lints
• [x] Completed the Contributor License Agreement ("CLA")

@macisamuele i'll resurrect Fabios Tox diff, and make it work for running all our tests . I think running against all supported py versions is strong signal

Insightful, I really missed out the point that both float and integer values can be valid range extremes. I'll try to update accordingly as per the comments you mentioned. Thanks for your review. ☺

In fact all of the commits were failing build test, until the last one.

I wanted to point the PR checks. as mentioned by deathowl here

I was curious why the build was failing so i checked for the build logs as i mentioned here

This is the failure i was mentioning about.

And, I'll remember the note.