profile
viewpoint
Carl Meyer carljm Instagram Rapid City, SD

ambv/flake8-mypy 107

A plugin for flake8 integrating Mypy.

carljm/django-adminfiles 66

[MIRROR] File uploader/manager/picker for Django admin; see demo screencast at http://vimeo.com/8940852

carljm/django-errorstack 8

[MIRROR] Django integration for ErrorStack.com error reporting service.

carljm/django-filch 7

Custom model fields that make de-normalizing data in Django easier

carljm/django-email-confirmation 3

simple email confirmation for the Django web framework

carljm/caseconductor-ui 2

testcase manager

carljm/datasource 2

Provides encapsulation and a clean interface for data sources.

carljm/django 2

The Web framework for perfectionists with deadlines. Now on GitHub.

carljm/django-deployment-workshop 2

Code and configuration used in my Django Deployment Workshop.

apollo13/django 1

The Web framework for perfectionists with deadlines. Now on GitHub.

pull request commentpypa/wheel

replace pep425tags with packaging

I'll have to defer to @jreese for that question, I haven't worked on that side of the system. Thanks for the heads-up!

mattip

comment created time in 15 days

pull request commentpypa/wheel

replace pep425tags with packaging

@agronholm Yes, we have our own internal tooling that selects the right wheel from our wheel archive based on platform/version/abi tags and then unpacks them where we need them. Wheel is ultimately a very easy format to use, that's one of its attractions :)

mattip

comment created time in 15 days

PullRequestReviewEvent

Pull request review commentoddbird/oddleventy

Pm2020

 --- title: Jobs-summary: |-  **We are not currently accepting applications for any positions** ---  thanks for your interest!+banner: Web Project Manager+image:+  svg: logos/oddbird+  type: angle+  attrs:+    style: '--align: center'+sub: Helping guide the web design and development process ---++OddBird is looking for a Project Manager to join the team.++## We’re Looking for Someone Who:++- Has knowledge of the web design and development process+- Can manage multiple projects simultaneously+- Will help plan, monitor, and facilitate projects on time and within budget +- Is an excellent verbal and written communicator+- Is not tied to a single methodology, +  but can help guide the team in finding & implementing +  processes that work for us+- Is located in a US time-zone

Technically this includes, like, Hawaii and American Samoa and Guam :) Would it be both more accurate and less US-centric to say e.g. "is located in a continental North America timezone"?

mirisuzanne

comment created time in 17 days

PullRequestReviewEvent

issue commentInstagram/MonkeyType

Generate all stubs available

I'm not aware that anyone is working on this currently. If you'd like to work on it, pull requests are welcome! IMO either "allow passing multiple modules to apply" or "add --recursive option to apply" are reasonable design options. In either case stub would remain unchanged.

thedrow

comment created time in 22 days

issue commentpypa/wheel

test_tagopy.test_plat_name_ext and setting plat-name on non-pure wheels

Described the use case in https://github.com/pypa/wheel/pull/346#issuecomment-682318511

mattip

comment created time in 25 days

pull request commentpypa/wheel

replace pep425tags with packaging

Oh! I see @mattip already supplied a PR to fix; thank you!

And I see in the description of that PR it also explains why the test failure was not noticed.

mattip

comment created time in 25 days

pull request commentpypa/wheel

replace pep425tags with packaging

Could you explain your case in detail?

Sure. At work we have custom versioned internal Linux platforms each with its own specific set of C library versions and compiler versions etc. So we build a wide variety of shared libraries for these platforms, including many Python wheels that link against libraries in the platform. So we pass --plat-name to bdist-wheel when building wheels so they are marked with the appropriate platform.

What I don't understand, and maybe @carljm could clarify, is how pip will install such a wheel: it should reject it.

I don't know about this either way, because we don't use pip to install them. As far as I know wheel is a standardized Python packaging format, it is not owned by pip and should not necessarily serve only pip-supported use cases.

Hmm, but the test_plat_name_ext test should check that my analysis is mistaken ...

Yes, to me it seems quite intentional in the previous code that this plat-name override was allowed, given that it's tested. Also I wonder how test_plat_name_ext is still passing after this PR and why didn't it catch the removal of this support?

I suggest filing an issue on pypa/packaging instead of trying to change only-wheel to do the right thing here.

I'm not sure why. This was functionality entirely contained within the wheel library that previously existed and was tested but has now regressed. Seems straightforward to fix as a regression in wheel, it doesn't require changes in any other tool. If other tools (e.g. pip) want to make changes to support the use case, that seems fine, but it shouldn't be a prerequisite to fixing the regression.

If a PR to restore this functionality would be accepted, I'd be happy to supply one.

mattip

comment created time in 25 days

pull request commentpypa/wheel

replace pep425tags with packaging

This PR broke the ability to set the plat_name attribute and have supported tags include that platform (previously supported by https://github.com/pypa/wheel/pull/346/files#diff-7af7e00575edbab17718ee16d91b8982L207 ). This breaks our usage, where we use a distutils config file to set a plat-name for bdist_wheel command, and is preventing us from upgrading wheel.

mattip

comment created time in a month

issue closedInstagram/MonkeyType

How to deal with embedded C modules

My project is meant to be used inside vim. This means that it relies on import vim which exists only inside an actual vim instance. The way my tests deal with it is by assigning a MagicMock object into sys.modules['vim'].

This worked perfectly fine for the monkeytype run $(which pytest) step. However, monkeytype apply <module name> and monkeytype stub <module name> can't use that sort of a trick as far as I know. Any idea how to work around this?

closed time in a month

bstaletic

issue commentInstagram/MonkeyType

How to deal with embedded C modules

Thanks for the report! I think the cli_context config method is intended precisely for this type of case, and it should work to just do your sys.modules insertion hack there.

Personally if I were writing your project, I would probably prefer to provide a shim module which does try: import vim; except ImportError: ... and explicitly defines a shim version of the functionality provided by the vim module if it is not available. Then such sys.modules tricks become unnecessary.

bstaletic

comment created time in a month

issue closedInstagram/MonkeyType

How do I annotate pyspark project ?

Hi,

I want to use monkeytype for my pyspark project.

I used to debug a job by :

export PYSPARK_DRIVER_PYTHON=ipython && python ./sales_forecast/src/main.py --job $action  --job-args $args

main.py


if __name__ == '__main__':

    parser = argparse.ArgumentParser(description='Run a PySpark job')
    parser.add_argument('--job', type=str, required=True, dest='job_name', help="The name of the job module you want to run. (ex: poc will run job on jobs.poc package)")
    parser.add_argument('--job-args', nargs='*', help="Extra arguments to send to the PySpark job (example: --job-args template=manual-email1 foo=bar")

    spark = start_spark_session(app_name=args.job_name, master=args.master, spark_config=spark_config)

    job_module = importlib.import_module('jobs.%s' % args.job_name)
    job_module.run(spark, **job_args)

    spark.stop()

jobs/sample_job1.py

def run(spark, start_date, end_date, limit_num, **kwargs):
    do_task1(....)
    do_task2(....)
    pass

I want to annotate jobs/sample_job1.py, do_task1, do_task2 , this function can not run without spark session .

But I don't actually can run this command in local , it would cost much time, I would like early stop since it is enough to generate annotation . Do you have any idea to support this?

closed time in 2 months

eromoe

issue commentInstagram/MonkeyType

How do I annotate pyspark project ?

In general it is up to you to provide a suitable process run for MonkeyType to acquire types from. This could be a test suite, or a production run, if you can configure it appropriately.

So this question really doesn't have anything to do with MonkeyType, it's a question about PySpark; how to configure your PySpark job to complete more quickly. Since I don't know anything about PySpark, unfortunately I don't have any ideas about how to do this.

eromoe

comment created time in 2 months

issue commentInstagram/MonkeyType

add `from __future__ import annotations` if adding forward-reference annotations

If you want to use from __future__ import annotations in your project, I would suggest just adding it to all files in your project once (this is pretty easy to automate even with just sed). Having it in some files and not others makes things confusing for later development that touches annotations.

Regarding automatically putting added imports under if TYPE_CHECKING:, that's a great idea, but probably only if we do add from __future__ import annotations (since it pretty much means all annotations would need to be stringified otherwise).

I think it would be reasonable to group both of these features under a --pep-563 flag.

Jongy

comment created time in 2 months

push eventInstagram/MonkeyType

Shane Harvey

commit sha f680c783c3aec6b0f613c4ea0268032cab23e788

Cleanup documentation for list-modules

view details

push time in 2 months

PR merged Instagram/MonkeyType

Cleanup documentation for list-modules CLA Signed

list_modules does not exist:

$ monkeytype list_modules
usage: monkeytype [-h] [--disable-type-rewriting] [--limit LIMIT] [--verbose] [--config CONFIG] {run,apply,stub,list-modules} ...
monkeytype: error: argument command: invalid choice: 'list_modules' (choose from 'run', 'apply', 'stub', 'list-modules')
+4 -10

1 comment

1 changed file

ShaneHarvey

pr closed time in 2 months

pull request commentInstagram/MonkeyType

Cleanup documentation for list-modules

Thanks!!

ShaneHarvey

comment created time in 2 months

IssuesEvent

issue commentInstagram/MonkeyType

MonkeyType doesn't use forward declarations when necessary

I don't think it's easy, but it should be doable. I still have some mixed feelings about MonkeyType doing that, but I think I would accept a PR for it. I'll reopen this issue for that.

Jongy

comment created time in 2 months

push eventInstagram/MonkeyType

Zedive

commit sha a5c694ddd1a627d3f22d60bc399ea1e9f81470d7

Update monkeytype version in requirements.txt

view details

push time in 2 months

PR merged Instagram/MonkeyType

Update monkeytype version in requirements.txt CLA Signed

Make it easier for python 3.7 users to run this demo

+1 -1

1 comment

1 changed file

Zedive

pr closed time in 2 months

pull request commentInstagram/MonkeyType

Update monkeytype version in requirements.txt

Thank you!

Zedive

comment created time in 2 months

issue commentInstagram/MonkeyType

AttributeError: __qualname__

Thanks for the report! MonkeyType doesn't have support for 3.9 yet, but it would be great to have. Pull requests gladly accepted :)

nschloe

comment created time in 2 months

issue closedInstagram/LibCST

Circular dependency on hypothesmith

I'm working on packaging LibCST for Guix (https://guix.gnu.org) and noticed that some LibCST tests were using hypothesmith. The problem is that hypothesis has libcst as a dependency.

Is this on purpose? Can those tests be safely disabled and the dev dependency on hypothesmith dropped in my package?

Thanks

closed time in 2 months

tlc28

issue commentInstagram/LibCST

Circular dependency on hypothesmith

It should be fine to just remove test_fuzz.py from your distribution altogether if needed. Thanks for the report!

tlc28

comment created time in 2 months

issue commentInstagram/MonkeyType

MonkeyType doesn't use forward declarations when necessary

Yeah, that's a reasonable thought. I'm a little hesitant to have MonkeyType do that by default, though, since from __future__ import annotations can be a breaking change (if your code is inspecting the annotations at runtime).

Jongy

comment created time in 2 months

issue closedInstagram/MonkeyType

Running monkeytype directly from venv

When I create a venv and install monkeytype like this:

python3 -m venv /tmp/venv
/tmp/venv/bin/pip install MonkeyType

And then I run monkeytype like this:

/tmp/venv/bin/monkeytype run foo.py
/tmp/venv/bin/monkeytype apply example

Then it fails with this error:

Traceback (most recent call last):
  File "/tmp/venv/bin/monkeytype", line 11, in <module>
    sys.exit(entry_point_main())
  File "/tmp/venv/lib/python3.6/site-packages/monkeytype/cli.py", line 382, in entry_point_main
    sys.exit(main(sys.argv[1:], sys.stdout, sys.stderr))
  File "/tmp/venv/lib/python3.6/site-packages/monkeytype/cli.py", line 367, in main
    handler(args, stdout, stderr)
  File "/tmp/venv/lib/python3.6/site-packages/monkeytype/cli.py", line 165, in apply_stub_handler
    proc = subprocess.run(retype_args, check=True, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'retype': 'retype'

As retype is not in my path.

It would be very great to run the retype execuable from the venv also, as it is installed to the bin directory in the venv.

I can workaround this issue by activating the venv before but it is a bit inconvenient for me.

closed time in 2 months

csernazs

issue commentInstagram/MonkeyType

Running monkeytype directly from venv

This was fixed with the switch from retype to LibCST for applying type annotations. Thanks for the nudge to close it!

csernazs

comment created time in 2 months

issue closedInstagram/MonkeyType

MonkeyType doesn't use forward declarations when necessary

$ cat a.py
class A:
    def f(self):
        return self

$ cat b.py
import a

a.A().f()

$ monkeytype run b.py
$ monkeytype apply a
class A:
    def f(self) -> A:
        return self

$ python b.py
Traceback (most recent call last):
  File "b.py", line 1, in <module>
    import a
  File "a.py", line 1, in <module>
    class A:
  File "a.py", line 2, in A
    def f(self) -> A:
NameError: name 'A' is not defined

As described here, I guess forward declarations should have been used here. If I manually use -> "A", then Mypy identifies it correctly.

I see references to typing.ForwardRef in MonkeyType's code, so either it was missed for some reason, or am I using it wrong?

MonkeyType version: 20.5.0.

closed time in 2 months

Jongy

issue commentInstagram/MonkeyType

MonkeyType doesn't use forward declarations when necessary

Thanks for the report! The ForwardRef references in MonkeyType's code are for cases where we get the existing annotation from the code and need to reproduce it in our stub. MonkeyType doesn't currently attempt to detect forward references in generated stubs and output stringified references. And I don't think it makes sense to add that feature at this point, because it's unnecessary with from __future__ import annotations, which in future will be the default behavior of annotations.

So if you aren't using from __future__ import annotations, this is just something you need to fix up as needed in the applied annotations. In general MonkeyType doesn't aim to provide ready-for-production code; it aims to provide an informative first draft of type information to be edited as needed by the developer. This also applies to things like generating concrete types when perhaps an abstract type would be more suitable, or unions where a common parent type would be better.

Jongy

comment created time in 2 months

issue commentInstagram/MonkeyType

Feature request: enable an optional limit at which the generated annotation for a type is `Any`

We actually already have RewriteConfigDict rewriter also in the default config, which should do precisely the Union[Dict[T, X], Dict[T, Y]] -> Dict[T, Union[X, Y]] transform in your first example. Interesting that there seem to be a couple deeply-nested cases in your example annotation where that should have applied but didn't. Maybe there's a bug in it that's causing it to not fully traverse the nested types.

The transforms in your second example look significantly harder to pull off algorithmically, but are probably possible; if you come up with a working rewriter we'd definitely look at it!

sirosen

comment created time in 2 months

issue commentInstagram/MonkeyType

Feature request: enable an optional limit at which the generated annotation for a type is `Any`

Hi, thanks for the report! So the default type rewriter in the default config already includes RewriteLargeUnion which is intended to address this problem. However, it defaults to rewriting any Union with more than 5 elements, and it looks like your particular type escapes this rewrite by being deeply nested but not having a Union of more than 5 elements at any one layer.

In principle I think it should be possible to write a type rewriter that in some way also collapses based on depth of nesting, though I think identifying the right heuristic precisely might be tricky; it's not clear that a deeply-nested but well-specified type should necessarily be collapsed, but deep nesting with lots of unions at multiple layers probably should be.

Given that MonkeyType's type-rewriter and config infrastructure allows you to easily do all of this outside of MonkeyType in your own config file, I would encourage you to experiment with writing a type-rewriter to do this and seeing how it behaves in practice on your codebases. If you reach a solution that you feel works well and want to submit it as a pull request, I'd certainly consider it.

FWIW, at Instagram we ended up disabling even RewriteLargeUnion. Our approach to using MonkeyType is that its purpose is to generate an informative first draft suitable for editing by a developer before committing. Given that it's relatively hard for a developer to find information that type rewriters choose to discard, but relatively easy to delete multiple lines of type annotation, we ended up feeling that it was preferable for MonkeyType to just provide all the information available and allow the developer to make the choice of how to best condense it, since with their understanding of the code context they can make a better choice than an automated type rewriter.

sirosen

comment created time in 2 months

issue commentInstagram/LibCST

Apply class method / attribute annotations from the base class to subclasses

Lol, sorry, clearly i somehow thought this was on the MonkeyType repo when I replied. My bad :)

Basically though my thoughts on it remain the same. I don’t think this should be built in behavior of the apply-annotations visitor, which should continue to do its single job well: apply exactly the annotations found in a stub to the marching source.

What you want to do should be written as a separate LibCST transform, to be run on the stubs before application to the source. I doubt that it’s widely needed enough to be included in the LibCST repo, but LibCST is a great tool for writing it!

mkurnikov

comment created time in 3 months

issue closedInstagram/LibCST

Apply class method / attribute annotations from the base class to subclasses

I'm trying to make a script which merges django-stubs stubs into the django source code.

Django inheritance chains could be quite big, so stubs are written in the way that some inherited (abstract or not) methods are defined only in the base class.

See for example,

class BaseConstraint:
    name: str
    def __init__(self, name: str) -> None: ...
    def constraint_sql(
        self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]
    ) -> str: ...
    def create_sql(self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]) -> str: ...
    def remove_sql(self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]) -> str: ...

class CheckConstraint(BaseConstraint):
    check: Q
    def __init__(self, *, check: Q, name: str) -> None: ...

and implementation with applied stubs is

class BaseConstraint:
    def __init__(self, name: str) -> None:
        self.name = name

    def constraint_sql(self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]) -> str:
        raise NotImplementedError('This method must be implemented by a subclass.')

    def create_sql(self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]) -> str:
        raise NotImplementedError('This method must be implemented by a subclass.')

    def remove_sql(self, model: Optional[Type[Model]], schema_editor: Optional[BaseDatabaseSchemaEditor]) -> str:
        raise NotImplementedError('This method must be implemented by a subclass.')

class CheckConstraint(BaseConstraint):
    def __init__(self, *, check: Q, name: str) -> None:
        self.check = check
        super().__init__(name)

    def constraint_sql(self, model, schema_editor):
        check = self._get_check_sql(model, schema_editor)
        return schema_editor._check_sql(self.name, check)

    def create_sql(self, model, schema_editor):
        check = self._get_check_sql(model, schema_editor)
        return schema_editor._create_check_sql(model, self.name, check)

    def remove_sql(self, model, schema_editor):
        return schema_editor._delete_check_sql(model, self.name)

I'd like name: str and annotations for the create_sql, remove_sql to be applied to the subclasses too.

closed time in 3 months

mkurnikov

issue commentInstagram/LibCST

Apply class method / attribute annotations from the base class to subclasses

This is really pretty much unrelated to MonkeyType, which only really creates stubs; the stub-application code is actually not in MonkeyType at all, it's in LibCST. For the sake of modularity and focused tools, I think the best strategy for you here would be to write your own LibCST transformation to create these subclass methods in the source stubs, and then apply that as a prior step to running application.

For your task you shouldn't involve MonkeyType at all, just the apply-stubs transformation in LibCST that MonkeyType calls out to.

mkurnikov

comment created time in 3 months

pull request commentInstagram/MonkeyType

MonkeyType: require a newer version of LibCST

Thank you! ✨

hauntsaninja

comment created time in 3 months

push eventInstagram/MonkeyType

hauntsaninja

commit sha e68c1858fc87e87f2127bbb8f3ec4f0be257d6e2

MonkeyType: require a newer version of LibCST Fixes #198

view details

push time in 3 months

PR merged Instagram/MonkeyType

MonkeyType: require a newer version of LibCST CLA Signed

Fixes #198

+1 -1

0 comment

1 changed file

hauntsaninja

pr closed time in 3 months

issue closedInstagram/MonkeyType

monkeytype destroys default values of keyword-only args

Here's a repro:

(.env) ~/tmp/bug λ cat mod.py   
def foo(a, *, b=5):
    return (a, b)


foo("asdf", b=10)
(.env) ~/tmp/bug λ cat script.py
from mod import *
(.env) ~/tmp/bug λ monkeytype run script.py
(.env) ~/tmp/bug λ monkeytype apply mod    
from typing import Tuple

def foo(a: str, *, b = ...) -> Tuple[str, int]:
    return (a, b)


foo("asdf", b=10)

(.env) ~/tmp/bug λ cat mod.py   
from typing import Tuple

def foo(a: str, *, b = ...) -> Tuple[str, int]:
    return (a, b)


foo("asdf", b=10)

closed time in 3 months

hauntsaninja

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Callable, Optional, Sequence, Set, Tuple, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++def leave_import_decorator(+    method: Callable[..., Union[cst.Import, cst.ImportFrom]]+) -> Callable[..., Union[cst.Import, cst.ImportFrom]]:+    # We want to record any 'as name' that is relevant but only after we leave the corresponding Import/ImportFrom node as we don't want the 'as name'+    # to interfere with children 'Name' and 'Attribute' nodes.+    def wrapper(+        self: VisitorBasedCodemodCommand,+        original_node: Union[cst.Import, cst.ImportFrom],+        updated_node: Union[cst.Import, cst.ImportFrom],+    ) -> Union[cst.Import, cst.ImportFrom]:+        updated_node = method(self, original_node, updated_node)+        if original_node != updated_node:+            getattr(self, "record_asname")(original_node)+        return updated_node++    return wrapper+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help=(+                "Full dotted name of replacement object. You may provide a single-colon-delimited name to specify how you want the new import to be structured."+                + "\nEg: `foo:bar.baz` will be translated to `from foo import bar`."+                + "\nIf no ':' character is provided, the import statement will default to `from foo.bar import baz` for a `new_name` value of `foo.bar.baz`."+            ),+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)++        new_module, has_colon, new_mod_or_obj = new_name.rpartition(":")+        # Exit early if improperly formatted args.+        if ":" in new_module:+            raise ValueError("Error: `new_name` should contain at most one colon.")+        if ":" in old_name:+            raise ValueError("Error: `old_name` should not contain any colons.")++        if not has_colon or not new_module:+            new_module, _, new_mod_or_obj = new_name.rpartition(".")++        self.new_name: str = new_name.replace(":", ".").strip(".")+        self.new_module: str = new_module.replace(":", ".").strip(".")+        self.new_mod_or_obj: str = new_mod_or_obj++        if not self.new_mod_or_obj:+            old_module = old_name+            old_mod_or_obj = ""+        else:+            old_module, _, old_mod_or_obj = old_name.rpartition(".")++        self.old_name: str = old_name.replace(":", ".")+        self.old_module: str = old_module.replace(":", ".")+        self.old_mod_or_obj: str = old_mod_or_obj.replace(":", ".")++        self.as_name: Optional[Tuple[str, str]] = None++        self.scheduled_removals: Set[cst.CSTNode] = set()+        self.bypass_imports = False+        self.schedule_import = False

I think that some comments on instance attributes like this, explaining what the meaning of each one is, can go a long way to helping readers understand the code faster (as well as ensuring that you as the author have a clear mental model of the possible states.)

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Callable, Optional, Sequence, Set, Tuple, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++def leave_import_decorator(+    method: Callable[..., Union[cst.Import, cst.ImportFrom]]+) -> Callable[..., Union[cst.Import, cst.ImportFrom]]:+    # We want to record any 'as name' that is relevant but only after we leave the corresponding Import/ImportFrom node as we don't want the 'as name'+    # to interfere with children 'Name' and 'Attribute' nodes.+    def wrapper(+        self: VisitorBasedCodemodCommand,+        original_node: Union[cst.Import, cst.ImportFrom],+        updated_node: Union[cst.Import, cst.ImportFrom],+    ) -> Union[cst.Import, cst.ImportFrom]:+        updated_node = method(self, original_node, updated_node)+        if original_node != updated_node:+            getattr(self, "record_asname")(original_node)+        return updated_node++    return wrapper+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help=(+                "Full dotted name of replacement object. You may provide a single-colon-delimited name to specify how you want the new import to be structured."+                + "\nEg: `foo:bar.baz` will be translated to `from foo import bar`."+                + "\nIf no ':' character is provided, the import statement will default to `from foo.bar import baz` for a `new_name` value of `foo.bar.baz`."+            ),+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)++        new_module, has_colon, new_mod_or_obj = new_name.rpartition(":")+        # Exit early if improperly formatted args.+        if ":" in new_module:+            raise ValueError("Error: `new_name` should contain at most one colon.")+        if ":" in old_name:+            raise ValueError("Error: `old_name` should not contain any colons.")++        if not has_colon or not new_module:+            new_module, _, new_mod_or_obj = new_name.rpartition(".")++        self.new_name: str = new_name.replace(":", ".").strip(".")+        self.new_module: str = new_module.replace(":", ".").strip(".")+        self.new_mod_or_obj: str = new_mod_or_obj++        if not self.new_mod_or_obj:+            old_module = old_name+            old_mod_or_obj = ""

It's not clear to me what case this is handling. How would self.new_mod_or_obj end up empty here?

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Callable, Optional, Sequence, Set, Tuple, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++def leave_import_decorator(+    method: Callable[..., Union[cst.Import, cst.ImportFrom]]+) -> Callable[..., Union[cst.Import, cst.ImportFrom]]:+    # We want to record any 'as name' that is relevant but only after we leave the corresponding Import/ImportFrom node as we don't want the 'as name'+    # to interfere with children 'Name' and 'Attribute' nodes.+    def wrapper(+        self: VisitorBasedCodemodCommand,+        original_node: Union[cst.Import, cst.ImportFrom],+        updated_node: Union[cst.Import, cst.ImportFrom],+    ) -> Union[cst.Import, cst.ImportFrom]:+        updated_node = method(self, original_node, updated_node)+        if original_node != updated_node:+            getattr(self, "record_asname")(original_node)+        return updated_node++    return wrapper+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help=(+                "Full dotted name of replacement object. You may provide a single-colon-delimited name to specify how you want the new import to be structured."+                + "\nEg: `foo:bar.baz` will be translated to `from foo import bar`."+                + "\nIf no ':' character is provided, the import statement will default to `from foo.bar import baz` for a `new_name` value of `foo.bar.baz`."+            ),+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)++        new_module, has_colon, new_mod_or_obj = new_name.rpartition(":")+        # Exit early if improperly formatted args.+        if ":" in new_module:+            raise ValueError("Error: `new_name` should contain at most one colon.")+        if ":" in old_name:+            raise ValueError("Error: `old_name` should not contain any colons.")++        if not has_colon or not new_module:+            new_module, _, new_mod_or_obj = new_name.rpartition(".")++        self.new_name: str = new_name.replace(":", ".").strip(".")+        self.new_module: str = new_module.replace(":", ".").strip(".")+        self.new_mod_or_obj: str = new_mod_or_obj++        if not self.new_mod_or_obj:+            old_module = old_name+            old_mod_or_obj = ""+        else:+            old_module, _, old_mod_or_obj = old_name.rpartition(".")++        self.old_name: str = old_name.replace(":", ".")+        self.old_module: str = old_module.replace(":", ".")+        self.old_mod_or_obj: str = old_mod_or_obj.replace(":", ".")

Why bother replacing colons here when you already checked above to ensure there aren't any in old_name?

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Callable, Optional, Sequence, Set, Tuple, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++def leave_import_decorator(+    method: Callable[..., Union[cst.Import, cst.ImportFrom]]+) -> Callable[..., Union[cst.Import, cst.ImportFrom]]:+    # We want to record any 'as name' that is relevant but only after we leave the corresponding Import/ImportFrom node as we don't want the 'as name'+    # to interfere with children 'Name' and 'Attribute' nodes.+    def wrapper(+        self: VisitorBasedCodemodCommand,+        original_node: Union[cst.Import, cst.ImportFrom],+        updated_node: Union[cst.Import, cst.ImportFrom],+    ) -> Union[cst.Import, cst.ImportFrom]:+        updated_node = method(self, original_node, updated_node)+        if original_node != updated_node:+            getattr(self, "record_asname")(original_node)

Should be able to avoid this getattr by annotating self as "RenameCommand"? Note the quoting to work around the forward reference.

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedName, QualifiedNameProvider, QualifiedNameSource+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--orig_module",+            dest="orig_module",+            help="Name of the module to rename. Leave blank if renaming a local variable.",+        )+        parser.add_argument(+            "--orig_object",+            dest="orig_object",+            required=True,+            help="Name of the object in module or variable to rename.",+        )+        parser.add_argument(+            "--new_module",+            dest="new_module",+            help="New name of the module. Leave blank if renaming a local variable, or if same as orig_module.",+        )+        parser.add_argument(+            "--new_object",+            dest="new_object",+            required=True,+            help="New name of the object in module or variable.",+        )++    def __init__(+        self,+        context: CodemodContext,+        orig_object: str,+        new_object: str,+        orig_module: Optional[str] = None,+        new_module: Optional[str] = None,+    ) -> None:+        super().__init__(context)++        self.orig_module: Optional[str] = orig_module+        self.orig_object: str = orig_object+        self.orig_name: str = (+            f"{orig_module}.{orig_object}" if orig_module is not None else orig_object+        )++        self.new_module: Optional[+            str+        ] = new_module if new_module is not None else orig_module+        self.new_object: str = new_object+        self.new_name: str = (+            f"{new_module}.{new_object}" if new_module is not None else new_object+        )++        self.source: QualifiedNameSource = (+            QualifiedNameSource.IMPORT+            if orig_module is not None+            else QualifiedNameSource.LOCAL+        )++    def leave_Import(+        self, original_node: cst.Import, updated_node: cst.Import+    ) -> Union[cst.Import, cst.RemovalSentinel]:+        for import_alias in original_node.names:+            name = get_full_name_for_node(import_alias.name)+            if name is not None and name == self.orig_module:+                # Schedule this import to be potentially removed+                RemoveImportsVisitor.remove_unused_import_by_node(+                    self.context, original_node+                )+        return original_node++    def leave_ImportFrom(

Looks like this case is now handled and there's a test for it!

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],+        }++        self.scheduled_imports = False++    def visit_Import(self, node: cst.Import) -> None:+        for import_alias in node.names:+            alias_name = get_full_name_for_node(import_alias.name)+            if alias_name is not None:+                self.record_asname(import_alias, alias_name)++                # If the import statement is exactly equivalent to the old name, replace it.+                if alias_name == self.old_name:+                    self.cleanup_imports(node=node, new_module=self.new_name)++    def leave_ImportFrom(+        self, original_node: cst.ImportFrom, updated_node: cst.ImportFrom+    ) -> cst.ImportFrom:+        module = updated_node.module+        if module is not None:+            imported_module_name = get_full_name_for_node(module)+            names = original_node.names++            if imported_module_name is None or not isinstance(names, Sequence):+                return updated_node+            elif imported_module_name == self.old_name:+                # Simply rename the imported module on the spot.+                return updated_node.with_changes(

I noticed that according to the coverage report no test actually hits this line?

(Test coverage of this PR is otherwise quite excellent!)

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict++from libcst.codemod import CodemodTest+from libcst.codemod.commands.rename import RenameCommand+++class TestRenameCommand(CodemodTest):++    ONCALL_SHORTNAME = "instagram_server_framework"++    TRANSFORM = RenameCommand++    def test_rename_name(self) -> None:++        before = """+            from foo import bar++            def test() -> None:+                bar(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(before, after, old_name="foo.bar", new_name="baz.qux")++    def test_rename_name_asname(self) -> None:++        before = """+            from foo import bar as bla++            def test() -> None:+                bla(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="baz.qux",+        )++    def test_rename_attr(self) -> None:++        before = """+            import a.b++            def test() -> None:+                a.b.c(5)+        """+        after = """+            import a.b+            import d.e++            def test() -> None:+                d.e.f(5)+        """++        self.assertCodemod(+            before, after, old_name="a.b.c", new_name="d.e.f",+        )++    def test_rename_attr_asname(self) -> None:++        before = """+            import foo as bar++            def test() -> None:+                bar.qux(5)+        """+        after = """+            import baz++            def test() -> None:+                baz.quux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.qux", new_name="baz.quux",+        )++    def test_rename_module_import(self) -> None:+        before = """+            import a.b++            class Foo(a.b.C):+                pass+        """+        after = """+            import c.b++            class Foo(c.b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",+        )++    def test_rename_module_import_from(self) -> None:+        before = """+            from a import b++            class Foo(b.C):+                pass+        """+        after = """+            from c import b++            class Foo(b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",+        )++    def test_rename_class(self) -> None:+        before = """+            from a.b import some_class++            class Foo(some_class):+                pass+        """+        after = """+            from c.b import some_class++            class Foo(some_class):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b.some_class", new_name="c.b.some_class",+        )++    def test_rename_object_same_module(self) -> None:+        before = """+            from a.b import Class_1, Class_2++            class Foo(Class_1):+                pass+        """+        after = """+            from a.b import Class_3, Class_2++            class Foo(Class_3):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b.Class_1", new_name="a.b.Class_3",+        )++    def test_rename_local_variable(self) -> None:+        before = """+            x = 5+            y = 5 + x+        """+        after = """+            z = 5+            y = 5 + z+        """++        self.assertCodemod(+            before, after, old_name="x", new_name="z",+        )++    def test_module_does_not_change(self) -> None:+        before = """+            from a import b++            class Foo(b):+                pass+        """+        after = """+            from a import c++            class Foo(c):+                pass+        """+        self.assertCodemod(before, after, old_name="a.b", new_name="a.c")++    def test_other_imports_untouched(self) -> None:+        before = """+            import a, b, c++            class Foo(a.z):+                bar: b.bar+                baz: c.baz+        """+        after = """+            import b, c+            import d++            class Foo(d.z):+                bar: b.bar+                baz: c.baz+        """+        self.assertCodemod(+            before, after, old_name="a.z", new_name="d.z",+        )++    def test_other_import_froms_untouched(self) -> None:+        before = """+            from a import b, c, d++            class Foo(b):+                bar: c.bar+                baz: d.baz+        """+        after = """+            from a import c, d+            from f import b++            class Foo(b):+                bar: c.bar+                baz: d.baz+        """+        self.assertCodemod(+            before, after, old_name="a.b", new_name="f.b",+        )++    def test_no_removal_of_import_in_use(self) -> None:+        before = """+            import a++            class Foo(a.b):+                pass+            class Foo2(a.c):+                pass+        """+        after = """+            import a+            import z++            class Foo(z.b):+                pass+            class Foo2(a.c):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b", new_name="z.b",+        )++    def test_no_removal_of_import_from_in_use(self) -> None:+        before = """+            from a import b++            class Foo(b.some_class):+                bar: b.some_other_class+        """+        after = """+            from a import b+            import blah++            class Foo(blah.some_class):+                bar: b.some_other_class+        """+        self.assertCodemod(+            before, after, old_name="a.b.some_class", new_name="blah.some_class",+        )++    def test_other_unused_imports_untouched(self) -> None:+        before = """+            import a+            import b++            class Foo(a.obj):+                pass+        """+        after = """+            import b+            import c++            class Foo(c.obj):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.obj", new_name="c.obj",+        )++    def test_complex_module_rename(self) -> None:+        before = """+            from a.b.c import d++            class Foo(d.e.f):+                pass+        """+        after = """+            import g.h.i++            class Foo(g.h.i.j):+                pass+        """+        self.assertCodemod(before, after, old_name="a.b.c.d.e.f", new_name="g.h.i.j")++    def test_names_with_repeated_substrings(self) -> None:+        before = """+            from aa import aaaa++            class Foo(aaaa.Bar):+                pass+        """+        after = """+            from b import c++            class Foo(c.Bar):+                pass+        """+        self.assertCodemod(+            before, after, old_name="aa.aaaa", new_name="b.c",+        )++    def test_import_star_attr(self) -> None:+        before = """+            from foo import *++            def baz():+                foo.bar(5)+        """+        after = """+            from foo import *+            import qux++            def baz():+                qux.bar(5)+        """+        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="qux.bar",+        )++    def test_import_star_obj(self) -> None:+        before = """+            from foo import *++            class Bar(foo.Bar):

again, like above, the combination of from foo import * with later use of foo.Bar doesn't make sense as working code in the first place (unless the foo has a submodule also named foo and the fully qualified name of Bar is foo.foo.Bar)

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict++from libcst.codemod import CodemodTest+from libcst.codemod.commands.rename import RenameCommand+++class TestRenameCommand(CodemodTest):++    ONCALL_SHORTNAME = "instagram_server_framework"++    TRANSFORM = RenameCommand++    def test_rename_name(self) -> None:++        before = """+            from foo import bar++            def test() -> None:+                bar(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(before, after, old_name="foo.bar", new_name="baz.qux")++    def test_rename_name_asname(self) -> None:++        before = """+            from foo import bar as bla++            def test() -> None:+                bla(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="baz.qux",+        )++    def test_rename_attr(self) -> None:++        before = """+            import a.b++            def test() -> None:+                a.b.c(5)+        """+        after = """+            import a.b+            import d.e++            def test() -> None:+                d.e.f(5)+        """++        self.assertCodemod(+            before, after, old_name="a.b.c", new_name="d.e.f",+        )++    def test_rename_attr_asname(self) -> None:++        before = """+            import foo as bar++            def test() -> None:+                bar.qux(5)+        """+        after = """+            import baz++            def test() -> None:+                baz.quux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.qux", new_name="baz.quux",+        )++    def test_rename_module_import(self) -> None:+        before = """+            import a.b++            class Foo(a.b.C):+                pass+        """+        after = """+            import c.b++            class Foo(c.b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",+        )++    def test_rename_module_import_from(self) -> None:+        before = """+            from a import b++            class Foo(b.C):+                pass+        """+        after = """+            from c import b++            class Foo(b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",+        )++    def test_rename_class(self) -> None:+        before = """+            from a.b import some_class++            class Foo(some_class):+                pass+        """+        after = """+            from c.b import some_class++            class Foo(some_class):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b.some_class", new_name="c.b.some_class",+        )++    def test_rename_object_same_module(self) -> None:+        before = """+            from a.b import Class_1, Class_2++            class Foo(Class_1):+                pass+        """+        after = """+            from a.b import Class_3, Class_2++            class Foo(Class_3):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b.Class_1", new_name="a.b.Class_3",+        )++    def test_rename_local_variable(self) -> None:+        before = """+            x = 5+            y = 5 + x+        """+        after = """+            z = 5+            y = 5 + z+        """++        self.assertCodemod(+            before, after, old_name="x", new_name="z",+        )++    def test_module_does_not_change(self) -> None:+        before = """+            from a import b++            class Foo(b):+                pass+        """+        after = """+            from a import c++            class Foo(c):+                pass+        """+        self.assertCodemod(before, after, old_name="a.b", new_name="a.c")++    def test_other_imports_untouched(self) -> None:+        before = """+            import a, b, c++            class Foo(a.z):+                bar: b.bar+                baz: c.baz+        """+        after = """+            import b, c+            import d++            class Foo(d.z):+                bar: b.bar+                baz: c.baz+        """+        self.assertCodemod(+            before, after, old_name="a.z", new_name="d.z",+        )++    def test_other_import_froms_untouched(self) -> None:+        before = """+            from a import b, c, d++            class Foo(b):+                bar: c.bar+                baz: d.baz+        """+        after = """+            from a import c, d+            from f import b++            class Foo(b):+                bar: c.bar+                baz: d.baz+        """+        self.assertCodemod(+            before, after, old_name="a.b", new_name="f.b",+        )++    def test_no_removal_of_import_in_use(self) -> None:+        before = """+            import a++            class Foo(a.b):+                pass+            class Foo2(a.c):+                pass+        """+        after = """+            import a+            import z++            class Foo(z.b):+                pass+            class Foo2(a.c):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.b", new_name="z.b",+        )++    def test_no_removal_of_import_from_in_use(self) -> None:+        before = """+            from a import b++            class Foo(b.some_class):+                bar: b.some_other_class+        """+        after = """+            from a import b+            import blah++            class Foo(blah.some_class):+                bar: b.some_other_class+        """+        self.assertCodemod(+            before, after, old_name="a.b.some_class", new_name="blah.some_class",+        )++    def test_other_unused_imports_untouched(self) -> None:+        before = """+            import a+            import b++            class Foo(a.obj):+                pass+        """+        after = """+            import b+            import c++            class Foo(c.obj):+                pass+        """+        self.assertCodemod(+            before, after, old_name="a.obj", new_name="c.obj",+        )++    def test_complex_module_rename(self) -> None:+        before = """+            from a.b.c import d++            class Foo(d.e.f):+                pass+        """+        after = """+            import g.h.i++            class Foo(g.h.i.j):+                pass+        """+        self.assertCodemod(before, after, old_name="a.b.c.d.e.f", new_name="g.h.i.j")++    def test_names_with_repeated_substrings(self) -> None:+        before = """+            from aa import aaaa++            class Foo(aaaa.Bar):+                pass+        """+        after = """+            from b import c++            class Foo(c.Bar):+                pass+        """+        self.assertCodemod(+            before, after, old_name="aa.aaaa", new_name="b.c",+        )++    def test_import_star_attr(self) -> None:+        before = """+            from foo import *++            def baz():+                foo.bar(5)+        """

assuming the fully-qualified name of the thing is foo.bar, this initial code is wrong, if you do from foo import * you'd have just bar in your namespace, not foo.bar.

I think this codemod should basically ignore * imports. So this test would be correct if we have an unrelated * import and then an import foo, but we should never make any assumptions about names coming from * imports.

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict++from libcst.codemod import CodemodTest+from libcst.codemod.commands.rename import RenameCommand+++class TestRenameCommand(CodemodTest):++    ONCALL_SHORTNAME = "instagram_server_framework"++    TRANSFORM = RenameCommand++    def test_rename_name(self) -> None:++        before = """+            from foo import bar++            def test() -> None:+                bar(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(before, after, old_name="foo.bar", new_name="baz.qux")++    def test_rename_name_asname(self) -> None:++        before = """+            from foo import bar as bla++            def test() -> None:+                bla(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="baz.qux",+        )++    def test_rename_attr(self) -> None:++        before = """+            import a.b++            def test() -> None:+                a.b.c(5)+        """+        after = """+            import a.b+            import d.e++            def test() -> None:+                d.e.f(5)+        """++        self.assertCodemod(+            before, after, old_name="a.b.c", new_name="d.e.f",+        )++    def test_rename_attr_asname(self) -> None:++        before = """+            import foo as bar++            def test() -> None:+                bar.qux(5)+        """+        after = """+            import baz++            def test() -> None:+                baz.quux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.qux", new_name="baz.quux",+        )++    def test_rename_module_import(self) -> None:+        before = """+            import a.b++            class Foo(a.b.C):+                pass+        """+        after = """+            import c.b++            class Foo(c.b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",+        )++    def test_rename_module_import_from(self) -> None:+        before = """+            from a import b++            class Foo(b.C):+                pass+        """+        after = """+            from c import b++            class Foo(b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",

And here also we should verify the same before/after with old_name="a", new_name="b"

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict++from libcst.codemod import CodemodTest+from libcst.codemod.commands.rename import RenameCommand+++class TestRenameCommand(CodemodTest):++    ONCALL_SHORTNAME = "instagram_server_framework"++    TRANSFORM = RenameCommand++    def test_rename_name(self) -> None:++        before = """+            from foo import bar++            def test() -> None:+                bar(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(before, after, old_name="foo.bar", new_name="baz.qux")++    def test_rename_name_asname(self) -> None:++        before = """+            from foo import bar as bla++            def test() -> None:+                bla(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="baz.qux",+        )++    def test_rename_attr(self) -> None:++        before = """+            import a.b++            def test() -> None:+                a.b.c(5)+        """+        after = """+            import a.b

Shouldn't this be removed as now-unused?

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict++from libcst.codemod import CodemodTest+from libcst.codemod.commands.rename import RenameCommand+++class TestRenameCommand(CodemodTest):++    ONCALL_SHORTNAME = "instagram_server_framework"++    TRANSFORM = RenameCommand++    def test_rename_name(self) -> None:++        before = """+            from foo import bar++            def test() -> None:+                bar(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(before, after, old_name="foo.bar", new_name="baz.qux")++    def test_rename_name_asname(self) -> None:++        before = """+            from foo import bar as bla++            def test() -> None:+                bla(5)+        """+        after = """+            from baz import qux++            def test() -> None:+                qux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.bar", new_name="baz.qux",+        )++    def test_rename_attr(self) -> None:++        before = """+            import a.b++            def test() -> None:+                a.b.c(5)+        """+        after = """+            import a.b+            import d.e++            def test() -> None:+                d.e.f(5)+        """++        self.assertCodemod(+            before, after, old_name="a.b.c", new_name="d.e.f",+        )++    def test_rename_attr_asname(self) -> None:++        before = """+            import foo as bar++            def test() -> None:+                bar.qux(5)+        """+        after = """+            import baz++            def test() -> None:+                baz.quux(5)+        """++        self.assertCodemod(+            before, after, old_name="foo.qux", new_name="baz.quux",+        )++    def test_rename_module_import(self) -> None:+        before = """+            import a.b++            class Foo(a.b.C):+                pass+        """+        after = """+            import c.b++            class Foo(c.b.C):+                pass+        """++        self.assertCodemod(+            before, after, old_name="a.b", new_name="c.b",

Per comments above, I think we should also add a test where old_name="a", new_name="c" with the same before and after as in this test.

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],+        }++        self.scheduled_imports = False++    def visit_Import(self, node: cst.Import) -> None:+        for import_alias in node.names:+            alias_name = get_full_name_for_node(import_alias.name)+            if alias_name is not None:+                self.record_asname(import_alias, alias_name)++                # If the import statement is exactly equivalent to the old name, replace it.+                if alias_name == self.old_name:+                    self.cleanup_imports(node=node, new_module=self.new_name)++    def leave_ImportFrom(+        self, original_node: cst.ImportFrom, updated_node: cst.ImportFrom+    ) -> cst.ImportFrom:+        module = updated_node.module+        if module is not None:+            imported_module_name = get_full_name_for_node(module)+            names = original_node.names++            if imported_module_name is None or not isinstance(names, Sequence):+                return updated_node+            elif imported_module_name == self.old_name:+                # Simply rename the imported module on the spot.+                return updated_node.with_changes(+                    module=cst.parse_expression(self.new_name)+                )+            else:+                new_names = []+                for import_alias in names:+                    alias_name = get_full_name_for_node(import_alias.name)+                    if alias_name is not None:+                        qual_name = f"{imported_module_name}.{alias_name}"+                        if self.old_name == qual_name:+                            self.record_asname(import_alias, alias_name)++                            replacement_module = self.gen_replacement(+                                imported_module_name + "."+                            )+                            replacement_obj = self.gen_replacement(alias_name)+                            new_import_alias_name: cst.BaseExpression = cst.parse_expression(+                                replacement_obj+                            )+                            if not isinstance(+                                new_import_alias_name, (cst.Attribute, cst.Name)+                            ):+                                raise Exception("Something went wrong!")

This is a pretty classic not-useful error message :) I understand it should never happen and you're just keeping the type-checker happy, but it still seems like nothing would be lost by changing it to "parse_expression() on dotted path returned non-Attribute-or-Name".

Another alternative to consider might be a simple assert: assert isinstance(new_import_alias_name, (cst.Attribute, cst.Name))

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],+        }++        self.scheduled_imports = False++    def visit_Import(self, node: cst.Import) -> None:+        for import_alias in node.names:+            alias_name = get_full_name_for_node(import_alias.name)+            if alias_name is not None:+                self.record_asname(import_alias, alias_name)++                # If the import statement is exactly equivalent to the old name, replace it.+                if alias_name == self.old_name:+                    self.cleanup_imports(node=node, new_module=self.new_name)++    def leave_ImportFrom(+        self, original_node: cst.ImportFrom, updated_node: cst.ImportFrom+    ) -> cst.ImportFrom:+        module = updated_node.module+        if module is not None:+            imported_module_name = get_full_name_for_node(module)+            names = original_node.names++            if imported_module_name is None or not isinstance(names, Sequence):+                return updated_node+            elif imported_module_name == self.old_name:

Similar to above, seems like we're missing the case here where the changed name is a prefix of imported_module_name, e.g. we're renaming a.b to x.y and we have from a.b.c import d, which should become from x.y.c import d

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],+        }++        self.scheduled_imports = False++    def visit_Import(self, node: cst.Import) -> None:+        for import_alias in node.names:+            alias_name = get_full_name_for_node(import_alias.name)+            if alias_name is not None:+                self.record_asname(import_alias, alias_name)++                # If the import statement is exactly equivalent to the old name, replace it.

What about the case where the import statement contains the old name as a prefix? If we have old_name='a.b' and new_name='x.y', shouldn't we replace import a.b.c.d with import x.y.c.d?

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],

I'm a bit confused in reading the code about why some keys in self.equivalents have a trailing dot and others don't. Is there a meaning to the difference (i.e. it wouldn't work just to ensure no-trailing-dot on all keys)? I'm guessing there is a meaning, otherwise you wouldn't go to the trouble of ensuring the trailing dot on the second key here (and various other places). This would be a great place for a comment explaining what the meaning is!

It also might be worth considering whether two separate dictionaries would be easier to manage and clearer for the reader, rather than a single dictionary with the subtle distinction of some keys with trailing dot and some without.

josieesh

comment created time in 3 months

Pull request review commentInstagram/LibCST

Add codemod RenameCommand

+# Copyright (c) Facebook, Inc. and its affiliates.+#+# This source code is licensed under the MIT license found in the+# LICENSE file in the root directory of this source tree.+#+# pyre-strict+import argparse+from typing import Dict, Optional, Sequence, Union++import libcst as cst+from libcst.codemod import CodemodContext, VisitorBasedCodemodCommand+from libcst.codemod.visitors import AddImportsVisitor, RemoveImportsVisitor+from libcst.helpers import get_full_name_for_node+from libcst.metadata import QualifiedNameProvider+++class RenameCommand(VisitorBasedCodemodCommand):+    """+    Rename all instances of a local or imported object.+    """++    DESCRIPTION: str = "Rename all instances of a local or imported object."++    METADATA_DEPENDENCIES = (QualifiedNameProvider,)++    @staticmethod+    def add_args(parser: argparse.ArgumentParser) -> None:+        parser.add_argument(+            "--old_name",+            dest="old_name",+            required=True,+            help="Full dotted name of object to rename. Eg: `foo.bar.baz`",+        )++        parser.add_argument(+            "--new_name",+            dest="new_name",+            required=True,+            help="Full dotted name of replacement object. Eg: `foo.bar.baz`",+        )++    def __init__(self, context: CodemodContext, old_name: str, new_name: str) -> None:+        super().__init__(context)+        self.old_name = old_name+        self.new_name = new_name++        old_names = self.old_name.rpartition(".")+        new_names = self.new_name.rpartition(".")++        # Mapping for easy lookup later.+        self.equivalents: Dict[str, str] = {+            old_names[2]: new_names[2],+            "".join(old_names[:2]): new_names[0],+        }++        self.scheduled_imports = False++    def visit_Import(self, node: cst.Import) -> None:+        for import_alias in node.names:+            alias_name = get_full_name_for_node(import_alias.name)+            if alias_name is not None:+                self.record_asname(import_alias, alias_name)++                # If the import statement is exactly equivalent to the old name, replace it.+                if alias_name == self.old_name:+                    self.cleanup_imports(node=node, new_module=self.new_name)++    def leave_ImportFrom(+        self, original_node: cst.ImportFrom, updated_node: cst.ImportFrom+    ) -> cst.ImportFrom:+        module = updated_node.module+        if module is not None:

minor nit: maybe if module is None: return updated_node as a short-circuit here to avoid indenting the entire body of the function an extra level?

josieesh

comment created time in 3 months

more