profile
viewpoint
Pavel Minaev int19h Microsoft North Bend, WA, USA

int19h/aow-patch 10

Unofficial patch for Age of Wonders

int19h/WarBender 10

WarBender: a Mount & Blade: Warband save game editor

int19h/csx 3

C# scripting for Mount & Blade 2: Bannerlord

int19h/warbend 3

WarBend: a Python library to read and write Mount & Blade: Warband saved games

int19h/cpython 0

The Python programming language

int19h/debugpy 0

An implementation of the Debug Adapter Protocol for Python

int19h/django 0

The Web framework for perfectionists with deadlines.

int19h/FirearmDeathRate 0

Use regression tree to predict firearm death rate with firearm law & CDC firearm death rate data.

int19h/kiwix-tools 0

Command Kiwix tools: kiwix-serve, kiwix-install, kiwix-manage, ...

int19h/mb_warband_module_system 0

Mount & Blade: Warband module system (mirror)

PullRequestReviewEvent

delete branch int19h/debugpy

delete branch : 330

delete time in 7 hours

push eventmicrosoft/debugpy

Pavel Minaev

commit sha 1727a3e109c6e82927eb49932530b0c95a8b5c11

Fix #330: Unable to debug pythonw apps on macOS In the launcher, use "python" instead of sys.executable to spawn the debuggee. Update test for "python" / "pythonPath" / "pythonArgs" to use a helper script to assert the use of a custom Python binary; and test -B instead of -v to minimize test log size and run time.

view details

push time in 7 hours

issue closedmicrosoft/debugpy

Unable to debug pythonw apps on macOS

After updating the extension, the debugger doesn't allow debugging pythonw apps anymore. Instead, it tries to launch python, which makes it unusable for wxpython development on MacOS.

Reverting to v2020.3.71659 solves the issue.

Environment data

  • VS Code version: 1.46.1 (cd9ea64)
  • Extension version (available under the Extensions sidebar): XXX
  • OS and version: MacOS Catalina 10.15.5
  • Python version (& distribution if applicable, e.g. Anaconda): Miniconda 4.8.3 + Python 3.7.0
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
  • Relevant/affected Python packages and their versions: wxPython v4+ (at least, older versions are possibly affected as well)
  • Relevant/affected Python-related VS Code extensions and their versions: XXX
  • Value of the python.languageServer setting: XXX

Expected behaviour

When debugging a python app with pythonw in the launch.json settings, the debugger should load the file using pythonw.

Actual behaviour

The following error occurs:

Exception has occurred: SystemExit
This program needs access to the screen. Please run with a
Framework build of python, and only when you are logged in
on the main display of your Mac.

Which is simply caused by MacOS not granting access to the screen to python. Normally, this is solved by running apps using pythonw, but the latest versions of the python extension ignore the setting in launch.json.

Steps to reproduce:

Create any wxPython app and try to debug it on a Mac. Example:

import wx
import sys

class WxApplication(wx.App):
    def __init__(self):
        wx.App.__init__(self)

def main():
    WxApplication()

if __name__ == "__main__":
    sys.exit(main())

This will happen regardless of the "pythonPath" setting in launch.json.

Logs

<details>

<summary>Output for <code>Python</code> in the <code>Output</code> panel (<code>View</code>→<code>Output</code>, change the drop-down the upper-right of the <code>Output</code> panel to <code>Python</code>) </summary>

<p>

XXX

</p> </details>

closed time in 7 hours

jrockwar

PR merged microsoft/debugpy

Fix #330: Unable to debug pythonw apps on macOS

In the launcher, use "python" instead of sys.executable to spawn the debuggee.

Update test for "python" / "pythonPath" / "pythonArgs" to use a helper script to assert the use of a custom Python binary; and test -B instead of -v to minimize test log size and run time.

+39 -18

1 comment

4 changed files

int19h

pr closed time in 7 hours

push eventint19h/debugpy

Pavel Minaev

commit sha 18a04ab659c1795f67aac641709def32fffead39

Fix #330: Unable to debug pythonw apps on macOS In the launcher, use "python" instead of sys.executable to spawn the debuggee. Update test for "python" / "pythonPath" / "pythonArgs" to use a helper script to assert the use of a custom Python binary; and test -B instead of -v to minimize test log size and run time.

view details

push time in 8 hours

PR opened microsoft/debugpy

Reviewers
Fix #330: Unable to debug pythonw apps on macOS

In the launcher, use "python" instead of sys.executable to spawn the debuggee.

Update test for "python" / "pythonPath" / "pythonArgs" to use a helper script to assert the use of a custom Python binary; and test -B instead of -v to minimize test log size and run time.

+37 -18

0 comment

4 changed files

pr created time in 8 hours

create barnchint19h/debugpy

branch : 330

created branch time in 8 hours

issue closedmicrosoft/debugpy

ValueFormat should support "rawValue"

pydevd supports "rawString" in ValueFormat to obtain the raw (non-repr()) value of the variable. However, VS documents the flag as "rawValue". We should support the latter for VS, and keep "rawString" for back-compat.

closed time in 8 hours

int19h

issue commentmicrosoft/debugpy

ValueFormat should support "rawValue"

Never mind - the spec is wrong, VS actually sends "rawString".

int19h

comment created time in 8 hours

issue openedmicrosoft/debugpy

ValueFormat should support "rawValue"

pydevd supports "rawString" in ValueFormat to obtain the raw (non-repr()) value of the variable. However, VS documents the flag as "rawValue". We should support the latter for VS, and keep "rawString" for back-compat.

created time in 9 hours

PullRequestReviewEvent

issue commentmicrosoft/debugpy

Messages sequence for attach different from that of PTVSD

@JohanMabille It's a legal sequence, but it's up to the debug adapter to signal the "initialized" event when the conditions are met. In case of debugpy, the conditions aren't met until after you issue the "launch" or "attach" request.

The reason why it needs it is because the adapter doesn't handle breakpoints - the debug server does. But, there's no debug server until one is either spawned (if "launch") or attached to. If "setBreakpoints" were an event, it could be deferred and propagated when the server is there - but it's a request, requiring a response, so you'd just end up with a deadlock where the client is waiting for a response, and the adapter is waiting for "launch" or "attach".

This is different from ptvsd 4, because in that one, the debug adapter proper was not a part of ptvsd at all, but rather a bunch of TypeScript code in the VSCode extension - ptvsd itself was strictly a debug server, and didn't meaningfully handle "launch" or "attach" at all, since, by definition, it's already in the process it's trying to debug. The old adapter was aware of ptvsd's non-coformance idiosyncracies, and massaged them into something that VSCode could handle; in particular, it would not send "initialized" event until it received "launch", and spawned the ptvsd server (and thus ready to receive "setBreakpoints"). The new adapter that's a part of debugpy itself does the same thing.

Previously, if you were using ptvsd directly as a client - e.g. by attaching to the socket - you were skipping the old adapter, and thus getting ptvsd's broken implementation of DAP. This is not an option with debugpy anymore.

DAP spec needs to be updated to reflect the diagram linked above. There was an issue about that on their issue tracker somewhere. But in any case, the client is supposed to follow the adapter lead here - it's up to the adapter to decide which sequence is more appropriate to its implementation (whether it's because that's the only way to do it, or because it's just easier). A conforming client should wait until the "initialized" event is received before sending "setBreakpoints", and should be able to handle either sequence in the diagram - meaning that it cannot wait for "launch" or "attach" response before sending "setBreakpoints" and/or "configurationDone". This is not something that's clear from the DAP spec by itself, but it doesn't contradict it, either.

(@weinand, please correct me if anything is wrong in the above.)

From our perspective, which sequence debugpy implements is an implementation detail that can change at any time in the future, and should not be relied upon. The only guarantee that we make here is conformance to the spec as clarified by the diagram.

JohanMabille

comment created time in a day

PullRequestEvent

PR closed microsoft/debugpy

Reviewers
Fix #93: null values not supported in "env"
+22 -9

1 comment

2 changed files

int19h

pr closed time in a day

PR opened microsoft/debugpy

Reviewers
Fix #93: null values not supported in "env"
+22 -9

0 comment

2 changed files

pr created time in a day

create barnchint19h/debugpy

branch : 93

created branch time in a day

PullRequestReviewEvent

issue commentmicrosoft/debugpy

Wrong evaluation of if statement in debugging

This sounds like https://github.com/microsoft/debugpy/issues/346/. Can you try the workaround there? In your launch.json:

"env": {"PYDEVD_USE_FRAME_EVAL": "NO"}

If this helps, it's the same bug, and we already fixed it - the fix will be in the next version of the extension.

sadjadanzabizadeh

comment created time in 2 days

issue closedmicrosoft/debugpy

Messages sequence for attach different from that of PTVSD

Environment data

  • debugpy version: 1.0.0b9
  • OS and version: gentoo (5.4.60-gentoo-x86_64)
  • Python version (& distribution if applicable, e.g. Anaconda): 3.8.6 from conda
  • Using VS Code or Visual Studio: No

Actual behavior

When trying to attach a process to the debugpy, the expected messages sequence is the following:

client -> debugpy: initialize request
client <- debugpy: intialize response
client <- debugpy: initialize event
client -> debugpy: attach request
client -> debugpy: setBreakpoints request
client <- debugpy: setBreakpoints response
client -> debugpy: configurationDone request
client <- debugpy: configurationDone response
client <- debugpy: attach response

This is different from the messages sequence required to attach a process to PTVSD:

client -> PTVSD: initialize request
client <- PTVSD: initialize response
client <- PTVSD: initialize event
client -> PTVSD: attach request
client <- PTVSD: attach response
client -> PTVSD: setBreakpoints request
client <- PTVSD: setBreakpoints response
client -> PTVSD: configurationDone request
client <- PTVSD: configurationDone response

Expected behavior

The sequence should be the same as that of PTVSD, that is, attach response should be sent once the process is attached instead of waiting for configurationDone request. Besides, this seems to break the "reply - response" pattern from the Debug Adapter protocol, and is not consistent with the initialization sequence described on this page

Steps to reproduce:

Build xeus-python from this branch: jupyter-xeus/xeus-python#329 Try to run the bugger

closed time in 2 days

JohanMabille

issue commentmicrosoft/debugpy

Messages sequence for attach different from that of PTVSD

As noted above, this is by design, and in conformance with the DAP specification. The older ptvsd behavior was actually not fully conforming, because we misunderstood the spec.

JohanMabille

comment created time in 2 days

issue closedmicrosoft/ptvsd

Using latest version of the debugger.

Download and install insiders build of the python extension: https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix

image

To use the new debugger please add the following to your user settings: image

    "python.experiments.optInto": [
        "DebugAdapterFactory - experiment",
        "PtvsdWheels37 - experiment"
    ],

Reload VSCode. if you look at Output tab > Python panel, you should see these lines: image

This should get you the new debugger.

closed time in 2 days

karthiknadig
PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1 -    The list comprises of 3-item tuples that contain the starting line number,-    ending line number and whether the statement is a single line.+    # If it is a single line statement: Dedent and skip to the end.+    # Else: Parse the multiline selection into a list of top-level blocks.+    if is_singleline:+        statements = [textwrap.dedent(selection)]

Wouldn't _get_multiline_statements() do essentially the same thing in the end, though?

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1 -    The list comprises of 3-item tuples that contain the starting line number,-    ending line number and whether the statement is a single line.+    # If it is a single line statement: Dedent and skip to the end.+    # Else: Parse the multiline selection into a list of top-level blocks.+    if is_singleline:+        statements = [textwrap.dedent(selection)]

I meant that erroring out (regardless of how it's reported) is the desired result... since the only other option leaves us at the continuation prompt in the REPL - and then follow-up submissions will not behave correctly.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

create barnchint19h/debugpy

branch : main

created branch time in 2 days

delete branch microsoft/debugpy

delete branch : master

delete time in 2 days

issue commentmicrosoft/vscode-python

Send selection to REPL will fail on a large file

This should also solve encoding issues on Python 2. As things are, if you try to send this:

print('🔨')

you get this:

>>> print('??')
??
kimadeline

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1 -    The list comprises of 3-item tuples that contain the starting line number,-    ending line number and whether the statement is a single line.+    # If it is a single line statement: Dedent and skip to the end.+    # Else: Parse the multiline selection into a list of top-level blocks.+    if is_singleline:+        statements = [textwrap.dedent(selection)]+    else:+        statements = _get_multiline_statements(selection) -    """-    tree = ast.parse(source)-    visitor = Visitor(lines)-    visitor.visit(tree)--    statement_ranges = []-    for index, line_number in enumerate(visitor.line_numbers_with_statements):-        remaining_line_numbers = visitor.line_numbers_with_statements[index + 1 :]-        end_line_number = (-            len(lines)-            if len(remaining_line_numbers) == 0-            else min(remaining_line_numbers) - 1-        )-        current_statement_is_oneline = line_number == end_line_number--        if len(statement_ranges) == 0:-            statement_ranges.append(-                (line_number, end_line_number, current_statement_is_oneline)-            )-            continue--        previous_statement = statement_ranges[-1]-        previous_statement_is_oneline = previous_statement[2]-        if previous_statement_is_oneline and current_statement_is_oneline:-            statement_ranges[-1] = previous_statement[0], end_line_number, True-        else:-            statement_ranges.append(-                (line_number, end_line_number, current_statement_is_oneline)-            )--    return statement_ranges---def normalize_lines(source):-    """Normalize blank lines for sending to the terminal.--    Blank lines within a statement block are removed to prevent the REPL-    from thinking the block is finished. Newlines are added to separate-    top-level statements so that the REPL does not think there is a syntax-    error.+    # Insert a newline between each top-level statement, and append a newline to the selection.+    source = "\n".join(statements) + "\n" -    """-    # Ensure to dedent the code (#2837)-    lines = textwrap.dedent(source).splitlines(False)-    # If we have two blank lines, then add two blank lines.-    # Do not trim the spaces, if we have blank lines with spaces, its possible-    # we have indented code.-    if (len(lines) > 1 and len("".join(lines[-2:])) == 0) or source.endswith(-        ("\n\n", "\r\n\r\n")-    ):-        trailing_newline = "\n" * 2-    # Find out if we have any trailing blank lines-    elif len(lines[-1].strip()) == 0 or source.endswith(("\n", "\r\n")):-        trailing_newline = "\n"-    else:-        trailing_newline = ""--    # Step 1: Remove empty lines.-    tokens = _tokenize(source)-    newlines_indexes_to_remove = (-        spos[0]-        for (toknum, tokval, spos, epos, line) in tokens-        if len(line.strip()) == 0-        and token.tok_name[toknum] == "NL"-        and spos[0] == epos[0]-    )--    for line_number in reversed(list(newlines_indexes_to_remove)):-        del lines[line_number - 1]--    # Step 2: Add blank lines between each global statement block.-    # A consecutive single lines blocks of code will be treated as a single statement,-    # just to ensure we do not unnecessarily add too many blank lines.-    source = "\n".join(lines)-    tokens = _tokenize(source)-    dedent_indexes = (-        spos[0]-        for (toknum, tokval, spos, epos, line) in tokens-        if toknum == token.DEDENT and _indent_size(line) == 0-    )--    global_statement_ranges = _get_global_statement_blocks(source, lines)-    start_positions = map(operator.itemgetter(0), reversed(global_statement_ranges))-    for line_number in filter(lambda x: x > 1, start_positions):-        lines.insert(line_number - 1, "")--    sys.stdout.write("\n".join(lines) + trailing_newline)+    # Finally, send the formatted selection to the REPL.+    sys.stdout.write(source)

source is a unicode instance at this point on Python 2, isn't it? If so, it will implicitly encode it using sys.getdefaultencoding(), which is almost certainly the wrong thing. I'm assuming the Node end will try to decode it as UTF-8? In this case, for maximum portability, it should be something like:

stdout = sys.stdout if sys.version_info < (3,) else sys.stdout.buffer
stdout.write(source.encode("utf-8"))
kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1 -    The list comprises of 3-item tuples that contain the starting line number,-    ending line number and whether the statement is a single line.+    # If it is a single line statement: Dedent and skip to the end.+    # Else: Parse the multiline selection into a list of top-level blocks.+    if is_singleline:+        statements = [textwrap.dedent(selection)]

This circumvents ast.parse(). So if it is a single-line statement which is unterminated, e.g.

x = [

it will go here, right? And because it won't try to parse, it'll submit that as is, and the user will end up at the ... continuation prompt.

Does this actually need to be a special case at all? If I'm following the code correctly, for any complete single-line statement, _get_multiline_statements() will do the same thing this does, anyway.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1 -    The list comprises of 3-item tuples that contain the starting line number,-    ending line number and whether the statement is a single line.+    # If it is a single line statement: Dedent and skip to the end.+    # Else: Parse the multiline selection into a list of top-level blocks.+    if is_singleline:+        statements = [textwrap.dedent(selection)]+    else:+        statements = _get_multiline_statements(selection)

Since _get_multiline_statements immediately splits selection, and never uses the original value again, this caller might as well stash the result of the earlier split, and just pass it in.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"++        statements.append(block)++    return statements+++def normalize_lines(selection):+    """+    Normalize the text selection received from the extension and send it to the REPL. +    If it is a single line selection, dedent it, append a newline and send it to the REPL.+    Otherwise, sanitize the multiline selection before sending it to the REPL:+    split it in a list of top-level statements+    and add newlines between each of them to tell the REPL where each block ends.+    """ -def _get_global_statement_blocks(source, lines):-    """Return a list of all global statement blocks.+    # Check if it is a singleline or multiline selection.+    is_singleline = len(selection.splitlines()) == 1

Same thing about splitlines() applies here, so that logic should probably go into a helper function for reuse.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""]

So, looking at the language spec, Python grammar only treats \r, \n, and \r\n as newlines. But splitlines() in Python 3 has a much larger list - it also includes \v, \f, and some even more exotic stuff.

I think you'll have to use re.split() here for this to work as it should across all Python versions.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):+        # Given this selection:+        # if (m > 0 and+        #        n < 3):+        #     print('foo')+        # value = 'bar'+        #+        # The first block would have lineno = 1,and the second block lineno = 4+        start = node.lineno - 1+        end = len(lines) if idx == last_idx else tree.body[idx + 1].lineno - 1+        block = "\n".join(lines[start:end])++        # If the block is multiline, add an extra newline character at its end.+        # This way, when joining blocks back together, there will be a blank line between each multiline statement+        # and no blank lines between single-line statements, or it would look like this:+        # >>> x = 22+        # >>>+        # >>> y = 30+        # >>>+        # >>> total = x + y+        # >>>+        if end - start > 1:+            block += "\n"

One thing that didn't occur to me when we were discussing this, is that for the multiline parentheses case, this newline is redundant, since the closing parenthesis terminates the statement already. So for this input:

x = [
   1
]
y = [
   2
]

The REPL will get a blank line between the two, even though it's not strictly needed:

>>> x = [
...   1
... ]
>>> 
>>> y = [
...   2
...]

I think that's okay, since that extra blank line doesn't change behavior, and appearance is not such a big deal for an uncommon code pattern. But it might be worth pointing out in the comments.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""] +    # Dedent the selection and parse it using the ast module.+    # Note that leading comments in the selection will be discarded during parsing.+    source = textwrap.dedent("\n".join(lines))+    tree = ast.parse(source) -def _indent_size(line):-    for index, char in enumerate(line):-        if not char.isspace():-            return index+    # We'll need the dedented lines to rebuild the selection.+    lines = source.splitlines(False)++    # Get the line ranges for top-level blocks returned from parsing the dedented text+    # and split the selection accordingly.+    # tree.body is a list of AST objects, which we rely on to extract top-level statements.+    # If we supported Python 3.8+ only we could use the lineno and end_lineno attributes of each object+    # to get the boundaries of each block.+    # However, earlier Python versions only have the lineno attribute, which is the range start position (1-indexed).+    # Therefore, to retrieve the end line of each block in a version-agnostic way we need to do+    # `end = next_block.lineno - 1`+    # for all blocks except the last one, which will will just run until the last line.+    last_idx = len(tree.body) - 1+    for idx, node in enumerate(tree.body):

zip() can come in handy here:

ends = [node.lineno - 1 for node in tree.body[1:]] + [len(lines)]
for node, end in zip(tree.body, ends):
    ...
kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""]

Also, if I remember correctly, splitlines() defaults to False.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = [] -def _tokenize(source):-    """Tokenize Python source code."""-    # Using an undocumented API as the documented one in Python 2.7 does not work as needed-    # cross-version.-    if sys.version_info < (3,) and isinstance(source, str):-        source = source.decode()-    return tokenize.generate_tokens(io.StringIO(source).readline)+    # Remove blank lines within the selection to prevent the REPL from thinking the block is finished.+    lines = [line for line in selection.splitlines(False) if line.strip() != ""]

Similarly, can use () instead of [] here to get a lazy iterator instead of a list temporary - join() works fine with iterators.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Code parsing for run selection in terminal - Python side

 # Licensed under the MIT License.  import ast-import io-import operator-import os-import sys import textwrap-import token-import tokenize---class Visitor(ast.NodeVisitor):-    def __init__(self, lines):-        self._lines = lines-        self.line_numbers_with_nodes = set()-        self.line_numbers_with_statements = []--    def generic_visit(self, node):-        if (-            hasattr(node, "col_offset")-            and hasattr(node, "lineno")-            and node.col_offset == 0-        ):-            self.line_numbers_with_nodes.add(node.lineno)-            if isinstance(node, ast.stmt):-                self.line_numbers_with_statements.append(node.lineno)+import sys -        ast.NodeVisitor.generic_visit(self, node) +def _get_multiline_statements(selection):+    """+    Process a multiline selection into a list of its top-level statements.+    This will remove empty newlines around and within the selection, dedent it,+    and split it using the result of `ast.parse()`.+    """+    statements = []

If you make the whole function a generator, you can just yield items one by one from within the loop instead of accumulating them.

kimadeline

comment created time in 2 days

PullRequestReviewEvent

issue closedmicrosoft/ptvsd

Press F5 does not hit breakpoint in remote debug.

Environment data

Build of Visual Studio Code - Insiders: 1.47.0-insider Build of python-insider: 2020.7.92197-dev

Actual behavior

the breakpoint can be hit and contiune pressing F5 , it should hit the breakpoint again.

Expected behavior

the breakpoint can be hit in the first time, but if continue pressing F5, the bp can not be hit and print all the result directly.

Steps to reproduce:

  1. Create python file named remote_debugging.py, enter content as following and set breakpoint in the sencond line:
for i in range(1, 10, 2):
    print (i)
print('demo done.')

image

2.Copy this file to ubuntu which should include ptvsd installed with same version as current machine a. install putty ( https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) b. paste the IP of ubuntu
c. input user and pwd of ubuntu d. install ptvsd in ubuntu (pip install ptvsd) e. find pscp.exe in PUTTY(C:\Program Files\PUTTY) f. copy pscp.exe in C:\Windows\System32 g. open cmd and input below command: pscp <workspace of windows> <user of ubuntu>@<IP of ubuntu>:<workspace of ubuntu> 3. Execute python file in ubuntu: a. python –m ptvsd --host 0.0.0.0 --port 5678 --wait remote_debugging.py 4.Select configuration "Python: attach", set the host value with IP of ubuntu/other windows 5.Press F5 to run the code

closed time in 2 days

zhouwangyang

issue commentmicrosoft/ptvsd

Press F5 does not hit breakpoint in remote debug.

Please try this with https://github.com/microsoft/debugpy, and if you are still having problems, open an issue in that repo.

zhouwangyang

comment created time in 2 days

issue closedmicrosoft/ptvsd

Debugger Timedout if kept idle for some time

Environment data

  • PTVSD version: 2020.6.90262
  • OS and version: Mac OS and 10.15.3
  • Python version (& distribution if applicable, e.g. Anaconda): 3.7.6
  • Using VS Code or Visual Studio: VS Code and Version: 1.46.1

Actual behavior

We are using ptvsd in some what different way. We embedded python interpreter in a C++ application. We are using ptvsd to debug the scripts which is written using the API provided by our application. To do that We load PTVSD module in the embedded python interpreter and open a socket to set up a remote debugging session with VSCode. It was working fine so far but now we started seeing timeout issue showing below message where debugging is stopped if kept idle for around 30 seconds.

Exception in thread ptvsd.Server: Traceback (most recent call last): File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/ipcjson.py", line 269, in process_one_message msg = self.__message.pop(0) IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/goyals/Library/Application Support/Autodesk/webdeploy/feature--ChangesFromAPIPerforceBranch/5fe9cf01e67436462626acce9bec5ab16b7b3fb7/Autodesk Fusion 360.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/Users/goyals/Library/Application Support/Autodesk/webdeploy/feature--ChangesFromAPIPerforceBranch/5fe9cf01e67436462626acce9bec5ab16b7b3fb7/Autodesk Fusion 360.app/Contents/Frameworks/Python.framework/Versions/Current/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/wrapper.py", line 536, in process_messages raise exc File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/wrapper.py", line 521, in process_messages self.process_messages() File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/ipcjson.py", line 258, in process_messages self.process_one_message() File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/ipcjson.py", line 272, in process_one_message self._wait_for_message() File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/ipcjson.py", line 154, in _wait_for_message line = self._buffered_read_line_as_ascii() File "/Users/goyals/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/ipcjson.py", line 113, in _buffered_read_line_as_ascii temp = self.__socket.recv(1024) socket.timeout: timed out

A similar issue is reported here https://github.com/Microsoft/ptvsd/issues/1113.

I tried setting timeout property to a higher number in launch.json as suggested by some other user's post but that did not help. I am not sure if the connection is dropped by VSCode debug adapter if there is no response in a specified time.

I appreciate if you can provide any suggestion on what might be going wrong here and there is any way we can increase the default time out duration from 30 seconds to some higher number. Thanks.

closed time in 2 days

shyamiisc

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { traceVerbose } from '../../../../common/logger';+import { chain, iterable } from '../../../../common/utils/async';+import {+    getEnvironmentVariable, getUserHomeDir,+} from '../../../../common/utils/platform';+import {+    PythonEnvInfo, PythonEnvKind, UNKNOWN_PYTHON_VERSION,+} from '../../../base/info';+import { buildEnvInfo } from '../../../base/info/env';+import { ILocator, IPythonEnvsIterator } from '../../../base/locator';+import { PythonEnvsWatcher } from '../../../base/watcher';+import { findInterpretersInDir } from '../../../common/commonUtils';+import { getFileInfo, pathExists } from '../../../common/externalDependencies';+import { isVenvEnvironment, isVirtualenvEnvironment, isVirtualenvwrapperEnvironment } from './virtualEnvironmentIdentifier';++const DEFAULT_SEARCH_DEPTH = 2;++/**+ * Gets all default virtual environment locations. This uses WORKON_HOME,+ * and user home directory to find some known locations where global virtual+ * environments are often created.+ */+async function getGlobalVirtualEnvDirs(): Promise<string[]> {+    const venvDirs:string[] = [];++    const workOnHome = getEnvironmentVariable('WORKON_HOME');+    if (workOnHome && await pathExists(workOnHome)) {+        venvDirs.push(workOnHome);+    }++    const homeDir = getUserHomeDir();+    if (homeDir && await pathExists(homeDir)) {+        const subDirs = ['Envs', 'envs', '.direnv', '.venvs', '.virtualenvs'];

Oh wait, I missed the part where it uses includes() to check! That's just wrong, though - it's not properly case-insensitive on Windows, and "envs" vs "Envs" only takes care of the first letter.

karthiknadig

comment created time in 3 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { traceVerbose } from '../../../../common/logger';+import { chain, iterable } from '../../../../common/utils/async';+import {+    getEnvironmentVariable, getUserHomeDir,+} from '../../../../common/utils/platform';+import {+    PythonEnvInfo, PythonEnvKind, UNKNOWN_PYTHON_VERSION,+} from '../../../base/info';+import { buildEnvInfo } from '../../../base/info/env';+import { ILocator, IPythonEnvsIterator } from '../../../base/locator';+import { PythonEnvsWatcher } from '../../../base/watcher';+import { findInterpretersInDir } from '../../../common/commonUtils';+import { getFileInfo, pathExists } from '../../../common/externalDependencies';+import { isVenvEnvironment, isVirtualenvEnvironment, isVirtualenvwrapperEnvironment } from './virtualEnvironmentIdentifier';++const DEFAULT_SEARCH_DEPTH = 2;++/**+ * Gets all default virtual environment locations. This uses WORKON_HOME,+ * and user home directory to find some known locations where global virtual+ * environments are often created.+ */+async function getGlobalVirtualEnvDirs(): Promise<string[]> {+    const venvDirs:string[] = [];++    const workOnHome = getEnvironmentVariable('WORKON_HOME');+    if (workOnHome && await pathExists(workOnHome)) {+        venvDirs.push(workOnHome);+    }++    const homeDir = getUserHomeDir();+    if (homeDir && await pathExists(homeDir)) {+        const subDirs = ['Envs', 'envs', '.direnv', '.venvs', '.virtualenvs'];

Windows shouldn't need it, since readdir() is going to be case-insensitive anyway, right? (and if not, then this is the wrong way to do case-insensitivity anyway). I thought it's here because it's a possibility on Unix.

karthiknadig

comment created time in 3 days

PullRequestReviewEvent

create barnchmicrosoft/debugpy

branch : main

created branch time in 3 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import {+    getEnvironmentVariable, getOSType, getUserHomeDir, OSType,+} from '../../../../common/utils/platform';+import { pathExists } from '../../../common/externalDependencies';++/**+ * Checks if the given interpreter belongs to a venv based environment.+ * @param {string} interpreterPath: Absolute path to the python interpreter.+ * @returns {boolean} : Returns true if the interpreter belongs to a venv environment.+ */+export async function isVenvEnvironment(interpreterPath:string): Promise<boolean> {+    const pyvenvConfigFile = 'pyvenv.cfg';++    // Check if the pyvenv.cfg file is in the directory as the interpreter.+    // env+    // |__ pyvenv.cfg  <--- check if this file exists+    // |__ python  <--- interpreterPath+    const venvPath1 = path.join(path.dirname(interpreterPath), pyvenvConfigFile);++    // Check if the pyvenv.cfg file is in the parent directory relative to the interpreter.+    // env+    // |__ pyvenv.cfg  <--- check if this file exists+    // |__ bin or Scripts+    //     |__ python  <--- interpreterPath+    const venvPath2 = path.join(path.dirname(path.dirname(interpreterPath)), pyvenvConfigFile);++    return [await pathExists(venvPath1), await pathExists(venvPath2)].includes(true);+}++/**+ * Checks if the given interpreter belongs to a virtualenv based environment.+ * @param {string} interpreterPath: Absolute path to the python interpreter.+ * @returns {boolean} : Returns true if the interpreter belongs to a virtualenv environment.+ */+export async function isVirtualenvEnvironment(interpreterPath:string): Promise<boolean> {+    // Check if there are any activate.* files in the same directory as the interpreter.+    //+    // env+    // |__ activate, activate.*  <--- check if any of these files exist+    // |__ python  <--- interpreterPath+    const directory = path.dirname(interpreterPath);+    const files = await fsapi.readdir(directory);+    const regex = /^activate(\.([A-z]|\d)+)?$/;

Do we care if the extension is non-alphanumeric or blank? It feels like (\..*)? would do fine here.

Also, this should probably be case-insensitive on Windows. Or maybe on all platforms, even?

karthiknadig

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import {+    getEnvironmentVariable, getOSType, getUserHomeDir, OSType,+} from '../../../../common/utils/platform';+import { pathExists } from '../../../common/externalDependencies';++/**+ * Checks if the given interpreter belongs to a venv based environment.+ * @param {string} interpreterPath: Absolute path to the python interpreter.+ * @returns {boolean} : Returns true if the interpreter belongs to a venv environment.+ */+export async function isVenvEnvironment(interpreterPath:string): Promise<boolean> {+    const pyvenvConfigFile = 'pyvenv.cfg';++    // Check if the pyvenv.cfg file is in the directory as the interpreter.+    // env+    // |__ pyvenv.cfg  <--- check if this file exists+    // |__ python  <--- interpreterPath+    const venvPath1 = path.join(path.dirname(interpreterPath), pyvenvConfigFile);++    // Check if the pyvenv.cfg file is in the parent directory relative to the interpreter.+    // env+    // |__ pyvenv.cfg  <--- check if this file exists+    // |__ bin or Scripts+    //     |__ python  <--- interpreterPath+    const venvPath2 = path.join(path.dirname(path.dirname(interpreterPath)), pyvenvConfigFile);++    return [await pathExists(venvPath1), await pathExists(venvPath2)].includes(true);

This is more expensive than it needs to be, since both pathExists() checks will always run even if the first one succeeds, and they will run sequentially rather than in parallel. I think this would be better with &&.

karthiknadig

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as path from 'path';+import { traceError, traceVerbose } from '../../../../common/logger';+import { chain, iterable } from '../../../../common/utils/async';+import {+    Architecture, getEnvironmentVariable, getUserHomeDir,+} from '../../../../common/utils/platform';+import {+    PythonEnvInfo, PythonEnvKind, PythonVersion, UNKNOWN_PYTHON_VERSION,+} from '../../../base/info';+import { parseVersion } from '../../../base/info/pythonVersion';+import { ILocator, IPythonEnvsIterator } from '../../../base/locator';+import { PythonEnvsWatcher } from '../../../base/watcher';+import { findInterpretersInDir } from '../../../common/commonUtils';+import { getFileInfo, pathExists } from '../../../common/externalDependencies';+import { isVenvEnvironment, isVirtualenvEnvironment, isVirtualenvwrapperEnvironment } from './virtualEnvironmentIdentifier';++const DEFAULT_SEARCH_DEPTH = 4;++/**+ * Gets all default virtual environment locations. This uses WORKON_HOME,+ * and user home directory to find some known locations where global virtual+ * environments are often created.+ */+async function getGlobalVirtualEnvDirs(): Promise<string[]> {+    const venvDirs:string[] = [];+    const dirPaths:string[] = [];++    const workOnHome = getEnvironmentVariable('WORKON_HOME');+    if (workOnHome) {+        dirPaths.push(workOnHome);+    }++    const homeDir = getUserHomeDir();+    if (homeDir) {+        dirPaths.push(path.join(homeDir, 'envs'));+        dirPaths.push(path.join(homeDir, '.direnv'));+        dirPaths.push(path.join(homeDir, '.venvs'));+        dirPaths.push(path.join(homeDir, '.virtualenvs'));+        dirPaths.push(path.join(homeDir, '.local', 'share', 'virtualenvs'));+    }++    const exists = await Promise.all(dirPaths.map(pathExists));+    exists.forEach((v, i) => {+        if (v) {+            venvDirs.push(dirPaths[i]);+        }+    });++    return venvDirs;+}++/**+ * Gets the virtual environment kind for a given interpreter path.+ * This only checks for environments created using venv, virtualenv,+ * and virtualenvwrapper based environments.+ * @param interpreterPath: Absolute path to the interpreter paths.+ */+async function getVirtualEnvKind(interpreterPath:string): Promise<PythonEnvKind> {+    if (await isVenvEnvironment(interpreterPath)) {+        return PythonEnvKind.Venv;+    }++    if (await isVirtualenvwrapperEnvironment(interpreterPath)) {+        return PythonEnvKind.VirtualEnvWrapper;+    }++    if (await isVirtualenvEnvironment(interpreterPath)) {+        return PythonEnvKind.VirtualEnv;+    }++    return PythonEnvKind.Unknown;+}++/**+ * Takes absolute path to the interpreter and environment kind to build+ * environment info.+ * @param {string} interpreterPath: Absolute path to interpreter.+ * @param {PythonEnvKind} kind: Kind for the given environment.+ */+async function buildEnvInfo(interpreterPath:string, kind:PythonEnvKind): Promise<PythonEnvInfo> {+    let version:PythonVersion;+    try {+        version = parseVersion(path.basename(interpreterPath));+    } catch (ex) {+        traceError(`Failed to parse version from path: ${interpreterPath}`, ex);+        version = UNKNOWN_PYTHON_VERSION;+    }+    return {+        name: '',+        location: '',+        kind,+        executable: {+            filename: interpreterPath,+            sysPrefix: '',+            ...(await getFileInfo(interpreterPath)),+        },+        version,+        arch: Architecture.Unknown,+        distro: { org: '' },+    };+}++/**+ * Finds and resolves virtual environments created in known global locations.+ */+export class GlobalVirtualEnvironmentLocator extends PythonEnvsWatcher implements ILocator {+    private virtualEnvKinds = [+        PythonEnvKind.Venv,+        PythonEnvKind.VirtualEnv,+        PythonEnvKind.VirtualEnvWrapper,+    ];++    public constructor(private readonly searchDepth?:number) {+        super();+    }++    public iterEnvs(): IPythonEnvsIterator {+        // Number of levels of sub-directories to recurse when looking for+        // interpreters+        const searchDepth = this.searchDepth ?? DEFAULT_SEARCH_DEPTH;++        async function* iterator(virtualEnvKinds:PythonEnvKind[]) {+            const envRootDirs = await getGlobalVirtualEnvDirs();+            const envGenerators = envRootDirs.map((envRootDir) => {+                async function* generator() {+                    traceVerbose(`Searching for global virtual envs in: ${envRootDir}`);++                    const envGenerator = findInterpretersInDir(envRootDir, searchDepth);++                    for await (const env of envGenerator) {+                    // We only care about python.exe (on windows) and python (on linux/mac)+                    // Other version like python3.exe or python3.8 are often symlinks to+                    // python.exe or python in the same directory in the case od virtual

Spelling ("od") and indentation.

karthiknadig

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as path from 'path';+import { traceError, traceVerbose } from '../../../../common/logger';+import { chain, iterable } from '../../../../common/utils/async';+import {+    Architecture, getEnvironmentVariable, getUserHomeDir,+} from '../../../../common/utils/platform';+import {+    PythonEnvInfo, PythonEnvKind, PythonVersion, UNKNOWN_PYTHON_VERSION,+} from '../../../base/info';+import { parseVersion } from '../../../base/info/pythonVersion';+import { ILocator, IPythonEnvsIterator } from '../../../base/locator';+import { PythonEnvsWatcher } from '../../../base/watcher';+import { findInterpretersInDir } from '../../../common/commonUtils';+import { getFileInfo, pathExists } from '../../../common/externalDependencies';+import { isVenvEnvironment, isVirtualenvEnvironment, isVirtualenvwrapperEnvironment } from './virtualEnvironmentIdentifier';++const DEFAULT_SEARCH_DEPTH = 4;++/**+ * Gets all default virtual environment locations. This uses WORKON_HOME,+ * and user home directory to find some known locations where global virtual+ * environments are often created.+ */+async function getGlobalVirtualEnvDirs(): Promise<string[]> {+    const venvDirs:string[] = [];+    const dirPaths:string[] = [];++    const workOnHome = getEnvironmentVariable('WORKON_HOME');+    if (workOnHome) {+        dirPaths.push(workOnHome);+    }++    const homeDir = getUserHomeDir();+    if (homeDir) {+        dirPaths.push(path.join(homeDir, 'envs'));+        dirPaths.push(path.join(homeDir, '.direnv'));+        dirPaths.push(path.join(homeDir, '.venvs'));+        dirPaths.push(path.join(homeDir, '.virtualenvs'));+        dirPaths.push(path.join(homeDir, '.local', 'share', 'virtualenvs'));+    }++    const exists = await Promise.all(dirPaths.map(pathExists));+    exists.forEach((v, i) => {+        if (v) {+            venvDirs.push(dirPaths[i]);+        }+    });++    return venvDirs;+}++/**+ * Gets the virtual environment kind for a given interpreter path.+ * This only checks for environments created using venv, virtualenv,+ * and virtualenvwrapper based environments.+ * @param interpreterPath: Absolute path to the interpreter paths.+ */+async function getVirtualEnvKind(interpreterPath:string): Promise<PythonEnvKind> {+    if (await isVenvEnvironment(interpreterPath)) {+        return PythonEnvKind.Venv;+    }++    if (await isVirtualenvwrapperEnvironment(interpreterPath)) {+        return PythonEnvKind.VirtualEnvWrapper;+    }++    if (await isVirtualenvEnvironment(interpreterPath)) {+        return PythonEnvKind.VirtualEnv;+    }++    return PythonEnvKind.Unknown;+}++/**+ * Takes absolute path to the interpreter and environment kind to build+ * environment info.+ * @param {string} interpreterPath: Absolute path to interpreter.+ * @param {PythonEnvKind} kind: Kind for the given environment.+ */+async function buildEnvInfo(interpreterPath:string, kind:PythonEnvKind): Promise<PythonEnvInfo> {+    let version:PythonVersion;+    try {+        version = parseVersion(path.basename(interpreterPath));+    } catch (ex) {+        traceError(`Failed to parse version from path: ${interpreterPath}`, ex);+        version = UNKNOWN_PYTHON_VERSION;+    }+    return {+        name: '',+        location: '',+        kind,+        executable: {+            filename: interpreterPath,+            sysPrefix: '',+            ...(await getFileInfo(interpreterPath)),+        },+        version,+        arch: Architecture.Unknown,+        distro: { org: '' },+    };+}++/**+ * Finds and resolves virtual environments created in known global locations.+ */+export class GlobalVirtualEnvironmentLocator extends PythonEnvsWatcher implements ILocator {+    private virtualEnvKinds = [+        PythonEnvKind.Venv,+        PythonEnvKind.VirtualEnv,+        PythonEnvKind.VirtualEnvWrapper,+    ];++    public constructor(private readonly searchDepth?:number) {+        super();+    }++    public iterEnvs(): IPythonEnvsIterator {+        // Number of levels of sub-directories to recurse when looking for+        // interpreters+        const searchDepth = this.searchDepth ?? DEFAULT_SEARCH_DEPTH;++        async function* iterator(virtualEnvKinds:PythonEnvKind[]) {+            const envRootDirs = await getGlobalVirtualEnvDirs();+            const envGenerators = envRootDirs.map((envRootDir) => {+                async function* generator() {+                    traceVerbose(`Searching for global virtual envs in: ${envRootDir}`);++                    const envGenerator = findInterpretersInDir(envRootDir, searchDepth);++                    for await (const env of envGenerator) {+                    // We only care about python.exe (on windows) and python (on linux/mac)+                    // Other version like python3.exe or python3.8 are often symlinks to+                    // python.exe or python in the same directory in the case od virtual+                    // environments.+                        const name = path.basename(env).toLowerCase();+                        if (name === 'python.exe' || name === 'python') {+                        // We should extract the kind here to avoid doing is*Environment()+                        // check multiple times. Those checks are file system heavy and+                        // we can use the kind to determine this anyway.

It seems that this comment describes a recurring pattern. If is...Environment() is so easy to misuse, should we just get rid of those altogether, and fold that logic into getVirtualEnvKind(), forcing the API clients to a better pattern?

karthiknadig

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Implement global virtual environment locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { chain, iterable } from '../../common/utils/async';+import { getOSType, OSType } from '../../common/utils/platform';+import { isPosixPythonBin } from './posixUtils';+import { isWindowsPythonExe } from './windowsUtils';++export async function* findInterpretersInDir(root:string, recurseLevels?:number): AsyncIterableIterator<string> {+    const dirContents = (await fsapi.readdir(root)).map((c) => path.join(root, c));+    const os = getOSType();+    const generators = dirContents.map((item) => {+        async function* generator() {+            const stat = await fsapi.lstat(item);++            if (stat.isDirectory()) {+                if (recurseLevels && recurseLevels > 0) {+                    const subItems = findInterpretersInDir(item, recurseLevels - 1);++                    for await (const subItem of subItems) {+                        yield subItem;+                    }+                }+            } else if (+                (os === OSType.Windows && isWindowsPythonExe(item))+                || (os !== OSType.Windows && isPosixPythonBin(item))

This would probably be a bit more readable with :? - or better yet, lift the check out of the loop, and set a local to point to the right function.

karthiknadig

comment created time in 5 days

PullRequestReviewEvent

push eventmicrosoft/ptvsd

Pavel Minaev

commit sha 533426eb77fe153c6b1171be809218cbe79a44c4

Deprecate ptvsd Update README to direct to debugpy.

view details

Pavel Minaev

commit sha 99c8513921021d2cc7cd82e132b65c644c256768

Add notice for VS

view details

push time in 7 days

PR merged microsoft/ptvsd

Deprecate ptvsd

Update README to direct to debugpy.

+10 -0

3 comments

1 changed file

int19h

pr closed time in 7 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

issue openedmicrosoft/vscode

Extension API for shell quoting & escaping

<!-- ⚠️⚠️ Do Not Delete This! feature_request_template ⚠️⚠️ --> <!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ --> <!-- Please search existing issues to avoid creating duplicates. -->

<!-- Describe the feature you'd like. -->

VSCode has an implementation of shell quoting & escaping for multiple shells in its implementation of the "runInTerminal" DAP request:

https://github.com/microsoft/vscode/blob/bd7746b82be92350c3ca9c91fefaad0252abf760/src/vs/workbench/contrib/debug/node/terminals.ts#L79-L198

https://github.com/microsoft/vscode/blob/bd7746b82be92350c3ca9c91fefaad0252abf760/src/vs/workbench/api/node/extHostDebugService.ts#L113-L114

Python extension for VSCode has to do very similar with terminal.sendText() a lot, in order to activate environments and to run external tools. Consequently, it also needs to quote & escape the commands that it generates. However, there's no way to reuse the VSCode implementation, because it is not exposed directly as a public extension API, but only via "runInTerminal".

The request is to have this functionality exposed as a public API. Given the shell type and separated command arguments, it should produce a command string suitable for submission to terminal.sendText() or child_process.exec(). The most straightforward implementation would be to publish prepareCommand() as is, since it already provides everything that's needed.

created time in 11 days

PullRequestReviewEvent

issue commentmicrosoft/debugpy

Call Stack empty when Debugging

You've mentioned that it behaves somewhat differently when there is a breakpoint that is hit - could you do that repro with logging enabled, as well?

tiagosvf

comment created time in 15 days

issue commentmicrosoft/debugpy

Breakpoints won't break in Pydantic package, specifically

One other thing that would be interesting to check here - what happens if, instead of setting the breakpoint in UI, you use breakpoint() inside that same code? Does it break, and if so, what does the call stack look like?

smcoll

comment created time in 15 days

issue commentmicrosoft/debugpy

Call Stack empty when Debugging

So far as I can see, VSCode never issued a "stackTrace" request at all.

Can you try it with the most recent version of Code (1.50 at the moment)? It might be something that they have already fixed.

tiagosvf

comment created time in 15 days

issue commentmicrosoft/debugpy

Python 3.9 support

If there's anything that's not working, and that isn't captured by the debugpy test suite (last I checked it was passing on 3.9), can you file test issues for that?

fabioz

comment created time in 16 days

PullRequestReviewEvent

issue commentmicrosoft/debugpy

Breakpoints won't break in Pydantic package, specifically

Looking at setup.py for that package, it is set up to cythonize all the .py files by default:

https://github.com/samuelcolvin/pydantic/blob/bf9cc4a5e7903ada2e819a63fe2da011f292a143/setup.py#L86-L92

This would make them native code, and thus not something that debugpy can trace. But, as a workaround, you should be able to build that package with SKIP_CYTHON=1, and then it should work. Can you give that a try?

smcoll

comment created time in 17 days

Pull request review commentmicrosoft/vscode-python

Add API to get data from registry

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import { HKCU, HKLM } from 'winreg';+import { getInterpreterDataFromRegistry, IRegistryInterpreterData } from '../../../common/windowsUtils';++export async function getRegistryInterpreters() : Promise<IRegistryInterpreterData[]> {+    let registryData:IRegistryInterpreterData[] = [];++    // Read from PythonCore locations+    for (const arch of ['x64', 'x86']) {+        for (const hive of [HKLM, HKCU]) {+            registryData = registryData.concat(await getInterpreterDataFromRegistry({ arch, hive, key: '\\SOFTWARE\\Python\\PythonCore' }));+        }+    }++    // Read from PythonCodingPack locations+    return registryData.concat(await getInterpreterDataFromRegistry({ arch: 'x64', hive: HKCU, key: '\\SOFTWARE\\Python\\PythonCodingPack' }));+}

Shouldn't this be scanning all subkeys of SOFTWARE\Python, not just PythonCore and PythonCodingPack? CPython distros packaged by third parties will, presumably, use other subkeys, per PEP 514.

karthiknadig

comment created time in 17 days

PullRequestReviewEvent

issue closedmicrosoft/vscode-python

Wrong binary detection (missing venv $PATH while debugging)

<!-- Please search existing issues to avoid creating duplicates. -->

Environment data

Versión: 1.49.2 Confirmación: e5e9e69aed6e1984f7499b7af85b3d05f9a6883a Fecha: 2020-09-24T16:26:09.944Z Electron: 9.2.1 Chrome: 83.0.4103.122 Node.js: 12.14.1 V8: 8.3.110.13-electron.0 Sistema Operativo: Linux x64 5.8.12-200.fc32.x86_64

  • VS Code version: 1.49.2
  • Extension version (available under the Extensions sidebar): v2020.9.112786
  • OS and version: Fedora Silverblue 32, Linux x64 5.8.12-200.fc32.x86_64
  • Python version (& distribution if applicable, e.g. Anaconda): 3.8.5
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): poetry (uses local .venv dir for compatibility with this extension)
  • Relevant/affected Python packages and their versions: Copier, in development mode
  • Value of the python.languageServer setting: Microsoft

Expected behaviour

This test (pytest) should fail with the exact same message if I run it or if I debug it:

def test_bin_detection():
    from shutil import which
    import os

    print(os.environ["PATH"])
    assert which("copier") == "THIS MUST FAIL"

Actual behaviour

When running the test, it detects the binary copier from the virtualenv (CORRECT):

imagen

When debugging it, it detects the system-wide copier binary (WRONG):

imagen

Steps to reproduce:

  1. Write a pytest that tries to find a binary that exists both inside the selected virtualenv and the global scope.
  2. Run it. Good.
  3. Debug it. Bad.

Logs

<details>

<summary>Output for <code>Python</code> in the <code>Output</code> panel (<code>View</code>→<code>Output</code>, change the drop-down the upper-right of the <code>Output</code> panel to <code>Python</code>) </summary>

<p>

User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'AA_testing - control'
User belongs to experiment group 'LocalZMQKernel - experiment'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
User belongs to experiment group 'EnableIPyWidgets - experiment'
User belongs to experiment group 'RunByLine - control'
User belongs to experiment group 'CustomEditorSupport - control'
User belongs to experiment group 'pythonaa'
> conda --version
> pyenv root
> python3.7 ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3.6 ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3 ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python2 ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> conda info --json
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import jupyter"
Starting Microsoft Python language server.
Python interpreter path: ./.venv/bin/python
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py -c "import notebook"
> conda --version
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/prodevel/copier --junit-xml=/tmp/tmp-122702RQIEICiCM0ln.xml
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
48,80,E,E501:line too long (85 > 79 characters)
53,80,E,E501:line too long (86 > 79 characters)
54,80,E,E501:line too long (87 > 79 characters)
115,80,E,E501:line too long (82 > 79 characters)
179,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
63,80,E,E501:line too long (86 > 79 characters)
64,80,E,E501:line too long (85 > 79 characters)
76,80,E,E501:line too long (85 > 79 characters)
##########Linting Output - flake8##########
50,80,E,E501:line too long (83 > 79 characters)
60,80,E,E501:line too long (83 > 79 characters)
90,80,E,E501:line too long (85 > 79 characters)
100,80,E,E501:line too long (85 > 79 characters)
142,80,E,E501:line too long (85 > 79 characters)
149,80,E,E501:line too long (81 > 79 characters)
166,80,E,E501:line too long (85 > 79 characters)
171,80,E,E501:line too long (82 > 79 characters)
176,80,E,E501:line too long (86 > 79 characters)
181,80,E,E501:line too long (87 > 79 characters)
184,80,E,E501:line too long (84 > 79 characters)
188,80,E,E501:line too long (87 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
54,80,E,E501:line too long (85 > 79 characters)
59,80,E,E501:line too long (86 > 79 characters)
60,80,E,E501:line too long (87 > 79 characters)
121,80,E,E501:line too long (82 > 79 characters)
185,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
54,80,E,E501:line too long (85 > 79 characters)
59,80,E,E501:line too long (86 > 79 characters)
60,80,E,E501:line too long (87 > 79 characters)
121,80,E,E501:line too long (82 > 79 characters)
185,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/prodevel/copier --junit-xml=/tmp/tmp-1227020w0VS6DZZjr1.xml ./tests/test_complex_questions.py::test_bin_detection
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
63,80,E,E501:line too long (86 > 79 characters)
64,80,E,E501:line too long (85 > 79 characters)
76,80,E,E501:line too long (85 > 79 characters)
##########Linting Output - flake8##########
50,80,E,E501:line too long (83 > 79 characters)
60,80,E,E501:line too long (83 > 79 characters)
90,80,E,E501:line too long (85 > 79 characters)
100,80,E,E501:line too long (85 > 79 characters)
142,80,E,E501:line too long (85 > 79 characters)
149,80,E,E501:line too long (81 > 79 characters)
166,80,E,E501:line too long (85 > 79 characters)
171,80,E,E501:line too long (82 > 79 characters)
176,80,E,E501:line too long (86 > 79 characters)
181,80,E,E501:line too long (87 > 79 characters)
184,80,E,E501:line too long (84 > 79 characters)
188,80,E,E501:line too long (87 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
56,80,E,E501:line too long (85 > 79 characters)
61,80,E,E501:line too long (86 > 79 characters)
62,80,E,E501:line too long (87 > 79 characters)
123,80,E,E501:line too long (82 > 79 characters)
187,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
52,28,E,E711:comparison to None should be 'if cond is None:'
56,80,E,E501:line too long (85 > 79 characters)
61,80,E,E501:line too long (86 > 79 characters)
62,80,E,E501:line too long (87 > 79 characters)
123,80,E,E501:line too long (82 > 79 characters)
187,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py black --diff --quiet ~/prodevel/copier/tests/test_complex_questions.py.4c9f0cfe2d24ed2cc31d8fcae33308c3.tmp
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_complex_questions.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
15,80,E,E501:line too long (85 > 79 characters)
56,80,E,E501:line too long (85 > 79 characters)
61,80,E,E501:line too long (86 > 79 characters)
62,80,E,E501:line too long (87 > 79 characters)
123,80,E,E501:line too long (82 > 79 characters)
187,80,E,E501:line too long (83 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/prodevel/copier -s --cache-clear
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/prodevel/copier --junit-xml=/tmp/tmp-122702Qcs6Vh7OdXIo.xml ./tests/test_complex_questions.py::test_bin_detection
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py flake8 --format=%(row)d,%(col)d,%(code).1s,%(code)s:%(text)s ~/prodevel/copier/tests/test_templated_prompt.py
cwd: ~/prodevel/copier
##########Linting Output - flake8##########
63,80,E,E501:line too long (86 > 79 characters)
64,80,E,E501:line too long (85 > 79 characters)
76,80,E,E501:line too long (85 > 79 characters)
##########Linting Output - flake8##########
50,80,E,E501:line too long (83 > 79 characters)
60,80,E,E501:line too long (83 > 79 characters)
90,80,E,E501:line too long (85 > 79 characters)
100,80,E,E501:line too long (85 > 79 characters)
142,80,E,E501:line too long (85 > 79 characters)
149,80,E,E501:line too long (81 > 79 characters)
166,80,E,E501:line too long (85 > 79 characters)
171,80,E,E501:line too long (82 > 79 characters)
176,80,E,E501:line too long (86 > 79 characters)
181,80,E,E501:line too long (87 > 79 characters)
184,80,E,E501:line too long (84 > 79 characters)
188,80,E,E501:line too long (87 > 79 characters)
> ~/prodevel/copier/.venv/bin/python ~/.vscode/extensions/ms-python.python-2020.9.112786/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/prodevel/copier --junit-xml=/tmp/tmp-12270255zvEOUPlXdC.xml ./tests/test_complex_questions.py::test_bin_detection
cwd: ~/prodevel/copier

</p> </details>

closed time in 17 days

Yajo

issue commentmicrosoft/vscode-python

Wrong binary detection (missing venv $PATH while debugging)

The root cause here is #4300 - i.e. the environment doesn't get activated properly.

The problem is that "Debug Tests", by default, uses "console": "internalConsole" - i.e. it doesn't run in the terminal. The extension does auto-activate terminals (with default settings), but in the absence of one, that logic doesn't enter the picture. The debugger, meanwhile, just runs the test using the specified path to the Python binary. This is usually enough (e.g. it'll use libraries from the environment etc), but it doesn't do anything about PATH; so if you rely on the latter, that's where it becomes observable.

You can force the tests to run in the terminal by adding a test config to your launch.json:

        {
            "name": "Python: Test",
            "type": "python",
            "request": "test",
            "console": "integratedTerminal",
        },

However, there is a separate bug with terminal activation, where there's a race condition between the debugger launch and the activation command, so there's still a chance of it getting things wrong on the first run. Subsequent runs should work fine, since it'll reuse the activated terminal.

Until we fix #4300, the most reliable workaround is to activate the environment manually in a terminal outside of VSCode (e.g. via poetry shell), and then launch Code from that activated terminal, so that it inherits the environment.

Yajo

comment created time in 17 days

issue closedmicrosoft/vscode-python

Debugger should allow for a timeout when debugging inside container

<!-- Please search existing issues to avoid creating duplicates. -->

Environment data

  • VS Code version: 1.49.2
  • Extension version: v2020.9.111407
  • OS and version: Linux x64 5.4.0-7642-generic
  • Python version: Python 3.8.2
  • Type of virtual environment used: Docker
  • Value of the python.languageServer setting: Jedi

Expected behaviour

The debugging session should start normally and correctly map breakpoints.

Actual behaviour

The debugging session should fails at start and cannot map some breakpoints until reloaded.

Steps to reproduce:

1.Setup a launch debugging configuration or compound where the preLaunchTask involves starting a Docker container in detached mode (like docker compose up -d) 2. Launch the debugging session 3. Observe the debugging session fail, not be launched or produce errors (like breakpoints not being mapped correctly) 4. Launch the session again and works normally.

closed time in 18 days

joao-p-marques

issue commentmicrosoft/vscode-python

Debugger should allow for a timeout when debugging inside container

Tasks in general are handled by VSCode proper, so adding some form of delay to them would have to be done in https://github.com/microsoft/vscode. But I think their intent is that for anything that requires this degree of complexity, a task spawning a script (that can then use the standard scripting facilities to wait on timer, or on some external condition) is the way to go.

joao-p-marques

comment created time in 18 days

Pull request review commentmicrosoft/vscode-python

Show 'View Tensor in Data Viewer' debug menu option for tensors

 export class DebuggerVariables implements IConditionalJupyterVariables, DebugAda             ? this.configService.getSettings().datascience.variableExplorerExclude?.split(';')             : []; -        const allowedVariables = variablesResponse.body.variables.filter((v) => {-            if (!v.name || !v.type || !v.value) {-                return false;-            }-            if (exclusionList && exclusionList.includes(v.type)) {-                return false;-            }-            if (v.name.startsWith('_')) {-                return false;-            }-            if (KnownExcludedVariables.has(v.name)) {-                return false;-            }-            if (v.type === 'NoneType') {-                return false;-            }-            return true;-        });+        const allowedVariables = variablesResponse.body.variables

It would be hard to make it quite as flexible as client-side filtering. If we hardcode the filters, debugpy would have to be touched every time this logic needs revising. And if we don't hardcode them, then the protocol would need to be expressive enough to express all those conditions, which would also make the implementation expensive.

I think client-side filtering is the better option here, unless there are important perf reasons to exclude some variables.

joyceerhl

comment created time in 18 days

PullRequestReviewEvent

push eventint19h/debugpy

Pavel Minaev

commit sha 49919cd40e6b99e100c08771f54df36f9338617c

Add a feature flag for userUnhandled exception filter.

view details

push time in 18 days

PullRequestReviewEvent

issue closedmicrosoft/debugpy

GitHub Release, but isn't a release?

Follow up of issue #427

Environment data

  • debugpy version: 1.0.0
  • OS and version:
  • Python version (& distribution if applicable, e.g. Anaconda):
  • Using VS Code or Visual Studio:

Actual behavior

In issue #427 I raised the release was missing on PyPI, which was closed and I was told it was invalid, as there is not release yet.

Hence, I open this issue. According to GitHub there is a stable release, which isn't true according to this response in #427: https://github.com/microsoft/debugpy/issues/427#issuecomment-702358979

@int19h: That's because it's not actually released yet - the commit that's tagged as v1.0.0 is the one that will become the release, but it's still in process (we have to tag it before we build it, because the version number in the build comes from the tag). It's expected to be published by the end of this week.

Expected behavior

I expect a release on GitHub to be a release, especially when it is marked a non-prerelease as well:

image

The previous issue was closed, but it talks about a tag, not release. Those are 2 different things in the GitHub world.

Steps to reproduce:

Browse to: https://github.com/microsoft/debugpy/releases/tag/v1.0.0

closed time in 21 days

frenck

issue commentmicrosoft/debugpy

GitHub Release, but isn't a release?

We are not going to change the release process in the middle of an ongoing release. It should be considered released when it shows up on PyPI. Normally, the gap between the two is less than 24 hours, but not this time, due to some technical issues. We'll revisit this for future releases to avoid such disparities..

frenck

comment created time in 21 days

issue commentmicrosoft/PTVS

Python 3.8 and Python/C++ mixed-mode debugging

If this is debugging unit tests, wouldn't it be using the regular debugger, not mixed-mode?

naughtong

comment created time in 22 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Know posix path locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { getPathEnvironmentVariable } from '../../common/utils/platform';++/**+ * Checks if a given path ends with python*.exe+ * @param {string} interpreterPath : Path to python interpreter.+ * @returns {boolean} : Returns true if the path matches pattern for windows python executable.+ */+export function isPosixPythonBin(interpreterPath:string): boolean {+    /**+     * This Reg-ex matches following file names:+     * python+     * python3+     * python38+     * python3.8+     */+    const posixPythonBinPattern = /^python(\d+(.\d+)?)?$/;++    return posixPythonBinPattern.test(path.basename(interpreterPath));+}++export async function commonPosixBinPaths(): Promise<string[]> {+    const searchPaths = (getPathEnvironmentVariable() || '')+        .split(path.delimiter)+        .filter((p) => p.length > 0);++    const paths: Set<string> = new Set(+        [+            '/bin',+            '/etc',+            '/lib',+            '/lib/x86_64-linux-gnu',+            '/lib64',+            '/sbin',+            '/snap/bin',+            '/usr/bin',+            '/usr/games',+            '/usr/include',+            '/usr/lib',+            '/usr/lib/x86_64-linux-gnu',+            '/usr/lib64',+            '/usr/libexec',+            '/usr/local',+            '/usr/local/bin',+            '/usr/local/etc',+            '/usr/local/games',+            '/usr/local/lib',+            '/usr/local/sbin',+            '/usr/sbin',+            '/usr/share',+            '~/.local/bin',+        ].concat(searchPaths),

That list comes partly from PATH, and partly from some hardcoded stuff in whereis, but I suspect that the latter is really old and outdated. That said, this is at most a minor perf issue, so we can revisit if and when it actually becomes important.

karthiknadig

comment created time in 22 days

PullRequestReviewEvent

Pull request review commentmicrosoft/vscode-python

Know posix path locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { getPathEnvironmentVariable } from '../../common/utils/platform';++/**+ * Checks if a given path ends with python*.exe+ * @param {string} interpreterPath : Path to python interpreter.+ * @returns {boolean} : Returns true if the path matches pattern for windows python executable.+ */+export function isPosixPythonBin(interpreterPath:string): boolean {+    /**+     * This Reg-ex matches following file names:+     * python+     * python3+     * python38+     * python3.8+     */+    const posixPythonBinPattern = /^python(\d+(.\d+)?)?$/;++    return posixPythonBinPattern.test(path.basename(interpreterPath));+}++export async function commonPosixBinPaths(): Promise<string[]> {+    const searchPaths = (getPathEnvironmentVariable() || '')+        .split(path.delimiter)+        .filter((p) => p.length > 0);++    const paths: Set<string> = new Set(+        [+            '/bin',+            '/etc',+            '/lib',+            '/lib/x86_64-linux-gnu',+            '/lib64',+            '/sbin',+            '/snap/bin',+            '/usr/bin',+            '/usr/games',+            '/usr/include',+            '/usr/lib',+            '/usr/lib/x86_64-linux-gnu',+            '/usr/lib64',+            '/usr/libexec',+            '/usr/local',+            '/usr/local/bin',+            '/usr/local/etc',+            '/usr/local/games',+            '/usr/local/lib',+            '/usr/local/sbin',+            '/usr/sbin',+            '/usr/share',+            '~/.local/bin',+        ].concat(searchPaths),

Some of these look out of place here - /etc and **/include shouldn't contain binaries, neither should top-level /usr/share or /usr/local (the latter one is especially weird, since there's no entry for /usr, which serves the same function).

karthiknadig

comment created time in 22 days

Pull request review commentmicrosoft/vscode-python

Know posix path locator

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import * as fsapi from 'fs-extra';+import * as path from 'path';+import { traceVerbose } from '../../../../common/logger';++import { Architecture } from '../../../../common/utils/platform';+import {+    PythonEnvInfo, PythonEnvKind, PythonReleaseLevel, PythonVersion,+} from '../../../base/info';+import { parseVersion } from '../../../base/info/pythonVersion';+import { ILocator, IPythonEnvsIterator } from '../../../base/locator';+import { PythonEnvsWatcher } from '../../../base/watcher';+import { getFileInfo, resolveSymbolicLink } from '../../../common/commonUtils';+import { commonPosixBinPaths, isPosixPythonBin } from '../../../common/posixUtils';++async function getPythonBinFromKnownPaths(): Promise<string[]> {+    const knownPaths = await commonPosixBinPaths();+    const pythonBins:Set<string> = new Set();+    // eslint-disable-next-line no-restricted-syntax+    for (const knownPath of knownPaths) {+        // eslint-disable-next-line no-await-in-loop+        const files = (await fsapi.readdir(knownPath))+            .map((filename:string) => path.join(knownPath, filename))+            .filter(isPosixPythonBin);++        // eslint-disable-next-line no-restricted-syntax+        for (const file of files) {+            // Ensure that we have a collection of unique global binaries by+            // resolving all symlinks to the target binaries.+            // eslint-disable-next-line no-await-in-loop+            const resolvedBin = await resolveSymbolicLink(file);+            pythonBins.add(resolvedBin);+            traceVerbose(`Found: ${file} --> ${resolvedBin}`);+        }+    }++    return Array.from(pythonBins);+}++export class PosixKnownPathsLocator extends PythonEnvsWatcher implements ILocator {+    private kind: PythonEnvKind = PythonEnvKind.OtherGlobal;++    public iterEnvs(): IPythonEnvsIterator {+        const buildEnvInfo = (bin:string) => this.buildEnvInfo(bin);+        const iterator = async function* () {+            const exes = await getPythonBinFromKnownPaths();+            yield* exes.map(buildEnvInfo);+        };+        return iterator();+    }++    public resolveEnv(env: string | PythonEnvInfo): Promise<PythonEnvInfo | undefined> {+        const executablePath = typeof env === 'string' ? env : env.executable.filename;+        return this.buildEnvInfo(executablePath);+    }++    private async buildEnvInfo(bin:string): Promise<PythonEnvInfo> {+        let version:PythonVersion;+        try {+            version = parseVersion(path.basename(bin));+        } catch (e) {

If it can't parse, it should probably log that somewhere, so that we can diagnose it if something goes wrong.

karthiknadig

comment created time in 22 days

more