profile
viewpoint
Don Jayamanne DonJayamanne Software engineer with a soft spot for .NET and Node.js. Loves tinkering with Arduino and Raspberry Pi. Lives in a cave.

beeware/vscode-beeware 31

A Visual Studio Code extension with support for BeeWare development

DonJayamanne/awesome-python 1

A curated list of awesome Python frameworks, libraries, software and resources

DonJayamanne/azure-pipelines-template 1

template for your azure pipelines

DonJayamanne/conda 1

OS-agnostic, system-level binary package manager and ecosystem

DonJayamanne/apollo-tooling 0

:pencil2: Tooling for development and production Apollo workflows

DonJayamanne/autoDocstring 0

vscode extension that generates docstrings for python files

DonJayamanne/awesome-jupyter 0

A curated list of awesome Jupyter projects, libraries and resources

DonJayamanne/azure-powershell 0

Microsoft Azure PowerShell

DonJayamanne/BlazorTest 0

A place to test Blazor features

PR closed microsoft/vscode-python

Reviewers
Refactor validation of kernels for VSC Notebooks skip news

For #12189 Validation moved to when we run a cell.

  • The kernel information returned to VSC from kernel provider now returns a stateless class
  • No more validation is performed when a kernel is changed (this happens when executing code)
  • Cleaned up IKernel interface
  • Move all execution of code & managing state of cells and notebook into IKernel class
    • Hence deleted the old ExzecutionService

@rchiodo @IanMatthewHuff @joyceerhl @DavidKutu We can discuss this PR tomorrow, I'd like to ensure we get these interfaces & code right.

TODO (will probably file issues for these)

  • [ ] As discussed, create an interface with an event for when a kernel has been selected
  • [ ] Update trust code to create new interface/function to check if a VSC Notebook is trusted, instead of relying on INotebookModel, else this old interface will never go away even though we want to stop using this.
+616 -566

2 comments

13 changed files

DonJayamanne

pr closed time in 7 hours

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha 885ca52e18dbf1588777a33b8118288fddd3fa2f

Fixes

view details

push time in 21 hours

Pull request review commentmicrosoft/vscode-python

Refactor validation of kernels for VSC Notebooks

  import { nbformat } from '@jupyterlab/coreutils'; import type { KernelMessage } from '@jupyterlab/services';+import { Subscription } from 'rxjs'; import { Observable } from 'rxjs/Observable'; import { Subject } from 'rxjs/Subject';-import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import {+    CancellationToken,+    CancellationTokenSource,+    Event,+    EventEmitter,+    NotebookCell,+    NotebookCellRunState,+    NotebookDocument,+    Uri+} from 'vscode'; import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { ICommandManager } from '../../../common/application/types';+import { wrapCancellationTokens } from '../../../common/cancellation'; import { traceError } from '../../../common/logger'; import { IDisposableRegistry } from '../../../common/types'; import { createDeferred, Deferred } from '../../../common/utils/async';+import { noop } from '../../../common/utils/misc';+import { StopWatch } from '../../../common/utils/stopWatch';+import { IInterpreterService } from '../../../interpreter/contracts';+import { captureTelemetry, sendTelemetryEvent } from '../../../telemetry';+import { Commands, Telemetry, VSCodeNativeTelemetry } from '../../constants';+import {+    handleUpdateDisplayDataMessage,+    hasTransientOutputForAnotherCell,+    updateCellExecutionCount,+    updateCellOutput,+    updateCellWithErrorStatus+} from '../../notebook/helpers/executionHelpers';+import {+    clearCellForExecution,+    getCellStatusMessageBasedOnFirstCellErrorOutput,+    updateCellExecutionTimes+} from '../../notebook/helpers/helpers';+import { MultiCancellationTokenSource } from '../../notebook/helpers/multiCancellationToken';+import { NotebookEditor } from '../../notebook/notebookEditor';+import { INotebookContentProvider } from '../../notebook/types'; import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';-import type {+import {     ICell,+    IDataScienceErrorHandler,     IJupyterKernelSpec,     INotebook,+    INotebookEditorProvider,     INotebookProvider,     INotebookProviderConnection,     InterruptResult,     KernelSocketInformation } from '../../types';-import type { IKernel, KernelSelection, LiveKernelModel } from './types';+import { KernelProvider } from './kernelProvider';+import type { IKernel, IKernelSelectionUsage, KernelSelection, LiveKernelModel } from './types';+// tslint:disable-next-line: no-var-requires no-require-imports+const vscodeNotebookEnums = require('vscode') as typeof import('vscode-proposed');++/**+ * Separate class that deals just with kernel execution.+ * Else the `Kernel` class gets very big.+ */+class KernelExecution {+    public notebook?: INotebook;+    private readonly registeredIOPubListeners = new WeakSet<IKernel>();+    private readonly cellExecutions = new WeakMap<NotebookCell, MultiCancellationTokenSource>();+    private readonly documentExecutions = new WeakMap<NotebookDocument, MultiCancellationTokenSource>();+    private readonly pendingExecutionCancellations = new Map<string, CancellationTokenSource[]>();+    private readonly documentsWithPendingCellExecutions = new WeakMap<NotebookDocument, NotebookCell | undefined>();+    private readonly tokensInterrupted = new WeakSet<CancellationToken>();+    private sentExecuteCellTelemetry: boolean = false;+    private readonly cellsQueueForExecutionButNotYetExecuting = new WeakSet<NotebookCell>();+    private readonly kernelValidated = new WeakMap<NotebookDocument, { kernel: IKernel; promise: Promise<void> }>();+    constructor(+        private readonly kernelProvider: KernelProvider,+        private readonly commandManager: ICommandManager,+        private readonly interpreterService: IInterpreterService,+        private readonly errorHandler: IDataScienceErrorHandler,+        private readonly contentProvider: INotebookContentProvider,+        private readonly editorProvider: INotebookEditorProvider,+        readonly kernelSelectionUsage: IKernelSelectionUsage+    ) {}+    @captureTelemetry(Telemetry.ExecuteNativeCell, undefined, true)+    public async executeCell(cell: NotebookCell): Promise<void> {+        if (!this.notebook) {+            throw new Error('executeObservable cannot be called if kernel has not been started!');+        }+        if (this.cellExecutions.has(cell)) {+            return;+        }+        const source = new MultiCancellationTokenSource();+        const token = source.token;+        this.cellExecutions.set(cell, source);++        // Cannot execute empty cells.+        if (cell.document.getText().trim().length === 0) {+            return;+        }+        const stopWatch = new StopWatch();+        const kernel = this.getKernel(cell.notebook);+        this.cellsQueueForExecutionButNotYetExecuting.add(cell);+        // Mark cells as busy (this way there's immediate feedback to users).+        // If it does not complete, then restore old state.+        const oldCellState = cell.metadata.runState;+        cell.metadata.runState = vscodeNotebookEnums.NotebookCellRunState.Running;++        // If we cancel running cells, then restore the state to previous values unless cell has completed.+        token.onCancellationRequested(() => {+            if (this.cellsQueueForExecutionButNotYetExecuting.has(cell)) {+                cell.metadata.runState = oldCellState;+            }+        });++        await this.executeIndividualCell(kernel, cell, token, stopWatch);+    }+    @captureTelemetry(Telemetry.ExecuteNativeCell, undefined, true)+    @captureTelemetry(VSCodeNativeTelemetry.RunAllCells, undefined, true)+    public async executeAllCells(document: NotebookDocument): Promise<void> {+        if (!this.notebook) {+            throw new Error('executeObservable cannot be called if kernel has not been started!');+        }+        if (this.documentExecutions.has(document)) {+            return;+        }+        const source = new MultiCancellationTokenSource();+        const token = source.token;+        this.documentExecutions.set(document, source);+        const stopWatch = new StopWatch();+        const kernel = this.getKernel(document);+        document.metadata.runState = vscodeNotebookEnums.NotebookRunState.Running;+        // Mark all cells as busy (this way there's immediate feedback to users).+        // If it does not complete, then restore old state.+        const oldCellStates = new WeakMap<NotebookCell, NotebookCellRunState | undefined>();+        document.cells.forEach((cell) => {+            if (+                cell.document.getText().trim().length === 0 ||+                cell.cellKind === vscodeNotebookEnums.CellKind.Markdown+            ) {+                return;+            }+            this.cellsQueueForExecutionButNotYetExecuting.add(cell);+            oldCellStates.set(cell, cell.metadata.runState);+            cell.metadata.runState = vscodeNotebookEnums.NotebookCellRunState.Running;+        });++        const restoreOldCellState = (cell: NotebookCell) => {+            if (oldCellStates.has(cell) && this.cellsQueueForExecutionButNotYetExecuting.has(cell)) {+                cell.metadata.runState = oldCellStates.get(cell);+            }+        };+        // If we cancel running cells, then restore the state to previous values unless cell has completed.+        token.onCancellationRequested(() => {+            if (!this.documentsWithPendingCellExecutions.has(document)) {+                document.metadata.runState = vscodeNotebookEnums.NotebookRunState.Idle;+            }+            document.cells.forEach(restoreOldCellState);+        });++        let executingAPreviousCellHasFailed = false;+        await document.cells.reduce((previousPromise, cellToExecute) => {+            return previousPromise.then((previousCellState) => {+                // If a previous cell has failed or execution cancelled, the get out.+                if (+                    executingAPreviousCellHasFailed ||+                    token.isCancellationRequested ||+                    previousCellState === vscodeNotebookEnums.NotebookCellRunState.Error+                ) {+                    executingAPreviousCellHasFailed = true;+                    restoreOldCellState(cellToExecute);+                    return;+                }+                if (+                    cellToExecute.document.getText().trim().length === 0 ||+                    cellToExecute.cellKind === vscodeNotebookEnums.CellKind.Markdown+                ) {+                    return;+                }+                return this.executeIndividualCell(kernel, cellToExecute, token, stopWatch);+            });+        }, Promise.resolve<NotebookCellRunState | undefined>(undefined));++        document.metadata.runState = vscodeNotebookEnums.NotebookRunState.Idle;+    }+    public cancelCell(cell: NotebookCell) {+        this.cellExecutions.get(cell)?.cancel(); // NOSONAR+    }+    public cancelAllCells(document: NotebookDocument) {+        this.documentExecutions.get(document)?.cancel(); // NOSONAR+        document.cells.forEach((cell) => this.cancelCell(cell));+    }+    public cancelPendingExecutions(document: NotebookDocument): void {+        this.pendingExecutionCancellations.get(document.uri.fsPath)?.forEach((cancellation) => cancellation.cancel()); // NOSONAR+    }+    private async getKernel(document: NotebookDocument): Promise<IKernel> {+        await this.validate(document);+        let kernel = this.kernelProvider.get(document.uri);+        if (!kernel) {+            const activeInterpreter = await this.interpreterService.getActiveInterpreter(document.uri);+            kernel = this.kernelProvider.getOrCreate(document.uri, {+                metadata: { interpreter: activeInterpreter!, kernelModel: undefined, kernelSpec: undefined },+                launchingFile: document.uri.fsPath+            });+        }+        if (!kernel) {+            throw new Error('Unable to create a Kernel to run cell');+        }+        await kernel.start();+        return kernel;+    }+    private sendPerceivedCellExecute(runningStopWatch: StopWatch) {+        const props = { notebook: true };+        if (!this.sentExecuteCellTelemetry) {+            this.sentExecuteCellTelemetry = true;+            sendTelemetryEvent(Telemetry.ExecuteCellPerceivedCold, runningStopWatch.elapsedTime, props);+        } else {+            sendTelemetryEvent(Telemetry.ExecuteCellPerceivedWarm, runningStopWatch.elapsedTime, props);+        }+    }++    private async executeIndividualCell(+        kernelPromise: Promise<IKernel>,+        cell: NotebookCell,+        token: CancellationToken,+        stopWatch: StopWatch+    ): Promise<NotebookCellRunState | undefined> {+        if (!this.notebook) {+            throw new Error('No notebook object');+        }+        if (token.isCancellationRequested) {+            return;+        }+        const kernel = await kernelPromise;+        if (token.isCancellationRequested) {+            return;+        }+        const document = cell.notebook;+        const editor = this.editorProvider.editors.find((e) => e.file.toString() === document.uri.toString());+        if (!editor) {+            throw new Error('No editor for Model');+        }+        if (editor && !(editor instanceof NotebookEditor)) {+            throw new Error('Executing Notebook with another Editor');+        }+        // If we need to cancel this execution (from our code, due to kernel restarts or similar, then cancel).+        const cancelExecution = new CancellationTokenSource();+        if (!this.pendingExecutionCancellations.has(document.uri.fsPath)) {+            this.pendingExecutionCancellations.set(document.uri.fsPath, []);+        }+        // If kernel is restarted while executing, then abort execution.+        const cancelExecutionCancellation = new CancellationTokenSource();+        this.pendingExecutionCancellations.get(document.uri.fsPath)?.push(cancelExecutionCancellation); // NOSONAR++        // Replace token with a wrapped cancellation, which will wrap cancellation due to restarts.+        const wrappedToken = wrapCancellationTokens(token, cancelExecutionCancellation.token, cancelExecution.token);+        const kernelDisposedDisposable = kernel.onDisposed(() => {+            cancelExecutionCancellation.cancel();+        });++        // tslint:disable-next-line: no-suspicious-comment+        // TODO: How can nb be null?+        // We should throw an exception or change return type to be non-nullable.+        // Else in places where it shouldn't be null we'd end up treating it as null (i.e. ignoring error conditions, like this).++        this.handleDisplayDataMessages(document, kernel);++        const deferred = createDeferred<NotebookCellRunState>();+        wrappedToken.onCancellationRequested(() => {+            if (deferred.completed) {+                return;+            }++            // Interrupt kernel only if original cancellation was cancelled.+            // I.e. interrupt kernel only if user attempts to stop the execution by clicking stop button.+            if (token.isCancellationRequested && !this.tokensInterrupted.has(token)) {+                this.tokensInterrupted.add(token);+                this.commandManager.executeCommand(Commands.NotebookEditorInterruptKernel).then(noop, noop);+            }+        });++        // Ensure we clear the cell state and trigger a change.+        clearCellForExecution(cell);+        const executionStopWatch = new StopWatch();+        cell.metadata.runStartTime = new Date().getTime();+        this.contentProvider.notifyChangesToDocument(document);+        this.cellsQueueForExecutionButNotYetExecuting.delete(cell);+        this.documentsWithPendingCellExecutions.set(document, cell);+        let subscription: Subscription | undefined;+        try {+            editor.notifyExecution(cell.document.getText());+            this.notebook.clear(cell.uri.toString());+            const observable = this.notebook.executeObservable(+                cell.document.getText(),+                document.fileName,+                0,+                cell.uri.toString(),+                false+            );+            subscription = observable?.subscribe(+                (cells) => {+                    const rawCellOutput = cells+                        .filter((item) => item.id === cell.uri.toString())+                        .flatMap((item) => (item.data.outputs as unknown) as nbformat.IOutput[])+                        .filter((output) => !hasTransientOutputForAnotherCell(output));++                    // Set execution count, all messages should have it+                    if (+                        cells.length &&+                        'execution_count' in cells[0].data &&+                        typeof cells[0].data.execution_count === 'number'+                    ) {+                        const executionCount = cells[0].data.execution_count as number;+                        if (updateCellExecutionCount(cell, executionCount)) {+                            this.contentProvider.notifyChangesToDocument(document);+                        }+                    }++                    if (updateCellOutput(cell, rawCellOutput)) {+                        this.contentProvider.notifyChangesToDocument(document);+                    }+                },+                (error: Partial<Error>) => {+                    updateCellWithErrorStatus(cell, error);+                    this.contentProvider.notifyChangesToDocument(document);+                    this.errorHandler.handleError((error as unknown) as Error).ignoreErrors();+                    deferred.resolve(cell.metadata.runState);+                },+                () => {+                    cell.metadata.lastRunDuration = executionStopWatch.elapsedTime;+                    cell.metadata.runState = wrappedToken.isCancellationRequested+                        ? vscodeNotebookEnums.NotebookCellRunState.Idle+                        : vscodeNotebookEnums.NotebookCellRunState.Success;+                    cell.metadata.statusMessage = '';+                    updateCellExecutionTimes(cell, {+                        startTime: cell.metadata.runStartTime,+                        duration: cell.metadata.lastRunDuration+                    });++                    // If there are any errors in the cell, then change status to error.+                    if (cell.outputs.some((output) => output.outputKind === vscodeNotebookEnums.CellOutputKind.Error)) {+                        cell.metadata.runState = vscodeNotebookEnums.NotebookCellRunState.Error;+                        cell.metadata.statusMessage = getCellStatusMessageBasedOnFirstCellErrorOutput(cell.outputs);+                    }++                    this.contentProvider.notifyChangesToDocument(document);+                    deferred.resolve(cell.metadata.runState);+                }+            );

I'm planning on moving all of this into a separate function or the like to improve readability and the future plan is to not use ICell instead give INotebook the NotebookCell and let is update the output directly in that object.

DonJayamanne

comment created time in 21 hours

Pull request review commentmicrosoft/vscode-python

Refactor validation of kernels for VSC Notebooks

  import { nbformat } from '@jupyterlab/coreutils'; import type { KernelMessage } from '@jupyterlab/services';+import { Subscription } from 'rxjs'; import { Observable } from 'rxjs/Observable'; import { Subject } from 'rxjs/Subject';-import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import {+    CancellationToken,+    CancellationTokenSource,+    Event,+    EventEmitter,+    NotebookCell,+    NotebookCellRunState,+    NotebookDocument,+    Uri+} from 'vscode'; import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { ICommandManager } from '../../../common/application/types';+import { wrapCancellationTokens } from '../../../common/cancellation'; import { traceError } from '../../../common/logger'; import { IDisposableRegistry } from '../../../common/types'; import { createDeferred, Deferred } from '../../../common/utils/async';+import { noop } from '../../../common/utils/misc';+import { StopWatch } from '../../../common/utils/stopWatch';+import { IInterpreterService } from '../../../interpreter/contracts';+import { captureTelemetry, sendTelemetryEvent } from '../../../telemetry';+import { Commands, Telemetry, VSCodeNativeTelemetry } from '../../constants';+import {+    handleUpdateDisplayDataMessage,+    hasTransientOutputForAnotherCell,+    updateCellExecutionCount,+    updateCellOutput,+    updateCellWithErrorStatus+} from '../../notebook/helpers/executionHelpers';+import {+    clearCellForExecution,+    getCellStatusMessageBasedOnFirstCellErrorOutput,+    updateCellExecutionTimes+} from '../../notebook/helpers/helpers';+import { MultiCancellationTokenSource } from '../../notebook/helpers/multiCancellationToken';+import { NotebookEditor } from '../../notebook/notebookEditor';+import { INotebookContentProvider } from '../../notebook/types'; import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';-import type {+import {     ICell,+    IDataScienceErrorHandler,     IJupyterKernelSpec,     INotebook,+    INotebookEditorProvider,     INotebookProvider,     INotebookProviderConnection,     InterruptResult,     KernelSocketInformation } from '../../types';-import type { IKernel, KernelSelection, LiveKernelModel } from './types';+import { KernelProvider } from './kernelProvider';+import type { IKernel, IKernelSelectionUsage, KernelSelection, LiveKernelModel } from './types';+// tslint:disable-next-line: no-var-requires no-require-imports+const vscodeNotebookEnums = require('vscode') as typeof import('vscode-proposed');++/**+ * Separate class that deals just with kernel execution.+ * Else the `Kernel` class gets very big.+ */+class KernelExecution {+    public notebook?: INotebook;+    private readonly registeredIOPubListeners = new WeakSet<IKernel>();+    private readonly cellExecutions = new WeakMap<NotebookCell, MultiCancellationTokenSource>();

I feel this class is super busy

  • Managing state of cells
  • Managing state of document
  • Interrupting cells/documents
  • Running multiple cells (chaining them & executing in order)
  • Supporting cancellations (from other events, etc)

Will try to clean this up:

  • I believe the code that does the actual execution and translation of output and updating cell outputs can be moved into separate function or the like (as we don't really update any state in the class)

I welcome other feedback to clean this busy class/file & improve readability.

DonJayamanne

comment created time in 21 hours

Pull request review commentmicrosoft/vscode-python

Refactor validation of kernels for VSC Notebooks

 export interface IKernel extends IAsyncDisposable {     readonly disposed: boolean;     readonly kernelSocket: Observable<KernelSocketInformation | undefined>;     start(): Promise<void>;-    interrupt(timeoutInMs: number): Promise<InterruptResult>;-    restart(timeoutInMs: number): Promise<void>;-    executeObservable(code: string, file: string, line: number, id: string, silent: boolean): Observable<ICell[]>;+    interrupt(): Promise<InterruptResult>;+    restart(): Promise<void>;+    executeCell(cell: NotebookCell): Promise<void>;+    executeAllCells(document: NotebookDocument): Promise<void>;

Please review this new API. I think we will need the ability to execute arbitrary code. I guess we can add executeCode(code: string) : Promise<nbformat.IOutput[]> This would always execute stuff silently, else one would use executeCell.

Not sure we need executeAllCells in a kernel. Coz kernels just execute code, and now we're introducing notion of execution an entier notebook. However this is required, coz this class manages state related to execution of a cell or an entire notebook.

Feels like we need a higher order class, but then we end up with INotebook Or we could move that code into just the IExecutionService simiarl to what we had, and its only responsible for executions and not used elsewhere.. I.e. if we want to execute code silently we use IKernel, but when user runs cells or a notebook thats the only time the class IExecutionService gets used.

Probably worth discussing (tomorrow) as I do not like having executeAllCells in here. If others feel the same, then great. Else lets leave this.

DonJayamanne

comment created time in a day

Pull request review commentmicrosoft/vscode-python

Refactor validation of kernels for VSC Notebooks

  import { nbformat } from '@jupyterlab/coreutils'; import type { KernelMessage } from '@jupyterlab/services';+import { Subscription } from 'rxjs'; import { Observable } from 'rxjs/Observable'; import { Subject } from 'rxjs/Subject';-import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import {+    CancellationToken,+    CancellationTokenSource,+    Event,+    EventEmitter,+    NotebookCell,+    NotebookCellRunState,+    NotebookDocument,+    Uri+} from 'vscode'; import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { ICommandManager } from '../../../common/application/types';+import { wrapCancellationTokens } from '../../../common/cancellation'; import { traceError } from '../../../common/logger'; import { IDisposableRegistry } from '../../../common/types'; import { createDeferred, Deferred } from '../../../common/utils/async';+import { noop } from '../../../common/utils/misc';+import { StopWatch } from '../../../common/utils/stopWatch';+import { IInterpreterService } from '../../../interpreter/contracts';+import { captureTelemetry, sendTelemetryEvent } from '../../../telemetry';+import { Commands, Telemetry, VSCodeNativeTelemetry } from '../../constants';+import {+    handleUpdateDisplayDataMessage,+    hasTransientOutputForAnotherCell,+    updateCellExecutionCount,+    updateCellOutput,+    updateCellWithErrorStatus+} from '../../notebook/helpers/executionHelpers';+import {+    clearCellForExecution,+    getCellStatusMessageBasedOnFirstCellErrorOutput,+    updateCellExecutionTimes+} from '../../notebook/helpers/helpers';+import { MultiCancellationTokenSource } from '../../notebook/helpers/multiCancellationToken';+import { NotebookEditor } from '../../notebook/notebookEditor';+import { INotebookContentProvider } from '../../notebook/types'; import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';-import type {+import {     ICell,+    IDataScienceErrorHandler,     IJupyterKernelSpec,     INotebook,+    INotebookEditorProvider,     INotebookProvider,     INotebookProviderConnection,     InterruptResult,     KernelSocketInformation } from '../../types';-import type { IKernel, KernelSelection, LiveKernelModel } from './types';+import { KernelProvider } from './kernelProvider';+import type { IKernel, IKernelSelectionUsage, KernelSelection, LiveKernelModel } from './types';+// tslint:disable-next-line: no-var-requires no-require-imports+const vscodeNotebookEnums = require('vscode') as typeof import('vscode-proposed');++/**+ * Separate class that deals just with kernel execution.+ * Else the `Kernel` class gets very big.+ */+class KernelExecution {

Created a separate class that deals with just execution of cells (starting, interrupting, etc is all handled in the other class). Didn't want to move this out, else we need to export this and its assumed to be re-usable. Or i could expose this, and assume that no one will attempt to use this (I guess that's safe too).

Open to feedback - I prefer moving it (but left in here, as the IKernel should be responsible for all execution & I agree. However the class gets big.

DonJayamanne

comment created time in a day

Pull request review commentmicrosoft/vscode-python

Refactor validation of kernels for VSC Notebooks

 import { KernelSelection } from '../jupyter/kernels/types'; import { INotebookStorageProvider } from '../notebookStorage/notebookStorageProvider'; import { INotebook, INotebookProvider } from '../types'; import { getNotebookMetadata, isJupyterNotebook, updateKernelInNotebookMetadata } from './helpers/helpers';-import { NotebookKernel } from './notebookKernel';-import { INotebookContentProvider, INotebookExecutionService } from './types';+import { INotebookContentProvider } from './types';++class VSCodeNotebookKernelMetadata implements VSCNotebookKernel {+    get preloads(): Uri[] {

As discussed, this class is used purely to populate dropdown list (without any state)

DonJayamanne

comment created time in a day

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha 0b46922883ba1aa227a9a0c493947b5be16443dd

Refactor code

view details

push time in a day

issue commentmicrosoft/vscode-python

Flaky Python Daemon Pool tests

The tests haven't failed since then. Can we close for now.

karrtikr

comment created time in a day

PR opened microsoft/vscode-python

Refactor validation of kernels for VSC Notebooks

For #12189 Validation moved to when we run a cell.

+32 -55

0 comment

2 changed files

pr created time in a day

pull request commentmicrosoft/vscode-python

Support launching with the same directory as a notebook

Hope this doesn't result in creation of some temp files or the like.

rchiodo

comment created time in a day

create barnchDonJayamanne/pythonVSCode

branch : noValidateKernels

created branch time in a day

push eventmicrosoft/vscode-python

Don Jayamanne

commit sha 48797c5ec3b665eb41e1de970b83f1bb68950f04

Ensure we do not switch kernel if already the same (#13297)

view details

push time in a day

delete branch DonJayamanne/pythonVSCode

delete branch : reselectKernel

delete time in a day

PR merged microsoft/vscode-python

Reviewers
Ensure we do not switch kernel if already the same skip news

For #13266 For #12189

+27 -12

2 comments

6 changed files

DonJayamanne

pr closed time in a day

push eventmicrosoft/vscode-python

Don Jayamanne

commit sha ddbea64d2d94bbf410291e16980918ebb8c24659

Revert "Notebook undo command is same as standard undo cmd (#13293)" (#13318) This reverts commit d7c934804d70e14604304c60e4d2ab25232b7724.

view details

push time in a day

delete branch DonJayamanne/pythonVSCode

delete branch : revertUndoNb

delete time in a day

PR merged microsoft/vscode-python

Reviewers
Revert "Notebook undo command is same as standard undo cmd (#13293)" skip news

This reverts commit d7c934804d70e14604304c60e4d2ab25232b7724.

For #13316

+1 -1

2 comments

1 changed file

DonJayamanne

pr closed time in a day

issue commentmicrosoft/vscode-python

Working directory for non-python kernels in a temp directory

I mentioned above. In this scenario the working directory of jupyter isn't necessarily the root folder of where your notebook i

Can we not use a CLI to tell jupyter to set the CWD. I tnink the notebook_dir can work, however this would mean we'd need to always start a new jupyter server for all languages and all notebooks. Or we have pre-execute scripts where we can do this either for users or users can do this via the pre-execute scripts.

jlperla

comment created time in a day

PR opened microsoft/vscode-python

Reviewers
Add missing service Injection skip news

For #13319

+7 -3

0 comment

2 changed files

pr created time in a day

create barnchDonJayamanne/pythonVSCode

branch : fixBrokenCI

created branch time in a day

issue openedmicrosoft/vscode-python

Nightly tests are broken due to missing injection

Error: No matching bindings found for serviceIdentifier: Symbol(IInterpreterHashProviderFactory)

 22) DataScience Native Editor
       With Custom Editor API
         Editor tests
           Selection/Focus
             "before each" hook for "None of the cells are selected by default":
     Error: No matching bindings found for serviceIdentifier: Symbol(IInterpreterHashProviderFactory)
      at _validateActiveBindingCount (node_modules/inversify/lib/planning/planner.js:62:23)
      at _getActiveBindings (node_modules/inversify/lib/planning/planner.js:48:5)
      at _createSubRequests (node_modules/inversify/lib/planning/planner.js:91:26)
      at /home/vsts/work/1/s/node_modules/inversify/lib/planning/planner.js:115:17
      at Array.forEach (<anonymous>)
      at /home/vsts/work/1/s/node_modules/inversify/lib/planning/planner.js:114:26
      at Array.forEach (<anonymous>)
      at _createSubRequests (node_modules/inversify/lib/planning/planner.js:94:20)
      at Object.plan (node_modules/inversify/lib/planning/planner.js:136:9)
      at /home/vsts/work/1/s/node_modules/inversify/lib/container/container.js:318:37
      at Container._get (node_modules/inversify/lib/container/container.js:311:44)
      at Container.get (node_modules/inversify/lib/container/container.js:230:21)
      at ServiceContainer.get (out/client/ioc/container.js:29:89)
      at DataScienceIocContainer.registerDataScienceTypes (out/test/datascience/dataScienceIocContainer.js:664:58)
      at initIoc (out/test/datascience/nativeEditor.functional.test.js:759:25)
      at Context.<anonymous> (out/test/datascience/nativeEditor.functional.test.js:875:31)
      at processImmediate (internal/timers.js:439:21)

Should be an easy enough fix.

created time in a day

PR opened microsoft/vscode-python

Revert "Notebook undo command is same as standard undo cmd (#13293)"

This reverts commit d7c934804d70e14604304c60e4d2ab25232b7724.

For #13316

+1 -1

0 comment

1 changed file

pr created time in a day

PR closed microsoft/vscode-python

Reviewers
Remove undo keybinding for notebook cells skip news

For #13316

+0 -8

2 comments

1 changed file

DonJayamanne

pr closed time in a day

create barnchDonJayamanne/pythonVSCode

branch : revertUndoNb

created branch time in a day

issue openedmicrosoft/vscode

Unable to map letter `z` to `undo` command with a when clause

<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ --> <!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ --> <!-- Please search existing issues to avoid creating duplicates. --> <!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

<!-- Use Help > Report Issue to prefill these. -->

  • VSCode Version: 1.48.0-insider
  • OS Version: Darwin x64 19.6.0 Commit: cfbd1999769f4f08dce29629fb92fdc0fac53829 Date: 2020-08-06T05:40:40.871Z Electron: 7.3.2 Chrome: 78.0.3904.130 Node.js: 12.8.1 V8: 7.8.279.23-electron.0

Steps to Reproduce:

  1. Add the following into package.json
            {
                "mac": "Z",
                "win": "Z",
                "linux": "Z",
                "key": "Z",
                "when": "editorLangId == abc",
                "command": "undo"
            },
  1. The expectation is this will never work unless a text editor is opened with the language abc
  2. However, when ever you type z it performs an undo

Observation Seems to be special for undo. I.e. if i replace undo with a custom command, then nothing happens & the letter z gets written into the text editor.

<!-- Launch with code --disable-extensions to check. --> Does this issue occur when all extensions are disabled?: Yes/No

created time in a day

PR opened microsoft/vscode-python

Remove undo keybinding for notebook cells

For #13316

+0 -8

0 comment

1 changed file

pr created time in a day

create barnchDonJayamanne/pythonVSCode

branch : removeUndo

created branch time in a day

issue openedmicrosoft/vscode-python

Undo keybinding not working for notebook cells

Add the following keybinding into package.json

            {
                "mac": "Z",
                "win": "Z",
                "linux": "Z",
                "key": "Z",
                "when": "notebookEditorFocused && !inputFocus && notebookViewType == jupyter-notebook",
                "command": "undo"
            },

The expectation is this key binding will kick in, only when the when clause is satisfied. However, when ever you type z in any text editor in VSCode, it performs an undo operation.

created time in a day

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha dc36fd265f5343a3230b4c43825ffb43b867543b

Fixes to trusting native notebooks (#13292)

view details

Don Jayamanne

commit sha d7c934804d70e14604304c60e4d2ab25232b7724

Notebook undo command is same as standard undo cmd (#13293)

view details

Don Jayamanne

commit sha ac1eb0ef111713c606fa84930920aeb855cd3e28

Merge branch 'master' into reselectKernel

view details

push time in a day

issue openedmicrosoft/vscode-python

Jupyter Activity bar

  • Variables
  • Display notebook metadata? with ability to edit this.
  • Display cell metadata? (tags, attachments, etc) with ability to edit this
  • Display remote Jupyter files (users have requested this)?
  • Should we display the current kernel yet again here?

(Note: This is a place holder for the list of requirements and spec).

created time in a day

issue openedmicrosoft/vscode-python

Formatting of notebooks on save

If formatting on save is configured, then we need to see if notebooks also get formatted and accordingly update our API

created time in a day

issue openedmicrosoft/vscode-python

Run by line for native notebooks

created time in a day

issue openedmicrosoft/vscode-python

Gather for native notebooks

created time in a day

issue openedmicrosoft/vscode-python

SVG plot is displayed in favor of png plot

Based on jupyter the SVG plots should have a higher priority over png and others when rendering output. However as a side effect we are no longer able to open the plot viewer as a result of this: Solutions:

  • Change priority (works for our case with matplot lib, but not in other cases where user has SVG + PNG)
  • Create a custom renderer for SVG and add the icon to this (if this is an SVG that was generated by our code)
  • Detect when SVGs are generated as a result of our changes to matplot lib rendering and only add PNG into output
  • etc...
  • NOte, we have also looked at removing this approach and using the ipywidget for matplot lib as that results in a smaller ipynb file as there will be no SVG generated by us.

created time in a day

issue openedmicrosoft/vscode-python

Support formatting of all cells in notebook

Currently we have support for formatting individual cells However formatting entire document does not seem to work (VSC Has a specific command for this)

created time in a day

issue openedmicrosoft/vscode-python

Support non Python kernels

created time in a day

issue closedmicrosoft/vscode-python

Bug

Bug: Notebook Editor, Interactive Window, Editor cells

when I use remote development and open a jupyter file(.ipynb), once I paste a piece of code to jupyter, the terminal(bash) will paste the same copy, too. <!----------------------------------------------------------------------------------------------- PLEASE READ If this issue doesn't relate to Jupyter Notebooks, Python Interactive Window features or other notebook or "cell"-based features of the Python extension, please use the main Python bug template instead of this one. Thank you! ------------------------------------------------------------------------------------------------->

Steps to cause the bug to occur

4TBBF6CJ1NML7FARV40NIR8 paste a piece of code to jupyter, e.g. print('hello world‘)

Actual behavior

jupyter will paste this code certainly. But the bash command line appears the same copy, too

Expected behavior

only jupyter will appear the pasted code <!----------------------------------------------------------------------------------------------- Animated GIFs can be effective means to describe a bug. Consider using a tool like https://github.com/phw/peek or https://www.screentogif.com/ to create one. ------------------------------------------------------------------------------------------------->

Your Jupyter and/or Python environment

Please provide as much info as you readily know

  • Jupyter server running: Local | Remote | N/A the client and the server are in the same LAN, using local connection
  • Extension version: 20YY.MM.#####-xxx
  • VS Code version: #.## 1.47.3
  • Setting python.jediEnabled: true | false
  • Setting python.languageServer: Jedi | Microsoft | None
  • Python and/or Anaconda version: #.#.# 3.7.4; 4.8.3
  • OS: Windows | Mac | Linux (distro): client Windows 10, server Ubuntu 16.04
  • Virtual environment: conda | venv | virtualenv | N/A | ... conda

Developer Tools Console Output

<!----------------------------------------------------------------------------------------------- Copy/paste the output in the "Console" tab in the "Developer Tools" panel (Help > Toggle Developer Tools). For better details, run the "Enable source map support for extension debugging" command in VS Code at least once beforehand. ------------------------------------------------------------------------------------------------->

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

closed time in a day

luomou97

issue commentmicrosoft/vscode-python

Add the cell toolbar to Jupyter notebooks

Moving back to triage, as I don't think this is GA (i.e. GA is parity with existing nb, but this is a new feature and we wouldn't want to be working on this before reaching parity).

epetrovski

comment created time in a day

issue closedmicrosoft/vscode-python

Kernel selection UI cannot be accessed through keyboard

Only accessible through mouse clicks

Good time to add tests for tab order to ensure we tab through the items in the UI in the right order.

closed time in a day

DonJayamanne

issue commentmicrosoft/vscode-python

Kernel selection UI cannot be accessed through keyboard

Closing this as there's a command to select a kernel in vs code.

DonJayamanne

comment created time in a day

issue commentmicrosoft/vscode-python

Try out the new VS code custom notebook provider API with ipywidgets

Confirmed this is working with the new VSC Notebook API (renderer). Closing this issue as it work, even though there are other issues (confirmed the issues are not related to renderer API or VS COde API, its something to do with bundling, I have a branch where I have been working in ipywidgets for native notebooks. Currently put on hold to work on kernels).

rchiodo

comment created time in a day

issue closedmicrosoft/vscode-python

Try out the new VS code custom notebook provider API with ipywidgets

We want to make sure ipywidgets is still capable with the current api. Mostly we need cross extension and UI comms.

closed time in a day

rchiodo

issue commentmicrosoft/vscode-python

Reopening: Python snippets not working with Jupyter Notebook (.ipynb) files

Closing this issue, as native notebook cells are now first class VS Code text editors supporting extnsions, snippets and the like. For VSCode native notebooks (see issue #12189), all python snippets show up in the native notebook cells.

rickhg12hs

comment created time in a day

issue closedmicrosoft/vscode-python

Reopening: Python snippets not working with Jupyter Notebook (.ipynb) files

As the previous issue #8978 has been closed and comments are restricted, reopening this issue here.

closed time in a day

rickhg12hs

issue commentmicrosoft/vscode-python

Latex not rendered correctly

Update nteract and related libraries to latest version in renderer extension.

DonJayamanne

comment created time in a day

issue commentmicrosoft/vscode-python

Raw cells are not preserved in VSC Native Notebooks

After discussing with VSCode, the solution is to set the language of raw cells to plaintext. I.e. any cell with a language of plaintext is considererd a raw cell.

DonJayamanne

comment created time in a day

issue commentmicrosoft/vscode-python

Native notebook becomes untrusted after rename

Should hook into VSC FS FileRenameEvent (workspace.onDidRenameFiles)

joyceerhl

comment created time in a day

issue commentmicrosoft/vscode-python

Some icons don't appear in the Insiders Notebook Editor until an untrusted notebook is trusted and then closed

Today we are hiding some of these commands, guess we need to show them and disable the commands To be decided, hence moving back to triage. FYI - VSC does not show the run cells icon (out of our control). Makes sense, cuz if a notebook is not runnable, then it is not displayed.

kimadeline

comment created time in a day

issue closedmicrosoft/vscode-python

Display editor icons when notebook is active

Currently we display icons when a cell is active not when a notebook is active.

closed time in a day

DonJayamanne

issue openedmicrosoft/vscode

Readonly notebooks allow copying and pasting of cells

<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ --> <!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ --> <!-- Please search existing issues to avoid creating duplicates. --> <!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

<!-- Use Help > Report Issue to prefill these. -->

  • VSCode Version: 1.48.0-insider
  • OS Version: Darwin x64 19.6.0 Commit: cfbd1999769f4f08dce29629fb92fdc0fac53829 Date: 2020-08-06T05:40:40.871Z Electron: 7.3.2 Chrome: 78.0.3904.130 Node.js: 12.8.1 V8: 7.8.279.23-electron.0

Steps to Reproduce:

1.Open a Notebook 2.Ensure the notebook is not editable (notebook and cells all have editable metadata set to false) 3. Cells cannot be added, deleted, edited 4. Copy and cell and paste (it will now insert a new cell below the copied cell)

Found that following three keyboard shortcuts result in notebook being edited:

  • Cut
  • Copy + Paste
  • Undo

@rebornix /cc

<!-- Launch with code --disable-extensions to check. --> Does this issue occur when all extensions are disabled?: Yes/No

created time in a day

issue closedmicrosoft/vscode-python

Verify mime types and order of rendering for Notebooks

  • What else apart from following list
  • Whats the priority order (check jupyter)
                'application/vnd.*',
                'application/vdom.*',
                'application/geo+json',
                'application/x-nteract-model-debug+json',
                'text/html',
                'application/javascript',
                'text/latex',
                'text/markdown',
                'application/json',
                'image/svg+xml',
                'image/png',
                'image/jpeg',
                'text/plain'

closed time in a day

DonJayamanne

issue commentmicrosoft/vscode-python

Verify mime types and order of rendering for Notebooks

Moved to renderers

DonJayamanne

comment created time in a day

push eventmicrosoft/vscode-python

Don Jayamanne

commit sha d7c934804d70e14604304c60e4d2ab25232b7724

Notebook undo command is same as standard undo cmd (#13293)

view details

push time in 2 days

delete branch DonJayamanne/pythonVSCode

delete branch : fixNotebookUndo

delete time in 2 days

PR merged microsoft/vscode-python

Reviewers
Notebook undo command is same as standard undo cmd skip news

For #12189 Account for changes in upstream VSCode.

+2 -2

2 comments

1 changed file

DonJayamanne

pr closed time in 2 days

push eventmicrosoft/vscode-python

Don Jayamanne

commit sha dc36fd265f5343a3230b4c43825ffb43b867543b

Fixes to trusting native notebooks (#13292)

view details

push time in 2 days

delete branch DonJayamanne/pythonVSCode

delete branch : fixTrustingNativeNotebooks

delete time in 2 days

PR merged microsoft/vscode-python

Reviewers
Fixes to trusting native notebooks skip news

For #12189

+445 -4

2 comments

4 changed files

DonJayamanne

pr closed time in 2 days

Pull request review commentmicrosoft/vscode-python

change language server from Jedi when creating native notebooks

 export class NotebookEditorProvider implements INotebookEditorProvider {     }     @captureTelemetry(Telemetry.CreateNewNotebook, undefined, false)     public async createNew(contents?: string): Promise<INotebookEditor> {+        const settings = this.configurationService.getSettings();+        if (settings.languageServer === LanguageServerType.Jedi) {

We identified that jedi is auto enabled in Jupyter.

Here's I'm referring to jedi running inside Jupyter (the kernel).

DavidKutu

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

change language server from Jedi when creating native notebooks

 export class NotebookEditorProvider implements INotebookEditorProvider {     }     @captureTelemetry(Telemetry.CreateNewNotebook, undefined, false)     public async createNew(contents?: string): Promise<INotebookEditor> {+        const settings = this.configurationService.getSettings();+        if (settings.languageServer === LanguageServerType.Jedi) {

Not sure this is the right thing. The original issue is the use of Jedi in the kernel. This is changing the lange server for entire python extension, i don't think this is a good idea.

DavidKutu

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Ensure we do not switch kernel if already the same

  'use strict'; +import * as fastDeepEqual from 'fast-deep-equal';

Using this instead of using JSON.stringify as all i'm after is equality of data.

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Ensure we do not switch kernel if already the same

 export interface IKernelSelectionUsage { export interface IKernel extends IAsyncDisposable {     readonly uri: Uri;     readonly kernelSpec?: IJupyterKernelSpec | LiveKernelModel;+    readonly metadata: Readonly<KernelSelection>;

This is what i'd like to solve with the meeting I have booked. I need the metadata property as well as the kernelSpec property. Ideally we should have just one property that describes all of this and use that everywhere. One type to rule them all 😄

Leaving for now, as i need to solve this issue, but will address this once we have a type or approach to generalize this into a single entity.

DonJayamanne

comment created time in 2 days

PR opened microsoft/vscode-python

Ensure we do not switch kernel if already the same

For #13266 For #12189

+25 -12

0 comment

6 changed files

pr created time in 2 days

create barnchDonJayamanne/pythonVSCode

branch : reselectKernel

created branch time in 2 days

issue closedmicrosoft/vscode-python

Review and add notebook toolbar

Some of the icons exist (provided by VSC), & some don't. These will need to be hooked to a command.

closed time in 2 days

DonJayamanne

issue commentmicrosoft/vscode-python

Review and add notebook toolbar

We have these now.

DonJayamanne

comment created time in 2 days

issue closedmicrosoft/vscode-python

Interrupting kernel in notebook doesn't work as expected

Upstream https://github.com/microsoft/vscode/issues/99203

Create a nb with 10 cells

  • First cell print statement
  • Second cell, time.sleep(10)
  • Rest of the cells, print statements
  • Run all cells
  • Interrupt when second cell is busy

Actual behavior

  • Second cell stops running as it was interrupted.
  • Rest of the cells have a status or running

Adding to VSC Notebook API, as this will need to be fixed for VSC Notebook API implementation.

closed time in 2 days

DonJayamanne

issue commentmicrosoft/vscode-python

Interrupting kernel in notebook doesn't work as expected

Yup, broken in master branch, hence closing this.

DonJayamanne

comment created time in 2 days

issue commentmicrosoft/vscode

Add `product` and `runCommand` on notebook renderer API

Why not allow extensions to contribure an outputId. If this is provided by extensions, then VS Code could use this, else VS Code generates an output Id (as is the case today). Yes, extensions use 1 for all output ids, but at this point, its garbage in garbage out

connor4312

comment created time in 2 days

issue commentmicrosoft/vscode

Execution Count Not Always Updated - Native Notebook Editor

can you confirm if the execution is set always from the Python extension?

I can confirm we set the execution count. @claudiaregio do we have a repro or the video for this.

claudiaregio

comment created time in 2 days

issue closedmicrosoft/vscode-python

Add restart icon to new VSC Notebook toolbar

@greazer is this MVP to GA?

closed time in 2 days

DonJayamanne

issue commentmicrosoft/vscode-python

Saving notebooks results in unnecessary changes to ipynb

Closig for now, as we are sorting the keys in json.

DonJayamanne

comment created time in 2 days

issue closedmicrosoft/vscode-python

Saving notebooks results in unnecessary changes to ipynb

When using VSC Native Notebooks, if we open an ipynb file and save it, it could end up with a lot of unnecessary changes to the ipynb file (json content on disc).

  • We don't have this problem today as we load json and treat this json object as the notebook object model. Any changes made are made directly to this object. Hence, when serializing this to string, most of the structure (order of properies) is maintained.
  • However with the new VSC approach, we load the JSON, pass this info to VSC, then when serializing to string, we construct a json object on the fly. The order of properties will never match the original JSON object.
Solution Pros Cons
Always sort JSON properties (Jupyter seems to do this explicitly) Works with Jupyter Won't work with old Notebooks or other editors creating ipynb
Keep original JSON object model in memory. When we generate the new json object based on VS Code noteook, we can update this original JSON object, then dump that original JSON into string Works best (old approach) We end up trying to sync notebooks again (probably not too bad, as we only need to sync the cells and find out which one is which). However copying/moving cells complicates this solution
Ignore changes & leave VSC Notebook diff to handle this. We don't do anything Not sure this will solve changes to order of properties

closed time in 2 days

DonJayamanne

PR opened microsoft/vscode-python

Notebook undo command is same as standard undo cmd

For #12189 Account for changes in upstream VSCode.

+2 -2

0 comment

1 changed file

pr created time in 2 days

create barnchDonJayamanne/pythonVSCode

branch : fixNotebookUndo

created branch time in 2 days

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha 0495250a72e7f53c4fde75e2aa5bbdf7b943ad22

Code review comments

view details

push time in 2 days

create barnchDonJayamanne/pythonVSCode

branch : fixTrustingNativeNotebooks

created branch time in 2 days

push eventmicrosoft/vscode-python

Don Jayamanne

commit sha 0a0493a29f94390200b4f454746636f36c79f664

Support kernel selection in VSC Native Notebooks (#13245)

view details

push time in 2 days

delete branch DonJayamanne/pythonVSCode

delete branch : nbKernels

delete time in 2 days

PR merged microsoft/vscode-python

Reviewers
Support kernel selection in VSC Native Notebooks skip news

For #12189

  • Supporting non-raw & remote kernels will be done separately
  • Supporting remembering of remote kernels when opening a notebook to be done seprately
+1629 -1075

2 comments

35 changed files

DonJayamanne

pr closed time in 2 days

Pull request review commentmicrosoft/vscode-python

Pass cell metadata and cell id as metadata for execute_request

 export class JupyterNotebookBase implements INotebook {      private generateRequest = (         code: string,-        silent?: boolean+        silent?: boolean,+        metadata?: JSONObject

Please use Record<string, any> or any. We don't need to use JSONObject just to represent arbitrary json objects (as this is an external type).

stisa

comment created time in 2 days

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha a113509b83ca7860a2b10d37d2a28bc1fb437750

Fixes

view details

push time in 2 days

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha fe539aedcab527671c18b4dd5a2ddce115f7f6ff

Code review comments

view details

push time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++'use strict';++import { inject, injectable } from 'inversify';+import { Uri } from 'vscode';+import { traceWarning } from '../../../common/logger';+import { IAsyncDisposableRegistry, IConfigurationService, IDisposableRegistry } from '../../../common/types';+import { INotebookProvider } from '../../types';+import { Kernel } from './kernel';+import { IKernel, KernelOptions } from './types';++@injectable()+export class KernelProvider {+    private readonly kernelsByUri = new Map<string, { options: KernelOptions; kernel: IKernel }>();+    constructor(+        @inject(IAsyncDisposableRegistry) private asyncDisposables: IAsyncDisposableRegistry,+        @inject(IDisposableRegistry) private disposables: IDisposableRegistry,+        @inject(INotebookProvider) private notebookProvider: INotebookProvider,+        @inject(IConfigurationService) private configService: IConfigurationService+    ) {}+    public get(uri: Uri): IKernel | undefined {+        return this.kernelsByUri.get(uri.toString())?.kernel;+    }+    public getOrCreate(uri: Uri, options: KernelOptions): IKernel | undefined {+        const existingKernelInfo = this.kernelsByUri.get(uri.toString());+        if (+            existingKernelInfo &&+            JSON.stringify(existingKernelInfo.options.metadata) === JSON.stringify(options.metadata)+        ) {+            return existingKernelInfo.kernel;+        }++        this.disposeOldKernel(uri);++        const waitForIdleTimeout =+            options?.waitForIdleTimeout ?? this.configService.getSettings(uri).datascience.jupyterLaunchTimeout;+        const kernel = new Kernel(+            uri,+            options.metadata,+            this.notebookProvider,+            this.disposables,+            waitForIdleTimeout,+            options.launchingFile+        );+        this.asyncDisposables.push(kernel);+        this.kernelsByUri.set(uri.toString(), { options, kernel });+        this.deleteMappingIfKernelIdDisposed(uri, kernel);

Ok, u found the easter egg I left for @IanMatthewHuff 😄

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

 suite('DataScience - VSCode Notebook - (Saving)', function () {             cell4 = vscodeNotebook.activeNotebookEditor?.document.cells![3]!;         }         initializeCells();-        await commands.executeCommand('notebook.execute');

This doesn't seem to work anymore, filed an issue. Probably blocked by VSC to prevent rogue extensions running code against kernel.

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++'use strict';++import { nbformat } from '@jupyterlab/coreutils';+import type { KernelMessage } from '@jupyterlab/services';+import { Observable } from 'rxjs/Observable';+import { Subject } from 'rxjs/Subject';+import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { traceError } from '../../../common/logger';+import { IDisposableRegistry } from '../../../common/types';+import { createDeferred, Deferred } from '../../../common/utils/async';+import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';+import type {+    ICell,+    IJupyterKernelSpec,+    INotebook,+    INotebookProvider,+    INotebookProviderConnection,+    KernelSocketInformation+} from '../../types';+import type { IKernel, KernelSelection, LiveKernelModel } from './types';++export class Kernel implements IKernel {

@rchiodo @IanMatthewHuff @joyceerhl @DavidKutu /cc

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++'use strict';++import { nbformat } from '@jupyterlab/coreutils';+import type { KernelMessage } from '@jupyterlab/services';+import { Observable } from 'rxjs/Observable';+import { Subject } from 'rxjs/Subject';+import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { traceError } from '../../../common/logger';+import { IDisposableRegistry } from '../../../common/types';+import { createDeferred, Deferred } from '../../../common/utils/async';+import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';+import type {+    ICell,+    IJupyterKernelSpec,+    INotebook,+    INotebookProvider,+    INotebookProviderConnection,+    KernelSocketInformation+} from '../../types';+import type { IKernel, KernelSelection, LiveKernelModel } from './types';++export class Kernel implements IKernel {+    get connection(): INotebookProviderConnection | undefined {+        return this._notebook?.connection;+    }+    get kernelSpec(): IJupyterKernelSpec | LiveKernelModel | undefined {+        if (this._notebook) {+            return this._notebook.getKernelSpec();+        }+        return this._metadata.kernelSpec || this._metadata.kernelModel;+    }+    get onStatusChanged(): Event<ServerStatus> {+        return this._onStatusChanged.event;+    }+    get onRestarted(): Event<void> {+        return this._onRestarted.event;+    }+    get onDisposed(): Event<void> {+        return this._onDisposed.event;+    }+    get status(): ServerStatus {+        return this._notebook?.status ?? ServerStatus.NotStarted;+    }+    get disposed(): boolean {+        return this._disposed === true || this._notebook?.disposed === true;+    }+    get kernelSocket(): Observable<KernelSocketInformation | undefined> {+        return this._kernelSocket.asObservable();+    }+    private _disposed?: boolean;+    private readonly _kernelSocket = new Subject<KernelSocketInformation | undefined>();+    private readonly _onStatusChanged = new EventEmitter<ServerStatus>();+    private readonly _onRestarted = new EventEmitter<void>();+    private readonly _onDisposed = new EventEmitter<void>();+    private _notebook?: INotebook;+    private _notebookPromise?: Promise<INotebook | undefined>;+    private readonly hookedNotebookForEvents = new WeakSet<INotebook>();+    private restarting?: Deferred<void>;+    // executeObservable(code: string, file: string, line: number, id: string, silent: boolean): Observable<ICell[]>;+    constructor(+        public readonly uri: Uri,+        private readonly _metadata: KernelSelection,

Will create a separate class in separaate PR to represent infomration required to start a kernel:

  • Local connection using kernel spec
  • Local connection using interrpeter
  • Remote connection using Jupyter connection info

Today we have some types that represent the above, however we don't have a single one. E.g. we have IKernelSelection (this alone is not sufficient to start a remote kernel). When we have a live kernel, and we want to get the kernel metadata, it doesn't return the above information back. When we want to start a new kernel (INotebook) it does not accept the above information Basically we have various shapes of data that represent similar bits of informaiton.

Ideally we need one object that can be treated like a IConnection object or the like. And we can have multiple classes that implement this interface one for each type of kernel connection (local raw, local interpreter, remote jupyter, etc...)

That's will be done separately, in another PR.

@rchiodo @IanMatthewHuff @joyceerhl @DavidKutu /cc

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++'use strict';++import { nbformat } from '@jupyterlab/coreutils';+import type { KernelMessage } from '@jupyterlab/services';+import { Observable } from 'rxjs/Observable';+import { Subject } from 'rxjs/Subject';+import { CancellationToken, Event, EventEmitter, Uri } from 'vscode';+import { ServerStatus } from '../../../../datascience-ui/interactive-common/mainState';+import { traceError } from '../../../common/logger';+import { IDisposableRegistry } from '../../../common/types';+import { createDeferred, Deferred } from '../../../common/utils/async';+import { getDefaultNotebookContent, updateNotebookMetadata } from '../../notebookStorage/baseModel';+import type {+    ICell,+    IJupyterKernelSpec,+    INotebook,+    INotebookProvider,+    INotebookProviderConnection,+    KernelSocketInformation+} from '../../types';+import type { IKernel, KernelSelection, LiveKernelModel } from './types';++export class Kernel implements IKernel {

As discussed I have added a kernel class

  • Pending (auto starting kernel) - to use this instead of INotebook
  • Restarting and interrupting should use this instead of INotebook
  • Only exposed required PEMs

Later will change executeObservable to get rid of ICell.

Above changes will be done separately. Current approach still works as calling restart or interrupt on the new class eventually goes to the INotebook object.

DonJayamanne

comment created time in 2 days

Pull request review commentmicrosoft/vscode-python

Support kernel selection in VSC Native Notebooks

+// Copyright (c) Microsoft Corporation. All rights reserved.+// Licensed under the MIT License.++import { inject, injectable } from 'inversify';+import { CancellationToken, Event, EventEmitter } from 'vscode';+import {+    NotebookCommunication,+    NotebookDocument,+    NotebookKernel as VSCNotebookKernel,+    NotebookKernelProvider+} from '../../../../types/vscode-proposed';+import { IVSCodeNotebook } from '../../common/application/types';+import { createPromiseFromCancellation } from '../../common/cancellation';+import { IDisposableRegistry } from '../../common/types';+import { noop } from '../../common/utils/misc';+import { KernelSelectionProvider } from '../jupyter/kernels/kernelSelections';+import { KernelSelector } from '../jupyter/kernels/kernelSelector';+import { KernelSwitcher } from '../jupyter/kernels/kernelSwitcher';+import { INotebook, INotebookProvider } from '../types';+import { updateKernelInNotebookMetadata } from './helpers/helpers';+import { NotebookKernel } from './notebookKernel';+import { INotebookContentProvider, INotebookExecutionService } from './types';+@injectable()+export class KernelProvider implements NotebookKernelProvider {+    public get onDidChangeKernels(): Event<void> {+        return this._onDidChangeKernels.event;+    }+    private readonly _onDidChangeKernels = new EventEmitter<void>();+    private notebookKernelChangeHandled = new WeakSet<INotebook>();+    constructor(+        @inject(INotebookExecutionService) private readonly execution: INotebookExecutionService,+        @inject(KernelSelectionProvider) private readonly kernelSelectionProvider: KernelSelectionProvider,+        @inject(KernelSelector) private readonly kernelSelector: KernelSelector,+        @inject(IVSCodeNotebook) private readonly notebook: IVSCodeNotebook,+        @inject(INotebookProvider) private readonly notebookProvider: INotebookProvider,+        @inject(KernelSwitcher) private readonly kernelSwitcher: KernelSwitcher,+        @inject(INotebookContentProvider) private readonly notebookContentProvider: INotebookContentProvider,+        @inject(IDisposableRegistry) private readonly disposables: IDisposableRegistry+    ) {+        this.kernelSelectionProvider.SelectionsChanged(() => this._onDidChangeKernels.fire(), this, disposables);+        this.notebook.onDidChangeActiveNotebookKernel(this.onDidChangeActiveNotebookKernel, this, disposables);+    }+    /**+     * Called before running code against a kernel. An initialization phase.+     * If the selected kernel is being validated, we can block here.+     */+    public async resolveKernel(+        kernel: NotebookKernel,+        document: NotebookDocument,+        _webview: NotebookCommunication,+        token: CancellationToken+    ): Promise<void> {+        return Promise.race([+            kernel.validate(document.uri).then(() => void 0),

Removed, to use top level await.

DonJayamanne

comment created time in 2 days

push eventDonJayamanne/pythonVSCode

Don Jayamanne

commit sha 5853245055841e599b5e6ed71f3afeb2383dc071

Fix tests and add a kernel class

view details

push time in 2 days

issue openedmicrosoft/vscode-python

Executing command notebook.execute no longer works

This is used to map old commands to the new VSC Command. Alternative:

  • Find active editor, & then the corresponding active kernel and execute executeAllCells on that. The problem is, activeKernel might not be available immediately.

created time in 2 days

Pull request review commentmicrosoft/vscode-python

Fix when clauses for new interactive cell shortcuts

     "downloading.file.progress": "{0}{1} of {2} KB ({3}%)",     "DataScience.jupyterDataRateExceeded": "Cannot view variable because data rate exceeded. Please restart your server with a higher data rate limit. For example, --NotebookApp.iopub_data_rate_limit=10000000000.0",     "DataScience.addCellBelowCommandTitle": "Add cell",-    "DataScience.debugCellCommandTitle": "Debug cell",+    "DataScience.debugCellCommandTitle": "Debug Cell",

Agreed, we have localize('DataScience.addCellBelowCommandTitle', 'Add cell');, shouldn't this be Add Cell FYI - in VS Code, they are capatalized, i.e. i'd make this change everywhere.

joyceerhl

comment created time in 3 days

Pull request review commentmicrosoft/vscode-python

Fix when clauses for new interactive cell shortcuts

                     "command": "python.datascience.insertCellBelowPosition",                     "title": "%python.command.python.datascience.insertCellBelowPosition.title%",                     "category": "Python",-                    "when": "python.datascience.hascodecells && editorFocus && editorLangId == python && python.datascience.featureenabled"+                    "when": "python.datascience.hascodecells && python.datascience.featureenabled && !notebookEditorFocused"

Don't you also need editorFocus && editorLangId == python So that we display these commands only for python files?

joyceerhl

comment created time in 3 days

issue openedmicrosoft/vscode-python

Kernel interrupt does not work for raw kernels

This no longer works in standard notebooks. This used to work, now on Mac it always times out with an option to restart the kernel.

created time in 3 days

more