profile
viewpoint

issue closedIRkernel/repr

large dataframe handling doesn't finish

Before submitting an issue please be sure to answer these questions with yes:

  • [x] Did you follow the official installation instructions?

    If you have an installation problem, a segfault, caught illegal operation, cannot open shared object file, … and installed IRkernel using Anaconda, please report your issue to them instead.

  • [x] Did you reproduce the error with the newest versions? If the following code installs packages, you didn’t yet!

    old_pkgs <- intersect(old.packages()[, 'Package'], c('IRkernel', 'IRdisplay', 'repr'))
    if (length(old_pkgs) > 0) install.packages(old_pkgs)
    
  • [ ] Did you include a minimal reproducible example?

  • [ ] Is this the right repository?

    If the way a value is displayed is ugly or you got ERROR while rich displaying an object, go to IRkernel/repr/issues/new instead.

closed time in 5 days

xu-huiyan

issue commentIRkernel/repr

large dataframe handling doesn't finish

Fixed in d25aad07474644a38bd146ad82ee29dd857de4e3.

Please try it out with devtools::install_github('IRkernel/repr') (then restart the kernel and try again)

xu-huiyan

comment created time in 5 days

push eventIRkernel/repr

Philipp A

commit sha d25aad07474644a38bd146ad82ee29dd857de4e3

Optimize has_row_names. Fixes #634

view details

push time in 5 days

issue commentIRkernel/IRkernel

large dataframe handling doesn't finish

You didn’t create a reproducible example. Please create a code block with some code that creates and displays a data.frame that takes much longer to display than in RStudio.

xu-huiyan

comment created time in 5 days

pull request commenttheislab/scanpy

Move adding `--internet-tests` option back to `conftest.py`

Yes, sorry, I wasn’t aware of this. Please add a small comment explaining this.

ivirshup

comment created time in 5 days

issue closedtheislab/scanpy

New logging clutters output

The rehauled logging module tends to clutter output, see, for instance

import scanpy as sc
sc.settings.verbosity = 2
adata_sc = sc.datasets.pbmc68k_reduced()
sc.pp.neighbors(adata_sc)

which produces image

closed time in 6 days

falexwolf

issue commenttheislab/scanpy

New logging clutters output

fixed in 709bafb8ed600daf5f9ee995a0dc845ac1e7e605

falexwolf

comment created time in 6 days

push eventtheislab/scanpy

Philipp A

commit sha 709bafb8ed600daf5f9ee995a0dc845ac1e7e605

strip microseconds

view details

push time in 6 days

push eventtheislab/scanpy

Philipp A

commit sha bf015ad10d8f7b3d7b257d9f56a801cb324163f9

really fixed logging tests

view details

push time in 6 days

issue commenttheislab/scanpy

New logging clutters output

Thank you!

step: removed logging of asctime, which we never had before

I took that from your logging code in the data science bowl notebook, but I agree: People should add {asctime} to all formatting themselves if they use scanpy in some non-interactive code. We should document how to do that.

Second step: reverted things that were logged at a level equal or higher than 4 to debug.

Good! FYI: I changed the log levels because the initial version of the overhaul only allowed timing information in info.

falexwolf

comment created time in 6 days

pull request commentpython/cpython

bpo-37173: Show passed class in inspect.getfile error

Great! Thank you for the tests @terryjreedy

flying-sheep

comment created time in 8 days

delete branch flying-sheep/cpython

delete branch : patch-1

delete time in 8 days

issue openednoahc3/SDSetup

Add AmiiSwap

Depends on emuiibo, so deselecting emuiibo should deselect Amiiswap and selecting AmiiSwap should also select emuiibo.

https://github.com/FuryBaguette/AmiiSwap

created time in 8 days

pull request commentpython/cpython

bpo-37173: Show passed class in inspect.getfile error

@asvetlov OK, done

flying-sheep

comment created time in 8 days

push eventflying-sheep/cpython

Philipp A

commit sha c3bf2b43365e840c966a3a2c14c971b041500b05

added tests

view details

push time in 8 days

pull request commenttheislab/anndata

AlignedMapping (obsm, varm, layers refactor)

Things are still incorrectly named “views”.

ivirshup

comment created time in 8 days

push eventtheislab/anndata

Philipp A

commit sha 4538dddee07bb540661e2938aa4a2bfbf34d341d

Simplified gitignore

view details

Philipp A

commit sha e9ccfc33fa4152ad4e137dea4e5700386c0e4da0

Remove os

view details

push time in 8 days

pull request commentpython/cpython

bpo-37173: Show passed class in inspect.getfile error

OK, I was opposed to just add a test for the thing I fixed (basically as a regression test), but proposing that the whole thing ought to be tested makes sense to me. I’ll add both news and tests

flying-sheep

comment created time in 8 days

push eventflying-sheep/stefanoborini.github.io

Philipp A

commit sha 723cc2193c776f009b576751965831e38461e0f8

Only ask for exact versions for apps, not packages

view details

push time in 8 days

push eventtheislab/scanpy

Philipp A

commit sha 51fc35c41bd8cd988dd4f3023df7b70480602082

Use download function for expression atlas downloading

view details

push time in 9 days

issue closedIRkernel/IRkernel

Capture output once browser tab is closed and code is still running

Is there a way to capture and store the output while code is still running but the browser tab is closed?

In IPython it is done by the magic line %%capture output, then one can run time consuming code, and when back run output.show(). Is there any way around this issue in IRkernel?

Thank you!

closed time in 9 days

orr161

push eventtheislab/destiny

Philipp A

commit sha 7041968f5b05b2a3602e25f55569e92b7f0b0ec6

Correctly construct factor

view details

push time in 9 days

push eventtheislab/destiny

Philipp A

commit sha 6c7bbf55d7145033f76dab0f655c9e3dabeadf01

label using all top_n genes

view details

push time in 9 days

pull request commentpython/cpython

bpo-37173: Show passed class in inspect.getfile error

May I convince you to look again instead? It’s 4 lines of very simple code…

  1. We go into the branch when object is a class
  2. object gets reassigned to the class’ module
  3. An error gets thrown, with the message talking about the class.

I simply introduce a new variable instead of reassigning object, so it stays assigned to the class and the error message is correct.

flying-sheep

comment created time in 10 days

pull request commentpython/cpython

bpo-37173: Show passed class in inspect.getfile error

I think we can skip the news entry on this tiny fix

flying-sheep

comment created time in 10 days

push eventKoncopd/anndata

falexwolf

commit sha 7284fd6389dbfa5072ebfe2b6d705242045dd73f

fixed read_loom tests

view details

Isaac Virshup

commit sha d9350c2153ba344f460556c0ffc97aa6359a61df

Release notes for `AnnData.{obs,var}_vector`.

view details

Isaac Virshup

commit sha c1926c871bee268b0e9b1b9e419462994da07c98

Merge pull request #158 from ivirshup/vector_release-notes Release notes for `AnnData.{obs,var}_vector`.

view details

falexwolf

commit sha de60eb11bd9f5b3001a9ad2ec16e32d7e201656f

reformated release notes

view details

falexwolf

commit sha d0fc7165df3bf7aa192b9484c4b60bd8312b070a

updated brief description

view details

falexwolf

commit sha e91e3bb32b25169fa5eaa93efb58bd967efffccf

removed brief release description

view details

Philipp A

commit sha db44e69e7922e80c77eec82e2b67d16651960ad4

Merge branch 'master' into write_fix

view details

push time in 10 days

pull request commenttheislab/anndata

fix write_zarr and write_h5ad

By the way, the previous version also worked with lists, i.e allowed multiple excluded attrs.

Huh? How? You had {excluded_attrs}, which creates a set containing “excluded_attrs” as its single element. excluded_attrs = ['a', 'b']; {excluded_attrs} will result in Error: A list object is not hashable and excluded_attrs = ('a', 'b'); {excluded_attrs} will be a Set[Tuple[str, str]]: {('a','b')}.

Need to test if this line is really required.

Yes. The line was introduced by @tomwhite implementing zarr storage in 2102629fbbc13c5948485c051bb6888680a681bf. Maybe he can help?

Koncopd

comment created time in 10 days

issue commentdavidhalter/jedi

Jedi Autocomplete failing to populate properties for @property wrapped functon that returns a class object

I disagree. A notebook is a static file containing code, like a module, so it makes sense that things defined statically in code cells have an influence on autocompletion.

cdknorow

comment created time in 10 days

issue commentdavidhalter/jedi

Jedi Autocomplete failing to populate properties for @property wrapped functon that returns a class object

I figured something out. If IPython did this internally, getsource would start to work. Would that make jedi work?

import sys
import linecache
from pathlib import Path
from nbformat import read, NO_CONVERT

with (Path() / notebook_name).open() as fp:
    notebook = read(fp, NO_CONVERT)
code_cells = [
    c['source']
    for c in notebook['cells']
    if c['cell_type'] == 'code' and not c['source'].startswith('%%')
]

code = '\n'.join(code_cells)

linecache.cache['__main__'] = len(code), None, code.splitlines(keepends=True), '__main__'
sys.modules['__main__'].__file__ = '__main__'
cdknorow

comment created time in 10 days

PR opened python/cpython

bpo-37173: Show passed class in inspect.getfile error

Currently, inspect.getfile(str) will report nonsense:

inspect.getfile(str) TypeError: <module 'builtins' (built-in)> is a built-in class

This fixes that

+3 -3

0 comment

1 changed file

pr created time in 10 days

push eventflying-sheep/cpython

Philipp A

commit sha cc7384e6135390ce78676e2a60f1929427c85cf5

Show passed class in inspect.getfile error Currently, inspect.getfile(str) will report nonsense: >>> inspect.getfile(str) TypeError: <module 'builtins' (built-in)> is a built-in class This fixes that

view details

push time in 10 days

issue commentdavidhalter/jedi

Jedi Autocomplete failing to populate properties for @property wrapped functon that returns a class object

What would be the best way to make Jedi as powerful as possible in IPython? Can we e.g. make it see all code cells? Jedi currectly only gets the code of the current cell I think.

Will Jedi use e.g. inspect.getsource? Could we make that function work for classes defined in the notebook?

cdknorow

comment created time in 10 days

push eventKoncopd/anndata

Philipp A

commit sha c49857fa51121d760f9a84ff9afde7b1e6e49b4d

Allow passing multiple excluded attrs

view details

push time in 10 days

issue commenttheislab/scanpy

HVG by cell_ranger flavor, n_top_genes not working

n_top_genes is used here:

https://github.com/theislab/scanpy/blob/6ac6440f154922027e7b416affc53be6d4a9978d/scanpy/preprocessing/_highly_variable_genes.py#L140-L148

I would assume that this only happens if there’s several genes with the exact same dispersion, is that possible?

We need a reproducible example, else we can’t help you further: Please give me some lines of code that I can paste into a notebook unchanged that will demonstrate the problem.

MonaLitvinukova

comment created time in 10 days

pull request commenttheislab/anndata

fix write_zarr and write_h5ad

Hi, I deduplicated this a bit. My IDE just broke and I need to leave soon so I didn’t test it.

Koncopd

comment created time in 11 days

push eventKoncopd/anndata

Philipp A

commit sha 4c6dc4ca1566994cbe6ecc5f0e72c29c249348f2

Whoops

view details

push time in 11 days

push eventKoncopd/anndata

Philipp A

commit sha 047153fafcb211e750b25d45255af61c33f30428

Deduplication

view details

push time in 11 days

issue openedtheislab/anndata

Broken logic in HDF5 writing

We have two separate copies (sigh) of some logic I broke (sorry) that need to be replaced with a fixed helper function:

https://github.com/theislab/anndata/blob/a9390d6fca537fafdf9e93f1b85a91c10af7ca58/anndata/readwrite/write.py#L151-L154

https://github.com/theislab/anndata/blob/a9390d6fca537fafdf9e93f1b85a91c10af7ca58/anndata/readwrite/write.py#L280-L286

Alex said the intention is to test if we’re dealing with a h5py.Dataset and dense array with matching dimensions and dtypes and if so, simply replace it.

created time in 11 days

pull request commenttheislab/scanpy

Logging overhaul

Well, the logging module is undocumented, only logging.print_versions() is public. Python has warnings.warn and logging.warning. I think we should follow suit.

If you want you can of course do from warnings import warn and use that.

flying-sheep

comment created time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 6ac6440f154922027e7b416affc53be6d4a9978d

Add docstrings

view details

push time in 11 days

delete branch theislab/scanpy

delete branch : logging-overhaul

delete time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 0480d21fbd4462cece9d1ebb2e4ab0675fb08c35

Set logging manager

view details

push time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha b3505e7e9e08a389dfff5788db2b6634abdeb1cd

Logging overhaul (#676)

view details

push time in 11 days

PR merged theislab/scanpy

Logging overhaul

Here we go. It’s not done yet:

  • [x] The timing stuff isn’t yet implemented. I think we just do timing stuff when doing “info” level logging, correct?

    If yes, we probably just need to override RootLogger.info so it sets self.handlers[0].formatter.passed_time = kwargs.get('t', False).

  • [x] All the places hackily using _settings_verbosity_greater_or_equal_than(2|3) have to work differently. I propose that we just add a kwarg deepinfo: str or so which only adds the passed string to the message if the active verbosity is higher than the function’s (i.e. calling logging.warn('foo', deepinfo='bar') and the loglevel/verbosity is 'INFO' or noisier adds 'bar')

  • [x] We now use vanilla log function syntax, so no more using it like print. We switched to Python 3.6+ though, so I propose f-strings everywhere! That looks better and works.

Fixes #256

+763 -569

5 comments

41 changed files

flying-sheep

pr closed time in 11 days

issue closedtheislab/scanpy

Get rid of logging module and use our own

We should replace the whole logging module with

import logging

root_logger = logging.getLogger('scanpy')
root_logger.setLevel('INFO')
root_logger.propagate = False  # Don’t pass log messages on to the root logger and its handler

handler = logging.StreamHandler(sys.stderr)  # Why did we use stdout?
handler.setFormatter(logging.Formatter('%(message)s'))
handler.setLevel('INFO')
root_logger.addHandler(handler)

def get_logger(name):
   return root_logger.getChild(name)

and in all submodules just do

from .logging import get_logger

logger = get_logger(__name__)

closed time in 11 days

flying-sheep

push eventtheislab/scanpy

Philipp A

commit sha a4f4607671e90d809401c37dd2da1f8181669720

Maybe works with capsys?

view details

push time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 4c9fb60d8e37f10a76b0fd304675904c7720534e

Tests succeed with pytest 4.4?

view details

push time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 63f0dd6be589e63f7e60c0a1e34b3051ddcc9e0c

Better hint at what the functions will do

view details

push time in 11 days

pull request commenttheislab/scanpy

Logging overhaul

OK, one more change. The log functions now all return the current time and have the optional parameter time: datetime. If you pass something there, the time will be logged:

start = log.info('foo')
# do stuff
log.hint('bar', time=start)  # --> bar (00:00:02)

You can customize where the time ends up via log.*('blah {time_passed}: blub', time=...)

flying-sheep

comment created time in 11 days

Pull request review commenttheislab/scanpy

Logging overhaul

 def neighbors(     adata.uns['neighbors']['connectivities'] = neighbors.connectivities     if neighbors.rp_forest is not None:         adata.uns['neighbors']['rp_forest'] = neighbors.rp_forest-    logg.info('    finished', time=True, end=' ' if _settings_verbosity_greater_or_equal_than(3) else '\n')-    logg.hint(-        'added to `.uns[\'neighbors\']`\n'-        '    \'distances\', distances for each pair of neighbors\n'-        '    \'connectivities\', weighted adjacency matrix')+    logg.info(+        '    finished',+        time=True,

I switched to explicit time passing

flying-sheep

comment created time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 9eca235ea4399ad5ee9e08d9cd5c0e40e2cc0cdb

Made log timing explicit

view details

push time in 11 days

pull request commenttheislab/scanpy

Logging overhaul

I’m doing all the tests @ivirshup proposed. If my explanations to the questions I left open are sufficient, I’ll merge this! :smile:

flying-sheep

comment created time in 11 days

push eventtheislab/scanpy

Philipp A

commit sha 53a3ccade990f7aad50d870ac83b7776cdc9df7c

Test timing

view details

push time in 11 days

Pull request review commenttheislab/scanpy

Logging overhaul

 def neighbors(     adata.uns['neighbors']['connectivities'] = neighbors.connectivities     if neighbors.rp_forest is not None:         adata.uns['neighbors']['rp_forest'] = neighbors.rp_forest-    logg.info('    finished', time=True, end=' ' if _settings_verbosity_greater_or_equal_than(3) else '\n')-    logg.hint(-        'added to `.uns[\'neighbors\']`\n'-        '    \'distances\', distances for each pair of neighbors\n'-        '    \'connectivities\', weighted adjacency matrix')+    logg.info(+        '    finished',+        time=True,

It doesn’t seem like there’s any info call like that at the moment.

But theoretically changing that behavior is of course possible.

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 """Logging and Profiling """+import logging+import time as time_+from logging import CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET+from datetime import datetime -import time as time_module-from datetime import timedelta, datetime-from typing import Union+import anndata.logging -from anndata import logging-from ._settings import settings +HINT = (INFO + DEBUG) // 2+logging.addLevelName(HINT, 'HINT') -_VERBOSITY_LEVELS_FROM_STRINGS = {-    'error': 0,-    'warn': 1,-    'info': 2,-    'hint': 3,-    'debug': 4,-} +class RootLogger(logging.RootLogger):+    def __init__(self, level):+        super().__init__(level)+        self.propagate = False+        self.last_time = time_.time() -def info(*args, **kwargs):-    return _msg(*args, v='info', **kwargs)+    def log(self, level, msg, *, extra=None, deep=None, time_passed=None):

yeah! I heavily modified your code, but based the implementation on it.

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 def palantir(adata: AnnData):      """ -    logg.info('Palantir diffusion maps', r=True)+    logg.info('Palantir diffusion maps')

I checked and it seems like automatically resetting at every info call works, as there seem to be no info loggings in the middle of some long running process.

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 def filter_cells(      s = np.sum(~cell_subset)     if s > 0:-        logg.info('filtered out {} cells that have'.format(s), end=' ')+        msg = f'filtered out {s} cells that have'         if min_genes is not None or min_counts is not None:-            logg.info('less than',-                   str(min_genes) + ' genes expressed'-                   if min_counts is None else str(min_counts) + ' counts', no_indent=True)+            msg += 'less than '+            msg += f'{min_genes} genes expressed' if min_counts is None else f'{min_counts} counts'         if max_genes is not None or max_counts is not None:-            logg.info('more than ',-                   str(max_genes) + ' genes expressed'-                   if max_counts is None else str(max_counts) + ' counts', no_indent=True)+            msg += 'more than '+            msg += f'{max_genes} genes expressed' if max_counts is None else f'{max_counts} counts'+        logg.info(msg)

I think I didn’t change that anywhere. The idea was to change as little as possible while changing the API

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 def logfile(self) -> Union[Path, TextIO]:      @logfile.setter     def logfile(self, logfile: Union[str, Path, TextIO, None]):-        _type_check(logfile, "logfile", (str, Path, type(None)))-        stream = sys.stdout if self._is_run_from_ipython() else sys.stderr-        self._logfile = Path(logfile) if logfile else stream+        if not hasattr(logfile, 'write') and logfile:+            self.logpath = logfile+        else:  # file object+            if not logfile:  # None or ''+                logfile = sys.stdout if self._is_run_from_ipython() else sys.stderr+            self._logfile = logfile+            self._logpath = None

Thank you, fixed

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 import sys from pathlib import Path from time import time-from typing import Tuple, Union, Any, List, Iterable, TextIO+from typing import Tuple, Union, Any, List, Iterable, TextIO, Optional++from . import logging+from .logging import _set_log_level, _set_log_file, RootLogger++_VERBOSITY_TO_LOGLEVEL = {+    'error': 'ERROR',+    'warn': 'WARNING',+    'info': 'INFO',+    'hint': 'HINT',+    'debug': 'DEBUG',+}+verbosity_names = list(_VERBOSITY_TO_LOGLEVEL.keys())+# Python 3.7 ensures iteration order+for v, level in enumerate(list(_VERBOSITY_TO_LOGLEVEL.values())):+    _VERBOSITY_TO_LOGLEVEL[v] = level

I just introduced an IntEnum that gets used instead of integers.

I guess we should simplify the code to use that one!

flying-sheep

comment created time in 12 days

Pull request review commenttheislab/scanpy

Logging overhaul

 import sys from pathlib import Path from time import time-from typing import Tuple, Union, Any, List, Iterable, TextIO+from typing import Tuple, Union, Any, List, Iterable, TextIO, Optional++from . import logging+from .logging import _set_log_level, _set_log_file, RootLogger++_VERBOSITY_TO_LOGLEVEL = {+    'error': 'ERROR',+    'warn': 'WARNING',

Good point, I switched everything to warning!

flying-sheep

comment created time in 12 days

push eventtheislab/scanpy

Philipp A

commit sha ef13d215d7c6438317840e1aa2b4a687b80a0c82

warn→warning, more tests

view details

push time in 12 days

push eventtheislab/scanpy

Philipp A

commit sha 0de73a5ce810a40ca70e38b3f26a1880680dc29c

Fixed logic error and added logging tests

view details

push time in 12 days

push eventtheislab/scanpy

Philipp A

commit sha e5a38fd86c1f979126321b916973f41e9fd8e3f0

Limited logging interface

view details

push time in 12 days

pull request commenttheislab/scanpy

Logging overhaul

Oh, whoops, 3.6 of course. That was the one introducing f-strings. Sorry, I’m not that good in ancient history :wink:

flying-sheep

comment created time in 12 days

PR opened theislab/scanpy

Logging overhaul

Here we go. It’s not done yet:

  1. The timing stuff isn’t yet implemented. I think we just do timing stuff when doing “info” level logging, correct?

    If yes, we probably just need to override RootLogger.info so it sets self.handlers[0].formatter.passed_time = kwargs.get('t', False).

  2. All the places hackily using _settings_verbosity_greater_or_equal_than(2|3) have to work differently. I propose that we just add a kwarg deepinfo: str or so which only adds the passed string to the message if the active verbosity is higher than the function’s (i.e. calling logging.warn('foo', deepinfo='bar') and the loglevel/verbosity is 'INFO' or noisier adds 'bar')

  3. We now use vanilla log function syntax, so no more using it like print. We switched to Python 3.7+ though, so I propose f-strings everywhere! That looks better and works.

+125 -148

0 comment

14 changed files

pr created time in 12 days

create barnchtheislab/scanpy

branch : logging-overhaul

created branch time in 12 days

push eventtheislab/scanpy

Philipp A

commit sha 2bb562c0a9ed0dcd9d955370109d8024d9edfc13

Clean up logging module

view details

push time in 13 days

push eventtheislab/scanpy

Philipp A

commit sha bcbc3d00e38f7f06c44c8e793fac8b2ce32e17a0

More pathlib

view details

push time in 13 days

push eventtheislab/scanpy

Philipp A

commit sha c241a91f4e99a8a8e158ef89fe609d6932688911

Fixed settings.logfile, disuse logging.msg Since settings.logfile was broken, nobody seems to ever have used it. My fix is pretty slow, as I reopen the logfile. We should keep it open when switching to Python’s logging. The log levels 5 and even 6 used to be used when calling logging.msg. I replaced them with logging.debug in preparation of moving to python’s logging. I also discovered that the _sim module is highly broken. I fixed a bit superficial stuff, but using anything that’s not tested will likely break

view details

push time in 13 days

CommitCommentEvent

push eventtheislab/scanpy

Philipp A

commit sha 70ed4c8ddc3e7f8e69b092317e61855c5f1671cb

Restored ignoring /docs/_build/

view details

push time in 13 days

issue commentipython/ipython

Interactive shell does not complete members for property values

I think that’s related to davidhalter/jedi#1299: Basically it’s impossible to correctly infer the type of properties without executing them. However, I think it should be possible to statically infer the types of properties that don’t have branching code and always return the same type.

vmalloc

comment created time in 13 days

issue commentdavidhalter/jedi

Jedi Autocomplete failing to populate properties for @property wrapped functon that returns a class object

For me, it doesn’t even work when I add a type hint, how come?

    @property
    def project(self) -> Project:
        return self._project
cdknorow

comment created time in 13 days

push eventtheislab/anndata

Ryan Williams

commit sha 76375e2145c0c4be716ac40df41fa809a10fbaa9

rm dead read_zarr code (copied from h5ad path) (#154)

view details

push time in 14 days

pull request commenttheislab/anndata

rm dead read_zarr code (copied from h5ad path)

Thank you!

ryan-williams

comment created time in 14 days

push eventtheislab/anndata2ri

Philipp A

commit sha 3fe0c7dbd24474dc7fa7ebd9c24ba04509b3615e

typo

view details

push time in 14 days

push eventtheislab/anndata2ri

Philipp A

commit sha 2c0b06600cd7e53c7639dd476cebd5f49d78d891

Added link to example notebook

view details

push time in 14 days

issue commenttheislab/scanpy

Converting Seurat object to loom format

Sounds like it runs out of memory, no?

I’m happy that my anndata2ri works for you!

ts3139

comment created time in 14 days

push eventtheislab/destiny

Philipp A

commit sha 37fd56a124a36e03525806c667c1e136cdd286d5

More WIP

view details

push time in 15 days

push eventtheislab/destiny

Philipp A

commit sha b59519e8f1823cc0bf0f22dce2198ba8200ee01f

Prevent broken params

view details

push time in 15 days

push eventtheislab/destiny

Philipp A

commit sha 2f6de2a543c4899e55d23457e74e6d29c3910914

WIP GR plotting rank

view details

push time in 15 days

issue closedtheislab/scanpy

Error message in tl.diffmap / why n_comps must be > 2

Hi,

I'm using your package tl.diffmap in my analysis, and I'm having some difficulties. I have a dataframe I have converted into an anndata object adata. I run the following lines to prepare it for tl.diffmap:

pp.pca(adata,n_comps=50) pp.neighbors(adata, knn = False, method = 'gauss', n_neighbors = 20)

I then perform the diffmap:

tl.diffmap(adata, n_comps = 3)

and I get the following error:

Screen Shot 2019-05-15 at 6 11 47 PM

I am not sure what I am doing incorrectly here, and I was hoping you could help!

Furthermore, I was wondering why n_comps must be greater than 2?

closed time in 16 days

lsiraj

issue commenttheislab/scanpy

Error message in tl.diffmap / why n_comps must be > 2

Fixed in d44d5f7351719643bd21042d560355b83188b9e5

lsiraj

comment created time in 16 days

push eventtheislab/scanpy

Philipp A

commit sha be24f126bfd2a86510f79193326440d02ddb5b1f

Allow scipy 1.2.x

view details

push time in 16 days

push eventtheislab/scanpy

Philipp A

commit sha d44d5f7351719643bd21042d560355b83188b9e5

Neighbors overhaul, fixed restoring n_neighbors from dense arrays

view details

push time in 16 days

issue commenttheislab/scanpy

Error message in tl.diffmap / why n_comps must be > 2

I think I see what’s going wrong. Scipy sparse matrices have the method <matrix class>.count_nonzero(self, ...) while numpy’s ndarrays don’t. One would use the static function numpy.count_nonzero(a, ...) for that.

I.e. the neighbors code assumes self._connectivities is a sparse matrix, while it isn’t in your case.

lsiraj

comment created time in 16 days

delete branch flying-sheep/RSpectra

delete branch : sym

delete time in 16 days

push eventtheislab/bartSeq

Philipp A

commit sha b97f67817a6874526f9561edd948866b323cc470

added disclaimer

view details

push time in 18 days

issue closedtheislab/scanpy

sc.tl.umap numba error when used with init_pos="paga"

Hi Alex,

UMAP throws an error if I use scanpy.tl.ump with initial positions from sc.tl.paga. Based on the error (see below) I thought it was a problem of UMAP itself. However, the error is not thrown when called without initial positions from paga.

Here is the output / error:

sc.tl.umap(adata, init_pos='paga')

computing UMAP
    using 'X_pca' with n_pcs = 50

---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-35-924452b37e5b> in <module>
----> 1 sc.tl.umap(adata, init_pos='paga')

/opt/conda/lib/python3.7/site-packages/scanpy/tools/_umap.py in umap(adata, min_dist, spread, n_components, maxiter, alpha, gamma, negative_sample_rate, init_pos, random_state, a, b, copy)
    137         neigh_params.get('metric', 'euclidean'),
    138         neigh_params.get('metric_kwds', {}),
--> 139         verbose=max(0, verbosity-3))
    140     adata.obsm['X_umap'] = X_umap  # annotate samples with UMAP coordinates
    141     logg.info('    finished', time=True, end=' ' if _settings_verbosity_greater_or_equal_than(3) else '\n')

/opt/conda/lib/python3.7/site-packages/umap/umap_.py in simplicial_set_embedding(data, graph, n_components, initial_alpha, a, b, gamma, negative_sample_rate, n_epochs, init, random_state, metric, metric_kwds, verbose)
    984         initial_alpha,
    985         negative_sample_rate,
--> 986         verbose=verbose,
    987     )
    988 

/opt/conda/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    348                 e.patch_message(msg)
    349 
--> 350             error_rewrite(e, 'typing')
    351         except errors.UnsupportedError as e:
    352             # Something unsupported is present in the user code, add help info

/opt/conda/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
    315                 raise e
    316             else:
--> 317                 reraise(type(e), e, None)
    318 
    319         argtypes = []

/opt/conda/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
    656             value = tp()
    657         if value.__traceback__ is not tb:
--> 658             raise value.with_traceback(tb)
    659         raise value
    660 

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of type(CPUDispatcher(<function rdist at 0x7fb3c827f840>)) with parameters (array(float64, 1d, C), array(float64, 1d, C))
Known signatures:
 * (array(float32, 1d, A), array(float32, 1d, A)) -> float32
 * parameterized
[1] During: resolving callee type: type(CPUDispatcher(<function rdist at 0x7fb3c827f840>))
[2] During: typing of call at /opt/conda/lib/python3.7/site-packages/umap/umap_.py (776)


File "../../../opt/conda/lib/python3.7/site-packages/umap/umap_.py", line 776:
def optimize_layout(
    <source elided>

                dist_squared = rdist(current, other)
                ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/dev/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/dev/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

What I basically do from raw UMI counts:

  1. total counts normalization / logarithmization
  2. PCA, bbknn, louvain
  3. combat, HVG, PCA, UMAP (works well)
  4. Paga (with louvain from 2., works well)
  5. UMAP (with positions from 4., does not work)

Any idea? Any further info needed? Best, Jens

closed time in 18 days

jenzopr

issue commenttheislab/scanpy

sc.tl.umap numba error when used with init_pos="paga"

Great, thank you. Seems like the issue is well placed over there.

jenzopr

comment created time in 18 days

issue commenttheislab/scanpy

sc.tl.umap numba error when used with init_pos="paga"

The error seems to be that rdist is called from umap.umap_.optimize_layout with float64 arrays while it can only handle float32 arrays.

There seem to have been a few changes in umap between 0.3.8 and 0.3.9 maybe you should try with that.

jenzopr

comment created time in 18 days

issue commenttheislab/destiny

why are eigenvalues in decreasing order?

That’s curious, the most popular numerical spectral decomposition algorithm (Arnoldi/Lanczos) is used everywhere in its ARPACK implementation. Scipy, igraph, MATLAB, you name it.

And ARPACK is best at finding extremal eigenvalues. Every single one of those popular libraries should give you the largest-magnitude eigenvalues by default and warn you that finding the small ones will be slow and maybe won’t converge.

So where did you see those unusual small eigenvalues?

taylorreiter

comment created time in 18 days

issue commentnmslib/hnswlib

Does this version of hnsw works with sparse matrix?

"Python bindings not being helpful"? For R, or in general?

For R! I can’t expect people using my R package to have such a complex setup.

You can pass data to NMSLIB using dense matrices, sparse matrices, and strings.

Thank you, I missed that in my cursory glance! Fixed

pommedeterresautee

comment created time in 18 days

delete branch flying-sheep/mermaid

delete branch : svg-links

delete time in 18 days

issue commentnmslib/hnswlib

Does this version of hnsw works with sparse matrix?

Yes, thank you! I’ve found that one, but you’re right, Python-based bindings aren’t that helpful.

Here’s a little overview I made about ANN libraries in R

pommedeterresautee

comment created time in 18 days

issue commentnmslib/hnswlib

Does this version of hnsw works with sparse matrix?

Oh! Well, I hadn’t found that, thank you! But yeah, a Python-based solution isn’t really helping me much sadly.

pommedeterresautee

comment created time in 18 days

more