profile
viewpoint

Wykks/Untamed-Now-Playing-Next 76

Untamed-Now-Playing

Senpos/aiogram 0

Is a pretty simple and fully asynchronous framework for Telegram Bot API written in Python 3.7 with asyncio and aiohttp.

Senpos/babi-grammars 0

pip installable package to provide syntax grammars for babi

Senpos/httpx 0

A next generation HTTP client for Python. 🦋

Senpos/poetry-pycharm-plugin 0

A PyCharm plugin for poetry

Senpos/pyquery 0

A jquery-like library for python

Senpos/super-bot 0

radio-t bot

Senpos/Untamed-Now-Playing-Next 0

Untamed-Now-Playing

startedpygame/pygame

started time in 2 days

startedart049/odmantic

started time in 11 days

startediafan/Config-Neat

started time in 17 days

startedmicrosoft/playwright-python

started time in 21 days

issue openedpyrogram/tgcrypto

Add wheels for Python 3.9

Hello!

Python 3.9 released on October 5th.

It would be cool if we could install TgCrypto from wheels. :-) Especially, this is useful for Windows users, as it is pretty common not to have a compiler installed, as it is quite bulky. 😿

At the moment. TgCrypto has wheels for Python versions from 3.5 to 3.8.

Please, let me know if I can help you with this somehow!

created time in 22 days

issue commentpython-poetry/poetry

Always use virtualenv 20.x

Since #2666 is merged, should I close the issue? :)

Senpos

comment created time in 23 days

startedencode/httpx-cli

started time in a month

startedthombashi/pytablewriter

started time in 2 months

startedcbrunet/python-poppler

started time in 2 months

push eventSenpos/evil-bot

Evgeny Kovalchuk

commit sha f4c4371d722f572297fdae59809a7605531e29b2

feat: show the name of a user who unblocked stickerpack

view details

push time in 2 months

PR opened Djaler/evil-bot

Показывать имя администратора, который разблокировал стикерпак

Сейчас, после вызова команды /unblock_stickerpack, любой из администраторов может нажать на inline-кнопку и разблокировать стикерпак. При этом, нельзя узнать кто именно это сделал.

Это изменение позволит видеть всем пользователям чата кто разблокировал конкретный стикерпак.

+4 -1

0 comment

1 changed file

pr created time in 2 months

create barnchSenpos/evil-bot

branch : show-admin-name-unblock-stickerpack

created branch time in 2 months

PR opened asottile/babi-grammars

Allow passing multiple args to the "--only" param

This allows to download/update several repositories using the --only parameter e.g.

python bin\download-syntax update --only MagicStack/MagicPython --only PowerShell/EditorSyntax

notice, that you have to add --only to each repository name

+7 -6

0 comment

1 changed file

pr created time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha f7b48963fc4fb490ae4499798371472ab8c03670

use upstream vscode-nim

view details

Anthony Sottile

commit sha 7663a22bfb3b09be235468213b348cf9ecc1ffe3

Merge pull request #20 from asottile/upstream_nim use upstream vscode-nim

view details

Anthony Sottile

commit sha 27c4010dbcb24472e2a8efe65a2e86392aaf2a3c

update grammars

view details

Anthony Sottile

commit sha 2f935d2a38f2be05f830c447a754af8be69deb58

Merge pull request #21 from asottile/update update grammars

view details

Anthony Sottile

commit sha 70357f54b478271c932d385d06fa05bd82d6dfe2

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha 63d81b4909d1f3aa791092b28e1ca34d07bcccd5

Merge pull request #22 from asottile/terraform_unfork vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha cd56458c8a112e49798b48d038afc5f2cf17f97c

v0.0.18

view details

Evgeny Kovalchuk

commit sha fa9effb2532bbae37275a34ee30ab01e811f04ca

Allow passing multiple args to the "--only" param

view details

push time in 2 months

PR opened asottile/babi-grammars

Add "encoding='utf-8'" to all "open" calls explicitly

UTF-8 is a default nowadays.

With this change, all commands should produce / read text files using utf-8 instead of other system default encodings (in my case, it is Russian version of Windows 10 with Windows-1251)

No more this: image

+8 -8

0 comment

2 changed files

pr created time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha f7b48963fc4fb490ae4499798371472ab8c03670

use upstream vscode-nim

view details

Anthony Sottile

commit sha 7663a22bfb3b09be235468213b348cf9ecc1ffe3

Merge pull request #20 from asottile/upstream_nim use upstream vscode-nim

view details

Anthony Sottile

commit sha 27c4010dbcb24472e2a8efe65a2e86392aaf2a3c

update grammars

view details

Anthony Sottile

commit sha 2f935d2a38f2be05f830c447a754af8be69deb58

Merge pull request #21 from asottile/update update grammars

view details

Anthony Sottile

commit sha 70357f54b478271c932d385d06fa05bd82d6dfe2

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha 63d81b4909d1f3aa791092b28e1ca34d07bcccd5

Merge pull request #22 from asottile/terraform_unfork vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha cd56458c8a112e49798b48d038afc5f2cf17f97c

v0.0.18

view details

Evgeny Kovalchuk

commit sha bf994924e6a72dc3ca337ec4603793ab0e11bb93

Add "encoding='utf-8'" to "open" calls explicitly

view details

push time in 2 months

PR opened asottile/babi-grammars

Add concurrency to the repos download command

Downloading all the repos took me ~35-40 seconds, which is quite a lot.

With this change it now takes approximately 6-8 seconds on my machine with 4 cores.

While it is not a huge deal, since the tool is used mostly internally, but it looks like a good "quality of life" change with the minimal code-base modifications

a good portion of the code was just moved to a separate function with no additional changes then, that function is used in the ThreadPool

+58 -45

0 comment

1 changed file

pr created time in 2 months

push eventSenpos/babi-grammars

Evgeny Kovalchuk

commit sha d16fe5271597062d25fc22651a39629208028f7d

Add concurrency to the repos download command

view details

push time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha f7b48963fc4fb490ae4499798371472ab8c03670

use upstream vscode-nim

view details

Anthony Sottile

commit sha 7663a22bfb3b09be235468213b348cf9ecc1ffe3

Merge pull request #20 from asottile/upstream_nim use upstream vscode-nim

view details

Anthony Sottile

commit sha 27c4010dbcb24472e2a8efe65a2e86392aaf2a3c

update grammars

view details

Anthony Sottile

commit sha 2f935d2a38f2be05f830c447a754af8be69deb58

Merge pull request #21 from asottile/update update grammars

view details

Anthony Sottile

commit sha 70357f54b478271c932d385d06fa05bd82d6dfe2

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha 63d81b4909d1f3aa791092b28e1ca34d07bcccd5

Merge pull request #22 from asottile/terraform_unfork vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha cd56458c8a112e49798b48d038afc5f2cf17f97c

v0.0.18

view details

Evgeny Kovalchuk

commit sha b1de0ec6f6cd43460fc8e9279b0b17ba48b919b5

Add concurrency to the repos download command

view details

Evgeny Kovalchuk

commit sha ac18c38d903658243591132e3c6673673fd28493

Merge branch 'download-repos-concurrently' of https://github.com/Senpos/babi-grammars into download-repos-concurrently

view details

push time in 2 months

push eventSenpos/babi-grammars

push time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha 5f439bfdecbdcc16f8a64541117f1f5e56f6e860

use upstream vscode-nim

view details

Anthony Sottile

commit sha 8c15fa0859fbf940e7e2d5a937a406cbd805c15d

update grammars

view details

Anthony Sottile

commit sha 825624cc7bb30ee47f0840e53b2044c1650d86b9

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha 536c20cb7e883e7342ab5112d4dbbb129085af04

v0.0.18

view details

push time in 2 months

push eventSenpos/babi-grammars

push time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha f7b48963fc4fb490ae4499798371472ab8c03670

use upstream vscode-nim

view details

Anthony Sottile

commit sha 7663a22bfb3b09be235468213b348cf9ecc1ffe3

Merge pull request #20 from asottile/upstream_nim use upstream vscode-nim

view details

Anthony Sottile

commit sha 27c4010dbcb24472e2a8efe65a2e86392aaf2a3c

update grammars

view details

Anthony Sottile

commit sha 2f935d2a38f2be05f830c447a754af8be69deb58

Merge pull request #21 from asottile/update update grammars

view details

Anthony Sottile

commit sha 70357f54b478271c932d385d06fa05bd82d6dfe2

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha 63d81b4909d1f3aa791092b28e1ca34d07bcccd5

Merge pull request #22 from asottile/terraform_unfork vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha cd56458c8a112e49798b48d038afc5f2cf17f97c

v0.0.18

view details

Evgeny Kovalchuk

commit sha 08175dd94b6d07a23c34e20b14096ff2baf60edb

Add concurrency to the repos download command

view details

Evgeny Kovalchuk

commit sha 6771c4e8acf4479caea4c6df66aa4a64ec7bb12b

Merge remote-tracking branch 'origin/download-repos-concurrently' into download-repos-concurrently

view details

push time in 2 months

push eventSenpos/babi-grammars

Anthony Sottile

commit sha 4414cdb308d95ac0454b96c03c50ff90b763bf8a

use upstream vscode-nim

view details

Anthony Sottile

commit sha 3b4eeff979bfa8d8d64bbbf91dcc6b857f9b416d

update grammars

view details

Anthony Sottile

commit sha b5e0a35f5e72bdc404577b75fae9e18f7d9ed820

vscode-terraform accepted my patch

view details

Anthony Sottile

commit sha b9e4a8697531d9491f1f9adb83a04fdea95404a0

v0.0.18

view details

push time in 2 months

startedpallets/jinja

started time in 2 months

delete branch Senpos/babi-grammars

delete branch : concurrency-and-multiple-only-args

delete time in 2 months

create barnchSenpos/babi-grammars

branch : download-repos-concurrently

created branch time in 2 months

create barnchSenpos/babi-grammars

branch : multiple-args-only-param

created branch time in 2 months

create barnchSenpos/babi-grammars

branch : explicit-encoding-for-open

created branch time in 2 months

PR closed asottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter
  • added concurrent (ThreadPoolExecutor) downloads for the _download command, had to refactor it a bit and break into smaller pieces On 4 cores, python .\bin\download-syntax download: Before: 33-47s After: 4.9-5s
  • added concurrent processing (ThreadPoolExecutor) for the _update command On 4 cores, python .\bin\download-syntax update: Before: 75-82s After: 37-39s Well, not THAT huge difference, but I suppose moby/moby repo is the bottleneck here, it takes forever to process it
  • if one concurrent task fails - log to the stderr, other tasks continue to work; we don't want to waste things we already processed :-)
  • replaced all path strings with pathlib.Path objects when possible; it allows to make reproducible results on both Linux and Windows (slashes/backslashes, uh-oh)
  • added encoding='utf-8' parameter to all open calls, because on some locales on Windows it creates files with incorrect symbols
  • added support for passing multiple arguments to the --only parameter for both _download and _update commands. Now you can do python .\bin\download-syntax download --only MagicStack/MagicPython asottile/language-xml or python .\bin\download-syntax download --only MagicStack/MagicPython --only asottile/language-xml, both "styles" should work (warning: it is a 3.8+ feature (scroll down to the extend action)

Yes, I messed up and all the changes were added to a single commit. Sorry! If you are interested in something particular from the list - please, let me know and I will create a separate PR for it.

If it looks good enough - I may add changes to the README. Let me know what you think. :-)

P. S. I learned my lesson

+193 -110

4 comments

1 changed file

Senpos

pr closed time in 2 months

pull request commentasottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter

Thanks, Anthony!

could be split out into separate PRs

That's what I will do. :-) So, according to your comments, I plan to create PRs for:

  • encoding='utf-8' thingy
  • ThreadPoolExecutor only for the _download command, because it has the most impact (5s vs old 40+s)
  • allow to pass multiple args to only parameter using append action

I am closing this PR ^^

Senpos

comment created time in 2 months

Pull request review commentasottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter

 def _ts_to_js(scope: str, dct: Dict[str, Any]) -> Dict[str, Any]:     return ret  -def _download(*, only: Optional[str]) -> int:-    assert os.path.exists(_GRAMMAR_DIR)-    assert os.path.exists(_LICENSE_DIR)+def _load_grammar_string(grammar_string: str, strategies: Sequence[Callable]):+    for strategy in strategies:+        try:+            loaded_grammar = strategy(grammar_string)+        except Exception:+            continue+        else:+            return loaded_grammar+    raise AssertionError(f'could not parse "{grammar_string}"')+ -    licenses: Set[str] = set()-    grammars: Set[str] = set()+def _download_grammar(repo: Repo, grammar: str) -> Path:+    grammar_string = urllib.request.urlopen(repo.url(grammar)).read()+    loaded_grammar = _load_grammar_string(grammar_string, _STRATEGIES)+    grammar_filepath = _GRAMMAR_DIR / f'{loaded_grammar["scopeName"]}.json'+    with grammar_filepath.open('w', encoding='utf-8') as f:+        json.dump(loaded_grammar, f)+        f.write('\n')+    return grammar_filepath -    for repo in REPOS:-        if only is not None and repo.name != only:++def _parse_license_with_header(license_string: str, header: str) -> Optional[str]:+    sections = HEADER_RE.split(license_string)+    for section in sections:+        if not section.startswith(header):             continue-        print(f'{repo.name}...')+        _, _, section_body = section.partition('\n')+        return section_body+++def _download_repo_license(repo: Repo):+    license_path, _, header = repo.license_path.partition('#')+    license_url = repo.url(license_path)+    license_string = urllib.request.urlopen(license_url).read().decode()++    if header:+        license_string = _parse_license_with_header(license_string, header)+        if license_string is None:+            raise AssertionError(f'license not found "{repo.license_path}" for "{repo.name}"')++    # some licenses have trailing whitespace, ugh+    license_string = TRAILING_WS_RE.sub('', license_string)++    with open(os.path.join(_LICENSE_DIR, repo.license_filename), 'w', encoding='utf-8') as f:+        f.write(f'LICENSE retrieved from {license_url}\n\n')+        f.write(f'{"-" * 79}\n\n')+        f.write(f'{license_string.rstrip()}\n')+++def _download_repo(repo: Repo) -> Tuple[Path, Set[Path]]:+    print(f'{repo.name}...')+    _download_repo_license(repo)+    license_filepath = _LICENSE_DIR / repo.license_filename++    grammars_filepaths = set()+    with ThreadPoolExecutor() as executor:+        futures = {executor.submit(_download_grammar, repo, grammar): grammar for grammar in repo.grammars}+        for future in as_completed(futures):+            try:+                downloaded_grammar_filepath = future.result()+            except Exception as e:+                print(f'error while downloading grammar "{futures[future]}": {e}', file=stderr)+                continue+            else:+                grammars_filepaths.add(downloaded_grammar_filepath)+    return license_filepath, grammars_filepaths -        licenses.add(repo.license_filename) -        license_path, _, header = repo.license_path.partition('#')-        license_url = repo.url(license_path)-        license_s = urllib.request.urlopen(license_url).read().decode()+def _download(*, only: Set[str]) -> int:+    assert _GRAMMAR_DIR.exists()+    assert _LICENSE_DIR.exists() -        if header:-            sections = HEADER_RE.split(license_s)-            for section in sections:-                if section.startswith(header):-                    _, _, license_s = section.partition('\n')-                    break-            else:-                raise AssertionError(f'not found {repo.license_path}')--        # some licenses have trailing whitespace, ugh-        license_s = TRAILING_WS_RE.sub('', license_s)--        with open(os.path.join(_LICENSE_DIR, repo.license_filename), 'w') as f:-            f.write(f'LICENSE retrieved from {license_url}\n\n')-            f.write(f'{"-" * 79}\n\n')-            f.write(f'{license_s.rstrip()}\n')--        for grammar in repo.grammars:-            grammar_s = urllib.request.urlopen(repo.url(grammar)).read()-            for strategy in _STRATEGIES:-                try:-                    loaded = strategy(grammar_s)-                except Exception:-                    continue-                else:-                    break-            else:-                raise AssertionError(f'could not parse {grammar}')+    licenses: Set[Path] = set()+    grammars: Set[Path] = set() -            grammar_name = f'{loaded["scopeName"]}.json'-            grammars.add(grammar_name)-            with open(os.path.join(_GRAMMAR_DIR, grammar_name), 'w') as f:-                json.dump(loaded, f)-                f.write('\n')+    repos = REPOS+    if only:+        repos = [repo for repo in repos if repo.name in only]++    with ThreadPoolExecutor() as executor:+        futures = {executor.submit(_download_repo, repo): repo for repo in repos}++        if not futures and only:+            print(+                'Nothing to do. Make sure you provided correct repo names to the "only" parameter',+                file=stderr+            )+            return 1++        for future in as_completed(futures):+            try:+                license_filepath, grammars_filepaths = future.result()+            except Exception as e:+                print(f'error downloading repo {futures[future]}: {e}', file=stderr)+                continue+            else:+                licenses.add(license_filepath)+                grammars.update(grammars_filepaths)      # similar to what vs code does, derive javascript from typescript-    if only is None or only == 'microsoft/TypeScript-TmLanguage':-        with open(os.path.join(_GRAMMAR_DIR, 'source.tsx.json')) as f:+    if 'microsoft/TypeScript-TmLanguage' in repos:+        with _GRAMMAR_DIR.joinpath('source.tsx.json').open(encoding='utf-8') as f:             tsx = json.load(f)          for scope in ('.js', '.js.jsx'):-            grammar_name = f'source{scope}.json'-            grammars.add(grammar_name)-            with open(os.path.join(_GRAMMAR_DIR, grammar_name), 'w') as f:+            grammar_filepath = _GRAMMAR_DIR / f'source{scope}.json'+            grammars.add(grammar_filepath)+            with grammar_filepath.open('w', encoding='utf-8') as f:                 json.dump(_ts_to_js(scope, tsx), f)                 f.write('\n') -    if only is None:-        existing_licenses = set(os.listdir(_LICENSE_DIR))+    if not only:+        existing_licenses = set(_LICENSE_DIR.iterdir())         for victim in existing_licenses - licenses:-            os.remove(os.path.join(_LICENSE_DIR, victim))-        existing_grammars = set(os.listdir(_GRAMMAR_DIR))+            victim.unlink()+        existing_grammars = set(_GRAMMAR_DIR.iterdir())         for victim in existing_grammars - grammars:-            os.remove(os.path.join(_GRAMMAR_DIR, victim))+            victim.unlink()     else:-        licenses = set(os.listdir(_LICENSE_DIR))-        grammars = set(os.listdir(_GRAMMAR_DIR))+        licenses = set(_LICENSE_DIR.iterdir())+        grammars = set(_GRAMMAR_DIR.iterdir())      cfg = configparser.ConfigParser()-    cfg.read('setup.cfg')+    setup_cfg_filepath = Path('setup.cfg')+    cfg.read(setup_cfg_filepath) -    license_files = sorted(os.path.join(_LICENSE_DIR, f) for f in licenses)+    # we use Path.as_posix() everywhere to make sure that paths (slashes vs backslashes)+    # are reproducible on each supported platform (hi Windows)+    license_files = sorted(license.as_posix() for license in licenses)     cfg['metadata']['license_files'] = '\n' + '\n'.join(license_files)-    grammar_files = sorted(os.path.join(_GRAMMAR_DIR, f) for f in grammars)-    cfg['options.data_files'][_GRAMMAR_DIR] = '\n' + '\n'.join(grammar_files)+    grammar_files = sorted(grammar.as_posix() for grammar in grammars)+    cfg['options.data_files'][_GRAMMAR_DIR.as_posix()] = '\n' + '\n'.join(grammar_files)      sio = io.StringIO()     cfg.write(sio)     new_contents = sio.getvalue().strip() + '\n'     new_contents = new_contents.replace('\t', '    ')     new_contents = new_contents.replace(' \n', '\n') -    with open('setup.cfg', 'w') as f:+    with setup_cfg_filepath.open('w', encoding='utf-8') as f:         f.write(new_contents)      return 0  -def _update(*, only: Optional[str]) -> int:-    new = []-    for repo in REPOS:-        if only is not None and repo.name != only:-            new.append(repo)-            continue+def _update_repo(repo: Repo) -> Repo:+    print(f'Updating {repo.name}... ') -        print(f'{repo.name} ... ', end='', flush=True)+    with tempfile.TemporaryDirectory() as tmpdir:+        git = ('git', '-C', tmpdir) -        with tempfile.TemporaryDirectory() as tmpdir:-            git = ('git', '-C', tmpdir)+        url = f'https://github.com/{repo.name}'+        clone_cmd = (*git, 'clone', '--no-checkout', '-q', url, '.')+        subprocess.check_call(clone_cmd) -            url = f'https://github.com/{repo.name}'-            clone_cmd = (*git, 'clone', '--no-checkout', '-q', url, '.')-            subprocess.check_call(clone_cmd)+        branches_cmd = (*git, 'branch', '-r', '--contains', repo.version)+        branches_s = subprocess.check_output(branches_cmd).decode()+        branches = [branch.strip() for branch in branches_s.splitlines()] -            branches_cmd = (*git, 'branch', '-r', '--contains', repo.version)-            branches_s = subprocess.check_output(branches_cmd).decode()-            branches = [branch.strip() for branch in branches_s.splitlines()]+        if not branches:+            raise SystemExit(f'orphaned commit {repo.name}')+        elif branches[0].startswith('origin/HEAD -> '):+            _, _, branch = branches[0].rpartition(' ')+        else:+            branch = branches[0] -            if not branches:-                raise SystemExit(f'orphaned commit {repo.name}')-            elif branches[0].startswith('origin/HEAD -> '):-                _, _, branch = branches[0].rpartition(' ')-            else:-                branch = branches[0]+        version_cmd = (+            *git, 'log', '-1', '--format=%h', branch, '--', *repo.grammars,+        )+        version = subprocess.check_output(version_cmd).decode().strip() -            version_cmd = (-                *git, 'log', '-1', '--format=%h', branch, '--', *repo.grammars,-            )-            version = subprocess.check_output(version_cmd).decode().strip()+        ancestor_cmd = (+            *git, 'merge-base', '--is-ancestor', version, repo.version,+        ) -            ancestor_cmd = (-                *git, 'merge-base', '--is-ancestor', version, repo.version,+        if (+                repo.version == 'HEAD' or+                version != repo.version and subprocess.call(ancestor_cmd)+        ):+            print(f'Updated {repo.name} {repo.version} => {version}')+            return repo._replace(version=version)++        print(f'{repo.name} is up to date!')+        return repo+++def _update(*, only: Set[str]) -> int:+    up_to_date_repos = []+    with ThreadPoolExecutor() as executor:+        futures = {}++        for repo in REPOS:+            if only and repo.name not in only:+                up_to_date_repos.append(repo)+                continue+            futures[executor.submit(_update_repo, repo)] = repo++        if not futures and only:+            print(+                'Nothing to do. Make sure you provided correct repo names to the "only" parameter',+                file=stderr             )-            if (-                    repo.version == 'HEAD' or-                    version != repo.version and subprocess.call(ancestor_cmd)-            ):-                print(f'{repo.version} => {version}')-                new.append(repo._replace(version=version))+            return 1++        for future in as_completed(futures):+            try:+                updated_repo = future.result()+            except Exception as e:+                print(f'error updating repo {futures[future]}', file=stderr)+                continue             else:-                print('up to date!')-                new.append(repo)+                up_to_date_repos.append(updated_repo) -    new.sort(key=lambda repo: repo.name)+    up_to_date_repos.sort(key=lambda repo: repo.name) -    if new != list(REPOS):-        with open(__file__) as f:+    if up_to_date_repos != list(REPOS):+        print('Repositories have changed after the update, preparing to write changes...')+        with open(__file__, encoding='utf-8') as f:             contents = f.read()          before, begin, rest = contents.partition('# BEGIN\n')         _, end, rest = rest.partition('# END\n')          new_contents = (-            'REPOS = (\n' +-            ''.join(f'{textwrap.indent(repr(r), " " * 4)},\n' for r in new) +-            ')\n'+                'REPOS = (\n' ++                ''.join(f'{textwrap.indent(repr(r), " " * 4)},\n' for r in up_to_date_repos) ++                ')\n'

Uh, sorry, I probably messed up something in my IDE.

Senpos

comment created time in 2 months

pull request commentasottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter

Hi Anthony!

Thanks for the comments! :-)

Yeah, I agree, mixing all the things into one huge PR was a mistake.

I will answer your questions anyway and close this PR as it is not unsuitable for the merge. It is a good experience for me though!

it only improved it by ~40%

It is almost x10 boost for the download command (5 seconds vs 40+ seconds). But for the update - yes, it is not necessary, I agree.

additionally I see some as_posix() which means pathlib is actually harmful to correctness here :)

While I understand that it is a personal preference either to use pathlib or not, but could you give me an insight why as_posix is harmful? The goal was to generate the same output on both Linux and Windows machines. Using the original os.path.join method script produces strings with / and \ mixed: share/babi/grammar_v1\scope.terraform.json

Thanks for your time!

Senpos

comment created time in 2 months

pull request commentasottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter

Oops, there's no cp38m-win_amd64 wheel for editdistance and I don't want to install Build Tools for Visual Studio on this machine. Will boot from my laptop tomorrow and fix it, sorry. :-)

Senpos

comment created time in 2 months

PR opened asottile/babi-grammars

[WIP] Add concurrency for "download-syntax" module & support many args for --only parameter
  • added concurrent (ThreadPoolExecutor) downloads for the _download command, had to refactor it a bit and break into smaller pieces On 4 cores, python .\bin\download-syntax download: Before: 33-47s After: 4.9-5s
  • added concurrent processing (ThreadPoolExecutor) for the _update command On 4 cores, python .\bin\download-syntax update: Before: 75-82s After: 37-39s Well, not THAT huge difference, but I suppose moby/moby repo is the bottleneck here, it takes forever to process it
  • if one concurrent task fails - log to the stderr, other tasks continue to work; we don't want to waste things we already processed :-)
  • replaced all path strings with pathlib.Path objects when possible; it allows to make reproducible results on both Linux and Windows (slashes/backslashes, uh-oh)
  • added support for passing multiple arguments to the --only parameter for both _download and _update commands. Now you can do python .\bin\download-syntax download --only MagicStack/MagicPython asottile/language-xml or python .\bin\download-syntax download --only MagicStack/MagicPython --only asottile/language-xml, both "styles" should work (warning: it is a 3.8+ feature (scroll down to the extend action)

Yes, I messed up and all the changes were added to a single commit. Sorry! If you are interested in something particular from the list - please, let me know and I will create a separate PR for it.

If it looks good enough - I may add changes to the README. Let me know what you think. :-)

P. S. I learned my lesson

+193 -110

0 comment

1 changed file

pr created time in 2 months

create barnchSenpos/babi-grammars

branch : concurrency-and-multiple-only-args

created branch time in 2 months

fork Senpos/babi-grammars

pip installable package to provide syntax grammars for babi

fork in 2 months

push eventSenpos/super-bot

Umputun

commit sha d3f0b9bbd6a512bbb4847b1f2ef1d5b550de23f2

remove logging noise

view details

Senpos

commit sha 23028c4b074f3e8536a3fff62df77359f16e37e0

fix mention in ban message if user has no username

view details

Umputun

commit sha faab73e624a1d5798db53038edaf3a587ccb7fa8

simplify mention if

view details

Maxim Kozlov

commit sha 0438c8fffc54ed6f277a13d12505adbd91fe7c4c

Повышаем покрытие #12 (#17) * Add tests for "anecdot" bot * Add tests for "news" bot * Add tests for storage * Fix build * Add tests for multibot * Add tests for "votes" bot

view details

Umputun

commit sha b7d595b09ffe414b08c1825dbfcfe766cf0c9e2a

disable vote bot

view details

Umputun

commit sha a2fe8143796a3215c328039b35995a33c65b454f

remove vote info from docs

view details

Umputun

commit sha 88d0363c7a9a8932ba6330830d9ba05251392347

add /bot commands

view details

Umputun

commit sha 253f969cad9a705ea94085d682af84f704c0724d

unify test and mock names, add missing generation

view details

Konstantin Chukhlomin

commit sha e90b324358a688a09da6d8694f2b074527d34668

Bot should not ban for old messages (#18)

view details

Umputun

commit sha 4db35ab3058633b46bed40ed1a9a43446e2daee8

minor: typos in comments, err handling in export

view details

Umputun

commit sha a758e4a49b24d77b9846e9ab1129e5a112851d1f

lint: rename HttpClient

view details

Umputun

commit sha 3a6f76f60dc57c8c215b05ce832f4125e6898a5d

missing HttpClients renamed

view details

Umputun

commit sha ab5b25425cc3512f89f0893abb4be12379b9f9a1

get rid of Fatalf inside Export

view details

Umputun

commit sha 11beb70f1a6ffc68787842c8c904c5a33070febf

initial implementation of podcast search bot #19

view details

Umputun

commit sha 9e5453fbc138c432cb022f111e592ff449486a98

fix comment on podcast bot OnMessage

view details

Umputun

commit sha e2a2df10e2d93a68395341c0797a42db12c5289e

update deps

view details

Umputun

commit sha 1519ea650b0f087b1f1d86a5df8105b435caf86a

disable DisableWebPagePreview for bot's response

view details

Umputun

commit sha 260f744fa44e9e0715bf57610f8760a1750308b9

upd help in docs

view details

Konstantin Chukhlomin

commit sha 705e9eb7d3cb61f40d10dc16699ae68f882c2574

Reporter: ignore empty text entities decorations (#26)

view details

Konstantin Chukhlomin

commit sha 570deceb4e4ec24b2e201112d5364cb77d2e73ff

Reporter: mention ≠ link, handle links w/o schema (#27) * Reporter: mention ≠ link, handle links w/o schema * Reporter: fixed error handling for url.Parse * Reporter: simplified link filter in template

view details

push time in 3 months

startedpikepdf/pikepdf

started time in 3 months

startedScille/umongo

started time in 3 months

startedRapptz/discord-ext-menus

started time in 3 months

startedfelix-hilden/tekore

started time in 3 months

more