profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/pixelb/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Pádraig Brady pixelb @facebook http://www.pixelbeat.org/ Facebook engineer, GNU coreutils maintainer

pixelb/ps_mem 1222

A utility to accurately report the in core memory usage for a program

omry/omegaconf 901

Flexible Python configuration system. The last one you will ever need.

pixelb/crudini 322

A utility for manipulating ini files

pixelb/fslint 237

Linux file system lint checker/cleaner

pixelb/dvd-vr 23

A utility to identify and optionally copy recordings from a DVD-VR format disc

dprince/firestack 18

smokestack helper tasks: packages, puppet, tests

pixelb/jsondiff 7

Diff JSON and JSON-like structures in Python

pixelb/coreutils 2

dev branches

pixelb/libs 2

libs from pixelbeat.org

pixelb/BLAKE2 1

BLAKE2 official implementations

pull request commentfacebookresearch/hydra

fix CI on 1.1_branch

Generally I would try to avoid changing versions of dependencies in release branches. That would be one of the reasons for folks following a release branch rather than trunk. So I'd pin version

Jasha10

comment created time in 15 hours

issue closedfacebookresearch/hydra

`Dict[not-str, ...]` crashes when looking for key `__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__`

🐛 Bug

Description

There is a bug in the logic to remove top-level defaults lists from structured configs. Specifically, if a DictConfig instance has key_type that is typed as something like int or as a subclass of Enum, then the defaults list logic raises a ConfigCompositionException when getattr(cfg, "__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__") is called.

To reproduce

** Minimal Code/Config snippet to reproduce ** Here is app.py:

from enum import Enum
from dataclasses import dataclass, field
from typing import Dict
import hydra
from hydra.core.config_store import ConfigStore

cs = ConfigStore.instance()

class Category(Enum):
    X = 0
    Y = 1
    Z = 2

@dataclass
class Conf:
    enum_dict: Dict[Category, str] = field(default_factory=dict)
    int_dict: Dict[int, str] = field(default_factory=dict)
    str_dict: Dict[str, str] = field(default_factory=dict)

cs.store(name="conf", node=Conf)

@hydra.main(config_path=None, config_name="conf")
def app(cfg):
    assert cfg == {"enum_dict": {}, "int_dict": {}, "str_dict": {}}

if __name__ == "__main__":
    app()

Here is the output from running app.py:

$ HYDRA_FULL_ERROR=1 python app.py
Traceback (most recent call last):
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 140, in load_configuration
    return self._load_configuration_impl(
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 248, in _load_configuration_impl
    cfg = self._compose_config_from_defaults_list(
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 537, in _compose_config_from_defaults_list
    strip_defaults(cfg)
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 535, in strip_defaults
    strip_defaults(value)
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 530, in strip_defaults
    if getattr(cfg, "__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__", False):
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 357, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 457, in _get_node
    key = self._validate_and_normalize_key(key)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 275, in _validate_and_normalize_key
    return self._s_validate_and_normalize_key(self._metadata.key_type, key)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 300, in _s_validate_and_normalize_key
    raise KeyValidationError(
omegaconf.errors.KeyValidationError: Key '__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__' is incompatible with the enum type 'Category', valid: [X, Y, Z]
    full_key: enum_dict.__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__
    reference_type=Dict[Category, str]
    object_type=dict

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jasha10/hydra_tmp/D30996424/app.py", line 27, in <module>
    app()
  File "/home/jasha10/hydra.git/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/home/jasha10/hydra.git/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/jasha10/hydra.git/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/jasha10/hydra.git/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/jasha10/hydra.git/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/home/jasha10/hydra.git/hydra/_internal/hydra.py", line 88, in run
    cfg = self.compose_config(
  File "/home/jasha10/hydra.git/hydra/_internal/hydra.py", line 559, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 147, in load_configuration
    raise ConfigCompositionException().with_traceback(sys.exc_info()[2]) from e
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 140, in load_configuration
    return self._load_configuration_impl(
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 248, in _load_configuration_impl
    cfg = self._compose_config_from_defaults_list(
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 537, in _compose_config_from_defaults_list
    strip_defaults(cfg)
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 535, in strip_defaults
    strip_defaults(value)
  File "/home/jasha10/hydra.git/hydra/_internal/config_loader_impl.py", line 530, in strip_defaults
    if getattr(cfg, "__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__", False):
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 357, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/_utils.py", line 821, in format_and_raise
    _raise(ex, cause)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/_utils.py", line 719, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 457, in _get_node
    key = self._validate_and_normalize_key(key)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 275, in _validate_and_normalize_key
    return self._s_validate_and_normalize_key(self._metadata.key_type, key)
  File "/home/jasha10/miniconda3/envs/hydra/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 300, in _s_validate_and_normalize_key
    raise KeyValidationError(
hydra.errors.ConfigCompositionException

Expected Behavior

The above repro should not crash, and the assertion in they python code should pass.

System information

  • Hydra Version : main branch
  • Python version : 3.9

closed time in 2 days

Jasha10

pull request commentfacebookresearch/hydra

fix CI on 1.1_branch

wrong commit I think

Jasha10

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventomry/omegaconf

Pádraig Brady

commit sha a6a6ba2bc1e8b208e6417847cf73a5b2abddded9

clarify message re OC_CAUSE=1 env var (#806) * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 3 days

PR merged omry/omegaconf

clarify message re OC_CAUSE=1 env var
  • omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace
+1 -1

2 comments

1 changed file

pixelb

pr closed time in 3 days

push eventpixelb/omegaconf

Jasha10

commit sha fdb78fa144b70d15b2d12b4aaeb19e27088b70a0

Compatibility with pip version 21.3.0 (#800)

view details

Jasha10

commit sha 055eef2e22fa3218af3099cda43598d8a532b157

Interpolation grammar: allow pipe `|` in unquoted strings (#799)

view details

Jasha10

commit sha ef6b457b7db96c4db2b620c4d3e2a74ea273d26a

Support for python 3.10 (#808)

view details

Pádraig Brady

commit sha c915ced297acd752113b408194f2726264962d1d

clarify message re OC_CAUSE=1 env var * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 3 days

PullRequestReviewEvent

Pull request review commentfacebookresearch/hydra

Fix CI

 def load_config(self, config_path: str) -> ConfigResult:      def available(self) -> bool:         try:-            ret = resources.is_resource(self.path, "__init__.py")  # type:ignore+            ret = any(+                f.name == "__init__.py" and f.is_file() for f in resources.files(self.path).iterdir()  # type: ignore+            )             assert isinstance(ret, bool)             return ret-        except ValueError:-            return False-        except ModuleNotFoundError:+        except Exception:

Is it better then to catch (ValueError, ModuleNotFoundError, TypeError) ?

Jasha10

comment created time in 3 days

PullRequestReviewEvent

delete branch pixelb/hydra

delete branch : i-1834

delete time in 4 days

push eventpixelb/omegaconf

Pádraig Brady

commit sha 6e2462d0d18b17e422d99bc0585126a0132da0cd

clarify message re OC_CAUSE=1 env var * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 4 days

pull request commentomry/omegaconf

clarify message re OC_CAUSE=1 env var

Even though this is a comment, it's shown in the truncated backtrace and so should give accurate info

pixelb

comment created time in 4 days

PR opened omry/omegaconf

clarify message re OC_CAUSE=1 env var
  • omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace
+1 -1

0 comment

1 changed file

pr created time in 4 days

create barnchpixelb/omegaconf

branch : OC_CAUSE_message

created branch time in 4 days

push eventpixelb/omegaconf

push time in 4 days

push eventpixelb/omegaconf

Pádraig Brady

commit sha 2f606b6f25b838f1622010cd883d434600eb01bc

clarify message re OC_CAUSE=1 env var * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 4 days

push eventpixelb/omegaconf

push time in 4 days

push eventpixelb/omegaconf

Pádraig Brady

commit sha 2f606b6f25b838f1622010cd883d434600eb01bc

clarify message re OC_CAUSE=1 env var * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 4 days

push eventpixelb/omegaconf

push time in 4 days

push eventpixelb/omegaconf

Pádraig Brady

commit sha 2f606b6f25b838f1622010cd883d434600eb01bc

clarify message re OC_CAUSE=1 env var * omegaconf/_utils.py: Fix typo, and clarify that one needs to set the environment variable to get the full backtrace

view details

push time in 4 days

fork pixelb/omegaconf

Flexible Python configuration system. The last one you will ever need.

fork in 4 days

push eventfacebookresearch/hydra

Pádraig Brady

commit sha 03fe5a397f8fe142471df4ee4781aff54e912336

Fix recent invalid processing of structured configs (#1839) Fixes issue #1834 caused by commit 7435e5739 We can't insert an aribtrary "__REMOVE_TOP_LEVEL_DEFAULTS__" key in the config, as that will fail various base OmegaConf checks, due to not being a valid enum etc, giving errors like: Key '__HYDRA_REMOVE_TOP_LEVEL_DEFAULTS__' is incompatible with the enum type ... Instead we set that flag using _set_flag on the cfg object. This was tested with: pytest tests/test_compose.py -k \ 'test_missing_node_with_defaults_list or test_enum_with_removed_defaults_list'

view details

push time in 7 days

issue commentomry/omegaconf

Value 'float64' is not a supported primitive type

Well float64 and python float are c doubles underneath, so one should not loose info by converting to/from python floats, like:

d = OmegaConf.create(dict(a=np.float64(3.2).item()))

Alternatively if you didn't mind conversion overhead but wanted an intermediate string representation, one could use hex float strings like:

d = OmegaConf.create(dict(a=np.float64(3.2).hex()))

joamatab

comment created time in 8 days

PullRequestReviewEvent

create barnchpixelb/vissl

branch : export-D31557803

created branch time in 11 days

PR opened facebookresearch/vissl

VISSL: make hydra version checking more robust

Summary: Upstream hydra now is at "1.2.0dev1" which the last component throws an exception with the current parsing scheme. Here we restrict to the numeric portion which should suffice. Another option may be to depend on packaging.version.parse().

Differential Revision: D31557803

+2 -1

0 comment

1 changed file

pr created time in 11 days

fork pixelb/vissl

VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.

https://vissl.ai

fork in 11 days