profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/NarineK/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

NarineK/captum-1 1

Model interpretability and understanding for PyTorch

NarineK/scikit-learn 1

scikit-learn: machine learning in Python

NarineK/spark 1

Mirror of Apache Spark

NarineK/analyzing-redundancy-in-pretrained-transformer-models 0

Code for Analyzing Redundancy in Pretrained Transformer Models accepted at EMNLP 2020

NarineK/breeze 0

Breeze is a numerical processing library for Scala.

NarineK/DeepExplain 0

A unified framework of perturbation and gradient-based attribution methods for Deep Neural Networks interpretability. DeepExplain also includes support for Shapley Values sampling. (ICLR 2018)

NarineK/Fairness-in-Classification-and-Representation-Learning 0

Implementation of Fairness method for Classification Task

pull request commentpytorch/captum

Lrp tutorial added

@JohannesK14 would you have time to address the comments in this PR as discussed previously ?

JohannesK14

comment created time in 2 days

issue commentpytorch/captum

Error while using baselines for GradientShap

@pupul07, GradientShap chooses a point between input and baseline and it assumes that the first dimension of your input is batch size. rand_coefficient assumes that your first dimension is batch size as well. In order to find a workaround you can define a custom forward function takes inputs transposed the way so that it has first dimension as batch size. In the custom forward function you can transpose it back and call your model:

# input dims: (N, 257), (N ,12, 73) and (1, N, 36)
def my_forward_func(inp1, inp2, inp2): 
   return model(inp1, inp2, inp3.view(inp3.shape[1], inp3.shape[0], inp3.shape[2]))

Call attribute with something like this:

gs.attribute(inp1, inp2, inp3.view(inp3.shape[1], inp3.shape[0], inp3.shape[2]), base1, base2, base3.base3(base3.shape[1], base3.shape[0], base3.shape[2]))

Let me know if this worked for you.

pupul07

comment created time in 2 days

issue commentpytorch/captum

Implementing captum with pytorch-lightning

Tagging same question on pytorch forums. https://discuss.pytorch.org/t/implementing-captum-with-pytorch-lightning/129292

ik362

comment created time in 3 days

pull request commentpytorch/captum

change TCAV to use captum/_utils/fb/av.py

@99warriors, it looks like some of the tests and linter failed. Do you mind looking into them ?

99warriors

comment created time in 21 days

PR closed pytorch/captum

Reviewers
Avoid forward_hook under DataParallel cla signed

Address same problem in the VQA Insights tutorial reported in issue #469

+8 -17

4 comments

1 changed file

bilalsal

pr closed time in 24 days

pull request commentpytorch/captum

Avoid forward_hook under DataParallel

Closing it, since the issue got addressed

bilalsal

comment created time in 24 days

PullRequestReviewEvent
PullRequestReviewEvent

PR opened pytorch/captum

Add concept and robust tutorials to the website

Add concept and robust tutorials to the website. In addition to that also did minor fixes in previous tutorial descriptions.

+32 -2

0 comment

2 changed files

pr created time in a month

create barnchNarineK/captum-1

branch : add_tutorials_to_website

created branch time in a month

push eventNarineK/captum-1

Narine Kokhlikyan

commit sha f1346fc32a1f191f2c9a9401428372f68a88aec1

Use large model for IMDB and fix minor issues (#709) Summary: allow-large-files Fixes the issues raised in https://github.com/pytorch/captum/issues/653 and switches to a larger imdb pre-trained model. Smaller imdb model was trained on text with fixed length 7, therefore the model doesn't perform well on text with larger vocabulary and this was evident when applying TCAV on IMDB. Pull Request resolved: https://github.com/pytorch/captum/pull/709 Reviewed By: bilalsal Differential Revision: D29601001 Pulled By: NarineK fbshipit-source-id: 9db3f425855847eaf897cd267b483091cd8db833

view details

Hanno Küpers

commit sha d201bc4ec122faacf669370860f371fa8d5c1736

Enhance error message for missing rule (#727) Summary: This PR provides additional information to an error message that is triggered when neither an explicit LRP rule is defined nor a default rule exists. The current error message is quite short and does not give enough information to the user on how to solve the problem as discussed in issue https://github.com/pytorch/captum/issues/723 . Feel free to propose a different wording. Pull Request resolved: https://github.com/pytorch/captum/pull/727 Reviewed By: aobo-y Differential Revision: D30045186 Pulled By: vivekmig fbshipit-source-id: 2dc7f7a29d014da12ebdc409d4abc676d5cbbc34

view details

Narine Kokhlikyan

commit sha 3a697e329d4a8ee3d52f428b51d0d4bbf152a086

Support TCAV on cuda (#725) Summary: Addresses the issues: https://github.com/pytorch/captum/issues/721 #719 https://github.com/pytorch/captum/issues/720 Pull Request resolved: https://github.com/pytorch/captum/pull/725 Reviewed By: bilalsal Differential Revision: D30356015 Pulled By: NarineK fbshipit-source-id: 010a5263bdfc33e8c4d3f9de523d9d3ba3969f49

view details

Zsolt Dollenstein

commit sha 714fd4a9d211ca8cb2ab9fd038aef4d0a7f10577

Enable BLACK by default Reviewed By: zertosh Differential Revision: D30368068 fbshipit-source-id: c470d33da8115865bdeaa15ef66f7152b2e3ca3c

view details

Reuben Dunn

commit sha e222265dff147710e28b97a6ac5ac08863b27e2c

Fix empty data loading in Insights (#728) Summary: Previously, when Captum Insights had already shown all existing batches from the dataset, it would stop showing data at all after fetches. This change makes it recycle some batches, so that the user can keep clicking on "Fetch Data" to see the changes that their setting have on the results. This was reported in https://github.com/pytorch/captum/issues/686 Pull Request resolved: https://github.com/pytorch/captum/pull/728 Test Plan: 1. Replace `captum/insights/attr_vis/app.py` with P434861588 2. Open Captum Insights with `python3 -m captum.insights.example` 3. Click "Fetch Data" twice 4. You can test before and after this diff to confirm that it fixes the issue Reviewed By: edward-io Differential Revision: D30059290 Pulled By: Reubend fbshipit-source-id: 0c7d8d081df037096e16098b1217a6002f184a7e

view details

Vivek Miglani

commit sha 406b317029b59c6ef152f3a1cbcecff87f959f0e

Adding Captum Robustness Tutorial Summary: Adds tutorial for Captum Robustness tools using CIFAR for a basic image classification task. First demonstrates FGSM and PGD, similar to existing tutorial, and then demonstrates usage of robustness metrics, attack comparator and minimal perturbation Reviewed By: NarineK, aobo-y Differential Revision: D29123003 fbshipit-source-id: f841028c5f0d4f374f125af2417afb782ba1d6c8

view details

Narine Kokhlikyan

commit sha 41547a9b6f070afa3c73be28981fe7b5b59e8c30

Open source TCAV Tutorials (#736) Summary: Pull Request resolved: https://github.com/pytorch/captum/pull/736 Open source TCAV Tutorials. Both image and text Reviewed By: vivekmig Differential Revision: D30227918 fbshipit-source-id: 06d8515cdcc97ed56392ad8a25a270fc814fa833

view details

push time in a month

create barnchNarineK/captum-1

branch : export-D30227918

created branch time in a month

PR opened pytorch/captum

Open source TCAV Tutorials

Summary: Open source TCAV Tutorials. Both image and text

Reviewed By: vivekmig

Differential Revision: D30227918

+2410 -0

0 comment

10 changed files

pr created time in a month

issue closedpytorch/captum

TCAV: cannot run compute_cavs() in cuda

🐛 Bug

When using a model in the GPU, when no cav vectors are precomputed (i.e. the ./cav folder has not been created), and when using the default classifier for TCAV() (i.e. passing classifier=None), running tcav.compute_cavs() throws a wrong-device error.

To Reproduce

Steps to reproduce the behavior:

  1. Remove precomputed cav vectors (if any), i.e. the ./cav folder
  2. Run the following code
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader
from captum.concept import TCAV, Concept

DEVICE = 'cuda'

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 10, 10)
        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.flatten = nn.Flatten()
        self.classifier = nn.Linear(10, 1)
    def forward(self, images):
        # images shape: batch_size, 3, height, width
        x = self.conv(images) # shape: batch_size, 10, features-height, features-width
        x = self.pool(x) # shape: batch_size, 10, 1, 1
        x = self.flatten(x) # shape: batch_size, 10
        x = self.classifier(x) # shape: batch_size, 1
        return x

class DummyDataset(Dataset):
    def __init__(self, device='cpu'):
        super().__init__()
        self.device = device
    def __getitem__(self, idx):
        image = torch.zeros(3, 256, 256)
        return image.to(self.device)
    def __len__(self):
        return 10

model = MyModel().to(DEVICE)

concept0 = Concept(0, 'concept0', DataLoader(DummyDataset(device=DEVICE), batch_size=10))
concept1 = Concept(1, 'concept1', DataLoader(DummyDataset(device=DEVICE), batch_size=10))

tcav = TCAV(model, layers='conv')

cavs = tcav.compute_cavs([[concept0, concept1]])
  1. Running the tcav.compute_cavs(...) line throws:
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

See the full stack here

Expected behavior

The method compute_cavs() should run without errors, in any of CPU or GPU (or the docs should state that only CPU is supported?)

Environment

Describe the environment used for Captum

  • Captum / PyTorch Version: captum 0.4.0, torch 1.7.1+cu110
  • OS (e.g., Linux): Ubuntu 18.04.5
  • How you installed Captum / PyTorch (conda, pip, source): source
  • Build command you used (if compiling from source): pip install -e ~/software/captum
  • Python version: 3.6
  • CUDA/cuDNN version: cu110
  • GPU models and configuration: using a GPU RTX 3090
  • Any other relevant information: I'm running captum in the master branch, latest commit is f658185

Additional context

  • If the device is cpu (i.e. use DEVICE = 'cpu' in the code above) the code runs without error, and saves the CAV vectors to the ./cav folder
  • If the CAV vectors are precomputed, the code runs without error (in GPU and CPU)

Possible culprit

        # ...
        self.lm.fit(DataLoader(TensorDataset(x_train, y_train)))

        self.lm.linear.to(x_test.device) # Add this

        predict = self.lm(x_test)

        predict = self.lm.classes()[torch.argmax(predict, dim=1)]
        predict = predict.to(x_test.device) # Add this
        # ...
  • Probably this is not an optimal solution, I'm not familiar enough with the codebase to know better or to further inspect this :grin:

Btw, TCAV is an awesome feature! :rocket:

closed time in a month

pdpino

issue commentpytorch/captum

TCAV: cannot run compute_cavs() in cuda

fixed with #725

pdpino

comment created time in a month

issue closedpytorch/captum

Docs: TCAV - Concept, specify dataloader yield type

📚 Documentation

For the captum.concept.Concept class, the docs should state what type of object the data_iter Dataloader must yield. I think it has to yield inputs only (i.e. without labels), by seeing this line from TCAV().

closed time in a month

pdpino

issue commentpytorch/captum

Docs: TCAV - Concept, specify dataloader yield type

fixed with #725

pdpino

comment created time in a month

issue closedpytorch/captum

TCAV: cannot run interpret() in cuda

🐛 Bug

When using a model in the GPU, running tcav.interpret(...) throws a wrong-device error.

To Reproduce

Steps to reproduce the behavior:

  1. Work-around the #719 bug by running tcav.compute_cavs() in the CPU, which will save CAV vectors to ./cav
  2. Run the following code:
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader
from captum.concept import TCAV, Concept

DEVICE = 'cuda'

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 10, 10)
        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.flatten = nn.Flatten()
        self.classifier = nn.Linear(10, 1)
    def forward(self, images):
        # images shape: batch_size, 3, height, width
        x = self.conv(images) # shape: batch_size, 10, features-height, features-width
        x = self.pool(x) # shape: batch_size, 10, 1, 1
        x = self.flatten(x) # shape: batch_size, 10
        x = self.classifier(x) # shape: batch_size, 1
        return x

class DummyDataset(Dataset):
    def __init__(self, device='cpu'):
        super().__init__()
        self.device = device
    def __getitem__(self, idx):
        image = torch.zeros(3, 256, 256)
        return image.to(self.device)
    def __len__(self):
        return 10

model = MyModel().to(DEVICE)

concept0 = Concept(0, 'concept0', DataLoader(DummyDataset(device=DEVICE), batch_size=10))
concept1 = Concept(1, 'concept1', DataLoader(DummyDataset(device=DEVICE), batch_size=10))

tcav = TCAV(model, layers='conv')

inputs = torch.rand(7, 3, 256, 256).to(DEVICE)
scores = tcav.interpret(inputs, [[concept0, concept1]])

The tcav.interpret(...) line throws: RuntimeError: Input, output and indices must be on the current device. The full stack:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-6775c2780feb> in <module>
     39 
     40 inputs = torch.rand(7, 3, 256, 256).to(DEVICE)
---> 41 scores = tcav.interpret(inputs, [[concept0, concept1]])

~/software/captum/captum/concept/_core/tcav.py in interpret(self, inputs, experimental_sets, target, additional_forward_args, processes, **kwargs)
    597                     cav_subset,
    598                     classes_subset,
--> 599                     experimental_subset_sorted,
    600                 )
    601                 i += 1

~/software/captum/captum/concept/_core/tcav.py in _tcav_sub_computation(self, scores, layer, attribs, cavs, classes, experimental_sets)
    646             scores[concepts_key][layer] = {
    647                 "sign_count": torch.index_select(
--> 648                     sign_count_score[i, :], dim=0, index=new_ord
    649                 ),
    650                 "magnitude": torch.index_select(

RuntimeError: Input, output and indices must be on the current device

Expected behavior

The method interpret() should run without errors, in any of CPU or GPU (or the docs should state that only CPU is supported?)

Environment

  • Captum / PyTorch Version: captum 0.4.0, torch 1.7.1+cu110
  • OS (e.g., Linux): Ubuntu 18.04.5
  • How you installed Captum / PyTorch (conda, pip, source): source
  • Build command you used (if compiling from source): pip install -e ~/software/captum
  • Python version: 3.6
  • CUDA/cuDNN version: cu110
  • GPU models and configuration: using a GPU RTX 3090
  • Any other relevant information: I'm running captum in the master branch, latest commit is f658185

Additional context

  • I was able to hot-fix it by changing this line in the TCAV()._tcav_sub_computation() method to:
new_ord = torch.tensor([concept_ord[cls] for cls in cls_set], device=sign_count_score.device)

i.e. specifying the device param

closed time in a month

pdpino

issue commentpytorch/captum

TCAV: cannot run interpret() in cuda

fixed with #725

pdpino

comment created time in a month

pull request commentpytorch/captum

Optim-wip: Add Activation Atlas setup tutorial

@ProGamerGov, thank you for the replies. Let me know if you create a new PR with cleaned commit history. I'll take a look into it.

ProGamerGov

comment created time in a month

Pull request review commentpytorch/captum

Support TCAV on cuda

 def sklearn_train_linear_model(         if hasattr(sklearn_model, "classes_")         else None     )++    # extract model device+    device = model.device if hasattr(model, "device") else "cpu"+     num_outputs = sklearn_model.coef_.shape[0] if sklearn_model.coef_.ndim > 1 else 1-    weight_values = torch.FloatTensor(sklearn_model.coef_)  # type: ignore-    bias_values = torch.FloatTensor([sklearn_model.intercept_])  # type: ignore+    weight_values = torch.FloatTensor(sklearn_model.coef_).to(device)  # type: ignore+    bias_values = torch.FloatTensor([sklearn_model.intercept_]).to(  # type: ignore+        device  # type: ignore+    )  # type: ignore

I tried to do for the first or last line but circle CI was complaining and I ended up adding # type: ignore in all three lines.

NarineK

comment created time in a month

PullRequestReviewEvent

Pull request review commentpytorch/captum

Optim-wip: Add Activation Atlas setup tutorial

 class type to replace in the model.         layers = cast(List[Type[nn.Module]], layers)         for target_layer in layers:             replace_layers(model, target_layer, SkipLayer)+++class MaxPool2dRelaxed(torch.nn.Module):+    """+    A relaxed pooling layer, that's useful for calculating attributions of spatial+    positions. This layer reduces Noise in the gradient through the use of a+    continuous relaxation of the gradient.++    Args:+        kernel_size (int or tuple of int): The size of the window to perform max &+            average pooling with.+        stride (int or tuple of int, optional): The stride window size to use.+        padding (int or tuple of int): The amount of zero padding to add to both sides+            in the nn.MaxPool2d & nn.AvgPool2d modules.+        ceil_mode (bool, optional): Whether to use ceil or floor for creating the+            output shape.+    """++    def __init__(+        self,+        kernel_size: Union[int, Tuple[int, ...]],+        stride: Optional[Union[int, Tuple[int, ...]]] = None,+        padding: Union[int, Tuple[int, ...]] = 0,+        ceil_mode: bool = False,+    ) -> None:+        super().__init__()+        self.maxpool = torch.nn.MaxPool2d(+            kernel_size=kernel_size, stride=stride, padding=padding, ceil_mode=ceil_mode+        )+        self.avgpool = torch.nn.AvgPool2d(+            kernel_size=kernel_size, stride=stride, padding=padding, ceil_mode=ceil_mode+        )++    def forward(self, x: torch.Tensor) -> torch.Tensor:+        return self.maxpool(x.detach()) + self.avgpool(x) - self.avgpool(x.detach())

This part is a bit unclear. One time we add and then subtract avgpool for the detached input ?

ProGamerGov

comment created time in a month

Pull request review commentpytorch/captum

Optim-wip: Add Activation Atlas setup tutorial

 class type to replace in the model.         layers = cast(List[Type[nn.Module]], layers)         for target_layer in layers:             replace_layers(model, target_layer, SkipLayer)+++class MaxPool2dRelaxed(torch.nn.Module):+    """+    A relaxed pooling layer, that's useful for calculating attributions of spatial+    positions. This layer reduces Noise in the gradient through the use of a+    continuous relaxation of the gradient.+

Can we perhaps cite it with a reference to the code in lucid ? Gradient of what ? Of output with respect to this layer ?

ProGamerGov

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

issue commentpytorch/captum

Ignore None input

@R-N, for the forward function in the described , scenario you can use a wrapper as you mentioned. When you mention unused input in the layers. Which algorithm are you using that you are having an issue ? A layer attribution method ?

R-N

comment created time in a month

issue commentpytorch/captum

Ignore None input

@R-N, if I understand correctly you can just pass those arguments as additional_forward_arg. additional_forward_arg are passed to forward function but gradients w.r.t. those aren't computed. Here is a usage of additional_forward_args https://captum.ai/tutorials/Bert_SQUAD_Interpret Have you tried it ?

R-N

comment created time in a month

pull request commentpytorch/captum

Optim-wip: Add Activation Atlas setup tutorial

Thank you for splitting this PRs, @ProGamerGov. It looks like new PRs show 95 other commits in the commit history. Is it possible to clean commit history, squash it into one commit and have the longer commits to be associated only with the original PR ?

  1. For the notebook Collecting Samples for Activation Atlases with captum.optim do you mind pointing to the notebook in lucid? Activation atlas has multiple notebooks and it is unclear which one does this notebook refer to.
  2. I think I'm not ver clear why By default the activation samples will not have the right class attributions. What do we mean by attribution here ? I think that it would be great to define and describe it in the notebook.
  3. Are attributions activations for the modified network ? Why are we enabling the gradients if we are computing only the activations ?
with torch.set_grad_enabled(True):
                    target_activ_attr_dict = opt.models.collect_activations(
                        attr_model, attr_targets, inputs
                    )
        ```
4. I think that you might want to describe how is the attribution computed using input * gradients and why are we calling autograd twice. We need to describe the trick that we are using here and reference to the documentation otherwise it's unclear if we try to understand it from the tutorial.
5. It looks like in the `attribute_samples` function we are saving activation and attribution vectors in a separate file. Wouldn't it be better to concatenate them in the memory and save them all together in one file instead of concatenating them later in `consolidate_samples` function ?
6. In the `consolidate_samples` function when you mention `[n_channels, n_samples].` with `n_channels` do you mean the number of target classes ?
7. I looked into the tutorial and tutorial related changes but it looks like there is a lot more code in this PR that is not used in the tutorial and there are same copies in PR #730.  For example `AngledNeuronDirection` is defined in both PR. I think that the changes that aren't related to this tutorial we can leave in one place. e.g. in PR#730 
ProGamerGov

comment created time in a month