Added micro/macro averaging option to MetricsWrapper

[hzavadil98]

In addition to #52 where now logits are passed directly to metrics, I added, as proposed, the micro/macro averaging option to be passed to each metric (except of entropy where I believe it doesn't make sense, right? @Seilmast ). Here is a pretty thorough description of the issue Micro/Macro. This resolves #30 for Johan. Please push your adjusted metrics directly to this Draft PR before we merge them.

[sot176]

I pushed my adjusted metric now to this draft PR :)

[Seilmast]

Micro/Macro averages doesn't make sense for Shannon Entropy @hzavadil98. I have added in some averaging functionality for batchwise mean or sum of entropy. As long as the option is passed as a argument then it shouldn't make any difference.

[c-salomonsen]

Yep, I'm working on mine sporadically. Will push in not too long.

Could anyone clarify to me: the input to the metric, what shapes are y_pred and y_true? Cuz if y_pred are logits sent in, I suppose they are one-hot encoded, but is y_true also that?

[Johanmkr]

Yep, I'm working on mine sporadically. Will push in not too long.

Could anyone clarify to me: the input to the metric, what shapes are y_pred and y_true? Cuz if y_pred are logits sent in, I suppose they are one-hot encoded, but is y_true also that?

I do the one-hot-encoding within the metric, but we should maybe decide on this.

[Seilmast]

Logits are not one-hot encoded. The shape you should expect is N x C where N is the batchsize and C is the number of classes.
y_true is probably just a Nx1 vector, with each row being the numerical label of the sample. That can be converted to a one-hot matrix should you need to do so.

[c-salomonsen]

Logits are not one-hot encoded. The shape you should expect is N x C where N is the batchsize and C is the number of classes.

Thats what one-hot encoding is.

y_true is probably just a Nx1 vector, with each row being the numerical label of the sample. That can be converted to a one-hot matrix should you need to do so.

Okay, good. I've noticed that this ambiguity w.r.t the shape of y_true and y_pred means that some of us expect different input shapes. Thus, I guess we should specify if the sizes you are describing is what we want to use, or if both y_pred and y_true should already have shape ($N \times C$) when sent to the MetricWrapper?

[Seilmast]

Maybe we're not refering to the same thing.
A one-hot encoded vector is a spare vector with a single value on the index representing the class.
Logits are not sparse and represents the networks map of the input onto some probability space (-inf to inf before softmax).
The logits should be the pure output of the network. It's the same as you would input into the CrossEntropyLoss function for example.

Yeah, we should specify the sized. I think working with the logits and labels in the form NxC and Nx1 is easier as

1. You usually have the labels on that form when doing training / interferance
2. You don't have to touch the logits after the network returns them

It allows us to send the input sample through the network, and just use the raw output on the metric instance.

[c-salomonsen]

I see, was mixing with the softmax output.

I think we could go for that 👍🏻

[c-salomonsen]

I just pushed my updated metric + tests to this PR. Note that the test will purposefully fail until the Precision metric is updated (see this commit message @Johanmkr)

The tests will likely need to be changed a bit to accomodate the differing input arguments to @Seilmast's entropy metric, but this works for now 😉 See here for test on metricwrapper.

[Johanmkr]

I just pushed my updated metric + tests to this PR. Note that the test will purposefully fail until the Precision metric is updated (see this commit message @Johanmkr)

The tests will likely need to be changed a bit to accomodate the differing input arguments to @Seilmast's entropy metric, but this works for now 😉 See here for test on metricwrapper.

I am resolving this now, but I am still confused about the shapes. So the logits will have shape NxC, but should we enforce y_true to have shape Nx1 rather than just N, which it seems to have in the metric wrapper test @salomaestro?

I will try to fix my code in the breaks of the ethics course today.

[c-salomonsen]

@Johanmkr as long as your inputs and outputs follow the agreed upon shape you may modify the vectors to your liking. For instance, if you want to use y_true with a shape of N, you could squeeze the dimensions using: y_true.squeeze(), which will do that for you 😉 Maybe we agree upon this shape in our meeting tomorrow?

[Johanmkr]

@Johanmkr as long as your inputs and outputs follow the agreed upon shape you may modify the vectors to your liking. For instance, if you want to use y_true with a shape of N, you could squeeze the dimensions using: y_true.squeeze(), which will do that for you 😉 Maybe we agree upon this shape in our meeting tomorrow?

Yes we can discuss during today's meeting, because it seems to me that the shape we all use for y_true in the tests is in fact (N) and not (N,1), but let's have a look later today :)

[Johanmkr]

[c-salomonsen]

@Johanmkr git push origin main --force 😆