Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 7 additions & 12 deletions doc/Magnus_page.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,24 +26,19 @@ Each input is flattened over the channel, height and width channels. Then they a


## Entropy Metric In-Depth

The EntropyPrediction class' main job is to take some inputs and return the Shannon Entropy metric of those inputs. The class has four methods with the following jobs:
The EntropyPrediction class' main job is to take some inputs from the MetricWrapper class and store the batchwise Shannon Entropy metric of those inputs. The class has four methods with the following jobs:
* __init__ : Initialize the class.
* __call__ : Main method which is used to calculate and store the batchwise shannon entropy.
* __returnmetric__ : Returns the collected metric.
* __reset__ : Removes all the stored values up until that point. Readies the instance for storing values from a new epoch.

The class is initialized with a single parameter called "averages". This is inspired from other PyTorch and NumPy implementations and controlls how values from different batches or within batches will be combined. The __init__ method checks the value of this argument with an assertion, which must be one of three string. We only allow "mean", "sum" and "none" as methods of combining the different entropy values. We'll come back to the specifics here.
Furthermore, this method will also store the different Shannon Entropy values as we pass values into the __call__ method.
The __init__ method has two arguments, both present for compatability issues. However, the num_classes argument is used as a check in the __call__ method to assert the input is of correctly assumed size.

In __call__ we get both true labels and model logit scores for each sample in the batch as input. We're calculating Shannon Entropy, not KL-divergence, so the true labels aren't actually needed.
With permission I've used the scipy implementation to calculate entropy here. We apply a softmax over the logit values, then calculate the Shannon Entropy, and make sure to remove any Inf values which might arise from a perfect guess/distribution.


In __call__ we get both true labels and model logit scores for each sample in the batch as input. We're calculating Shannon Entropy, not KL-divergence, so the true labels aren't needed.
With permission I've used the scipy implementation to calculate entropy here. We apply a softmax over the logit values, then calculate the Shannon Entropy, and make sure to remove any NaN or Inf values which might arise from a perfect guess/distribution.

Next we have the __returnmetric__ method which is used to retrive the stored metric. Here the averages argument comes into play.
Depending on what has been chosen as the averaging metric when initializing the class, one of the following operations will be applied to the stored values:
* Mean: Calculate the mean of the stored entropy values.
* Sum: Sum the stored entropy values.
* None: Do nothing with the stored entropy values.
Then the value(s) are returned.
Next we have the __returnmetric__ method which is used to retrive the stored metric. This returns the mean over all stored values. Effectively, this will return the average Shannon Entropy of the dataset.

Lastly we have the __reset__ method which simply emptied the variable which stores the entropy values to prepare it for the next epoch.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name = "collaborative-coding-exam"
version = "0.1.0"
description = "Exam project in the collaborative coding course."
readme = "README.md"
requires-python = ">=3.12"
requires-python = ">=3.11.5"
dependencies = [
"black>=25.1.0",
"h5py>=3.12.1",
Expand Down