[FEATURE] bottlenecks in WaveNet layers #185
Merged
+340
−68
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…bottleneck parameter - Modified the constructors of _Layer and _LayerArray to accept a new bottleneck parameter, enhancing the flexibility of the layer configurations. - Updated relevant method calls and test cases to reflect the new parameter, ensuring consistency across the codebase. - Adjusted JSON configuration handling to support the bottleneck parameter, maintaining backward compatibility with existing configurations.
- Added a bottleneck parameter to the constructors of _Layer and _LayerArray in various test cases, enhancing the flexibility of layer configurations. - Implemented new test cases for layers with bottleneck configurations, including both gated and non-gated scenarios. - Updated existing tests to utilize the bottleneck parameter, ensuring comprehensive coverage and consistency across the codebase.
… initialization comments - Commented out tests for bottleneck and gated bottleneck layers in run_tests.cpp while investigating a resize error. - Updated weight initialization logic in test_layer.cpp to clarify the layout for Conv1D and 1x1 convolutions, ensuring consistency with the new bottleneck parameter. - Adjusted comments for better clarity on weight patterns and dimensions in the test cases.
- Adjusted the WaveNet Layer's SetMaxBufferSize and Process methods to correctly use the bottleneck parameter for resizing internal buffers. - Updated the handling of activation functions to ensure they operate on the correct number of channels based on the bottleneck. - Modified test cases to reflect changes in the Layer constructor and ensure proper functionality with the bottleneck configuration. - Enhanced comments for clarity regarding the internal channel structure and weight initialization in tests.
- Modified the headInput matrix resizing in test_layer.cpp to utilize the bottleneck parameter instead of channels, ensuring alignment with recent changes in the WaveNet layer configuration. - This adjustment enhances the accuracy of the test cases by reflecting the updated architecture that incorporates the bottleneck parameter.
- Introduced a new test case, test_layer_bottleneck_process_realtime_safe(), to validate that the Layer::Process() method operates correctly when the bottleneck parameter differs from the number of channels. - Ensured that the test checks for memory allocation during processing, maintaining real-time safety. - Updated run_tests.cpp to include this new test, enhancing coverage for bottleneck scenarios in the WaveNet layer.
sdatkinson
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a configurable
bottleneckparameter toWaveNet::_Layerthat allows the intermediate number of channels (going into the activation) to differ from the input/output channels. This resolves issue #170.Changes
Core Implementation
bottleneckparameter to_Layerconstructor_bottleneckmember variable to store the actual bottleneck value (not doubled when gated)_Layer::SetMaxBufferSize()to correctly size_zand_output_headbased on bottleneck_Layer::Process()to use_bottleneckinstead ofchannelsfor internal operations_output_headto havebottleneckrows instead ofchannelsrowsLayerArray Updates
bottleneckparameter toLayerArrayParamsconstructor and member variable_LayerArrayconstructor to accept and passbottleneckto layers_head_rechannelto takebottleneckinput channels instead ofchannels_head_inputsto be resized tobottleneckrowsFactory Function
bottleneckfrom JSON configchannelsifbottleneckis not specified (maintains backward compatibility)Tests
bottleneckparameter (using named variables for legibility)test_layer_bottleneck()- tests bottleneck < channels for non-gated layerstest_layer_bottleneck_gated()- tests bottleneck with gated activationbottleneckrows for head outputsBug Fixes
NDEBUG(was using-U_NDEBUGinstead of-UNDEBUG)Testing
All tests pass, including the new bottleneck-specific tests.
Backward Compatibility
The factory function defaults
bottlenecktochannelswhen not specified in JSON, ensuring existing models continue to work without modification.Resolves #170