[Feature] Allow CONCURRENT requests and Multiple Instances management; Add API authentication; and configuration improvements#225
Draft
CodePothunter wants to merge 11 commits intoremsky:masterfrom
Draft
Conversation
CodePothunter
commented
Mar 7, 2025
CodePothunter
commented
- Implement OpenAI-compatible API key authentication
- Add configuration options for GPU instances, concurrency, and request handling
- Update README with authentication instructions
- Modify configuration and routing to support optional API key verification
- Enhance system information and debug endpoints to expose authentication status.
- Implement OpenAI-compatible API key authentication - Add configuration options for GPU instances, concurrency, and request handling - Update README with authentication instructions - Modify configuration and routing to support optional API key verification - Enhance system information and debug endpoints to expose authentication status
Owner
|
This looks great, will take a look through today |
- Modify audio chunk concatenation to handle float32 audio data - Add explicit conversion from float32 to int16 using amplitude scaling - Remove unnecessary dtype specification in np.concatenate
- Create GPU-specific startup script - Set environment variables for GPU and project configuration - Use uv to install GPU extras and run FastAPI server
Collaborator
|
Is there a reason you deleted start-gpu and not start-cpu aswell |
Collaborator
Author
|
It's a misoperation I've added it back in my new commit. |
Collaborator
fireblade2534
left a comment
fireblade2534
left a comment
There was a problem hiding this comment.
As far as I can tell this pr breaks streaming. (I tested by running Test.py) It produced an empty wav file (outputstream.wav).
The reason is that when stream=True, the audio conversion functions are not really called.
Author
It has been solved in the most recent commit. The reason is that when stream=True, the audio conversion functions were not really called compared to the non-stream mode. |
Collaborator
|
When I run the docker container using the config in: It doesn't seem to respect env vars when I put them in a .env file or when I add them to the docker compose file |
Author
Sorry, I did not consider the docker-related issues. |
Collaborator
|
Also when running it in a docker container (I havn't tested this outsidee of one) running Test.py twice to generate four querys causes the container to exit with code 139 (Note that I am using the gpu container): |
…easing audio container Refactor StreamingAudioWriter to improve audio encoding reliability - Restructure audio encoding logic for better error handling - Create a new method `_create_container()` to manage container creation - Improve handling of different audio formats and encoding scenarios - Add error logging for audio chunk encoding failures - Simplify container and stream management in write_chunk method
fireblade2534
requested changes
Mar 10, 2025
Collaborator
There was a problem hiding this comment.
This pr currently breaks or removes the following features:
- The webui does not actually stream or receive any audio
- All text normalization that was in there is no longer being called
- There is no option to change the speed as it is not being passed into the generation system
- _process_chunk is in tts_service but it never gets called
- Captioned speech is broken because no timestamps are ever requested
- Streaming is broken as only the first chunk of text is returned
- Triming audio is always disabled even though it makes sense to do for chunks that contain speech
- smart_split is never called so I'm not really sure how it is suppost to split text in a sensible way
- process_text_chunk is never called
Honestly this pr feels unfinished and untested
- Update InstancePool to accept and process speed parameter - Modify TTSService to pass speed to instance pool - Update Test.py with new port and authentication - Adjust start-gpu.sh to use port 50888
Collaborator
|
Why did u change the gpu port to 50888 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.