non-streaming create_turn still streams under the covers

[thoraxe]

System Info

N/A

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

A PR was merged for supporting "non-streaming," however, its code still relies on _create_turn_streaming under the covers:

0be2852#diff-b38e3fcdf146b8d15b63f07ee0a4b6f0080d202da0381373bac8e8edf289f5c1R81-R93

This was from #104

It doesn't look like this "supports" non-streaming, other than not breaking if stream=False is set on the create_turn.

Also note that because of llamastack/llama-stack#1045 it may be impossible for community members to fix this.

Error logs

N/A

Expected behavior

create_turn when stream=false should not use streaming on the provider request.

[terrytangyuan]

cc @ehhuang

[ehhuang]

Yea we implemented this on the client side, cc @ashwinb. What is the issue you're seeing with this?

[ashwinb]

We did this because we don't believe supporting extremely long request on the server side over HTTP (as an agent turn is likely to be, far more than inference) does not seem like a good idea. Any decent web-server will time out.

[thoraxe]

Can you show where you have tested and come up with a scenario where an inference server doesn't respond within the HTTP request timeout window?

I've not done extensive testing with other frameworks, but I've never run into this problem.

[thoraxe]

@ashwinb do you have any existing scenarios that prove that this timeout will be a problem?

[terrytangyuan]

I wonder if we should support both scenario regardless of the reality but have streaming as the default. Otherwise, the stream parameter is not effective and can be misleading.