^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m [I 2025-06-06 14:32:54,777] Evaluating flow with config: {'few_shot_enabled': False, 'rag_mode': 'sub_question_rag', 'template_name': 'CoT', 'response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B', 'rag_method': 'sparse', 'rag_query_decomposition_enabled': False, 'rag_top_k': 1, 'splitter_method': 'recursive', 'splitter_chunk_exp': 8, 'splitter_chunk_overlap_frac': 0.5, 'reranker_enabled': False, 'hyde_enabled': False, 'additional_context_enabled': False, 'subquestion_engine_llm': 'microsoft/Phi-4-multimodal-instruct', 'subquestion_response_synthesizer_llm': 'nvidia/Llama-3_3-Nemotron-Super-49B'}
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m [D 2025-06-06 14:41:04,641] Model input: {"messages": ["ChatMessage(role=<MessageRole.USER: 'user'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text='You are a world class state of the art agent.\\n\\nYou have access to multiple tools, each representing a different data source or API.\\nEach of the tools has a name and a description, formatted as a JSON dictionary.\\nThe keys of the dictionary are the names of the tools and the values are the descriptions.\\nYour purpose is to help answer a complex user question by generating a list of sub questions that can be answered by the tools.\\n\\nThese are the guidelines you consider when completing your task:\\n* Be as specific as possible\\n* The sub questions should be relevant to the user question\\n* The sub questions should be answerable by the tools provided\\n* You can generate multiple sub questions for each tool\\n* Tools must be specified by their name, not their description\\n* You don\\'t need to use a tool if you don\\'t think it\\'s relevant\\n\\nOutput the list of sub questions by calling the SubQuestionList function.\\n\\n## Tools\\n```json\\n{\\n \"bright_hf\": \"This resource contains everything about psychology, including cognitive, developmental, social, and clinical psychology. It contains historical data, live data, just about everything about psychology, for instance, information about brain structures, psychological disorders, or therapeutic techniques, key events or theories in psychology history, information about psychological research methods or techniques, and information about psychological assessments or tests.\"\\n}\\n```\\n\\n## User Question\\nHas there ever been a split brain pianist?\\nAsked 2 years, 6 months ago\\nModified 2 years, 5 months ago\\nViewed 634 times\\n0\\n\\nHas there ever been a split brain pianist?\\n\\nI was thinking about this yesterday and it occurred to me that a split brain pianist might have an advantage over a non-split brain pianist; they could potentially read the bass and treble clef lines independently and thus process the lines independently. Perhaps this advantage would be meaningless at the professional level, but it surely could have the potential to make it easier for beginner pianists. I spent some time searching but was unfortunately unable to find any case studies of this.\\n\\nI did, however, find a case study of a ex-professional pianist with alien hand syndrome whose symptoms were exacerbated when playing the piano, but it wasn\\'t what I was looking for.\\n\\nI\\'d also be interested in any examples of split brain patients who were able to play any instrument at all \u2013 I\\'d find the outcomes fascinating.\\n')])"], "kwargs": {"tools": [{"type": "function", "function": {"name": "SubQuestionList", "description": "A pydantic object wrapping a list of sub-questions.\n\nThis is mostly used to make getting a json schema easier.", "parameters": {"$defs": {"SubQuestion": {"properties": {"sub_question": {"title": "Sub Question", "type": "string"}, "tool_name": {"title": "Tool Name", "type": "string"}}, "required": ["sub_question", "tool_name"], "title": "SubQuestion", "type": "object"}}, "description": "A pydantic object wrapping a list of sub-questions.\n\nThis is mostly used to make getting a json schema easier.", "properties": {"items": {"items": {"$ref": "#/$defs/SubQuestion"}, "title": "Items", "type": "array"}}, "required": ["items"], "title": "SubQuestionList", "type": "object"}}}], "tool_choice": {"type": "function", "function": {"name": "SubQuestionList"}}}}
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m Traceback (most recent call last):
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_6738eef6fec17802/syftr/evaluation.py", line 181, in exception_catcher
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m results = await func(**kwargs)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_6738eef6fec17802/syftr/flows.py", line 149, in agenerate
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m response, duration = await self._agenerate(query, invocation_id)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m result = await func(*args, **kwargs)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/py_modules_files/_ray_pkg_6738eef6fec17802/syftr/flows.py", line 347, in _agenerate
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m response = await self.query_engine.aquery(query)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m result = await func(*args, **kwargs)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/base/base_query_engine.py", line 64, in aquery
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m query_result = await self._aquery(str_or_query_bundle)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 370, in async_wrapper
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m result = await func(*args, **kwargs)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 193, in _aquery
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m qa_pairs_all = await asyncio.gather(*tasks)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/home/alex/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m future.result()
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/home/alex/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m result = coro.send(None)
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m File "/tmp/ray/session_2025-06-05_07-19-04_642991_1676/runtime_resources/pip/dd5f956fcc327946303d03fcf07dea86900ea86c/virtualenv/lib/python3.12/site-packages/llama_index/core/query_engine/sub_question_query_engine.py", line 227, in _aquery_subq
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m query_engine = self._query_engines[sub_q.tool_name]
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
^[[36m(<lambda> pid=2119979, ip=172.16.4.67)^[[0m KeyError: ''
