Users/robrandao/data driven tests by rodrigobr-msft · Pull Request #244 · microsoft/Agents-for-python

rodrigobr-msft · 2025-11-13T20:59:44Z

No description provided.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…erate_token.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ion/core/environment.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…osoft/Agents-for-python into users/robrandao/integration

Copilot

Pull Request Overview

This PR introduces a data-driven testing framework to the microsoft-agents-testing package. The implementation adds assertion utilities for validating bot activities, an integration testing framework with support for various test runners, and configuration files to support data-driven test scenarios.

Key changes:

New assertion framework with field-level and activity-level validation supporting multiple assertion types (EQUALS, CONTAINS, RE_MATCH, etc.)
Integration testing infrastructure with agent and response clients for testing bot interactions
Data-driven test configuration support via YAML files
Fixed package naming issue in pyproject.toml (was "microsoft-agents-hosting-core", now correctly "microsoft-agents-testing")

Reviewed Changes

Copilot reviewed 45 out of 51 changed files in this pull request and generated 30 comments.

Show a summary per file

File	Description
tests/activity/test_sub_channels.py	Removed empty test class placeholder
dev/microsoft-agents-testing/tests/assertions/test_check_field.py	Comprehensive tests for field assertion logic across various types
dev/microsoft-agents-testing/tests/assertions/test_assert_activity.py	Tests for activity assertion functionality with multiple assertion types
dev/microsoft-agents-testing/tests/assertions/_common.py	Pytest fixtures for activity testing
dev/microsoft-agents-testing/pytest.ini	Pytest configuration with test discovery and warning filters
dev/microsoft-agents-testing/pyproject.toml	Fixed incorrect package name
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/*.py	Core assertion framework implementation
dev/microsoft-agents-testing/microsoft_agents/testing/integration/*/.py	Integration testing framework with client implementations
dev/microsoft-agents-testing/microsoft_agents/testing/utils/*.py	Utility functions for activity normalization and URL parsing
dev/microsoft-agents-testing/microsoft_agents/testing/auth/*.py	Added copyright headers
dev/integration/*/.py	Example integration tests and data-driven test implementations
dev/integration/data_driven_tests/*.yaml	YAML configuration files for data-driven tests

Comments suppressed due to low confidence (3)

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py:52

Undefined variable assertion on line 51. The variable assertion_config is defined as a parameter, but on line 51, it's being accessed as assertion, which doesn't exist in this scope.

    for activity in activities:
        res, assertion_error_data = check_activity_verbose(activity, assertion)
        assert res, str(assertion_error_data)

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py:45

Variable quantifier is not used.

    quantifier = assertion_config.get(

dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/ddt.py:29

Syntax Error (in Python 3).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-13T21:07:36Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/select_activity.py

+    """Selects activities from a list based on the provided selector configuration.
+
+    :param activities: List of activities to select from.
+    :param selector_config: Configuration dict containing 'selector', 'quantifier', and optionally '


The docstring on line 34 has a truncated sentence ending with ' (single quote). The description appears to be cut off mid-sentence.

Suggested change

:param selector_config: Configuration dict containing 'selector', 'quantifier', and optionally '

:param selector_config: Configuration dict containing 'selector', 'quantifier', and optionally 'index'.

Copilot · 2025-11-13T21:07:37Z

dev/integration/test_basics.py

+from microsoft_agents.testing import DataDrivenTester
+
+


Missing import statement. The DataDrivenTester class is imported and used on lines 1 and 4, but this class doesn't appear to be exported from microsoft_agents.testing based on the __init__.py file shown in the PR.

Suggested change

from microsoft_agents.testing import DataDrivenTester

# Minimal DataDrivenTester base class definition

class DataDrivenTester:

def _run_data_driven_test(self, filename):

# Placeholder implementation

pass

Copilot · 2025-11-13T21:07:37Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/select_activity.py

+        activities,
+        {"selector": selector, "quantifier": SelectorQuantifier.ONE, "index": index},
+    )
+    return res[index] if res else None


Potential index out of bounds error on line 25. The function returns res[index] without checking if the index is valid for the result list. If res is shorter than index + 1, this will raise an IndexError.

Suggested change

return res[index] if res else None

return res[index] if res and 0 <= index < len(res) else None

Copilot · 2025-11-13T21:07:38Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/ddt.py

+        async def func(self, agent_client, response_client) -> None:
+            ddt = DataDrivenTest(test_path)
+
+            responses = []
+
+            await for step in ddt:
+                if isinstance(step, Activity):
+                    await agent_client.send_activity(step)
+                elif isinstance(step, dict):
+                    # assertion
+                    responses.extend(await response_client.pop())
+


The async def func function at line 29 contains invalid async syntax at line 34. The statement await for step in ddt: is incorrect - it should be either async for step in ddt: (for async iteration) or the function needs to be redesigned. Additionally, the ddt object is not shown to implement async iteration, and Activity is not imported.

Copilot · 2025-11-13T21:07:38Z

dev/microsoft-agents-testing/tests/assertions/test_check_field.py

+    def test_invalid_assertion_type(self):
+        # Passing an unsupported assertion type should return False
+        assert check_field("test", "test", "INVALID_TYPE") is False


The check_field function can raise a ValueError for unsupported assertion types (line 71), but in the test on line 170, an invalid assertion type "INVALID_TYPE" is expected to return False instead of raising an exception. This is inconsistent behavior - the implementation raises an error but the test expects False.

Copilot · 2025-11-13T21:07:44Z

dev/integration/integration/integration/foundational/test_suite.py

@@ -0,0 +1,141 @@
+import json


Syntax Error (in Python 3).

Copilot · 2025-11-13T21:07:44Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py

+
+from microsoft_agents.activity import Activity
+
+from .type_defs import FieldAssertionType, AssertionQuantifier


Import of 'AssertionQuantifier' is not used.

Suggested change

from .type_defs import FieldAssertionType, AssertionQuantifier

from .type_defs import FieldAssertionType

Copilot · 2025-11-13T21:07:45Z

dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_activity.py

+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+
+from typing import Any, TypeVar, Optional


Import of 'TypeVar' is not used.

Suggested change

from typing import Any, TypeVar, Optional

from typing import Any, Optional

Copilot · 2025-11-13T21:07:45Z

...icrosoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py

+from microsoft_agents.testing.assertions import (
+    ActivityAssertion,
+    assert_activity,
+)


Import of 'assert_activity' is not used.

Copilot · 2025-11-13T21:07:45Z

dev/microsoft-agents-testing/microsoft_agents/testing/integration/ddt.py

+import asyncio
+
+from copy import deepcopy
+from typing import Awaitable, Callable


Import of 'Awaitable' is not used.
Import of 'Callable' is not used.

rodrigobr-msft added 30 commits October 21, 2025 14:16

basic integration utility

cfb8847

Integration test suite factory implementation

4d93a46

Implementing core models for integration testing

be2fb1c

AutoClient mockup

0d5539a

Adding runner starter code

bf4c006

Adding foundational classes

ebe622d

Drafting AgentClient and ResponseClient implementations

45678e9

Spec test

c2b84cb

Filling in more implementation details

2816e68

More files

2214827

Cleaning up implementations

c90c8e4

Adding expect replies sending method

36f5c3b

Beginning unit tests

dfdd959

Adding integration decor from sample test cases

d4828fb

Integration from service url tests

91b3c44

_handle_conversation implementation for response_client

1eefe44

AgentClient tests completed

a06da9a

Creating response client tests

d36ec7a

Hosting server for response client

c096048

Response client tests completed

5451ddf

Beginning refactor of aiohttp runner

c2cdd40

Unit test updates

7d91ef9

Fixing issues in refactor

87610a5

Fixed TestIntegrationFromSample unit test

ddfdd0b

Another commit

dd0a87d

Reorganizing files

6a4e8ef

Completed TestIntegrationFromServiceUrl unit tests

7793790

Reformatting with black

3ebbd44

Quickstart integration test beginning

a6681bf

Draft of quickstart sample setup

1a7264b

rodrigobr-msft and others added 25 commits November 6, 2025 11:28

Update dev/microsoft-agents-testing/pyproject.toml

a6e305a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update dev/microsoft-agents-testing/microsoft_agents/testing/auth/gen…

c3877a3

…erate_token.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update dev/microsoft-agents-testing/microsoft_agents/testing/integrat…

fb5b76d

…ion/core/environment.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Removing unused code

5eb9bfb

Merge branch 'users/robrandao/integration' of https://github.com/micr…

81b5d6b

…osoft/Agents-for-python into users/robrandao/integration

Adding field assertion helpers

e451459

Implementing field assertion types

f241df8

Adding data driven test foundation

cb3c6aa

Separating assertions from checkers

b8f3304

Adjusting return values for check_activity with refactor

42b4ea1

Fixing package name in pyproject.toml

2f79824

Assertion test cases

cfded79

_parse_assertion unit tests

b6618b2

More assertion tests and fixes to data normalization logic

1e8f19e

Fixing more test cases with newest refactor

d9eb5fa

Fixing merge conflicts

f02b875

assert_activity tests

57b87eb

Reorganizing tests

6925093

Copyright comments

6d55b29

DDT implementation outline

6866f5d

DataDrivenTest class

1db2bf5

select_activity implementation for assertions

e44a2da

inverted assertions for quantifiers

1cf561b

ActivityAssertion class implementation

e651f92

DataDrivenTest implementation

ed7f2f1

Copilot AI review requested due to automatic review settings November 13, 2025 20:59

rodrigobr-msft closed this Nov 13, 2025

Copilot started reviewing on behalf of rodrigobr-msft November 13, 2025 21:00 View session

Copilot finished reviewing on behalf of rodrigobr-msft November 13, 2025 21:03

Copilot AI reviewed Nov 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Users/robrandao/data driven tests#244

Users/robrandao/data driven tests#244
rodrigobr-msft wants to merge 75 commits intomainfrom
users/robrandao/data-driven-tests

rodrigobr-msft commented Nov 13, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Copilot AI Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

	:param selector_config: Configuration dict containing 'selector', 'quantifier', and optionally '
	:param selector_config: Configuration dict containing 'selector', 'quantifier', and optionally 'index'.

-from microsoft_agents.testing import DataDrivenTester
+# Minimal DataDrivenTester base class definition
+class DataDrivenTester:
+    def _run_data_driven_test(self, filename):
+        # Placeholder implementation
+        pass

	return res[index] if res else None
	return res[index] if res and 0 <= index < len(res) else None


		from microsoft_agents.activity import Activity

		from .type_defs import FieldAssertionType, AssertionQuantifier

	from typing import Any, TypeVar, Optional
	from typing import Any, Optional

Conversation

rodrigobr-msft commented Nov 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments