Cutover test runs

.github/workflows/azure-sdk-tools.yml

@@ -21,7 +21,7 @@ jobs:
 
       - name: Install azure-sdk-tools
         run: |
-          python -m pip install -e eng/tools/azure-sdk-tools[build,ghtools,conda]
+          python -m pip install -e eng/tools/azure-sdk-tools[ghtools,conda]
           python -m pip freeze
         shell: bash
 
@@ -43,7 +43,7 @@ jobs:
 
       - name: Install azure-sdk-tools
         run: |
-          python -m pip install -e eng/tools/azure-sdk-tools[build,ghtools,conda]
+          python -m pip install -e eng/tools/azure-sdk-tools[ghtools,conda]
           python -m pip install black==24.4.0
           python -m pip freeze
         shell: bash
@@ -70,7 +70,7 @@ jobs:
 
       - name: Install azure-sdk-tools on in global uv, discover azpysdk checks
         run: |
-          uv pip install --system eng/tools/azure-sdk-tools[build,ghtools,conda]
+          uv pip install --system eng/tools/azure-sdk-tools[ghtools,conda,systemperf]
 
           # Discover available azpysdk commands from the {command1,command2,...} line in help output
           CHECKS=$(azpysdk -h 2>&1 | \
@@ -92,19 +92,20 @@ jobs:
 
       - name: Run all discovered checks against azure-template using uv as package manager
         run: |
-          python eng/scripts/dispatch_checks.py --checks "$AZPYSDK_CHECKS" azure-template
+          sdk_build azure-template -d $(pwd)/wheels --build_id 20250101.1
+          python eng/scripts/dispatch_checks.py --checks "$AZPYSDK_CHECKS" --wheel_dir $(pwd)/wheels azure-template
         shell: bash
         env:
           TOX_PIP_IMPL: "uv"
 
       - name: Install azure-sdk-tools on global pip env
         run: |
-          python -m pip install -e eng/tools/azure-sdk-tools[build,ghtools,conda]
+          python -m pip install -e eng/tools/azure-sdk-tools[ghtools,conda]
         shell: bash
 
       - name: Run all discovered checks against azure-template using pip as package manager
         run: |
-          python eng/scripts/dispatch_checks.py --checks "$AZPYSDK_CHECKS" azure-template
+          python eng/scripts/dispatch_checks.py --checks "$AZPYSDK_CHECKS" --wheel_dir $(pwd)/wheels azure-template
         shell: bash
 
   dev-setup-and-import:

.gitignore

@@ -1,5 +1,6 @@
 # Default Assets restore directory
 .assets
+.assets_distributed
 
 # Python cache
 __pycache__/

CONTRIBUTING.md

@@ -11,9 +11,7 @@ If you want to contribute to a file that is generated (header contains `Code gen
 
 We utilize a variety of tools to ensure smooth development, testing, and code quality for the Azure Python SDK. Below is a list of key tools and their purpose in the workflow:
 
-- Tox: [Tox](https://tox.wiki/en/latest/) is our primary tool for managing test environments. It allows us to distribute tests to virtual environments, install dependencies, and maintain consistency between local and CI builds. Tox is configured to handle various testing scenarios, including linting, type checks, and running unit tests.
-
-- Virtualenv: [Virtualenv](https://virtualenv.pypa.io/en/latest/) is leveraged by Tox to create isolated environments for each test suite, ensuring consistent dependencies and reducing conflicts.
+- azpysdk: The `azpysdk` CLI is our primary tool for running checks locally and in CI. It is an entrypoint provided by the `eng/tools/azure-sdk-tools` package and abstracts all checks (linting, type checking, tests, doc generation, etc.) behind a single command. See the [Tool Usage Guide](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/tool_usage_guide.md) for full details.
 
 - UV: [UV](https://docs.astral.sh/uv/) is a fast package manager that can manage Python versions, run and install Python packages, and be used instead of pip, virtualenv, and more.
 
@@ -31,140 +29,65 @@ We utilize a variety of tools to ensure smooth development, testing, and code qu
 
 ## Building and Testing
 
-The Azure SDK team's Python CI leverages the tool `tox` to distribute tests to virtual environments, handle test dependency installation, and coordinate tooling reporting during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing.
-
-[A Brief Overview of Tox](https://tox.wiki/en/latest/)
-
-#### A Monorepo and Tox in Harmony
-
-Traditionally, the `tox.ini` file for a package sits _alongside the setup.py_ in source code. The `azure-sdk-for-python` necessarily does not adhere to this policy. There are over one-hundred packages contained here-in. That's a lot of `tox.ini` files to maintain!
-
-Instead, the CI system leverages the `--root` argument which is new to `tox4`. The `--root` argument allows `tox` to act as if the `tox.ini` is located in whatever directory you specify!
-
-#### Tox Environments
-
-A given `tox.ini` works on the concept of `test environments`. A given test environment is a combination of:
-
-1. An identifier (or identifiers)
-2. A targeted Python version
-    1. `tox` will default to base python executing the `tox` command if no Python environment is specified
-3. (optionally) an OS platform
-
-Internally `tox` leverages `virtualenv` to create each test environment's virtual environment.
+The Azure SDK team's Python CI leverages the `azpysdk` CLI tool to run checks, tests, and linters during PR/CI builds. This means that a dev working locally can reproduce _exactly_ what the build machine is doing.
 
-This means that once the `tox` workflow is in place, all tests will be executed _within a virtual environment._
+The `azpysdk` entrypoint is provided by the `eng/tools/azure-sdk-tools` package. For full setup instructions and the list of available checks, see the [Tool Usage Guide](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/tool_usage_guide.md).
 
-You can use the command `tox list` to list all the environments provided by a `tox.ini` file. You can either use that command in the
-same directory as the file itself, or use the `--conf` argument to specify the path to it directly.
+### Quick Setup
 
-
-Sample output of `tox list`:
+From the root of your target package:
 
 ```
-sdk-for-python/eng/tox> tox list
-default environments:
-whl              -> Builds a wheel and runs tests
-sdist            -> Builds a source distribution and runs tests
-
-additional environments:
-pylint           -> Lints a package with a pinned version of pylint
-next-pylint      -> Lints a package with pylint (version 2.15.8)
-mypy             -> Typechecks a package with mypy (version 1.9.0)
-next-mypy        -> Typechecks a package with the latest version of mypy
-pyright          -> Typechecks a package with pyright (version 1.1.287)
-next-pyright     -> Typechecks a package with the latest version of static type-checker pyright
-verifytypes      -> Verifies the "type completeness" of a package with pyright
-whl_no_aio       -> Builds a wheel without aio and runs tests
-develop          -> Tests a package
-sphinx           -> Builds a package's documentation with sphinx
-depends          -> Ensures all modules in a target package can be successfully imported
-verifywhl        -> Verify directories included in whl and contents in manifest file
-verifysdist      -> Verify directories included in sdist and contents in manifest file. Also ensures that py.typed configuration is correct within the setup.py
-devtest          -> Tests a package against dependencies installed from a dev index
-latestdependency -> Tests a package against the released, upper-bound versions of its azure dependencies
-mindependency    -> Tests a package against the released, lower-bound versions of its azure dependencies
-apistub          -> Generate an api stub of a package ( for https://apiview.dev )
-bandit           -> Runs bandit, a tool to find common security issues, against a package
-samples          -> Runs a package's samples
-breaking         -> Runs the breaking changes checker against a package
+pip install -r dev_requirements.txt
 ```
 
-### Example Usage of the common Azure SDK For Python `tox.ini`
-
-Basic usage of `tox` within this monorepo is:
-
-1. `pip install "tox<5"`
-2. Run `tox run -e ENV_NAME -c path/to/tox.ini --root path/to/python_package`
-  * **Note**: You can use environment variables to provide defaults for tox config values
-    * With `TOX_CONFIG_FILE` set to the absolute path of `tox.ini`, you can avoid needing `-c path/to/tox.ini` in your tox invocations
-    * With `TOX_ROOT_DIR` set to the absolute path to your python package, you can avoid needing `--root path/to/python_package`
-
-The common `tox.ini` location is `eng/tox/tox.ini` within the repository.
-
-If at any time you want to blow away the tox created virtual environments and start over, simply append `-r` to any tox invocation!
-
-#### Example `azure-core` mypy
-
-1. Run `tox run -e mypy -c ./eng/tox/tox.ini --root sdk/core/azure-core`
+This installs `azure-sdk-tools` (which provides `azpysdk`) along with the package's dev dependencies.
 
-#### Example `azure-storage-blob` tests
+### Available Checks
 
-2. Execute `tox run -c ./eng/tox/tox.ini --root sdk/storage/azure-storage-blob`
-
-Note that we didn't provide an `environment` argument for this example. Reason here is that the _default_ environment selected by our common `tox.ini` file is one that runs `pytest`.
-
-#### `whl` environment
-Used for test execution across the spectrum of all the platforms we want to support. Maintained at a `platform specific` level just in case we run into platform-specific bugs.
-
-* Installs the wheel, runs tests using the wheel
+You can discover all available checks by running `azpysdk --help`. Some common checks:
 
 ```
-\> tox run -e whl -c <path to tox.ini> --root <path to python package>
-
+azpysdk pylint .        # Lint with pylint
+azpysdk mypy .          # Type check with mypy
+azpysdk pyright .       # Type check with pyright
+azpysdk verifytypes .   # Verify type completeness
+azpysdk sphinx .        # Build documentation
+azpysdk bandit .        # Security analysis
+azpysdk black .         # Code formatting
+azpysdk verifywhl .     # Verify wheel contents
+azpysdk verifysdist .   # Verify sdist contents
+azpysdk import_all .    # Verify all imports resolve
+azpysdk apistub .       # Generate API stub
+azpysdk samples .       # Run samples
+azpysdk breaking .      # Check for breaking changes
+azpysdk devtest .       # Test against dev feed dependencies
 ```
 
-#### `sdist` environment
-Used for the local dev loop.
+### Running from the repo root
 
-* Installs package in editable mode
-* Runs tests using the editable mode installation, not the wheel
+`azpysdk` also supports globbing and comma-separated package names when invoked from the repo root:
 
 ```
-
-\> tox run -e sdist -c <path to tox.ini> --root <path to python package>
-
+azure-sdk-for-python> azpysdk import_all azure-storage*
+azure-sdk-for-python> azpysdk pylint azure-storage-blob,azure-core
 ```
 
-#### `pylint` environment
-Pylint install and run.
-
-```
-\> tox run -e pylint -c <path to tox.ini> --root <path to python package>
-```
+### Isolated environments
 
-
-#### `mypy` environment
-Mypy install and run.
-
-```
-\> tox run -e mypy -c <path to tox.ini> --root <path to python package>
-```
-
-#### `sphinx` environment
-Generate sphinx doc for this package.
+To run a check in a completely fresh virtual environment, add `--isolate`:
 
 ```
-\> tox run -e sphinx -c <path to tox.ini> --root <path to python package>
+azpysdk pylint . --isolate
 ```
 
 ### Custom Pytest Arguments
 
-`tox` supports custom arguments, and the defined pytest environments within the common `tox.ini` also allow these. Essentially, separate the arguments you want passed to `pytest` by a `--` in your tox invocation.
-
-[Tox Documentation on Positional Arguments](https://tox.wiki/en/latest/config.html#substitutions-for-positional-arguments-in-commands)
+When running test-related checks, you can pass additional arguments to `pytest` after `--`:
 
-**Example: Invoke tox, breaking into the debugger on failure**
-`tox run -e whl -c <path to tox.ini> --root <path to python package> -- --pdb`
+```
+azpysdk devtest . -- --pdb
+```
 
 ### Performance Testing
 
@@ -175,7 +98,7 @@ SDK performance testing is supported via the custom `perfstress` framework. For
 We maintain an [additional document](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/eng_sys_checks.md) that has a ton of detail as to what is actually _happening_ in these executions.
 
 ### Dev Feed
-Daily dev build version of Azure sdk packages for python are available and are uploaded to Azure devops feed daily. We have also created a tox environment to test a package against dev built version of dependent packages. Below is the link to Azure devops feed.
+Daily dev build version of Azure sdk packages for python are available and are uploaded to Azure devops feed daily. Below is the link to Azure devops feed.
 [`https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python`](https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python)
 
 ##### To install latest dev build version of a package
@@ -191,13 +114,13 @@ pip install azure-appconfiguration==1.0.0b6.dev20191205001 --extra-index-url htt
 
 To test a package being developed against latest dev build version of dependent packages:
 a. cd to package root folder
-b. run tox environment devtest
+b. run `azpysdk devtest`
 
 ```
-\> tox run -e devtest -c <path to tox.ini> --root <path to python package>
+azpysdk devtest .
 ```
 
-This tox test( devtest) will fail if installed dependent packages are not dev build version.
+This check will fail if installed dependent packages are not dev build version.
 
 ## Samples
 

doc/dev/conda-builds.md

@@ -22,7 +22,7 @@ Follow the instructions [here](https://docs.conda.io/projects/conda-build/en/lat
 
 ```bash
 # cd <repo root>
-pip install "eng/tools/azure-sdk-tools[build,conda]"
+pip install "eng/tools/azure-sdk-tools[conda]"
 ```
 
 ### Get the configuration blob

doc/dev/dev_setup.md

@@ -33,11 +33,10 @@ or execute the various commands available in the toolbox.
 
 4. Setup your development environment
 
-    Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library), [Tox][tox] and an editable install of your library. For example, to install requirements for `azure-ai-formrecognizer`:
+    Install the development requirements for a specific library (located in the `dev_requirements.txt` file at the root of the library) and an editable install of your library. This will also install `azure-sdk-tools` which provides the `azpysdk` CLI for running checks. For example, to install requirements for `azure-ai-formrecognizer`:
     ```
     azure-sdk-for-python> cd sdk/formrecognizer/azure-ai-formrecognizer
     azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -r dev_requirements.txt
-    azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install "tox<5"
     azure-sdk-for-python/sdk/formrecognizer/azure-ai-formrecognizer> pip install -e .
     ```
 
@@ -54,5 +53,4 @@ After following the steps above, you'll be able to run recorded SDK tests with `
 [python_website]: https://www.python.org/downloads/
 [python_312]: https://apps.microsoft.com/detail/9ncvdn91xzqp
 [tests]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md
-[tox]: https://tox.wiki/en/latest/
 [virtual_environment]: https://docs.python.org/3/tutorial/venv.html

doc/dev/engineering_assumptions.md

@@ -13,7 +13,7 @@ universal=1
 Build CI for `azure-sdk-for-python` essentially builds and tests packages in one of two methodologies.
 
 ### Individual Packages
-1. Leverage `tox` to create wheel, install, and execute tests against newly installed wheel
+1. Leverage `azpysdk` to create wheel, install, and execute tests against newly installed wheel
 2. Tests each package in isolation (outside of dev_requirements.txt dependencies + necessary `pylint` and `mypy`)
 
 ### Global Method

doc/dev/pylint_checking.md

@@ -22,11 +22,11 @@ In the Azure SDK for Python repository, in addition to the standard pylint libra
 
 ## How to run Pylint?
 
-One way to run pylint is to run at the package level with tox:
+The recommended way to run pylint is with `azpysdk` at the package level:
 
-    .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid>tox run -e pylint -c ../../../eng/tox/tox.ini --root .
+    .../azure-sdk-for-python/sdk/eventgrid/azure-eventgrid> azpysdk pylint .
 
-If you don't want to use tox, you can also install and run pylint on its own:
+If you don't want to use `azpysdk`, you can also install and run pylint on its own:
 
  - If taking this approach, in order to run with the pylintrc formatting and the custom pylint checkers you must also install the custom checkers and `SET` the pylintrc path. 
 
@@ -36,8 +36,7 @@ If you don't want to use tox, you can also install and run pylint on its own:
         .../azure-sdk-for-python>SET PYLINTRC="./pylintrc"
         .../azure-sdk-for-python>pylint ./sdk/eventgrid/azure-eventgrid
 
-    Note that you may see different errors if running a different [version of pylint or azure-pylint-guidelines-checker](https://github.com/Azure/azure-sdk-for-python/blob/fdf7c49ea760b1e1698ebbbac48794e8382d8de5/eng/tox/tox.ini#L90) than the one in CI.
-
+    Note that you may see different errors if running a different version of [pylint](https://github.com/Azure/azure-sdk-for-python/blob/main/eng/tools/azure-sdk-tools/azpysdk/pylint.py#L17) or [azure-pylint-guidelines-checker](https://github.com/Azure/azure-sdk-for-python/blob/main/eng/tools/azure-sdk-tools/azpysdk/pylint.py#L61) than the one in CI.
 
 # Ignoring Pylint Checkers
 
@@ -58,12 +57,12 @@ In addition to being a part of the CI, the custom pylint checkers are also integ
 
 There is now a new step on the CI pipeline called `Run Pylint Next`. This is merely a duplicate of the `Run Pylint` step with the exception that `Run Pylint Next` uses the latest version of pylint and the latest version of the custom pylint checkers.
 
-This next-pylint environment can also be run locally through tox:
+This next-pylint check can also be run locally:
 
-    tox run -e next-pylint -c ../../../eng/tox/tox.ini --root <path to python package>
+    azpysdk pylint --next=True <path to python package>
 
 The errors generated by the `Run Pylint Next` step will not break your weekly test pipelines, but make sure to fix the warnings so that your client library is up to date for the next pylint release. 
 
 # How to prepare your SDK for a new pylint update?
 
-Check each client library's `Run Pylint Next` output in the [test-weekly CI pipeline](https://dev.azure.com/azure-sdk/internal/_build?pipelineNameFilter=python%20*%20tests-weekly). If there is no corresponding test-weekly pipeline, run `next-pylint` locally with `tox` as described in [How to run Pylint?](#how-to-run-pylint). In order to ensure that the SDK pipeline will not break when pylint is updated, make sure to address all pylint warnings present. 
+Check each client library's `Run Pylint Next` output in the [test-weekly CI pipeline](https://dev.azure.com/azure-sdk/internal/_build?pipelineNameFilter=python%20*%20tests-weekly). If there is no corresponding test-weekly pipeline, run `next-pylint` locally with `azpysdk pylint --next=True .` as described in [How to run Pylint?](#how-to-run-pylint). In order to ensure that the SDK pipeline will not break when pylint is updated, make sure to address all pylint warnings present.