`fix --aggressive` raises `UnicodeDecodeError` when reading a file and doesn't print the file path

[bkeryan]

Running poetry run ni-python-styleguide fix --aggressive on https://github.com/ni/measurementlink-python produces this error:

Traceback (most recent call last):
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\dev\measurementlink-python\.venv\Scripts\ni-python-styleguide.exe\__main__.py", line 7, in <module>
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\click\decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\ni_python_styleguide\_cli.py", line 186, in fix
    _fix.fix(
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\ni_python_styleguide\_fix.py", line 110, in fix
    _acknowledge_existing_errors.remove_auto_suppressions_from_file(file)
  File "C:\dev\measurementlink-python\.venv\lib\site-packages\ni_python_styleguide\_acknowledge_existing_errors\__init__.py", line 138, in remove_auto_suppressions_from_file
    lines = file.read_text(encoding=_utils.DEFAULT_ENCODING).splitlines()
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\pathlib.py", line 1267, in read_text
    return f.read()
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 122: invalid start byte

For debugging, I added calls to print (but logging is more appropriate):

def remove_auto_suppressions_from_file(file: pathlib.Path):
    """Removes auto-suppressions from file."""
    print(str(file))
    lines = file.read_text(encoding=_utils.DEFAULT_ENCODING).splitlines()
    stripped_lines = [_filter_suppresion_from_line(line) for line in lines]
    file.write_text("\n".join(stripped_lines) + "\n", encoding=_utils.DEFAULT_ENCODING)


def _suppress_errors_in_file(bad_file, errors_in_file, encoding):
    path = pathlib.Path(bad_file)
    print(str(path))
    lines = path.read_text(encoding=encoding).splitlines(keepends=True)

This shows that fix --aggressive is reading files from .venv, even though they should be excluded.

C:\dev\measurementlink-python\.venv\Lib\site-packages\win32com\demos\excelAddin.py
C:\dev\measurementlink-python\.venv\Lib\site-packages\win32com\demos\excelRTDServer.py
C:\dev\measurementlink-python\.venv\Lib\site-packages\win32com\demos\iebutton.py
Traceback (most recent call last):
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\bkeryan\.pyenv\pyenv-win\versions\3.9.13\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)

[mshafer-NI]

@bkeryan, I'm not able to reproduce this in latest ni-python-styleguide, can you confirm if this is still an issue with the source version of 0.5?

[bkeryan]

@bkeryan, I'm not able to reproduce this in latest ni-python-styleguide, can you confirm if this is still an issue with the source version of 0.5?

I can reproduce it with v0.4.7:

• cd c:\dev\measurement-plugin-python\packages\generator
• poetry install
• Edit tests\utilities\measurements\localized_measurement\__init__.py and add some extra whitespace.
• poetry run nps fix --aggressive

output:

C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\flake8_import_order\styles.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
reformatted C:\Dev\measurement-plugin-python\packages\generator\tests\utilities\measurements\localized_measurement\__init__.py

All done! ✨ 🍰 ✨
1 file reformatted.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Scripts\nps.exe\__main__.py", line 7, in <module>
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\decorators.py", line 46, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\ni_python_styleguide\_cli.py", line 186, in fix
    _fix.fix(
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\ni_python_styleguide\_fix.py", line 155, in fix
    raise Exception(
Exception: Failed to format files:
C:\Dev\measurement-plugin-python\packages\generator\tests\utilities\measurements\localized_measurement\__init__.py: 'charmap' codec can't decode byte 0x8d in position 687: character maps to <undefined>

And I can still reproduce it with 0.5.0a0 3d83096 from main, after I re-add extra whitespace:

PS C:\dev\measurement-plugin-python\packages\generator> poetry install
Installing dependencies from lock file

Package operations: 2 installs, 1 update, 0 removals

  - Installing better-diff (0.1.4)
  - Installing flake8-tidy-imports (4.12.0)
  - Updating ni-python-styleguide (0.4.7 -> 0.5.0a0 3d83096)

Installing the current project: ni_measurement_plugin_sdk_generator (2.4.0-dev1)
PS C:\dev\measurement-plugin-python\packages\generator> poetry run nps fix --aggressive
C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\flake8_import_order\styles.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
PS C:\dev\measurement-plugin-python\packages\generator> notepad .\tests\utilities\measurements\localized_measurement\__init__.py
PS C:\dev\measurement-plugin-python\packages\generator> poetry run nps fix --aggressive
C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\flake8_import_order\styles.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import iter_entry_points
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Scripts\nps.exe\__main__.py", line 7, in <module>
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\click\decorators.py", line 46, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\ni_python_styleguide\_cli.py", line 184, in fix
    _fix.fix(
  File "C:\dev\measurement-plugin-python\packages\generator\.venv\Lib\site-packages\ni_python_styleguide\_fix.py", line 139, in fix
    raise Exception(
Exception: Failed to format files:
C:\Dev\measurement-plugin-python\packages\generator\tests\utilities\measurements\localized_measurement\__init__.py: 'charmap' codec can't decode byte 0x8d in position 687: character maps to <undefined>

[mshafer-NI]

@bkeryan, can you check if this is fixed (in source) as of #242 ?

[bkeryan]

Yes, I tried my test case from #129 (comment) with 7201b40 and it undid the whitespace changes without errors.