Skip to content

Fix case when undecodable characters are encountered on Windows #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
StanFromIreland wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Fix case when undecodable characters are encountered on Windows #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b2f0bcf
Commit
StanFromIreland Nov 2, 2025
3754074
Fix Windows quirkiness
StanFromIreland Nov 2, 2025
688b00d
Fix Windows quirkiness
StanFromIreland Nov 2, 2025
a1e0101
Fix test skip on macOS
StanFromIreland Nov 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions iconvcodec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ def _iconv_encode_impl(encoder, msg, errors, bufsize=None):
return encoder.iconv(msg, bufsize), len(msg)
except iconv.error as e:
errstring, code, inlen, outres = e.args
if code == 0:
# Windows quirk: raises with code=0 when partial conversion succeeded
# Just return what was converted
return outres, inlen
if code == errno.E2BIG:
# outbuffer was too small, increase size a bit and try to encode rest
out1, len1 = _iconv_encode_impl(
Expand Down Expand Up @@ -53,6 +57,10 @@ def _iconv_decode_impl(decoder, msg, errors, bufsize=None):
return decoder.iconv(msg, bufsize).decode(), len(msg)
except iconv.error as e:
errstring, code, inlen, outres = e.args
if code == 0:
# Windows quirk: raises with code=0 when partial conversion succeeded
# Just return what was converted
return outres.decode(), inlen
if code == errno.E2BIG:
# buffer too small
out1, len1 = _iconv_decode_impl(
Expand All @@ -77,6 +85,7 @@ def _iconv_decode_impl(decoder, msg, errors, bufsize=None):
else:
raise ValueError("unsupported error handling")
return outres.decode() + out1, inlen + len1 + 1
raise
Copy link
Contributor Author

@StanFromIreland StanFromIreland Nov 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consistent with encode implementation, I assume this was an oversight.

Copy link
Owner

@bodograumann bodograumann Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should definitely always return a tuple or raise an error.



def codec_factory(encoding):
Expand Down
7 changes: 5 additions & 2 deletions test_iconvcodec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,14 @@ def test_decode(self):
string = b"Hallo".decode("T.61")
self.assertEqual(string, "Hallo")

@unittest.skipUnless(sys.platform.startswith("linux"), "Linux only test")
@unittest.skipIf(sys.platform == "darwin", "Test fails on macOS")
def test_transliterate(self):
string = "abc ß α € àḃç"
bytestring = string.encode("ASCII//TRANSLIT")
self.assertEqual(bytestring, b"abc ss ? EUR abc")
if sys.platform.startswith("linux"):
self.assertEqual(bytestring, b"abc ss ? EUR abc")
else: # Windows
self.assertEqual(bytestring, b"abc ss ")

def test_incremental_encode(self):
encoder = codecs.getincrementalencoder("ASCII//TRANSLIT")()
Expand Down