DOC: Address other string types in Pandas 3 string migration guide

[mattharrison]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/migration-3-strings.rst

Documentation problem

In Pandas 2.3, we had 'string[pyarrow]' and pd.ArrowDtype(pa.string())

The new guide at https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/migration-3-strings.rst doesn't mention either.

It would be worthwhile to mention them or deprecate them.

I assume the recommended best practice is to just use 'str' for the type and disregard the other two.

Suggested fix for documentation

Mention the 2.3 types and what to migrate them to. Also whether or not it makes sense to use them at all.

[mattharrison]

Forgot, there's also 'string'...

[mattharrison]

Also, confused as to why the referred doc says:

"Similarly, functions like [:func:`read_csv`](https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/migration-3-strings.rst#id1), [:func:`read_parquet`](https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/migration-3-strings.rst#id3), and others will now use the new string dtype when reading string data.

When I load a CSV with pyarrow engine the dtype for string is string[pyarrow]

>>> import io

>>> data = """name,age
... Alice,30
... Bob,25
... Charlie,35"""
>>> df = pd.read_csv(io.StringIO(data), dtype_backend='pyarrow')
>>> df.dtypes
name    string[pyarrow]
age      int64[pyarrow]
dtype: object

and

>>> df.name.astype(str).dtype == df.name.dtype
False

Note this is with '3.0.0rc0'

[jorisvandenbossche]

Yeah, I added this TODO to the migration guide to add a section targeted at existing users of those other string dtypes:

pandas/doc/source/user_guide/migration-3-strings.rst

Lines 460 to 463 in e450f0c

	.. For existing users of the nullable ``StringDtype``
	.. --------------------------------------------------
	.. TODO

but didn't get to adding that before the RC. But so fully agreed that would be helpful.

As for the actual advice, I think there might generally be two motivations: 1) do you specifically want the dtypes that support NA or the ArrowDtypes (and string dtype is one of them), then just keep using what you have been using. 2) were you mostly using just the nullable string dtype (and not so much the other nullable or arrow dtypes) to have an actual or faster string dtype, then you can now use the default str dtype.

[jorisvandenbossche]

When I load a CSV with pyarrow engine the dtype for string is string[pyarrow]

Your example is not using the pyarrow "engine", but the "dtype backend". The engine is about which library to use to do the actual csv parsing, or the dtype backend is about which dtypes you want to use for the resulting dataframe. Both options are independent of each other (and you can in theory have all 4 combinations).

Using your example:

>>> df = pd.read_csv(io.StringIO(data))  # by default will typically use `engine="c"`
>>> df.dtypes
name      str
age     int64
dtype: object
>>> df = pd.read_csv(io.StringIO(data), engine="pyarrow")
>>> df.dtypes
name      str
age     int64
dtype: object
>>> df = pd.read_csv(io.StringIO(data), dtype_backend="pyarrow")  # use the default c engine, but return ArrowDtypes
>>> df.dtypes
name    string[pyarrow]
age      int64[pyarrow]
dtype: object
>>> df = pd.read_csv(io.StringIO(data), engine="pyarrow", dtype_backend="pyarrow")
>>> df.dtypes
name    string[pyarrow]
age      int64[pyarrow]
dtype: object

[mattharrison]

I guess my motivition was twofold:

• I had heard that Pandas 3 would be arrow native. I guess I misunderstood or that changed overtime
• I'm updating Effective Pandas for Pandas 3 and trying to come up with best practices for readers, which was hopefully going to tow the line of the previous point and clean up the messy discrepancies that 2.3 had.

But apparently, there really isn't much of a change from 2.3 (basically str works differently if pyarrow is installed and pl.col is new) . There is not clean arrow everywhere. And with 4 (or however many string types), and data moving out of arrow semi-randomly, I'm left to cover everything and provide a bunch of caveats for readers/users.

[rhshadrach]

In Pandas 2.3, we had 'string[pyarrow]' and pd.ArrowDtype(pa.string())

The new guide at https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/migration-3-strings.rst doesn't mention either.

The four variants are mentioned here: https://pandas.pydata.org/pandas-docs/dev/user_guide/text.html#the-four-stringdtype-variants

Edit: I should have also pointed at https://pandas.pydata.org/pandas-docs/dev/user_guide/text.html#specifying-stringdtype-explicitly

If you were specifying the dtypes to be string[pyarrow] or pd.ArrowDtype(pa.string()), then I think the expectation is you will not see a change. I was under the intention that the migration guide is aimed at users not doing so.

[mroeschke]

I guess my motivition was twofold:

* I had heard that Pandas 3 would be arrow native. I guess I misunderstood or that changed overtime

* I'm updating `Effective Pandas` for Pandas 3 and trying to come up with best practices for readers, which was hopefully going to tow the line of the previous point and clean up the messy discrepancies that 2.3 had.

But apparently, there really isn't much of a change from 2.3 (basically str works differently if pyarrow is installed and pl.col is new) . There is not clean arrow everywhere. And with 4 (or however many string types), and data moving out of arrow semi-randomly, I'm left to cover everything and provide a bunch of caveats for readers/users.

IMO 1.5-2 years ago there was a lot of "marketing" (conference talks, etc) around all arrow-backed types in pandas, but only string types were explicitly planned to be arrow backed by 3.0 (PDEP-14). The decided state for the default string type in 3.0 was shaped by user feedback, and I also agree with you that all the variations of string types is not great.

This long thread is essentially the latest discussion about arrow types by default in pandas. My TLDR is:

• Backward compatibility and migration path is still taking shape and would likely require multiple major releases to get to an "end state"
• The are some Large Topics that are not formalized yet (NA vs NaN as missing values, role of numpy types)