How to migrate pandas code to the string dtype

Migrating pandas code to the string dtype means updating tests, dtype checks, and text-column writes that still assume strings live in NumPy object columns. pandas 3 infers text as str in constructors and IO readers, so code that checks only for object can skip real string columns after an upgrade.

The pandas 3 str dtype stores strings or missing values, and missing text is represented with NaN rather than preserving None. Existing code that intentionally uses the nullable StringDtype through dtype=“string” can usually stay on that dtype, but code that relied on flexible object columns needs a deliberate fallback when mixed Python objects are valid data.

Run the migration with fixtures that cover dtype checks, missing text values, Series.str operations, non-string writes, and array access. Keep object only for columns that must hold non-string values, invalid Unicode, bytes, or other Python objects, and verify the rewritten code against project data before removing temporary checks.

1. Enable pandas 3 string inference before the final pandas 2.3 test run.

   import pandas as pd
    
   pd.options.future.infer_string = True

   Skip this setting when tests already run on pandas 3. String inference is already enabled there.

2. Replace object dtype checks with a string dtype predicate.

   from pandas.api.types import is_string_dtype
    
   # Before
   if df["name"].dtype == "object":
       validate_text(df["name"])
    
   # After
   if is_string_dtype(df["name"]):
       validate_text(df["name"])

   Pass the Series when older data may still be object backed. Passing only the dtype token cannot distinguish a text-only object column from a mixed-object column.

3. Stop forcing text columns to object.

   # Before
   names = pd.Series(["Ada", "Lin", None], dtype="object")
    
   # After
   names = pd.Series(["Ada", "Lin", None], dtype="str")

   Use dtype=“str” for the pandas 3 default string dtype. Keep dtype=“string” only when existing code intentionally uses nullable StringDtype and pd.NA.

4. Check missing text values with pd.isna().

   # Before
   if value is None:
       handle_missing(value)
    
   # After
   if pd.isna(value):
       handle_missing(value)

   None becomes NaN in the pandas 3 str dtype, while pd.isna() still reports the value as missing.
   Related: How to find missing values in pandas

5. Keep non-string object data explicit.

   # Text-only column after the pandas 3 upgrade
   df["name"] = df["name"].astype("str")
    
   # Mixed Python objects that are not text-only data
   df["payload"] = df["payload"].astype("object")

   A str column rejects non-string values. Use object only when the column is intentionally mixed, not to keep old string checks working.

6. Replace values assumptions with the array form the next operation needs.

   array_for_numpy = df["name"].to_numpy()
   extension_array = df["name"].array

   Series.values can return an ArrowStringArray for str columns when pyarrow backs the dtype. Use to_numpy() when NumPy code needs an array, or array when extension-array behavior is expected.

7. Save a string dtype migration check script.

   string_dtype_migration_check.py

   import pandas as pd
   from pandas.api.types import is_string_dtype
    
    
   print(f"pandas {pd.__version__}")
    
   names = pd.Series(["Ada", "Lin", None], name="name")
   mixed_probe = pd.Series(["Ada", 2.5], dtype="object", name="mixed")
    
   print(f"inferred dtype: {names.dtype}")
   print(f"object dtype check: {names.dtype == 'object'}")
   print(f"string column check: {is_string_dtype(names)}")
   print(f"mixed object check: {is_string_dtype(mixed_probe)}")
   print(f"missing value is None: {names.iloc[2] is None}")
   print(f"missing value is NA: {pd.isna(names.iloc[2])}")
   print(f"str accessor result: {names.str.upper().tolist()}")
    
   try:
       names.iloc[1] = 2.5
   except TypeError as exc:
       print(f"non-string write: {exc}")
    
   mixed = names.astype("object")
   mixed.iloc[1] = 2.5
   print(f"mixed opt-out dtype: {mixed.dtype}")
   print(f"mixed opt-out value: {mixed.iloc[1]}")
   print(f"values object: {type(names.values).__name__}")
   print(f"numpy object: {type(names.to_numpy()).__name__}")

   Replace the sample Series objects with fixtures from the project when tests depend on real column names, IO readers, or validation rules.

8. Run the migration check script.

   $ python3 string_dtype_migration_check.py
   pandas 3.0.3
   inferred dtype: str
   object dtype check: False
   string column check: True
   mixed object check: False
   missing value is None: False
   missing value is NA: True
   str accessor result: ['ADA', 'LIN', nan]
   non-string write: Invalid value '2.5' for dtype 'str'. Value should be a string or missing value, got 'float' instead.
   mixed opt-out dtype: object
   mixed opt-out value: 2.5
   values object: ArrowStringArray
   numpy object: ndarray

9. Remove the temporary check script after project tests cover the same cases.

   $ rm string_dtype_migration_check.py