How to remove duplicate rows in pandas

Repeated records can distort totals, counts, and exports when the same observation appears more than once. The pandas DataFrame.drop_duplicates() method removes repeated rows while retaining one copy for downstream work.

By default, drop_duplicates() compares every column, ignores index labels, and keeps the first occurrence. The subset argument can limit matching to business-key columns when two rows should count as duplicates even if their other values differ.

The method returns a new DataFrame unless inplace=True is selected. Assigning the result preserves the source rows for comparison, while ignore_index=True gives the retained rows consecutive labels.

1. Create the input DataFrame in remove_duplicate_rows.py.

   remove_duplicate_rows.py

   import pandas as pd
    
   orders = pd.DataFrame(
       {
           "order_id": [1001, 1002, 1002, 1003, 1004],
           "customer": ["Ada", "Lin", "Lin", "Maya", "Omar"],
           "total_usd": [150.0, 240.0, 240.0, 875.0, 95.0],
       }
   )

2. Append exact duplicate removal after the DataFrame definition.

   deduplicated = orders.drop_duplicates(ignore_index=True)

   The subset=[“order_id”] argument limits matching to the identifier, while keep=“last” retains the final matching row.

3. Append fail-capable duplicate checks and output after the removal.

   assert len(deduplicated) == 4
   assert not deduplicated.duplicated().any()
   assert len(orders) == 5
    
   print(deduplicated.to_string(index=False))
   print(f"rows removed: {len(orders) - len(deduplicated)}")
   print(f"source rows: {len(orders)}")

4. Confirm duplicate removal with the completed program.

   $ python3 remove_duplicate_rows.py
    order_id customer  total_usd
        1001      Ada      150.0
        1002      Lin      240.0
        1003     Maya      875.0
        1004     Omar       95.0
   rows removed: 1
   source rows: 5