Dropping Rows of Data Using Pandas

You ‘ve heard the cliché before : it is frequently cited that roughly % 80~ of a data scientist ‘s role is dedicated to cleaning data sets. I personally have n’t looked in to the papers or clinical trials which prove this number ( that was a joke ), but the estimate holds true : in the data profession, we find ourselves doing away with blatantly corrupt or useless data. The simplistic set about is to discard such data wholly, thus here we are. What constitutes ‘filthy ‘ data is project-specific, and at times boundary line subjective. occasionally, the offenders are more obvious : these might include chunks of data which are evacuate, ailing formatted, or plainly irrelevant. While ‘bad ‘ data can occasionally be fixed or salvaged via transforms, in many cases it ‘s best to do away with rows wholly to ensure that lone the fittest exist .

Drop Empty Rows or Columns

If you’re looking to drop rows ( or column ) containing empty data, you’re in luck : Pandas ‘ dropna() method is specifically for this. Using dropna() is a simpleton one-liner which accepts a number of utilitarian arguments :

import pandas as pd


# Create a Dataframe from a CSV
df = pd.read_csv('example.csv')  

# Drop rows with any empty cells
df.dropna(
    axis=0,
    how='any',
    thresh=None,
    subset=None,
    inplace=True
)

Drop rows containing empty values in any column technically you could run df.dropna() without any parameters, and this would default to dropping all rows where are completely evacuate. If thats all you needed, well, I guess you’re done already. otherwise, here are the parameters you can include :

• Axis: Specifies to drop by row or column. 0 means row, 1 means column.
• How: Accepts one of two possible values: any or all. This will either drop an axis which is completely empty (all), or an axis with even just a single empty cell (any).
• Thresh: Here’s an interesting one: thresh accepts an integer, and will drop an axis only if that number threshold of empty cells is breached.
• Subset: Accepts an array of which axis’ to consider, as opposed to considering all by default.
• Inplace: If you haven’t come across inplace yet, learn this now: changes will NOT be made to the DataFrame you’re touching unless this is set to True. It’s False by default.

Pandas’ .drop() Method

The Pandas .drop() method acting is used to remove rows or column. For both of these entities, we have two options for specifying what is to be removed :

• Labels: This removes an entire row or column based on its “label”, which translates to column name for columns, or a named index for rows (if one exists)
• Position: Passing an array of integers to drop() will remove rows or columns by their default position in table. Passing an array [0, 1] to drop() would either drop the first two rows of a table, or the first two columns, depending on the axis we specify.

To better illustrate this, let ‘s look at the possible arguments drop() accepts :

df.drop(
    labels=None,
    axis=0,
    index=None,
    columns=None,
    level=None,
    inplace=True,
    errors='raise'
)

Possible arguments to pass into Pandas’ drop() method

• Labels: Accepts either an array of strings (ie: labels=['column_1', 'column_2'] ) or an array of integers  (ie: labels=[0, 1] ). Passed an array of strings will drop based on column name/row index, whereas an array of integers will drop based on position.
• Axis: Specifies whether we’re dropping rows or columns. A value of 0 denotes row index, where a value of 1 specifies column name.
• Index: This is shorthand way of dropping rows by index name. Passing a single array of strings to index is effectively the same as passing the same array to labels and passing an axis of 0.
• Columns: Shorthand for accomplishing the reverse of index. Passing a single array of strings to columns is effectively the same as passing the same array to labels and passing an axis of 1.
• Levels: Used in sets of data which contain multiple hierarchical indexes (this likely doesn’t concern you).
• Inplace: As always, methods performed on DataFrames are not committed unless explicitly stated. The purpose of this is to presumably preserve the original set of data during ad hoc manipulation.This adheres to the Python style-guide which states that actions should not be performed on live sets of data unless explicitly stated. Here is a video of some guy describing this for some reason.
• Errors: Accepts either ignore or raise, with ‘raise’ set as default. When errors=’ignore’ is set, no errors will be thrown and existing labels are dropped.

Dropping Columns

Let ‘s say we have a DataFrame which contains a column we ‘ve deemed useless. To removing a column named preferred_icecream_flavor from our DataFrame looks like this :

df.drop(
    labels=["preferred_icecream_flavor"],
    axis=1,
)

Drop column preferred_icecream_flavor from DataFrame alternatively :

df.drop(
    columns=["preferred_icecream_flavor"]
)

Drop by column name If we wanted to drop columns based on the order in which they ‘re arranged ( for some argue ), we can achieve this as so

df.drop(
    labels=[0, 1],
    axis=1,
)

Drop first two columns from a DataFrame

Dropping Rows

The row equivalent of drop() looks similar. Let ‘s drop a rows where our DataFrame has been index with first gear names, like Todd and Kyle :

df.drop(
    labels=["todd", "kyle"],
    axis=0,
)

Yes, I’ve seen the George Carlin bit Or of class :

df.drop(
    index=["todd", "kyle"],
)

Shorthand for dropping rows by index

Drop Duplicates

It’s common to run into datasets which contain double rows, either as a resultant role of dirty data or some preliminary cultivate on the dataset. Pandas has a method specifically for purging these rows called drop_duplicates(). When we run drop_duplicates() on a DataFrame without passing any arguments, Pandas will refer to dropping rows where all data across column is precisely the like. Running this will keep one case of the double row, and remove all those after :

import pandas as pd

# Drop rows where all data is the same
df = df.drop_duplicates()

Drop rows which are identical duplicates  drop_duplicates() has a few options we can play with :

• Subset: Let’s say we wanted to detect duplicates only in a certain row, or even number of rows. We can pass either a column name (string) or a collection of columns (list) via the subset attribute to perform duplicate checking only against the provided columns. Note: even though we’re only using certain columns to determine duplicates, any detected duplicates will drop the entire row.
• Keep: If we find duplicates, how do we know which of the duplicates to keep? By default, Pandas will keep the first appearance of that row, and discard all others thereafter ( keep='first' ). To keep the last, we would use keep='last. If we just want to drop all duplicates, we use keep=False.
• Inplace: Using df.drop_duplicates(inplace=True) is the same as our example above: df = df.drop_duplicates()

Drop by Criteria

We can besides remove rows or columns based on whichever criteria your little affection desires. For exercise, if you in truth hate people named Chad, you can drop all rows in your Customer database who have the diagnose Chad. Screw Chad.

Unlike former methods, the popular means of handling this is simply by saving your DataFrame over itself give a excrete value. here ‘s how we ‘d get rid of Chad :

import pandas as pd

# Create a Dataframe from CSV
df = pd.read_csv('example.csv')

# Drop via logic (similar to a SQL 'WHERE' clause)
df = df[df.employee_name != 'chad']

Drop rows where cells meet a condition The syntax may seem a bit off-putting to newcomers ( note the repeat of df 3 times ). The format of df[CONDITION] just returns a modify translation of df, where only the data matching the given discipline is affected. Since we ‘re purging this datum wholly, stating df = df[CONDITION] is an easy ( albeit destructive ) method for shedding data and moving on with our lives .

reference : https://coinselected
Category : coin 4u