Python Pandas / loc, iloc, Nan, unique, drop

Python Pandas DataFrame 데이터 활용2

loc vs iloc 비교, Nan (결측값), unique, drop

 

BASIC_03.Pandas_DataFrame.ipynb¶

In [1]:

import pandas as pd
import numpy as np

In [2]:

# DataFrame 생성시 index, column 설정안하면 정수형 숫자로 기본값이 입력된다.
df = pd.DataFrame(np.random.randn(10,3)) # 랜덤값 10x3개의 np array객체입력
df.columns = ["one", "two", "three"]
df

Out[2]:

	one	two	three
0	-1.141389	-1.597709	-0.593118
1	-0.896467	0.323917	-0.015165
2	1.093343	-2.020293	0.099026
3	1.971081	-0.379402	0.751828
4	-0.569713	-0.061500	-0.489205
5	-0.502000	0.545283	0.037430
6	1.407763	2.647069	0.217019
7	-1.627089	-0.195426	0.157814
8	-1.544826	-1.033046	-0.704031
9	-0.919720	-0.968175	-1.126654

In [3]:

# columns 리스트가 매우 많을때 쉽게 나열해 볼수 있는 방법
df_col_check = pd.DataFrame(df.columns)
df_col_check

Out[3]:

	0
0	one
1	two
2	three

loc vs iloc¶

iloc[ 행선택, 열선택 ] loc [ 행조건, 열선택 ]

iloc[row_num, col_num] row_num , col_num 에는 int값만 들어갈 수 있다. loc[row_select, col_select] row_select, col_select 에는 조건문이 들어갈 수 있다.

In [4]:

# shift + tab + tab 클릭시 api정보 확인 할 수 있으나
# 가끔 안뜨는 경우가 있어서 필요시 print해서 확인 한다.

print(df.loc.__doc__)

Access a group of rows and columns by label(s) or a boolean array.

``.loc[]`` is primarily label based, but may also be used with a
boolean array.

Allowed inputs are:

- A single label, e.g. ``5`` or ``'a'``, (note that ``5`` is
  interpreted as a *label* of the index, and **never** as an
  integer position along the index).
- A list or array of labels, e.g. ``['a', 'b', 'c']``.
- A slice object with labels, e.g. ``'a':'f'``.

  .. warning:: Note that contrary to usual python slices, **both** the
      start and the stop are included

- A boolean array of the same length as the axis being sliced,
  e.g. ``[True, False, True]``.
- An alignable boolean Series. The index of the key will be aligned before
  masking.
- An alignable Index. The Index of the returned selection will be the input.
- A ``callable`` function with one argument (the calling Series or
  DataFrame) and that returns valid output for indexing (one of the above)

See more at :ref:`Selection by Label <indexing.label>`.

Raises
------
KeyError
    If any items are not found.
IndexingError
    If an indexed key is passed and its index is unalignable to the frame index.

See Also
--------
DataFrame.at : Access a single value for a row/column label pair.
DataFrame.iloc : Access group of rows and columns by integer position(s).
DataFrame.xs : Returns a cross-section (row(s) or column(s)) from the
    Series/DataFrame.
Series.loc : Access group of values using labels.

Examples
--------
**Getting values**

>>> df = pd.DataFrame([[1, 2], [4, 5], [7, 8]],
...      index=['cobra', 'viper', 'sidewinder'],
...      columns=['max_speed', 'shield'])
>>> df
            max_speed  shield
cobra               1       2
viper               4       5
sidewinder          7       8

Single label. Note this returns the row as a Series.

>>> df.loc['viper']
max_speed    4
shield       5
Name: viper, dtype: int64

List of labels. Note using ``[[]]`` returns a DataFrame.

>>> df.loc[['viper', 'sidewinder']]
            max_speed  shield
viper               4       5
sidewinder          7       8

Single label for row and column

>>> df.loc['cobra', 'shield']
2

Slice with labels for row and single label for column. As mentioned
above, note that both the start and stop of the slice are included.

>>> df.loc['cobra':'viper', 'max_speed']
cobra    1
viper    4
Name: max_speed, dtype: int64

Boolean list with the same length as the row axis

>>> df.loc[[False, False, True]]
            max_speed  shield
sidewinder          7       8

Alignable boolean Series:

>>> df.loc[pd.Series([False, True, False],
...        index=['viper', 'sidewinder', 'cobra'])]
            max_speed  shield
sidewinder          7       8

Index (same behavior as ``df.reindex``)

>>> df.loc[pd.Index(["cobra", "viper"], name="foo")]
       max_speed  shield
foo
cobra          1       2
viper          4       5

Conditional that returns a boolean Series

>>> df.loc[df['shield'] > 6]
            max_speed  shield
sidewinder          7       8

Conditional that returns a boolean Series with column labels specified

>>> df.loc[df['shield'] > 6, ['max_speed']]
            max_speed
sidewinder          7

Callable that returns a boolean Series

>>> df.loc[lambda df: df['shield'] == 8]
            max_speed  shield
sidewinder          7       8

**Setting values**

Set value for all items matching the list of labels

>>> df.loc[['viper', 'sidewinder'], ['shield']] = 50
>>> df
            max_speed  shield
cobra               1       2
viper               4      50
sidewinder          7      50

Set value for an entire row

>>> df.loc['cobra'] = 10
>>> df
            max_speed  shield
cobra              10      10
viper               4      50
sidewinder          7      50

Set value for an entire column

>>> df.loc[:, 'max_speed'] = 30
>>> df
            max_speed  shield
cobra              30      10
viper              30      50
sidewinder         30      50

Set value for rows matching callable condition

>>> df.loc[df['shield'] > 35] = 0
>>> df
            max_speed  shield
cobra              30      10
viper               0       0
sidewinder          0       0

**Getting values on a DataFrame with an index that has integer labels**

Another example using integers for the index

>>> df = pd.DataFrame([[1, 2], [4, 5], [7, 8]],
...      index=[7, 8, 9], columns=['max_speed', 'shield'])
>>> df
   max_speed  shield
7          1       2
8          4       5
9          7       8

Slice with integer labels for rows. As mentioned above, note that both
the start and stop of the slice are included.

>>> df.loc[7:9]
   max_speed  shield
7          1       2
8          4       5
9          7       8

**Getting values with a MultiIndex**

A number of examples using a DataFrame with a MultiIndex

>>> tuples = [
...    ('cobra', 'mark i'), ('cobra', 'mark ii'),
...    ('sidewinder', 'mark i'), ('sidewinder', 'mark ii'),
...    ('viper', 'mark ii'), ('viper', 'mark iii')
... ]
>>> index = pd.MultiIndex.from_tuples(tuples)
>>> values = [[12, 2], [0, 4], [10, 20],
...         [1, 4], [7, 1], [16, 36]]
>>> df = pd.DataFrame(values, columns=['max_speed', 'shield'], index=index)
>>> df
                     max_speed  shield
cobra      mark i           12       2
           mark ii           0       4
sidewinder mark i           10      20
           mark ii           1       4
viper      mark ii           7       1
           mark iii         16      36

Single label. Note this returns a DataFrame with a single index.

>>> df.loc['cobra']
         max_speed  shield
mark i          12       2
mark ii          0       4

Single index tuple. Note this returns a Series.

>>> df.loc[('cobra', 'mark ii')]
max_speed    0
shield       4
Name: (cobra, mark ii), dtype: int64

Single label for row and column. Similar to passing in a tuple, this
returns a Series.

>>> df.loc['cobra', 'mark i']
max_speed    12
shield        2
Name: (cobra, mark i), dtype: int64

Single tuple. Note using ``[[]]`` returns a DataFrame.

>>> df.loc[[('cobra', 'mark ii')]]
               max_speed  shield
cobra mark ii          0       4

Single tuple for the index with a single label for the column

>>> df.loc[('cobra', 'mark i'), 'shield']
2

Slice from index tuple to single label

>>> df.loc[('cobra', 'mark i'):'viper']
                     max_speed  shield
cobra      mark i           12       2
           mark ii           0       4
sidewinder mark i           10      20
           mark ii           1       4
viper      mark ii           7       1
           mark iii         16      36

Slice from index tuple to index tuple

>>> df.loc[('cobra', 'mark i'):('viper', 'mark ii')]
                    max_speed  shield
cobra      mark i          12       2
           mark ii          0       4
sidewinder mark i          10      20
           mark ii          1       4
viper      mark ii          7       1

Please see the :ref:`user guide<advanced.advanced_hierarchical>`
for more details and explanations of advanced indexing.

In [5]:

df.loc[1]

Out[5]:

one     -0.896467
two      0.323917
three   -0.015165
Name: 1, dtype: float64

In [6]:

#df.loc[ df.index == 9 , :] # row값 9인 모든 column값, : => min:max 으로 All을 의미함
df.loc[ 0:1, :]

Out[6]:

	one	two	three
0	-1.141389	-1.597709	-0.593118
1	-0.896467	0.323917	-0.015165

In [7]:

df.loc[0:3, ['one', 'two']]

Out[7]:

	one	two
0	-1.141389	-1.597709
1	-0.896467	0.323917
2	1.093343	-2.020293
3	1.971081	-0.379402

─────────────────────────────────────¶

In [8]:

print(df.iloc.__doc__)

Purely integer-location based indexing for selection by position.

``.iloc[]`` is primarily integer position based (from ``0`` to
``length-1`` of the axis), but may also be used with a boolean
array.

Allowed inputs are:

- An integer, e.g. ``5``.
- A list or array of integers, e.g. ``[4, 3, 0]``.
- A slice object with ints, e.g. ``1:7``.
- A boolean array.
- A ``callable`` function with one argument (the calling Series or
  DataFrame) and that returns valid output for indexing (one of the above).
  This is useful in method chains, when you don't have a reference to the
  calling object, but would like to base your selection on some value.
- A tuple of row and column indexes. The tuple elements consist of one of the
  above inputs, e.g. ``(0, 1)``.

``.iloc`` will raise ``IndexError`` if a requested indexer is
out-of-bounds, except *slice* indexers which allow out-of-bounds
indexing (this conforms with python/numpy *slice* semantics).

See more at :ref:`Selection by Position <indexing.integer>`.

See Also
--------
DataFrame.iat : Fast integer location scalar accessor.
DataFrame.loc : Purely label-location based indexer for selection by label.
Series.iloc : Purely integer-location based indexing for
               selection by position.

Examples
--------
>>> mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4},
...           {'a': 100, 'b': 200, 'c': 300, 'd': 400},
...           {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }]
>>> df = pd.DataFrame(mydict)
>>> df
      a     b     c     d
0     1     2     3     4
1   100   200   300   400
2  1000  2000  3000  4000

**Indexing just the rows**

With a scalar integer.

>>> type(df.iloc[0])
<class 'pandas.core.series.Series'>
>>> df.iloc[0]
a    1
b    2
c    3
d    4
Name: 0, dtype: int64

With a list of integers.

>>> df.iloc[[0]]
   a  b  c  d
0  1  2  3  4
>>> type(df.iloc[[0]])
<class 'pandas.core.frame.DataFrame'>

>>> df.iloc[[0, 1]]
     a    b    c    d
0    1    2    3    4
1  100  200  300  400

With a `slice` object.

>>> df.iloc[:3]
      a     b     c     d
0     1     2     3     4
1   100   200   300   400
2  1000  2000  3000  4000

With a boolean mask the same length as the index.

>>> df.iloc[[True, False, True]]
      a     b     c     d
0     1     2     3     4
2  1000  2000  3000  4000

With a callable, useful in method chains. The `x` passed
to the ``lambda`` is the DataFrame being sliced. This selects
the rows whose index label even.

>>> df.iloc[lambda x: x.index % 2 == 0]
      a     b     c     d
0     1     2     3     4
2  1000  2000  3000  4000

**Indexing both axes**

You can mix the indexer types for the index and columns. Use ``:`` to
select the entire axis.

With scalar integers.

>>> df.iloc[0, 1]
2

With lists of integers.

>>> df.iloc[[0, 2], [1, 3]]
      b     d
0     2     4
2  2000  4000

With `slice` objects.

>>> df.iloc[1:3, 0:3]
      a     b     c
1   100   200   300
2  1000  2000  3000

With a boolean array whose length matches the columns.

>>> df.iloc[:, [True, False, True, False]]
      a     c
0     1     3
1   100   300
2  1000  3000

With a callable function that expects the Series or DataFrame.

>>> df.iloc[:, lambda df: [0, 2]]
      a     c
0     1     3
1   100   300
2  1000  3000

In [9]:

df.iloc[2]

Out[9]:

one      1.093343
two     -2.020293
three    0.099026
Name: 2, dtype: float64

In [10]:

df.iloc[1, 2] # row 1, cal 2 에 해당하는 값

Out[10]:

-0.015164940108393593

In [11]:

#df.iloc[:, 0:3]
df.iloc[:3, :2] # 0~3, 0~2, df.iloc[0:3, 0:2] 와 같음

Out[11]:

	one	two
0	-1.141389	-1.597709
1	-0.896467	0.323917
2	1.093343	-2.020293

In [12]:

# Column 조건문으로 행 추출하기
df["two"].values > 0.3
#df[df["two"].values > 0.3]
#df["two"].isin([-0.643932, 0.417865])

Out[12]:

array([False,  True, False, False, False,  True,  True, False, False,
       False])

NaN 결측값 다루기¶

np.nan은 결측값을 의미한다. Null과 같은 의미이다.

In [13]:

df["four"] = [np.nan, np.nan, np.nan, 4, 5, 6, 7, 8, 9, 10]
df

Out[13]:

	one	two	three	four
0	-1.141389	-1.597709	-0.593118	NaN
1	-0.896467	0.323917	-0.015165	NaN
2	1.093343	-2.020293	0.099026	NaN
3	1.971081	-0.379402	0.751828	4.0
4	-0.569713	-0.061500	-0.489205	5.0
5	-0.502000	0.545283	0.037430	6.0
6	1.407763	2.647069	0.217019	7.0
7	-1.627089	-0.195426	0.157814	8.0
8	-1.544826	-1.033046	-0.704031	9.0
9	-0.919720	-0.968175	-1.126654	10.0

In [14]:

df.dropna() # 데이터 내에 있는 모든 결측치 제거

Out[14]:

	one	two	three	four
3	1.971081	-0.379402	0.751828	4.0
4	-0.569713	-0.061500	-0.489205	5.0
5	-0.502000	0.545283	0.037430	6.0
6	1.407763	2.647069	0.217019	7.0
7	-1.627089	-0.195426	0.157814	8.0
8	-1.544826	-1.033046	-0.704031	9.0
9	-0.919720	-0.968175	-1.126654	10.0

In [15]:

df.fillna(value=-1) # nan값에 값 넣기

Out[15]:

	one	two	three	four
0	-1.141389	-1.597709	-0.593118	-1.0
1	-0.896467	0.323917	-0.015165	-1.0
2	1.093343	-2.020293	0.099026	-1.0
3	1.971081	-0.379402	0.751828	4.0
4	-0.569713	-0.061500	-0.489205	5.0
5	-0.502000	0.545283	0.037430	6.0
6	1.407763	2.647069	0.217019	7.0
7	-1.627089	-0.195426	0.157814	8.0
8	-1.544826	-1.033046	-0.704031	9.0
9	-0.919720	-0.968175	-1.126654	10.0

In [16]:

df.isnull()

Out[16]:

	one	two	three	four
0	False	False	False	True
1	False	False	False	True
2	False	False	False	True
3	False	False	False	False
4	False	False	False	False
5	False	False	False	False
6	False	False	False	False
7	False	False	False	False
8	False	False	False	False
9	False	False	False	False

In [17]:

df.isnull()["four"]

Out[17]:

0     True
1     True
2     True
3    False
4    False
5    False
6    False
7    False
8    False
9    False
Name: four, dtype: bool

In [18]:

df.loc[df.isnull()["four"],:]

Out[18]:

	one	two	three	four
0	-1.141389	-1.597709	-0.593118	NaN
1	-0.896467	0.323917	-0.015165	NaN
2	1.093343	-2.020293	0.099026	NaN

unique¶

In [19]:

df["four"].unique() # 특정 column에 중복값을 제외한 unique한 값 array

Out[19]:

array([nan,  4.,  5.,  6.,  7.,  8.,  9., 10.])

In [30]:

np.unique(df["four"])

Out[30]:

array([ 4.,  5.,  6.,  7.,  8.,  9., 10., nan])

In [31]:

len(np.unique(df["four"]))

Out[31]:

8

drop¶

In [20]:

mydict = [{'a': 1, 'b': 2, 'c': 3, 'd': 4},
          {'a': 100, 'b': 200, 'c': 300, 'd': 400},
          {'a': 1000, 'b': 2000, 'c': 3000, 'd': 4000 }]
df_dict = pd.DataFrame(mydict)
df_dict

Out[20]:

	a	b	c	d
0	1	2	3	4
1	100	200	300	400
2	1000	2000	3000	4000

In [21]:

df_dict = df_dict.drop(0, axis=0)
df_dict

Out[21]:

	a	b	c	d
1	100	200	300	400
2	1000	2000	3000	4000

In [22]:

df_dict = df_dict.drop("b", axis=1)
df_dict

Out[22]:

	a	c	d
1	100	300	400
2	1000	3000	4000