Categorical data

This is an introduction to pandas categorical data type, including a short comparisonwith R’s factor.

Categoricals are a pandas data type corresponding to categorical variables instatistics. A categorical variable takes on a limited, and usually fixed,number of possible values (categories; levels in R). Examples are gender,social class, blood type, country affiliation, observation time or rating viaLikert scales.

In contrast to statistical categorical variables, categorical data might have an order (e.g.‘strongly agree’ vs ‘agree’ or ‘first observation’ vs. ‘second observation’), but numericaloperations (additions, divisions, …) are not possible.

All values of categorical data are either in categories or np.nan. Order is defined bythe order of categories, not lexical order of the values. Internally, the data structureconsists of a categories array and an integer array of codes which point to the real value inthe categories array.

The categorical data type is useful in the following cases:

  • A string variable consisting of only a few different values. Converting such a stringvariable to a categorical variable will save some memory, see here.
  • The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”).By converting to a categorical and specifying an order on the categories, sorting andmin/max will use the logical order instead of the lexical order, see here.
  • As a signal to other Python libraries that this column should be treated as a categoricalvariable (e.g. to use suitable statistical methods or plot types).

See also the API docs on categoricals.

Object creation

Series creation

Categorical Series or columns in a DataFrame can be created in several ways:

By specifying dtype="category" when constructing a Series:

  1. In [1]: s = pd.Series(["a", "b", "c", "a"], dtype="category")
  2.  
  3. In [2]: s
  4. Out[2]: 
  5. 0    a
  6. 1    b
  7. 2    c
  8. 3    a
  9. dtype: category
  10. Categories (3, object): [a, b, c]

By converting an existing Series or column to a category dtype:

  1. In [3]: df = pd.DataFrame({"A": ["a", "b", "c", "a"]})
  2.  
  3. In [4]: df["B"] = df["A"].astype('category')
  4.  
  5. In [5]: df
  6. Out[5]: 
  7.    A  B
  8. 0  a  a
  9. 1  b  b
  10. 2  c  c
  11. 3  a  a

By using special functions, such as cut(), which groups data intodiscrete bins. See the example on tiling in the docs.

  1. In [6]: df = pd.DataFrame({'value': np.random.randint(0, 100, 20)})
  2.  
  3. In [7]: labels = ["{0} - {1}".format(i, i + 9) for i in range(0, 100, 10)]
  4.  
  5. In [8]: df['group'] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)
  6.  
  7. In [9]: df.head(10)
  8. Out[9]: 
  9.    value    group
  10. 0     65  60 - 69
  11. 1     49  40 - 49
  12. 2     56  50 - 59
  13. 3     43  40 - 49
  14. 4     43  40 - 49
  15. 5     91  90 - 99
  16. 6     32  30 - 39
  17. 7     87  80 - 89
  18. 8     36  30 - 39
  19. 9      8    0 - 9

By passing a pandas.Categorical object to a Series or assigning it to a DataFrame.

  1. In [10]: raw_cat = pd.Categorical(["a", "b", "c", "a"], categories=["b", "c", "d"],
  2.    ....:                          ordered=False)
  3.    ....: 
  4.  
  5. In [11]: s = pd.Series(raw_cat)
  6.  
  7. In [12]: s
  8. Out[12]: 
  9. 0    NaN
  10. 1      b
  11. 2      c
  12. 3    NaN
  13. dtype: category
  14. Categories (3, object): [b, c, d]
  15.  
  16. In [13]: df = pd.DataFrame({"A": ["a", "b", "c", "a"]})
  17.  
  18. In [14]: df["B"] = raw_cat
  19.  
  20. In [15]: df
  21. Out[15]: 
  22.    A    B
  23. 0  a  NaN
  24. 1  b    b
  25. 2  c    c
  26. 3  a  NaN

Categorical data has a specific category dtype:

  1. In [16]: df.dtypes
  2. Out[16]: 
  3. A      object
  4. B    category
  5. dtype: object

DataFrame creation

Similar to the previous section where a single column was converted to categorical, all columns in aDataFrame can be batch converted to categorical either during or after construction.

This can be done during construction by specifying dtype="category" in the DataFrame constructor:

  1. In [17]: df = pd.DataFrame({'A': list('abca'), 'B': list('bccd')}, dtype="category")
  2.  
  3. In [18]: df.dtypes
  4. Out[18]: 
  5. A    category
  6. B    category
  7. dtype: object

Note that the categories present in each column differ; the conversion is done column by column, soonly labels present in a given column are categories:

  1. In [19]: df['A']
  2. Out[19]: 
  3. 0    a
  4. 1    b
  5. 2    c
  6. 3    a
  7. Name: A, dtype: category
  8. Categories (3, object): [a, b, c]
  9.  
  10. In [20]: df['B']
  11. Out[20]: 
  12. 0    b
  13. 1    c
  14. 2    c
  15. 3    d
  16. Name: B, dtype: category
  17. Categories (3, object): [b, c, d]

New in version 0.23.0.

Analogously, all columns in an existing DataFrame can be batch converted using DataFrame.astype():

  1. In [21]: df = pd.DataFrame({'A': list('abca'), 'B': list('bccd')})
  2.  
  3. In [22]: df_cat = df.astype('category')
  4.  
  5. In [23]: df_cat.dtypes
  6. Out[23]: 
  7. A    category
  8. B    category
  9. dtype: object

This conversion is likewise done column by column:

  1. In [24]: df_cat['A']
  2. Out[24]: 
  3. 0    a
  4. 1    b
  5. 2    c
  6. 3    a
  7. Name: A, dtype: category
  8. Categories (3, object): [a, b, c]
  9.  
  10. In [25]: df_cat['B']
  11. Out[25]: 
  12. 0    b
  13. 1    c
  14. 2    c
  15. 3    d
  16. Name: B, dtype: category
  17. Categories (3, object): [b, c, d]

Controlling behavior

In the examples above where we passed dtype='category', we used the defaultbehavior:

  • Categories are inferred from the data.
  • Categories are unordered.To control those behaviors, instead of passing 'category', use an instanceof CategoricalDtype.
  1. In [26]: from pandas.api.types import CategoricalDtype
  2.  
  3. In [27]: s = pd.Series(["a", "b", "c", "a"])
  4.  
  5. In [28]: cat_type = CategoricalDtype(categories=["b", "c", "d"],
  6.    ....:                             ordered=True)
  7.    ....: 
  8.  
  9. In [29]: s_cat = s.astype(cat_type)
  10.  
  11. In [30]: s_cat
  12. Out[30]: 
  13. 0    NaN
  14. 1      b
  15. 2      c
  16. 3    NaN
  17. dtype: category
  18. Categories (3, object): [b < c < d]

Similarly, a CategoricalDtype can be used with a DataFrame to ensure that categoriesare consistent among all columns.

  1. In [31]: from pandas.api.types import CategoricalDtype
  2.  
  3. In [32]: df = pd.DataFrame({'A': list('abca'), 'B': list('bccd')})
  4.  
  5. In [33]: cat_type = CategoricalDtype(categories=list('abcd'),
  6.    ....:                             ordered=True)
  7.    ....: 
  8.  
  9. In [34]: df_cat = df.astype(cat_type)
  10.  
  11. In [35]: df_cat['A']
  12. Out[35]: 
  13. 0    a
  14. 1    b
  15. 2    c
  16. 3    a
  17. Name: A, dtype: category
  18. Categories (4, object): [a < b < c < d]
  19.  
  20. In [36]: df_cat['B']
  21. Out[36]: 
  22. 0    b
  23. 1    c
  24. 2    c
  25. 3    d
  26. Name: B, dtype: category
  27. Categories (4, object): [a < b < c < d]

Note

To perform table-wise conversion, where all labels in the entire DataFrame are used ascategories for each column, the categories parameter can be determined programmatically bycategories = pd.unique(df.to_numpy().ravel()).

If you already have codes and categories, you can use thefrom_codes() constructor to save the factorize stepduring normal constructor mode:

  1. In [37]: splitter = np.random.choice([0, 1], 5, p=[0.5, 0.5])
  2.  
  3. In [38]: s = pd.Series(pd.Categorical.from_codes(splitter,
  4.    ....:                                         categories=["train", "test"]))
  5.    ....:

Regaining original data

To get back to the original Series or NumPy array, useSeries.astype(original_dtype) or np.asarray(categorical):

  1. In [39]: s = pd.Series(["a", "b", "c", "a"])
  2.  
  3. In [40]: s
  4. Out[40]: 
  5. 0    a
  6. 1    b
  7. 2    c
  8. 3    a
  9. dtype: object
  10.  
  11. In [41]: s2 = s.astype('category')
  12.  
  13. In [42]: s2
  14. Out[42]: 
  15. 0    a
  16. 1    b
  17. 2    c
  18. 3    a
  19. dtype: category
  20. Categories (3, object): [a, b, c]
  21.  
  22. In [43]: s2.astype(str)
  23. Out[43]: 
  24. 0    a
  25. 1    b
  26. 2    c
  27. 3    a
  28. dtype: object
  29.  
  30. In [44]: np.asarray(s2)
  31. Out[44]: array(['a', 'b', 'c', 'a'], dtype=object)

Note

In contrast to R’s factor function, categorical data is not converting input values tostrings; categories will end up the same data type as the original values.

Note

In contrast to R’s factor function, there is currently no way to assign/change labels atcreation time. Use categories to change the categories after creation time.

CategoricalDtype

Changed in version 0.21.0.

A categorical’s type is fully described by

  • categories: a sequence of unique values and no missing values
  • ordered: a booleanThis information can be stored in a CategoricalDtype.The categories argument is optional, which implies that the actual categoriesshould be inferred from whatever is present in the data when thepandas.Categorical is created. The categories are assumed to be unorderedby default.
  1. In [45]: from pandas.api.types import CategoricalDtype
  2.  
  3. In [46]: CategoricalDtype(['a', 'b', 'c'])
  4. Out[46]: CategoricalDtype(categories=['a', 'b', 'c'], ordered=None)
  5.  
  6. In [47]: CategoricalDtype(['a', 'b', 'c'], ordered=True)
  7. Out[47]: CategoricalDtype(categories=['a', 'b', 'c'], ordered=True)
  8.  
  9. In [48]: CategoricalDtype()
  10. Out[48]: CategoricalDtype(categories=None, ordered=None)

A CategoricalDtype can be used in any place pandasexpects a dtype. For example pandas.read_csv(),pandas.DataFrame.astype(), or in the Series constructor.

Note

As a convenience, you can use the string 'category' in place of aCategoricalDtype when you want the default behavior ofthe categories being unordered, and equal to the set values present in thearray. In other words, dtype='category' is equivalent todtype=CategoricalDtype().

Equality semantics

Two instances of CategoricalDtype compare equalwhenever they have the same categories and order. When comparing twounordered categoricals, the order of the categories is not considered.

  1. In [49]: c1 = CategoricalDtype(['a', 'b', 'c'], ordered=False)
  2.  
  3. # Equal, since order is not considered when ordered=False
  4. In [50]: c1 == CategoricalDtype(['b', 'c', 'a'], ordered=False)
  5. Out[50]: True
  6.  
  7. # Unequal, since the second CategoricalDtype is ordered
  8. In [51]: c1 == CategoricalDtype(['a', 'b', 'c'], ordered=True)
  9. Out[51]: False

All instances of CategoricalDtype compare equal to the string 'category'.

  1. In [52]: c1 == 'category'
  2. Out[52]: True

Warning

Since dtype='category' is essentially CategoricalDtype(None, False),and since all instances CategoricalDtype compare equal to 'category',all instances of CategoricalDtype compare equal to aCategoricalDtype(None, False), regardless of categories orordered.

Description

Using describe() on categorical data will produce similaroutput to a Series or DataFrame of type string.

  1. In [53]: cat = pd.Categorical(["a", "c", "c", np.nan], categories=["b", "a", "c"])
  2.  
  3. In [54]: df = pd.DataFrame({"cat": cat, "s": ["a", "c", "c", np.nan]})
  4.  
  5. In [55]: df.describe()
  6. Out[55]: 
  7.        cat  s
  8. count    3  3
  9. unique   2  2
  10. top      c  c
  11. freq     2  2
  12.  
  13. In [56]: df["cat"].describe()
  14. Out[56]: 
  15. count     3
  16. unique    2
  17. top       c
  18. freq      2
  19. Name: cat, dtype: object

Working with categories

Categorical data has a categories and a ordered property, which list theirpossible values and whether the ordering matters or not. These properties areexposed as s.cat.categories and s.cat.ordered. If you don’t manuallyspecify categories and ordering, they are inferred from the passed arguments.

  1. In [57]: s = pd.Series(["a", "b", "c", "a"], dtype="category")
  2.  
  3. In [58]: s.cat.categories
  4. Out[58]: Index(['a', 'b', 'c'], dtype='object')
  5.  
  6. In [59]: s.cat.ordered
  7. Out[59]: False

It’s also possible to pass in the categories in a specific order:

  1. In [60]: s = pd.Series(pd.Categorical(["a", "b", "c", "a"],
  2.    ....:               categories=["c", "b", "a"]))
  3.    ....: 
  4.  
  5. In [61]: s.cat.categories
  6. Out[61]: Index(['c', 'b', 'a'], dtype='object')
  7.  
  8. In [62]: s.cat.ordered
  9. Out[62]: False

Note

New categorical data are not automatically ordered. You must explicitlypass ordered=True to indicate an ordered Categorical.

Note

The result of unique() is not always the same as Series.cat.categories,because Series.unique() has a couple of guarantees, namely that it returns categoriesin the order of appearance, and it only includes values that are actually present.

  1. In [63]: s = pd.Series(list('babc')).astype(CategoricalDtype(list('abcd')))
  2.  
  3. In [64]: s
  4. Out[64]: 
  5. 0    b
  6. 1    a
  7. 2    b
  8. 3    c
  9. dtype: category
  10. Categories (4, object): [a, b, c, d]
  11.  
  12. # categories
  13. In [65]: s.cat.categories
  14. Out[65]: Index(['a', 'b', 'c', 'd'], dtype='object')
  15.  
  16. # uniques
  17. In [66]: s.unique()
  18. Out[66]: 
  19. [b, a, c]
  20. Categories (3, object): [b, a, c]

Renaming categories

Renaming categories is done by assigning new values to theSeries.cat.categories property or by using therename_categories() method:

  1. In [67]: s = pd.Series(["a", "b", "c", "a"], dtype="category")
  2.  
  3. In [68]: s
  4. Out[68]: 
  5. 0    a
  6. 1    b
  7. 2    c
  8. 3    a
  9. dtype: category
  10. Categories (3, object): [a, b, c]
  11.  
  12. In [69]: s.cat.categories = ["Group %s" % g for g in s.cat.categories]
  13.  
  14. In [70]: s
  15. Out[70]: 
  16. 0    Group a
  17. 1    Group b
  18. 2    Group c
  19. 3    Group a
  20. dtype: category
  21. Categories (3, object): [Group a, Group b, Group c]
  22.  
  23. In [71]: s = s.cat.rename_categories([1, 2, 3])
  24.  
  25. In [72]: s
  26. Out[72]: 
  27. 0    1
  28. 1    2
  29. 2    3
  30. 3    1
  31. dtype: category
  32. Categories (3, int64): [1, 2, 3]
  33.  
  34. # You can also pass a dict-like object to map the renaming
  35. In [73]: s = s.cat.rename_categories({1: 'x', 2: 'y', 3: 'z'})
  36.  
  37. In [74]: s
  38. Out[74]: 
  39. 0    x
  40. 1    y
  41. 2    z
  42. 3    x
  43. dtype: category
  44. Categories (3, object): [x, y, z]

Note

In contrast to R’s factor, categorical data can have categories of other types than string.

Note

Be aware that assigning new categories is an inplace operation, while most other operationsunder Series.cat per default return a new Series of dtype category.

Categories must be unique or a ValueError is raised:

  1. In [75]: try:
  2.    ....:     s.cat.categories = [1, 1, 1]
  3.    ....: except ValueError as e:
  4.    ....:     print("ValueError:", str(e))
  5.    ....: 
  6. ValueError: Categorical categories must be unique

Categories must also not be NaN or a ValueError is raised:

  1. In [76]: try:
  2.    ....:     s.cat.categories = [1, 2, np.nan]
  3.    ....: except ValueError as e:
  4.    ....:     print("ValueError:", str(e))
  5.    ....: 
  6. ValueError: Categorial categories cannot be null

Appending new categories

Appending categories can be done by using theadd_categories() method:

  1. In [77]: s = s.cat.add_categories([4])
  2.  
  3. In [78]: s.cat.categories
  4. Out[78]: Index(['x', 'y', 'z', 4], dtype='object')
  5.  
  6. In [79]: s
  7. Out[79]: 
  8. 0    x
  9. 1    y
  10. 2    z
  11. 3    x
  12. dtype: category
  13. Categories (4, object): [x, y, z, 4]

Removing categories

Removing categories can be done by using theremove_categories() method. Values which are removedare replaced by np.nan.:

  1. In [80]: s = s.cat.remove_categories([4])
  2.  
  3. In [81]: s
  4. Out[81]: 
  5. 0    x
  6. 1    y
  7. 2    z
  8. 3    x
  9. dtype: category
  10. Categories (3, object): [x, y, z]

Removing unused categories

Removing unused categories can also be done:

  1. In [82]: s = pd.Series(pd.Categorical(["a", "b", "a"],
  2.    ....:               categories=["a", "b", "c", "d"]))
  3.    ....: 
  4.  
  5. In [83]: s
  6. Out[83]: 
  7. 0    a
  8. 1    b
  9. 2    a
  10. dtype: category
  11. Categories (4, object): [a, b, c, d]
  12.  
  13. In [84]: s.cat.remove_unused_categories()
  14. Out[84]: 
  15. 0    a
  16. 1    b
  17. 2    a
  18. dtype: category
  19. Categories (2, object): [a, b]

Setting categories

If you want to do remove and add new categories in one step (which has somespeed advantage), or simply set the categories to a predefined scale,use set_categories().

  1. In [85]: s = pd.Series(["one", "two", "four", "-"], dtype="category")
  2.  
  3. In [86]: s
  4. Out[86]: 
  5. 0     one
  6. 1     two
  7. 2    four
  8. 3       -
  9. dtype: category
  10. Categories (4, object): [-, four, one, two]
  11.  
  12. In [87]: s = s.cat.set_categories(["one", "two", "three", "four"])
  13.  
  14. In [88]: s
  15. Out[88]: 
  16. 0     one
  17. 1     two
  18. 2    four
  19. 3     NaN
  20. dtype: category
  21. Categories (4, object): [one, two, three, four]

Note

Be aware that Categorical.set_categories() cannot know whether some category is omittedintentionally or because it is misspelled or (under Python3) due to a type difference (e.g.,NumPy S1 dtype and Python strings). This can result in surprising behaviour!

Sorting and order

If categorical data is ordered (s.cat.ordered == True), then the order of the categories has ameaning and certain operations are possible. If the categorical is unordered, .min()/.max() will raise a TypeError.

  1. In [89]: s = pd.Series(pd.Categorical(["a", "b", "c", "a"], ordered=False))
  2.  
  3. In [90]: s.sort_values(inplace=True)
  4.  
  5. In [91]: s = pd.Series(["a", "b", "c", "a"]).astype(
  6.    ....:     CategoricalDtype(ordered=True)
  7.    ....: )
  8.    ....: 
  9.  
  10. In [92]: s.sort_values(inplace=True)
  11.  
  12. In [93]: s
  13. Out[93]: 
  14. 0    a
  15. 3    a
  16. 1    b
  17. 2    c
  18. dtype: category
  19. Categories (3, object): [a < b < c]
  20.  
  21. In [94]: s.min(), s.max()
  22. Out[94]: ('a', 'c')

You can set categorical data to be ordered by using asordered() or unordered by using as_unordered(). These will bydefault return a _new object.

  1. In [95]: s.cat.as_ordered()
  2. Out[95]: 
  3. 0    a
  4. 3    a
  5. 1    b
  6. 2    c
  7. dtype: category
  8. Categories (3, object): [a < b < c]
  9.  
  10. In [96]: s.cat.as_unordered()
  11. Out[96]: 
  12. 0    a
  13. 3    a
  14. 1    b
  15. 2    c
  16. dtype: category
  17. Categories (3, object): [a, b, c]

Sorting will use the order defined by categories, not any lexical order present on the data type.This is even true for strings and numeric data:

  1. In [97]: s = pd.Series([1, 2, 3, 1], dtype="category")
  2.  
  3. In [98]: s = s.cat.set_categories([2, 3, 1], ordered=True)
  4.  
  5. In [99]: s
  6. Out[99]: 
  7. 0    1
  8. 1    2
  9. 2    3
  10. 3    1
  11. dtype: category
  12. Categories (3, int64): [2 < 3 < 1]
  13.  
  14. In [100]: s.sort_values(inplace=True)
  15.  
  16. In [101]: s
  17. Out[101]: 
  18. 1    2
  19. 2    3
  20. 0    1
  21. 3    1
  22. dtype: category
  23. Categories (3, int64): [2 < 3 < 1]
  24.  
  25. In [102]: s.min(), s.max()
  26. Out[102]: (2, 1)

Reordering

Reordering the categories is possible via the Categorical.reorder_categories() andthe Categorical.set_categories() methods. For Categorical.reorder_categories(), allold categories must be included in the new categories and no new categories are allowed. This willnecessarily make the sort order the same as the categories order.

  1. In [103]: s = pd.Series([1, 2, 3, 1], dtype="category")
  2.  
  3. In [104]: s = s.cat.reorder_categories([2, 3, 1], ordered=True)
  4.  
  5. In [105]: s
  6. Out[105]: 
  7. 0    1
  8. 1    2
  9. 2    3
  10. 3    1
  11. dtype: category
  12. Categories (3, int64): [2 < 3 < 1]
  13.  
  14. In [106]: s.sort_values(inplace=True)
  15.  
  16. In [107]: s
  17. Out[107]: 
  18. 1    2
  19. 2    3
  20. 0    1
  21. 3    1
  22. dtype: category
  23. Categories (3, int64): [2 < 3 < 1]
  24.  
  25. In [108]: s.min(), s.max()
  26. Out[108]: (2, 1)

Note

Note the difference between assigning new categories and reordering the categories: the firstrenames categories and therefore the individual values in the Series, but if the firstposition was sorted last, the renamed value will still be sorted last. Reordering means that theway values are sorted is different afterwards, but not that individual values in theSeries are changed.

Note

If the Categorical is not ordered, Series.min() and Series.max() will raiseTypeError. Numeric operations like +, -, *, / and operations based on them(e.g. Series.median(), which would need to compute the mean between two values if the lengthof an array is even) do not work and raise a TypeError.

Multi column sorting

A categorical dtyped column will participate in a multi-column sort in a similar manner to other columns.The ordering of the categorical is determined by the categories of that column.

  1. In [109]: dfs = pd.DataFrame({'A': pd.Categorical(list('bbeebbaa'),
  2.    .....:                                         categories=['e', 'a', 'b'],
  3.    .....:                                         ordered=True),
  4.    .....:                     'B': [1, 2, 1, 2, 2, 1, 2, 1]})
  5.    .....: 
  6.  
  7. In [110]: dfs.sort_values(by=['A', 'B'])
  8. Out[110]: 
  9.    A  B
  10. 2  e  1
  11. 3  e  2
  12. 7  a  1
  13. 6  a  2
  14. 0  b  1
  15. 5  b  1
  16. 1  b  2
  17. 4  b  2

Reordering the categories changes a future sort.

  1. In [111]: dfs['A'] = dfs['A'].cat.reorder_categories(['a', 'b', 'e'])
  2.  
  3. In [112]: dfs.sort_values(by=['A', 'B'])
  4. Out[112]: 
  5.    A  B
  6. 7  a  1
  7. 6  a  2
  8. 0  b  1
  9. 5  b  1
  10. 1  b  2
  11. 4  b  2
  12. 2  e  1
  13. 3  e  2

Comparisons

Comparing categorical data with other objects is possible in three cases:

  • Comparing equality (== and !=) to a list-like object (list, Series, array,…) of the same length as the categorical data.
  • All comparisons (==, !=, >, >=, <, and <=) of categorical data toanother categorical Series, when ordered==True and the categories are the same.
  • All comparisons of a categorical data to a scalar.

All other comparisons, especially “non-equality” comparisons of two categoricals with differentcategories or a categorical with any list-like object, will raise a TypeError.

Note

Any “non-equality” comparisons of categorical data with a Series, np.array, list orcategorical data with different categories or ordering will raise a TypeError because customcategories ordering could be interpreted in two ways: one with taking into account theordering and one without.

  1. In [113]: cat = pd.Series([1, 2, 3]).astype(
  2.    .....:     CategoricalDtype([3, 2, 1], ordered=True)
  3.    .....: )
  4.    .....: 
  5.  
  6. In [114]: cat_base = pd.Series([2, 2, 2]).astype(
  7.    .....:     CategoricalDtype([3, 2, 1], ordered=True)
  8.    .....: )
  9.    .....: 
  10.  
  11. In [115]: cat_base2 = pd.Series([2, 2, 2]).astype(
  12.    .....:     CategoricalDtype(ordered=True)
  13.    .....: )
  14.    .....: 
  15.  
  16. In [116]: cat
  17. Out[116]: 
  18. 0    1
  19. 1    2
  20. 2    3
  21. dtype: category
  22. Categories (3, int64): [3 < 2 < 1]
  23.  
  24. In [117]: cat_base
  25. Out[117]: 
  26. 0    2
  27. 1    2
  28. 2    2
  29. dtype: category
  30. Categories (3, int64): [3 < 2 < 1]
  31.  
  32. In [118]: cat_base2
  33. Out[118]: 
  34. 0    2
  35. 1    2
  36. 2    2
  37. dtype: category
  38. Categories (1, int64): [2]

Comparing to a categorical with the same categories and ordering or to a scalar works:

  1. In [119]: cat > cat_base
  2. Out[119]: 
  3. 0     True
  4. 1    False
  5. 2    False
  6. dtype: bool
  7.  
  8. In [120]: cat > 2
  9. Out[120]: 
  10. 0     True
  11. 1    False
  12. 2    False
  13. dtype: bool

Equality comparisons work with any list-like object of same length and scalars:

  1. In [121]: cat == cat_base
  2. Out[121]: 
  3. 0    False
  4. 1     True
  5. 2    False
  6. dtype: bool
  7.  
  8. In [122]: cat == np.array([1, 2, 3])
  9. Out[122]: 
  10. 0    True
  11. 1    True
  12. 2    True
  13. dtype: bool
  14.  
  15. In [123]: cat == 2
  16. Out[123]: 
  17. 0    False
  18. 1     True
  19. 2    False
  20. dtype: bool

This doesn’t work because the categories are not the same:

  1. In [124]: try:
  2.    .....:     cat > cat_base2
  3.    .....: except TypeError as e:
  4.    .....:     print("TypeError:", str(e))
  5.    .....: 
  6. TypeError: Categoricals can only be compared if 'categories' are the same. Categories are different lengths

If you want to do a “non-equality” comparison of a categorical series with a list-like objectwhich is not categorical data, you need to be explicit and convert the categorical data back tothe original values:

  1. In [125]: base = np.array([1, 2, 3])
  2.  
  3. In [126]: try:
  4.    .....:     cat > base
  5.    .....: except TypeError as e:
  6.    .....:     print("TypeError:", str(e))
  7.    .....: 
  8. TypeError: Cannot compare a Categorical for op __gt__ with type <class 'numpy.ndarray'>.
  9. If you want to compare values, use 'np.asarray(cat) <op> other'.
  10.  
  11. In [127]: np.asarray(cat) > base
  12. Out[127]: array([False, False, False])

When you compare two unordered categoricals with the same categories, the order is not considered:

  1. In [128]: c1 = pd.Categorical(['a', 'b'], categories=['a', 'b'], ordered=False)
  2.  
  3. In [129]: c2 = pd.Categorical(['a', 'b'], categories=['b', 'a'], ordered=False)
  4.  
  5. In [130]: c1 == c2
  6. Out[130]: array([ True,  True])

Operations

Apart from Series.min(), Series.max() and Series.mode(), thefollowing operations are possible with categorical data:

Series methods like Series.value_counts() will use all categories,even if some categories are not present in the data:

  1. In [131]: s = pd.Series(pd.Categorical(["a", "b", "c", "c"],
  2.    .....:               categories=["c", "a", "b", "d"]))
  3.    .....: 
  4.  
  5. In [132]: s.value_counts()
  6. Out[132]: 
  7. c    2
  8. b    1
  9. a    1
  10. d    0
  11. dtype: int64

Groupby will also show “unused” categories:

  1. In [133]: cats = pd.Categorical(["a", "b", "b", "b", "c", "c", "c"],
  2.    .....:                       categories=["a", "b", "c", "d"])
  3.    .....: 
  4.  
  5. In [134]: df = pd.DataFrame({"cats": cats, "values": [1, 2, 2, 2, 3, 4, 5]})
  6.  
  7. In [135]: df.groupby("cats").mean()
  8. Out[135]: 
  9.       values
  10. cats        
  11. a        1.0
  12. b        2.0
  13. c        4.0
  14. d        NaN
  15.  
  16. In [136]: cats2 = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])
  17.  
  18. In [137]: df2 = pd.DataFrame({"cats": cats2,
  19.    .....:                     "B": ["c", "d", "c", "d"],
  20.    .....:                     "values": [1, 2, 3, 4]})
  21.    .....: 
  22.  
  23. In [138]: df2.groupby(["cats", "B"]).mean()
  24. Out[138]: 
  25.         values
  26. cats B        
  27. a    c     1.0
  28.      d     2.0
  29. b    c     3.0
  30.      d     4.0
  31. c    c     NaN
  32.      d     NaN

Pivot tables:

  1. In [139]: raw_cat = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])
  2.  
  3. In [140]: df = pd.DataFrame({"A": raw_cat,
  4.    .....:                    "B": ["c", "d", "c", "d"],
  5.    .....:                    "values": [1, 2, 3, 4]})
  6.    .....: 
  7.  
  8. In [141]: pd.pivot_table(df, values='values', index=['A', 'B'])
  9. Out[141]: 
  10.      values
  11. A B        
  12. a c       1
  13.   d       2
  14. b c       3
  15.   d       4

Data munging

The optimized pandas data access methods .loc, .iloc, .at, and .iat,work as normal. The only difference is the return type (for getting) andthat only values already in categories can be assigned.

Getting

If the slicing operation returns either a DataFrame or a column of typeSeries, the category dtype is preserved.

  1. In [142]: idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])
  2.  
  3. In [143]: cats = pd.Series(["a", "b", "b", "b", "c", "c", "c"],
  4.    .....:                  dtype="category", index=idx)
  5.    .....: 
  6.  
  7. In [144]: values = [1, 2, 2, 2, 3, 4, 5]
  8.  
  9. In [145]: df = pd.DataFrame({"cats": cats, "values": values}, index=idx)
  10.  
  11. In [146]: df.iloc[2:4, :]
  12. Out[146]: 
  13.   cats  values
  14. j    b       2
  15. k    b       2
  16.  
  17. In [147]: df.iloc[2:4, :].dtypes
  18. Out[147]: 
  19. cats      category
  20. values       int64
  21. dtype: object
  22.  
  23. In [148]: df.loc["h":"j", "cats"]
  24. Out[148]: 
  25. h    a
  26. i    b
  27. j    b
  28. Name: cats, dtype: category
  29. Categories (3, object): [a, b, c]
  30.  
  31. In [149]: df[df["cats"] == "b"]
  32. Out[149]: 
  33.   cats  values
  34. i    b       2
  35. j    b       2
  36. k    b       2

An example where the category type is not preserved is if you take one singlerow: the resulting Series is of dtype object:

  1. # get the complete "h" row as a Series
  2. In [150]: df.loc["h", :]
  3. Out[150]: 
  4. cats      a
  5. values    1
  6. Name: h, dtype: object

Returning a single item from categorical data will also return the value, not a categoricalof length “1”.

  1. In [151]: df.iat[0, 0]
  2. Out[151]: 'a'
  3.  
  4. In [152]: df["cats"].cat.categories = ["x", "y", "z"]
  5.  
  6. In [153]: df.at["h", "cats"]  # returns a string
  7. Out[153]: 'x'

Note

The is in contrast to R’s factor function, where factor(c(1,2,3))[1]returns a single value factor.

To get a single value Series of type category, you pass in a list witha single value:

  1. In [154]: df.loc[["h"], "cats"]
  2. Out[154]: 
  3. h    x
  4. Name: cats, dtype: category
  5. Categories (3, object): [x, y, z]

String and datetime accessors

The accessors .dt and .str will work if the s.cat.categories are ofan appropriate type:

  1. In [155]: str_s = pd.Series(list('aabb'))
  2.  
  3. In [156]: str_cat = str_s.astype('category')
  4.  
  5. In [157]: str_cat
  6. Out[157]: 
  7. 0    a
  8. 1    a
  9. 2    b
  10. 3    b
  11. dtype: category
  12. Categories (2, object): [a, b]
  13.  
  14. In [158]: str_cat.str.contains("a")
  15. Out[158]: 
  16. 0     True
  17. 1     True
  18. 2    False
  19. 3    False
  20. dtype: bool
  21.  
  22. In [159]: date_s = pd.Series(pd.date_range('1/1/2015', periods=5))
  23.  
  24. In [160]: date_cat = date_s.astype('category')
  25.  
  26. In [161]: date_cat
  27. Out[161]: 
  28. 0   2015-01-01
  29. 1   2015-01-02
  30. 2   2015-01-03
  31. 3   2015-01-04
  32. 4   2015-01-05
  33. dtype: category
  34. Categories (5, datetime64[ns]): [2015-01-01, 2015-01-02, 2015-01-03, 2015-01-04, 2015-01-05]
  35.  
  36. In [162]: date_cat.dt.day
  37. Out[162]: 
  38. 0    1
  39. 1    2
  40. 2    3
  41. 3    4
  42. 4    5
  43. dtype: int64

Note

The returned Series (or DataFrame) is of the same type as if you used the.str.<method> / .dt.<method> on a Series of that type (and not oftype category!).

That means, that the returned values from methods and properties on the accessors of aSeries and the returned values from methods and properties on the accessors of thisSeries transformed to one of type category will be equal:

  1. In [163]: ret_s = str_s.str.contains("a")
  2.  
  3. In [164]: ret_cat = str_cat.str.contains("a")
  4.  
  5. In [165]: ret_s.dtype == ret_cat.dtype
  6. Out[165]: True
  7.  
  8. In [166]: ret_s == ret_cat
  9. Out[166]: 
  10. 0    True
  11. 1    True
  12. 2    True
  13. 3    True
  14. dtype: bool

Note

The work is done on the categories and then a new Series is constructed. This hassome performance implication if you have a Series of type string, where lots of elementsare repeated (i.e. the number of unique elements in the Series is a lot smaller than thelength of the Series). In this case it can be faster to convert the original Seriesto one of type category and use .str.<method> or .dt.<property> on that.

Setting

Setting values in a categorical column (or Series) works as long as thevalue is included in the categories:

  1. In [167]: idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])
  2.  
  3. In [168]: cats = pd.Categorical(["a", "a", "a", "a", "a", "a", "a"],
  4.    .....:                       categories=["a", "b"])
  5.    .....: 
  6.  
  7. In [169]: values = [1, 1, 1, 1, 1, 1, 1]
  8.  
  9. In [170]: df = pd.DataFrame({"cats": cats, "values": values}, index=idx)
  10.  
  11. In [171]: df.iloc[2:4, :] = [["b", 2], ["b", 2]]
  12.  
  13. In [172]: df
  14. Out[172]: 
  15.   cats  values
  16. h    a       1
  17. i    a       1
  18. j    b       2
  19. k    b       2
  20. l    a       1
  21. m    a       1
  22. n    a       1
  23.  
  24. In [173]: try:
  25.    .....:     df.iloc[2:4, :] = [["c", 3], ["c", 3]]
  26.    .....: except ValueError as e:
  27.    .....:     print("ValueError:", str(e))
  28.    .....: 
  29. ValueError: Cannot setitem on a Categorical with a new category, set the categories first

Setting values by assigning categorical data will also check that the categories match:

  1. In [174]: df.loc["j":"k", "cats"] = pd.Categorical(["a", "a"], categories=["a", "b"])
  2.  
  3. In [175]: df
  4. Out[175]: 
  5.   cats  values
  6. h    a       1
  7. i    a       1
  8. j    a       2
  9. k    a       2
  10. l    a       1
  11. m    a       1
  12. n    a       1
  13.  
  14. In [176]: try:
  15.    .....:     df.loc["j":"k", "cats"] = pd.Categorical(["b", "b"],
  16.    .....:                                              categories=["a", "b", "c"])
  17.    .....: except ValueError as e:
  18.    .....:     print("ValueError:", str(e))
  19.    .....: 
  20. ValueError: Cannot set a Categorical with another, without identical categories

Assigning a Categorical to parts of a column of other types will use the values:

  1. In [177]: df = pd.DataFrame({"a": [1, 1, 1, 1, 1], "b": ["a", "a", "a", "a", "a"]})
  2.  
  3. In [178]: df.loc[1:2, "a"] = pd.Categorical(["b", "b"], categories=["a", "b"])
  4.  
  5. In [179]: df.loc[2:3, "b"] = pd.Categorical(["b", "b"], categories=["a", "b"])
  6.  
  7. In [180]: df
  8. Out[180]: 
  9.    a  b
  10. 0  1  a
  11. 1  b  a
  12. 2  b  b
  13. 3  1  b
  14. 4  1  a
  15.  
  16. In [181]: df.dtypes
  17. Out[181]: 
  18. a    object
  19. b    object
  20. dtype: object

Merging

You can concat two DataFrames containing categorical data together,but the categories of these categoricals need to be the same:

  1. In [182]: cat = pd.Series(["a", "b"], dtype="category")
  2.  
  3. In [183]: vals = [1, 2]
  4.  
  5. In [184]: df = pd.DataFrame({"cats": cat, "vals": vals})
  6.  
  7. In [185]: res = pd.concat([df, df])
  8.  
  9. In [186]: res
  10. Out[186]: 
  11.   cats  vals
  12. 0    a     1
  13. 1    b     2
  14. 0    a     1
  15. 1    b     2
  16.  
  17. In [187]: res.dtypes
  18. Out[187]: 
  19. cats    category
  20. vals       int64
  21. dtype: object

In this case the categories are not the same, and therefore an error is raised:

  1. In [188]: df_different = df.copy()
  2.  
  3. In [189]: df_different["cats"].cat.categories = ["c", "d"]
  4.  
  5. In [190]: try:
  6.    .....:     pd.concat([df, df_different])
  7.    .....: except ValueError as e:
  8.    .....:     print("ValueError:", str(e))
  9.    .....:

The same applies to df.append(df_different).

See also the section on merge dtypes for notes about preserving merge dtypes and performance.

Unioning

New in version 0.19.0.

If you want to combine categoricals that do not necessarily have the samecategories, the union_categoricals() function willcombine a list-like of categoricals. The new categories will be the union ofthe categories being combined.

  1. In [191]: from pandas.api.types import union_categoricals
  2.  
  3. In [192]: a = pd.Categorical(["b", "c"])
  4.  
  5. In [193]: b = pd.Categorical(["a", "b"])
  6.  
  7. In [194]: union_categoricals([a, b])
  8. Out[194]: 
  9. [b, c, a, b]
  10. Categories (3, object): [b, c, a]

By default, the resulting categories will be ordered asthey appear in the data. If you want the categories tobe lexsorted, use sort_categories=True argument.

  1. In [195]: union_categoricals([a, b], sort_categories=True)
  2. Out[195]: 
  3. [b, c, a, b]
  4. Categories (3, object): [a, b, c]

union_categoricals also works with the “easy” case of combining twocategoricals of the same categories and order information(e.g. what you could also append for).

  1. In [196]: a = pd.Categorical(["a", "b"], ordered=True)
  2.  
  3. In [197]: b = pd.Categorical(["a", "b", "a"], ordered=True)
  4.  
  5. In [198]: union_categoricals([a, b])
  6. Out[198]: 
  7. [a, b, a, b, a]
  8. Categories (2, object): [a < b]

The below raises TypeError because the categories are ordered and not identical.

  1. In [1]: a = pd.Categorical(["a", "b"], ordered=True)
  2. In [2]: b = pd.Categorical(["a", "b", "c"], ordered=True)
  3. In [3]: union_categoricals([a, b])
  4. Out[3]:
  5. TypeError: to union ordered Categoricals, all categories must be the same

New in version 0.20.0.

Ordered categoricals with different categories or orderings can be combined byusing the ignore_ordered=True argument.

  1. In [199]: a = pd.Categorical(["a", "b", "c"], ordered=True)
  2.  
  3. In [200]: b = pd.Categorical(["c", "b", "a"], ordered=True)
  4.  
  5. In [201]: union_categoricals([a, b], ignore_order=True)
  6. Out[201]: 
  7. [a, b, c, c, b, a]
  8. Categories (3, object): [a, b, c]

union_categoricals() also works with aCategoricalIndex, or Series containing categorical data, but note thatthe resulting array will always be a plain Categorical:

  1. In [202]: a = pd.Series(["b", "c"], dtype='category')
  2.  
  3. In [203]: b = pd.Series(["a", "b"], dtype='category')
  4.  
  5. In [204]: union_categoricals([a, b])
  6. Out[204]: 
  7. [b, c, a, b]
  8. Categories (3, object): [b, c, a]

Note

union_categoricals may recode the integer codes for categorieswhen combining categoricals. This is likely what you want,but if you are relying on the exact numbering of the categories, beaware.

  1. In [205]: c1 = pd.Categorical(["b", "c"])
  2.  
  3. In [206]: c2 = pd.Categorical(["a", "b"])
  4.  
  5. In [207]: c1
  6. Out[207]: 
  7. [b, c]
  8. Categories (2, object): [b, c]
  9.  
  10. # "b" is coded to 0
  11. In [208]: c1.codes
  12. Out[208]: array([0, 1], dtype=int8)
  13.  
  14. In [209]: c2
  15. Out[209]: 
  16. [a, b]
  17. Categories (2, object): [a, b]
  18.  
  19. # "b" is coded to 1
  20. In [210]: c2.codes
  21. Out[210]: array([0, 1], dtype=int8)
  22.  
  23. In [211]: c = union_categoricals([c1, c2])
  24.  
  25. In [212]: c
  26. Out[212]: 
  27. [b, c, a, b]
  28. Categories (3, object): [b, c, a]
  29.  
  30. # "b" is coded to 0 throughout, same as c1, different from c2
  31. In [213]: c.codes
  32. Out[213]: array([0, 1, 2, 0], dtype=int8)

Concatenation

This section describes concatenations specific to category dtype. See Concatenating objects for general description.

By default, Series or DataFrame concatenation which contains the same categoriesresults in category dtype, otherwise results in object dtype.Use .astype or union_categoricals to get category result.

  1. # same categories
  2. In [214]: s1 = pd.Series(['a', 'b'], dtype='category')
  3.  
  4. In [215]: s2 = pd.Series(['a', 'b', 'a'], dtype='category')
  5.  
  6. In [216]: pd.concat([s1, s2])
  7. Out[216]: 
  8. 0    a
  9. 1    b
  10. 0    a
  11. 1    b
  12. 2    a
  13. dtype: category
  14. Categories (2, object): [a, b]
  15.  
  16. # different categories
  17. In [217]: s3 = pd.Series(['b', 'c'], dtype='category')
  18.  
  19. In [218]: pd.concat([s1, s3])
  20. Out[218]: 
  21. 0    a
  22. 1    b
  23. 0    b
  24. 1    c
  25. dtype: object
  26.  
  27. In [219]: pd.concat([s1, s3]).astype('category')
  28. Out[219]: 
  29. 0    a
  30. 1    b
  31. 0    b
  32. 1    c
  33. dtype: category
  34. Categories (3, object): [a, b, c]
  35.  
  36. In [220]: union_categoricals([s1.array, s3.array])
  37. Out[220]: 
  38. [a, b, b, c]
  39. Categories (3, object): [a, b, c]

Following table summarizes the results of Categoricals related concatenations.

arg1arg2result
categorycategory (identical categories)category
categorycategory (different categories, both not ordered)object (dtype is inferred)
categorycategory (different categories, either one is ordered)object (dtype is inferred)
categorynot categoryobject (dtype is inferred)

Getting data in/out

You can write data that contains category dtypes to a HDFStore.See here for an example and caveats.

It is also possible to write data to and reading data from Stata format files.See here for an example and caveats.

Writing to a CSV file will convert the data, effectively removing any information about thecategorical (categories and ordering). So if you read back the CSV file you have to convert therelevant columns back to category and assign the right categories and categories ordering.

  1. In [221]: import io
  2.  
  3. In [222]: s = pd.Series(pd.Categorical(['a', 'b', 'b', 'a', 'a', 'd']))
  4.  
  5. # rename the categories
  6. In [223]: s.cat.categories = ["very good", "good", "bad"]
  7.  
  8. # reorder the categories and add missing categories
  9. In [224]: s = s.cat.set_categories(["very bad", "bad", "medium", "good", "very good"])
  10.  
  11. In [225]: df = pd.DataFrame({"cats": s, "vals": [1, 2, 3, 4, 5, 6]})
  12.  
  13. In [226]: csv = io.StringIO()
  14.  
  15. In [227]: df.to_csv(csv)
  16.  
  17. In [228]: df2 = pd.read_csv(io.StringIO(csv.getvalue()))
  18.  
  19. In [229]: df2.dtypes
  20. Out[229]: 
  21. Unnamed: 0     int64
  22. cats          object
  23. vals           int64
  24. dtype: object
  25.  
  26. In [230]: df2["cats"]
  27. Out[230]: 
  28. 0    very good
  29. 1         good
  30. 2         good
  31. 3    very good
  32. 4    very good
  33. 5          bad
  34. Name: cats, dtype: object
  35.  
  36. # Redo the category
  37. In [231]: df2["cats"] = df2["cats"].astype("category")
  38.  
  39. In [232]: df2["cats"].cat.set_categories(["very bad", "bad", "medium",
  40.    .....:                                 "good", "very good"],
  41.    .....:                                inplace=True)
  42.    .....: 
  43.  
  44. In [233]: df2.dtypes
  45. Out[233]: 
  46. Unnamed: 0       int64
  47. cats          category
  48. vals             int64
  49. dtype: object
  50.  
  51. In [234]: df2["cats"]
  52. Out[234]: 
  53. 0    very good
  54. 1         good
  55. 2         good
  56. 3    very good
  57. 4    very good
  58. 5          bad
  59. Name: cats, dtype: category
  60. Categories (5, object): [very bad, bad, medium, good, very good]

The same holds for writing to a SQL database with to_sql.

Missing data

pandas primarily uses the value np.nan to represent missing data. It is bydefault not included in computations. See the Missing Data section.

Missing values should not be included in the Categorical’s categories,only in the values.Instead, it is understood that NaN is different, and is always a possibility.When working with the Categorical’s codes, missing values will always havea code of -1.

  1. In [235]: s = pd.Series(["a", "b", np.nan, "a"], dtype="category")
  2.  
  3. # only two categories
  4. In [236]: s
  5. Out[236]: 
  6. 0      a
  7. 1      b
  8. 2    NaN
  9. 3      a
  10. dtype: category
  11. Categories (2, object): [a, b]
  12.  
  13. In [237]: s.cat.codes
  14. Out[237]: 
  15. 0    0
  16. 1    1
  17. 2   -1
  18. 3    0
  19. dtype: int8

Methods for working with missing data, e.g. isna(), fillna(),dropna(), all work normally:

  1. In [238]: s = pd.Series(["a", "b", np.nan], dtype="category")
  2.  
  3. In [239]: s
  4. Out[239]: 
  5. 0      a
  6. 1      b
  7. 2    NaN
  8. dtype: category
  9. Categories (2, object): [a, b]
  10.  
  11. In [240]: pd.isna(s)
  12. Out[240]: 
  13. 0    False
  14. 1    False
  15. 2     True
  16. dtype: bool
  17.  
  18. In [241]: s.fillna("a")
  19. Out[241]: 
  20. 0    a
  21. 1    b
  22. 2    a
  23. dtype: category
  24. Categories (2, object): [a, b]

Differences to R’s factor

The following differences to R’s factor functions can be observed:

  • R’s levels are named categories.
  • R’s levels are always of type string, while categories in pandas can be of any dtype.
  • It’s not possible to specify labels at creation time. Use s.cat.rename_categories(new_labels)afterwards.
  • In contrast to R’s factor function, using categorical data as the sole input to create anew categorical series will not remove unused categories but create a new categorical serieswhich is equal to the passed in one!
  • R allows for missing values to be included in its levels (pandas’ categories). Pandasdoes not allow NaN categories, but missing values can still be in the values.

Gotchas

Memory usage

The memory usage of a Categorical is proportional to the number of categories plus the length of the data. In contrast,an object dtype is a constant times the length of the data.

  1. In [242]: s = pd.Series(['foo', 'bar'] * 1000)
  2.  
  3. # object dtype
  4. In [243]: s.nbytes
  5. Out[243]: 16000
  6.  
  7. # category dtype
  8. In [244]: s.astype('category').nbytes
  9. Out[244]: 2016

Note

If the number of categories approaches the length of the data, the Categorical will use nearly the same ormore memory than an equivalent object dtype representation.

  1. In [245]: s = pd.Series(['foo%04d' % i for i in range(2000)])
  2.  
  3. # object dtype
  4. In [246]: s.nbytes
  5. Out[246]: 16000
  6.  
  7. # category dtype
  8. In [247]: s.astype('category').nbytes
  9. Out[247]: 20000

Categorical is not a numpy array

Currently, categorical data and the underlying Categorical is implemented as a Pythonobject and not as a low-level NumPy array dtype. This leads to some problems.

NumPy itself doesn’t know about the new dtype:

  1. In [248]: try:
  2.    .....:     np.dtype("category")
  3.    .....: except TypeError as e:
  4.    .....:     print("TypeError:", str(e))
  5.    .....: 
  6. TypeError: data type "category" not understood
  7.  
  8. In [249]: dtype = pd.Categorical(["a"]).dtype
  9.  
  10. In [250]: try:
  11.    .....:     np.dtype(dtype)
  12.    .....: except TypeError as e:
  13.    .....:     print("TypeError:", str(e))
  14.    .....: 
  15. TypeError: data type not understood

Dtype comparisons work:

  1. In [251]: dtype == np.str_
  2. Out[251]: False
  3.  
  4. In [252]: np.str_ == dtype
  5. Out[252]: False

To check if a Series contains Categorical data, use hasattr(s, 'cat'):

  1. In [253]: hasattr(pd.Series(['a'], dtype='category'), 'cat')
  2. Out[253]: True
  3.  
  4. In [254]: hasattr(pd.Series(['a']), 'cat')
  5. Out[254]: False

Using NumPy functions on a Series of type category should not work as _Categoricals_are not numeric data (even in the case that .categories is numeric).

  1. In [255]: s = pd.Series(pd.Categorical([1, 2, 3, 4]))
  2.  
  3. In [256]: try:
  4.    .....:     np.sum(s)
  5.    .....: except TypeError as e:
  6.    .....:     print("TypeError:", str(e))
  7.    .....: 
  8. TypeError: Categorical cannot perform the operation sum

Note

If such a function works, please file a bug at https://github.com/pandas-dev/pandas!

dtype in apply

Pandas currently does not preserve the dtype in apply functions: If you apply along rows you geta Series of object dtype (same as getting a row -> getting one element will return abasic type) and applying along columns will also convert to object. NaN values are unaffected.You can use fillna to handle missing values before applying a function.

  1. In [257]: df = pd.DataFrame({"a": [1, 2, 3, 4],
  2.    .....:                    "b": ["a", "b", "c", "d"],
  3.    .....:                    "cats": pd.Categorical([1, 2, 3, 2])})
  4.    .....: 
  5.  
  6. In [258]: df.apply(lambda row: type(row["cats"]), axis=1)
  7. Out[258]: 
  8. 0    <class 'int'>
  9. 1    <class 'int'>
  10. 2    <class 'int'>
  11. 3    <class 'int'>
  12. dtype: object
  13.  
  14. In [259]: df.apply(lambda col: col.dtype, axis=0)
  15. Out[259]: 
  16. a          int64
  17. b         object
  18. cats    category
  19. dtype: object

Categorical index

CategoricalIndex is a type of index that is useful for supportingindexing with duplicates. This is a container around a Categoricaland allows efficient indexing and storage of an index with a large number of duplicated elements.See the advanced indexing docs for a more detailedexplanation.

Setting the index will create a CategoricalIndex:

  1. In [260]: cats = pd.Categorical([1, 2, 3, 4], categories=[4, 2, 3, 1])
  2.  
  3. In [261]: strings = ["a", "b", "c", "d"]
  4.  
  5. In [262]: values = [4, 2, 3, 1]
  6.  
  7. In [263]: df = pd.DataFrame({"strings": strings, "values": values}, index=cats)
  8.  
  9. In [264]: df.index
  10. Out[264]: CategoricalIndex([1, 2, 3, 4], categories=[4, 2, 3, 1], ordered=False, dtype='category')
  11.  
  12. # This now sorts by the categories order
  13. In [265]: df.sort_index()
  14. Out[265]: 
  15.   strings  values
  16. 4       d       1
  17. 2       b       2
  18. 3       c       3
  19. 1       a       4

Side effects

Constructing a Series from a Categorical will not copy the inputCategorical. This means that changes to the Series will in most caseschange the original Categorical:

  1. In [266]: cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
  2.  
  3. In [267]: s = pd.Series(cat, name="cat")
  4.  
  5. In [268]: cat
  6. Out[268]: 
  7. [1, 2, 3, 10]
  8. Categories (5, int64): [1, 2, 3, 4, 10]
  9.  
  10. In [269]: s.iloc[0:2] = 10
  11.  
  12. In [270]: cat
  13. Out[270]: 
  14. [10, 10, 3, 10]
  15. Categories (5, int64): [1, 2, 3, 4, 10]
  16.  
  17. In [271]: df = pd.DataFrame(s)
  18.  
  19. In [272]: df["cat"].cat.categories = [1, 2, 3, 4, 5]
  20.  
  21. In [273]: cat
  22. Out[273]: 
  23. [5, 5, 3, 5]
  24. Categories (5, int64): [1, 2, 3, 4, 5]

Use copy=True to prevent such a behaviour or simply don’t reuse Categoricals:

  1. In [274]: cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
  2.  
  3. In [275]: s = pd.Series(cat, name="cat", copy=True)
  4.  
  5. In [276]: cat
  6. Out[276]: 
  7. [1, 2, 3, 10]
  8. Categories (5, int64): [1, 2, 3, 4, 10]
  9.  
  10. In [277]: s.iloc[0:2] = 10
  11.  
  12. In [278]: cat
  13. Out[278]: 
  14. [1, 2, 3, 10]
  15. Categories (5, int64): [1, 2, 3, 4, 10]

Note

This also happens in some cases when you supply a NumPy array instead of a Categorical:using an int array (e.g. np.array([1,2,3,4])) will exhibit the same behavior, while usinga string array (e.g. np.array(["a","b","c","a"])) will not.