Finding and removing duplicate records

Problem

You want to find and/or remove duplicate entries from a vector or data frame.

Solution

With vectors:

  1. # Generate a vector 
  2. set.seed(158)
  3. x <- round(rnorm(20, 10, 5))
  4. x
  5. #>  [1] 14 11  8  4 12  5 10 10  3  3 11  6  0 16  8 10  8  5  6  6
  7. # For each element: is this one a duplicate (first instance of a particular value
  8. # not counted)
  9. duplicated(x)
  10. #>  [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE FALSE
  11. #> [15]  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
  13. # The values of the duplicated entries
  14. # Note that '6' appears in the original vector three times, and so it has two
  15. # entries here.
  16. x[duplicated(x)]
  17. #> [1] 10  3 11  8 10  8  5  6  6
  19. # Duplicated entries, without repeats
  20. unique(x[duplicated(x)])
  21. #> [1] 10  3 11  8  5  6
  23. # The original vector with all duplicates removed. These do the same:
  24. unique(x)
  25. #>  [1] 14 11  8  4 12  5 10  3  6  0 16
  26. x[!duplicated(x)]
  27. #>  [1] 14 11  8  4 12  5 10  3  6  0 16

With data frames:

  1. # A sample data frame:
  2. df <- read.table(header=TRUE, text='
  3.  label value
  4.      A     4
  5.      B     3
  6.      C     6
  7.      B     3
  8.      B     1
  9.      A     2
  10.      A     4
  11.      A     4
  12. ')
  15. # Is each row a repeat?
  16. duplicated(df)
  17. #> [1] FALSE FALSE FALSE  TRUE FALSE FALSE  TRUE  TRUE
  19. # Show the repeat entries
  20. df[duplicated(df),]
  21. #>   label value
  22. #> 4     B     3
  23. #> 7     A     4
  24. #> 8     A     4
  26. # Show unique repeat entries (row names may differ, but values are the same)
  27. unique(df[duplicated(df),])
  28. #>   label value
  29. #> 4     B     3
  30. #> 7     A     4
  32. # Original data with repeats removed. These do the same:
  33. unique(df)
  34. #>   label value
  35. #> 1     A     4
  36. #> 2     B     3
  37. #> 3     C     6
  38. #> 5     B     1
  39. #> 6     A     2
  40. df[!duplicated(df),]
  41. #>   label value
  42. #> 1     A     4
  43. #> 2     B     3
  44. #> 3     C     6
  45. #> 5     B     1
  46. #> 6     A     2