How to remove duplicates from an array?

Removing duplicate items in the array is a very common operation is needed in actual development work. Therefore, it is often asked in JavaScript interviews. So if you are not sure how to implement the removal method, be sure to practice a few times before the interview.

What does the duplicate removal question look like?

This type of interview question usually looks like the following, given an array with duplicate numbers in it, write a removeDuplicate function, the input is the original array, and the output is the array with duplicate numbers removed.

const originalArr = [9,1,2,2,3,4,2,4,8,1,9]

function removeDuplicate(array){
...
}

const ans = removeDuplicate(originalArr);
console.log(ans) // [9,1,2,3,4,8]

Four solutions

Solution 1: Use Set to de-duplicate

Set is similar to Array, but one feature is that only the unique value can be stored in Set, so you can first convert Array to Set, then the duplicated value will be is removed. Then convert the Set back to an Array, and you will now have an array without duplicated value.

The solution is as follows:

function removeDuplicate(arr) {
  return Array.from(new Set(arr));
}

// You can also use spread operator to simplify the code
function removeDuplicate(arr) {
  return [...new Set(arr)];
}

let arr = [1, 2, 3, 2, 3, 8];
let arrAfter = removeDuplicate(arr);

console.log(arr1After); // [1, 2, 3, 8]

Solution 2: filter with indexOf

This solution first uses the filter method with indexOf method. We only keep the value that appears for the first time. As long as it is the second time, it will be filtered out. With this approach, we can ensure that the results will not have duplicated value.

The solution is as follows:

function removeDuplicate(arr) {
  // indexOf will return the index of the first item in the array equal to the value,
  // If the index returned by indexOf is equal to the current filtered value,
  // it means it is the first time the value appears, so we keep it,
  // Conversely, if the index is not equal, it means that the value has already appeared in the array, so we filter it out.
  return arr.filter((item, index, array) => array.indexOf(item) === index);
}

let arr = [1, 2, 3, 2, 3, 8];
let arrAfter = removeDuplicate(arr);

console.log(arrAfter); // [1, 2, 3, 8]

Solution 3: Nested for loop

Nested for loop is a brute force solution. It first iterates through the entire array in order, and find duplicate values and remove them through the nested for loop.

The solution is as follows:

function removeDuplicate(arr) {
  // In the outer for loop, i starts from index 0 and ends at the end of arr
  for (let i = 0, len = arr.length; i < len; i++) {
    // In the inner for loop, j starts from i + 1, to check whether the value is duplicated
    for (let j = i + 1; j < len; j++) {
      // If the value is duplicated, remove the value at position j from arr through the splice method
      if (arr[i] == arr[j]) {
        arr.splice(j, 1);
        // After removing duplicate values, the length of arr length will need to be reduced by 1
        len--;
        // The value at position j is removed, so index j is also decremented by 1
        j--;
      }
    }
  }
  return arr;
}

let arr = [1, 2, 3, 2, 3, 8];
let arrAfter = removeDuplicate(arr);

console.log(arrAfter); // [1, 2, 3, 8]

Solution 4: Object or Map

We can use object or Map to store the items that have been traversed to find out whether they already exist in the array. If not, put them into the array. If they already exist, don't put them in again. With the check, we ensure that there are no duplicate values in the array.

The solution is as follows:

function removeDuplicate(arr) {
  let seen = {};
  let newArray = [];

  // traverse the original array
  for (let item of arr) {
    // check whether the currently traversed item is in the object or map
    if (seen[item] !== true) {
      newArray.push(item); // If not yet, put in the new array that we will return eventually
      seen[item] = true; // place the item into the object or map, so we know it is already appear once
    }
  }
  return newArray;
}

let arr = [1, 2, 3, 2, 3, 8];
let arrAfter = removeDuplicate(arr);

console.log(arrAfter); // [1, 2, 3, 8]