When we talk about the weight reduction of a set, we can definitely think of a set of unique elements, as well as the key in a map. So our traditional way of de-duplication is definitely to choose set or map to implement. In addition, when de-duplicating the entity object type, we may specify a field in the entity. This situation belongs to the majority, so how do we achieve it?

1. Implementation of Traditional map

First, let's look at the map implementation. Because the objects in the list are not fixed, I've written a general example of de-duplication according to id.

/**
 * De-duplication based on attribute id
 * @param objList  Set list to be de-duplicated
 * @param <E>      list Content generics
 * @return         Return to the list after de-duplication
 * @throws NoSuchFieldException
 * @throws IllegalAccessException
 */
public static <E> List<E> removeDuplicateId(List<E> objList) throws NoSuchFieldException, IllegalAccessException {
	List<E> newList = new ArrayList<>();
	HashMap<String, String> map = new HashMap<>();
	for (E t : objList) {
        // If it is empty, no comparison will be made. (Determine if you want to add it as needed)
		if (org.springframework.util.StringUtils.isEmpty(getFieldValue(t,"id"))) {
			continue;
		}
		String id = String.valueOf(getFieldValue(t,"id"));
		String value = map.get(id);
		if (org.springframework.util.StringUtils.isEmpty(value)) { //If the value is empty, the name is taken for the first time
			map.put(id, id);
			newList.add(t); //newList is the result of the de-duplication we want
		}
	}
	return newList;
}

Of course, some customized configurations are added here, such as discarding fields when they are empty. It can be seen that they are all based on map characteristics. If a value is found in the map, it is no longer added.

If it is extended to de-duplicate the field according to the specified field, the code can be modified to pass in the string of the field:

public static <E> List<E> removeDuplicateId(List<E> objList,String filedName) throws NoSuchFieldException, IllegalAccessException {
    if (StringUtils.isBlank(filedName)){
        return objList;
    }
	List<E> newList = new ArrayList<>();
	HashMap<String, String> map = new HashMap<>();
	for (E t : objList) {
		// If it is empty, no comparison will be made. (Determine if you want to add it as needed)
		if (org.springframework.util.StringUtils.isEmpty(getFieldValue(t,filedName))) {
			continue;
		}
		String id = String.valueOf(getFieldValue(t,filedName));
		String value = map.get(id);
		if (org.springframework.util.StringUtils.isEmpty(value)) { //If the value is empty, the name is taken for the first time
			map.put(id, id);
			newList.add(t); //newList is the result of the de-duplication we want
		}
	}
	return newList;
}

What if you want to duplicate multiple fields? Extend the second parameter to an array of characters:

public static <E> List<E> removeDuplicateId(List<E> objList,String... filedNames) throws NoSuchFieldException, IllegalAccessException {
	if (filedNames==null||filedNames.length==0){
		return objList;
	}
	List<E> newList = new ArrayList<>();
	HashMap<String, String> map = new HashMap<>();
	for (E t : objList) {
		String key = "";
		for (int i = 0; i < filedNames.length; i++) {
			key += String.valueOf(getFieldValue(t,filedNames[i]));
		}

		String value = map.get(key);
		if (org.springframework.util.StringUtils.isEmpty(value)) { //If the value is empty, the name is taken for the first time
			map.put(key, key);
			newList.add(t); //newList is the result of the de-duplication we want
		}
	}
	return newList;
}

With the implementation of map, set is not difficult, just need to modify the code to judge and add map to set object.

 

2. Implementation of labmda

The implementation of lambda is relatively simple. Take a look at three consecutive projects:

Single field de-duplication:

List<Member> memberList= members.stream() .collect(
                    Collectors.collectingAndThen(Collectors.toCollection(() -> new TreeSet<>(Comparator.comparing(member -> member.getName()))), ArrayList::new));

Multiple field de-duplication:

List<Member> memberList = members.stream() .collect(
        Collectors.collectingAndThen(
                Collectors.toCollection(() -> new TreeSet<>(Comparator.comparing(member -> member.getName()+"-"+member.getId()))), ArrayList::new));

Another group grouped according to one or more fields can also satisfy this requirement:

Map<String,List<Member>> map = members.stream().collect(Collectors.groupingBy(Member::getName));

Summary

Comparatively speaking, using lambda for de-rewriting is relatively simple, the code is concise and easy to read, while the traditional way is too cumbersome and needs to write a special util for processing, but from another point of view, the traditional way is more flexible, which can increase the judgment that lambda does not have, if the field needs to be de-duplicated In the case of null, using lambda grouping or de-duplication will result in null pointer errors.