List deduplication + Java8-Stream stream operation List deduplication distinct, and specified field deduplication
Create a new list array:
List list = new ArrayList();
list.add(26);
list.add(39);
list.add(39);
list.add(39);
list.add(39);
list.add(5);
list.add(40);
list.add(39);
list.add(25);
System.out.println(list);
Method 1: Use the new java8 feature stream to deduplicate List [commonly used]
Note: to deduplicate objects, he can only deduplicate all fields in the same object, not for a single deduplication
List newList = list.stream().distinct().collect(Collectors.toList());
System.out.println(“java8新特性stream去重:”+newList);
list.add(39);
Method 3: Set the set to judge and deduplicate, without disrupting the order
protected final <T> List<T> removeDuplicates(List<T> list) {
return new ArrayList<>(new LinkedHashSet<>(list))
}
Java8-Stream stream operation List deduplication distinct, and deduplication of specified fields
Method 1: Java8-Stream stream operation List deduplication distinct, and deduplication of specified fields
The distinct deduplication method of stream is based on the two methods of Object.equals and Object.hashCode to determine whether it is a duplicate.
So we can use this feature to rewrite the two methods of Object.equals and Object.hashCode of pojo to achieve.
1. Rewrite the equals and hashCode methods of the Book class, use the name to determine whether the comparison is the same, and then use the distinct method of the stream to deduplicate
class Book {
...
@Override
public String toString() {
return String.format("(%s,%s,%s)", id, name, DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss").format(createTime.toInstant().atZone(ZoneId.systemDefault()).toLocalDateTime()));
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Book book = (Book) o;
return Objects.equals(name, book.name);
}
}
List<Book> distinctNameBooks1 = books.stream().distinct().collect(Collectors.toList());
System.out.println(distinctNameBooks1);
Summary: By rewriting the equals and hashCode methods and comparing them according to actual needs, you can directly use the distinct method of stream to deduplicate, which is more convenient; sometimes the object class is inconvenient or cannot be modified, such as it has been implemented or the referenced third-party package cannot Modification, this method cannot flexibly deduplicate by field.
2. Through Collectors.toCollection of Collectors.collectingAndThen, use TreeSet to specify fields in the constructor
A field deduplication
List<ProjectInfoVo> vo = jdbcTemplate.query(sql, new BeanPropertyRowMapper<>(ProjectInfoVo.class));
ArrayList<ProjectInfoVo> collect = vo.stream().collect(Collectors.collectingAndThen(
Collectors.toCollection(() -> new TreeSet<>(
Comparator.comparing(p -> p.getProjectId()))), ArrayList::new));
Deduplication of multiple fields
List<ProjectInfoVo> vo = jdbcTemplate.query(sql, new BeanPropertyRowMapper<>(ProjectInfoVo.class));
ArrayList<ProjectInfoVo> collect = vo.stream().collect(Collectors.collectingAndThen(
Collectors.toCollection(() -> new TreeSet<>(
Comparator.comparing(p -> p.getProjectId()+";"+p.getMember()))), ArrayList::new));
Summary:
Using the method provided by stream, the code is very concise, but the disadvantage is that although the deduplication effect is achieved, the order in the list changes, and some scenes need to maintain the order.
3. Custom method Comparator.comparing(p -> p.get***())
Custom method class - distinctByKey
public class StreamUtils {
public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
Map<Object, Boolean> seen = new ConcurrentHashMap<>();
return t -> seen.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
}
}
single field
List<ProjectInfoVo> acceptances = vo.stream()
.filter(StreamUtils.distinctByKey(b -> b.getProjectId()))
.collect(Collectors.toList());
multiple fields
List<ProjectInfoVo> acceptances = vo.stream()
.filter(StreamUtils.distinctByKey(b -> b.getProjectId()))
.filter(StreamUtils.distinctByKey(b -> b.getMember()))
.collect(Collectors.toList());
Summary: By encapsulating and defining a deduplication method, combined with the filter method, it can flexibly deduplicate by field and maintain the order of the original list. The disadvantage is that a HashMap is defined internally, which occupies a certain amount of memory, and there is an additional method definition.
4. Use the filter method of the stream to deduplicate, do not define the deduplication method, and create a HashMap outside
Map<Object, Boolean> map = new HashMap<>();
List<Book> distinctNameBooks4 = books.stream().filter(i -> map.putIfAbsent(i.getName(), Boolean.TRUE) == null).collect(Collectors.toList());
System.out.println(distinctNameBooks4);
Summary: It still cooperates with the filter method to achieve deduplication. There is no separate creation method, and a HashMap is temporarily defined to maintain the order of the original list. The disadvantage is that it takes up a certain amount of memory.