The premise of deduplication in this article is to ensure that the order remains unchanged. This article gives a variety of implementation methods. Friends who need it can refer to the following
1. Intuitive method
The simplest idea is:
ids = [1,2,3,3,4,2,3,4,5,6,1] news_ids = [] for id in ids: if id not in news_ids: news_ids.append(id) print news_ids
2. Use set
ids = [1,4,3,3,4,2,3,4,5,6,1]
ids = list(set(ids))
The result is that the original order is not maintained
ids = [1,4,3,3,4,2,3,4,5,6,1] news_ids = list(set(ids)) news_ids.sort(ids.index)
3. Use itertools.grouby
ids = [1,4,3,3,4,2,3,4,5,6,1] ids.sort() it = itertools.groupby(ids) for k, g in it: print k
The principle of itertools.groupby can be seen here: http://docs.python.org/2/library/itertools.html#itertools.groupby
4. Use reduce
其中的 lambda x,y:x if y in x else x + [y] 等价于 lambda x,y: y in x and x or x+[y] 。
The idea is to change the ids to [[], 1,4,3,...] first, and then use the reduce feature. See here for an explanation of reduce: http://docs.python.org/2/library/functions.html#reduce