bloom

算法分析:使用布隆过滤器(Bloom Filter)进行大数据量排序

半世苍凉 提交于 2020-02-29 21:53:27
题目大意:移动公司需要对已经发放的所有139段的号码进行统计排序,已经发放的139号码段的文件都存放在一个文本文件中(原题是放在两个文件中),一个号码一行,现在需要将文件里的所有号码进行排序,并写入到一个新的文件中;号码可能会有很多,最多可能有一亿个不同的号码(所有的139段号码),存入文本文件中大概要占1.2G的空间;JVM最大的内存在300以内,程序要考虑程序的可执行性及效率;只能使用Java标准库,不得使用第三方工具。 这是个典型的大数据量的排序算法问题,首先要考虑空间问题,一下把.2G的数据读入内存是不太可能的,就算把壹亿条数据都转换成INT类型存储也要占接近400M的空间。当时做个题目我并没有想太多的执行效率问题,主要就考虑了空间,而且习惯性的想到合并排序,基本思想是原文件分割成若干个小文件并排序,再将排序好的小文件合并得到最后结果,算法大概如下: 1、顺序读取存放号码文件的中所有号码,并取139之后的八位转换为int类型;每读取号码数满一百万个(这个数据可配置)将已经读取的号码排序并存入新建的临时文件。 2、将所有生成的号码有序的临时文件合并存入结果文件。 这个算法虽然解决了空间问题,但是运行效率极低,由于IO读写操作太多,加上步骤1中的排序的算法(快速排序)本来效率就不高(对于电话排序这种特殊情况来说),导致1亿条数据排序运行3个小时才有结果。

布隆过滤器(Bloom Filter)

∥☆過路亽.° 提交于 2020-01-07 17:16:42
【推荐】2019 Java 开发者跳槽指南.pdf(吐血整理) >>> 基本概念 如果想判断一个元素是不是在一个集合里,一般想到的是将所有元素保存起来,然后通过比较确定。链表、树、散列表(又叫哈希表,Hash table)等等数据结构都是这种思路。但是随着集合中元素的增加,我们需要的存储空间越来越大。同时检索速度也越来越慢,上述三种结构的检索时间复杂度分别为O(n),O(\log n),O(n/k)。 布隆过滤器的原理是,当一个元素被加入集合时,通过K个散列函数将这个元素映射成一个位数组中的K个点,把它们置为1。检索时,我们只要看看这些点是不是都是1就(大约)知道集合中有没有它了:如果这些点有任何一个0,则被检元素一定不在;如果都是1,则被检元素很可能在。这就是布隆过滤器的基本思想。 优点 相比于其它的数据结构,布隆过滤器在空间和时间方面都有巨大的优势。布隆过滤器存储空间和插入/查询时间都是常数(O(k))。另外, 散列函数相互之间没有关系,方便由硬件并行实现。布隆过滤器不需要存储元素本身,在某些对保密要求非常严格的场合有优势。 布隆过滤器可以表示全集,其它任何数据结构都不能; k和m相同,使用同一组散列函数的两个布隆过滤器的交并差运算可以使用位操作进行。 缺点 但是布隆过滤器的缺点和优点一样明显。误算率是其中之一。随着存入的元素数量增加,误算率随之增加。但是如果元素数量太少

How to add glowing effect to a line for OpenGL? [closed]

吃可爱长大的小学妹 提交于 2019-12-17 22:37:51
问题 Closed . This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Update the question so it focuses on one problem only by editing this post. Closed 2 years ago . How can I add a glowing effect to a line that I draw? I'm using OpenGL for Linux. 回答1: You can implement the radial blur effect described on Nehe Lesson 36. The main idea is to render the drawing to a texture, and do that N times with a small offset after each render, until the

XNA screenshot shows pre-Bloom, not final render

五迷三道 提交于 2019-12-06 06:43:40
问题 I have a windows platform game coded in C# XNA 4.0 using the Reach graphics settings. My project is based on the GameStateManagement sample but I later added Bloom and spriteSheet/spriteBatch functionality to it. I desire to have a screenshot saved of the final screen output. However, when I save my screenshot it only shows the render before Bloom was applied and before my HUD text is displayed (which I draw after the Bloom). I have my screenshot saved at the end of my Draw method, after

XNA screenshot shows pre-Bloom, not final render

ぐ巨炮叔叔 提交于 2019-12-04 12:54:13
I have a windows platform game coded in C# XNA 4.0 using the Reach graphics settings. My project is based on the GameStateManagement sample but I later added Bloom and spriteSheet/spriteBatch functionality to it. I desire to have a screenshot saved of the final screen output. However, when I save my screenshot it only shows the render before Bloom was applied and before my HUD text is displayed (which I draw after the Bloom). I have my screenshot saved at the end of my Draw method, after these two processes. I have tried all kinds of things. Andrew's answer here Take screen shot in XNA was

How to add glowing effect to a line for OpenGL? [closed]

别来无恙 提交于 2019-11-28 18:54:48
How can I add a glowing effect to a line that I draw? I'm using OpenGL for Linux. You can implement the radial blur effect described on Nehe Lesson 36 . The main idea is to render the drawing to a texture, and do that N times with a small offset after each render, until the drawing is ready to be copied to the framebuffer. I've written a small demo that uses Qt and OpenGL . You can see the original drawing (without the blur) below: The next image shows the drawing with the blur effect turned on: I know it's not much, but it's a start. I too once hoped there was a very simple solution to this,