一.简介
1.RF-IDF【term frequency-inverse document frequency】是一种用于检索与探究的常用加权技术。
2.TF-IDF是一种统计方法，用于评估一个词对于一个文件集或一个语料库中的其中一个文件的重要程度。
3.词的重要性随着它在文件中出现的次数的增加而增加，但同时也会随着它在语料库中出现的频率的升高而降低。
二.词频
指的是某一个给定的词语在一份给定的文件中出现的次数。这个数字通常会被归一化，以防止它偏向长的文件【同一个词语在文件里可能会比短文件有更高的词频，而不管该词重要与否】。
公式：
ni,j:是该词在文件dj中出现的次数，而分母则是在文件dj中所有词出现的次数之和。
三.逆文档频率
是一个词普遍重要性的度量。某一个特定词的IDF可以由总文件数目除以包含该词语的文件数据，再将得到的商取对数得到。
公式：
|D|：语料库中的文件总数
|{j:ti€dj}|:包含ti的文件总数
四.TF-IDF
公式：TF-IDF = TF * IDF
特点：某一特定文件内的高频率词语，以及该词语在整个语料库中的低文件频率，可以产生高权重的TF-IDF。因此，TF-IDF倾向于过滤掉常见的词语，保留重要的词语。
思想：如果某个词或短语在一篇文章中出现的频率TF高，并且在其它文章中很少出现，则认为此词或短语具有很好的类别区分能力，适合用来分类。
五.代码实现
1 package big.data.analyse.tfidf
2
3 import org.apache.log4j.{Level, Logger}
4 import org.apache.spark.sql.SparkSession
5
6 /**
7 * Created by zhen on 2019/05/28.
8 */
9 object TF_IDF {
10 /**
11 * 设置日志级别
12 */
13 Logger.getLogger("org").setLevel(Level.WARN)
14 def main(args: Array[String]) {
15 val spark = SparkSession
16 .builder()
17 .appName("TF_IDF")
18 .master("local[2]")
19 .config("spark.sql.warehouse.dir", "file:///D://warehouse").getOrCreate()
20 val sc = spark.sparkContext
21 /**
22 * 计算TF
23 */
24 val tf = sc.textFile("src/big/data/analyse/tfidf/TF.txt")
25 .map(row => row.replace(",", " ").replace(".", " ").replace(" ", " ")) // 数据清洗
26 .flatMap(row => row.split(" ")) // 拆分
27 .map(row => (row, 1.0))
28 .reduceByKey(_+_)
29
30 val tfSize = tf.map(row => row._2).sum() // 计算总词数
31
32 val tfed = tf.map(row => (row._1, row._2 / tfSize.toDouble)) //求词频
33 println("TF：")
34 tfed.foreach(println)
35
36 /**
37 * 计算IDF
38 */
39 val idf_0 = tf.map(row => (row._1, 1.0))
40 println("加载IDF1文件数据。。。")
41 val idf_1 = sc.textFile("src/big/data/analyse/tfidf/IDF1.txt")
42 .map(row => row.replace(",", " ").replace(".", " ").replace(" ", " "))
43 .flatMap(row => row.split(" "))
44 .map(row => (row, 1.0))
45 .reduceByKey(_+_)
46 .map(row => (row._1, 1.0))
47
48 println("加载IDF2文件数据。。。")
49 val idf_2 = sc.textFile("src/big/data/analyse/tfidf/IDF2.txt")
50 .map(row => row.replace(",", " ").replace(".", " ").replace(" ", " "))
51 .flatMap(row => row.split(" "))
52 .map(row => (row, 1.0))
53 .reduceByKey(_+_)
54 .map(row => (row._1, 1.0))
55
56 /**
57 * 整合语料库数据
58 */
59 val idf = idf_0.union(idf_1).union(idf_2)
60 .reduceByKey(_+_)
61 .map(row => (row._1, 3 / row._2))
62 println("IDF：")
63 idf.foreach(println)
64
65 /**
66 * 关联TF和IDF，计算TF-IDF
67 */
68 println("TF-IDF：")
69 tfed.join(idf).map(row => (row._1, (row._2._1 * row._2._2).formatted("%.4f")))
70 .foreach(println)
71 }
72 }
六.结果
TF： (GraphX,0.011494252873563218) (are,0.011494252873563218) (learning,0.011494252873563218) (Python,0.011494252873563218) (provides,0.011494252873563218) (is,0.022988505747126436) (Please,0.011494252873563218) (higher-level,0.011494252873563218) (general,0.011494252873563218) (Security,0.034482758620689655) (R,0.011494252873563218) (fast,0.011494252873563218) (SQL,0.022988505747126436) (Apache,0.011494252873563218) (Java,0.011494252873563218) (data,0.011494252873563218) (attack,0.011494252873563218) (This,0.011494252873563218) (cluster,0.011494252873563218) (graph,0.011494252873563218) (execution,0.011494252873563218) (MLlib,0.011494252873563218) (Scala,0.011494252873563218) (computing,0.011494252873563218) (downloading,0.011494252873563218) (Streaming,0.011494252873563218) (supports,0.022988505747126436) (engine,0.011494252873563218) (set,0.011494252873563218) (running,0.011494252873563218) (Spark,0.08045977011494253) (you,0.011494252873563218) (Overview,0.011494252873563218) (general-purpose,0.011494252873563218) (rich,0.011494252873563218) (APIs,0.011494252873563218) (vulnerable,0.011494252873563218) (that,0.011494252873563218) (a,0.022988505747126436) (high-level,0.011494252873563218) (processing,0.022988505747126436) (OFF,0.011494252873563218) (before,0.011494252873563218) (including,0.011494252873563218) (could,0.011494252873563218) (optimized,0.011494252873563218) (in,0.022988505747126436) (to,0.011494252873563218) (see,0.011494252873563218) (graphs,0.011494252873563218) (of,0.011494252873563218) (also,0.011494252873563218) (by,0.022988505747126436) (structured,0.011494252873563218) (tools,0.011494252873563218) (It,0.022988505747126436) (for,0.034482758620689655) (mean,0.011494252873563218) (an,0.011494252873563218) (machine,0.011494252873563218) (and,0.06896551724137931) (system,0.011494252873563218) (default,0.022988505747126436) 加载IDF1文件数据。。。 加载IDF2文件数据。。。 IDF： (running,1.5) (For,3.0) (visit,3.0) (The,3.0) (you,1.0) (website,1.5) (than,3.0) (7,3.0) (PATH,3.0) (that,1.0) (was,1.5) (a,1.0) (main,3.0) (old,3.0) (high-level,1.5) (be,1.5) (quick,3.0) (processing,1.5) (could,1.5) (all,3.0) (augmenting,3.0) (optimized,1.5) (Downloads,3.0) (follow,3.0) (applications,3.0) (classpath,3.0) (structured,1.5) (like,1.5) (along,3.0) (support,3.0) (Spark’s,1.5) (If,3.0) (but,3.0) (and,1.0) (reference,3.0) (1,3.0) (g,3.0) (system,1.5) (your,3.0) (10,3.0) (It’s,3.0) (are,1.0) (learning,1.5) (download,1.5) (its,3.0) (After,3.0) (Building,3.0) (can,1.5) (Security,1.5) (have,3.0) (runs,3.0) (6,3.0) (build,3.0) (0,1.5) (SQL,1.0) (with,1.5) (locally,3.0) (projects,3.0) (their,3.0) (Get,3.0) (UNIX-like,3.0) (This,1.0) (,1.5) (first,3.0) (documentation,3.0) (Since,3.0) (still,3.0) (Downloading,3.0) (packaged,3.0) (better,3.0) (However,3.0) (switch,3.0) (hood,3.0) (Linux,3.0) (Streaming,1.5) (supports,1.5) (PyPI,3.0) ((2,3.0) (vulnerable,1.5) (RDD,3.0) (Dataset,3.0) (package,3.0) (this,3.0) (under,3.0) (Python,1.0) (provides,1.0) (API,1.5) (higher-level,1.5) (introduction,3.0) (Apache,1.5) (will,1.5) (Java,1.0) (2,1.5) (data,1.5) (as,3.0) (YARN,3.0) (installed,3.0) (pointing,3.0) (optimizations,3.0) (get,3.0) (cluster,1.5) (tutorial,3.0) (graph,1.5) (easy,3.0) (execution,1.5) (MLlib,1.5) (We,3.0) (you’d,3.0) (supported,3.0) (downloading,1.5) (shell,3.0) (handful,3.0) (1+,3.0) (Users,3.0) (engine,1.5) (version,1.5) (11,3.0) (set,1.5) (performance,3.0) (rich,1.5) (systems,3.0) (replaced,3.0) (Spark,1.0) (project,3.0) (Overview,1.5) (APIs,1.5) (Mac,3.0) (or,1.5) (popular,3.0) (Support,3.0) (richer,3.0) (downloads,3.0) (OFF,1.5) (future,3.0) (detailed,3.0) (GraphX,1.5) (removed,3.0) (4,3.0) (installation,3.0) (Please,1.5) (is,1.0) (guide,3.0) (recommend,3.0) (R,1.5) (general,1.5) (JAVA_HOME,3.0) (fast,1.5) (include,3.0) (need,3.0) (one,3.0) (attack,1.5) (how,3.0) (uses,3.0) (compatible,3.0) (information,3.0) (we,3.0) (interactive,3.0) (—,3.0) (using,1.5) (Note,1.5) (7+/3,3.0) (java,3.0) (pre-packaged,3.0) (Scala,1.0) (any,1.5) (computing,1.5) (variable,3.0) (users,3.0) (from,1.5) (has,3.0) (won’t,3.0) (through,3.0) (at,3.0) (more,3.0) (3,3.0) (versions,3.0) (of,1.0) (tools,1.5) (8+,3.0) (by,1.0) (mean,1.5) (RDDs,3.0) ((e,3.0) (It,1.5) (for,1.0) (To,3.0) (were,3.0) (both,3.0) (an,1.0) (12,3.0) (which,3.0) (machine,1.5) (libraries,3.0) (introduce,3.0) (environment,3.0) ((in,3.0) (programming,3.0) (See,3.0) (use,1.5) (default,1.5) (the,1.5) (write,3.0) (highly,3.0) (release,3.0) (Resilient,3.0) (interface,3.0) (strongly-typed,3.0) (about,3.0) (run,3.0) (general-purpose,1.5) (5,3.0) (Distributed,3.0) (on,3.0) (You,3.0) (source,3.0) (Scala),3.0) (show,3.0) (then,3.0) (before,1.0) (including,1.5) (to,1.0) (in,1.0) (client,3.0) (see,1.5) (HDFS,1.5) (graphs,1.5) (Hadoop’s,3.0) (also,1.5) (“Hadoop,3.0) (binary,3.0) (x),3.0) (free”,3.0) (Maven,3.0) (coordinates,3.0) (Windows,3.0) (deprecated,3.0) (install,3.0) ((RDD),3.0) (4+,3.0) (page,3.0) (OS),3.0) (Hadoop,1.5) TF-IDF： (you,0.0115) (that,0.0115) (a,0.0230) (high-level,0.0172) (processing,0.0345) (could,0.0172) (optimized,0.0172) (structured,0.0172) (and,0.0690) (system,0.0172) (are,0.0115) (learning,0.0172) (Security,0.0517) (SQL,0.0230) (This,0.0115) (Streaming,0.0172) (supports,0.0345) (vulnerable,0.0172) (Spark,0.0805) (Overview,0.0172) (APIs,0.0172) (OFF,0.0172) (of,0.0115) (tools,0.0172) (by,0.0230) (mean,0.0172) (It,0.0345) (for,0.0345) (an,0.0115) (machine,0.0172) (default,0.0345) (Python,0.0115) (provides,0.0115) (higher-level,0.0172) (Apache,0.0172) (GraphX,0.0172) (Please,0.0172) (is,0.0230) (R,0.0172) (general,0.0172) (fast,0.0172) (attack,0.0172) (Java,0.0115) (Scala,0.0115) (computing,0.0172) (data,0.0172) (cluster,0.0172) (graph,0.0172) (execution,0.0172) (MLlib,0.0172) (downloading,0.0172) (engine,0.0172) (set,0.0172) (rich,0.0172) (general-purpose,0.0172) (before,0.0115) (including,0.0172) (to,0.0115) (in,0.0230) (see,0.0172) (graphs,0.0172) (also,0.0172) Process finished with exit code 0
来源：https://www.cnblogs.com/yszd/p/10939583.html