Python大數(shù)據(jù)分析與應(yīng)用實(shí)戰(zhàn)

第 1 章 Python 語法基礎(chǔ) ··························· 1
1.1 安裝 Anaconda ····································· 1
1.1.1 代碼提示 ······························· 4
1.1.2 變量瀏覽 ······························· 5
1.1.3 安裝第三方庫 ························· 5
1.2 語法基礎(chǔ) ············································ 6
1.2.1 字符串、列表、元組、字典和
集合 ····································· 6
1.2.2 條件判斷、循環(huán)和函數(shù) ··········· 13
1.2.3 異常 ··································· 17
1.2.4 特殊函數(shù) ····························· 20
1.3 Python 基礎(chǔ)庫應(yīng)用入門 ························ 22
1.3.1 NumPy 庫應(yīng)用入門 ················ 23
1.3.2 Pandas 庫應(yīng)用入門 ················· 29
1.3.3 Matplotlib 庫應(yīng)用入門 ············· 40
1.4 本章小結(jié) ·········································· 45
第 2 章 天氣數(shù)據(jù)的獲取與建模分析 ·········· 52
2.1 準(zhǔn)備工作 ·········································· 52
2.2 利用抓取方法獲取天氣數(shù)據(jù) ·················· 54
2.2.1 網(wǎng)頁解析 ····························· 54
2.2.2 抓取一個(gè)靜態(tài)頁面中的天氣
數(shù)據(jù) ··································· 57
2.2.3 抓取歷史天氣數(shù)據(jù) ················· 60
2.3 天氣數(shù)據(jù)可視化 ································· 63
2.3.1 查看數(shù)據(jù)基本信息 ················· 63
2.3.2 變換數(shù)據(jù)格式 ······················· 64
2.3.3 氣溫走勢的折線圖 ················· 66
2.3.4 歷年氣溫對(duì)比圖 ···················· 67
2.3.5 天氣情況的柱狀圖 ················· 69
2.3.6 使用 Tableau 制作天氣情況的
氣泡云圖 ····························· 70
2.3.7 風(fēng)向占比的餅圖 ···················· 73
2.3.8 使用 windrose 庫繪制風(fēng)玫瑰圖 ·· 74
2.4 機(jī)器學(xué)習(xí)在天氣預(yù)報(bào)中的應(yīng)用 ··············· 76
2.4.1 線性回歸的基本概念 ·············· 76
2.4.2 使用一元線性回歸預(yù)測氣溫 ····· 77
2.4.3 使用多元線性回歸預(yù)測氣溫 ····· 85
2.5 本章小結(jié) ·········································· 91
第 3 章 養(yǎng)成游戲中人物的數(shù)據(jù)搭建 ·········· 92
3.1 準(zhǔn)備工作 ·········································· 92
3.2 利用 Pyecharts 庫進(jìn)行數(shù)據(jù)基本情況分析 ··· 93
3.2.1 感染人數(shù)分布圖 ···················· 94
3.2.2 病情分布圖 ·························· 96
3.2.3 病癥情況堆疊圖 ···················· 97
3.2.4 繪制出院、死亡情況折線圖 ····· 98
3.2.5 病情熱力圖 ························· 100
3.2.6 病情分布象形圖 ··················· 101
3.2.7 人口流動(dòng)示意圖 ··················· 103
| Python 大數(shù)據(jù)分析與應(yīng)用實(shí)戰(zhàn) |
VI
3.3 感染病例分析 ··································· 105
3.3.1 基本信息統(tǒng)計(jì) ······················ 106
3.3.2 使用直方圖展示感染周期 ······· 108
3.3.3 使用詞云圖展示死亡病例情況 ··· 111
3.4 疫情趨勢預(yù)測 ··································· 114
3.4.1 利用邏輯方程預(yù)測感染人數(shù) ···· 115
3.4.2 利用 SIR 模型進(jìn)行疫情預(yù)測 ···· 120
3.4.3 Logistic 模型和 SIR 模型的
對(duì)比 ·································· 128
3.5 本章小結(jié) ········································· 131
第 4 章 航空數(shù)據(jù)分析 ···························· 132
4.1 準(zhǔn)備工作 ········································· 132
4.2 基本情況統(tǒng)計(jì)分析 ····························· 135
4.2.1 查看數(shù)據(jù)的基本信息 ············· 135
4.2.2 航空公司、機(jī)型分布 ············· 137
4.2.3 展示各個(gè)城市航班數(shù)量的 3D
地圖 ·································· 139
4.2.4 從首都機(jī)場出發(fā)的?；鶊D ······· 142
4.2.5 通過關(guān)系圖展示航線 ············· 145
4.3 利用 Floyd 算法計(jì)算最短飛行時(shí)間 ········· 148
4.3.1 Floyd 算法簡介 ···················· 148
4.3.2 Floyd 算法的流程 ················· 150
4.3.3 算法程序?qū)崿F(xiàn) ······················ 150
4.3.4 結(jié)果分析 ···························· 154
4.4 本章小結(jié) ········································· 158
第 5 章 市民服務(wù)熱線文本數(shù)據(jù)分析 ········· 160
5.1 準(zhǔn)備工作 ········································· 160
5.2 基本情況分析 ··································· 162
5.2.1 數(shù)據(jù)分布基本信息 ················ 162
5.2.2 每日平均工單量分析 ············· 165
5.2.3 來電時(shí)間分析 ······················ 166
5.2.4 工單類型分析 ······················ 167
5.3 利用詞云圖展示工單內(nèi)容 ···················· 171
5.3.1 工單分詞 ···························· 171
5.3.2 去除停用詞 ························· 172
5.3.3 詞頻統(tǒng)計(jì) ···························· 173
5.3.4 市民反映問題詞云圖 ············· 175
5.3.5 保存數(shù)據(jù) ···························· 176
5.4 基于樸素貝葉斯的工單自動(dòng)分類轉(zhuǎn)辦 ····· 177
5.4.1 需求概述 ···························· 177
5.4.2 樸素貝葉斯模型的基本概念 ···· 177
5.4.3 樸素貝葉斯文本分類算法的
流程 ·································· 181
5.4.4 程序?qū)崿F(xiàn) ···························· 182
5.5 基于 K-Means 算法和 PCA 方法降維的
熱點(diǎn)問題挖掘 ··································· 189
5.5.1 應(yīng)用場景 ···························· 189
5.5.2 K-Means 算法和 PCA 方法的
基本原理 ···························· 189
5.5.3 熱點(diǎn)問題挖掘算法的流程 ······· 193
5.5.4 程序?qū)崿F(xiàn) ···························· 194
5.6 本章小結(jié) ········································· 205
第 6 章 決策樹信貸風(fēng)險(xiǎn)控制 ·················· 206
6.1 準(zhǔn)備工作 ········································· 206
6.2 數(shù)據(jù)集基本情況分析 ·························· 209
6.2.1 查看數(shù)據(jù)大小和缺失情況 ······· 209
6.2.2 繪制直方圖查看數(shù)據(jù)的分布
情況 ·································· 211
6.2.3 繪制直方圖的 3 種方法 ·········· 212
| 目錄 |
VII
6.2.4 通過箱型圖查看異常值的情況 ···· 213
6.2.5 異常值和缺失值的處理 ·········· 217
6.2.6 使用小提琴圖展示預(yù)處理后的
數(shù)據(jù) ·································· 218
6.3 利用決策樹進(jìn)行信貸數(shù)據(jù)建模 ·············· 219
6.3.1 決策樹原理簡介 ··················· 219
6.3.2 決策樹信貸建模流程 ············· 225
6.3.3 利用 scikit-learn 庫實(shí)現(xiàn)決策樹
風(fēng)險(xiǎn)控制算法 ······················ 226
6.3.4 模型優(yōu)化 ···························· 231
6.4 本章小結(jié) ········································· 233
第 7 章 利用深度學(xué)習(xí)進(jìn)行垃圾圖片分類 ···· 234
7.1 準(zhǔn)備工作 ········································· 234
7.2 深度學(xué)習(xí)的基本原理 ·························· 237
7.2.1 CNN 的基本原理 ·················· 237
7.2.2 Keras 庫簡介 ······················· 240
7.3 利用 Keras 庫實(shí)現(xiàn)基于 CNN 的垃圾
圖片分類 ········································ 241
7.3.1 算法流程 ···························· 241
7.3.2 數(shù)據(jù)預(yù)處理 ························· 241
7.3.3 CNN 模型實(shí)現(xiàn) ····················· 247
7.4 優(yōu)化 CNN 模型 ································· 252
7.4.1 選擇優(yōu)化器 ························· 252
7.4.2 選擇損失函數(shù) ······················ 254
7.4.3 調(diào)整模型 ···························· 256
7.4.4 圖片增強(qiáng) ···························· 259
7.4.5 改變學(xué)習(xí)率 ························· 263
7.5 模型應(yīng)用 ········································· 265
7.6 本章小結(jié) ········································· 268
第 8 章 協(xié)同過濾和矩陣分解推薦算法
分析 ········································· 269
8.1 準(zhǔn)備工作 ········································· 269
8.2 基于協(xié)同過濾算法的短視頻完播情況
分析 ··············································· 271
8.2.1 基于用戶的協(xié)同過濾算法的
原理 ·································· 271
8.2.2 算法流程 ···························· 274
8.2.3 程序?qū)崿F(xiàn) ···························· 275
8.3 基于矩陣分解算法的短視頻完播情況
預(yù)測 ·············································· 283
8.3.1 算法原理 ···························· 283
8.3.2 利用 Surprise 庫實(shí)現(xiàn) SVD
算法 ·································· 286
8.4 幾種方法在測試集中的表現(xiàn) ················· 289
8.5 本章小結(jié) ········································· 291
第 9 章 《紅樓夢》文本數(shù)據(jù)分析 ············ 292
9.1 準(zhǔn)備工作 ········································· 292
9.1.1 編程環(huán)境 ···························· 292
9.1.2 數(shù)據(jù)情況簡介 ······················ 293
9.2 分詞 ··············································· 294
9.2.1 讀取數(shù)據(jù) ···························· 295
9.2.2 數(shù)據(jù)預(yù)處理 ························· 298
9.2.3 分詞及去除停用詞 ················ 306
9.2.4 制作詞云圖 ························· 307
9.3 文本聚類分析 ··································· 316
9.3.1 構(gòu)建分詞 TF-IDF 矩陣 ··········· 317
9.3.2 K-Means 聚類 ······················ 318
9.3.3 MDS 降維 ··························· 320
9.3.4 PCA 降維 ··························· 321
| Python 大數(shù)據(jù)分析與應(yīng)用實(shí)戰(zhàn) |
VIII
9.3.5 HC 聚類 ····························· 323
9.3.6 t -SNE 高維數(shù)據(jù)可視化 ·········· 325
9.4 LDA 主題模型 ·································· 326
9.5 人物社交網(wǎng)絡(luò)分析 ····························· 332
9.6 本章小結(jié) ········································· 338
附錄 A 抓取數(shù)據(jù)請(qǐng)求頭查詢 ··················· 339
附錄 B GraphViz 庫的安裝方法 ·············· 341
附錄 C 在 Windows 10 中安裝 TensorFlow
的方法 ······································ 343
參考文獻(xiàn) ··············································· 346
致謝 ····················································· 34