惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

S
Securelist
Engineering at Meta
Engineering at Meta
MongoDB | Blog
MongoDB | Blog
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
A
About on SuperTechFans
S
Security @ Cisco Blogs
L
LangChain Blog
Cloudbric
Cloudbric
B
Blog
Google Online Security Blog
Google Online Security Blog
I
InfoQ
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
GbyAI
GbyAI
WordPress大学
WordPress大学
月光博客
月光博客
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
小众软件
小众软件
Cyberwarzone
Cyberwarzone
Martin Fowler
Martin Fowler
AWS News Blog
AWS News Blog
C
Cyber Attacks, Cyber Crime and Cyber Security
PCI Perspectives
PCI Perspectives
T
The Exploit Database - CXSecurity.com
Hugging Face - Blog
Hugging Face - Blog
Microsoft Azure Blog
Microsoft Azure Blog
C
CERT Recently Published Vulnerability Notes
博客园 - 司徒正美
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
F
Full Disclosure
P
Palo Alto Networks Blog
Hacker News - Newest:
Hacker News - Newest: "LLM"
V2EX - 技术
V2EX - 技术
G
GRAHAM CLULEY
T
Threat Research - Cisco Blogs
P
Privacy International News Feed
S
Schneier on Security
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Attack and Defense Labs
Attack and Defense Labs
V
Visual Studio Blog
N
News and Events Feed by Topic
C
Cybersecurity and Infrastructure Security Agency CISA
S
Security Affairs
Spread Privacy
Spread Privacy
Vercel News
Vercel News
量子位
Apple Machine Learning Research
Apple Machine Learning Research
Security Latest
Security Latest
T
The Blog of Author Tim Ferriss
IT之家
IT之家
Stack Overflow Blog
Stack Overflow Blog

博客园 - 拓子

.net编码规则 tensorflow mnist The tensorflow simplest calculate python opencv english opencv 图片识别 随机概率 从excel 导入数据绘制 散点图 tensor flow 线性回归 一些搞笑,但有意义的图片 基于python玩转人工智能最火框架之TensorFlow人工智能&深度学习介绍 win10 64下anaconda4.2.0(python3.5) PYTHON 爬虫 baidu美女图片 falkonry python hadoop wordcout测试 CENTOS重新安装JDK 目录和权限 centos基本命令
搭建Hadoop的环境
拓子 · 2019-01-06 · via 博客园 - 拓子

准备实验的环境:
1、安装Linux、JDK
2、配置主机名、免密码登录
3、约定:安装目录:/usr/local/bin

安装:
1、解压 : tar -zxvf hadoop-2.7.7.tar.gz -C /usr/local/bin/
2、设置环境变量: vi ~/.bash_profile
HADOOP_HOME=/usr/local/bin/hadoop-2.7.7
export HADOOP_HOME

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export PATH
/usr/local/bin/hadoop-2.7.7
生效环境变量: source ~/.bash_profile

第一节:Hadoop的目录结构

第二节:Hadoop的本地模式
1、特点:不具备HDFS,只能测试MapReduce程序
2、修改hadoop-env.sh

修改第27行:export JAVA_HOME=/root/training/jdk1.7.0_75

3、演示Demo: $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar
命令:hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount /root/data/input/data.txt /root/data/output/wc
日志:17/08/04 23:28:38 INFO mapreduce.Job: map 100% reduce 100%

注意:MR有一个默认的排序规则

第三节:Hadoop的伪分布模式
1、特点:具备Hadoop的所有功能,在单机上模拟一个分布式的环境
(1)HDFS:主:NameNode,数据节点:DataNode
(2)Yarn:容器,运行MapReduce程序
主节点:ResourceManager
从节点:NodeManager

2、步骤:
(1)hdfs-site.xml
<!--配置HDFS的冗余度-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<!--配置是否检查权限-->
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

(2)core-site.xml
<!--配置HDFS的NameNode-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.88.11:9000</value>
</property>

<!--配置DataNode保存数据的位置-->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/bin/hadoop-2.7.7/tmp</value>
</property>

(3) mapred-site.xml
<!--配置MR运行的框架-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

(4) yarn-site.xml
<!--配置ResourceManager的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.43.128</value>
</property>

<!--配置NodeManager执行任务的方式-->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

(5) 格式化NameNode
hdfs namenode -format
日志:Storage directory/usr/local/bin/hadoop-2.7.7/tmp/dfs/name has been successfully formatted.

(6) 启动:start-all.sh
(*) HDFS: 存储数据
(*) Yarn:执行计算

(7) 访问:(*)命令行
(*)Java API
(*)Web Console:
HDFS:http://192.168.43.128:50070
Yarn:http://192.168.43.128:8088