Hadoop配置

#Hadoop配置
*翻译至Hadoop The Definitive Guide 4th Edition*

There are a handful of files for controlling the configuration of a Hadoop installation; the most important ones are listed in Table 10-1.
>Hadoop的安装需要一些配置文件；比较重要的一些文件已经在表10-1中罗列出来了。

<center>Table 10-1. Hadoop configuration files</center>
Filename|	Formate|	Description
---|---|---
hadoop-env.sh|	Bash script|	Environment variables that are used in the scripts to run Hadoop.【译文】运行Hadoop的脚本所需要的环境变量。
mapred-env.sh||Environment variables that are used in the scripts to run MapReduce(overrides variables set in hadoop-env.sh)【译文】MapReduce脚本运行需要的环境变量。（可以覆盖hadoop-evn.sh中的环境变量）
yarn-env.sh||Environment variables that are used in the scripts to run YARN (overridesvariables set in hadoop-env.sh)【译文】运行YARN的脚本需要的环境变量。（可以覆盖hadoop-env.sh中的变量）
core-site.xml||Configuration settings for Hadoop Core, such as I/O settings that are commonto HDFS, MapReduce, and YARN【译文】Hadoop中HDFS/MapReduce/YARN通用核心设置，例如I/O设置。
hdfs-site.xml||Configuration settings for HDFS daemons: the namenode, the secondary namenode, and the datanodes 【译文】HDFS守护进程配置
mapred-site.xml||Configuration settings for MapReduce daemons: the job history server【译文】MapReduce守护进程配置
yarn-site.xml||Configuration settings for YARN daemons: the resource manager, the web app proxy server, and the node managers【译文】YARN守护进程配置
slaves||A list of machines (one per line) that each run a datanode and a node manager【译文】datenode和node manager的主机地址（按行记录）
hadoop-metrics2.properties||Properties for controlling how metrics are published in Hadoop (see Metrics and JMX)【译文】Mertrics的配置
log4j.properties||Properties for system logfiles, the namenode audit log, and the task log for the task JVM process (Hadoop Logs)【译文】系统日志配置
hadoop-policy.xml||Configuration settings for access control lists when running Hadoop in secure mode【译文】Hadoop安全模式下访问控制列表配置
These files are all found in the etc/hadoop directory of the Hadoop distribution. The configuration directory can be relocated to another part of the filesystem (outside the Hadoop installation, which makes upgrades marginally easier) as long as daemons are started with the --config option (or, equivalently, with the HADOOP_CONF_DIR environment variable set) specifying the location of this directory on the local filesystem.

> 这些文件都可以在安装目录下的etc/hadoop目录下找到。这个配置文件目录可以迁移到文件系统的其他文职，只要在守护进程启动是设置参数--config（等同于设置环境变量HADOOP_CONF_DIR）指定本地的配置文件目录即可。
##Configuration Management 配置管理
Hadoop does not have a single, global location for configuration information. Instead, each Hadoop node in the cluster has its own set of configuration files, and it is up to administrators to ensure that they are kept in sync across the system. There are parallel shell tools that can help do this, such as dsh or pdsh. This is an area where Hadoop cluster management tools like Cloudera Manager and Apache Ambari really shine, since they take care of propagating changes across the cluster.

>【译文】Hadoop集群中没有统一的全局设置。集群中每一个Hadoop节点都有自己的配置文件，同时要确保它们在系统中保持同步。一些同步shell工具可以实现这个目的，例如dsh或者pdsh。这正是一些诸如Cloudera Manager和Apache Ambari的闪光点，这些工具可以很好的在集群中间处理配置文件的变化。

Hadoop is designed so that it is possible to have a single set of configuration files that are used for all master and worker machines. The great advantage of this is simplicity, both conceptually (since there is only one configuration to deal with) and operationally (as the Hadoop scripts are sufficient to manage a single configuration setup).

>【译文】Hadoop被设计成，在集群中的主从服务器上，仅使用一套配置文件。这种设计的有点就是简单，在概念上（仅有唯一的一套配置）和操作性（Hadoop的脚本足够管理唯一的一套配置）上的简便化。

For some clusters, the one-size-fits-all configuration model breaks down. For example, if you expand the cluster with new machines that have a different hardware specification from the existing ones, you need a different configuration for the new machines to take advantage of their extra resources.

>【译文】在一些集群中，one-size-fits-all（统一配置）配置模式有可能会不适用。例如，如果你使用一台与集群中其他机器硬件配置完全不同的服务器来进行扩展集群，你就需要优化配置，使它的额外资源得到充分利用。

In these cases, you need to have the concept of a class of machine and maintain a separate configuration for each class. Hadoop doesn’t provide tools to do this, but there are several excellent tools for doing precisely this type of configuration management, such as Chef, Puppet, CFEngine, and Bcfg2.

>【译文】在这种情况下，你需要定义一个主机类，每一类主机拥有独立配置。Hadoop没有提供这样的工具，但是有一些很好的工具可以精确的实现这种配置管理，例如Chef，Puppet，CFEngine和Bcfg2。

For a cluster of any size, it can be a challenge to keep all of the machines in sync. Consider what happens if the machine is unavailable when you push out an update. Who ensures it gets the update when it becomes available? This is a big problem and can lead to divergent installations, so even if you use the Hadoop control scripts for managing Hadoop, it may be a good idea to use configuration management tools for maintaining the cluster. These tools are also excellent for doing regular maintenance, such as patching security holes and updating system packages.

>【译文】保持集群中的主机同步，是任何规模集群需要解决的一个挑战性问题。试想一下，当你向集群提交一个更新动作，但是集群中的某个主机处于不可用状态，那么会发生什么呢？谁来确保在这台主机恢复可用状态后，能够接收到更新请求呢？在集群中不同的设备中间，这是一个大问题，所以使用维护集群的控制管理工具来调用Hadoop的控制脚本管理Hadoop，是一个很好的选择。这些工具同样也可以实现一些常规维护，例如安装系统安全漏洞的升级包。

##Environment Settings 环境变量设置

In this section, we consider how to set the variables in hadoop-env.sh. There are also analogous configuration files for MapReduce and YARN (but not for HDFS), called mapred-env.sh and yarn-env.sh, where variables pertaining to those components can be set. Note that the MapReduce and YARN files override the values set in hadoop-env.sh.

>【译文】这一节我们讲述如何在hadoop-env.sh中设置环境变量。同样，MapReduce和YARN也有相似的配置文件，分别是mapred-env.sh和yarn-env.sh。注意，MapReduce和YARN配置文件中的值，将覆盖hadoop-env.sh中的值。

###Java
The location of the Java implementation to use is determined by the JAVA_HOME setting in hadoop-env.sh or the JAVA_HOME shell environment variable, if not set in hadoop-env.sh. It’s a good idea to set the value in hadoop-env.sh, so that it is clearly defined in one place and to ensure that the whole cluster is using the same version of Java.

>【译文】hadoop-env.sh中JAVA_HOME的值指定了Java运行环境的路径，如果hadoop-env.sh中没有设置，就是用默认shell环境变量中的JAVA_HOME。最好在hadoop-env.sh中设置JAVA_HOME，因为这样做很清楚的将变量定义在同一个地方，并且整个集群统一使用一个相同版本的Java。

###Memory heap size
By default, Hadoop allocates 1,000 MB (1 GB) of memory to each daemon it runs. This is controlled by the HADOOP_HEAPSIZE setting in hadoop-env.sh. There are also environment variables to allow you to change the heap size for a single daemon. For example, you can set YARN_RESOURCEMANAGER_HEAPSIZE in yarn-env.sh to override the heap size for the resource manager.

>【译文】默认情况下Hadoop为每个守护进程分配1G内存用来运行。分配内存的大小由hadoop-env.sh文件中的HADOOP_HEAPSIZE来设置。同样可以使用单独的环境变量来设置各个守护进程的heap size。例如，你可以在yarn-env.sh中设置YARN_RESOURCEMANAGER_HEAPSIZE来决定YARN进程的heap size，同事覆盖掉hadoop-env.sh中的设置。

Surprisingly, there are no corresponding environment variables for HDFS daemons, despite it being very common to give the namenode more heap space. There is another way to set the namenode heap size, however; this is discussed in the following sidebar.

>【译文】出乎意料，HDFS守护进程并没有heap size这样的变量设置，虽然给namenode分配更多的heap space是一个很常见的需求。其实有另外一个方法能够设置namenode的heap size，我们将在以下框定的内容中讨论。

HOW MUCH MEMORY DOES A NAMENODE NEED?|
---|---
A namenode can eat up memory, since a reference to every block of every file is maintained in memory. It’s difficult to give a precise formula because memory usage depends on the number of blocks per file, the filename length, and the number of directories in the filesystem; plus, it can change from one Hadoop release to another.The default of 1,000 MB of namenode memory is normally enough for a few million files, but as a rule of thumb for sizing purposes, you can conservatively allow 1,000 MB per million blocks of storage.For example, a 200-node cluster with 24 TB of disk space per node, a block size of 128 MB, and a replication factor of 3 has room for about 12 million blocks (or more): 200 × 24,000,000 MB ⁄ (128 MB × 3). So in this case, setting the namenode memory to 12,000 MB would be a good starting point.You can increase the namenode’s memory without changing the memory allocated to other Hadoop daemons by setting HADOOP_NAMENODE_OPTS in hadoop-env.sh to include a JVM option for setting the memory size. HADOOP_NAMENODE_OPTS allows you to pass extra options to the namenode’s JVM. So, for example, if you were using a Sun JVM, -Xmx2000m would specify that 2,000 MB of memory should be allocated to the namenode.If you change the namenode’s memory allocation, don’t forget to do the same for the secondary namenode (using the HADOOP_SECONDARYNAMENODE_OPTS variable), since its memory requirements are comparable to the primary namenode’s.|
>【译文】因为在内存中管理的每个文件的每个block都需要占用一定的内存，所以namenode将吃掉一部分内存空间。namenode占用的内存空间很难去估算，因为内存的用量依赖于每个文件的block数量、文件名称的长度和文件所在文件系统目录的级数；另外，不同版本的Hadoop中这些数值也不相同。默认namenode内存占用为1000MB，已经足够数以百万计的文件使用了，但是从经验上来讲，你可以保守的估算，每一百万个block存储占用1000MB内存。举个列子，有一个200个节点的集群，每个节点由 200TB的硬盘存储空间，每个block的大小为128MB，每个block有3个副本，那么将会有超过2百万个block需要存储，200 × 24,000,000 MB ⁄ (128 MB × 3)。所以这个例子中，namenode的起始内存最好设置成12,000MB。通过设置hadoop-env.sh中的配置项HADOOP_NAMENODE_OPTS，你可以增加namenode的占用内存，并且这个值将会被记为JVM分配的内存大小，而且不用修改其他Hadoop守护进程分配的内存大小。HADOOP_NAMENODE_OPTS将一个额外设置给namenode的JVM。例如，你适用的JVM参数-Xmx2000m，那么将指定应该有2000MB的内存分配给namenode。如果改变了namenode的内存分配，不要忘了同样修改一下secondary namenode（使用HADOOP_SECONDARYNAMENODE_OPTS），因为sencondary namenode的内存要求跟主namenode的相匹配。

In addition to the memory requirements of the daemons, the node manager allocates containers to applications, so we need to factor these into the total memory footprint of a worker machine; see Memory settings in YARN and MapReduce.

>【译文】另外，对于守护进程的内存要求，node manager分配容器给应用程序，所以也要把这部分内存消耗作为工作主机的内存占用因子；参见Memory settings in YARN and MapReduce

##System logfiles

System logfiles produced by Hadoop are stored in $HADOOP_HOME/logs by default. This can be changed using the HADOOP_LOG_DIR setting in hadoop-env.sh. It’s a good idea to change this so that logfiles are kept out of the directory that Hadoop is installed in. Changing this keeps logfiles in one place, even after the installation directory changes due to an upgrade. A common choice is /var/log/hadoop, set by including the following line in hadoop-env.sh:

>【译文】日志文件默认在$HADOOP_HOME/logs目录下。这个路径可以通过hadoop-env.sh文件中的HADOOP_LOG_DIR变量设置来修改。建议修改这个配置，将日志文件存放在Hadoop的安装目录以外。假设将来升级操作修改了安装目录，那么不会对日志文件有任何影响。通常我们选择/var/log/hadoop这个目录来存放日志文件，我们在hadoop-env.sh文件中做如下设置：

`
export HADOOP_LOG_DIR=/var/log/hadoop
`

The log directory will be created if it doesn’t already exist. (If it does not exist, confirm that the relevant Unix Hadoop user has permission to create it.) Each Hadoop daemon running on a machine produces two logfiles. The first is the log output written via log4j. This file, whose name ends in .log, should be the first port of call when diagnosing problems because most application log messages are written here. The standard Hadoop log4j configuration uses a daily rolling file appender to rotate logfiles. Old logfiles are never deleted, so you should arrange for them to be periodically deleted or archived, so as to not run out of disk space on the local node.
>【译文】如果日志文件目录不存在程序会自动创建一个。（如果不存日志目录，需要确定操作系统Hadoop账户有权限创建文件目录。）每一个Hadoop守护进程运行时会产生两个日志文件。第一个日志文件是有log4j框架输出的。大部分的应用日志信息都将被写入这个扩展名.log的日志文件，所以在诊断系统问题时我们应该首选查看这个日志文件。Hadoop的log'4j配置适用Daily Rolling File Apander截断日志文件。旧日志文件永远不会被删除，所以需要定期的删除日志文件来释放节点磁盘空间。

The second logfile is the combined standard output and standard error log. This logfile, whose name ends in .out, usually contains little or no output, since Hadoop uses log4j for logging. It is rotated only when the daemon is restarted, and only the last five logs are retained. Old logfiles are suffixed with a number between 1 and 5, with 5 being the oldest file.

>【译文】第二个日志文件结合了标准输出和标准错误输出。这个日志文件的扩展名.out，通常包含一点日志或者根本没有日志，应为Hadoop使用log4j框架输出日志。只有当守护进程重新启动时，这个日志文件才会被滚动截断，而且仅保留之前最后5行日志记录。旧的日志文件名以数字1-5结尾，5代表最早的日志文件。

Logfile names (of both types) are a combination of the name of the user running the daemon, the daemon name, and the machine hostname. For example, hadoop-hdfsdatanode-ip-10-45-174-112.log.2014-09-20 is the name of a logfile after it has been rotated. This naming structure makes it possible to archive logs from all machines in the cluster in a single directory, if needed, since the filenames are unique.

>【译文】日志文件名由运行进程的用户名、进程名称和主机名称组合而成。例如，hadoop-hdfsdatanode-ip-10-45-174-122.log.2014-09-20作为一个滚动截断后的日志文件名。这样的日志文件名结构，使集群中所有机器的日志都可以保存在一个单一的目录下，因为文件名是唯一的。

The username in the logfile name is actually the default for the HADOOP_IDENT_STRING setting in hadoop-env.sh. If you wish to give the Hadoop instance a different identity for the purposes of naming the logfiles, change HADOOP_IDENT_STRING to be the identifier you want.

>【译文】日志文件名中的用户名，默认有hadoop-env.sh中的HADOOP_IDENT_STRING配置的。如果你想修改一个不一样的而标识，来给日志文件命名，那么就可以修改这个变量。

##SSH settings
The control scripts allow you to run commands on (remote) worker nodes from the master node using SSH. It can be useful to customize the SSH settings, for various reasons. For example, you may want to reduce the connection timeout (using the ConnectTimeout option) so the control scripts don’t hang around waiting to see whether a dead node is going to respond. Obviously, this can be taken too far. If the timeout is too low, then busy nodes will be skipped, which is bad.

>【译文】控制脚本可以在主机上通过SSH在从机上执行命令。由于很多原因，自定义SSH设置变得很有用处。例如，我们可能想减少连接的超时时长，使控制脚本不用不停的一直等待一个死掉的节点应答。如果超时时长过短，那么繁忙的节点将被不断的跳过，这样同样也不是一个明智的做法。

Another useful SSH setting is StrictHostKeyChecking, which can be set to no to automatically add new host keys to the known hosts files. The default, ask, prompts the user to confirm that the key fingerprint has been verified, which is not a suitable setting in a large cluster environment.[72]

>【译文】另外一个用途大的SSH设置是StrictHostKeyChecking，如果设置成no，将会使新的主机的key自动添加的the known hosts files。默认情况，系统会提示用户key fingerprint已经被认证过，但是这种设置在一个大型的集群环境中并不适合。

To pass extra options to SSH, define the HADOOP_SSH_OPTS environment variable in hadoop-env.sh. See the ssh and ssh_config manual pages for more SSH settings.

>【译文】关于额外的SSH设置，在hadoop-env.sh文件的HADOOP_SSH_OPTS中设置。

##Important Hadoop Daemon Properties
Hadoop has a bewildering number of configuration properties. In this section, we address the ones that you need to define (or at least understand why the default is ppropriate) for any real-world working cluster. These properties are set in the Hadoop site files: coresite. xml, hdfs-site.xml, and yarn-site.xml. Typical instances of these files are shown in Examples 10-1, 10-2, and 10-3.[73] You can learn more about the format of Hadoop’s configuration files in The Configuration API.

>【译文】Hadoop有很多配置属性，并且这些属性有时会令人感到迷惑。这一节，我们将讲解一些在真实的集群中必须的配置项，至少让读者理解这些配置项的默认值。这些设置包含在Hadoop的site文件中：core-site.xml，hdfs-site.xml和yarn-site.xml。

To find the actual configuration of a running daemon, visit the /conf page on its web server. For example, http://resource-manager-host:8088/conf shows the configuration that the resource manager is running with. This page shows the combined site and default configuration files that the daemon is running with, and also shows which file each property was picked up from.

>【译文】我们可以通过访问守护进程的web服务路径/conf，查看正在运行的守护进程的配置。例如，访问http://resource-manager-host:8088/conf我们可以查看资源管理进程的配置。这个页面显示了site配置文件和默认配置文件的内容，并且指明配置项来至于哪些文件。

<center>Example 10-1. A typical core-site.xml configuration file</center>
```
<?xml version="1.0"?>

<configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://namenode/</value>
 </property>
</configuration>
```
<center>Example 10-2. A typical hdfs-site.xml configuration file</center>
```
<?xml version="1.0"?>

<configuration>
 <property>
 <name>dfs.namenode.name.dir</name>
 <value>/disk1/hdfs/name,/remote/hdfs/name</value>
 </property>
 <property>
 <name>dfs.datanode.data.dir</name>
 <value>/disk1/hdfs/data,/disk2/hdfs/data</value>
 </property>
 <property>
 <name>dfs.namenode.checkpoint.dir</name>
 <value>/disk1/hdfs/namesecondary,/disk2/hdfs/namesecondary </value>
 </property>
</configuration> 
```
<center>Example 10-3. A typical yarn-site.xml configuration file</center>
```
<?xml version="1.0"?>

<configuration>
 <property>
 <name>yarn.resourcemanager.hostname</name>
 <value>resourcemanager</value>
 </property>
 <property>
 <name>yarn.nodemanager.local-dirs</name>
 <value>/disk1/nm-local-dir,/disk2/nm-local-dir</value>
 </property>
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce.shuffle</value>
 </property>
 <property>
 <name>yarn.nodemanager.resource.memory-mb</name>
 <value>16384</value>
 </property>
 <property>
 <name>yarn.nodemanager.resource.cpu-vcores</name>
 <value>16</value>
 </property>
</configuration> 
```
##HDFS
To run HDFS, you need to designate one machine as a namenode. In this case, the property fs.defaultFS is an HDFS filesystem URI whose host is the namenode’s hostname or IP address and whose port is the port that the namenode will listen on for RPCs. If no port is specified, the default of 8020 is used.

>【译文】运行HDFS之前，你需要指定一个主机作为namenode。在这种情况下，属性fs.defaultFS将作为一个HDFS的文件系统URI，它由namenode的主机名或者IP地址和namenode在RPC上监听的端口组成。如果没有指定端口，那么默认的端口是8020。

The fs.defaultFS property also doubles as specifying the default filesystem. The default filesystem is used to resolve relative paths, which are handy to use because they save typing (and avoid hardcoding knowledge of a particular namenode’s address). For example, with the default filesystem defined in Example 10-1, the relative URI /a/b is resolved to hdfs://namenode/a/b.

>【译文】fs.defaulstFS同时指明了默认文件系统。这个默认文件系统使用相对路径来定位文件位置或者目录，这种方式节省了键盘输入，所以使用起来很简便（并且避免对特别的namenode地址的硬编码）。例如，如例子10-1中的配置，/a/b这样一个URI将被解析为hdfs://namenode/a/b。

NOTE|
---|
If you are running HDFS, the fact that fs.defaultFS is used to specify both the HDFS namenode and the default filesystem means HDFS has to be the default filesystem in the server configuration. Bear in mind, however, that it is possible to specify a different filesystem as the default in the client configuration, for convenience.|

>【译文】如果我们使用了一个正在运行的HDFS，事实上fs.defaultFS同时指定了server端配置HDFS的namenode和默认文件系统是HDFS。请注意，我们很有可能在客户端配置中指定一个不同的文件系统。那么fs.defaultFS就显示出它的便捷性了。

For example, if you use both HDFS and S3 filesystems, then you have a choice of specifying either as the default in the client configuration, which allows you to refer to the default with a relative URI and the other with an absolute URI.

>例如，如果你同时有HDFS和S3两种文件系统可供选择，那么接下来你需要选择指定其中一个做为客户端配置的默认文件系统，这种配置可以允许你考虑使用一个相对URI来做为默认的文件系统，并使用绝对URI来作为其他的文件系统。
 
There are a few other configuration properties you should set for HDFS: those that set the storage directories for the namenode and for datanodes. The property dfs.namenode.name.dir specifies a list of directories where the namenode stores persistent filesystem metadata (the edit log and the filesystem image). A copy of each metadata file is stored in each directory for redundancy. It’s common to configure dfs.namenode.name.dir so that the namenode metadata is written to one or two local disks, as well as a remote disk, such as an NFS-mounted directory. Such a setup guards against failure of a local disk and failure of the entire namenode, since in both cases the files can be recovered and used to start a new namenode. (The secondary namenode takes only periodic checkpoints of the namenode, so it does not provide an up-to-date backup of the namenode.)

>【译文】接下来我们需要配置namenode和datanode的存储路径。dfs.namenode.name.dir用来指定namenode的文件系统metadata（编辑日志和文件系统镜像）存储目录列表。metadata的副本将被做为冗余存储在列表中的目录中。通常我们通过设置dfs.namenode.name.dir，使namenode的metadata写在1或者2块本地磁盘上，或者像NDF-mounted这样的远程磁盘上。这种保护措施，防止本地磁盘损坏导致整个namenode出现故障时，可以启动一个新的namenode恢复文件。（Secandory namenode对namenode仅执行周期性的检查，所以不会对namenode进行实时备份。）

You should also set the dfs.datanode.data.dir property, which specifies a list of directories for a datanode to store its blocks in. Unlike the namenode, which uses multiple directories for redundancy, a datanode round-robins writes between its storage directories, so for performance you should specify a storage directory for each local disk. Read performance also benefits from having multiple disks for storage, because blocks will be spread across them and concurrent reads for distinct blocks will be correspondingly spread across disks.

>【译文】我们同时需要设置dfs.datanode.data.dir属性，指定一个目录列表用来存储blocks。与namenode不同，目录列表并不是用来作为冗余的，datanode会按照轮循的方式写到目录列表的路径下。

TIP|
---|
For maximum performance, you should mount storage disks with the noatime option. This setting means that last accessed time information is not written on file reads, which gives significant performance gains.|

Finally, you should configure where the secondary namenode stores its checkpoints of the filesystem. The dfs.namenode.checkpoint.dir property specifies a list of directories where the checkpoints are kept. Like the storage directories for the namenode, which keep redundant copies of the namenode metadata, the checkpointed filesystem image is stored in each checkpoint directory for redundancy.

>  最后，设置secondary namenode的checkpoint的存储路径。dfs.namenode.checkpoint.dir指定了一个目录列表，checkpoint将被存储在这些路径上。就像namenode一样，checkpoint的副本将以冗余的方式保存在目录列表的每个目录上。

<center>*Table 10-2 summarizes the importan t configuration properties for HDFS.*</center>

Property name |Type |Default value |Description
---|---|---|---
fs.defaultFS| URI |file:/// |The default filesystem. The URI defines the hostname and port that the namenode’s RPC server runs on. The default port is 8020. This property is set in core-site.xml.
dfs.namenode.name.dir |Commaseparated directory names|file://\${hadoop.tmp.dir}/dfs/name| The list of directories where the namenode stores its persistent metadata. The namenode stores a copy of the metadata in each directory in the list.
dfs.datanode.data.dir|Commaseparated directory names|file://\${hadoop.tmp.dir}/dfs/data| A list of directories where the datanode stores blocks. Each block is stored in only one of these directories.
dfs.datanode.data.dir| Commaseparated directory names|file://\${hadoop.tmp.dir}/dfs/data| A list of directories where the datanode stores blocks. Each block is stored in only one of these directories.
dfs.namenode.checkpoint.dir| Commaseparated directory names| file://\${hadoop.tmp.dir}/dfs/namesecondary|A list of directories where the secondary namenode stores checkpoints. It stores a copy of the checkpoint in each directory in the list.

WARNING|
---|
Note that the storage directories for HDFS are under Hadoop’s temporary directory by default (this is configured via the hadoop.tmp.dir property, whose default is /tmp/hadoop-${user.name}). Therefore, it is critical that these properties are set so that data is not lost by the system when it clears out temporary directories.|
>注意HDFS的默认存储路径是在Hadoop的临时目录下（有hadoop.tmp.dir配置指定的/tmp/hadoop-\${user.name}目录下）。因此，为了防止操作系统清理临时文件时而被删除，需要设置到其他的目录上。
## **YARN**
To run YARN, you need to designate one machine as a resource manager. The simplest way to do this is to set the property yarn.resourcemanager.hostname to the hostname or IP address of the machine running the resource manager. Many of the resource manager’s server addresses are derived from this property. For example, yarn.resourcemanager.address takes the form of a host-port pair, and the host defaults to yarn.resourcemanager.hostname. In a MapReduce client configuration, this property is used to connect to the resource manager over RPC.
> 要运行YARN需要制定一台机器为resource manager。最简单的做法就是设置yarn.reourcemanager.hostname，把这个值设置成运行resource manager的服务器的IP地址或者主机名。很多地方都会从这个属性得到resource manager服务器I的IP或者主机名。例如，yarn.resourcemanager.address是一个host-port对，他的主机就是yarn.resourcemanager.hostname所指的主机。在MapReduce客户端的配置中，客户端通过这个属性使用RPC协议来连接resource manager。

During a MapReduce job, intermediate data and working files are written to temporary local files. Because this data includes the potentially very large output of map tasks, you need to ensure that the yarn.nodemanager.local-dirs property, which controls the location of local temporary storage for YARN containers, is configured to use disk partitions that are large enough. The property takes a comma-separated list of directory names, and you should use all available local disks to spread disk I/O (the directories are used in round-robin fashion). Typically, you will use the same disks and partitions (but different directories) for YARN local storage as you use for datanode block storage, as governed by the dfs.datanode.data.dir property, which was discussed earlier.
>yarn.nodemanager.local-dirs属性用来指定YARN容器的本地临时存储的位置。 在MapReduce job执行的过程中，数据和文件将被写在临时的本地文件中。map task会有潜在的巨大输出的可能，所以yarn.nodemanager.local-dirs应该配置到有足够大的磁盘分区的路径上。这个属性用逗号分割的形式指定了一个目录列表，我们尽量使用所有的可用本地磁盘来分别负载磁盘I/O（路径列表中的路径将被以轮循方式的选择使用）。经典的情况是，你将使用相同的磁盘和分区作为YARN的本地存储，做法就像我们之前导论过的使用dfs.datanode.data.dir属性设置datanode block的存储一样。

Unlike MapReduce 1, YARN doesn’t have tasktrackers to serve map outputs to reduce tasks, so for this function it relies on shuffle handlers, which are long-running auxiliary services running in node managers. Because YARN is a general-purpose service, the MapReduce shuffle handlers need to be enabled explicitly in yarn-site.xml by setting the yarn.nodemanager.aux-services property to mapreduce_shuffle. 
 Table 10-3 summarizes the important configuration properties for YARN. The resourcerelated settings are covered in more detail in the next sections.
> 与MapReduce 1不同的是，YARN没有tashtrackers提供map的输出给reduce tasks，这个功能现在由一个长期运行在node manager上的辅助服务shuffle handlers来代替。由于YARN是一个通用的服务，所以如果要使用MapReduce shuffle handler，需要在yarn-size.xml文件中配置yarn.nodemanager.aux-services=mapreduce_shuffle。

<center>*Table 10-3. Important YARN daemon properties*</center>
Property name |Type |Default value |Description
---|---|---|---
yarn.resourcemanager.hostname| Hostname| 0.0.0.0| The hostname of the machine the resource manager runs on. Abbreviated \${y.rm.hostname} below.
yarn.resourcemanager.address |Hostname and port|${y.rm.hostname}:8032| The hostname and port that the resource manager’s RPC server runs on.
yarn.nodemanager.local-dirs| Commaseparated directory names|\${hadoop.tmp.dir}/nmlocal-dir|A list of directories where node managers allow containers to store intermediate data. The data is cleared out when the application ends.
yarn.nodemanager.aux-services| Commaseparated service names||A list of auxiliary services run by the node manager. A service is implemented by the class defined by the property yarn.nodemanager.auxservices. service-name.class. By default, no auxiliary services are specified.
yarn.nodemanager.resource.memorymb| int |8192 |The amount of physical memory (in MB) that may be allocated to containers being run by the node manager.
yarn.nodemanager.vmem-pmem-ratio| float| 2.1 |The ratio of virtual to physical memory for containers. Virtual memory usage may exceed the allocation by this amount.
yarn.nodemanager.resource.cpuvcores|int| 8 |The number of CPU cores that may be allocated to containers being run by the node manager.

## **Memory settings in YARN and MapReduce**
YARN treats memory in a more fine-grained manner than the slot-based model used in MapReduce 1. Rather than specifying a fixed maximum number of map and reduce slots that may run on a node at once, YARN allows applications to request an arbitrary amount of memory (within limits) for a task. In the YARN model, node managers allocate memory from a pool, so the number of tasks that are running on a particular node depends on the sum of their memory requirements, and not simply on a fixed number of slots.
> YARN处理内存的方法比MapReduce 1的slot-based模型具有更细的粒度。

The calculation for how much memory to dedicate to a node manager for running containers depends on the amount of physical memory on the machine. Each Hadoop daemon uses 1,000 MB, so for a datanode and a node manager, the total is 2,000 MB. Set aside enough for other processes that are running on the machine, and the remainder can be dedicated to the node manager’s containers by setting the configuration property yarn.nodemanager.resource.memory-mb to the total allocation in MB. (The default is 8,192 MB, which is normally too low for most setups.)
 
 The next step is to determine how to set memory options for individual jobs. There are two main controls: one for the size of the container allocated by YARN, and another for the heap size of the Java process run in the container.
NOTE|
---|
The memory controls for MapReduce are all set by the client in the job configuration. The YARN settings are cluster settings and cannot be modified by the client.|
 
Container sizes are determined by mapreduce.map.memory.mb and mapreduce.reduce.memory.mb; both default to 1,024 MB. These settings are used by the application master when negotiating for resources in the cluster, and also by the node manager, which runs and monitors the task containers. The heap size of the Java process is set by mapred.child.java.opts, and defaults to 200 MB. You can also set the Java options separately for map and reduce tasks (see Table 10-4).
 
<center>*Table 10-4. MapReduce job memory properties (set by the client)*</center>
Property name |Type |Default value |Description
---|---|---|---
mapreduce.map.memory.mb |int| 1024| The amount of memory for map containers.
mapreduce.reduce.memory.mb| int| 1024| The amount of memory for reduce containers.
mapred.child.java.opts |String| \-Xmx200m|The JVM options used to launch the container process that runs map and reduce tasks. In addition to memory settings, this property can include JVM properties for debugging, for example.
mapreduce.map.java.opts| String| \-Xmx200m|The JVM options used for the child process that runs map tasks.
mapreduce.reduce.java.opts| String| \-Xmx200m|The JVM options used for the child process that runs reduce tasks.
 
 For example, suppose mapred.child.java.opts is set to -Xmx800m and mapreduce.map.memory.mb is left at its default value of 1,024 MB. When a map task is run, the node manager will allocate a 1,024 MB container (decreasing the size of its pool by that amount for the duration of the task) and will launch the task JVM configured with an 800 MB maximum heap size. Note that the JVM process will have a larger memory footprint than the heap size, and the overhead will depend on such things as the native libraries that are in use, the size of the permanent generation space, and so on. The important thing is that the physical memory used by the JVM process, including any processes that it spawns, such as Streaming processes, does not exceed its allocation (1,024 MB). If a container uses more memory than it has been allocated, then it may be terminated by the node manager and marked as failed.
 YARN schedulers impose a minimum or maximum on memory allocations. The default minimum is 1,024 MB (set by yarn.scheduler.minimum-allocation-mb), and the default maximum is 8,192 MB (set by yarn.scheduler.maximum-allocation-mb).
 
 There are also virtual memory constraints that a container must meet. If a container’s virtual memory usage exceeds a given multiple of the allocated physical memory, the node manager may terminate the process. The multiple is expressed by the yarn.nodemanager.vmem-pmem-ratio property, which defaults to 2.1. In the example used earlier, the virtual memory threshold above which the task may be terminated is 2,150 MB, which is 2.1 × 1,024 MB.
 
 When configuring memory parameters it’s very useful to be able to monitor a task’s actual memory usage during a job run, and this is possible via MapReduce task counters. The counters PHYSICAL_MEMORY_BYTES, VIRTUAL_MEMORY_BYTES, and COMMITTED_HEAP_BYTES (described in Table 9-2) provide snapshot values of memory usage and are therefore suitable for observation during the course of a task attempt.
 Hadoop also provides settings to control how much memory is used for MapReduce operations. These can be set on a per-job basis and are covered in Shuffle and Sort.

## **CPU settings in YARN and MapReduce**
 In addition to memory, YARN treats CPU usage as a managed resource, and applications can request the number of cores they need. The number of cores that a node manager can allocate to containers is controlled by the yarn.nodemanager.resource.cpu-vcores property. It should be set to the total number of cores on the machine, minus a core for each daemon process running on the machine (datanode, node manager, and any other long-running processes).
 
 MapReduce jobs can control the number of cores allocated to map and reduce containers by setting mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores. Both default to 1, an appropriate setting for normal single-threaded MapReduce tasks, which can only saturate a single core.
 
WARNING|
---|
While the number of cores is tracked during scheduling (so a container won’t be allocated on a machine where there are no spare cores, for example), the node manager will not, by default, limit actual CPU usage of running containers. This means that a container can abuse its allocation by using more CPU than it was given, possibly starving other containers running on the same host. YARN has support for enforcing CPU limits using Linux cgroups. The node manager’s container executor class (yarn.nodemanager.container-executor.class) must be set to use the LinuxContainerExecutor class, which in turn must be configured to use cgroups (see the properties under yarn.nodemanager.linux-container-executor).|

## **Hadoop Daemon Addresses and Ports**
 Hadoop daemons generally run both an RPC server for communication between daemons (Table 10-5) and an HTTP server to provide web pages for human consumption (Table 10-6). Each server is configured by setting the network address and port number to listen on. A port number of 0 instructs the server to start on a free port, but this is generally discouraged because it is incompatible with setting cluster-wide firewall policies. In general, the properties for setting a server’s RPC and HTTP addresses serve double duty: they determine the network interface that the server will bind to, and they are used by clients or other machines in the cluster to connect to the server. For example, node managers use the yarn.resourcemanager.resource-tracker.address property to find the address of their resource manager.
 
 It is often desirable for servers to bind to multiple network interfaces, but setting the network address to 0.0.0.0, which works for the server, breaks the second case, since the address is not resolvable by clients or other machines in the cluster. One solution is to have separate configurations for clients and servers, but a better way is to set the bind host for the server. By setting yarn.resourcemanager.hostname to the (externally resolvable) hostname or IP address and yarn.resourcemanager.bind-host to 0.0.0.0, you ensure that the resource manager will bind to all addresses on the machine, while at the same time providing a resolvable address for node managers and clients.
 
 In addition to an RPC server, datanodes run a TCP/IP server for block transfers. The server address and port are set by the dfs.datanode.address property , which has a default value of 0.0.0.0:50010.

Holy Null 's Blog

导航

最近发表

友情链接