描述错误
在 windows 中,执行 MapReduce 时,报错如下:
2021-11-05 19:21:43,340 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Cleaning up the staging area file:/tmp/hadoop/mapred/staging/mym1580568728/.staging/job_local1580568728_0001
Exception in thread "main" 0: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:236)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:773)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:726)
at org.apache.hadoop.mapreduce.JobResourceUploader.mkdirs(JobResourceUploader.java:658)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:172)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
at top.malaoshi.Main.main(Main.java:41)
分析
跟踪 NativeIO.java 的 236 行代码可知,执行到 233 行代码
chmodImpl(path, mode);
报错,
好像是检查 D:\tmp\hadoop\mapred\staging\mym510547688\.staging\job_local510547688_0001 目录的权限
解决
在 windows 中,hadoop所在的盘下,有 tmp 目录,将其删除 或 改名
如:
我的 hadoop 安装在 D:\devtools\hadoop-3.0.3 目录下
那么就有 D:\tmp 目录,该目录可能是因为权限等一些原因,导致 MapReduce 无法在该目录下创建文件夹
所以将 D:\tmp 目录 删除 或 改名 即可