活了二十多年,没能为祖国、为人民做点什么,每思及此,伤心欲绝 !

Hive常见问题

大数据 keguang 98℃ 0评论

1. 内存溢出

虚拟内存溢出:

Current usage: 1.1gb of 2.0gb physical memory used; 4.6gb of 4.2gb virtual memory used. Killing container.==【即虚拟内存溢出】==;

方法一:提高yarn.nodemanager.vmem-pmem-ratio = 5或者更高;【推荐】

方法二:yarn.nodemanager.vmem-check-enabled =false ;关闭虚拟内存检查;不推荐

方法三:提高物理内存分配,相应的虚拟内存自然就多了,但是这样不是最优

物理内存溢出:

Current usage: 2.1gb of 2.0gb physical memory used; 3.6gb of 4.2gb virtual memory used. Killing container.【即物理内存溢出】;

方法一:mapreduce.map.memory.mb = 3GB以上,然后测试这个map/reduce task需要使用多少内存才够用,提高这个值直到不报错为止。

方法二:提高yarn.scheduler.minimum-allocation-mb = 3GB以上,同理【不推荐】


作者:z3935212
来源:CSDN
原文:https://blog.csdn.net/z3935212/article/details/78637157?utm_source=copy
版权声明:本文为博主原创文章,转载请附上博文链接!

打开低版本hive报错:

ls: cannot access /app/local/spark-2.0.2-bin-hadoop2.6/lib/spark-assembly-*.jar: No such file or directory

修改hive启动文件

vim /app/local/hive/bin/hive
找到下面这一行:
# add Spark assembly jar to the classpath
if [[ -n "$SPARK_HOME" ]]
then
  # sparkAssemblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`
  sparkAssemblyPath=`ls ${SPARK_HOME}/jars/*.jar`
  CLASSPATH="${CLASSPATH}:${sparkAssemblyPath}"
fi

2. 关联查询

2018-11-25 14:43:04,199 main ERROR Unable to invoke factory method in class class org.apache.hadoop.hive.ql.log element HushableMutableRandomAccess. java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:132)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:2
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
    at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.ja
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
    at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
    at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
    at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabili
    at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
    at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1993)
    at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1852)
    at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
    at org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionHandler.<init>(MapJoinMemoryExhaustion
    at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:129)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:358)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:546)
    at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:498)
    at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:514)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:418)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:393)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:774)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.RandomAccessctory@6dac64ea] unable to create manager for [/var/log/hive/operation_logs/5396b439-4945-483d-b8eb-b5c478e6fbb5ae-9f97-23f95080e4be] with data [org.apache.logging.log4j.core.appender.RandomAccessFileManager$FactoryData@5de
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
    at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManage
    at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFi
    ... 40 more
  • 异常原因:mr将数据量小的表识别成了大表,数据量大的识别成小表,导致将数据量大的表加入到内存,导致程序异常
  • 处理方法:
    set hive.execution.engine=mr;
    set hive.mapjoin.smalltable.filesize=55000000;
    set hive.auto.convert.join = false;  #取消小表加载至内存中

    ==通常情况下==,设置取消小表加载至内存中即可:set hive.auto.convert.join = false;

3. hive on spark问题

Job aborted due to stage failure: Aborting TaskSet 2.0 because task 8 (partition 8) cannot run anywhere due to node and executor blacklist. Blacklisting behavior can be configured via spark.blacklist.*.

临时解决办法:

set hive.execution.engine = mr;

https://www.iteblog.com/archives/1907.html

4. hive 事务表

执行的操作

delete from hm2.history_helper_back where starttime = '2019-06-12';

报错信息

FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.

解决办法:

set hive.support.concurrency = true;
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

5. 修复大量分区

hive> MSCK REPAIR TABLE employee;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

设置:

set hive.msck.path.validation=ignore;

https://stackoverflow.com/questions/40043986/hdinsight-hive-msck-repair-table-table-name-throwing-error

6. hiveserver2 不识别udf函数

在无法使用UDF的 HiveServer2 上,执行 reload function 命令,将MetaStore中新增的UDF信息同步到HiveServer2内存中。

7. 动态分区

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:941)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:712)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:147)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:487)
    ... 9 more

设置每个节点最大动态分区个数.

set hive.exec.max.dynamic.partitions=3000;
set hive.exec.max.dynamic.partitions.pernode=1000;

8. block块丢失

Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-808991319-10.1.0.62-1541662386662:blk_1110742285_40900ile=/user/hive/warehouse/hm4.db/hm4_format_log_his_tmp/dt=2019-09-16/hour=11/product=mini/event=click/part-r-00013_copy_1

由于block块受损,无法恢复,只能删除。

转载请注明:柯广的博客 » Hive常见问题

喜欢 (0)or分享 (0)
发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址