Skip to main content
版本:Current

Monitoring Spark

Collect and monitor the general performance Metrics of Spark.

Protocol Use:JMX

Spark App Enable JMX Protocol

  1. Add Spark VM options When Start Server ⚠️ customIP

Refer: https://spark.apache.org/docs/latest/spark-standalone.html

监控配置spark的监控主要分为Master、Worker、driver、executor监控。Master和Worker的监控在spark集群运行时即可监控,Driver和Excutor的监控需要针对某一个app来进行监控。 如果都要监控,需要根据以下步骤来配置

第一步

修改$SPARK_HOME/conf/spark-env.sh,添加以下语句:

# JMX Port to use
SPARK_DAEMON_JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

# export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
export SPARK_DAEMON_JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8712 "

语句中有$JMX_PORT,这个的值可以自定义,也可以获取一个随机数作为端口号。 如果端口自定义为一个具体的值,而 spark 的 Master 和其中之一的 Worker 在同一台机器上,会出现端口冲突的情况。

第二步

vim $SPARK_HOME/conf/metrics.properties 添加如下内容

*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

第三步

vim $SPARK_HOME/conf/spark-defaults.conf,添加以下项为driver和executor设置监控端口,在有程序运行的情况下,此端口会被打开。

spark.metrics.conf /opt/bigdata/spark/conf/metrics.properties
spark.driver.extraJavaOptions -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.mana
gement.jmxremote.port=8712

spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.mana
gement.jmxremote.port=8711

在spark的Master和Worker正常运行以及spark-submit提交了一个程序的情况下,可以从linux中查询出端口号码。

Configuration parameter

Parameter nameParameter help description
Monitoring HostMonitored IPV4, IPV6 or domain name. Note⚠️Without protocol header (eg: https://, http://)
Monitoring nameIdentify the name of this monitoring. The name needs to be unique
PortPort provided by JMX
UsernameJMX connection user name, optional
PasswordJMX connection password, optional
Collection intervalInterval time of monitor periodic data collection, unit: second, and the minimum interval that can be set is 30 seconds
Whether to detectWhether to detect and check the availability of monitoring before adding monitoring. Adding and modifying operations will continue only after the detection is successful
Description remarksFor more information about identifying and describing this monitoring, users can note information here

Collection Metrics

Metrics Set:memory_pool

Metric nameMetric unitMetric help description
namemetrics name
committedkbtotal size
initkbinit size
maxkbmax size
usedkbused size

Metrics Set:code_cache (Only Support JDK8)

Metric nameMetric unitMetric help description
committedkbtotal size
initkbinit size
maxkbmax size
usedkbused size

Metrics Set:class_loading

Metric nameMetric unitMetric help description
LoadedClassCountLoaded Class Count
TotalLoadedClassCountTotal Loaded Class Count
UnloadedClassCountUnloaded Class Count

Metrics Set:thread

Metric nameMetric unitMetric help description
TotalStartedThreadCountTotal Started Thread Count
ThreadCountThread Count
PeakThreadCountPeak Thread Count
DaemonThreadCountDaemon Thread Count
CurrentThreadUserTimemsCurrent Thread User Time
CurrentThreadCpuTimemsCurrent Thread Cpu Time