我有一个带有结构化Spark流的应用程序,我想获取一些指标,例如调度延迟,延迟等。通常,这些指标可以在Spark UI Streaming选项卡中找到,但是,结构化流不存在这种功能我所知道的。 那么如何获得这些指标值?
目前,我尝试使用查询进度,但是在结果中找不到所有必需的指标:
QueryProgress {
"timestamp" : "2019-11-19T20:14:07.011Z","batchId" : 1,"numInputRows" : 8,"inputRowsPerSecond" : 0.8429038036034138,"processedRowsPerSecond" : 1.1210762331838564,"durationms" : {
"addBatch" : 6902,"getBatch" : 1,"getEndOffset" : 0,"queryPlanning" : 81,"setOffsetRange" : 20,"triggerExecution" : 7136,"walCommit" : 41
},"stateOperators" : [ {
"numRowsTotal" : 2,"numRowsUpdated" : 2,"memoryUsedBytes" : 75415,"customMetrics" : {
"loadedMapCacheHitCount" : 400,"loadedMapCacheMissCount" : 0,"stateonCurrentVersionSizeBytes" : 17815
}
} ],"sources" : [ {
"description" : "KafkaV2[Subscribe[tweets]]","startOffset" : {
"tweets" : {
"0" : 579
}
},"endOffset" : {
"tweets" : {
"0" : 587
}
},"processedRowsPerSecond" : 1.1210762331838564
} ]