结构化Spark流指标检索

我有一个带有结构化Spark流的应用程序,我想获取一些指标,例如调度延迟,延迟等。通常,这些指标可以在Spark UI Streaming选项卡中找到,但是,结构化流不存在这种功能我所知道的。 那么如何获得这些指标值?

目前,我尝试使用查询进度,但是在结果中找不到所有必需的指标:

QueryProgress {
  "timestamp" : "2019-11-19T20:14:07.011Z","batchId" : 1,"numInputRows" : 8,"inputRowsPerSecond" : 0.8429038036034138,"processedRowsPerSecond" : 1.1210762331838564,"durationms" : {
    "addBatch" : 6902,"getBatch" : 1,"getEndOffset" : 0,"queryPlanning" : 81,"setOffsetRange" : 20,"triggerExecution" : 7136,"walCommit" : 41
  },"stateOperators" : [ {
    "numRowsTotal" : 2,"numRowsUpdated" : 2,"memoryUsedBytes" : 75415,"customMetrics" : {
      "loadedMapCacheHitCount" : 400,"loadedMapCacheMissCount" : 0,"stateonCurrentVersionSizeBytes" : 17815
    }
  } ],"sources" : [ {
    "description" : "KafkaV2[Subscribe[tweets]]","startOffset" : {
      "tweets" : {
        "0" : 579
      }
    },"endOffset" : {
      "tweets" : {
        "0" : 587
      }
    },"processedRowsPerSecond" : 1.1210762331838564
  } ]
yicaoyimu_ye 回答:结构化Spark流指标检索

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/3063838.html

大家都在问