Apache Flink Kryo序列化器-ClassNotFoundException

我在Apache Flink 1.8.1中有一个项目,它具有Scala 2.11和Java8。我曾经使用Maven进行编译和所有依赖项管理,但是切换到Gradle ...这导致我在下面遇到此问题:

j.l.ClassnotFoundException: om.tinker.my.project.ProjectPayload
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    ... 3 frames excluded
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:172)
    ... 15 common frames omitted
    Wrapped by: c.e.kryo.KryoException: Unable to find class: om.tinker.my.project.ProjectPayload
    Serialization trace:
        eventOutputTag (com.my.project.contexts.ProjectContext)
        at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:178)
        at c.e.k.u.DefaultClassResolver.readClass(DefaultClassResolver.java:147)
        at c.e.kryo.Kryo.readClass(Kryo.java:674)
        at c.e.k.s.ReflectField.read(ReflectField.java:107)
        at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
        at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
        at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
        at o.a.f.r.p.NonReusingDeserializationDelegate.read(NonReusin...

首先,错误消息中缺少'c'。类路径应为“ com.tinker.my.project.ProjectPayload” ...我使用该代码检查了文件,并且导入语句中没有丢失的“ c” ...

我还编辑Flink conf文件以使用父级优先策略...

更多背景信息: 我还有一个名为ProjectContext的文件,其中有一个ArrayList<ProjectPayload>。它还具有eventOutputTag(如序列化跟踪中所述)...当我注释掉ArrayList<ProjectPayload>及其吸气剂/吸气剂时,一切正常!

当我将实例变量及其获取器/设置器放回ProjectContext中时,就会发生ClassnotFoundException ...

此外,我散布了大量打印语句,并且能够创建ProjectPayload实例,并将其注销。

###编辑(2020年6月30日)###

根据this serialization issue,我添加了以下代码: env.getconfig.registerTypeWithKryoSerializer(classOf[ProjectPayload],classOf[JavaSerializer[ProjectPayload]])

现在我遇到了这个尴尬(但类似)的错误:

"j.l.ClassnotFoundException: \u0005sr\u00008com.tinker.my.project.ProjectPayload+\"v
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    ... 3 frames excluded
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:172)
    ... 15 common frames omitted
    Wrapped by: c.e.kryo.KryoException: Unable to find class: \u0005sr\u00008com.tinker.my.project.ProjectPayload+\"v
    Serialization trace:
    allMyPayloads (com.tinker.my.project.ProjectContext)
    at c.e.k.u.DefaultClassResolver.readName(DefaultClassResolver.java:178)
    at c.e.k.u.DefaultClassResolver.readClass(DefaultClassResolver.java:147)
    at c.e.kryo.Kryo.readClass(Kryo.java:674)
    at c.e.k.s.ReflectField.read(ReflectField.java:107)
    at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
    at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
    at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
    at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
    at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
    at o.a.f.r.p.NonReusingDeserializationDelegate....
    

结果是\u0005是Unicode字符'ENQUIRY'。并且\u00008导致Google搜索结果出现乱码...稍后会报告

###编辑(2020年7月1日)### 一些进展:我正在ArrayList<ProjectPayload>内部初始化ProjectContext。当我删除该初始化,将其移到外部,然后设置ArrayList值时,我的代码进行得更深入了。然后它也抱怨一个HashMap<String,String>实例变量-由于没有使用它,我最终删除了它。

现在把我带到IndexOutOfBoundsException

j.l.IndexOutOfBoundsException: Index: 93,Size: 9
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at c.e.k.u.MapReferenceResolver.getReadObject(MapReferenceResolver.java:62)
    at c.e.kryo.Kryo.readReferenceOrNull(Kryo.java:838)
    at c.e.kryo.Kryo.readObjectOrNull(Kryo.java:761)
    at c.e.k.s.ReflectField.read(ReflectField.java:120)
    ... 12 common frames omitted
    Wrapped by: c.e.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 93,Size: 9
    Serialization trace:
        fooBarStr (com.tinker.my.project.contexts.ProjectContext)
        at c.e.k.s.ReflectField.read(ReflectField.java:133)
        at c.e.k.s.FieldSerializer.read(FieldSerializer.java:122)
        at c.e.kryo.Kryo.readClassAndObject(Kryo.java:793)
        at o.a.f.a.j.t.r.k.KryoSerializer.deserialize(KryoSerializer.java:346)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:202)
        at o.a.f.s.r.s.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
        at o.a.f.r.p.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
        at o.a.f.r.i.n.a.s.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRec...

以及有关Kryo的Github问题:https://github.com/EsotericSoftware/kryo/issues/456

iCMS 回答:Apache Flink Kryo序列化器-ClassNotFoundException


尝试一下:

env.getConfig.registerTypeWithKryoSerializer(classOf[ProjectPayload],classOf[JavaSerializer[ProjectPayload]])
env.getConfig.registerTypeWithKryoSerializer(classOf[ProjectContext],classOf[JavaSerializer[ProjectContext]])

并确保您正在导入org.apache.flink.api.java.typeutils.runtime.kryo.JavaSerializer

https://ci.apache.org/projects/flink/flink-docs-stable/dev/custom_serializers.html#issue-with-using-kryos-javaserializer

本文链接:https://www.f2er.com/2048259.html

大家都在问