2022年11月

TL;DR：热更新问题

要增强的目标类ThreadPoolExecutor，在增强前（bytebuddy.installOn → instrument.addTransformer / redefine / retransform）已经被classloader装载到jvm里

导致instrument对已加载的类增强有些问题，这个问题也许可以通过深入redefine或retransform的机制解决，快速且稳定的解决方案为在插件装载完成前不加载ThreadPoolExecutor类

0.前置

instrument中的几个基本逻辑：

`ClassFileTransformer`接口

ClassFile实际指的是java字节码，class文件格式，这个对象在内存中，和文件没有关系

Note the term class file is used as defined in section 3.1 of The Java™ Virtual Machine Specification, to mean a sequence of bytes in class file format, whether or not they reside in a file.

jdk8中接口只有一个transform方法，类在加载、redefined或retransformed时会被调用来增强（the transformer's transform method is invoked when classes are loaded, redefined, or retransformed.）

`redefineClasses`方法

java 5+，已经加载的类重新进行转换处理，即会触发重新加载类定义，需要注意的是，新加载的类不能修改旧有的类声明，譬如不能增加属性、不能修改方法声明

`retransformClasses`方法

java 6+，与如上类似，但不是重新进行转换处理，而是直接把处理结果(bytecode)直接给JVM

“Agents use these methods to retransform previously loaded classes without needing to access their class files.”

redefine和retransform的区别：https://stackoverflow.com/questions/19009583/difference-between-redefine-and-retransform-in-javaagent

1.原始问题

自定义executors插件未生效，调试发现onInstall时抛UnsupportedOperationException，且未打印出来，这里异常栈顶是sun.instrument.InstrumentationImpl#retransformClasses，由SkyWalking调用ByteBuddy时指定了AgentBuilder.RedefinitionStrategy.RETRANSFORMATION策略加载

对比相同使用bytebuddy的自定义agent生效：https://gitlab.sunyongfei.com/platform-basic/java-agents/tree/threadpool-qy/thread-agent

其中onInstall前的RedefinitionStrategy不同：

SkyWalking采用AgentBuilder.RedefinitionStrategy.RETRANSFORMATION，自定义agent采用AgentBuilder.RedefinitionStrategy.REDEFINITION

net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy:

/**
 * <p>
 * A redefinition strategy regulates how already loaded classes are modified by a built agent.
 * </p>
 * <p>
 * <b>Important</b>: Most JVMs do not support changes of a class's structure after a class was already
 * loaded. Therefore, it is typically required that this class file transformer was built while enabling
 * {@link AgentBuilder#disableClassFormatChanges()}.
 * </p>
 */
enum RedefinitionStrategy {
 
    /**
     * Disables redefinition such that already loaded classes are not affected by the agent.
     */
    DISABLED(false, false) {
        @Override
        public void apply(Instrumentation instrumentation,
                          AgentBuilder.Listener listener,
                          CircularityLock circularityLock,
                          PoolStrategy poolStrategy,
                          LocationStrategy locationStrategy,
                          DiscoveryStrategy discoveryStrategy,
                          BatchAllocator redefinitionBatchAllocator,
                          Listener redefinitionListener,
                          LambdaInstrumentationStrategy lambdaInstrumentationStrategy,
                          DescriptionStrategy descriptionStrategy,
                          FallbackStrategy fallbackStrategy,
                          RawMatcher matcher) {
            /* do nothing */
        }
 
        @Override
        protected void check(Instrumentation instrumentation) {
            throw new IllegalStateException("Cannot apply redefinition on disabled strategy");
        }
 
        @Override
        protected Collector make() {
            throw new IllegalStateException("A disabled redefinition strategy cannot create a collector");
        }
    },
 
    /**
     * <p>
     * Applies a <b>redefinition</b> to all classes that are already loaded and that would have been transformed if
     * the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is <b>not</b>
     * registered for applying retransformations.
     * </p>
     * <p>
     * Using this strategy, a redefinition is applied as a single transformation request. This means that a single illegal
     * redefinition of a class causes the entire redefinition attempt to fail.
     * </p>
     * <p>
     * <b>Note</b>: When applying a redefinition, it is normally required to use a {@link TypeStrategy} that applies
     * a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
     * the constrains given by this type strategy.
     * </p>
     */
    REDEFINITION(true, false) {
        @Override
        protected void check(Instrumentation instrumentation) {
            if (!instrumentation.isRedefineClassesSupported()) {
                throw new IllegalStateException("Cannot apply redefinition on " + instrumentation);
            }
        }
 
        @Override
        protected Collector make() {
            return new Collector.ForRedefinition();
        }
    },
 
    /**
     * <p>
     * Applies a <b>retransformation</b> to all classes that are already loaded and that would have been transformed if
     * the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is registered
     * for applying retransformations.
     * </p>
     * <p>
     * Using this strategy, a retransformation is applied as a single transformation request. This means that a single illegal
     * retransformation of a class causes the entire retransformation attempt to fail.
     * </p>
     * <p>
     * <b>Note</b>: When applying a retransformation, it is normally required to use a {@link TypeStrategy} that applies
     * a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
     * the constrains given by this type strategy.
     * </p>
     */
    RETRANSFORMATION(true, true) {
        @Override
        protected void check(Instrumentation instrumentation) {
            if (!DISPATCHER.isRetransformClassesSupported(instrumentation)) {
                throw new IllegalStateException("Cannot apply retransformation on " + instrumentation);
            }
        }
 
        @Override
        protected Collector make() {
            return new Collector.ForRetransformation();
        }
    };

RedefinitionStrategy的存在是因为“Most JVMs do not support changes of a class's structure after a class was already loaded. Therefore, it is typically required that this class file transformer was built while enabling disableClassFormatChanges().”

即大部分JVM不支持在类被装载后修改，需要指定对这些已经被装载的类如何Redefine策略

open-jdk8 HotSpot VM instrumentation的支持：支持Redefine，不支持Retransform

最上面抛出异常的图调用方法：this.retransformClasses为反射获取的sun.instrument.InstrumentationImpl#retransformClasses方法：

打断点查看native方法的支持情况，原来这里才会懒加载，即上面instrumentation对retransform支持到这里才是正确的：

支持retransform，也就是说执行retransformClasses0(mNativeAgent, classes);原生方法抛了上述异常，此时ThreadPoolExecutor类应该是被加载了的：

验证：自定义agent也改为和SkyWalking一致的RETRAINSFORMATION策略，期望如果也变得不生效，则说明增强时ThreadPoolExecutor类已被装载，且是策略选择问题

结果：自定义agent依旧生效

说明要么自定义agent没有装载ThreadPoolExecutor类，这个策略自然也就对ThreadPoolExecutor类无效；要么可能压根就不是这个问题

验证自定义agent在增强时是否加载了目标类：断点打在apply:4812, AgentBuilder$RedefinitionStrategy

stack:

apply:4812, AgentBuilder$RedefinitionStrategy (net.bytebuddy.agent.builder)
doInstall:9463, AgentBuilder$Default (net.bytebuddy.agent.builder)
installOn:9384, AgentBuilder$Default (net.bytebuddy.agent.builder)
instrumentation:58, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
premain:38, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
invoke0:-1, NativeMethodAccessorImpl (sun.reflect)
invoke:62, NativeMethodAccessorImpl (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
loadClassAndStartAgent:386, InstrumentationImpl (sun.instrument)
loadClassAndCallPremain:401, InstrumentationImpl (sun.instrument)

发现SkyWalking已经加载了ThreadPoolExecutor，自定义agent没有加载

2.问题定位

通过在ThreadPoolExecutor构造器方法上打断点，定位到加载ThreadPoolExecutor在日志组件中，获取单例FIleWriter时会通过ThreadPoolExecutor创建异步线程

org.apache.skywalking.apm.agent.core.logging.core.FileWriter#FileWriter

private FileWriter() {
    logBuffer = new ArrayBlockingQueue(1024);
    final ArrayList<String> outputLogs = new ArrayList<String>(200);
    Executors.newSingleThreadScheduledExecutor(new DefaultNamedThreadFactory("LogFileWriter"))
             .scheduleAtFixedRate(new RunnableWithExceptionProtection(new Runnable() {
                 @Override
                 public void run() {
                     try {
                         logBuffer.drainTo(outputLogs);
                         for (String log : outputLogs) {
                             writeToFile(log + Constants.LINE_SEPARATOR);
                         }
                         try {
                             fileOutputStream.flush();
                         } catch (IOException e) {
                             e.printStackTrace();
                         }
                     } finally {
                         outputLogs.clear();
                     }
                 }
             }, new RunnableWithExceptionProtection.CallbackWhenException() {
                 @Override
                 public void handle(Throwable t) {
                 }
             }), 0, 1, TimeUnit.SECONDS);
}

3.修复

org.apache.skywalking.apm.agent.core.logging.core.WriterFactory，增加FILE_WRITTER_INIT_FLAG开关，在所有插件和bytebuddy installlOn执行结束前不允许初始化FileWriter，日志只能输出到STDOUT
executors-plugin插件本身在插桩时不能打印日志，否则会死循环递归调用interpretor逻辑，引发stackoverflow等问题

2022年11月

SkyWalking中异步Executors调用不生效问题排查修复

0.前置

`ClassFileTransformer`接口

`redefineClasses`方法

`retransformClasses`方法

1.原始问题

2.问题定位

3.修复

最新文章

最近回复

分类

归档

其它

2022年11月

SkyWalking中异步Executors调用不生效问题排查修复

0.前置

ClassFileTransformer接口

redefineClasses方法

retransformClasses方法

1.原始问题

2.问题定位

3.修复

最新文章

最近回复

分类

归档

其它

`ClassFileTransformer`接口

`redefineClasses`方法

`retransformClasses`方法