SkyWalking中异步Executors调用不生效问题排查修复
TL;DR：热更新问题
要增强的目标类ThreadPoolExecutor，在增强前（bytebuddy.installOn → instrument.addTransformer / redefine / retransform）已经被classloader装载到jvm里
导致instrument对已加载的类增强有些问题，这个问题也许可以通过深入redefine或retransform的机制解决，快速且稳定的解决方案为在插件装载完成前不加载ThreadPoolExecutor类
0.前置
instrument中的几个基本逻辑：
ClassFileTransformer接口
ClassFile实际指的是java字节码，class文件格式，这个对象在内存中，和文件没有关系
Note the term class file is used as defined in section 3.1 of The Java™ Virtual Machine Specification, to mean a sequence of bytes in class file format, whether or not they reside in a file.
jdk8中接口只有一个transform方法，类在加载、redefined或retransformed时会被调用来增强（the transformer's transform method is invoked when classes are loaded, redefined, or retransformed.）
redefineClasses方法
java 5+，已经加载的类重新进行转换处理，即会触发重新加载类定义，需要注意的是，新加载的类不能修改旧有的类声明，譬如不能增加属性、不能修改方法声明
retransformClasses方法
java 6+，与如上类似，但不是重新进行转换处理，而是直接把处理结果(bytecode)直接给JVM
“Agents use these methods to retransform previously loaded classes without needing to access their class files.”
redefine和retransform的区别：https://stackoverflow.com/questions/19009583/difference-between-redefine-and-retransform-in-javaagent
1.原始问题
自定义executors插件未生效，调试发现onInstall时抛UnsupportedOperationException，且未打印出来，这里异常栈顶是sun.instrument.InstrumentationImpl#retransformClasses，由SkyWalking调用ByteBuddy时指定了AgentBuilder.RedefinitionStrategy.RETRANSFORMATION策略加载
对比相同使用bytebuddy的自定义agent生效：https://gitlab.sunyongfei.com/platform-basic/java-agents/tree/threadpool-qy/thread-agent
其中onInstall前的RedefinitionStrategy不同：
SkyWalking采用AgentBuilder.RedefinitionStrategy.RETRANSFORMATION，自定义agent采用AgentBuilder.RedefinitionStrategy.REDEFINITION
net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy:
/**
* <p>
* A redefinition strategy regulates how already loaded classes are modified by a built agent.
* </p>
* <p>
* <b>Important</b>: Most JVMs do not support changes of a class's structure after a class was already
* loaded. Therefore, it is typically required that this class file transformer was built while enabling
* {@link AgentBuilder#disableClassFormatChanges()}.
* </p>
*/
enum RedefinitionStrategy {
/**
* Disables redefinition such that already loaded classes are not affected by the agent.
*/
DISABLED(false, false) {
@Override
public void apply(Instrumentation instrumentation,
AgentBuilder.Listener listener,
CircularityLock circularityLock,
PoolStrategy poolStrategy,
LocationStrategy locationStrategy,
DiscoveryStrategy discoveryStrategy,
BatchAllocator redefinitionBatchAllocator,
Listener redefinitionListener,
LambdaInstrumentationStrategy lambdaInstrumentationStrategy,
DescriptionStrategy descriptionStrategy,
FallbackStrategy fallbackStrategy,
RawMatcher matcher) {
/* do nothing */
}
@Override
protected void check(Instrumentation instrumentation) {
throw new IllegalStateException("Cannot apply redefinition on disabled strategy");
}
@Override
protected Collector make() {
throw new IllegalStateException("A disabled redefinition strategy cannot create a collector");
}
},
/**
* <p>
* Applies a <b>redefinition</b> to all classes that are already loaded and that would have been transformed if
* the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is <b>not</b>
* registered for applying retransformations.
* </p>
* <p>
* Using this strategy, a redefinition is applied as a single transformation request. This means that a single illegal
* redefinition of a class causes the entire redefinition attempt to fail.
* </p>
* <p>
* <b>Note</b>: When applying a redefinition, it is normally required to use a {@link TypeStrategy} that applies
* a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
* the constrains given by this type strategy.
* </p>
*/
REDEFINITION(true, false) {
@Override
protected void check(Instrumentation instrumentation) {
if (!instrumentation.isRedefineClassesSupported()) {
throw new IllegalStateException("Cannot apply redefinition on " + instrumentation);
}
}
@Override
protected Collector make() {
return new Collector.ForRedefinition();
}
},
/**
* <p>
* Applies a <b>retransformation</b> to all classes that are already loaded and that would have been transformed if
* the built agent was registered before they were loaded. The created {@link ClassFileTransformer} is registered
* for applying retransformations.
* </p>
* <p>
* Using this strategy, a retransformation is applied as a single transformation request. This means that a single illegal
* retransformation of a class causes the entire retransformation attempt to fail.
* </p>
* <p>
* <b>Note</b>: When applying a retransformation, it is normally required to use a {@link TypeStrategy} that applies
* a redefinition instead of rebasing classes such as {@link TypeStrategy.Default#REDEFINE}. Also, consider
* the constrains given by this type strategy.
* </p>
*/
RETRANSFORMATION(true, true) {
@Override
protected void check(Instrumentation instrumentation) {
if (!DISPATCHER.isRetransformClassesSupported(instrumentation)) {
throw new IllegalStateException("Cannot apply retransformation on " + instrumentation);
}
}
@Override
protected Collector make() {
return new Collector.ForRetransformation();
}
};RedefinitionStrategy的存在是因为“Most JVMs do not support changes of a class's structure after a class was already loaded. Therefore, it is typically required that this class file transformer was built while enabling disableClassFormatChanges().”
即大部分JVM不支持在类被装载后修改，需要指定对这些已经被装载的类如何Redefine策略
open-jdk8 HotSpot VM instrumentation的支持：支持Redefine，不支持Retransform
最上面抛出异常的图调用方法：this.retransformClasses为反射获取的sun.instrument.InstrumentationImpl#retransformClasses方法：
打断点查看native方法的支持情况，原来这里才会懒加载，即上面instrumentation对retransform支持到这里才是正确的：
支持retransform，也就是说执行retransformClasses0(mNativeAgent, classes);原生方法抛了上述异常，此时ThreadPoolExecutor类应该是被加载了的：
验证：自定义agent也改为和SkyWalking一致的RETRAINSFORMATION策略，期望如果也变得不生效，则说明增强时ThreadPoolExecutor类已被装载，且是策略选择问题
结果：自定义agent依旧生效
说明要么自定义agent没有装载ThreadPoolExecutor类，这个策略自然也就对ThreadPoolExecutor类无效；要么可能压根就不是这个问题
验证自定义agent在增强时是否加载了目标类：断点打在apply:4812, AgentBuilder$RedefinitionStrategy
stack:
apply:4812, AgentBuilder$RedefinitionStrategy (net.bytebuddy.agent.builder)
doInstall:9463, AgentBuilder$Default (net.bytebuddy.agent.builder)
installOn:9384, AgentBuilder$Default (net.bytebuddy.agent.builder)
instrumentation:58, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
premain:38, ThreadPoolAgent (com.sunyongfei.platform.basic.agent.threadpool)
invoke0:-1, NativeMethodAccessorImpl (sun.reflect)
invoke:62, NativeMethodAccessorImpl (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
loadClassAndStartAgent:386, InstrumentationImpl (sun.instrument)
loadClassAndCallPremain:401, InstrumentationImpl (sun.instrument)发现SkyWalking已经加载了ThreadPoolExecutor，自定义agent没有加载
2.问题定位
通过在ThreadPoolExecutor构造器方法上打断点，定位到加载ThreadPoolExecutor在日志组件中，获取单例FIleWriter时会通过ThreadPoolExecutor创建异步线程
org.apache.skywalking.apm.agent.core.logging.core.FileWriter#FileWriter
private FileWriter() {
logBuffer = new ArrayBlockingQueue(1024);
final ArrayList<String> outputLogs = new ArrayList<String>(200);
Executors.newSingleThreadScheduledExecutor(new DefaultNamedThreadFactory("LogFileWriter"))
.scheduleAtFixedRate(new RunnableWithExceptionProtection(new Runnable() {
@Override
public void run() {
try {
logBuffer.drainTo(outputLogs);
for (String log : outputLogs) {
writeToFile(log + Constants.LINE_SEPARATOR);
}
try {
fileOutputStream.flush();
} catch (IOException e) {
e.printStackTrace();
}
} finally {
outputLogs.clear();
}
}
}, new RunnableWithExceptionProtection.CallbackWhenException() {
@Override
public void handle(Throwable t) {
}
}), 0, 1, TimeUnit.SECONDS);
}3.修复
org.apache.skywalking.apm.agent.core.logging.core.WriterFactory，增加FILE_WRITTER_INIT_FLAG开关，在所有插件和bytebuddy installlOn执行结束前不允许初始化FileWriter，日志只能输出到STDOUT
executors-plugin插件本身在插桩时不能打印日志，否则会死循环递归调用interpretor逻辑，引发stackoverflow等问题
