Choosing log file dynamically

Question

I have an application with a number of components that create data in a database. Each component logs what it is doing when creating data. There are many such components and the application is flexible so that it does not always need to execute the same set of these data-creation components each time it runs.

Currently, everything logs to a single file, which is producing files that are starting to get unmanageable. I'd like it if each component could log to a file whose name describes which component wrote it - ComponentA should log to ComponentA-dataCreationPhase.log.

Most solutions I've seen seem to assume that the different loggers will be static so that it's OK to look them up by name, such as LogManager.getLogger("ComponentA"); - assuming a logger with that name is already configured in my log4j2.xml. Other solutions that I've seen have used Routing and ThreadContexts - but I'm not sure this will work since these components will probably all execute in the same thread.

How can I get each component (many are distinct classes, but some are just different instances of the same class, just configured differently) to log to its own log file? Ideally, this would be done based on the existing log4j2.xml file, as the log4j2.xml might have some user-specified configurations that I'd want to propagate to the component-specific loggers, such as logging path and logging level.

Answer 1

I found this answer:

https://stackoverflow.com/a/38096181/192801

And I used it as the basis of a new function that I added to all Components, via a default method in the top-level interface.

Added to the interface for all components:

public interface Component {

default Logger createLogger(String logFileName, String oldAppenderName, String newAppenderName, boolean append, Level level)
{
    LoggerContext context = (LoggerContext) LogManager.getContext(false);
    Configuration configuration = context.getConfiguration();

    Appender oldAppender = configuration.getAppender(oldAppenderName);

    Layout<? extends Serializable> oldLayout = oldAppender.getLayout();

    // create new appender/logger
    LoggerConfig loggerConfig = new LoggerConfig(logFileName, level, false);

    Appender appender ;
    // In my case, it is possible that the old appender could
    // either be a simple FileAppender, or a  RollingRandomAccessFileAppender,
    // so I'd like the new one to be of the same type as the old one.
    // I have yet to find a more elegant way to do create a new Appender
    // of *any* type and then copy all relevant config.
    if (oldAppender instanceof RollingRandomAccessFileAppender)
    {
        int bufferSize = ((RollingRandomAccessFileAppender)oldAppender).getBufferSize();

        RollingRandomAccessFileManager oldMananger = (RollingRandomAccessFileManager)((RollingRandomAccessFileAppender) oldAppender).getManager();

        TriggeringPolicy triggerPolicy = oldMananger.getTriggeringPolicy();
        RolloverStrategy rollStrategy = oldMananger.getRolloverStrategy();
        Filter filter = ((RollingRandomAccessFileAppender)oldAppender).getFilter();
        // Inject new log file name into filePattern so that file rolling will work properly 
        String pattern = ((RollingRandomAccessFileAppender)oldAppender).getFilePattern().replaceAll("/[^/]*-\\%d\\{yyyy-MM-dd\\}\\.\\%i\\.log\\.gz", "/"+logFileName+"-%d{yyyy-MM-dd}.%i.log.gz");
        appender = RollingRandomAccessFileAppender.newBuilder()
                                           .withFileName("logs/" + logFileName + ".log")
                                           .withFilePattern(pattern)
                                           .withAppend(append)
                                           .withName(newAppenderName)
                                           .withBufferSize(bufferSize)
                                           .withPolicy(triggerPolicy)
                                           .withStrategy(rollStrategy)
                                           .withLayout(oldLayout)
                                           .withImmediateFlush(true)
                                           .withFilter(filter)
                                           .build();
    }
    else
    {
        appender = FileAppender.newBuilder()
                               .withFileName("logs/" + logFileName + ".log")
                                .withAppend(append)
                                .withName(newAppenderName)
                                .withLayout(oldLayout)
                                .setConfiguration(configuration)
                                .withLocking(false)
                                .withImmediateFlush(true)
                                .withIgnoreExceptions(true)
                                .withBufferSize(8192)
                                .withFilter(null)
                                .withAdvertise(false)
                                .withAdvertiseUri("")
                            .build();
    }
    appender.start();
    loggerConfig.addAppender(appender, level, null);
    configuration.addLogger(logFileName, loggerConfig);
    context.updateLoggers();

    return context.getLogger(logFileName);
}

Use of this function happens inside the constructor of the components:

public class ComponentA implements Component
{
    private Logger logger = LogManager.getLogger();
    public ComponentA(String componentName)
    {
        this.logger = this.createLogger(componentName, "MyOriginalFileAppender", componentName+"Appender", true, Level.DEBUG, this.logger);
    }
}

and elsewhere:

ComponentA fooSubtypeAComponent = new ComponentA("FooA");
ComponentA barSubtypeAComponent = new ComponentA("BarA");
ComponentB fooVariantTypeBComponent = new ComponentB("FooB");

The original Appenders + Loggers snippet from the Log4j config:

<Appenders>
    <!-- STDOUT appender. -->
    <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout
            pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36}#%M - %msg%n" />
    </Console>

    <RollingRandomAccessFile name="MyOriginalFileAppender" fileName="${baseDir}/${defaultLogName}.log" filePattern="${baseDir}/$${date:yyyy-MM}/${defaultLogName}-%d{MM-dd-yyyy}.%i.log.gz">
        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36}#%M - %msg%n" />
        <Policies>
            <!-- <TimeBasedTriggeringPolicy /> -->
            <!-- Let's try cron triggering policy configured to trigger every day at midnight -->
            <CronTriggeringPolicy schedule="0 0 0 * * ?"/>
            <SizeBasedTriggeringPolicy size="25 MB" />
        </Policies>
    </RollingRandomAccessFile>
</Appenders>
<Loggers>
    <Root level="debug">
        <!-- only write INFO level to the file. -->
        <AppenderRef ref="MyOriginalFileAppender" level="debug"/>
        <!-- Console shows everything at DEBUG level-->
        <AppenderRef ref="Console" level="info" />
    </Root>
</Loggers>

The result should be three logs files: logs/FooA.log, logs/BarA.log, logs/FooB.log - one log file for each instance shown above. Still has some kinks to iron out, but I think this will work fine.