Mask sensitive data in logs with logback

Question

I need to be able to search an event for any one of a number of patterns and replace the text in the pattern with a masked value. This is a feature in our application intend

Answer 1

You need to wrap layout using LayoutWrappingEncoder. And also I believe you cannot use spring here as logback is not managed by spring.

Here is the updated class.

public class MaskingPatternLayout extends PatternLayout {

    private String patternsProperty;

    public String getPatternsProperty() {
        return patternsProperty;
    }

    public void setPatternsProperty(String patternsProperty) {
        this.patternsProperty = patternsProperty;
    }

    @Override
    public String doLayout(ILoggingEvent event) {
        String message = super.doLayout(event);
        
        if (patternsProperty != null) {
            String[] patterns = patternsProperty.split("\\|");
            for (int i = 0; i < patterns.length; i++) {
                Pattern pattern = Pattern.compile(patterns[i]);

                Matcher matcher = pattern.matcher(event.getMessage());
                if (matcher.find()) {
                    message = matcher.replaceAll("*");
                }
            }
        } else {

        }

        return message;
    }

}

And sample logback.xml

<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
    <file>c:/logs/kp-ws.log</file>
    <append>true</append>
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="com.kp.MaskingPatternLayout">
            <patternsProperty>.*password.*|.*karthik.*</patternsProperty>
            <pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
        </layout>
    </encoder>
</appender>
<root level="DEBUG">
    <appender-ref ref="fileAppender1" />
</root>

---

UPDATE

Here its better approach, set Pattern during init itself. such that we can avoid recreating Pattern again and again and this implementation is close to realistic usecase.

public class MaskingPatternLayout extends PatternLayout {

    private String patternsProperty;
    private Optional<Pattern> pattern;

    public String getPatternsProperty() {
        return patternsProperty;
    }

    public void setPatternsProperty(String patternsProperty) {
        this.patternsProperty = patternsProperty;
        if (this.patternsProperty != null) {
            this.pattern = Optional.of(Pattern.compile(patternsProperty, Pattern.MULTILINE));
        } else {
            this.pattern = Optional.empty();
        }
    }

        @Override
        public String doLayout(ILoggingEvent event) {
            final StringBuilder message = new StringBuilder(super.doLayout(event));
    
            if (pattern.isPresent()) {
                Matcher matcher = pattern.get().matcher(message);
                while (matcher.find()) {
    
                    int group = 1;
                    while (group <= matcher.groupCount()) {
                        if (matcher.group(group) != null) {
                            for (int i = matcher.start(group); i < matcher.end(group); i++) {
                                message.setCharAt(i, '*');
                            }
                        }
                        group++;
                    }
                }
            }
            return message.toString();
        }
    
    }

And the updated Configuration file.

<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
    <file>c:/logs/kp-ws.log</file>
    <append>true</append>
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="com.kp.MaskingPatternLayout">
            <patternsProperty>(password)|(karthik)</patternsProperty>
            <pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
        </layout>
    </encoder>
</appender>
<root level="DEBUG">
    <appender-ref ref="fileAppender1" />
</root>

Output

My username=test and password=*******

Answer 2

From the documentation:

replace(p){r, t}    

The pattern p can be arbitrarily complex and in particular can contain multiple conversion keywords.

Facing same problem having to replace 2 patterns in a message, I just tried to chain so p is just an invocation of replace, in my case:

%replace(  %replace(%msg){'regex1', 'replacement1'}  ){'regex2', 'replacement2'}

Worked great, though I wonder if I'm pushing it a bit and p can be indeed that arbitrarily complex.

Answer 3

I've used censor based on RegexCensor from library https://github.com/tersesystems/terse-logback. In logback.xml

<!--censoring information-->
<newRule pattern="*/censor" actionClass="com.tersesystems.logback.censor.CensorAction"/>
<conversionRule conversionWord="censor" converterClass="com.tersesystems.logback.censor.CensorConverter" />
<!--impl inspired by com.tersesystems.logback.censor.RegexCensor -->
<censor name="censor-sensitive" class="com.mycompaqny.config.logging.SensitiveDataCensor"></censor>

where i put list regex replacements.

@Getter@Setter    
public class SensitiveDataCensor extends ContextAwareBase implements Censor, LifeCycle {
    protected volatile boolean started = false;
    protected String name;
    private List<Pair<Pattern, String>> replacementPhrases = new ArrayList<>();

    public void start() {

        String ssnJsonPattern = "\"(ssn|socialSecurityNumber)(\"\\W*:\\W*\".*?)-(.*?)\"";
        replacementPhrases.add(Pair.of(Pattern.compile(ssnJsonPattern), "\"$1$2-****\""));

        String ssnXmlPattern = "<(ssn|socialSecurityNumber)>(\\W*.*?)-(.*?)</";
        replacementPhrases.add(Pair.of(Pattern.compile(ssnXmlPattern), "<$1>$2-****</"));

        started = true;
    }

    public void stop() {
        replacementPhrases.clear();
        started = false;
    }

    public CharSequence censorText(CharSequence original) {
        CharSequence outcome = original;
        for (Pair<Pattern, String> replacementPhrase : replacementPhrases) {
            outcome = replacementPhrase.getLeft().matcher(outcome).replaceAll(replacementPhrase.getRight());
        } 
        return outcome;
    }
}

and used it in logback.xml like this

<message>[ignore]</message> <---- IMPORTANT to disable original message field so you get only censored message
...
<pattern>
    {"message": "%censor(%msg){censor-sensitive}"}
</pattern>

Answer 4

Here is my approach, maybe it can help somebody

Try this one. 1. First of all, we should create a class for handling our logs (each row)

public class PatternMaskingLayout extends PatternLayout {

private Pattern multilinePattern;
private List<String> maskPatterns = new ArrayList<>();

public void addMaskPattern(String maskPattern) { // invoked for every single entry in the xml
    maskPatterns.add(maskPattern);
    multilinePattern = Pattern.compile(
            String.join("|", maskPatterns), // build pattern using logical OR
            Pattern.MULTILINE
    );
}

@Override
public String doLayout(ILoggingEvent event) {
    return maskMessage(super.doLayout(event)); // calling superclass method is required
}

private String maskMessage(String message) {
    if (multilinePattern == null) {
        return message;
    }
    StringBuilder sb = new StringBuilder(message);
    Matcher matcher = multilinePattern.matcher(sb);
    while (matcher.find()) {
        if (matcher.group().contains("creditCard")) {
            maskCreditCard(sb, matcher);
        } else if (matcher.group().contains("email")) {
            // your logic for this case
        }
    }
    return sb.toString();
}
private void maskCreditCard(StringBuilder sb, Matcher matcher) {
    //here is our main logic for masking sensitive data
    String targetExpression = matcher.group();
    String[] split = targetExpression.split("=");
    String pan = split[1];
    String maskedPan = Utils.getMaskedPan(pan);
    int start = matcher.start() + split[0].length() + 1;
    int end = matcher.end();
    sb.replace(start, end, maskedPan);
}

}

2. The second step is we should create appender for logback into logback.xml

   <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
   <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
       <layout class="com.bpcbt.micro.utils.PatternMaskingLayout">
           <maskPattern>creditCard=\d+</maskPattern> <!-- SourcePan pattern -->
           <pattern>%d{dd/MM/yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n%ex</pattern>-->
       </layout>
   </encoder>
   

3. Now we can use logger into our code

   log.info("card context set for creditCard={}", creditCard);

4. As a result, we will see

   one row from logs

   card context set for creditCard=11111******111

without these options, our logs would be like this row

card context set for creditCard=1111111111111

Answer 5

A very similar but slightly different approach evolves around customizing CompositeConverter and defining a <conversionRule ...> within the logback that references the custom converter.

In one of my tech-demo projects I defined a MaskingConverter class that defines a series of patterns the logging event is analyzed with and on a match updated which is used inside my logback configuration.

As link-only answers are not that beloved here at SO I'll post the important parts of the code here and explain what it does and why it is set up like that. Starting with the Java-based custom converter class:

public class MaskingConverter<E extends ILoggingEvent> extends CompositeConverter<E> {

  public static final String CONFIDENTIAL = "CONFIDENTIAL";
  public static final Marker CONFIDENTIAL_MARKER = MarkerFactory.getMarker(CONFIDENTIAL);

  private Pattern keyValPattern;
  private Pattern basicAuthPattern;
  private Pattern urlAuthorizationPattern;

  @Override
  public void start() {
    keyValPattern = Pattern.compile("(pw|pwd|password)=.*?(&|$)");
    basicAuthPattern = Pattern.compile("(B|b)asic ([a-zA-Z0-9+/=]{3})[a-zA-Z0-9+/=]*([a-zA-Z0-9+/=]{3})");
    urlAuthorizationPattern = Pattern.compile("//(.*?):.*?@");
    super.start();
  }

  @Override
  protected String transform(E event, String in) {
    if (!started) {
      return in;
    }
    Marker marker = event.getMarker();
    if (null != marker && CONFIDENTIAL.equals(marker.getName())) {
      // key=value[&...] matching
      Matcher keyValMatcher = keyValPattern.matcher(in);
      // Authorization: Basic dXNlcjpwYXNzd29yZA==
      Matcher basicAuthMatcher = basicAuthPattern.matcher(in);
      // sftp://user:password@host:port/path/to/resource
      Matcher urlAuthMatcher = urlAuthorizationPattern.matcher(in);

      if (keyValMatcher.find()) {
        String replacement = "$1=XXX$2";
        return keyValMatcher.replaceAll(replacement);
      } else if (basicAuthMatcher.find()) {
        return basicAuthMatcher.replaceAll("$1asic $2XXX$3");
      } else if (urlAuthMatcher.find()) {
        return urlAuthMatcher.replaceAll("//$1:XXX@");
      }
    }
    return in;
  }
}

This class defines a number of RegEx patterns the respective log-line should be compared against and on a match lead to an update of the event by masking the passwords.

Note that this code sample assumes that a log line only contains one kind of password. You are of course free to adapt the bahvior to your needs in case you want to probe each line for multiple pattern matches.

To apply this converter one simply has to add the following line to the logback configuration:

<conversionRule conversionWord="mask" converterClass="at.rovo.awsxray.utils.MaskingConverter"/>

which defines a new function mask which can be used in a pattern in order to mask any log events matching any of the patterns defined in the custom converter. This function can now be used inside a pattern to tell Logback to perform the logic on each log event. The respective pattern might be something along the lines below:

<property name="patternValue"
          value="%date{yyyy-MM-dd HH:mm:ss} [%-5level] - %X{FILE_ID} - %mask(%msg) [%thread] [%logger{5}] %n"/>

<!-- Appender definitions-->

<appender class="ch.qos.logback.core.ConsoleAppender" name="console">
    <encoder>
        <pattern>${patternValue}</pattern>
    </encoder>
</appender>

where %mask(%msg) will take the original log-line as input and perform the password masking on each of the lines passed to that function.

As probing each line for one or multiple pattern matches might be costly, the Java code above includes Markers that can be used in log statements to send certain meta information on the log statement itself to Logback/SLF4J. Based on such markers different behaviors might be achievable. In the scenario presented a marker interface can be used to tell Logback that the respective log line contains confidential information and thus requires masking if it matches. Any log line that isn't marked as confidential will be ignored by this converter which helps in pumping out the lines faster as no pattern matching needs to be performed on those lines.

In Java such a marker can be added to a log statement like this:

LOG.debug(MaskingConverter.CONFIDENTIAL_MARKER, "Received basic auth header: {}",
      connection.getBasicAuthentication());

which might produce a log line similar to Received basic auth header: Basic QlRXXXlQ= for the above mentioned custom converter, which leaves the first and last couple of characters in tact but obfuscates the middle bits with XXX.

Answer 6

I've got censors in https://github.com/tersesystems/terse-logback that allow you to define a censor in one place and then refer to it in multiple appenders.