How to silently truncate strings while storing them when they are longer than the column length definition?

Question

I have a web app, using EclipseLink and MySQL for storing data. Some of these data are strings, ie varchars in the DB. In the code of entities, the strings have attributes

Answer 1

There is another way to do it, may be even faster (at least it works on 5th version of MySql):

First, check your sql_mode setting: there is a detailed description how to do it. This setting should have value of "" for windows, and "modes" for Unix.

That didn't help me, so I found another setting, this time in jdbc:

jdbcCompliantTruncation=false. 

In my case (I used persistence) it is defined in persistence.xml:

<property name="javax.persistence.jdbc.url" value="jdbc:mysql://127.0.0.1:3306/dbname?jdbcCompliantTruncation=false"/>

These 2 settings work only together, I tried to use them separately and had no effect.

Notice: remember that by setting sql_mode as described above you switch off important database checks, so please do it carefully.

Answer 2

Two very important features of UI design:

1. You should never silently alter user data - the user should be aware, in control and consent to such changes.
2. You should never allow data to be entered that cannot be handled - use UI/HTML attributes and validation to restrict data to legal values

The answer to your problem is very simple. Simply limit your UI input fields to 256 characters, to match the database field length:

<input type="text" name="widgetDescription" maxlength="256">

This is a system limitation - the user should not be able to enter more data than this. If this is insufficient, change the database.

Answer 3

One can truncate a string according to the JPA annotations in the setter for the corresponding field:

public void setX(String x) {
    try {
        int size = getClass().getDeclaredField("x").getAnnotation(Column.class).length();
        int inLength = x.length();
        if (inLength>size)
        {
            x = x.substring(0, size);
        }
    } catch (NoSuchFieldException ex) {
    } catch (SecurityException ex) {
    }
    this.x = x;
}

The annotation itself should look like:

@Column(name = "x", length=100)
private String x;

(Based on https://stackoverflow.com/a/1946901/16673)

The annotations can be recreated from the database if the database changes, as hinted in the comment to https://stackoverflow.com/a/7648243/16673

Answer 4

Since the Exception seems to be raised at the database level during the commit() process, a before-insert Trigger on the target table/s could truncate the new values before they are committed, bypassing the error.

Answer 5

You have different solutions and false solutions.

Using trigger or any database level trick

This will create inconsistency between the objects in the ORM and their serialized form in the DB. If you use second level caching: it can leads to lots of troubles. In my opinion this is not a real solution for a general use case.

Using pre-insert, pre-update hooks

You will silently modify the user data just before persisting it. So your code may behave differently depending on the fact that the object is already persisted or not. It may cause trouble too. Additionally, you must be careful with the order of hooks invocation: be sure that your "field-truncator-hook" is the very first one called by the persistence provider.

Using aop to intercept call to setters

This solution will more or less silently modify user/automatic input, but your objects won't be modified after you use them to do some business logic. So this is more acceptable than the 2 previous solutions, but the setters won't follow the contract of usual setters. Additionally, if you use field injection: it will bypass the aspect (depending on your configuration, jpa provider may use field injection. Most of the time: Spring use setters. I guess some other frameworks may use field injection, so even if you don't use it explicitelly be aware of the underlying implementation of frameworks you are using).

Using aop to intercept field modification

Similar to the previous solution except that field injection will also be intercepted by the aspect. (note that I never wrote an aspect doing this kind of things, but I think it's feasible)

Adding a controller layer to check field length before calling the setter

Probably the best solution in terms of data integrity. But it may requires a lot of refactoring. For a general use case, this is (in my opinion) the only acceptable solution.

Depending on your use case, you may choose any of those solutions. Be aware of the drawbacks.

Answer 6

There was already answer mentioned Converters, but I want to add more details. My answer also assumes Converters from JPA, not EclipseLink specific.

At first create this class - special type converter which responsibility will be to truncate value at the persistence moment:

import javax.persistence.AttributeConverter;
import javax.persistence.Convert;

@Convert
public class TruncatedStringConverter implements AttributeConverter<String, String> {
  private static final int LIMIT = 999;

  @Override
  public String convertToDatabaseColumn(String attribute) {
    if (attribute == null) {
      return null;
    } else if (attribute.length() > LIMIT) {
      return attribute.substring(0, LIMIT);
    } else {
      return attribute;
    }
  }

  @Override
  public String convertToEntityAttribute(String dbData) {
    return dbData;
  }
}

Then you can use it in your entities like this:

@Entity(name = "PersonTable")
public class MyEntity {
    
    @Convert(converter = TruncatedStringConverter.class)
    private String veryLongValueThatNeedToBeTruncated;
 
    //...
}

Related article about JPA Converters: http://www.baeldung.com/jpa-attribute-converters