Is there a common Java library that will handle URL encoding/decoding for a collection of strings?

Question

I often have to url encode or decode a large collection or array of strings. Besides iterating through them and using the static URLDecoder.decode(string, "UTF-8"), are there any libraries out there that will make this type of operation more performant?

A colleague insists that using the static method to decode the strings in-place is not thread safe. Why would that be?

Answer 1

The JDK URLDecoder wasn't implemented efficiently. Most notably, internally it relies on StringBuffer (which unnecessarily introduces synchronization in the case of URLDecoder). The Apache commons provides URLCodec, but it has also been reported to have similar issues in regards to performance but I haven't verified that's still the case in most recent version.

Mark A. Ziesemer wrote a post a while back regarding the issues and performance with URLDecoder. He logged some bug reports and ended up writing a complete replacement. Because this is SO, I'll quote some key excerpts here, but you should really read the entire source article here: http://blogger.ziesemer.com/2009/05/improving-url-coder-performance-java.html

Selected quotes:

Java provides a default implementation of this functionality in java.net.URLEncoder and java.net.URLDecoder. Unfortunately, it is not the best performing, due to both how the API was written as well as details within the implementation. A number of performance-related bugs have been filed on sun.com in relation to URLEncoder.

There is an alternative: org.apache.commons.codec.net.URLCodec from Apache Commons Codec. (Commons Codec also provides a useful implementation for Base64 encoding.) Unfortunately, Commons' URLCodec suffers some of the same issues as Java's URLEncoder/URLDecoder.

...

Recommendations for both the JDK and Commons:

When constructing any of the "buffer" classes, e.g. ByteArrayOutputStream, CharArrayWriter, StringBuilder, or StringBuffer, estimate and pass-in an estimated capacity. The JDK's URLEncoder currently does this for its StringBuffer, but should do this for its CharArrayWriter instance as well. Common's URLCodec should do this for its ByteArrayOutputStream instance. If the classes' default buffer sizes are too small, they may have to resize by copying into new, larger buffers - which isn't exactly a "cheap" operation. If the classes' default buffer sizes are too large, memory may be unnecessarily wasted.

Both implementations are dependent on Charsets, but only accept them as their String name. Charset provides a simple and small cache for name lookups - storing only the last 2 Charsets used. This should not be relied upon, and both should accept Charset instances for other interoperability reasons as well.

Both implementations only handle fixed-size inputs and outputs. The JDK's URLEncoder only works with String instances. Commons' URLCodec is also based on Strings, but also works with byte[] arrays. This is a design-level constraint that essentially prevents efficient processing of larger or variable-length inputs. Instead, the "stream-supporting" interfaces such as CharSequence, Appendable, and java.nio's Buffer implementations of ByteBuffer and CharBuffer should be supported.

...

Note that com.ziesemer.utils.urlCodec is over 3x as fast as the JDK URLEncoder, and over 1.5x as fast as the JDK URLDecoder. (The JDK's URLDecoder was faster than the URLEncoder, so there wasn't as much room for improvement.)

I think your colleague is wrong to suggest URLDecode is not thread-safe. Other answers here explain in detail.

EDIT [2012-07-03] - Per later comment posted by OP

Not sure if you were looking for more ideas or not? You are correct that if you intend to operate on the list as an atomic collection, then you would have to synchronize all access to the list, including references outside of your method. However, if you are okay with the returned list contents potentially differing from the original list, then a brute force approach for operating on a "batch" of strings from a collection that might be modified by other threads could look something like this:

/**
 * @param origList will be copied by this method so that origList can continue
 *                 to be read/write by other threads. 
 * @return list containing  decoded strings for each entry that was 
           in origList at time of copy.
 */
public List<String> decodeListOfStringSafely(List<String> origList)
        throws UnsupportedEncodingException {
    List<String> snapshotList = new ArrayList<String>(origList);
    List<String> newList  = new ArrayList<String>(); 

    for (String urlStr : snapshotList) {
      String decodedUrlStr  = URLDecoder.decode(urlStr, "UTF8");
          newList.add(decodedUrlStr);
    }

    return newList;
}

If that does not help, then I'm still not sure what you are after and you would be better served to create a new, more concise, question. If that is what you were asking about, then be careful because this example out of context is not a good idea for many reasons.

Answer 2

Thread Safety is actually never really necessary with static functions (or it is a design failure). Especially not, if the don't even access static Variables in the class.

I would suggest using the function you used before, and iterating through the collection

Answer 3

Basically there's no magic thread-safety applied to static methods or instance methods or constructors. They can all be called on multiple threads concurrently unless synchronization is applied. If they don't fetch or change any shared data, they will generally be safe - if they do access shared data, you need to be more careful.

so In your case you can write synchronized method on top of this urldecoding or encoding by which you can enforce thread safety externally.

Answer 4

Apache has URLCodec which can be used for encoding decoding.

If your static method just works on the local variables or final initialized variables then it is completely thread safe.

As parameters live on stack and they are completely thread safe, final constants are immutable hence cannot be changed.

Following code is completely thread safe:

public static String encodeMyValue(String value){
  // do encoding here
}

Care should be taken if final variables are mutable meaning you cannot reassign it but you can change its internal representation(properties).