PreparedStatement IN clause alternatives?

Question

What are the best workarounds for using a SQL IN clause with instances of java.sql.PreparedStatement, which is not supported for multiple values du

Answer 1

Limitations of the in() operator is the root of all evil.

It works for trivial cases, and you can extend it with "automatic generation of the prepared statement" however it is always having its limits.

• if you're creating a statement with variable number of parameters, that will make an sql parse overhead at each call
• on many platforms, the number of parameters of in() operator are limited
• on all platforms, total SQL text size is limited, making impossible for sending down 2000 placeholders for the in params
• sending down bind variables of 1000-10k is not possible, as the JDBC driver is having its limitations

The in() approach can be good enough for some cases, but not rocket proof :)

The rocket-proof solution is to pass the arbitrary number of parameters in a separate call (by passing a clob of params, for example), and then have a view (or any other way) to represent them in SQL and use in your where criteria.

A brute-force variant is here http://tkyte.blogspot.hu/2006/06/varying-in-lists.html

However if you can use PL/SQL, this mess can become pretty neat.

function getCustomers(in_customerIdList clob) return sys_refcursor is 
begin
    aux_in_list.parse(in_customerIdList);
    open res for
        select * 
        from   customer c,
               in_list v
        where  c.customer_id=v.token;
    return res;
end;

Then you can pass arbitrary number of comma separated customer ids in the parameter, and:

• will get no parse delay, as the SQL for select is stable
• no pipelined functions complexity - it is just one query
• the SQL is using a simple join, instead of an IN operator, which is quite fast
• after all, it is a good rule of thumb of not hitting the database with any plain select or DML, since it is Oracle, which offers lightyears of more than MySQL or similar simple database engines. PL/SQL allows you to hide the storage model from your application domain model in an effective way.

The trick here is:

• we need a call which accepts the long string, and store somewhere where the db session can access to it (e.g. simple package variable, or dbms_session.set_context)
• then we need a view which can parse this to rows
• and then you have a view which contains the ids you're querying, so all you need is a simple join to the table queried.

The view looks like:

create or replace view in_list
as
select
    trim( substr (txt,
          instr (txt, ',', 1, level  ) + 1,
          instr (txt, ',', 1, level+1)
             - instr (txt, ',', 1, level) -1 ) ) as token
    from (select ','||aux_in_list.getpayload||',' txt from dual)
connect by level <= length(aux_in_list.getpayload)-length(replace(aux_in_list.getpayload,',',''))+1

where aux_in_list.getpayload refers to the original input string.

---

A possible approach would be to pass pl/sql arrays (supported by Oracle only), however you can't use those in pure SQL, therefore a conversion step is always needed. The conversion can not be done in SQL, so after all, passing a clob with all parameters in string and converting it witin a view is the most efficient solution.

Answer 2

I just worked out a PostgreSQL-specific option for this. It's a bit of a hack, and comes with its own pros and cons and limitations, but it seems to work and isn't limited to a specific development language, platform, or PG driver.

The trick of course is to find a way to pass an arbitrary length collection of values as a single parameter, and have the db recognize it as multiple values. The solution I have working is to construct a delimited string from the values in the collection, pass that string as a single parameter, and use string_to_array() with the requisite casting for PostgreSQL to properly make use of it.

So if you want to search for "foo", "blah", and "abc", you might concatenate them together into a single string as: 'foo,blah,abc'. Here's the straight SQL:

select column from table
where search_column = any (string_to_array('foo,blah,abc', ',')::text[]);

You would obviously change the explicit cast to whatever you wanted your resulting value array to be -- int, text, uuid, etc. And because the function is taking a single string value (or two I suppose, if you want to customize the delimiter as well), you can pass it as a parameter in a prepared statement:

select column from table
where search_column = any (string_to_array($1, ',')::text[]);

This is even flexible enough to support things like LIKE comparisons:

select column from table
where search_column like any (string_to_array('foo%,blah%,abc%', ',')::text[]);

Again, no question it's a hack, but it works and allows you to still use pre-compiled prepared statements that take *ahem* discrete parameters, with the accompanying security and (maybe) performance benefits. Is it advisable and actually performant? Naturally, it depends, as you've got string parsing and possibly casting going on before your query even runs. If you're expecting to send three, five, a few dozen values, sure, it's probably fine. A few thousand? Yeah, maybe not so much. YMMV, limitations and exclusions apply, no warranty express or implied.

But it works.

Answer 3

SetArray is the best solution but its not available for many older drivers. The following workaround can be used in java8

String baseQuery ="SELECT my_column FROM my_table where search_column IN (%s)"

String markersString = inputArray.stream().map(e -> "?").collect(joining(","));
String sqlQuery = String.format(baseSQL, markersString);

//Now create Prepared Statement and use loop to Set entries
int index=1;

for (String input : inputArray) {
     preparedStatement.setString(index++, input);
}

This solution is better than other ugly while loop solutions where the query string is built by manual iterations

Answer 4

My workaround (JavaScript)

    var s1 = " SELECT "

 + "FROM   table t "

 + "  where t.field in ";

  var s3 = '(';

  for(var i =0;i<searchTerms.length;i++)
  {
    if(i+1 == searchTerms.length)
    {
     s3  = s3+'?)';
    }
    else
    {
        s3  = s3+'?, ' ;
    }
   }
    var query = s1+s3;

    var pstmt = connection.prepareStatement(query);

     for(var i =0;i<searchTerms.length;i++)
    {
        pstmt.setString(i+1, searchTerms[i]);
    }

SearchTerms is the array which contains your input/keys/fields etc

Answer 5

My workaround is:

create or replace type split_tbl as table of varchar(32767);
/

create or replace function split
(
  p_list varchar2,
  p_del varchar2 := ','
) return split_tbl pipelined
is
  l_idx    pls_integer;
  l_list    varchar2(32767) := p_list;
  l_value    varchar2(32767);
begin
  loop
    l_idx := instr(l_list,p_del);
    if l_idx > 0 then
      pipe row(substr(l_list,1,l_idx-1));
      l_list := substr(l_list,l_idx+length(p_del));
    else
      pipe row(l_list);
      exit;
    end if;
  end loop;
  return;
end split;
/

Now you can use one variable to obtain some values in a table:

select * from table(split('one,two,three'))
  one
  two
  three

select * from TABLE1 where COL1 in (select * from table(split('value1,value2')))
  value1 AAA
  value2 BBB

So, the prepared statement could be:

  "select * from TABLE where COL in (select * from table(split(?)))"

Regards,

Javier Ibanez

Answer 6

Following Adam's idea. Make your prepared statement sort of select my_column from my_table where search_column in (#) Create a String x and fill it with a number of "?,?,?" depending on your list of values Then just change the # in the query for your new String x an populate