Store big JSON files into Oracle DB

Question

I\'m using the following script in order to read data from MongoDB as JSON files.

DECLARE
  l_param_list     VARCHAR2(512);
 
  l_http_request   UTL_HTTP.req;         


        

Answer 1

Your problem is in your call to UTL_HTTP.read_text. You are passing a CLOB, but read_text only accepts VARCHAR2, so it can return a maximum of 32k bytes.

You need to call it in a loop, using a VARCHAR2 buffer, and concatenate the results into your CLOB, e.g.:

DECLARE
  buf VARCHAR2(32767);
BEGIN
  LOOP
    UTL_HTTP.read_text(l_http_response, buf);
    l_response_text := l_response_text || buf;
  END LOOP;
EXCEPTION
  WHEN UTL_HTTP.end_of_body THEN
    NULL;
END;

http://docs.oracle.com/cd/E11882_01/appdev.112/e40758/u_http.htm#ARPLS71074

Your second problem is that json_ext.get_string only returns a VARCHAR2, which means it is limited to 32767 byes max. I've had a browse of the PL/json wiki, you might need to reach out to one of the authors to find out how to use it to get a CLOB value.

Answer 2

CLOB have a size limit of 4G

But the limitation here is with UTL_HTTP.read_text that returns the result as a VARCHAR2 (you have an implicit conversion here).

To retrieve large text objects from the web easily, you probably need HttpUriType.getClob

---

If for some reason you want to stick with UTL_HTTP, you will have to loop to read your data chunk by chunk. Something like that:

BEGIN
  ...
  l_clob           CLOB;
  l_text           VARCHAR2(32767);
BEGIN
  DBMS_LOB.createtemporary(l_clob, FALSE);

  ...
  l_http_request  := UTL_HTTP.begin_request(your_URI);
  l_http_response := UTL_HTTP.get_response(l_http_request);

  -- Loop to read data chunk by chunk up to the end
  BEGIN
    LOOP
      UTL_HTTP.read_text(l_http_response, l_text, 32766);
      DBMS_LOB.writeappend (l_clob, LENGTH(l_text), l_text);
    END LOOP;
  EXCEPTION
    WHEN UTL_HTTP.end_of_body THEN
      UTL_HTTP.end_response(l_http_response);
  END;

See http://oracle-base.com/articles/misc/retrieving-html-and-binaries-into-tables-over-http.php vor various examples