Android: Using WebView outside an Activity context

Question

I am trying to achieve Web Scraping through a background IntentService that periodically scrape a website without a view displaying on the users phone.

• Since

Answer 1

Why don't you create a Backend Service that does the scraping for you?

And then you just poll results from a RESTful Webservice or even use a messaging middleware (e.g. ZeroMQ).

Maybe more elegant if it fits your use case: let the Scraping Service send your App Push Messages via GCM :)

Answer 2

You can display a webview from a service. Code below creates a window which your service has access to. The window isn't visible because the size is 0 by 0.

public class ServiceWithWebView extends Service {

    @Override
    public void onCreate() {
        super.onCreate();

        WindowManager windowManager = (WindowManager) getSystemService(WINDOW_SERVICE);
        params = new WindowManager.LayoutParams(WindowManager.LayoutParams.WRAP_CONTENT, WindowManager.LayoutParams.WRAP_CONTENT, WindowManager.LayoutParams.TYPE_SYSTEM_OVERLAY, WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE, PixelFormat.TRANSLUCENT);
        params.gravity = Gravity.TOP | Gravity.LEFT;
        params.x = 0;
        params.y = 0;
        params.width = 0;
        params.height = 0;

        LinearLayout view = new LinearLayout(this);
        view.setLayoutParams(new RelativeLayout.LayoutParams(RelativeLayout.LayoutParams.MATCH_PARENT, RelativeLayout.LayoutParams.MATCH_PARENT));

        WebView wv = new WebView(this);
        wv.setLayoutParams(new LinearLayout.LayoutParams(LinearLayout.LayoutParams.MATCH_PARENT, LinearLayout.LayoutParams.MATCH_PARENT));
        view.addView(wv);
        wv.loadUrl("http://google.com");

        windowManager.addView(view, params);
    }
}

Also this will require the android.permission.SYSTEM_ALERT_WINDOW permission.

Answer 3

I am not sure if this is a silver bullet to the given problem. As per @Pierre's accepted answer (sounds correct to me)

there is NO possible way to use a WebView in the background while the user is doing other things on the phone without interrupting the user by means of an Activity.

Thus, I believe there must be some architectural/flow/strategy changes that must be done in order to solve this problem.

Proposed Solution #1: Instead of getting a push notification from the server and run a background job and followed by running some JS code or WebView. Instead, Whenever user launch the application one should query the backend server to know whether there is any need to perform any scrapping or not. And on the basis of backend input android client can run JS code or WebView and pass the result back to the server.

I haven't tried this solution. But hope it is feasible.

---

This will also solve the following problem stated in the comments:

Reason for this is because the backend will get detected as a bot scraping from the same IP and get blocked (in addition to backend resources needed to do a lot of scraping on different pages).

Data might be unavailable for some time (until some user scrap it for you). But surely we can provide a better user experience to the end users using this strategy.

Answer 4

I used the following code to get round this problem:

Handler handler = new Handler(Looper.getMainLooper());
try
{
    handler.post(
        new Runnable()
        {
            @Override
            public void run()
            {
                ProcessRequest(); // Where this method runs the code you're needing
            }
        }
    );
} catch (Exception e)
{
    e.printStackTrace();
}

Answer 5

A WebView cannot exist outside of an Activity or Fragment due to it being a UI. However, this means that an Activity is only needed to create the WebView, not handle all its requests.

If you create the invisible WebView in your main activity and have it accessible from a static context, you should be able to perform tasks in the view in the background from anywhere, since I believe all of WebView's IO is done asynchronously.

To take away the ick of that global access, you could always launch a Service with a reference to the WebView to do the work you need.

Answer 6

the solution was like this, but with Looper.getMainLooper() :

https://github.com/JonasCz/save-for-offline/blob/master/app/src/main/java/jonas/tool/saveForOffline/ScreenshotService.java

@Override
public void onCreate() {
    super.onCreate();
    //HandlerThread thread = new HandlerThread("ScreenshotService", Process.THREAD_PRIORITY_BACKGROUND);
    //thread.start();
    //mServiceHandler = new ServiceHandler(thread.getLooper()); // not working
    mServiceHandler = new ServiceHandler(Looper.getMainLooper()); // working
}

with help of @JonasCz : https://stackoverflow.com/a/28234761/466363