Looking for a working example of addTimedTextSource for adding subtitle to a video from an .srt file in Android 4.1

Question

I have been trying to use an .srt file for a timed text source (Only available in android 4.1+ http://developer.android.com/about/versions/android-4.1.html#Multimedia) . The first problem has to do with getting a file descriptor for the .srt file (in the assets folder, how else would you bundle it in your app?). The file gets compressed automatically so you won't be able to even see the file without changing compile settings or doing a custom build. The easiest solution was to rename the .srt file to .jpg so that it does not get compressed and the openFD method still works. I am now adding the TimedTextSource with:

_myMP.addTimedTextSource(getAssets().openFd("captions.jpg").getFileDescriptor(),   MediaPlayer.MEDIA_MIMETYPE_TEXT_SUBRIP);

Now the file loads correctly and using myMP.getTrackInfo() to get a list of tracks, can see that after adding the timed text source, the 6th track has type "3" which is timed text track type. I have used selectTrack to choose this track as said in the google documentation but after doing so no captions ever appear and on my TimedTextListener:

 _myMP.setOnTimedTextListener(new OnTimedTextListener(){
        @Override
        public void onTimedText(MediaPlayer mp, TimedText text) {
                if (text!=null)
                   Log.d("TimedText", text.getText());  
            }       
        });

Fires only once (I have like 20 timed text events in the file) but the text parameter is always null. I have done searches and cannot find a single working code example of using timeText and it does not appear in any sample projects, there is literally no documentation other than the api docs from google but as far as I can tell, NO one has posted a working example of it yet. I am testing this on a google Nexus updated to Android 4.2

Answer 1

I was able to get this to work and since it is still an open question I will include the complete solution here.

Although the idea of changing the file extension to prevent the compression is nice, but I prefer to copy the srt file from the resources to the app local directory on the device, but anyways for the sake of completeness here is a list of extensions that won't be compressed.

".jpg", ".jpeg", ".png", ".gif", ".wav", ".mp2", ".mp3", ".ogg", ".aac", ".mpg", ".mpeg", ".mid", ".midi", ".smf", ".jet", ".rtttl", ".imy", ".xmf", ".mp4", ".m4a", ".m4v", ".3gp", ".3gpp", ".3g2", ".3gpp2",".amr", ".awb", ".wma", ".wmv"

The solution steps are simple:

1. Create a MediaPlayer instance and prepare it by either calling MediaPlayer.create() or player.setDataSource() then player.prepare()

2. If the subtitle files does not already exists on the android device, copy it from the resource folder to the device

3. Call player.addTimedTextSource() with the first argument a String that contains the full path of the subtitle file on the device and MediaPlayer.MEDIA_MIMETYPE_TEXT_SUBRIP as the second argument

4. Select the TimedText track by calling player.selectTrack() and pass the index of timedTextType by searching the TrackInfo[] returned from player.getTrackInfo() (I find it usually 2)

5. Set up a listener with player.setOnTimedTextListener() and then start playing the media file player.start()

Here is the complete class:

To run this exact class you will need two files under your res/raw folder sub.srt and video.mp4 (or whatever extensions). Then define a TextView with the id txtDisplay. Finally your project/device/emulator must support API 16

public class MainActivity extends Activity implements OnTimedTextListener {
    private static final String TAG = "TimedTextTest";
    private TextView txtDisplay;
    private static Handler handler = new Handler();

    @Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    txtDisplay = (TextView) findViewById(R.id.txtDisplay);
    MediaPlayer player = MediaPlayer.create(this, R.raw.video);
    try {
        player.addTimedTextSource(getSubtitleFile(R.raw.sub),
                MediaPlayer.MEDIA_MIMETYPE_TEXT_SUBRIP);
        int textTrackIndex = findTrackIndexFor(
                TrackInfo.MEDIA_TRACK_TYPE_TIMEDTEXT, player.getTrackInfo());
        if (textTrackIndex >= 0) {
            player.selectTrack(textTrackIndex);
        } else {
            Log.w(TAG, "Cannot find text track!");
        }
        player.setOnTimedTextListener(this);
        player.start();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

private int findTrackIndexFor(int mediaTrackType, TrackInfo[] trackInfo) {
    int index = -1;
    for (int i = 0; i < trackInfo.length; i++) {
        if (trackInfo[i].getTrackType() == mediaTrackType) {
            return i;
        }
    }
    return index;
}

private String getSubtitleFile(int resId) {
    String fileName = getResources().getResourceEntryName(resId);
    File subtitleFile = getFileStreamPath(fileName);
    if (subtitleFile.exists()) {
        Log.d(TAG, "Subtitle already exists");
        return subtitleFile.getAbsolutePath();
    }
    Log.d(TAG, "Subtitle does not exists, copy it from res/raw");

    // Copy the file from the res/raw folder to your app folder on the
    // device
    InputStream inputStream = null;
    OutputStream outputStream = null;
    try {
        inputStream = getResources().openRawResource(resId);
        outputStream = new FileOutputStream(subtitleFile, false);
        copyFile(inputStream, outputStream);
        return subtitleFile.getAbsolutePath();
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        closeStreams(inputStream, outputStream);
    }
    return "";
}

private void copyFile(InputStream inputStream, OutputStream outputStream)
        throws IOException {
    final int BUFFER_SIZE = 1024;
    byte[] buffer = new byte[BUFFER_SIZE];
    int length = -1;
    while ((length = inputStream.read(buffer)) != -1) {
        outputStream.write(buffer, 0, length);
    }
}

// A handy method I use to close all the streams
private void closeStreams(Closeable... closeables) {
    if (closeables != null) {
        for (Closeable stream : closeables) {
            if (stream != null) {
                try {
                    stream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

@Override
public void onTimedText(final MediaPlayer mp, final TimedText text) {
    if (text != null) {
        handler.post(new Runnable() {
            @Override
            public void run() {
                int seconds = mp.getCurrentPosition() / 1000;

                txtDisplay.setText("[" + secondsToDuration(seconds) + "] "
                        + text.getText());
            }
        });
    }
}

// To display the seconds in the duration format 00:00:00
public String secondsToDuration(int seconds) {
    return String.format("%02d:%02d:%02d", seconds / 3600,
            (seconds % 3600) / 60, (seconds % 60), Locale.US);
}
}

And here is the subtitle file I am using as example:

1
00:00:00,220 --> 00:00:01,215
First Text Example

2
00:00:03,148 --> 00:00:05,053
Second Text Example

3
00:00:08,004 --> 00:00:09,884
Third Text Example

4
00:00:11,300 --> 00:00:12,900
Fourth Text Example

5
00:00:15,500 --> 00:00:16,700
Fifth Text Example

6
00:00:18,434 --> 00:00:20,434
Sixth Text Example

7
00:00:22,600 --> 00:00:23,700
Last Text Example

Here are few screenshots from the test app showing that the TextView is changing automatically (i.e. reading from the subtitle file) as the media file progresses

Edit:

Here is the code for an example project

Answer 2

EDIT: I should point out that in the last few years, post-KitKat versions of Android have become most of the app-using android device market share. The implementation below was an attempt to bridge compatibility with older devices. At this point, I suggest using the TimedText framework (which worked fine in KitKat), or newer alternatives released by android, as a custom solution might have significant maintenance cost.

---

I went through 2 days looking at android source trying to iron out all the bugs that this TimedText Framework was causing.

My recommendation is to skip their implementation altogether. It is incomplete and inconsistent. In earlier versions, A lot of the Text synchronization is done in the native mediaplayer, so it's prone to state errors.

My alternative is to use a Textview Subclass:

package ca.yourpackage.yourapp;

import android.content.Context;
import android.media.MediaPlayer;
import android.util.AttributeSet;
import android.util.Log;
import android.widget.TextView;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.LineNumberReader;
import java.util.Locale;
import java.util.Map;
import java.util.TreeMap;

/**
 * Created by MHDante on 2015-07-26.
 */
public class SubtitleView extends TextView implements Runnable{
    private static final String TAG = "SubtitleView";
    private static final boolean DEBUG = false;
    private static final int UPDATE_INTERVAL = 300;
    private MediaPlayer player;
    private TreeMap<Long, Line> track;

    public SubtitleView(Context context) {
        super(context);
    }


    public SubtitleView(Context context, AttributeSet attrs) {
        super(context, attrs);
    }
    public SubtitleView(Context context, AttributeSet attrs, int defStyleAttr) {
        super(context, attrs, defStyleAttr);
    }


    @Override
    public void run() {
        if (player !=null && track!= null){
            int seconds = player.getCurrentPosition() / 1000;
            setText((DEBUG?"[" + secondsToDuration(seconds) + "] ":"")
                    + getTimedText(player.getCurrentPosition()));
        }
        postDelayed(this, UPDATE_INTERVAL);
    }

    private String getTimedText(long currentPosition) {
        String result = "";
        for(Map.Entry<Long, Line> entry: track.entrySet()){
            if (currentPosition < entry.getKey()) break;
            if (currentPosition < entry.getValue().to) result = entry.getValue().text;
        }
        return result;
    }

    // To display the seconds in the duration format 00:00:00
    public String secondsToDuration(int seconds) {
        return String.format("%02d:%02d:%02d", seconds / 3600,
                (seconds % 3600) / 60, (seconds % 60), Locale.US);
    }

    @Override
    protected void onAttachedToWindow() {
        super.onAttachedToWindow();
        postDelayed(this, 300);
    }

    @Override
    protected void onDetachedFromWindow() {
        super.onDetachedFromWindow();
        removeCallbacks(this);
    }
    public void setPlayer(MediaPlayer player) {
        this.player = player;
    }

    public void setSubSource(int ResID, String mime){
        if(mime.equals(MediaPlayer.MEDIA_MIMETYPE_TEXT_SUBRIP))
            track = getSubtitleFile(ResID);
        else
            throw new UnsupportedOperationException("Parser only built for SRT subs");
    }

    /////////////Utility Methods:
    //Based on https://github.com/sannies/mp4parser/
    //Apache 2.0 Licence at: https://github.com/sannies/mp4parser/blob/master/LICENSE

    public static TreeMap<Long, Line> parse(InputStream is) throws IOException {
        LineNumberReader r = new LineNumberReader(new InputStreamReader(is, "UTF-8"));
        TreeMap<Long, Line> track = new TreeMap<>();
        while ((r.readLine()) != null) /*Read cue number*/{
            String timeString = r.readLine();
            String lineString = "";
            String s;
            while (!((s = r.readLine()) == null || s.trim().equals(""))) {
                lineString += s + "\n";
            }
            long startTime = parse(timeString.split("-->")[0]);
            long endTime = parse(timeString.split("-->")[1]);
            track.put(startTime, new Line(startTime, endTime, lineString));
        }
        return track;
    }

    private static long parse(String in) {
        long hours = Long.parseLong(in.split(":")[0].trim());
        long minutes = Long.parseLong(in.split(":")[1].trim());
        long seconds = Long.parseLong(in.split(":")[2].split(",")[0].trim());
        long millies = Long.parseLong(in.split(":")[2].split(",")[1].trim());

        return hours * 60 * 60 * 1000 + minutes * 60 * 1000 + seconds * 1000 + millies;

    }

    private TreeMap<Long, Line> getSubtitleFile(int resId) {
        InputStream inputStream = null;
        try {
            inputStream = getResources().openRawResource(resId);
            return parse(inputStream);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (inputStream != null) {
                try {
                    inputStream.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
        return null;
    }

    public static class Line {
        long from;
        long to;
        String text;


        public Line(long from, long to, String text) {
            this.from = from;
            this.to = to;
            this.text = text;
        }
    }
}

Usage:

//I used and reccomend asyncPrepare()
MediaPlayer mp = MediaPlayer.create(context, R.raw.video);
SubtitleView subView = (SubtitleView) getViewbyId(R.id.subs_box);
subView.setPlayer(mp);
subView.setSubSource(R.raw.subs_intro, MediaPlayer.MEDIA_MIMETYPE_TEXT_SUBRIP);

In your layout xml file, just create a textView as you would want the subtitles to show, then change the class to ca.yourpagckage.yourapp.SubtitleView

<ca.yourpagckage.yourapp.SubtitleView
    android:layout_width="300dp"
    android:layout_height="300dp"
    android:text="Subtitles go Here"
    android:id="@+id/subs_box"/>

Good Luck.

Answer 3

To get it to work with .mp3 files, call player.start(); immediately after declaring a new mediaplayer and before the addtimedtext code. Right after the line below

MediaPlayer player = MediaPlayer.create(this, R.raw.video);