Generically read any file format and convert it to .txt format

Question

I need to make sure that the file given by the user is converted to the .txt file if contains text, before any further processing.

At the moment I have a switch stat

Answer 1

Following L.B's advice I am going to reincarnate this.

Using the Tika Java Library In Your .Net Application With IKVM

This may sound scary and heretical but did you know it is possible to leverage Java libraries from .Net applications with no TCP sockets or web services getting caught in the crossfire? Let me introduce you to IKVM, which is frankly magic:

IKVM.NET is an implementation of Java for Mono and the Microsoft .NET Framework. It includes the following components:

• A Java Virtual Machine implemented in .NET
• A .NET implementation of the Java class libraries
• Tools that enable Java and .NET interoperability

Using IKVM we have been able to successfully integrate our Dovetail Seeker search application with the Tika text extraction library implemented in Java. With Tika we can easily pull text out of rich documents from many supported formats. Why Tika? Because there is nothing comparable in the .Net world as Tika.

This post will review how we integrated with Tika. If you like code you can find this example in a repo up on Github.

Compiling a Jar Into An Assembly

First thing, we need to get our hands on the latest version of Tika. I downloaded and built the Tika source using Maven as instructed. The result of this was a few jar files. The one we are interested in is tika-app-x.x.jar which has everything we need bundled into one useful container.

Next up we need to convert this jar we’ve built to a .Net assembly. Do this using ikvmc.exe.

tika\build>ikvmc.exe -target:library tika-app-0.7.jar

Unfortunately, you will see tons of troublesome looking warnings but the end result is a .Net assembly wrapping the Java jar which you can reference in your projects.

Using Tika From .Net

IKVM is pretty transparent. You simply reference the the Tika app assembly and your .Net code is talking to Java types. It is a bit weird at first as you have Java versions of types and .Net versions. Next you’ll want to make sure that all the dependent IKVM runtime assemblies are included with your project. Using Reflector I found that the Tika app assembly referenced a lot of IKVM assemblies which do not appear to be used. I had to figure out through trial and error which assemblies where not being touched by the rich document extractions being done. If need be you could simple include all of the referenced IKVM assemblies with your application. Below I have done the work for you and eliminated all references to all the IKVM assemblies which appear to be in play.

16 assemblies down to 5. A much smaller deployment.

Using Tika

To do some text extraction we’ll ask Tika, very nicely, to parse the files we throw at it. For my purposes this involved having Tika automatically determine how to parse the stream and extract the text and metadata about the document.

public TextExtractionResult Extract(string filePath)
{
var parser = new AutoDetectParser();
var metadata = new Metadata();
var parseContext = new ParseContext();
java.lang.Class parserClass = parser.GetType();
parseContext.set(parserClass, parser);

try
{
var file = new File(filePath);
var url = file.toURI().toURL();
using (var inputStream = MetadataHelper.getInputStream(url, metadata))
{
parser.parse(inputStream, getTransformerHandler(), metadata, parseContext);
inputStream.close();
}

return assembleExtractionResult(_outputWriter.toString(), metadata);
}
catch (Exception ex)
{
throw new ApplicationException("Extraction of text from the file '{0}' failed.".ToFormat(filePath), ex);
}
}

One Important Cavet

Java has a concept called a ClassLoader which has something to do with how Java types are found and loaded. There is probably a better way around this but for some reason if you do not implement a custom ClassLoader and also set an application setting cueing the IKVM runtime about which .Net type to use as the ClassLoader.

public class MySystemClassLoader : ClassLoader
{
public MySystemClassLoader(ClassLoader parent)
: base(new AppDomainAssemblyClassLoader(typeof(MySystemClassLoader).Assembly))
{
}
}

Here is an example app.config telling IKVM where the ClassLoader is found.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key="ikvm:java.system.class.loader" value="TikaOnDotNet.MySystemClassLoader, TikaOnDotNet" />
</appSettings>
</configuration>

This step is very important. If IKVM cannot find a class loader, for some horrible reason, Tika will work fine but extract only empty documents with no metadata. The main reason this is troubling is that no exception is raised. For this reason we actually have a validation step in our application ensuring that the app setting is present and that it resolves to a valid type.

Demo

Here is a test demonstrating an extraction and the result.

[Test]
public void should_extract_from_pdf()
{
var textExtractionResult = new TextExtractor().Extract("Tika.pdf");

textExtractionResult.Text.ShouldContain("pack of pickled almonds");

Console.WriteLine(textExtractionResult);
}

Put simply rich documents like this go in.

And a TextExtractionResult comes out:

public class TextExtractionResult
{
public string Text { get; set; }
public string ContentType { get; set; }
public IDictionary<string, string> Metadata { get; set; }
//toString() override
}

Here is the raw output from Tika:

Conclusion

I hope this helps boost your confidence that you can use Java libraries in your .Net code and I hope my example repo will be of assistance if you need to do some work with Tika on the .Net platform. Enjoy.

---

Info to set this up:

Use Nuget to look up TikaOnDotnet and install both TikaOnDotnet & TikaOnDotnet.TextExtractor to your project. Here's the code to test it out on a Winform App:

public partial class Form1 : Form
{
    private System.Windows.Forms.TextBox textBox1;
    private TextExtractor _textExtractor;
    public Form1()
    {
        InitializeComponent();
        _textExtractor = new TextExtractor();

        textBox1 = new System.Windows.Forms.TextBox();
        textBox1.Dock = System.Windows.Forms.DockStyle.Fill;
        textBox1.Multiline = true;
        textBox1.Name = "textBox1";
        textBox1.ScrollBars = System.Windows.Forms.ScrollBars.Vertical;
        textBox1.AllowDrop = true;
        textBox1.DragDrop += new System.Windows.Forms.DragEventHandler(this.textBox1_DragDrop);
        textBox1.DragOver += new System.Windows.Forms.DragEventHandler(this.textBox1_DragOver);
        Controls.Add(this.textBox1);
        Name = "Drag/Drop any file on to the TextBox";
        ClientSize = new System.Drawing.Size(867, 523);
    }

    private void textBox1_DragOver(object sender, DragEventArgs e)
    {
        if (e.Data.GetDataPresent(DataFormats.FileDrop))
            e.Effect = DragDropEffects.Copy;
        else
            e.Effect = DragDropEffects.None;
    }

    private void textBox1_DragDrop(object sender, DragEventArgs e)
    {
        string[] files = (string[])e.Data.GetData(DataFormats.FileDrop);
        if (files != null && files.Length != 0)
        {
            TextExtractionResult textExtractionResult = _textExtractor.Extract(files[0]);
            textBox1.Text = textExtractionResult.Text;
        }
    }
}

Edit: The original blog page has moved to but there is no 302 perm redirect http://clarify.dovetailsoftware.com/kmiller/2010/07/02/using-the-tika-java-library-in-your-net-application-with-ikvm/