How to read EBCDIC data with a non standard codepage, and not mess up numbers?

Question

Here is one for the old(er) hands :-)

I\'m reading a binary dump from a mainframe DB2 table. The table has varchar, char, smallint, integer and float columns. To m

Answer 1

Do not use a StreamReader to read this file. It is going to interpret the binary numbers in the file as though they are characters and that will mess up their value. Use a FileStream and a BinaryReader. Only use Encoding.GetString() when you are translating a group of bytes from the file that represents a string.

Answer 2

@Hans Passant is correct. If you are reading a file that contains binary data (as your discription indicates), then it is incorrect to read the file as though it were text.

Fortunately, the BinaryReader class includes a constructor that takes a character encoding as one of the parameters. You may use this to automatically convert any Hebrew EBCDIC strings in the file to ordinary Unicode strings without affecting the interpretation of the non-text (binary) portion.

Also, you should probably use the two-byte VARCHAR length field to read your strings instead of just throwing it away!

The ReadString() method will not work in this case, since the file was not encoded with the .NET BinaryWriter class. Instead you should get the length of the VARCHAR (or the hard-coded length of the CHAR field) and pass that to the ReadChars(int) method. Then construct your resulting string from the character array that is returned.

Answer 3

You can't read something like an EBCDIC file dump as a stream. The StreamReader class is a type of TextReader and exists for reading characters. You're reading a record -- a complex data structure containing mixed binary and text.

You need to do the reads with a FileStream and read blocks of octets as needed. You'll need some trivial helper methods like these:

private byte[] ReadOctets( Stream input , int size )
{
    if ( size < 0 ) throw new ArgumentOutOfRangeException() ;

    byte[] octets      = new byte[size] ;
    int    octets_read = input.Read( octets , 0 , size ) ;

    if ( octets_read != size ) throw new InvalidDataException() ;

    return octets ;
}

public string readCharVarying( Stream input )
{
    short    size        = readShort( input ) ;

    return readCharFixed( input , size ) ;
}

public string readCharFixed( Stream input , int size )
{
    Encoding e           = System.Text.Encoding.GetEncoding(20424) ;
    byte[]   octets      = ReadOctets( input , size ) ;
    string   value       = e.GetString( octets ) ;

    return value ;
}

private short readShort( Stream input )
{
    byte[] octets            = ReadOctets(input,2) ;
    short  bigEndianValue    = BitConverter.ToInt16(octets,0) ;
    short  littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

private int readInt( Stream input )
{
    byte[] octets            = ReadOctets(input,4) ;
    int    bigEndianValue    = BitConverter.ToInt32(octets,0) ;
    int    littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

private long readLong( Stream input )
{
    byte[] octets            = ReadOctets(input,8) ;
    long   bigEndianValue    = BitConverter.ToInt64(octets,0) ;
    long   littleEndianValue = System.Net.IPAddress.NetworkToHostOrder( bigEndianValue ) ;

    return littleEndianValue ;
}

The IBM mainframe typically has fixed or variable length records in its file system. Fixed length is easy: you just need to know the record length and you can read all the bytes for the record in a single call to the Read() method, then convert the pieces as needed.

Variable length records are a little trickier, they start with 4-octet record descriptor word, consisting of 2-octet (16-bit) logical record length, followed by a 2-octet (16-bit) 0 value. the logical record length is exclusive of the 4-octet record descriptor word.

You might also see variable, spanned records. These are similar to variable length records, except that the 4-octet prefix is a segment descriptor word. the first 2 octets contains the segment length, the next octet identifies the segment type and the last octet is NUL (0x00). Segment types are as follows:

• 0x00 indicates a complete logical record
• 0x01 indicates that this is the first segment of a spanned record
• 0x10 indicates that this is the last segment of a spanned record
• 0x11 indicates that this is an "internal" segment of a spanned record, that is, a "Segment of a multisegment record other than the first or last segment."

You can treat variable length and variable spanned records as identical. To read one of these, you first need to parse out the segment/record/descriptor word and read/assemble the complete record into a byte[] from its constituent segment(s), then do whatever needs to be done to convert that byte[] into a form that you can use.