Split a byte array at a delimiter

Question

I\'m having a bit of an issue and, the other questions here didn\'t help me much.

I am a security student and I am trying to write a crypter for a project. For those

Answer 1

Your approach has a number of flaws - you're reading an entire Byte[] into memory, but decryption is a streamable process, so you're needlessly wasting memory. Secondly you cannot "split" an array (or a string, for that matter) in the CLR. When you split a CLR String it creates copies, which wastes memory.

Try this:

public static void Main(String[] args) {

    using(FileStream fs = new FileStream( @"path\to\fileName.exe", FileMode.Read)) {

        BinaryReader rdr = new BinaryReader( fs );
        SeekToEndOfDelimiter( rdr );

        // Use an implementation of RC4 decryption that accepts Streams as arguments, then pass fs directly as an argument:
        using(FileStream output = new FileStream( @"path\to\out.exe", FileMode.Write)) {
            // Providing the key arguments is an exercise for the reader
            MyRc4Implementation.DecryptStream( fs, output, key );
        }
    }

}

private static void SeekToEndOfDelimiter(BinaryReader rdr) {
    // Implementing this code is an exercise left up to the reader.
    // But just iterate through each byte (assuming ASCII-compatible encoding) until you encounter the end of the delimiter
}

There, no messy byte[] arrays :)

Answer 2

I know I'm really, really late to the party, but... You can of course modify this to return a List easily if preferred. I left comments/writelines in case it would be helpful... This may not be the most optimal/optimized code but works well for my specific use case and I thought I would share.

    public static byte[][] SplitBytesByDelimiter(byte[] data, byte delimiter)
    {
        if (data == null) throw new ArgumentNullException(nameof(data));
        if (data.Length < 1) return null;

        List<byte[]> retList = new List<byte[]>();

        int start = 0;
        int pos = 0;
        byte[] remainder = null; // in case data found at end without terminating delimiter

        while (true)
        {
            // Console.WriteLine("pos " + pos + " start " + start);
            if (pos >= data.Length) break;

            if (data[pos] == delimiter)
            {
                // Console.WriteLine("delimiter found at pos " + pos + " start " + start);

                // separator found
                if (pos == start)
                {
                    // Console.WriteLine("first char is delimiter, skipping");
                    // skip if first character is delimiter
                    pos++;
                    start++;
                    if (pos >= data.Length)
                    {
                        // last character is a delimiter, yay!
                        remainder = null;
                        break;
                    }
                    else
                    {
                        // remainder exists
                        remainder = new byte[data.Length - start];
                        Buffer.BlockCopy(data, start, remainder, 0, (data.Length - start));
                        continue;
                    }
                }
                else
                {
                    // Console.WriteLine("creating new byte[] at pos " + pos + " start " + start);
                    byte[] ba = new byte[(pos - start)];
                    Buffer.BlockCopy(data, start, ba, 0, (pos - start));
                    retList.Add(ba);

                    start = pos + 1;
                    pos = start;

                    if (pos >= data.Length)
                    {
                        // last character is a delimiter, yay!
                        remainder = null;
                        break;
                    }
                    else
                    {
                        // remainder exists
                        remainder = new byte[data.Length - start];
                        Buffer.BlockCopy(data, start, remainder, 0, (data.Length - start));
                    }
                }
            }
            else
            {
                // payload character, continue;
                pos++;
            }
        }

        if (remainder != null)
        {
            // Console.WriteLine("adding remainder");
            retList.Add(remainder);
        }

        return retList.ToArray();
    }

Answer 3

byte[] SeparateAndGetLast(byte[] source, byte[] separator)
{
  for (var i = 0; i < source.Length; ++i)
  {
     if(Equals(source, separator, i))
     {
       var index = i + separator.Length;
       var part = new byte[source.Length - index];
       Array.Copy(source, index, part, 0, part.Length);
       return part;
     }
  }
  throw new Exception("not found");
}

public static byte[][] Separate(byte[] source, byte[] separator)
{
    var Parts = new List<byte[]>();
    var Index = 0;
    byte[] Part;
    for (var I = 0; I < source.Length; ++I)
    {
        if (Equals(source, separator, I))
        {
            Part = new byte[I - Index];
            Array.Copy(source, Index, Part, 0, Part.Length);
            Parts.Add(Part);
            Index = I + separator.Length;
            I += separator.Length - 1;
        }
    }
    Part = new byte[source.Length - Index];
    Array.Copy(source, Index, Part, 0, Part.Length);
    Parts.Add(Part);
    return Parts.ToArray();
}

bool Equals(byte[] source, byte[] separator, int index)
{
  for (int i = 0; i < separator.Length; ++i)
    if (index + i >= source.Length || source[index + i] != separator[i])
      return false;
  return true;
}

Answer 4

Here's mine. It only does the split once though. I made no attempt at making it performant.

public static byte[][] Split(this byte[] composite, byte[] seperator)
{
    bool found = false;

    int i = 0;
    for (; i < composite.Length - seperator.Length; i++)
    {
        var compositeView = new byte[seperator.Length];
        Array.Copy(composite, i, compositeView, 0, seperator.Length);

            found = compositeView.SequenceEqual(seperator);
        if (found) break;
    }

    if(found == false)
    {
        return null;
    }

    var component1Length = i;
    var component1 = new byte[component1Length];

    var component2Length = composite.Length - seperator.Length - component1Length;
    var component2 = new byte[component2Length];
    var component2Index = i + seperator.Length;

    Array.Copy(composite, 0, component1, 0, component1Length);
    Array.Copy(composite, component2Index, component2, 0, component2Length);

    return new byte[][]
    {
        component1,
        component2
    };
}

tested (partially):

byte[] b1 = new byte[] { 1, 2, 3, 4, 1, 1, 5 };
byte[] b2 = new byte[] { 1, 1 };
var parts1 = b1.Split(b2); // [1,2,3,4],[5]

byte[] b3 = new byte[] { 1, 1, 3, 4, 4, 1, 5 };
byte[] b4 = new byte[] { 1, 1 };
var parts2 = b3.Split(b4); // [],[3,4,4,1,5]

byte[] b5 = new byte[] { 0, 0, 3, 4, 4, 1, 1 };
byte[] b6 = new byte[] { 1, 1 };
var parts3 = b5.Split(b6); // [0,0,3,4,4],[]

byte[] b7 = new byte[] { 1, 2, 3, 4, 5 };
byte[] b8 = new byte[] { 1, 2, 3, 4 };
var parts4 = b7.Split(b8); // [],[5]

byte[] b9 = new byte[] { 1, 2, 3, 4, 5 };
byte[] b0 = new byte[] { 2, 3, 4, 5 };
var parts5 = b9.Split(b0); // [1],[]

byte[] c1 = new byte[] { 1, 2, 3, 4, 5 };
byte[] c2 = new byte[] { 6 };
var parts6 = c1.Split(c2); // null

Answer 5

Here is Generic version

    public static IList<ArraySegment<T>> Split<T>(this T[] arr, params T[] delimiter)
    {

        var result = new List<ArraySegment<T>>();
        var segStart = 0;
        for (int i = 0, j = 0; i < arr.Length; i++)
        {
            //If is match
            if (arr.Skip(i).Take(delimiter.Length).SequenceEqual(delimiter))
            {

                //Skip first empty segment
                if (i > 0)
                {
                    result.Add(new ArraySegment<T>(arr, segStart, i - segStart));
                }

                //Reset
                segStart = i;
            }      
        }

        //Add last item
        if (segStart < arr.Length)
        {
            result.Add(new ArraySegment<T>(arr, segStart, arr.Length - segStart));
        }

        return result;
    }

Answer 6

For people who want to use the bytes in-place, instead of copying them to new arrays, you can use ArraySegment for this purpose.

Here's an implementation:

private static List<ArraySegment<byte>> Split(byte[] arr, byte[] delimiter)
{
    var result = new List<ArraySegment<byte>>();
    var segStart = 0;
    for (int i = 0, j = 0; i < arr.Length; i++)
    {
        if (arr[i] != delimiter[j]) continue;
        if (j++ != delimiter.Length - 1) continue;
        var segLen = i - segStart - (delimiter.Length - 1);
        if (segLen > 0) result.Add(new ArraySegment<byte>(arr, segStart, segLen));
        segStart = i + 1;
        j = 0;
    }

    if (segStart < arr.Length) result.Add(new ArraySegment<byte>(arr, segStart, arr.Length - segStart));
    return result;
}