How to shuffle a “DenseMatrix” in F#

Question

In Matlab we can write the following code to shuffle a matrix:

data = data(:, randperm(size(data,2)));

there is what I write with Math.NET:

let csvfile = @"../UFLDL-tutorial-F#/housing.csv"
let housingAsLines = 
    File.ReadAllLines(csvfile)
        |> Array.map (fun t -> t.Split(',')
                            |> Array.map (fun t -> float t))
let housingAsMatrix= DenseMatrix.OfRowArrays housingAsLines
let housingAsMatrixT = housingAsMatrix.Transpose()

let v1 = DenseVector.Create(housingAsMatrixT.ColumnCount,1.0)
housingAsMatrixT.InsertRow(0,v1)

// How to shuffle a "DenseMatrix" in F#

To simulate matrix operation in Matlab, using the F# slice syntax and zero-based indexing. However, it doesn't work.

housingAsMatrixT.[*,0]

And I got the error message in vscode.

The field, constructor or member 'GetSlice' is not defined

Answer 1

You actually have two questions, 1) how to slice Matrices ala matlab and 2) how to shuffle the columns of a matrix.

For 1) actually Issue 277 you linked in the comment does indeed provide the solution. However you might be using an old version or you might not be referencing the F# extensions correctly:

#r @"..\packages\MathNet.Numerics.3.13.1\lib\net40\MathNet.Numerics.dll"
#r @"..\packages\MathNet.Numerics.FSharp.3.13.1\lib\net40\MathNet.Numerics.FSharp.dll"

open MathNet.Numerics
open MathNet.Numerics.LinearAlgebra
open MathNet.Numerics.Distributions
open System

//let m = DenseMatrix.randomStandard<float> 5 5
let m = DenseMatrix.random<float> 5 5 (ContinuousUniform(0., 1.))
let m' = m.[*,0]
m'
//val it : Vector<float> =
//seq [0.4710989485; 0.2220238937; 0.566367266; 0.2356496324; ...]

This extracts the first column of the matrix.

Now for 2), assuming you need to shuffle the matrix or the arrays containing a matrix you can use some of the approaches below. There might be a more elegant method within mathnet.numerics.

To permute the vector above: m'.SelectPermutation() or SelectPermutationInplace for arrays. There are other convenience function like .Column(idx),.EnumerateColumnsIndexed() or EnumerateColumns(), etc.

So m'.SelectPermutation() will shuffle the elements of m'. Or to shuffle the columns (which your matlab function does):

let idx = Combinatorics.GeneratePermutation 5
idx
//val it : int [] = [|2; 0; 1; 4; 3|]
let m2 = idx |> Seq.map (fun i -> m.Column(i)) |> DenseMatrix.ofColumnSeq
m2.Column(1) = m.Column(0)
//val it : bool = true

Since the first column of the original matrix moved to the second column of the new matrix, the two should be equal.

Answer 2

With Neural Networks I had to shuffle an array of Matrices and used the following code. Note the base data structure is an array ([]) and each item in the array is a Matrix. This is not shuffling a matrix, but an array. It should give you some idea of how to proceed for your problem.

type Random() = 
    static member Shuffle (a : 'a[]) =
        let rand = new System.Random()
        let swap (a: _[]) x y =
            let tmp = a.[x]
            a.[x] <- a.[y]
            a.[y] <- tmp
        Array.iteri (fun i _ -> swap a i (rand.Next(i, Array.length a))) a

and called it like

Random.Shuffle trainingData

Addendum

Here is the code to convert a byte[] to a DenseMatrix of double

let byteArrayToMatrix (bytes : byte[]) : Matrix<double> =
    let (x : Vector<byte>) = Vector<byte>.Build.DenseOfArray bytes
    let (y : Vector<double>) = x.Map(fun x -> double x)
    let (z : Matrix<double>) = Matrix<double>.Build.DenseOfRowVectors y
    z

Answer 3

@GuyCoder and @s952163, thanks for your help. I implemented a quick-and-dirty version. It is not good enough but it works.

Please feel free to comment. Thank you.

#load "../packages/FsLab.1.0.2/FsLab.fsx"
open System
open System.IO
open MathNet.Numerics.LinearAlgebra.Double
open MathNet.Numerics
open MathNet.Numerics.LinearAlgebra
open MathNet.Numerics.Distributions

// implementation of the Fisher-Yates shuffle by Mathias
// http://www.clear-lines.com/blog/post/Optimizing-some-old-F-code.aspx
let swap fst snd i =
   if i = fst then snd else
   if i = snd then fst else
   i
let shuffle items (rng: Random) =
   let rec shuffleTo items upTo =
      match upTo with
      | 0 -> items
      | _ ->
         let fst = rng.Next(upTo)
         let shuffled = List.permute (swap fst (upTo - 1)) items
         shuffleTo shuffled (upTo - 1)
   let length = List.length items
   shuffleTo items length

let csvfile = @"/eUSB/sync/fsharp/UFLDL-tutorial-F#/housing.csv"
let housingAsLines = 
    File.ReadAllLines(csvfile)
        |> Array.map (fun t -> t.Split(',')
                            |> Array.map (fun t -> float t))
let housingAsMatrix= DenseMatrix.OfRowArrays housingAsLines
let housingAsMatrixTmp = housingAsMatrix.Transpose()
let v1 = DenseVector.Create(housingAsMatrixTmp.ColumnCount,1.0)
let housingAsMatrixT = housingAsMatrixTmp.InsertRow(0,v1)

let m = housingAsMatrixT.RowCount - 1
let listOfArray = [0..m]
let random = new Random()
let shuffled = shuffle listOfArray random

let z = [for i in shuffled -> (housingAsMatrixT.[i, *])]
let final = DenseMatrix.OfRowVectors z