Supported Dense Matrix Microbenchmarks

The dense matrix microbenchmarks supported by the dense matrix benchmark are: Cholesky factorization, matrix cross product, matrix determinant, eigendecomposition, linear solve with multiple right hand sides, least squares fit, matrix deformation and transpose, matrix-matrix multiplication, matrix-vector multiplication, QR decomposition, and singular value decomposition.

Each microbenchmark is tested with several matrices of increasing size. The tested matrix dimensions are parameterized by \(N\) with values of \(N\) equal to: 1000, 2000, 4000, 8000, 10000, 15000, and 20000. The lsfit microbenchmark tests matrices of dimension \(2N\)-by-\(N/2\), while the remaining microbenchmarks all test \(N\)-by-\(N\) (square) matrices.

The table below lists the name of each microbenchmark supported along with a brief description of the functionality it tests. The kernel functions are applied to matrix or vector operands, where A and B are input matrices and x is an input vector generated by the corresponding allocator function.

For fast performance of the dense matrix kernels, it is crucial to link the R programming environment with optimized BLAS and LAPACK libraries. It is also important to have substantial amounts of memory (16GB minimum) to run most of the microbenchmarks.

The function GetDenseMatrixDefaultMicrobenchmarks defines the default microbenchmarks referenced in the table. It returns a vector of DenseMatrixMicrobenchmark objects specifying each microbenchmark. Microbenchmark parameters that can be specified include the dimensions of the matrices to be performance tested with, number of performance trials per matrix, and the allocator and microbenchmarking functions to be used. By default, the benchmark function RunDenseMatrixBenchmark calls GetDenseMatrixDefaultMicrobenchmarks to specify the microbenchmarks to be executed. See the object documentation for GetDenseMatrixDefaultMicrobenchmarks for more details.

The allocator functions for the dense matrix microbenchmarks take a DenseMatrixMicrobenchmark object specifying the microbenchmark and an integer index indicating which of the above values of \(N\) is to be applied. The allocator must return a list of allocated data objects, including the matrix, for the microbenchmark to operate on. The microbenchmarking functions take the DenseMatrixMicrobenchmark object defining the microbenchmark and the list of data objects returned by the allocator function.

Supported Sparse Matrix Microbenchmarks

The sparse matrix microbenchmarks supported by the sparse matrix benchmark are: matrix-vector multiplication, Cholesky factorization, LU factorization, and QR factorization. Each sparse matrix kernel tested is from the Matrix library included with the R distribution, and each kernel is performance tested with two or three sparse matrices from different application domains. The microbenchmarks, their associated identifiers and brief descriptions of the tested matrices are given in the table below. The table below lists each sparse matrix tested in the sparse matrix benchmark along with some fundamental characteristics of each matrix. The Laplacian matrices were generated for use with the benchmark, the rest of the matrices come from the University of Florida Sparse Matrix Collection. More detailed descriptions of the matrices from the Sparse Matrix Collection can be found on the web site. The R data files containing the sparse matrices can be downloaded in the companion package RHPCBenchmarkData.

The following table lists the microbenchmarks supported by the sparse matrix benchmark, the matrices each microbenchmark is executed with, and the kernel function that is microbenchmarked. The kernel functions are applied to matrix or vector operands, where A and B are input matrices and x is an input vector generated by the corresponding allocator function.

The functions GetSparseMatrixVectorDefaultMicrobenchmarks, GetSparseCholeskyDefaultMicrobenchmarks, GetSparseLuDefaultMicrobenchmarks, and GetSparseQrDefaultMicrobenchmarks define the default microbenchmarks referenced in the table. Each returns a vector of SparseMatrixMicrobenchmark objects specifying each microbenchmark. Microbenchmark parameters that can be specified include the dimensions of the matrices to be performance tested with, the number of performance trials to be conducted per matrix, and the allocator and microbenchmarking functions to be used. By default, the benchmark function RunSparseMatrixBenchmark calls each of the GetSparse* functions to specify the microbenchmarks to be executed. See the object documentation for RunSparseMatrixBenchmarks and the GetSparse* functions for more details.

The allocator functions for the sparse matrix microbenchmarks take a SparseMatrixMicrobenchmark object specifying the microbenchmark and an integer index indicating which matrix is being tested. The integer index is unused by the microbenchmarks specified by the GetSparse* default functions because the sparse matrix microbenchmarks read the test matrices from files as opposed to dynamically generating them. The allocator returns a list of data objects, including the matrix, for the microbenchmark to operate on. The microbenchmarking functions take the SparseMatrixMicrobenchmark object defining the microbenchmark and the list of data objects returned by the allocator function.

Supported Machine Learning Microbenchmarks

Currently, clustering microbenchmarks are the only microbenchmarks supported by the machine learning benchmark. The clustering benchmark generates clusters of normally distributed feature vectors in an \(N\)-dimensional, real-valued feature space where the mean of one cluster is located at the origin and the means of two clusters are each located at positions -1 and 1 of one or more of the axes. See the object documentation for the function GenerateClusterData for more details on how the clusters are generated. Microbenchmarking is performed with two cluster identification functions pam and clara, from the cluster package provided with the R distribution. The pam function implements the partitioning around medoids algorithm which has quadratic time complexity. The clara function is used for large data sets and operates in linear time complexity. The pam function is called with cluster.only=TRUE so that only assignments of feature vectors to clusters is reported. The clara function is called with samples=50, keep.data=FALSE, rngG=TRUE, and pamLike=TRUE. See the object documentation for the RunMachineLearningBenchmark, pam, and clara functions for more details.

The function GetClusteringDefaultMicrobenchmarks defines the default microbenchmarks referenced in the table. The function returns a vector of ClusteringMicrobenchmark objects specifying each microbenchmark. By default, the benchmark function RunMachineLearningBenchmark calls the GetClusteringDefaultMicrobenchmarks function to specify the microbenchmarks given in the table. See the object documentation for RunMachineLearningBenchmarks and the GetClusteringDefaultMicrobenchmarks functions for more details.

The allocator functions for the clustering microbenchmarks take a ClusteringMicrobenchmark object specifying the microbenchmark. The allocator must return a list of allocated data objects, including the matrix of feature vectors, for the microbenchmark to operate on. The microbenchmarking functions take the ClusteringMicrobenchmark object defining the microbenchmark and the list of data objects returned by the allocator function.

Dense Matrix Microbenchmark Examples

This example runs all of the default dense matrix microbenchmarks, saves the summary statistics for each microbenchmark in the directory DenseMatrixResults, and saves the data frame returned from the dense matrix benchmark to a file named allResultsFrame.RData.

# Create the directory where the CSV files and data frames will be saved.
resultsDirectory <- "./DenseMatrixResults"
dir.create(resultsDirectory)
# Set the run identifier to be appended to the names of the CSV files generated
# so that they can be identified with this run.
runIdentifier <- "all"
# Run all of the default microbenchmarks
allResults <- RunDenseMatrixBenchmark(runIdentifier, resultsDirectory)
save(allResults, file="./DenseMatrixResults/allResultsFrame.RData")

This example runs all but the matrix transpose microbenchmarks, which tend to run very slowly, and saves the results to the same directory as in the previous example.

# Get the default microbenchmark definition objects and deactivate
# the unwanted matrix transpose microbenchmarks.
myDenseMatrixMicrobenchmarks <- GetDenseMatrixDefaultMicrobenchmarks()
myDenseMatrixMicrobenchmarks[["deformtrans"]]$active <- FALSE
myDenseMatrixMicrobenchmarks[["transpose"]]$active <- FALSE
# Set an appropriate run identifier
runIdentifier <- "no_transpose"
exTransposeResults <- RunDenseMatrixBenchmark(runIdentifier, resultsDirectory, myDenseMatrixMicrobenchmarks)
save(exTransposeResults, file="./DenseMatrixResults/exTransposeResultsFrame.RData")

This example runs only the matrix-matrix microbenchmark, and it modifies the default matrix dimensions to test only a few small matrices.

# Get the default microbenchmark definition objects and select only
# the matrix-matrix multiplication microbenchmark to run.
myMatmatMicrobenchmark <- list()
myMatmatMicrobenchmark[["matmat"]] <- GetDenseMatrixDefaultMicrobenchmarks()[["matmat"]]
# Set the matrix dimensions to be tested to 2000 and 3000.  The
# number of trials and number of warmup trials per matrix must also be
# updated.
myMatmatMicrobenchmark[["matmat"]]$dimensionParameters <- as.integer(c(2000, 3000))
myMatmatMicrobenchmark[["matmat"]]$numberOfTrials <- as.integer(c(3, 3))
myMatmatMicrobenchmark[["matmat"]]$numberOfWarmupTrials <- as.integer(c(1, 1))
# Set an appropriate run identifier
runIdentifier <- "matmat"
matmatResults <- RunDenseMatrixBenchmark(runIdentifier, resultsDirectory, microbenchmarks=myMatmatMicrobenchmark)
save(matmatResults, file="./DenseMatrixResults/matmatResultsFrame.RData")

Sparse Matrix Microbenchmark Examples

This example runs all of the default sparse matrix microbenchmarks, saves the summary statistics for each microbenchmark in the directory SparseMatrixResults, and saves the data frame returned from the dense matrix benchmark to a file named allResultsFrame.RData.

# Create the directory where the CSV files and data frames will be saved.
resultsDirectory <- "./SparseMatrixResults"
dir.create(resultsDirectory)
# Set the run identifier to be appended to the names of the CSV files generated
# so that they can be identified with this run.
runIdentifier <- "all"
# Run all of the default microbenchmarks
allResults <- RunSparseMatrixBenchmark(runIdentifier, resultsDirectory)
save(allResults, file="./SparseMatrixResults/allResultsFrame.RData")

This example runs only the Cholesky factorization microbenchmarks and saves the results to the same directory as in the previous example.

# Set an appropriate run identifier
runIdentifier <- "cholesky_only"
# Run only the sparse Cholesky factorization microbenchmarks
choleskyResults <- RunSparseMatrixBenchmark(runIdentifier, resultsDirectory,
   matrixVectorMicrobenchmarks=NULL, luMicrobenchmarks=NULL, qrMicrobenchmarks=NULL)
save(choleskyResults, file="./SparseMatrixResults/choleskyResultsFrame.RData")

Machine Learning Microbenchmark Examples

This example runs all of the default machine learning microbenchmarks, saves the summary statistics for each microbenchmark in the directory MachineLearningResults, and saves the data frame returned from the dense matrix benchmark to a file named allResultsFrame.RData.

# Create the directory where the CSV files and data frames will be saved.
resultsDirectory <- "./MachineLearningResults"
dir.create(resultsDirectory)
# Set the run identifier to be appended to the names of the CSV files generated
# so that they can be identified with this run.
runIdentifier <- "all"
# Run all of the default microbenchmarks
allResults <- RunMachineLearningMatrixBenchmark(runIdentifier, resultsDirectory)
save(allResults, file="./MachineLearningResults/allResultsFrame.RData")

This example shows how to specify a new clustering microbenchmark and run it.

# Create a new clustering microbenchmark that tests the `clara` method from the
# `cluster` package using a data set with 250 features, 50 clusters, and
# 2000 normally distributed feature vectors per cluster.
claraMicrobenchmark <- list()
claraMicrobenchmark[["clara_cluster_250_50_2000"]] <- methods::new(
      "ClusteringMicrobenchmark",
      active = TRUE,
      benchmarkName = "clara_cluster_250_50_2000",
      benchmarkDescription = "Clustering of 100000 250-dimensional feature vectors into 50 clusters using clara function",
      numberOfFeatures = as.integer(250),
      numberOfClusters = as.integer(50),
      numberOfFeatureVectorsPerCluster = as.integer(2000),
      dataObjectName = NA_character_,
      numberOfTrials = as.integer(3),
      numberOfWarmupTrials = as.integer(1),
      allocatorFunction = ClusteringAllocator,
      benchmarkFunction = ClaraClusteringBenchmark
   )
# Set an appropriate run identifier
runIdentifier <- "clara"
# Run the clara microbenchmark
claraResults <- RunMachineLearningBenchmark(runIdentifier, resultsDirectory,
   clusteringMicrobenchmarks=claraMicrobenchmark)
save(claraResults, file="./MachineLearningResults/claraResultsFrame.RData")