criterion performance measurements
overview
want to understand this report?
map/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.664195987012732e-3 | 7.706460434539382e-3 | 7.758570367641186e-3 |
| Standard deviation | 1.0318115112864062e-4 | 1.339351774750734e-4 | 1.830565164538128e-4 |
Outlying measurements have slight (2.8546712802768284e-2%) effect on estimated standard deviation.
map/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1805570592262756e-2 | 1.1893213876041885e-2 | 1.2011640943675243e-2 |
| Standard deviation | 2.025703239541312e-4 | 2.687168537925407e-4 | 3.5628216439165066e-4 |
Outlying measurements have slight (3.443877551020426e-2%) effect on estimated standard deviation.
map/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6142454790076753e-2 | 2.625764152486135e-2 | 2.641988208858288e-2 |
| Standard deviation | 2.3211625260223087e-4 | 3.048639568917941e-4 | 4.1634668388289926e-4 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
map/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.0968538872750632e-2 | 3.1091531154319712e-2 | 3.127942794921602e-2 |
| Standard deviation | 2.1855825678125372e-4 | 3.0510016671977356e-4 | 3.7796113784534165e-4 |
Outlying measurements have slight (5.536332179930796e-2%) effect on estimated standard deviation.
map/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.588299752770539e-2 | 7.633607647494987e-2 | 7.682044832450491e-2 |
| Standard deviation | 5.672633849899718e-4 | 7.648311615267001e-4 | 1.0182291692452613e-3 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
drop/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2358268996602268e-2 | 1.244352113503839e-2 | 1.2564073045516847e-2 |
| Standard deviation | 1.8153846336343464e-4 | 2.518146100589175e-4 | 3.91315941342877e-4 |
Outlying measurements have slight (3.5665294924554364e-2%) effect on estimated standard deviation.
drop/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3407721667178666e-2 | 2.3682245607526842e-2 | 2.4023001141364353e-2 |
| Standard deviation | 4.7535253681354767e-4 | 7.002300788089312e-4 | 1.2170793094946034e-3 |
Outlying measurements have slight (9.106377144983405e-2%) effect on estimated standard deviation.
drop/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6559113272618456e-2 | 1.666116061302038e-2 | 1.6752475248407797e-2 |
| Standard deviation | 1.925198762632938e-4 | 2.4078289378039994e-4 | 3.310018822138518e-4 |
Outlying measurements have slight (4.158790170132303e-2%) effect on estimated standard deviation.
drop/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8892306659838532e-2 | 1.9195048389217945e-2 | 1.973699384324364e-2 |
| Standard deviation | 5.471897914969618e-4 | 9.689389775988682e-4 | 1.6008843186319933e-3 |
Outlying measurements have moderate (0.17260398392323495%) effect on estimated standard deviation.
drop/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.834886652212081e-2 | 5.8872605385994044e-2 | 5.921362490243664e-2 |
| Standard deviation | 5.414813866770924e-4 | 7.25930370168767e-4 | 9.932356446250535e-4 |
Outlying measurements have slight (7.638888888888856e-2%) effect on estimated standard deviation.
dropWhile/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.601900226948132e-3 | 9.6607051401978e-3 | 9.727854626326641e-3 |
| Standard deviation | 1.4966119801495135e-4 | 1.7795745564242168e-4 | 2.2707091673986415e-4 |
Outlying measurements have slight (3.1217481789802177e-2%) effect on estimated standard deviation.
dropWhile/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0502859724229643e-2 | 1.0652269072227718e-2 | 1.0801626933237702e-2 |
| Standard deviation | 3.505854482652131e-4 | 4.0575917773372314e-4 | 4.8032747658826924e-4 |
Outlying measurements have moderate (0.1308663370358982%) effect on estimated standard deviation.
dropWhile/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.093977242530554e-2 | 2.107911507638897e-2 | 2.124243827605706e-2 |
| Standard deviation | 2.5741933944281516e-4 | 3.545516433157348e-4 | 5.522646764565457e-4 |
Outlying measurements have slight (4.5351473922902376e-2%) effect on estimated standard deviation.
dropWhile/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3421407499148254e-2 | 1.3513919278223782e-2 | 1.3703763668841185e-2 |
| Standard deviation | 1.8278263870999962e-4 | 3.1926707483344877e-4 | 5.449744793447745e-4 |
Outlying measurements have slight (3.6982248520709776e-2%) effect on estimated standard deviation.
dropWhile/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.333359636547483e-2 | 4.3672679351900494e-2 | 4.405143834531719e-2 |
| Standard deviation | 4.8231960168906596e-4 | 6.83078614200647e-4 | 1.038848601536271e-3 |
Outlying measurements have slight (6.632653061224489e-2%) effect on estimated standard deviation.
scan/vector (my)
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6764770289800845e-2 | 1.6831854368541597e-2 | 1.689270398107368e-2 |
| Standard deviation | 1.2061611255635429e-4 | 1.5221525715928476e-4 | 1.926380056162513e-4 |
Outlying measurements have slight (4.158790170132282e-2%) effect on estimated standard deviation.
scan/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.98999892229787e-2 | 7.861255630525979e-2 | 8.393883857344726e-2 |
| Standard deviation | 5.590274015285743e-3 | 1.0764751203043096e-2 | 1.714591629591292e-2 |
Outlying measurements have moderate (0.4791677643906871%) effect on estimated standard deviation.
scan/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3210760651097586e-2 | 2.361555279420595e-2 | 2.3966983710510054e-2 |
| Standard deviation | 6.555201388168087e-4 | 8.31428932250436e-4 | 1.0442158206462908e-3 |
Outlying measurements have slight (9.791184039174387e-2%) effect on estimated standard deviation.
scan/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.788543430559119e-2 | 4.8232737793891305e-2 | 4.844513330358605e-2 |
| Standard deviation | 2.369034683847413e-4 | 4.958310947102829e-4 | 7.543580821062196e-4 |
Outlying measurements have slight (7.100591715976325e-2%) effect on estimated standard deviation.
scan/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.8336083469370244e-2 | 3.8646035101196276e-2 | 3.916269203083667e-2 |
| Standard deviation | 4.697902061901515e-4 | 8.057300354403599e-4 | 1.3456817522478006e-3 |
Outlying measurements have slight (6.222222222222222e-2%) effect on estimated standard deviation.
scan/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.262184653877795e-2 | 9.324148806909709e-2 | 9.400081130164523e-2 |
| Standard deviation | 8.371173463800308e-4 | 1.125776934451821e-3 | 1.3686862395863812e-3 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
take/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1215495138537371e-2 | 1.1265357471280456e-2 | 1.1324881005871111e-2 |
| Standard deviation | 1.180102552104323e-4 | 1.442509592575308e-4 | 1.9062079193155062e-4 |
Outlying measurements have slight (3.443877551020405e-2%) effect on estimated standard deviation.
take/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.044627651774144e-2 | 3.1075396430238076e-2 | 3.197858486533724e-2 |
| Standard deviation | 1.1580143795781538e-3 | 1.5512199245357294e-3 | 1.9300557488848961e-3 |
Outlying measurements have moderate (0.1649837004655392%) effect on estimated standard deviation.
take/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.666108926397772e-2 | 3.693536988823679e-2 | 3.7423047884390034e-2 |
| Standard deviation | 4.7717105204407443e-4 | 7.405565199074874e-4 | 1.1303000785427353e-3 |
Outlying measurements have slight (6.222222222222222e-2%) effect on estimated standard deviation.
take/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.6873419525797514e-2 | 3.748642606873969e-2 | 3.8023130859238904e-2 |
| Standard deviation | 8.752243764641169e-4 | 1.1354359077540498e-3 | 1.5801521396840996e-3 |
Outlying measurements have slight (6.222222222222219e-2%) effect on estimated standard deviation.
take/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.742662381575833e-2 | 9.820064846071357e-2 | 9.916132014733818e-2 |
| Standard deviation | 1.0704471594775971e-3 | 1.3955577842177933e-3 | 1.8541230867409103e-3 |
Outlying measurements have slight (9.876543209876533e-2%) effect on estimated standard deviation.
takeWhile/vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.090215146553627e-3 | 9.536205630168972e-3 | 1.0145188690084354e-2 |
| Standard deviation | 1.1649542494043838e-3 | 1.4427254702484996e-3 | 1.9312095417105722e-3 |
Outlying measurements have severe (0.7380001341511269%) effect on estimated standard deviation.
takeWhile/streaming
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1830960271072232e-2 | 1.1889437576366376e-2 | 1.1956260808141312e-2 |
| Standard deviation | 1.3939021318890526e-4 | 1.7161265583158625e-4 | 2.2784287700056727e-4 |
Outlying measurements have slight (3.443877551020408e-2%) effect on estimated standard deviation.
takeWhile/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3238518797867246e-2 | 3.351808233335489e-2 | 3.3822629987886564e-2 |
| Standard deviation | 4.7326810926462263e-4 | 6.199559448022187e-4 | 8.039966553381469e-4 |
Outlying measurements have slight (5.859375e-2%) effect on estimated standard deviation.
takeWhile/pipes
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.935775928617393e-2 | 2.997438932131526e-2 | 3.1111420525295086e-2 |
| Standard deviation | 1.1344435048965976e-3 | 1.741804015174237e-3 | 2.6077885418342417e-3 |
Outlying measurements have moderate (0.22022473858338446%) effect on estimated standard deviation.
takeWhile/machines
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.354108240818006e-2 | 8.410305575024146e-2 | 8.434659771805639e-2 |
| Standard deviation | 1.5046363302281552e-4 | 6.105997453518562e-4 | 1.0005914605601281e-3 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.