it.grypho.scala.leonardo.statistics

Members list

Type members

Classlikes

enum Alternative(val word: String) extends Keyworded

The alternative hypothesis of a test (F_0130), with the slot word that names it. The words are matched only in a test's trailing slots and are not reserved (the c2d precedent).

The alternative hypothesis of a test (F_0130), with the slot word that names it. The words are matched only in a test's trailing slots and are not reserved (the c2d precedent).

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
trait Keyworded
class Object
trait Matchable
class Any
Show all
object PairStatKind

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
enum PairStatKind extends Keyworded

The two-sample statistics.

The two-sample statistics.

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
trait Keyworded
class Object
trait Matchable
class Any
Show all
enum RlsFailure

Why the numeric rlsUpdate declined (F_0308, AIACE L_0010).

Why the numeric rlsUpdate declined (F_0308, AIACE L_0010).

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
enum StatKind extends Keyworded

The single-sample statistics.

The single-sample statistics.

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
trait Keyworded
class Object
trait Matchable
class Any
Show all
object StatKind

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
StatKind.type
object TestKind

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
TestKind.type
enum TestKind extends Keyworded

The inference tests.

The inference tests.

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
trait Keyworded
class Object
trait Matchable
class Any
Show all
case class _Anova(groups: List[_Expression]) extends _Expression

anova(g1, g2, …) — one-way analysis of variance over two or more samples (F_0131), answering the p-value of F = MSB/MSW on (k − 1, N − k), the upper tail read directly. With two groups it is the pooled two-sample t-test (F = t²), which the tests pin. A single group, or groups with no spread inside them, has no test.

anova(g1, g2, …) — one-way analysis of variance over two or more samples (F_0131), answering the p-value of F = MSB/MSW on (k − 1, N − k), the upper tail read directly. With two groups it is the pooled two-sample t-test (F = t²), which the tests pin. A single group, or groups with no spread inside them, has no test.

Value parameters

groups

the samples, one per group

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _BinCounts(x: _Expression, lo: _Expression, hi: _Expression, k: _Expression) extends _Expression

bincounts(x, lo, hi, k) — binCounts at the prompt (F_0135): a k × 1 column of counts, the data behind a histogram. An exact sample is read as the numbers it holds. k is capped by the environment's Limit.MaxMatrixCells, the size of the column it builds.

bincounts(x, lo, hi, k) — binCounts at the prompt (F_0135): a k × 1 column of counts, the data behind a histogram. An exact sample is read as the numbers it holds. k is capped by the environment's Limit.MaxMatrixCells, the size of the column it builds.

Value parameters

hi

the upper edge

k

the number of bins

lo

the lower edge

x

the sample

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _ChiSqTable(table: _Expression) extends _Expression

chisqtest(table) — Pearson's test of independence (or homogeneity, the same computation) on an r × c table of observed counts (F_0130): expected counts from the margins, (r − 1)(c − 1) degrees of freedom. A single row or column, or a zero margin, has no test.

chisqtest(table) — Pearson's test of independence (or homogeneity, the same computation) on an r × c table of observed counts (F_0130): expected counts from the margins, (r − 1)(c − 1) degrees of freedom. A single row or column, or a zero margin, has no test.

Value parameters

table

the observed counts

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _PairStatistic(kind: PairStatKind, a: _Expression, b: _Expression) extends _Expression

A statistic of two equally long samples.

A statistic of two equally long samples.

Value parameters

a

the first sample

b

the second

kind

the statistic

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _PropTest(args: List[_Expression], alt: Alternative) extends _Expression

proptest(x, n, p0 [, alt]) or proptest(x1, n1, x2, n2 [, alt]) — the one- and two-proportion z-tests of Introductory Statistics 2e (F_0130): the normal approximation, the two-sample form on the pooled proportion (x₁ + x₂)/(n₁ + n₂). Counts must be whole and within their trial counts, p0 strictly inside (0, 1); otherwise the node stays unevaluated.

proptest(x, n, p0 [, alt]) or proptest(x1, n1, x2, n2 [, alt]) — the one- and two-proportion z-tests of Introductory Statistics 2e (F_0130): the normal approximation, the two-sample form on the pooled proportion (x₁ + x₂)/(n₁ + n₂). Counts must be whole and within their trial counts, p0 strictly inside (0, 1); otherwise the node stays unevaluated.

Value parameters

alt

the alternative hypothesis

args

x, n, p0 or x1, n1, x2, n2

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Quantile(x: _Expression, p: _Expression, method: Option[_Expression]) extends _Expression

quantile(x, p) or quantile(x, p, method) — dispatching on its first argument like _Statistic (F_0129): a distribution answers its inverse cdf, exactly as before, and a sample its order-statistic quantile by sampleQuantile, type 7 unless method is 6. A method given with a distribution is refused: there is nothing for it to choose between.

quantile(x, p) or quantile(x, p, method) — dispatching on its first argument like _Statistic (F_0129): a distribution answers its inverse cdf, exactly as before, and a sample its order-statistic quantile by sampleQuantile, type 7 unless method is 6. A method given with a distribution is refused: there is nothing for it to choose between.

Value parameters

method

6 or 7, when given

p

the probability

x

the sample or distribution

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Regress(x: _Expression, y: _Expression) extends _Expression

Ordinary least squares: regress(X, y).

Ordinary least squares: regress(X, y).

Returns the p × 1 coefficient vector. No intercept column is added: a constant term is a column of ones the caller supplies, because a silently inserted column would make regress compute something other than what was written, with no way to opt out.

Value parameters

x

the design matrix

y

the response vector

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _RidgeRegress(x: _Expression, y: _Expression, lambda: _Expression) extends _Expression

Ridge regression: regress(X, y, lambda), minimising ‖Xβ − y‖² + λ‖β‖² (F_0062).

Ridge regression: regress(X, y, lambda), minimising ‖Xβ − y‖² + λ‖β‖² (F_0062).

The three-argument spelling of regress, as a node of its own rather than a third field on _Regress: that is a published case class, and widening it would be a binary break for a spelling. Solved by ridgeLeastSquares (augmented QR). Stays unevaluated for a λ that is not a number ≥ 0, and wherever regress would.

Value parameters

lambda

the penalty

x

the design matrix

y

the response vector

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Rls(theta: _Expression, p: _Expression, phi: _Expression, y: _Expression, lambda: _Expression) extends _Expression

rls(theta, P, phi, y, lambda) — one update, as the row [[theta', P']] (the lu shape), so tuple assignment carries the state to the next call.

rls(theta, P, phi, y, lambda) — one update, as the row [[theta', P']] (the lu shape), so tuple assignment carries the state to the next call.

Value parameters

lambda

the forgetting factor

p

the covariance

phi

the regressor

theta

the estimate

y

the observation

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Sort(x: _Expression) extends _Expression

sort(x) — the entries of a sample in ascending order (F_0129): a row as a row, a column as a column, any other matrix as a column of all its entries. An exact sample stays exact, on the symbolic _Matrix carrier exact matrices use.

sort(x) — the entries of a sample in ascending order (F_0129): a row as a row, a column as a column, any other matrix as a column of all its entries. An exact sample stays exact, on the symbolic _Matrix carrier exact matrices use.

Value parameters

x

the sample

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Statistic(kind: StatKind, arg: _Expression) extends _Expression

A statistic of one argument, which may be a sample or a distribution.

A statistic of one argument, which may be a sample or a distribution.

Dispatching on the argument is what lets mean and variance mean one thing each. A distribution delegates to probability, which owns its own moments; a matrix is treated as a sample. This is why statistics imports probability and not the reverse.

Note the n/n − 1 distinction that follows from the dispatch: variance(sample) is the unbiased estimator, while variance(distribution) is the population variance, because a distribution is not a sample and has no n to correct for. See the package overview.

Value parameters

arg

the sample or distribution

kind

the statistic

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _TTest(a: _Expression, b: _Expression, pooled: Boolean, alt: Alternative) extends _Expression

ttest(x, m_or_y [, pooled] [, less | greater]) — a t-test with its options written (F_0130). A node of its own because the published _Test has two fields; ttest(a, b) with no option still builds _Test, which answers the two-sample test too.

ttest(x, m_or_y [, pooled] [, less | greater]) — a t-test with its options written (F_0130). A node of its own because the published _Test has two fields; ttest(a, b) with no option still builds _Test, which answers the two-sample test too.

pooled is a two-sample option: with a null mean in place of a second sample it has nothing to pool, and the call stays unevaluated rather than silently dropping a word the user wrote (F_0150) — ttest(x, 3) is the one-sample test.

Value parameters

a

the sample

alt

the alternative hypothesis

b

the null mean, or the second sample

pooled

whether two samples share a variance

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _Test(kind: TestKind, a: _Expression, b: _Expression) extends _Expression

An inference test.

An inference test.

Thin wrappers: the work was in the incomplete beta and gamma kernels and in the distribution carrier, both of which already existed. What this adds is the test statistics themselves and the routing.

Value parameters

a

the sample (or observed counts)

b

the null mean, the confidence level, or the expected counts

kind

the test

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all
case class _VarTest(a: _Expression, b: _Expression, alt: Alternative) extends _Expression

vartest(x, y[, less | greater]) — the F-test of two variances (F_0131): F = s₁²/s₂² on (n₁ − 1, n₂ − 1); greater is P(F ≥ f), less is P(F ≤ f), and the two-sided p-value is twice the smaller tail.

vartest(x, y[, less | greater]) — the F-test of two variances (F_0131): F = s₁²/s₂² on (n₁ − 1, n₂ − 1); greater is P(F ≥ f), less is P(F ≤ f), and the two-sided p-value is twice the smaller tail.

Value parameters

a

the first sample

alt

the alternative hypothesis

b

the second sample

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all

Weighted least squares: wregress(X, y, w), minimising Σ wᵢ(xᵢβ − yᵢ)² (F_0062).

Weighted least squares: wregress(X, y, w), minimising Σ wᵢ(xᵢβ − yᵢ)² (F_0062).

Solved by weightedLeastSquares (row scaling by √wᵢ, then QR). Stays unevaluated for a negative weight, a length mismatch, or a design the weights leave rank-deficient.

Value parameters

w

the weights, one per observation

x

the design matrix

y

the response vector

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
trait _Expression
class Object
trait Matchable
class Any
Show all

Value members

Concrete methods

def binCounts(xs: Vector[Double], lo: Double, hi: Double, k: Int): Option[Vector[Int]]

The counts of a sample in k equal bins over [lo, hi] (F_0135).

The counts of a sample in k equal bins over [lo, hi] (F_0135).

Each bin is half-open, [lo + i·w, lo + (i+1)·w), except the last, which is closed so that hi itself — typically the sample's maximum — is counted. Values outside [lo, hi] are left out. A value's bin is computed, not searched for, so the cost is one pass.

Value parameters

hi

the upper edge, above lo

k

the number of bins, at least 1

lo

the lower edge

xs

the sample

Attributes

Returns

the k counts, or None for lo ≥ hi, k < 1 or a non-finite edge

def fitted(x: _MatrixValue, beta: _MatrixValue): Option[_MatrixValue]

The fitted values X·β.

The fitted values X·β.

Value parameters

beta

the coefficients

x

the design matrix

Attributes

Fits y ≈ X·β by ordinary least squares.

Fits y ≈ X·β by ordinary least squares.

No intercept is added. A constant term is a column of ones in X, which the caller supplies — silently inserting one would make regress mean something different from what was written, and there is no way to opt out of a hidden column.

Value parameters

x

the design matrix, n × p, with n ≥ p

y

the response, n × 1 or 1 × n

Attributes

Returns

the p × 1 coefficient vector, or None when the shapes do not conform or the design is rank-deficient (qrDecompose refuses it, so nonsense is not produced)

def residualSumOfSquares(x: _MatrixValue, y: _MatrixValue, beta: _MatrixValue): Option[Double]

The residual sum of squares of a fit, which the inference tier needs.

The residual sum of squares of a fit, which the inference tier needs.

Value parameters

beta

the coefficients

x

the design matrix

y

the response, n × 1

Attributes

def ridgeLeastSquares(x: _MatrixValue, y: _MatrixValue, lambda: Double): Option[_MatrixValue]

Fits y ≈ X·β by ridge regression, minimising ‖Xβ − y‖² + λ‖β‖² (F_0062).

Fits y ≈ X·β by ridge regression, minimising ‖Xβ − y‖² + λ‖β‖² (F_0062).

Solved as the ordinary problem on the augmented design [X; √λ·I] against [y; 0], by the same QR — never as (XᵀX + λI)⁻¹Xᵀy, which squares the condition number exactly as the unregularised normal equations do, and for the same small λ that is meant to rescue a nearly collinear design. λ = 0 is leastSquares, called directly so the two agree bit for bit. For λ > 0 the augmented design has full column rank whatever X is, so a rank-deficient or wide X that ordinary regression declines is solved — which is what ridge is for. No intercept is added, and the intercept column is penalised like any other.

Value parameters

lambda

the penalty, ≥ 0

x

the design matrix, n × p

y

the response, n entries either way up

Attributes

Returns

the p × 1 coefficients, or None for a negative or non-finite λ, a response of the wrong length, or (at λ = 0) a rank-deficient design

One recursive-least-squares update.

One recursive-least-squares update.

Value parameters

env

the environment the operands evaluate in, and the exact reduction policy

lambda

the forgetting factor, in (0, 1]

p

the covariance, n×n

phi

the regressor, n×1

theta

the estimate, n×m (n×1 for one output)

y

the observation, m×1 (a number for one output)

Attributes

Returns

(theta', P') — exact when every input is, dense otherwise — or None when λ is out of range, a shape disagrees, an operand is not numeric, or λ + φᵀPφ is not positive

def sampleCorrelation(xs: Vector[_Value], ys: Vector[_Value], env: Environment): Option[_Value]

The Pearson correlation coefficient.

The Pearson correlation coefficient.

Scale-invariant, so the n − 1 cancels between the covariance and the two standard deviations — this is the one statistic here for which the population/sample choice makes no difference at all.

Value parameters

env

supplies the working precision

xs

the first sample

ys

the second, which must be the same length

Attributes

Returns

the correlation in [-1, 1], or None if either sample is constant

def sampleCovariance(xs: Vector[_Value], ys: Vector[_Value], env: Environment): Option[_Value]

The covariance of two equally long samples, dividing by n − 1.

The covariance of two equally long samples, dividing by n − 1.

Value parameters

env

supplies the reduction policy

xs

the first sample

ys

the second, which must be the same length

Attributes

Returns

the covariance — exact when both samples are

def sampleMean(xs: Vector[_Value], env: Environment): Option[_Value]

The arithmetic mean.

The arithmetic mean.

Value parameters

env

supplies the reduction policy for the exact path

xs

the sample, which must be non-empty

Attributes

Returns

the mean — exact when the sample is

def sampleOf(r: Either[_Expression, _Value]): Option[Vector[_Value]]

Reads an evaluated expression as a sample: every entry of a matrix, row-major.

Reads an evaluated expression as a sample: every entry of a matrix, row-major.

Accepts both carriers, which matters for exact input: an exactly-written matrix stays a symbolic _Matrix rather than collapsing to the dense one, and that is precisely the case where an exact statistic is worth having.

Value parameters

r

the evaluated argument

Attributes

Returns

the entries, or None when the argument is not a matrix of values

def sampleQuantile(xs: Vector[_Value], p: _Value, method: Int, env: Environment): Option[_Value]

The p-quantile of a sample by Hyndman and Fan's type 7 or type 6 (F_0129, Decision 3).

The p-quantile of a sample by Hyndman and Fan's type 7 or type 6 (F_0129, Decision 3).

Both interpolate linearly between neighbouring order statistics; they differ in where p falls. Type 7 (the default, R's and numpy's) puts it at the 0-based position (n − 1)p, so p = 0 and p = 1 are the extremes. Type 6 puts it at the 1-based position (n + 1)p, the percentile formula of Introductory Statistics 2e, clamped to the extremes below 1 and past n. The median is the same under both.

Value parameters

env

supplies the reduction policy for the exact path

method

7 or 6

p

the probability, in [0, 1]

xs

the sample, non-empty

Attributes

Returns

the quantile — exact when the sample and p are — or None for an empty sample, a p outside [0, 1] or another method

def sampleStdDev(xs: Vector[_Value], ddof: Int, env: Environment): Option[_Value]

The standard deviation — the square root of sampleVariance.

The standard deviation — the square root of sampleVariance.

A square root is not closed over the rationals, so an exact sample gives an exact variance but only a working-precision standard deviation. That is the same contract every irrational result in the library carries, and it goes through the same kernel.

Value parameters

ddof

1 for the sample standard deviation, 0 for the population one

env

supplies the working precision

xs

the sample

Attributes

def sampleVariance(xs: Vector[_Value], ddof: Int, env: Environment): Option[_Value]

The variance, dividing by n − ddof.

The variance, dividing by n − ddof.

Two-pass, deliberately. The textbook one-pass form E[X²] − E[X]² subtracts two nearly equal quantities whenever the sample has a large mean relative to its spread, and loses most of its significant digits doing so — the same catastrophic cancellation that already cost this project the quadratic formula's small root. Computing the mean first and then summing squared deviations has no such subtraction in it.

Value parameters

ddof

1 for the unbiased sample variance, 0 for the population variance

env

supplies the reduction policy for the exact path

xs

the sample

Attributes

Returns

the variance — exact when the sample is

def sortedSample(xs: Vector[_Value]): Option[Vector[_Value]]

The sample in ascending order (F_0129) — compared exactly when every entry is exact, else as Doubles. Selection is not arithmetic, so an exact sample stays exact.

The sample in ascending order (F_0129) — compared exactly when every entry is exact, else as Doubles. Selection is not arithmetic, so an exact sample stays exact.

Value parameters

xs

the sample

Attributes

Returns

the entries sorted, or None when an entry is not a real number

Fits y ≈ X·β by weighted least squares, minimising Σ wᵢ(xᵢβ − yᵢ)² (F_0062).

Fits y ≈ X·β by weighted least squares, minimising Σ wᵢ(xᵢβ − yᵢ)² (F_0062).

Row scaling by √wᵢ, then the same QR — never (XᵀWX)⁻¹XᵀWy, for the condition-number reason leastSquares states. Equal weights give the ordinary fit; a zero weight removes its row, so a design left rank-deficient by zeros is declined as an ordinary one would be.

Value parameters

w

the weights, n entries either way up, each ≥ 0 and finite

x

the design matrix, n × p

y

the response, n entries either way up

Attributes

Returns

the p × 1 coefficients, or None for a negative or non-finite weight, a length mismatch, or a rank-deficient weighted design