• Understand the difference between zero-truncated and zero-inflated data

• Understand how to model zero-truncated data

• Understand the differences between zero-altered and zero-inflated models

Zero-truncated data cannot take a value of 0

Although somewhat rare in ecological studies, examples include

• time a whale is at the surface before diving

• herd size in elk

• number of fin rays on a fish

Zero-truncated data are not necessarily a problem

Rather, an underlying assumption of Poisson or neg binomial may be the problem

Both of these examples contain zeros

Recall that for \(y_i \sim \text{Poisson}(\lambda)\)

its probability mass function is

\[ f(y_i; \lambda_i) = \frac{\exp (\text{-} \lambda_i) \lambda_{i}^{y_i}}{y_i!} \]

\(f(y_i; \lambda_i)\) gives the probability of \(y_i \geq 0\)

The probability that \(y_i = 0\) is

\[ f(y_i; \lambda_i) = \frac{\exp (\text{-} \lambda_i) \lambda_{i}^{y_i}}{y_i!} \\ \Downarrow \\ \begin{align} f(y_i = 0; \lambda_i) &= \frac{\exp (\text{-} \lambda_i) \lambda_{i}^0}{0!} \\ &= \exp (\text{-} \lambda_i) \end{align} \]

The probability that \(y_i \neq 0\) is therefore

\[ f(y_i = 0; \lambda_i) = \exp (\text{-} \lambda_i) \\ \Downarrow \\ f(y_i \neq 0; \lambda_i) = 1 - \exp (\text{-} \lambda_i) \]

We can now exclude the probability that \(y_i = 0\) by dividing the pmf by the probability that \(y_i \neq 0\)

\[ f(y_i; \lambda_i) = \frac{\exp (\text{-} \lambda_i) \lambda_{i}^{y_i}}{y_i!} \\ \Downarrow \\ f(y_i; \lambda_i | y_i > 0) = \frac{\exp (\text{-} \lambda_i) \lambda_{i}^{y_i}}{y_i!} \cdot \frac{1}{1 - \exp (\text{-} \lambda_i)} \\ \Downarrow \\ \log \mathcal{L} = \left( y_i \log \lambda_i - \lambda_i \right) - \left( 1 - \exp (\text{-} \lambda_i) \right) \]

Let’s consider some data presented in Zuur et al. (2009), which detail the number of days that carcasses from road-killed snakes stay on roads

The predictors are the total rainfall (mm) and an indicator of where on the pavement the snake was killed (lane or shoulder)

Let’s first consider a regular Poisson regression model

## Poisson regression
smod_pois <- glm(n_days ~ location + rain, data = snakes,
                 family = poisson(link = "log"))

Now let’s fit a zero-truncated Poisson regression model with vglm() from VGAM

library(VGAM)
## zero-truncated Poisson regression
smod_ztpois <- vglm(n_days ~ location + rain, data = snakes,
                    family = pospoisson)

Here are the parameter estimates and SE’s for both models

##               Poisson   Poisson SE   +Poisson   +Poisson SE
## (Intercept)     0.439        0.089      0.041         0.125
## locationV       0.463        0.119      0.711         0.149
## rain            0.022        0.003      0.027         0.003

Recall that for \(y_i \sim \text{negBinom}(r, \mu)\)

its probability mass function is

\[ f(y; \mu, r) = \frac{(y+r-1) !}{(r-1) ! y !} \left( \frac{r}{\mu + r} \right)^{r}\left( \frac{\mu}{\mu + r} \right)^{y} \]

\(f(y_i; \mu, r)\) gives the probability of \(y_i \geq 0\)

The probability that \(y_i = 0\) is

\[ f(y; r, \mu) = \frac{(y+r-1) !}{(r-1) ! y !} \left( \frac{r}{\mu + r} \right)^{r}\left( \frac{\mu}{\mu + r} \right)^{y} \\ \Downarrow \\ \begin{align} f(y_i = 0; r, \mu) &= \frac{(0+r-1) !}{(r-1) ! 0 !} \left( \frac{r}{\mu + r} \right)^{r} \left( \frac{\mu}{\mu + r} \right)^{0} \\ &= \left( \frac{r}{\mu + r} \right)^{r} \end{align} \]

The probability that \(y_i \neq 0\) is therefore

\[ f(y_i = 0; r, \mu_i) = \left( \frac{r}{\mu + r} \right)^{r} \\ \Downarrow \\ f(y_i \neq 0; r, \mu_i) = 1 - \left( \frac{r}{\mu + r} \right)^{r} \]

We can now exclude the probability that \(y_i = 0\) by dividing the pmf by the probability that \(y_i \neq 0\)

\[ f(y; r, \mu) = \frac{(y+r-1) !}{(r-1) ! y !} \left( \frac{r}{\mu + r} \right)^{r} \left( \frac{\mu}{\mu + r} \right)^{y} \\ \Downarrow \\ f(y_i; \lambda_i | y_i > 0) = \frac{ \frac{(y+r-1) !}{(r-1) ! y !} \left( \frac{r}{\mu + r} \right)^{r} \left( \frac{\mu}{\mu + r} \right)^{y} }{ 1 - \left( \frac{r}{\mu + r} \right)^{r} } \\ \Downarrow \\ \log \mathcal{L} = \log \mathcal{L}(\text{NB}) - \log \left( 1 - \left( \frac{r}{\mu + r} \right)^{r} \right) \]

Let’s first consider a regular negative binomial regression model

## load MASS pkg
library(MASS)
## neg binomial regression
smod_nb <- glm.nb(n_days ~ location + rain, data = snakes,
                  link = "log")

Now let’s fit a zero-truncated neg binomial regression model with vglm() from VGAM

library(VGAM)
## zero-truncated neg binomial regression
smod_ztnb <- vglm(n_days ~ location + rain, data = snakes,
                  family = posnegbinomial)

Here are the parameter estimates and SE’s for both models

##                NB   NB SE       +NB  +NB SE
## (Intercept) 0.418   0.107   -17.972   1.948
## locationV   0.454   0.152     0.965   2.933
## rain        0.025   0.005     0.065   0.103

Lots of count data are zero-inflated

The data contain more zeros than would be expected under a Poisson or negative binomial distribution

In general, there are 4 different types of errors that cause zeros

1. Structural (an animal is absent because the habitat is unsuitable)

In general, there are 4 different types of errors that cause zeros

1. Structural (an animal is absent because the habitat is unsuitable)

2. Design (sampling is limited temporally or spatially)

In general, there are 4 different types of errors that cause zeros

1. Structural (an animal is absent because the habitat is unsuitable)

2. Design (sampling is limited temporally or spatially)

3. Observer error (inexperience or difficult circumstances)

In general, there are 4 different types of errors that cause zeros

1. Structural (an animal is absent because the habitat is unsuitable)

2. Design (sampling is limited temporally or spatially)

3. Observer error (inexperience or difficult circumstances)

4. Process error (habitat is suitable but unused)

There are 2 general approaches for dealing with zero-inflated data

1. Zero-altered (“hurdle”) models

2. Zero-inflated (“mixture”) models

Hurdle models do not discriminate among the 4 types of zeros

The data are treated as 2 distinct groups:

1. Zeros

2. Non-zero counts

Hurdle models consist of 2 parts

1. Use a binomial model to determine the probability of a zero

2. If non-zero (“over the hurdle”), use a truncated Poisson or negative binomial to model the positive counts

A zero-altered Poisson (ZAP) model is given by

\[ f_{\text{ZAP}}(y; \pi, \lambda) = \left\{ \begin{array}{lc} f_{\text{binomial}}(y = 0; \pi) \\ \left[1 - f_{\text{binomial}}(y = 0; \pi) \right] \times \left( \frac{f_{\text{Poisson}}(y = 0; \lambda)}{1 - f_{\text{Poisson}}(y = 0; \lambda)} \right) \end{array} \right. \]

\(\pi\) is the probability of finding any individuals

\(\lambda\) is the mean (and variance) of the positive counts

We can model both parameters as functions of covariates

Probability of detection

\[ \text{logit}(\pi) = \mathbf{X}_d \boldsymbol{\beta}_d \]

Mean and variance of the positive counts

\[ \log(\lambda) = \mathbf{X}_c \boldsymbol{\beta}_c \]

Let’s apply a ZAP model to survey data for hippos

We’ll assume the following

• the probability of finding hippos increases with water availability

• the number of hippos increases with tree density

Detection as a function of water availability \(W\)

\[ z_i \sim \text{Bernoulli}(\pi_i) \\ \text{logit}(\pi) = \gamma_0 + \gamma_1 W_i \]

Positive counts as a function of tree density \(T\)

\[ c_i \sim \text{Poisson}^+(\lambda_i) \\ \log(\lambda) = \beta_0 + \beta_1 T_i \]

Total counts as a function of detections and positive counts

\[ y_i = z_i c_i \]

We can fit ZAP models in R with hurdle() from the pscl package

The formula for ZAP models is specified as

y ~ predictors_of_counts | predictors_for_detection

## load pscl
library(pscl)
## fit hurdle model
hippo_zap <- hurdle(y ~ trees | water)

A zero-altered negative binomial (ZANB) model is given by

\[ f_{\text{ZANB}}(y; \pi, \mu, r) = \left\{ \begin{array}{lc} f_{\text{binomial}}(y = 0; \pi) \\ \left[1 - f_{\text{binomial}}(y = 0; \pi) \right] \times \left( \frac{f_{\text{NB}}(y = 0; \mu, r)}{1 - f_{\text{NB}}(y = 0; \mu, r)} \right) \end{array} \right. \]

\(\pi\) is the probability of finding any individuals

\(\mu\) is the mean the positive counts

\(r\) is the scale for the positive counts

Zero-inflated (mixture) models treat the zeros as coming from 2 sources

1. observation errors (missed detections)

2. ecological (function of environment)

Zero-inflated (mixture) models consist of 2 parts

1. Use a binomial model to determine the probability of a zero

2. Use a Poisson or negative binomial to model counts, which can include zeros

Probability of a zero count comes from 2 sources:

1. false zeros (missed detections)

2. true zeros (ecological reasons)

Pr(zero) = Pr(false zero) + Pr(true zero) \(\times\) Pr(count = 0)

A zero-inflated Poisson (ZIP) model is given by

\[ \begin{align} f_{\text{ZIP}}(y = 0) &= f_{\text{Binomial}}(\pi) + [1 - f_{\text{Binomial}}(\pi)] f_{\text{Poisson}}(y = 0; \lambda) \\ ~ \\ f_{\text{ZIP}}(y | y > 0) &= [1 - f_{\text{Binomial}}(\pi)] f_{\text{Poisson}}(y; \lambda) \\ \end{align} \]

\(\pi\) is the probability of false zeros (missed detections)

\(\lambda\) is the mean (and variance) of all counts (including zeros)

We can model both parameters as functions of covariates

Probability of detection

\[ \text{logit}(\pi) = \mathbf{X}_d \boldsymbol{\beta}_d \]

Mean and variance of the counts

\[ \log(\lambda) = \mathbf{X}_c \boldsymbol{\beta}_c \]

Let’s apply a ZIP model to survey data for white tailed deer

We’ll assume the following

• the probability of detecting deer decreases with tree density

• the number of deer increases with tree density

Non-detection as a function of tree density \(T\)

\[ z_i \sim \text{Bernoulli}(\pi_i) \\ \text{logit}(\pi) = \gamma_0 + \gamma_1 T_i \]

Counts as a function of tree density \(T\)

\[ c_i \sim \text{Poisson}(\lambda_i) \\ \log(\lambda) = \beta_0 + \beta_1 T_i \]

Total counts as a function of detections and positive counts

\[ y_i = (1 - z_i) c_i \]

We can fit ZIP models in R with zeroinfl() from the pscl package

The formula for ZIP models is specified as

y ~ predictors_of_counts | predictors_for_detection

## fit hurdle model
deer_zip <- zeroinfl(y ~ trees | trees)

A zero-inflated negative binomial (ZINB) model is given by

\[ \begin{align} f_{\text{ZIP}}(y = 0) &= f_{\text{Binomial}}(\pi) + [1 - f_{\text{Binomial}}(\pi)] f_{\text{NB}}(y = 0; \mu, r) \\ ~ \\ f_{\text{ZIP}}(y | y > 0) &= [1 - f_{\text{Binomial}}(\pi)] f_{\text{NB}}(y; \mu, r) \\ \end{align} \]

\(\pi\) is the probability of false zeros (missed detections)

\(\mu\) is the mean of all counts (including zeros)

\(r\) is the scale of the counts

1. Understand the system of interest

Formulate good hypotheses and create a robust study design

1. Understand the system of interest

2. Detect and classify zeros

Remove false zeros due to design or observer errors

1. Understand the system of interest

2. Detect and classify zeros

3. Identify suitable covariates for zeros & non-zeros

What are the causes of zeros (non-zeros)

1. Understand the system of interest

2. Detect and classify zeros

3. Identify suitable covariates for zeros & non-zeros

4. Test for overdispersion

1. Understand the system of interest

2. Detect and classify zeros

3. Identify suitable covariates for zeros & non-zeros

4. Test for overdispersion

5. Choose appropriate model