American Oil and Gas Reporter - April 2016 - 76

SpecialReport: Drilling Technology
Quality Of Model Fit
Many measures exist to evaluate the
quality of a model fit, including average
absolute error (AAE) and mean squared
error (MSE). These two metrics are
similar, and both attempt to capture the
overall closeness of predictions to the
evaluation data.
AAE is defined as the average magnitude of the difference between the true
response and the predicted response (i.e.,
the average size of the residuals). MSE
measures the average squared difference
between truth and prediction, rather than
the absolute value. AAE has units matching
those of the response, while MSE is
measured in squared units of the response.
Values closer to zero are desirable, since
they indicate smaller deviations between
the truth and predictions (i.e., more accurate prediction).
Figure 2 summarizes the model fitting
results on the Wolfcamp dataset. Each

plot shows the true response (M12CO)
on the horizontal axis and the predicted
response on the vertical axis. Points on
the diagonal dotted line indicate perfect
predictions. Each row of plots shows predictions from one type of model (OLS,
RF, GBM, SVR and KM), while each
column shows results for different model
evaluation types.
The left column shows independent
validation results, where a random 20
percent subset of the wells was held out.
The model was then fit to the remaining
80 percent of the dataset and evaluated
using the held-out dataset. The points in
the plots in the left column correspond
to only those predictions on the held-out
data.
For k-fold cross-validation predictions
(center column), a 10-fold cross-validation
was used as a further refinement of the
fivefold cross-validation approach. The
points in these plots show the actual

FIGURE 3
Decision Tree Separating Top 25 Percent
And Bottom 25 Percent of Producing Wells
PROP< 1,406e+06

LATLEN< 2,756

Bottom 25%
26/0

TVDSS>=-8,100

TVDSS>=-8,294
Bottom 25%
8/2

LATLEN>=5,362

Bottom 25%
7/3
Bottom 25%
21/2

Top 25%
6/15

76 THE AMERICAN OIL & GAS REPORTER

Top 25%
12/58

versus the cross-validated predictions of
each well. The right column shows the
results from training and predicting using
the full dataset.
The independent validation and crossvalidation results tell a much different
story than the full training set predictions.
Compared with OLS regression, many
of the models show a dramatic reduction
in error in both AAE and MSE metrics.
However, this reduction is more modest
for the other methods of model evaluation.
The extreme case is the kriging model,
which is a perfect interpolator, and therefore, forces the model fit through the
training observations by design.
Predictions with the full training set
yield an apparent perfect predictive ability,
which clearly is not going to hold up in
future datasets. However, for a case such
as random forest, it is not so clear that
predictions on the training data are biased.
It is only by comparing with independent
validation and cross-validation plots that
overfitting is revealed.
Basing expectations solely on the full
training-set-based predictions could lead
to an optimistic perception of model accuracy for future data, and disappointment
when those levels of accuracy are not
met. Accordingly, it may be wiser to
adopt a more robust method of model
evaluation, such as a k-fold cross-validation. The results should align better with
actual model performance on future data
collections.
Decision Tree Analysis
One strategy for tackling predictive
problems is to simplify the question being
asked. Rather than using predictive regression models to pinpoint the exact cumulative first-year production for a given
set of well characteristics, suppose the
aim of the exercise is to give a simple
"go/no go" solution on well decisions.
In this case, accurate first-year production prediction is not necessarily required. Instead, the goal is to predict
whether a well will be "good" or "bad"
(relatively high or low M12CO). This
problem can be simplified by changing
the modeling effort from regression to
classification. That is, the response can
be binned into categories, and models
can be used to predict which category a
well falls into.
Decision trees are useful for building
simple, interpretive models to describe
how these categories relate to the predictor
values. The common approach to con-

American Oil and Gas Reporter - April 2016

Table of Contents for the Digital Edition of American Oil and Gas Reporter - April 2016