Machine learning for geological modelling: neural networks, SVMs and Gaussian processes

Two goals: classification and regression

Geological modelling has two main tasks: classification, which models categorical data such as lithology, and numeric interpolation, also called regression, which models values such as assays.

Gaussian processes (GPs)

A Gaussian process, or Gaussian process regression, estimates the boundary between a unit and its surroundings and also gives the variance between known points, so it indicates how reliable the model is. This is very close to kriging combined with conditional simulation, and the Gaussian process literature itself lists kriging as a recommended reference. This ‘machine learning technique’ is not new to geology.

Support vector machines (SVMs)

Implicit modelling converts categorical data into numbers: +1 for points inside the unit, 0 for boundary points and −1 for points outside. RBFs are then fitted through all of the points. SVMs use the same kind of function, called a kernel, but instead of fitting exactly through every point they find the best-fit boundary between the two groups, ignoring samples that do not matter.

The difference is that an unoptimised RBF honours boundary points exactly, while an SVM may not if the inside and outside points sit at unequal distances from the boundary. SVMs learn quickly, give good estimates where contacts are uncertain, and control accuracy with a penalty on misclassified points. Results are reproducible and controlled.

Neural networks (NNs)

Neural networks, including convolutional and deep networks, can reproduce any continuous function given enough data and enough nodes. A basic network has an input layer (X, Y and Z), hidden nodes that each contain a function, often a Gaussian, and an output compared with the known indicator values. Training starts from random weights, which means every training run gives a slightly different result. They are slow to train, hard to define (how many nodes and layers?), and a network trained for one project does not transfer to another.

Technique	Closest geological tool	Strength	Caution
Gaussian process	Kriging, conditional simulation	Gives variance as well as an estimate	Assumes normally distributed data
Support vector machine	Implicit modelling	Fast, reproducible, good at uncertain contacts	Does not give uncertainty directly
Neural network	Fitting RBFs by training	Very fast to evaluate once trained	Random start, slow to train, not reproducible

Three cautions

1. Anisotropy. A machine learning model only separates one group from another. It could draw a small sphere around each inside point, which is a perfect answer to the maths and a useless one geologically. Anisotropy has to be specified before data goes in.
2. Discrete changes. Like implicit modelling, machine learning assumes continuous space, so faults smear unless they are handled separately.
3. Extrapolation. Outside the input points, the model extrapolates, and that needs great caution. In machine learning language this is over-fitting when done badly and good generalisation when done well.

Recommendations

• Understand the technique. Gaussian processes assume normal data, and geology rarely justifies that, so variography is still needed. SVMs are a fast, controlled way to model classifications.
• Understand the data. SVM results depend on how close samples sit to boundaries, so sampling and the conversion of drill logs to points matter.
• Understand the geology. A black box gives an answer whatever you do. Whether it makes geological sense, with discontinuities and anisotropy, is for the geologist to judge.