Chapter 8 Predicting Risk Impact Score Model – Artificial Intelligence for Risk Management

CHAPTER 8

Predicting Risk Impact Score Model

Define Goal

Define the goal of the model for predicting the risk impact score model. Predict the impact of risk based on expert knowledge or derived from historical occurrences.

Evaluation Steps

Evaluation Measure or Key Performance Indicator

From a given measure

Input:

  • Risk measure identified in the “risk categorization/classification” step.

Output:

  • Risk impact score 1 to 10 (minimum to high)

Evaluate the Business Process

From the given business process (retail industry Ex: sales transaction)

Input:

  • Feed the list of risk measures identified in the “risk categorization/classification” step.

Output:

  • List respective risk impact scores

Identify the list of measures > Identify the risks list > Identify the list of risk impact scores

Evaluate Given Dataset

From given data (retail industry Ex: POS data)

Input:

  • Feed the list of risk measures identified in the “risk categorization/classification” step.

Output:

  • List the respective risk impact scores

Identify the business process > Identify the list of measures > Identify the risks list > Identify the list of risk impact scores.

Evaluate Project-Related Documents

From the given project-related documents:

Collect risk impact data from historical occurrences or from an expert.

Use expert knowledge to capture the system of annotation and historical dataset.

Input:

  • Management plan
    • Evaluate stores and store sales performance to continue, close down, or improve sales.
  • Risk supporting documents
    • Historically identified risk datasets (past risk faced, its impact, and mitigation action taken).
  • Risk policy and standards
    • Company survival minimum requirements data (if revenue goes below 40 percent for three months, close down the company).
    • Hacker-avoidance policies
  • Organization rules, regulations, and policy
    • Organization rules (ex: monthly sales below 20 percent is a negative risk).
  • Dataset
    • POS transaction dataset

Data Collection

Input:

  • Output of risk categorization/classification model.
    • Measure risk name
    • Sales
    • Store
    • Location
    • Date
    • Industry average sales prev month
    • Industry average sales prev month − 1
    • Minimum monthly sales decrease percent
    • Three-month revenue decrease
    • Past sales transaction amount
    • Past risk flag
    • Past impact score
    • Past action taken
    • Risk category

Output:

  • What is the risk impact score?
  • Risk impact score (1 to 10)

Design Algorithm

Design an Algorithm to Predict the Risk Impact

The impact score is based on a range of numbers, typically 1 to 10; 1 means minimal impact and 10 means maximum impact. The corporation determines the range and impact based on expert knowledge related to the risk. The goal of predicting an algorithm is to determine whole numbers ranging from 1 to 10, based on historical data and expert knowledge through human annotation.

ML/AI Use Case

The machine learning (ML)/artificial intelligence (AI) use case is a regression analysis with continuous dependent variables. Now, we dive deeply into the regression steps.

Identify the List of Regression Models

The following algorithms can be used for regression analysis with continuous dependent variables (Table 8.1 and 8.2)

  • Linear regression
  • Polynomial regression
  • Stepwise regression and best subsets regression
  • Least absolute shrinkage and selection operator (LASSO) regression
  • Ridge regression
  • Partial least squares
    • Nonlinear regression
    • Support Vector Machine (SVM) with a nonlinear kernel
    • Quantile regression
    • Bayesian inference
    • Gauss–Newton algorithm
    • Gradient descent algorithm
    • Levenberg–Marquardt algorithm

    Finalized

  • Polynomial regression
  • SVM with a nonlinear kernel
  • Deep neural networks (deep learning)

Data Preparation


Table 8.1 Sample data prep

Measure risk name

Risk Category

Change Percentage

Risk Impact Score

Sales amount decrease

Financial/competitive risk

−10%

10

Sales amount increase

Inventory risk

30%

5



Table 8.2 Data sample with all features

Risk name (from step 1)

Sales Amount Decrease

Sales amount

10M

Store name

store1

Location

Atlanta

Date

1/3/18

Industry average sales amount previous month

30M

Industry average sales amount previous month − 1

40M

Minimum monthly sales amount decrease percent

15%

Three-month revenue decrease

20%

Past sales transaction amount

5M

Past risk flag

Y

Risk impact

High

Action taken

Store closed

Risk impact score (label)

10


Data Preprocessing

  • Missing values in categorical variables (e.g., store or location) need to be corrected or removed.
  • Reduce some levels if categorical predictors have many levels. A store has thousands of stores, so, group them based on sales volume and reduce them to three to five levels.
  • When using hundreds of variables, no regression will be interpretable. Reduce the number of variables using LASSO or least angle regression, factor analysis, substantive knowledge, correlation matrices, principal component analysis, or partial least squares.

Start with sensitivity analysis. Check the influence of each independent variable on the dependent(s) variable. Try to find out how much you can change each independent variable to change the dependent(s) by (for example) 10 percent.

For continuous variables: Use Pearson correlation coefficients. If Pearson correlation is near 1 or −1 among two, one should disappear in multiple regression (Figure 8.1).

Figure 8.1 Pearson correlation scatter plot

For category variables: If most counts appear in the diagonal of the contingency table, one of the two category variables should disappear. Draw scatter plots with continuous outcomes to see the association. Confirm if a linear trend is shown, the factor is in, if a nonlinear effect is shown, or transformation is needed. If no trend emerges, like the scatter plot is random, the independent factor could be out of sync (Figure 8.2).

Figure 8.2 Correlation sample

Using 14 variables, reduce it to 7 variables using aforementioned dimensionality reduction technique. The reduced variables list follows:

  • Measure risk name
  • Sales
  • Store
  • Risk category
  • Industry average sales previous month
  • Industry average sales previous month − 1
  • Minimum monthly sales decrease percent

Check for outliers by leverage or CooksD or Residual. If outliers are present, we can delete them to improve the quality of the data (Figure 8.3).

Figure 8.3 Trend with or without outlier

Check normality. If violated, use transformation on some independent factor. If variance homogeneity exists, transform the dependent factor. These processes may improve model fitting.

Train the Model

Using Deep Neural Network Architecture

Here we train the deep neural network to predict the risk impact score.

We present details of deep neural network architecture for this risk impact score model.

See Figure 2.6 for the deep neural network.

Activation function: see Figure 8.4.

Figure 8.4 Activation function of neural network

This deep neural network consists of

  • One input layer with 14 nodes.
  • Three hidden layers with 256 nodes each with a “relu” activation function and a “normal” initializer as the kernal_intializer.
  • Mean absolute error is a loss function.
  • The output layer has only one node.
  • Use “linear” as the activation function for the output layer (refer to Figure 8.4 and Table 8.3).

Table 8.3 Deep neural network layers

Layer (type)

Output Shape

Param #

Input layer (Dense)

(none, 128)

19,200

Hidden layer 1 (Dense)

(none, 256)

33,024

Hidden layer 1 (Dense)

(none, 256)

65,792

Hidden layer 1 (Dense)

(none, 256)

65,792

Output layer (Dense)

(none, 1)

257


Note: Total params:184,065; trainable params:184,065; nontrainable params: 0.


See that the validation loss of the best model is 18520.23 (Figure 8.5).

Figure 8.5 Training log output

Using Other Algorithms

Use the identified datasets to train the model with an annotated dataset.

Fit a model using linear regression first, then determine whether the linear model provides an adequate fit by checking the residual plots. If you cannot obtain a good fit using linear regression, try a nonlinear model because it can fit a wider variety of curves. We recommend using ordinary least squares first because it is easier to perform and interpret.

Use Akaike’s information criterion, Bayesian information criterion, or Mallows’ CP to decide how many factors should be included (Figure 8.6). Using them is better than comparing R2.

Figure 8.6 Principal components versus Bayesian information criterion

Perform multiple regression. If the sample size is large enough, you may use the autoselect option, such as forward, backward, or best, which will select independent factors using sampling techniques.

Use sampling techniques.

Test the Model

Test the model using the test dataset and expert knowledge.

Analyze different metrics like the statistical significance of parameters, R2, adjusted R2, Akaike information criterion, Bayesian information criterion, and the error term (Figure 8.7). Another one to use is Mallow’s Cp criterion, which checks for possible bias in the model by comparing the model with all possible submodels (or a careful selection of them).

Figure 8.7 R2 comparison graph

The model on the left is more accurate.

If the dataset has multiple confounding variables, you should not choose an automatic model selection method because you do not want to put these in a model at the same time.

Regression regularization methods (LASSO, Ridge, and ElasticNet) work well in case of high dimensionality and multicollinearity among the variables in the dataset.

Evaluate the Model

Evaluate the model using accuracy and mean square error and determine the learning rate. Cross-validation is the best way to evaluate models used for prediction. Here, divide the dataset into two groups (train and validate). A simple mean squared difference between the observed and predicted values gives a measure to predict accuracy (Figure 8.8).

Figure 8.8 Cross-validation comparison

Cross-validation graph:

Predicting the risk impact score model using a linear model:

Check predicted versus actual and root mean square error (RMSE) score as follows (Figure 8.9).

Figure 8.9 Evaluate the ensemble model predicted versus actual

Test RMSE score: 5.125877

Predicting the risk impact score model using bagged model using a randomForest algorithm:

Now, fit a bagged model using the randomForest algorithm. Bagging is a special case of a random forest where mtry (the number of variables randomly sampled as candidates at each split) is equal to p, the number of predictors. Try using 13 predictors.

Test RMSE score: 3.843966.

See two interesting results. First, the predicted versus actual plot no longer has a small number of predicted values. Second, the test error has dropped dramatically. Also note that the “mean of squared residuals,” which is output from randomForest, is the out-of-bag (OOB) estimate of the error.

Predicting the risk impact score model using a randomForest algorithm:

Now try using a random forest algorithm. For regression, we suggest using mtry equal to p/3 = 4 predictors

Test RMSE score: 3.701805.

Here note three RMSEs. The training RMSE (which is optimistic), the OOB RMSE (which is a reasonable estimate of the test error), and the test RMSE. Also note that we calculated variable importance. See Table 8.4 that is comparison of data and error.


Table 8.4 Data versus error comparison

No

Data

Error

1

Training Data

1.558111

2

OOB

3.576229

3

Test

3.701805


Note. OOB = out-of-bag.


Predicting the risk impact score model using a boosted model:

Last, try using a boosted model, which by default will produce a nice variable importance plot as well as plots of the marginal effects of the predictors. Based on this analysis, decide which variable has a greater influence on the risk impact score.

See TABLE 8.5 for the results of test error by models.


Table 8.5 Results

Model

Test Error

1

Single Tree

5.45808

2

Linear Model

5.12587

3

Bagging

3.84396

4

Random Forest

3.70180

5

Boosting

3.43382


Model Conclusion

The ensemble boosting model performed better than other algorithms in predicting a risk impact score.

Publish/Produce the Model

Same as risk categorization.

Conclusion

Same as risk categorization.

The same ML/AI model can be used to score the skills mentioned as follows.

Skills AI Scoring Models

  • Being respectfulness score (helping others retain their autonomy)
  • Courteousness score
  • Friendliness score
  • Kindness score
  • Honesty score
  • Trustworthiness score
  • Loyalty score
  • Ethical score

Skills scoring models can be used in customer, customer support executives, project resources, vendors, suppliers, buyers, and much more.