Contents
- Non-negative Least Squares
- L1-Regularization Least Squares
- Logistic Regression with Bounded Coefficients
- Dual Support Vector Machines (no bias or regularized bias)
- Ordinal Logistic Regression
- Kernel Ordinal Logistic Regression
- Graphical LASSO
- Associative Conditional Random Fields (trained with pseudo-likelihood)
Non-negative Least Squares
Solve min_w (1/2)norm(X*w-y)^2, s.t. w >= 0.
nInstances = 1000; nVars = 100; X = randn(nInstances,nVars); w = randn(nVars,1); y = X*w + randn(nInstances,1); funObj = @(w)SquaredError(w,X,y); LB = zeros(nVars,1); UB = inf(nVars,1); fprintf('Solving non-negative least-squares problem...\n'); w = minConf_TMP(funObj,zeros(nVars,1),LB,UB); stem(w);title('Non-negative least-squares coefficients'); pause
Solving non-negative least-squares problem...
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 1.24808e-05 8.89869e+04 7.79994e+04
2 3 1.00000e+00 4.69101e+04 3.71020e+03
3 4 1.00000e+00 4.67942e+04 8.92785e+02
4 5 1.00000e+00 4.67885e+04 2.15416e+02
5 6 1.00000e+00 4.67882e+04 3.74646e+01
6 7 1.00000e+00 4.67882e+04 1.24189e+01
7 8 1.00000e+00 4.67882e+04 2.13076e+00
8 9 1.00000e+00 4.67882e+04 3.42795e-01
9 10 1.00000e+00 4.67882e+04 1.60818e-01
Function value changing by less than optTol
L1-Regularization Least Squares
Solve min_w (1/2)norm(X*w-y)^2 + lambda*sum(abs(w)) by formulating as min_w (1/2)norm([X -X]*w - y)^2 + lambda*sum(w), s.t. w >= 0
lambda = 1000*ones(nVars,1); regObj = @(w)nonNegGrad(w,lambda,funObj); LB = zeros(2*nVars,1); UB = inf(2*nVars,1); fprintf('Solving L1-regularized least-squares problem...\n'); w = minConf_TMP(regObj,zeros(2*nVars,1),LB,UB); w = w(1:nVars)-w(nVars+1:end); stem(w);title('L1-regularized least-squares coefficients'); pause
Solving L1-regularized least-squares problem...
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 1.33277e-05 8.92479e+04 7.27051e+04
2 3 1.00000e+00 5.65089e+04 1.01573e+04
3 4 1.00000e+00 5.60961e+04 5.85940e+03
4 5 1.00000e+00 5.58765e+04 1.27821e+03
5 6 1.00000e+00 5.58677e+04 8.21728e+02
6 7 1.00000e+00 5.58628e+04 1.75008e+02
7 8 1.00000e+00 5.58626e+04 8.13803e+01
8 9 1.00000e+00 5.58626e+04 1.96652e+01
9 10 1.00000e+00 5.58626e+04 9.32875e+00
10 11 1.00000e+00 5.58626e+04 3.11834e+00
11 12 1.00000e+00 5.58626e+04 1.39486e+00
12 13 1.00000e+00 5.58626e+04 4.80418e-01
13 14 1.00000e+00 5.58626e+04 1.97328e-01
Directional Derivative below optTol
Logistic Regression with Bounded Coefficients
Solve min_w sum_i log(1 + exp(-y(i)*w*X(i,:)), s.t. -1 <= w <= 1
y = sign(y); funObj = @(w)LogisticLoss(w,X,y); LB = -ones(nVars,1); UB = ones(nVars,1); fprintf('Solving bounded logistic regression problem...\n'); w = minConf_TMP(funObj,zeros(nVars,1),LB,UB); stem(w);title('Bounded logistic regression coefficients'); pause
Solving bounded logistic regression problem...
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 2.94363e-04 6.43855e+02 3.11272e+03
2 3 1.00000e+00 3.07472e+02 1.04097e+03
3 4 1.00000e+00 2.30254e+02 6.41644e+02
4 5 1.00000e+00 1.69623e+02 5.34782e+02
5 6 1.00000e+00 1.37826e+02 1.81410e+02
6 7 1.00000e+00 1.33822e+02 8.91419e+01
7 8 1.00000e+00 1.32733e+02 3.77198e+01
8 9 1.00000e+00 1.32525e+02 2.07840e+01
9 10 1.00000e+00 1.32484e+02 8.77170e+00
10 11 1.00000e+00 1.32475e+02 4.18951e+00
11 12 1.00000e+00 1.32472e+02 2.13249e+00
12 13 1.00000e+00 1.32471e+02 1.75750e+00
13 14 1.00000e+00 1.32471e+02 4.65247e-01
14 15 1.00000e+00 1.32471e+02 3.07314e-01
15 16 1.00000e+00 1.32471e+02 1.51880e-01
16 17 1.00000e+00 1.32471e+02 1.09456e-01
17 18 1.00000e+00 1.32471e+02 4.28841e-02
Function value changing by less than optTol
Dual Support Vector Machines (no bias or regularized bias)
Solve min_alpha (1/2)alpha'*A*alpha - sum(alpha), s.t. 0 <= alpha <= C, where A_ij = y_iy_jK(x_i,x_j), and we use the RBF kernel with sigma = 1.
% Generate data nInstances = 250; [X,y] = makeData('classificationNonlinear',nInstances,2,2); sigma = 1; K = kernelRBF(X,X,sigma); A = diag(y)*K*diag(y); C = 1; funObj = @(alpha)dualSVMLoss_noBias(alpha,A,y); LB = zeros(nInstances,1); UB = C*ones(nInstances,1); fprintf('Solving dual SVM problem (no bias)...\n'); alpha = minConf_TMP(funObj,zeros(nInstances,1),LB,UB); stem(alpha);title('Dual SVM coefficients'); yhat = sign(sum((diag(alpha.*y)*K)))'; trainErr = sum(yhat~=y)/numel(y) pause
Solving dual SVM problem (no bias)...
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 4.00000e-03 -9.26917e-01 2.13459e+02
2 3 1.00000e+00 -5.75999e+00 1.22354e+02
3 4 1.00000e+00 -9.03139e+00 1.90347e+02
4 5 1.00000e+00 -1.97189e+01 2.80367e+02
5 6 1.00000e+00 -2.72854e+01 4.65150e+02
6 7 1.00000e+00 -3.13428e+01 2.20405e+02
7 8 1.00000e+00 -4.02030e+01 2.28478e+02
8 9 1.00000e+00 -4.41960e+01 1.27701e+02
9 10 1.00000e+00 -4.78569e+01 8.68537e+01
10 11 1.00000e+00 -4.97190e+01 6.82303e+01
11 12 1.00000e+00 -5.34507e+01 4.15025e+01
Cubic Backtracking
12 14 1.19267e-01 -5.40919e+01 2.06099e+01
13 15 1.00000e+00 -5.46082e+01 1.64300e+01
14 16 1.00000e+00 -5.54528e+01 5.04322e+00
15 17 1.00000e+00 -5.55415e+01 4.37118e+00
16 18 1.00000e+00 -5.57400e+01 3.02294e+00
17 19 1.00000e+00 -5.57908e+01 3.68629e+00
18 20 1.00000e+00 -5.58890e+01 1.91893e+00
19 21 1.00000e+00 -5.59069e+01 1.60164e+00
20 22 1.00000e+00 -5.59286e+01 1.29089e+00
21 23 1.00000e+00 -5.59386e+01 9.61066e-01
22 24 1.00000e+00 -5.59445e+01 1.19811e+00
23 25 1.00000e+00 -5.59631e+01 1.00647e+00
24 26 1.00000e+00 -5.59823e+01 8.48519e-01
25 27 1.00000e+00 -5.59933e+01 7.11928e-01
26 28 1.00000e+00 -5.60026e+01 6.59629e-01
27 29 1.00000e+00 -5.60134e+01 9.78925e-01
Cubic Backtracking
28 31 4.48271e-02 -5.60148e+01 8.91318e-01
29 32 1.00000e+00 -5.60191e+01 5.14404e-01
30 33 1.00000e+00 -5.60256e+01 3.83046e-01
31 34 1.00000e+00 -5.60383e+01 3.81827e-01
Cubic Backtracking
32 36 2.39554e-01 -5.60446e+01 4.20466e-01
33 37 1.00000e+00 -5.60522e+01 5.11830e-01
Cubic Backtracking
34 39 6.78098e-02 -5.60539e+01 4.49363e-01
35 40 1.00000e+00 -5.60619e+01 4.95361e-01
36 41 1.00000e+00 -5.60745e+01 2.96566e-01
Cubic Backtracking
37 43 3.00525e-02 -5.60766e+01 2.83966e-01
Cubic Backtracking
38 45 3.40924e-02 -5.60787e+01 2.22022e-01
Cubic Backtracking
Cubic Backtracking
39 48 7.82637e-03 -5.60821e+01 2.37324e-01
Cubic Backtracking
Cubic Backtracking
40 51 3.79133e-02 -5.60858e+01 2.32438e-01
41 52 1.00000e+00 -5.60889e+01 3.69327e-01
Cubic Backtracking
Cubic Backtracking
42 55 2.72854e-02 -5.60916e+01 2.11027e-01
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
43 59 8.19016e-03 -5.60921e+01 2.12031e-01
Cubic Backtracking
44 61 1.33245e-01 -5.60973e+01 1.47493e-01
Cubic Backtracking
Cubic Backtracking
45 64 8.24514e-03 -5.60982e+01 1.57583e-01
46 65 1.00000e+00 -5.61010e+01 1.27420e-01
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
47 90 3.36137e-07 -5.61020e+01 1.47095e-01
48 91 1.00000e+00 -5.61037e+01 1.13463e-01
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
49 114 1.21311e-06 -5.61045e+01 6.81895e-02
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
50 131 6.01075e-06 -5.61046e+01 5.04040e-02
51 132 1.00000e+00 -5.61054e+01 5.23710e-02
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
52 159 9.35330e-09 -5.61054e+01 4.80074e-02
53 160 1.00000e+00 -5.61058e+01 5.61451e-02
54 161 1.00000e+00 -5.61065e+01 6.50190e-02
55 162 1.00000e+00 -5.61072e+01 5.95579e-02
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
56 172 3.29516e-04 -5.61073e+01 4.05834e-02
57 173 1.00000e+00 -5.61078e+01 2.31386e-02
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Interpolated value too large, Adjusting
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
58 183 4.67633e-05 -5.61079e+01 2.40555e-02
59 184 1.00000e+00 -5.61082e+01 1.99010e-03
60 185 1.00000e+00 -5.61082e+01 1.26033e-03
Directional Derivative below optTol
trainErr =
0.0720
Ordinal Logistic Regression
Solve min_{w,gamma} -sum(log(F(gamma(y+1) - X*w) - F(gamma(y) - X*w))), where F(x) = 1/(1+exp(-x), -inf < 0 < gamma(1) < gamma(2) < ... < inf
% Generate Data nInstances = 1000; nVars = 10; nClasses = 5; X = randn(nInstances,nVars); w = randn(nVars,1); gamma = sort(randn(nClasses-1,1)); z = X*w; y = zeros(nInstances,1); y(z < gamma(1)) = 1; for class = 2:nClasses-1 y(z >= gamma(class-1) & z < gamma(class)) = class; end y(z >= gamma(nClasses-1)) = nClasses; % Standardize columns and add bias X = standardizeCols(X); X = [ones(nInstances,1) X]; nVars = nVars+1; % First try Multinomial Logistic fprintf('Training multinomial logistic classifier\n'); model = classificationSoftmax(X,y,struct('nClasses',nClasses)); yhat = model.predictFunc(model,X); trainErr_MLR = sum(abs(yhat-y)); % Ordinal Logistic funObj = @(w)OrdinalLogisticLoss2(w,X,y,nClasses); LB = [-inf(nVars,1);zeros(nClasses-2,1)]; UB = inf(nVars+nClasses-2,1); w_init = zeros(nVars,1); gamma_init = sort(rand(nClasses-2,1)); fprintf('Training ordinal logistic classifier\n'); wGamma = minConf_TMP(funObj,[w_init;gamma_init],LB,UB); w = wGamma(1:nVars); gamma = [-inf;0;cumsum(wGamma(nVars+1:end));inf]; % Predict labels on training data z = X*w; yhat = zeros(nInstances,1); for c = 1:nClasses yhat(z > gamma(c)) = c; end trainErr_OLR = sum(abs(yhat-y)); fprintf('Training error of multinomial logistic regression: %f\n',trainErr_MLR/nInstances); fprintf('Training error of ordinal logistic regression: %f\n',trainErr_OLR/nInstances); pause
Training multinomial logistic classifier
Iteration FunEvals Step Length Function Val Opt Cond
1 2 5.73173e-04 1.47346e+03 2.38265e+02
2 3 1.00000e+00 8.92317e+02 7.51778e+01
3 4 1.00000e+00 7.61403e+02 4.14169e+01
4 5 1.00000e+00 6.31136e+02 5.74177e+01
5 6 1.00000e+00 5.53854e+02 7.22049e+01
6 7 1.00000e+00 5.05579e+02 2.31216e+01
7 8 1.00000e+00 4.59853e+02 1.34245e+01
8 9 1.00000e+00 4.00381e+02 1.74819e+01
9 11 3.83398e-01 3.64375e+02 3.30776e+01
10 12 1.00000e+00 2.75826e+02 2.54871e+01
11 13 1.00000e+00 2.12170e+02 2.13205e+01
12 14 1.00000e+00 1.78035e+02 2.22624e+01
13 15 1.00000e+00 1.57119e+02 1.16054e+01
14 16 1.00000e+00 1.32527e+02 9.24013e+00
15 17 1.00000e+00 1.11340e+02 1.06892e+01
16 18 1.00000e+00 1.05829e+02 2.40035e+01
17 19 1.00000e+00 7.77646e+01 1.14787e+01
18 20 1.00000e+00 6.47514e+01 9.04428e+00
19 24 1.25000e-01 6.28998e+01 8.71543e+00
20 32 7.81250e-03 6.26581e+01 8.68506e+00
21 42 1.95312e-03 6.25575e+01 8.67661e+00
22 53 9.76562e-04 6.25213e+01 8.67219e+00
23 73 1.90735e-06 6.25212e+01 8.67218e+00
24 95 4.76837e-07 6.25212e+01 8.67217e+00
25 118 2.38419e-07 6.25212e+01 8.67217e+00
26 142 1.19209e-07 6.25212e+01 8.67217e+00
27 167 5.96046e-08 6.25212e+01 8.67217e+00
28 194 1.49012e-08 6.25212e+01 8.67217e+00
29 222 7.45058e-09 6.25212e+01 8.67217e+00
30 254 4.65661e-10 6.25212e+01 8.67217e+00
31 288 1.16415e-10 6.25212e+01 8.67217e+00
32 326 0.00000e+00 6.25212e+01 8.67217e+00
Step Size below progTol
Training ordinal logistic classifier
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 5.80099e-04 1.20021e+03 1.33661e+03
2 3 1.00000e+00 7.97824e+02 9.60976e+02
3 4 1.00000e+00 6.05549e+02 3.56318e+02
4 5 1.00000e+00 5.16553e+02 2.65410e+02
5 6 1.00000e+00 3.73899e+02 1.78636e+02
6 7 1.00000e+00 3.23800e+02 4.56331e+02
7 8 1.00000e+00 2.38817e+02 1.98526e+02
8 9 1.00000e+00 1.87020e+02 5.15878e+01
9 10 1.00000e+00 1.53108e+02 6.96180e+01
10 11 1.00000e+00 1.17863e+02 1.04043e+02
11 12 1.00000e+00 8.43583e+01 1.26589e+02
12 13 1.00000e+00 6.44784e+01 6.13154e+01
13 14 1.00000e+00 5.03957e+01 3.06560e+01
14 15 1.00000e+00 4.08723e+01 2.43359e+01
15 16 1.00000e+00 3.75675e+01 6.16553e+01
16 17 1.00000e+00 2.97615e+01 2.56469e+01
17 18 1.00000e+00 2.63149e+01 1.39195e+01
18 19 1.00000e+00 2.33707e+01 1.37799e+01
19 20 1.00000e+00 1.99202e+01 1.33241e+01
Cubic Backtracking
20 22 2.90055e-01 1.84987e+01 1.92716e+01
21 23 1.00000e+00 1.60015e+01 1.22602e+01
22 24 1.00000e+00 1.41525e+01 9.60725e+00
23 25 1.00000e+00 1.29064e+01 1.17556e+01
24 26 1.00000e+00 1.17360e+01 1.10115e+01
25 27 1.00000e+00 1.13209e+01 1.01109e+01
26 28 1.00000e+00 1.08303e+01 9.44927e+00
27 29 1.00000e+00 1.03740e+01 9.41551e+00
28 30 1.00000e+00 9.83721e+00 8.33298e+00
29 31 1.00000e+00 9.35614e+00 9.25077e+00
30 32 1.00000e+00 8.99625e+00 9.78427e+00
31 33 1.00000e+00 8.52401e+00 1.01430e+01
32 34 1.00000e+00 8.06894e+00 9.62485e+00
33 35 1.00000e+00 6.87749e+00 7.33461e+00
34 36 1.00000e+00 5.90297e+00 5.44936e+00
35 37 1.00000e+00 4.65697e+00 3.79086e+00
36 38 1.00000e+00 3.61993e+00 4.69347e+00
37 39 1.00000e+00 2.56286e+00 6.36885e+00
38 40 1.00000e+00 1.44746e+00 5.52026e+00
39 41 1.00000e+00 5.91079e-01 2.47167e+00
40 42 1.00000e+00 2.92462e-01 1.18089e+00
41 43 1.00000e+00 1.44326e-01 5.46954e-01
42 44 1.00000e+00 7.29289e-02 2.62604e-01
43 45 1.00000e+00 3.68666e-02 1.27048e-01
44 46 1.00000e+00 1.86729e-02 6.18512e-02
45 47 1.00000e+00 9.43570e-03 3.05331e-02
46 48 1.00000e+00 4.75981e-03 1.51698e-02
47 49 1.00000e+00 2.39742e-03 7.72119e-03
48 50 1.00000e+00 1.20658e-03 3.88779e-03
49 51 1.00000e+00 6.07011e-04 1.94112e-03
50 52 1.00000e+00 3.05357e-04 9.62478e-04
51 53 1.00000e+00 1.53628e-04 4.74365e-04
52 54 1.00000e+00 7.73091e-05 2.32521e-04
53 55 1.00000e+00 3.89143e-05 1.13401e-04
54 56 1.00000e+00 1.95930e-05 5.50488e-05
55 57 1.00000e+00 9.86679e-06 2.66103e-05
56 58 1.00000e+00 4.96927e-06 1.28172e-05
57 59 1.00000e+00 2.50263e-06 6.27295e-06
58 60 1.00000e+00 1.26018e-06 3.07341e-06
Directional Derivative below optTol
Training error of multinomial logistic regression: 0.021000
Training error of ordinal logistic regression: 0.000000
Kernel Ordinal Logistic Regression
% Generate data nInstances = 500; nVars = 2; nClasses = 5; X = randn(nInstances,nVars); nExamplePoints = 3; examplePoints = randn(nExamplePoints,nVars); thresholds = [0;cumsum(2*rand(nClasses-1,1))]; y = zeros(nInstances,1); for i = 1:nInstances dists = sum((repmat(X(i,:),nExamplePoints,1) - examplePoints).^2,2); y(i,1) = max(find(min(dists) > thresholds)); end X = [ones(nInstances,1) standardizeCols(X)]; nTrain = nInstances/2; Xtrain = X(1:nTrain,:); ytrain = y(1:nTrain); Xtest = X(nTrain+1:end,:); ytest = y(nTrain+1:end); % First try kernel multinomial logistic sigma = 1; lambda = 1e-5; model = classificationKernelSoftmax(Xtrain,ytrain,struct('nClasses',nClasses,'kernelFunc',@kernelRBF,'kernelArgs',sigma,'lambda',lambda)); yhat = model.predictFunc(model,Xtest); size(yhat) size(ytest) testErr_KMLR = sum(yhat~=ytest)/length(ytest); testDist_KMLR = sum(abs(yhat-ytest))/length(ytest); % Now try kernel ordinal logistic Ktrain = kernelRBF(Xtrain,Xtrain,sigma); % Set up problem w = zeros(nTrain,1); gamma = ones(nClasses-2,1); LB = [-inf(nTrain,1);zeros(nClasses-2,1)]; UB = inf(nTrain+nClasses-2,1); funObj_sub = @(w)OrdinalLogisticLoss2(w,Ktrain,ytrain,nClasses); funObj = @(w)penalizedKernelL2_subset(w,Ktrain,1:nTrain,funObj_sub,lambda); % Solve optiamization wGamma = minConf_TMP(funObj,[w;gamma],LB,UB); w = wGamma(1:nTrain); gamma = [-inf;0;cumsum(wGamma(nTrain+1:end));inf]; % Predict on test data Ktest = kernelRBF(Xtest,Xtrain,sigma); z = Ktest*w; yhat = zeros(size(ytest)); for c = 1:nClasses yhat(z > gamma(c)) = c; end testErr_KOLR = sum(yhat~=ytest)/length(ytest); testDist_KOLR = sum(abs(yhat-ytest))/length(ytest); fprintf('Test error of kernel multinomial logistic regression: %f (distance = %f)\n',testErr_KMLR,testDist_KMLR); fprintf('Test error of kernel ordinal logistic regression: %f (distance = %f)\n',testErr_KOLR,testDist_KOLR);
Iteration FunEvals Step Length Function Val Opt Cond
1 3 2.13324e-03 3.54854e+02 8.33044e+00
2 4 1.00000e+00 3.18694e+02 5.00661e+00
3 5 1.00000e+00 2.91184e+02 3.43676e+00
4 7 4.41823e-01 2.79312e+02 5.95776e+00
5 8 1.00000e+00 2.61627e+02 2.99960e+00
6 9 1.00000e+00 2.55434e+02 2.48155e+00
7 10 1.00000e+00 2.49740e+02 2.40483e+00
8 11 1.00000e+00 2.38219e+02 2.71464e+00
9 12 1.00000e+00 2.22439e+02 2.29682e+00
10 13 1.00000e+00 2.02191e+02 1.89955e+00
11 14 1.00000e+00 1.88408e+02 2.23119e+00
12 15 1.00000e+00 1.80382e+02 1.23146e+00
13 16 1.00000e+00 1.77407e+02 9.90214e-01
14 17 1.00000e+00 1.72991e+02 7.49834e-01
15 18 1.00000e+00 1.69072e+02 9.58311e-01
16 19 1.00000e+00 1.63715e+02 1.02299e+00
17 20 1.00000e+00 1.57243e+02 1.16288e+00
18 21 1.00000e+00 1.48634e+02 1.34369e+00
19 22 1.00000e+00 1.43571e+02 2.63967e+00
20 23 1.00000e+00 1.37850e+02 8.31940e-01
21 24 1.00000e+00 1.36567e+02 6.93175e-01
22 25 1.00000e+00 1.32644e+02 7.76764e-01
23 26 1.00000e+00 1.31193e+02 6.92903e-01
24 27 1.00000e+00 1.29112e+02 5.46202e-01
25 28 1.00000e+00 1.27049e+02 4.39825e-01
26 29 1.00000e+00 1.25549e+02 6.00014e-01
27 30 1.00000e+00 1.23622e+02 1.00955e+00
28 31 1.00000e+00 1.20992e+02 1.23773e+00
29 32 1.00000e+00 1.16444e+02 1.06610e+00
30 33 1.00000e+00 1.12199e+02 4.46398e-01
31 34 1.00000e+00 1.09975e+02 4.64400e-01
32 35 1.00000e+00 1.08262e+02 5.00294e-01
33 36 1.00000e+00 1.06224e+02 5.84955e-01
34 37 1.00000e+00 1.03314e+02 6.14773e-01
35 38 1.00000e+00 9.99716e+01 6.43601e-01
36 39 1.00000e+00 9.65581e+01 6.11731e-01
37 40 1.00000e+00 9.26869e+01 4.34685e-01
38 41 1.00000e+00 9.00205e+01 4.89075e-01
39 42 1.00000e+00 8.81637e+01 4.26777e-01
40 43 1.00000e+00 8.55292e+01 4.81282e-01
41 44 1.00000e+00 8.33355e+01 5.30098e-01
42 45 1.00000e+00 8.05128e+01 5.88957e-01
43 46 1.00000e+00 7.80316e+01 5.90707e-01
44 47 1.00000e+00 7.60752e+01 5.23750e-01
45 48 1.00000e+00 7.49316e+01 3.28400e-01
46 49 1.00000e+00 7.38952e+01 2.25505e-01
47 50 1.00000e+00 7.30590e+01 2.10095e-01
48 51 1.00000e+00 7.24767e+01 2.31355e-01
49 52 1.00000e+00 7.19834e+01 2.59055e-01
50 53 1.00000e+00 7.11627e+01 2.83822e-01
51 54 1.00000e+00 6.98997e+01 3.32865e-01
52 55 1.00000e+00 6.80022e+01 4.46002e-01
53 56 1.00000e+00 6.64773e+01 3.24199e-01
54 57 1.00000e+00 6.55106e+01 2.90727e-01
55 58 1.00000e+00 6.48317e+01 2.73925e-01
56 59 1.00000e+00 6.43246e+01 2.28600e-01
57 60 1.00000e+00 6.35772e+01 1.96386e-01
58 61 1.00000e+00 6.29918e+01 2.27184e-01
59 62 1.00000e+00 6.23547e+01 3.18440e-01
60 63 1.00000e+00 6.17582e+01 2.76975e-01
61 64 1.00000e+00 6.08353e+01 2.86630e-01
62 65 1.00000e+00 5.97196e+01 2.04173e-01
63 66 1.00000e+00 5.89917e+01 1.48082e-01
64 67 1.00000e+00 5.85154e+01 1.47877e-01
65 68 1.00000e+00 5.82248e+01 1.39592e-01
66 69 1.00000e+00 5.79756e+01 1.36212e-01
67 70 1.00000e+00 5.76751e+01 1.27488e-01
68 71 1.00000e+00 5.71578e+01 1.63947e-01
69 72 1.00000e+00 5.65886e+01 2.11440e-01
70 73 1.00000e+00 5.59795e+01 1.77772e-01
71 74 1.00000e+00 5.53545e+01 1.94517e-01
72 75 1.00000e+00 5.47410e+01 2.08660e-01
73 76 1.00000e+00 5.41624e+01 2.09681e-01
74 77 1.00000e+00 5.34943e+01 2.11299e-01
75 78 1.00000e+00 5.28778e+01 1.98703e-01
76 79 1.00000e+00 5.24042e+01 1.87274e-01
77 80 1.00000e+00 5.19629e+01 1.68527e-01
78 81 1.00000e+00 5.14987e+01 1.45424e-01
79 82 1.00000e+00 5.10795e+01 1.99074e-01
80 83 1.00000e+00 5.07154e+01 1.65776e-01
81 84 1.00000e+00 5.05044e+01 1.51417e-01
82 85 1.00000e+00 5.01888e+01 1.30166e-01
83 86 1.00000e+00 4.98565e+01 1.21368e-01
84 87 1.00000e+00 4.95596e+01 1.26365e-01
85 88 1.00000e+00 4.93135e+01 1.08510e-01
86 89 1.00000e+00 4.90967e+01 1.32252e-01
87 90 1.00000e+00 4.89327e+01 6.75589e-02
88 91 1.00000e+00 4.88255e+01 4.64096e-02
89 92 1.00000e+00 4.87354e+01 6.86643e-02
90 93 1.00000e+00 4.86320e+01 7.94012e-02
91 94 1.00000e+00 4.84475e+01 1.12847e-01
92 95 1.00000e+00 4.81182e+01 1.27913e-01
93 96 1.00000e+00 4.76946e+01 1.01822e-01
94 97 1.00000e+00 4.72271e+01 1.54352e-01
95 98 1.00000e+00 4.67618e+01 1.40412e-01
96 99 1.00000e+00 4.65337e+01 1.41989e-01
97 100 1.00000e+00 4.61213e+01 1.05646e-01
98 101 1.00000e+00 4.56980e+01 1.19658e-01
99 102 1.00000e+00 4.54045e+01 1.58827e-01
100 103 1.00000e+00 4.49888e+01 1.47646e-01
101 104 1.00000e+00 4.47842e+01 1.00358e-01
102 105 1.00000e+00 4.46401e+01 7.24773e-02
103 106 1.00000e+00 4.44948e+01 6.02935e-02
104 107 1.00000e+00 4.43679e+01 1.16092e-01
105 108 1.00000e+00 4.42176e+01 1.54380e-01
106 109 1.00000e+00 4.40809e+01 1.64771e-01
107 110 1.00000e+00 4.39516e+01 2.27888e-01
108 111 1.00000e+00 4.38029e+01 7.98124e-02
109 112 1.00000e+00 4.36656e+01 1.05189e-01
110 113 1.00000e+00 4.35948e+01 1.17858e-01
111 114 1.00000e+00 4.35087e+01 1.34100e-01
112 116 4.75793e-01 4.34494e+01 1.57602e-01
113 117 1.00000e+00 4.33641e+01 9.48323e-02
114 118 1.00000e+00 4.32502e+01 9.36612e-02
115 119 1.00000e+00 4.31783e+01 8.55612e-02
116 120 1.00000e+00 4.30811e+01 1.01411e-01
117 121 1.00000e+00 4.29434e+01 8.03430e-02
118 122 1.00000e+00 4.28102e+01 9.30981e-02
119 123 1.00000e+00 4.26689e+01 1.01123e-01
120 124 1.00000e+00 4.24702e+01 5.04561e-02
121 125 1.00000e+00 4.23408e+01 6.80296e-02
122 126 1.00000e+00 4.23262e+01 1.63732e-01
123 127 1.00000e+00 4.22512e+01 7.79125e-02
124 128 1.00000e+00 4.22225e+01 5.18407e-02
125 129 1.00000e+00 4.21794e+01 4.31252e-02
126 130 1.00000e+00 4.21503e+01 3.25736e-02
127 131 1.00000e+00 4.21167e+01 3.18588e-02
128 132 1.00000e+00 4.20949e+01 9.30823e-02
129 133 1.00000e+00 4.20300e+01 7.71008e-02
130 134 1.00000e+00 4.19062e+01 7.22154e-02
131 135 1.00000e+00 4.16971e+01 8.39423e-02
132 136 1.00000e+00 4.13794e+01 1.17511e-01
133 137 1.00000e+00 4.09690e+01 1.22768e-01
134 139 1.69431e-01 4.08770e+01 2.29873e-01
135 140 1.00000e+00 4.04544e+01 1.27484e-01
136 141 1.00000e+00 4.01541e+01 6.10053e-02
137 142 1.00000e+00 3.99784e+01 6.04868e-02
138 143 1.00000e+00 3.99036e+01 8.45610e-02
139 144 1.00000e+00 3.97892e+01 5.97570e-02
140 145 1.00000e+00 3.96635e+01 9.30054e-02
141 146 1.00000e+00 3.94677e+01 1.15860e-01
142 147 1.00000e+00 3.91579e+01 1.29154e-01
143 148 1.00000e+00 3.84125e+01 1.40913e-01
144 149 1.00000e+00 3.81053e+01 1.47753e-01
145 150 1.00000e+00 3.76143e+01 1.29875e-01
146 152 3.62470e-01 3.73281e+01 9.44510e-02
147 153 1.00000e+00 3.70085e+01 6.88468e-02
148 154 1.00000e+00 3.68434e+01 5.20709e-02
149 155 1.00000e+00 3.67717e+01 1.21176e-01
150 156 1.00000e+00 3.66803e+01 7.10954e-02
151 157 1.00000e+00 3.66114e+01 4.47466e-02
152 158 1.00000e+00 3.65334e+01 5.53025e-02
153 159 1.00000e+00 3.64045e+01 8.20290e-02
154 160 1.00000e+00 3.61355e+01 1.12661e-01
155 161 1.00000e+00 3.55249e+01 1.17523e-01
156 162 1.00000e+00 3.46061e+01 1.62455e-01
157 164 5.68194e-01 3.43029e+01 1.66512e-01
158 165 1.00000e+00 3.40478e+01 7.70333e-02
159 166 1.00000e+00 3.39672e+01 3.06119e-02
160 167 1.00000e+00 3.38942e+01 4.20252e-02
161 168 1.00000e+00 3.38093e+01 6.39820e-02
162 169 1.00000e+00 3.37452e+01 5.29902e-02
163 170 1.00000e+00 3.36923e+01 4.76744e-02
164 171 1.00000e+00 3.36580e+01 3.44912e-02
165 172 1.00000e+00 3.36378e+01 3.30893e-02
166 173 1.00000e+00 3.36079e+01 3.42213e-02
167 174 1.00000e+00 3.35744e+01 4.45973e-02
168 175 1.00000e+00 3.35207e+01 3.22907e-02
169 176 1.00000e+00 3.34506e+01 4.31404e-02
170 177 1.00000e+00 3.33617e+01 3.37945e-02
171 178 1.00000e+00 3.32406e+01 4.94269e-02
172 179 1.00000e+00 3.30708e+01 7.80331e-02
173 180 1.00000e+00 3.28954e+01 7.83057e-02
174 181 1.00000e+00 3.27255e+01 8.26793e-02
175 182 1.00000e+00 3.25748e+01 6.46835e-02
176 183 1.00000e+00 3.24279e+01 4.34906e-02
177 184 1.00000e+00 3.23623e+01 3.86816e-02
178 185 1.00000e+00 3.22893e+01 3.65844e-02
179 186 1.00000e+00 3.22372e+01 3.63775e-02
180 187 1.00000e+00 3.21410e+01 3.67822e-02
181 188 1.00000e+00 3.20466e+01 4.47560e-02
182 189 1.00000e+00 3.19515e+01 5.15427e-02
183 190 1.00000e+00 3.17619e+01 4.84240e-02
184 191 1.00000e+00 3.15260e+01 5.75948e-02
185 192 1.00000e+00 3.13484e+01 4.88022e-02
186 193 1.00000e+00 3.11844e+01 9.80825e-02
187 194 1.00000e+00 3.10681e+01 3.76497e-02
188 195 1.00000e+00 3.10222e+01 2.04167e-02
189 196 1.00000e+00 3.09716e+01 2.02444e-02
190 197 1.00000e+00 3.09225e+01 1.98161e-02
191 198 1.00000e+00 3.09015e+01 7.02329e-02
192 199 1.00000e+00 3.08594e+01 2.38482e-02
193 200 1.00000e+00 3.08375e+01 1.18977e-02
194 201 1.00000e+00 3.08214e+01 1.22985e-02
195 202 1.00000e+00 3.07969e+01 3.04791e-02
196 203 1.00000e+00 3.07690e+01 1.97455e-02
197 204 1.00000e+00 3.07357e+01 2.75860e-02
198 205 1.00000e+00 3.06800e+01 3.29343e-02
199 206 1.00000e+00 3.06118e+01 3.72502e-02
200 207 1.00000e+00 3.05165e+01 4.53003e-02
201 208 1.00000e+00 3.04566e+01 7.14695e-02
202 209 1.00000e+00 3.03769e+01 2.23796e-02
203 210 1.00000e+00 3.03322e+01 2.22776e-02
204 211 1.00000e+00 3.03202e+01 1.52094e-02
205 212 1.00000e+00 3.03151e+01 1.18358e-02
206 213 1.00000e+00 3.03137e+01 1.69261e-02
207 214 1.00000e+00 3.03056e+01 9.14823e-03
208 215 1.00000e+00 3.02841e+01 2.29854e-02
209 216 1.00000e+00 3.02621e+01 5.54825e-02
210 217 1.00000e+00 3.02101e+01 5.03800e-02
211 219 4.95752e-02 3.02033e+01 1.27666e-01
212 220 1.00000e+00 3.01568e+01 7.23437e-02
213 221 1.00000e+00 3.01094e+01 2.95132e-02
214 222 1.00000e+00 3.00693e+01 5.00521e-02
215 223 1.00000e+00 3.00057e+01 7.41982e-02
216 224 1.00000e+00 2.98973e+01 1.02199e-01
217 225 1.00000e+00 2.98465e+01 2.03804e-01
218 226 1.00000e+00 2.96591e+01 3.39804e-02
219 227 1.00000e+00 2.95613e+01 6.30799e-02
220 228 1.00000e+00 2.94821e+01 4.00120e-02
221 229 1.00000e+00 2.94658e+01 2.28889e-02
222 230 1.00000e+00 2.94520e+01 1.38365e-02
223 232 2.86999e-01 2.94481e+01 2.02530e-02
224 233 1.00000e+00 2.94449e+01 9.53947e-03
225 234 1.00000e+00 2.94430e+01 1.03153e-02
226 235 1.00000e+00 2.94415e+01 1.14361e-02
227 236 1.00000e+00 2.94386e+01 9.99272e-03
228 237 1.00000e+00 2.94368e+01 1.55556e-02
229 238 1.00000e+00 2.94345e+01 8.48339e-03
230 239 1.00000e+00 2.94320e+01 1.10855e-02
231 240 1.00000e+00 2.94300e+01 1.40278e-02
232 241 1.00000e+00 2.94246e+01 2.02731e-02
233 242 1.00000e+00 2.94132e+01 2.28292e-02
234 243 1.00000e+00 2.93902e+01 2.62066e-02
235 244 1.00000e+00 2.93405e+01 1.87045e-02
236 245 1.00000e+00 2.92423e+01 3.21045e-02
237 246 1.00000e+00 2.90070e+01 7.85549e-02
238 248 5.59840e-01 2.88402e+01 8.90409e-02
239 250 1.27368e-01 2.87612e+01 1.97536e-01
240 251 1.00000e+00 2.84379e+01 9.33830e-02
241 252 1.00000e+00 2.81936e+01 3.07257e-02
242 253 1.00000e+00 2.81112e+01 2.33671e-02
243 255 2.85376e-01 2.80812e+01 2.96049e-02
244 256 1.00000e+00 2.80615e+01 2.54678e-02
245 257 1.00000e+00 2.80527e+01 2.72494e-02
246 258 1.00000e+00 2.80453e+01 2.71344e-02
247 259 1.00000e+00 2.80384e+01 2.30478e-02
248 260 1.00000e+00 2.80292e+01 2.54061e-02
249 261 1.00000e+00 2.80153e+01 2.13541e-02
250 262 1.00000e+00 2.79842e+01 3.61825e-02
251 263 1.00000e+00 2.79340e+01 3.99559e-02
252 264 1.00000e+00 2.78582e+01 6.76647e-02
253 265 1.00000e+00 2.77547e+01 5.93795e-02
254 266 1.00000e+00 2.76246e+01 6.92120e-02
255 268 5.31821e-01 2.75487e+01 1.34255e-01
256 269 1.00000e+00 2.74509e+01 4.94029e-02
257 270 1.00000e+00 2.73987e+01 5.64874e-02
258 271 1.00000e+00 2.73168e+01 1.44504e-02
259 272 1.00000e+00 2.72734e+01 2.11049e-02
260 273 1.00000e+00 2.72532e+01 1.72707e-02
261 274 1.00000e+00 2.72471e+01 1.55459e-02
262 275 1.00000e+00 2.72343e+01 1.19456e-02
263 276 1.00000e+00 2.72299e+01 8.79717e-03
264 277 1.00000e+00 2.72243e+01 8.55031e-03
265 278 1.00000e+00 2.72173e+01 8.83136e-03
266 279 1.00000e+00 2.72063e+01 1.00044e-02
267 280 1.00000e+00 2.71862e+01 1.20220e-02
268 281 1.00000e+00 2.71625e+01 8.90839e-03
269 282 1.00000e+00 2.71219e+01 9.70785e-03
270 283 1.00000e+00 2.70673e+01 1.87904e-02
271 284 1.00000e+00 2.70103e+01 1.78864e-02
272 285 1.00000e+00 2.69360e+01 2.72790e-02
273 286 1.00000e+00 2.68905e+01 5.43844e-02
274 287 1.00000e+00 2.68466e+01 2.99264e-02
275 288 1.00000e+00 2.68115e+01 2.59628e-02
276 289 1.00000e+00 2.67862e+01 3.23528e-02
277 290 1.00000e+00 2.67679e+01 2.08323e-02
278 291 1.00000e+00 2.67543e+01 9.08080e-03
279 292 1.00000e+00 2.67483e+01 9.91359e-03
280 293 1.00000e+00 2.67438e+01 8.96554e-03
281 294 1.00000e+00 2.67403e+01 6.00181e-03
282 295 1.00000e+00 2.67366e+01 6.91457e-03
283 296 1.00000e+00 2.67329e+01 7.70078e-03
284 297 1.00000e+00 2.67266e+01 9.47308e-03
285 298 1.00000e+00 2.67184e+01 9.24212e-03
286 300 4.15455e-01 2.67100e+01 1.09366e-02
287 301 1.00000e+00 2.66846e+01 1.73159e-02
288 302 1.00000e+00 2.66366e+01 7.29436e-03
289 303 1.00000e+00 2.65750e+01 2.13661e-02
290 304 1.00000e+00 2.64902e+01 2.70449e-02
291 305 1.00000e+00 2.64255e+01 3.22887e-02
292 306 1.00000e+00 2.63627e+01 3.35494e-02
293 307 1.00000e+00 2.63064e+01 1.19987e-02
294 308 1.00000e+00 2.62610e+01 2.10776e-02
295 309 1.00000e+00 2.62295e+01 4.81601e-02
296 310 1.00000e+00 2.62076e+01 1.58605e-02
297 311 1.00000e+00 2.61927e+01 8.59292e-03
298 312 1.00000e+00 2.61854e+01 1.57499e-02
299 313 1.00000e+00 2.61812e+01 1.18851e-02
300 314 1.00000e+00 2.61777e+01 7.06455e-03
301 315 1.00000e+00 2.61756e+01 7.58195e-03
302 316 1.00000e+00 2.61738e+01 7.94838e-03
303 317 1.00000e+00 2.61711e+01 8.19341e-03
304 318 1.00000e+00 2.61659e+01 8.64237e-03
305 319 1.00000e+00 2.61556e+01 9.26629e-03
306 321 5.22271e-01 2.61472e+01 4.49733e-02
307 322 1.00000e+00 2.61241e+01 2.89460e-02
308 323 1.00000e+00 2.60425e+01 5.46943e-02
309 324 1.00000e+00 2.59595e+01 8.44463e-02
310 325 1.00000e+00 2.57956e+01 1.07032e-01
311 326 1.00000e+00 2.57362e+01 5.94891e-02
312 327 1.00000e+00 2.56807e+01 9.00210e-02
313 329 3.39744e-01 2.56640e+01 2.77529e-02
314 330 1.00000e+00 2.56445e+01 2.28717e-02
315 331 1.00000e+00 2.56336e+01 2.65292e-02
316 332 1.00000e+00 2.56223e+01 9.18636e-03
317 333 1.00000e+00 2.56180e+01 6.99928e-03
318 334 1.00000e+00 2.56149e+01 7.73560e-03
319 335 1.00000e+00 2.56141e+01 9.47073e-03
320 336 1.00000e+00 2.56120e+01 6.77255e-03
321 337 1.00000e+00 2.56103e+01 6.26592e-03
322 338 1.00000e+00 2.56074e+01 5.87769e-03
323 339 1.00000e+00 2.56041e+01 6.43137e-03
324 340 1.00000e+00 2.55991e+01 8.38271e-03
325 341 1.00000e+00 2.55877e+01 1.27723e-02
326 342 1.00000e+00 2.55721e+01 3.20210e-02
327 343 1.00000e+00 2.55503e+01 1.02927e-02
328 344 1.00000e+00 2.55071e+01 1.60746e-02
329 345 1.00000e+00 2.54514e+01 3.24633e-02
330 346 1.00000e+00 2.53793e+01 3.88586e-02
331 348 2.13924e-01 2.53624e+01 4.96568e-02
332 349 1.00000e+00 2.52949e+01 2.36512e-02
333 350 1.00000e+00 2.52463e+01 1.91633e-02
334 351 1.00000e+00 2.52241e+01 1.45881e-02
335 352 1.00000e+00 2.52171e+01 1.37720e-02
336 353 1.00000e+00 2.52086e+01 1.20246e-02
337 354 1.00000e+00 2.52038e+01 4.76758e-03
338 355 1.00000e+00 2.52007e+01 3.77340e-03
339 356 1.00000e+00 2.51983e+01 3.87452e-03
340 357 1.00000e+00 2.51980e+01 2.26410e-02
341 358 1.00000e+00 2.51959e+01 1.07287e-02
342 359 1.00000e+00 2.51940e+01 3.67110e-03
343 360 1.00000e+00 2.51909e+01 1.06478e-02
344 361 1.00000e+00 2.51859e+01 1.65802e-02
345 362 1.00000e+00 2.51763e+01 1.80168e-02
346 363 1.00000e+00 2.51552e+01 1.70757e-02
347 364 1.00000e+00 2.51139e+01 2.37655e-02
348 365 1.00000e+00 2.50747e+01 3.73052e-02
349 366 1.00000e+00 2.50021e+01 1.92621e-02
350 367 1.00000e+00 2.49101e+01 3.41625e-02
351 368 1.00000e+00 2.48541e+01 2.62886e-02
352 370 4.01985e-01 2.48240e+01 1.64217e-02
353 371 1.00000e+00 2.47781e+01 1.68553e-02
354 372 1.00000e+00 2.47243e+01 6.66864e-02
355 373 1.00000e+00 2.46615e+01 3.83054e-02
356 374 1.00000e+00 2.46023e+01 2.65901e-02
357 375 1.00000e+00 2.45768e+01 2.31162e-02
358 376 1.00000e+00 2.45608e+01 1.37715e-02
359 377 1.00000e+00 2.45594e+01 2.60802e-02
360 378 1.00000e+00 2.45495e+01 1.48365e-02
361 379 1.00000e+00 2.45447e+01 5.91351e-03
362 380 1.00000e+00 2.45362e+01 1.43631e-02
363 381 1.00000e+00 2.45306e+01 1.98067e-02
364 382 1.00000e+00 2.45242e+01 1.11928e-02
365 383 1.00000e+00 2.45204e+01 1.02427e-02
366 384 1.00000e+00 2.45179e+01 9.44899e-03
367 385 1.00000e+00 2.45168e+01 7.02636e-03
368 386 1.00000e+00 2.45153e+01 5.33594e-03
369 387 1.00000e+00 2.45133e+01 6.13122e-03
370 388 1.00000e+00 2.45105e+01 8.96674e-03
371 389 1.00000e+00 2.45053e+01 8.64318e-03
372 390 1.00000e+00 2.44947e+01 2.11485e-02
373 391 1.00000e+00 2.44797e+01 1.37217e-02
374 392 1.00000e+00 2.44732e+01 2.31176e-02
375 393 1.00000e+00 2.44657e+01 8.57644e-03
376 395 3.10863e-01 2.44625e+01 2.04165e-02
377 396 1.00000e+00 2.44590e+01 6.93848e-03
378 397 1.00000e+00 2.44571e+01 5.46501e-03
379 398 1.00000e+00 2.44555e+01 7.06469e-03
380 399 1.00000e+00 2.44526e+01 5.68369e-03
381 400 1.00000e+00 2.44483e+01 7.55429e-03
382 401 1.00000e+00 2.44388e+01 1.76874e-02
383 402 1.00000e+00 2.44235e+01 3.22140e-02
384 403 1.00000e+00 2.43946e+01 4.66094e-02
385 404 1.00000e+00 2.43517e+01 4.83179e-02
386 405 1.00000e+00 2.42904e+01 4.19260e-02
387 407 5.15645e-01 2.42537e+01 5.32171e-02
388 408 1.00000e+00 2.41914e+01 4.24810e-02
389 409 1.00000e+00 2.41189e+01 2.64852e-02
390 410 1.00000e+00 2.40669e+01 3.00664e-02
391 411 1.00000e+00 2.40352e+01 3.17958e-02
392 412 1.00000e+00 2.39943e+01 4.99683e-02
393 413 1.00000e+00 2.39774e+01 2.24926e-02
394 414 1.00000e+00 2.39702e+01 1.58115e-02
395 415 1.00000e+00 2.39651e+01 7.89845e-03
396 416 1.00000e+00 2.39641e+01 8.85209e-03
397 417 1.00000e+00 2.39622e+01 8.28971e-03
398 418 1.00000e+00 2.39618e+01 8.31722e-03
399 419 1.00000e+00 2.39601e+01 8.75020e-03
400 420 1.00000e+00 2.39593e+01 8.75963e-03
401 422 3.00872e-01 2.39577e+01 2.26824e-02
402 423 1.00000e+00 2.39543e+01 1.83850e-02
403 424 1.00000e+00 2.39466e+01 8.56938e-03
404 425 1.00000e+00 2.39319e+01 1.19659e-02
405 426 1.00000e+00 2.39068e+01 2.86816e-02
406 427 1.00000e+00 2.38684e+01 4.83226e-02
407 428 1.00000e+00 2.38273e+01 3.66681e-02
408 430 5.30141e-01 2.38042e+01 6.20594e-02
409 431 1.00000e+00 2.37729e+01 1.75063e-02
410 432 1.00000e+00 2.37507e+01 1.44546e-02
411 433 1.00000e+00 2.37431e+01 1.35318e-02
412 434 1.00000e+00 2.37416e+01 1.29760e-02
413 435 1.00000e+00 2.37376e+01 3.60228e-03
414 436 1.00000e+00 2.37370e+01 1.97760e-03
415 437 1.00000e+00 2.37367e+01 6.28328e-03
416 438 1.00000e+00 2.37367e+01 8.77765e-03
417 439 1.00000e+00 2.37364e+01 1.71496e-03
418 440 1.00000e+00 2.37364e+01 1.74979e-03
419 441 1.00000e+00 2.37363e+01 1.62166e-03
420 442 1.00000e+00 2.37359e+01 1.92984e-03
421 444 2.15508e-01 2.37356e+01 2.86346e-03
422 445 1.00000e+00 2.37346e+01 2.72545e-03
423 446 1.00000e+00 2.37309e+01 7.44360e-03
424 447 1.00000e+00 2.37235e+01 7.92363e-03
425 448 1.00000e+00 2.37063e+01 1.55918e-02
426 449 1.00000e+00 2.36659e+01 2.24012e-02
427 450 1.00000e+00 2.36130e+01 2.73267e-02
428 452 2.03018e-01 2.36031e+01 2.24518e-02
429 453 1.00000e+00 2.35560e+01 7.11714e-03
430 454 1.00000e+00 2.35352e+01 1.05376e-02
431 455 1.00000e+00 2.35259e+01 4.46291e-03
432 456 1.00000e+00 2.35218e+01 4.93204e-03
433 458 5.11905e-01 2.35207e+01 4.02000e-03
434 459 1.00000e+00 2.35195e+01 4.43380e-03
435 460 1.00000e+00 2.35185e+01 4.50371e-03
436 461 1.00000e+00 2.35175e+01 3.91763e-03
437 462 1.00000e+00 2.35161e+01 3.39180e-03
438 463 1.00000e+00 2.35138e+01 5.82242e-03
439 464 1.00000e+00 2.35084e+01 1.54905e-02
440 465 1.00000e+00 2.34972e+01 2.52770e-02
441 466 1.00000e+00 2.34747e+01 3.24918e-02
442 467 1.00000e+00 2.34366e+01 2.45183e-02
443 468 1.00000e+00 2.33659e+01 3.98665e-02
444 470 1.82663e-01 2.33488e+01 2.17608e-02
445 471 1.00000e+00 2.32616e+01 1.24277e-02
446 472 1.00000e+00 2.31905e+01 1.18941e-02
447 473 1.00000e+00 2.31728e+01 1.88516e-02
448 474 1.00000e+00 2.31560e+01 1.87211e-02
449 475 1.00000e+00 2.31516e+01 5.55212e-03
450 476 1.00000e+00 2.31499e+01 4.16077e-03
451 477 1.00000e+00 2.31487e+01 3.16337e-03
452 478 1.00000e+00 2.31471e+01 2.99031e-03
453 479 1.00000e+00 2.31456e+01 3.00954e-03
454 480 1.00000e+00 2.31443e+01 3.68186e-03
455 481 1.00000e+00 2.31437e+01 7.25491e-03
456 482 1.00000e+00 2.31423e+01 3.43972e-03
457 483 1.00000e+00 2.31412e+01 3.04449e-03
458 484 1.00000e+00 2.31389e+01 5.87739e-03
459 485 1.00000e+00 2.31356e+01 1.05838e-02
460 486 1.00000e+00 2.31320e+01 2.04623e-02
461 487 1.00000e+00 2.31257e+01 1.19428e-02
462 488 1.00000e+00 2.31126e+01 7.76018e-03
463 489 1.00000e+00 2.30948e+01 1.64888e-02
464 490 1.00000e+00 2.30651e+01 2.37466e-02
465 491 1.00000e+00 2.30285e+01 1.71763e-02
466 493 3.78106e-01 2.30130e+01 4.33541e-02
467 494 1.00000e+00 2.29777e+01 1.13696e-02
468 495 1.00000e+00 2.29629e+01 9.33313e-03
469 496 1.00000e+00 2.29600e+01 5.30803e-03
470 498 4.19377e-01 2.29590e+01 3.79786e-03
471 499 1.00000e+00 2.29584e+01 2.48131e-03
472 500 1.00000e+00 2.29575e+01 2.71134e-03
473 502 4.90488e-01 2.29570e+01 4.82526e-03
474 503 1.00000e+00 2.29562e+01 2.50364e-03
475 504 1.00000e+00 2.29555e+01 1.93177e-03
476 505 1.00000e+00 2.29550e+01 2.25900e-03
477 507 1.96156e-01 2.29548e+01 4.27373e-03
478 508 1.00000e+00 2.29544e+01 3.02387e-03
479 509 1.00000e+00 2.29538e+01 3.88725e-03
480 510 1.00000e+00 2.29532e+01 2.72546e-03
481 511 1.00000e+00 2.29519e+01 3.23956e-03
482 512 1.00000e+00 2.29505e+01 2.81186e-03
483 513 1.00000e+00 2.29488e+01 4.71549e-03
484 514 1.00000e+00 2.29455e+01 6.11357e-03
485 515 1.00000e+00 2.29400e+01 8.49084e-03
486 516 1.00000e+00 2.29352e+01 2.27402e-02
487 517 1.00000e+00 2.29165e+01 1.79676e-02
488 518 1.00000e+00 2.28891e+01 1.20347e-02
489 519 1.00000e+00 2.28387e+01 1.37714e-02
490 521 2.42526e-02 2.28363e+01 1.59818e-02
491 522 1.00000e+00 2.28087e+01 1.03203e-02
492 523 1.00000e+00 2.27943e+01 1.01622e-02
493 525 3.51512e-01 2.27865e+01 1.18775e-02
494 526 1.00000e+00 2.27850e+01 1.33197e-02
495 527 1.00000e+00 2.27706e+01 6.48670e-03
496 528 1.00000e+00 2.27584e+01 9.50515e-03
497 529 1.00000e+00 2.27488e+01 3.77761e-03
498 530 1.00000e+00 2.27413e+01 2.82148e-03
499 531 1.00000e+00 2.27387e+01 3.03526e-03
500 532 1.00000e+00 2.27384e+01 1.09332e-02
Reached Maximum Number of Iterations
ans =
250 1
ans =
250 1
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 4.95326e-04 4.14754e+02 1.61957e+03
2 3 1.00000e+00 3.56426e+02 1.67151e+03
3 4 1.00000e+00 3.36207e+02 1.19204e+03
4 5 1.00000e+00 3.06043e+02 3.91437e+02
5 6 1.00000e+00 2.95438e+02 7.53368e+02
6 7 1.00000e+00 2.81971e+02 5.13258e+02
7 8 1.00000e+00 2.58591e+02 5.05433e+02
8 9 1.00000e+00 2.30304e+02 2.67402e+02
9 10 1.00000e+00 2.23343e+02 1.39943e+02
10 11 1.00000e+00 2.19976e+02 1.17064e+02
11 12 1.00000e+00 2.13990e+02 1.53892e+02
12 13 1.00000e+00 2.02760e+02 1.92951e+02
13 14 1.00000e+00 1.90914e+02 1.58021e+02
14 15 1.00000e+00 1.83917e+02 8.03570e+01
15 16 1.00000e+00 1.79728e+02 8.95175e+01
16 17 1.00000e+00 1.76159e+02 1.08320e+02
17 18 1.00000e+00 1.70111e+02 1.21786e+02
18 19 1.00000e+00 1.56963e+02 9.89316e+01
19 20 1.00000e+00 1.46553e+02 1.09969e+02
20 21 1.00000e+00 1.35878e+02 1.47104e+02
21 22 1.00000e+00 1.25182e+02 4.42663e+01
22 23 1.00000e+00 1.18903e+02 6.72439e+01
23 24 1.00000e+00 1.15247e+02 5.73144e+01
24 25 1.00000e+00 1.11739e+02 1.13761e+02
25 26 1.00000e+00 1.09110e+02 8.20114e+01
26 27 1.00000e+00 1.06807e+02 6.08763e+01
27 28 1.00000e+00 1.04876e+02 3.18232e+01
28 29 1.00000e+00 1.04155e+02 3.32192e+01
29 30 1.00000e+00 1.03617e+02 3.44535e+01
30 31 1.00000e+00 1.03045e+02 3.45866e+01
31 32 1.00000e+00 1.01982e+02 3.65316e+01
32 33 1.00000e+00 1.00041e+02 4.01535e+01
33 34 1.00000e+00 9.70583e+01 2.47913e+01
34 35 1.00000e+00 9.47439e+01 1.82661e+01
35 36 1.00000e+00 9.30508e+01 3.67281e+01
36 37 1.00000e+00 9.18658e+01 2.42113e+01
37 38 1.00000e+00 9.10988e+01 2.14335e+01
38 39 1.00000e+00 8.98613e+01 2.06537e+01
39 40 1.00000e+00 8.88808e+01 1.80946e+01
40 41 1.00000e+00 8.77843e+01 3.17394e+01
41 42 1.00000e+00 8.68548e+01 3.18080e+01
42 43 1.00000e+00 8.57145e+01 2.41548e+01
43 44 1.00000e+00 8.48271e+01 1.16200e+01
44 45 1.00000e+00 8.41132e+01 1.91924e+01
45 46 1.00000e+00 8.31875e+01 2.76160e+01
46 47 1.00000e+00 8.19785e+01 3.13311e+01
47 48 1.00000e+00 8.00798e+01 2.67228e+01
48 49 1.00000e+00 7.84550e+01 2.35401e+01
49 50 1.00000e+00 7.76431e+01 1.29071e+01
50 51 1.00000e+00 7.73034e+01 8.45101e+00
51 52 1.00000e+00 7.69885e+01 2.05713e+01
52 53 1.00000e+00 7.64413e+01 3.60059e+01
53 54 1.00000e+00 7.59750e+01 3.20636e+01
54 55 1.00000e+00 7.52687e+01 9.25764e+00
55 56 1.00000e+00 7.47837e+01 1.72501e+01
56 57 1.00000e+00 7.44123e+01 2.24122e+01
57 58 1.00000e+00 7.37758e+01 2.40236e+01
58 59 1.00000e+00 7.27080e+01 3.02029e+01
59 60 1.00000e+00 7.11823e+01 3.49535e+01
60 61 1.00000e+00 6.96410e+01 3.83668e+01
61 62 1.00000e+00 6.81612e+01 1.80875e+01
62 63 1.00000e+00 6.72560e+01 1.93715e+01
63 64 1.00000e+00 6.68913e+01 1.53838e+01
64 65 1.00000e+00 6.65676e+01 3.03215e+01
65 66 1.00000e+00 6.62205e+01 2.78809e+01
66 67 1.00000e+00 6.56664e+01 1.38443e+01
67 68 1.00000e+00 6.50261e+01 2.54167e+01
68 69 1.00000e+00 6.45483e+01 4.68627e+01
69 70 1.00000e+00 6.40170e+01 6.21956e+01
70 71 1.00000e+00 6.34461e+01 6.88977e+01
71 72 1.00000e+00 6.26691e+01 6.80799e+01
72 73 1.00000e+00 6.16516e+01 5.65556e+01
73 74 1.00000e+00 6.05219e+01 2.77899e+01
74 75 1.00000e+00 5.99923e+01 1.70344e+01
75 76 1.00000e+00 5.95684e+01 1.20098e+01
76 77 1.00000e+00 5.91300e+01 1.72347e+01
77 78 1.00000e+00 5.87014e+01 2.69104e+01
78 79 1.00000e+00 5.83724e+01 2.96716e+01
79 80 1.00000e+00 5.80703e+01 2.57501e+01
80 81 1.00000e+00 5.76934e+01 2.30180e+01
81 82 1.00000e+00 5.72863e+01 2.21201e+01
82 83 1.00000e+00 5.66472e+01 2.76538e+01
83 84 1.00000e+00 5.57283e+01 3.52021e+01
84 85 1.00000e+00 5.43995e+01 3.60621e+01
85 86 1.00000e+00 5.28790e+01 2.01920e+01
86 87 1.00000e+00 5.20576e+01 4.38487e+00
87 88 1.00000e+00 5.18123e+01 8.49740e+00
88 89 1.00000e+00 5.17055e+01 8.62655e+00
89 90 1.00000e+00 5.15723e+01 5.77641e+00
90 91 1.00000e+00 5.14626e+01 9.25767e+00
91 92 1.00000e+00 5.13832e+01 1.38009e+01
92 93 1.00000e+00 5.12915e+01 1.59360e+01
93 94 1.00000e+00 5.11971e+01 1.22116e+01
94 95 1.00000e+00 5.10980e+01 7.10873e+00
95 96 1.00000e+00 5.09902e+01 3.40927e+00
96 97 1.00000e+00 5.07901e+01 1.17853e+01
97 98 1.00000e+00 5.04495e+01 2.31557e+01
98 99 1.00000e+00 4.98427e+01 3.57057e+01
99 100 1.00000e+00 4.90216e+01 4.16413e+01
100 101 1.00000e+00 4.81607e+01 3.56512e+01
101 102 1.00000e+00 4.72626e+01 1.45926e+01
102 103 1.00000e+00 4.70846e+01 7.88166e+00
103 104 1.00000e+00 4.69956e+01 4.87191e+00
104 105 1.00000e+00 4.69507e+01 3.22198e+00
105 106 1.00000e+00 4.69162e+01 8.09681e+00
106 107 1.00000e+00 4.68818e+01 2.53300e+00
107 108 1.00000e+00 4.68501e+01 9.43511e+00
108 109 1.00000e+00 4.68011e+01 5.25747e+00
109 110 1.00000e+00 4.66884e+01 6.20122e+00
110 111 1.00000e+00 4.64491e+01 1.24893e+01
111 112 1.00000e+00 4.60793e+01 1.16336e+01
112 113 1.00000e+00 4.53511e+01 1.80182e+01
113 114 1.00000e+00 4.37453e+01 1.34056e+01
114 115 1.00000e+00 4.27735e+01 5.37748e+01
115 116 1.00000e+00 4.23030e+01 1.28953e+01
116 117 1.00000e+00 4.19269e+01 4.22004e+01
117 118 1.00000e+00 4.14477e+01 4.54762e+01
118 119 1.00000e+00 4.13729e+01 2.64740e+01
119 120 1.00000e+00 4.09800e+01 1.34432e+01
120 121 1.00000e+00 4.09307e+01 5.21012e+00
121 122 1.00000e+00 4.09028e+01 3.95834e+00
122 123 1.00000e+00 4.08930e+01 4.32013e+00
123 124 1.00000e+00 4.08865e+01 1.75179e+00
124 125 1.00000e+00 4.08821e+01 1.27368e+00
125 126 1.00000e+00 4.08791e+01 1.39728e+00
126 127 1.00000e+00 4.08753e+01 1.04765e+00
127 128 1.00000e+00 4.08636e+01 3.21494e+00
128 129 1.00000e+00 4.08473e+01 5.38778e+00
129 130 1.00000e+00 4.08141e+01 7.29220e+00
130 131 1.00000e+00 4.07636e+01 7.20268e+00
131 132 1.00000e+00 4.06679e+01 4.23810e+00
132 133 1.00000e+00 4.04925e+01 4.68193e+00
133 134 1.00000e+00 4.01828e+01 1.62878e+01
134 135 1.00000e+00 3.97891e+01 2.49668e+01
135 136 1.00000e+00 3.96011e+01 1.29646e+01
136 137 1.00000e+00 3.95133e+01 3.79964e+00
137 138 1.00000e+00 3.94893e+01 2.74573e+00
138 139 1.00000e+00 3.94797e+01 1.77869e+00
139 140 1.00000e+00 3.94743e+01 1.09483e+00
140 141 1.00000e+00 3.94697e+01 8.09017e-01
141 142 1.00000e+00 3.94664e+01 9.77093e-01
142 143 1.00000e+00 3.94614e+01 1.58935e+00
143 144 1.00000e+00 3.94546e+01 2.03134e+00
144 145 1.00000e+00 3.94450e+01 2.79685e+00
145 146 1.00000e+00 3.94340e+01 2.00357e+00
146 147 1.00000e+00 3.94246e+01 1.86606e+00
147 148 1.00000e+00 3.94157e+01 1.49920e+00
148 149 1.00000e+00 3.94074e+01 2.39104e+00
149 150 1.00000e+00 3.93884e+01 4.68522e+00
150 151 1.00000e+00 3.93676e+01 6.01503e+00
151 152 1.00000e+00 3.92953e+01 8.31609e+00
152 153 1.00000e+00 3.91652e+01 9.28238e+00
153 154 1.00000e+00 3.89409e+01 1.03650e+01
Cubic Backtracking
154 156 3.61701e-01 3.88495e+01 7.66182e+00
155 157 1.00000e+00 3.87578e+01 1.92221e+01
156 158 1.00000e+00 3.85470e+01 2.75533e+00
157 159 1.00000e+00 3.85176e+01 1.82391e+00
158 160 1.00000e+00 3.85088e+01 2.75127e+00
159 161 1.00000e+00 3.85058e+01 1.16158e+00
160 162 1.00000e+00 3.85045e+01 4.08239e-01
161 163 1.00000e+00 3.85039e+01 7.21748e-01
162 164 1.00000e+00 3.85032e+01 1.29159e+00
163 165 1.00000e+00 3.85018e+01 1.76088e+00
164 166 1.00000e+00 3.84993e+01 1.65749e+00
165 167 1.00000e+00 3.84931e+01 4.30330e+00
166 168 1.00000e+00 3.84837e+01 3.57073e+00
167 169 1.00000e+00 3.84565e+01 2.08236e+00
168 170 1.00000e+00 3.84214e+01 2.46173e+00
169 171 1.00000e+00 3.83414e+01 5.29836e+00
170 172 1.00000e+00 3.81998e+01 1.11993e+01
171 173 1.00000e+00 3.79270e+01 9.67111e+00
Cubic Backtracking
172 175 5.14302e-01 3.76695e+01 2.00813e+01
173 176 1.00000e+00 3.72911e+01 8.73439e+00
174 177 1.00000e+00 3.70035e+01 1.13620e+01
175 178 1.00000e+00 3.69409e+01 4.35823e+00
Cubic Backtracking
176 180 4.02733e-01 3.69149e+01 2.22771e+00
Cubic Backtracking
177 182 3.49373e-01 3.69043e+01 1.16683e+00
178 183 1.00000e+00 3.69015e+01 2.39658e+00
179 184 1.00000e+00 3.68991e+01 1.03009e+00
180 185 1.00000e+00 3.68984e+01 4.45865e-01
181 186 1.00000e+00 3.68973e+01 7.50619e-01
182 187 1.00000e+00 3.68952e+01 2.08606e+00
183 188 1.00000e+00 3.68907e+01 4.21239e+00
184 189 1.00000e+00 3.68801e+01 7.49473e+00
185 190 1.00000e+00 3.68546e+01 1.24622e+01
186 191 1.00000e+00 3.67933e+01 1.96201e+01
187 192 1.00000e+00 3.66543e+01 2.87125e+01
188 193 1.00000e+00 3.63857e+01 3.63326e+01
189 194 1.00000e+00 3.60008e+01 3.33977e+01
190 195 1.00000e+00 3.57709e+01 1.80371e+01
191 196 1.00000e+00 3.56986e+01 8.52648e+00
192 197 1.00000e+00 3.55945e+01 1.82507e+00
193 198 1.00000e+00 3.55476e+01 3.25718e+00
194 199 1.00000e+00 3.54968e+01 7.29407e-01
195 200 1.00000e+00 3.54876e+01 4.19090e-01
196 201 1.00000e+00 3.54871e+01 1.09819e+00
197 202 1.00000e+00 3.54827e+01 5.15087e-01
198 203 1.00000e+00 3.54817e+01 4.76339e-01
199 204 1.00000e+00 3.54773e+01 2.15709e+00
200 205 1.00000e+00 3.54744e+01 1.67220e+00
201 206 1.00000e+00 3.54706e+01 7.36873e-01
202 207 1.00000e+00 3.54638e+01 1.19891e+00
203 208 1.00000e+00 3.54518e+01 3.10195e+00
204 209 1.00000e+00 3.54286e+01 5.18506e+00
205 210 1.00000e+00 3.53910e+01 6.44785e+00
206 211 1.00000e+00 3.53421e+01 5.62661e+00
Cubic Backtracking
207 213 4.14287e-01 3.53284e+01 5.85326e+00
208 214 1.00000e+00 3.52998e+01 2.55959e+00
209 215 1.00000e+00 3.52677e+01 1.39932e+00
210 216 1.00000e+00 3.52445e+01 3.04385e+00
211 217 1.00000e+00 3.52106e+01 3.02110e+00
Cubic Backtracking
212 219 2.47601e-01 3.51915e+01 2.70407e+00
213 220 1.00000e+00 3.51649e+01 1.18019e+00
214 221 1.00000e+00 3.51607e+01 1.07630e+00
215 222 1.00000e+00 3.51571e+01 7.31499e-01
216 223 1.00000e+00 3.51521e+01 7.20430e-01
217 224 1.00000e+00 3.51434e+01 3.09821e+00
218 225 1.00000e+00 3.51166e+01 3.98925e+00
Cubic Backtracking
219 227 1.59383e-01 3.51017e+01 1.17152e+01
220 228 1.00000e+00 3.50315e+01 1.12917e+01
221 229 1.00000e+00 3.47430e+01 1.22976e+01
222 230 1.00000e+00 3.45315e+01 1.65496e+01
223 231 1.00000e+00 3.41820e+01 3.29961e+00
224 232 1.00000e+00 3.39614e+01 1.42738e+01
225 233 1.00000e+00 3.38631e+01 6.83530e+00
226 234 1.00000e+00 3.38364e+01 1.68396e+00
227 235 1.00000e+00 3.38233e+01 2.35392e+00
228 236 1.00000e+00 3.38129e+01 3.74178e+00
229 237 1.00000e+00 3.38073e+01 2.67827e+00
230 238 1.00000e+00 3.38020e+01 1.15128e+00
231 239 1.00000e+00 3.37997e+01 5.81783e-01
232 240 1.00000e+00 3.37990e+01 6.06349e-01
233 241 1.00000e+00 3.37983e+01 4.57007e-01
234 242 1.00000e+00 3.37965e+01 6.39988e-01
235 243 1.00000e+00 3.37942e+01 1.61257e+00
236 244 1.00000e+00 3.37886e+01 3.15287e+00
237 245 1.00000e+00 3.37732e+01 5.59916e+00
238 246 1.00000e+00 3.37420e+01 8.02828e+00
239 247 1.00000e+00 3.36645e+01 1.05009e+01
240 248 1.00000e+00 3.35001e+01 1.32090e+01
Cubic Backtracking
241 250 4.68393e-01 3.34188e+01 7.63911e+00
242 251 1.00000e+00 3.31804e+01 5.62168e+00
Cubic Backtracking
243 253 2.66278e-01 3.31469e+01 1.87864e+01
244 254 1.00000e+00 3.28962e+01 7.48853e+00
Cubic Backtracking
245 256 3.04875e-01 3.28361e+01 4.13573e+00
Cubic Backtracking
246 258 2.71917e-01 3.28145e+01 4.35747e+00
Cubic Backtracking
247 260 3.75557e-01 3.27891e+01 1.48042e+00
248 261 1.00000e+00 3.27709e+01 1.24956e+00
249 262 1.00000e+00 3.27597e+01 1.75535e+00
250 263 1.00000e+00 3.27548e+01 1.10692e+00
251 264 1.00000e+00 3.27495e+01 6.48352e-01
252 265 1.00000e+00 3.27387e+01 1.03373e+00
253 266 1.00000e+00 3.27210e+01 1.65052e+00
254 267 1.00000e+00 3.26888e+01 1.95816e+00
Cubic Backtracking
255 269 1.84281e-01 3.26852e+01 2.59428e+00
256 270 1.00000e+00 3.26725e+01 9.19563e-01
257 271 1.00000e+00 3.26679e+01 5.94338e-01
258 272 1.00000e+00 3.26664e+01 2.11847e-01
259 273 1.00000e+00 3.26659e+01 2.80356e-01
260 274 1.00000e+00 3.26656e+01 2.49121e-01
261 275 1.00000e+00 3.26653e+01 2.37447e-01
262 276 1.00000e+00 3.26646e+01 3.59666e-01
263 277 1.00000e+00 3.26632e+01 8.14548e-01
264 278 1.00000e+00 3.26592e+01 1.41390e+00
265 279 1.00000e+00 3.26494e+01 2.45434e+00
266 280 1.00000e+00 3.26284e+01 3.20563e+00
267 281 1.00000e+00 3.25796e+01 6.04733e+00
268 282 1.00000e+00 3.24797e+01 5.35195e+00
Cubic Backtracking
269 284 4.03979e-02 3.24492e+01 5.17246e+00
Halving Step Size
Cubic Backtracking
270 287 2.35678e-02 3.24183e+01 5.02509e+00
Halving Step Size
Cubic Backtracking
271 290 1.06775e-02 3.24026e+01 5.02368e+00
Halving Step Size
Cubic Backtracking
272 293 8.75930e-03 3.23885e+01 4.85792e+00
Halving Step Size
Cubic Backtracking
273 296 8.64847e-03 3.23754e+01 4.82113e+00
Halving Step Size
Cubic Backtracking
274 299 8.48495e-03 3.23626e+01 4.79578e+00
Halving Step Size
Cubic Backtracking
275 302 6.61829e-03 3.23498e+01 4.46895e+00
Halving Step Size
Cubic Backtracking
276 305 7.68426e-03 3.23386e+01 4.43754e+00
Halving Step Size
Cubic Backtracking
277 308 7.81298e-03 3.23274e+01 4.44386e+00
Halving Step Size
Cubic Backtracking
278 311 7.17260e-03 3.23171e+01 4.33599e+00
Halving Step Size
Cubic Backtracking
279 314 7.68182e-03 3.23065e+01 4.32466e+00
Halving Step Size
Cubic Backtracking
280 317 7.71695e-03 3.22943e+01 3.87005e+00
Halving Step Size
Cubic Backtracking
281 320 8.10289e-03 3.22841e+01 3.84298e+00
Cubic Backtracking
282 322 9.79256e-03 3.22724e+01 3.78237e+00
Cubic Backtracking
283 324 1.17060e-02 3.22575e+01 3.74932e+00
Halving Step Size
Cubic Backtracking
284 327 9.68386e-03 3.22429e+01 3.65198e+00
Cubic Backtracking
285 329 1.89846e-02 3.22231e+01 3.56988e+00
Cubic Backtracking
286 331 1.15832e-01 3.21300e+01 2.96703e+00
Cubic Backtracking
287 333 5.09732e-01 3.19430e+01 2.86055e+00
288 334 1.00000e+00 3.19154e+01 4.38107e+00
Cubic Backtracking
289 336 5.89807e-01 3.18386e+01 3.11372e+00
290 337 1.00000e+00 3.18208e+01 1.55565e+00
Cubic Backtracking
291 339 1.66635e-01 3.18086e+01 3.11696e+00
292 340 1.00000e+00 3.17757e+01 3.18632e+00
293 341 1.00000e+00 3.17619e+01 6.40450e-01
294 342 1.00000e+00 3.17557e+01 1.09718e+00
295 343 1.00000e+00 3.17509e+01 8.20982e-01
Cubic Backtracking
296 345 2.92566e-01 3.17495e+01 9.46843e-01
297 346 1.00000e+00 3.17475e+01 3.32476e-01
298 347 1.00000e+00 3.17472e+01 1.76763e-01
299 348 1.00000e+00 3.17471e+01 7.57184e-02
300 349 1.00000e+00 3.17470e+01 5.94572e-02
301 350 1.00000e+00 3.17470e+01 9.44698e-02
302 351 1.00000e+00 3.17468e+01 1.32735e-01
303 352 1.00000e+00 3.17465e+01 1.28331e-01
Cubic Backtracking
304 354 2.42799e-01 3.17463e+01 5.56811e-01
305 355 1.00000e+00 3.17453e+01 3.96231e-01
306 356 1.00000e+00 3.17420e+01 2.76898e-01
307 357 1.00000e+00 3.17346e+01 5.49474e-01
308 358 1.00000e+00 3.17157e+01 1.79599e+00
309 359 1.00000e+00 3.17091e+01 1.77513e+01
310 360 1.00000e+00 3.16568e+01 3.72509e+00
311 361 1.00000e+00 3.16097e+01 5.82722e+00
312 362 1.00000e+00 3.15805e+01 6.63709e+00
Cubic Backtracking
313 364 2.48522e-01 3.15781e+01 4.05377e+00
314 365 1.00000e+00 3.15717e+01 2.82682e-01
315 366 1.00000e+00 3.15705e+01 3.52940e-01
316 367 1.00000e+00 3.15700e+01 7.22268e-01
317 368 1.00000e+00 3.15699e+01 2.00866e-01
318 369 1.00000e+00 3.15698e+01 1.14195e-01
319 370 1.00000e+00 3.15698e+01 1.33759e-01
320 371 1.00000e+00 3.15698e+01 8.59354e-02
321 372 1.00000e+00 3.15697e+01 4.95021e-01
322 373 1.00000e+00 3.15695e+01 1.19025e-01
323 374 1.00000e+00 3.15692e+01 3.98859e-01
324 375 1.00000e+00 3.15682e+01 1.17773e+00
325 376 1.00000e+00 3.15661e+01 2.05229e+00
326 377 1.00000e+00 3.15609e+01 3.29520e+00
327 378 1.00000e+00 3.15491e+01 4.89311e+00
328 379 1.00000e+00 3.15231e+01 6.30076e+00
329 380 1.00000e+00 3.14842e+01 1.51262e+01
330 381 1.00000e+00 3.14234e+01 1.00023e+01
331 382 1.00000e+00 3.13572e+01 2.51240e+00
Cubic Backtracking
332 384 3.74550e-01 3.13385e+01 1.59041e+00
333 385 1.00000e+00 3.13327e+01 4.16123e+00
334 386 1.00000e+00 3.13232e+01 6.35067e-01
335 387 1.00000e+00 3.13205e+01 6.34061e-01
336 388 1.00000e+00 3.13184e+01 6.72524e-01
337 389 1.00000e+00 3.13169e+01 5.32616e-01
338 390 1.00000e+00 3.13136e+01 1.12510e+00
339 391 1.00000e+00 3.13085e+01 8.00750e-01
340 392 1.00000e+00 3.13002e+01 1.18025e+00
341 393 1.00000e+00 3.12971e+01 8.84212e-01
342 394 1.00000e+00 3.12964e+01 1.28956e-01
343 395 1.00000e+00 3.12963e+01 1.16082e-01
344 396 1.00000e+00 3.12962e+01 1.61351e-01
345 397 1.00000e+00 3.12960e+01 4.37006e-01
346 398 1.00000e+00 3.12957e+01 1.88843e-01
347 399 1.00000e+00 3.12948e+01 3.61698e-01
348 400 1.00000e+00 3.12931e+01 4.54863e-01
Cubic Backtracking
349 402 3.59254e-01 3.12920e+01 1.30738e+00
350 403 1.00000e+00 3.12876e+01 1.04277e+00
351 404 1.00000e+00 3.12721e+01 7.02117e-01
352 405 1.00000e+00 3.12406e+01 3.14311e+00
353 406 1.00000e+00 3.11707e+01 4.21645e+00
354 407 1.00000e+00 3.08783e+01 1.32348e+01
355 408 1.00000e+00 3.08000e+01 1.44335e+01
Cubic Backtracking
356 410 8.44908e-02 3.07515e+01 4.60054e+01
Cubic Backtracking
357 412 3.97596e-01 3.03163e+01 6.34538e+00
358 413 1.00000e+00 3.00511e+01 2.74914e+00
359 414 1.00000e+00 2.98543e+01 5.14810e+00
360 415 1.00000e+00 2.98000e+01 6.01401e+00
Cubic Backtracking
361 417 3.34427e-01 2.97790e+01 3.81520e+00
362 418 1.00000e+00 2.97473e+01 1.49947e+00
363 419 1.00000e+00 2.97178e+01 1.31986e+00
364 420 1.00000e+00 2.96823e+01 2.62300e+00
365 421 1.00000e+00 2.96561e+01 1.63761e+00
366 422 1.00000e+00 2.96470e+01 2.36013e+00
367 423 1.00000e+00 2.96440e+01 2.09950e+00
368 424 1.00000e+00 2.96425e+01 6.98349e-01
369 425 1.00000e+00 2.96423e+01 9.86017e-02
370 426 1.00000e+00 2.96421e+01 2.52698e-01
371 427 1.00000e+00 2.96420e+01 2.11325e-01
372 428 1.00000e+00 2.96420e+01 8.16760e-02
373 429 1.00000e+00 2.96419e+01 6.16555e-02
374 430 1.00000e+00 2.96419e+01 9.64174e-02
375 431 1.00000e+00 2.96419e+01 1.22986e-01
376 432 1.00000e+00 2.96418e+01 1.99052e-01
377 433 1.00000e+00 2.96417e+01 1.36791e-01
378 434 1.00000e+00 2.96416e+01 2.52362e-01
379 435 1.00000e+00 2.96414e+01 9.48728e-02
380 436 1.00000e+00 2.96411e+01 1.45948e-01
381 437 1.00000e+00 2.96406e+01 1.88371e-01
Cubic Backtracking
382 439 9.12161e-02 2.96405e+01 4.81745e-01
383 440 1.00000e+00 2.96400e+01 3.32081e-01
384 441 1.00000e+00 2.96393e+01 1.12679e-01
385 442 1.00000e+00 2.96387e+01 2.37074e-01
386 443 1.00000e+00 2.96380e+01 5.39816e-01
387 444 1.00000e+00 2.96368e+01 9.00243e-01
388 445 1.00000e+00 2.96347e+01 1.31573e+00
389 446 1.00000e+00 2.96311e+01 1.64989e+00
390 447 1.00000e+00 2.96242e+01 1.66699e+00
391 448 1.00000e+00 2.96087e+01 1.84037e+00
392 449 1.00000e+00 2.95736e+01 1.38667e+00
393 450 1.00000e+00 2.95105e+01 2.57109e+00
394 451 1.00000e+00 2.93946e+01 3.23500e+00
395 452 1.00000e+00 2.93211e+01 8.28270e+00
396 453 1.00000e+00 2.91742e+01 3.01079e+00
397 454 1.00000e+00 2.91179e+01 2.28757e+00
398 455 1.00000e+00 2.90911e+01 1.38059e+00
399 456 1.00000e+00 2.90835e+01 1.05466e+00
400 457 1.00000e+00 2.90819e+01 1.87667e+00
401 458 1.00000e+00 2.90774e+01 2.80874e-01
402 459 1.00000e+00 2.90764e+01 1.35869e-01
403 460 1.00000e+00 2.90747e+01 4.88821e-01
404 461 1.00000e+00 2.90737e+01 3.76585e-01
Cubic Backtracking
405 463 4.54085e-01 2.90733e+01 9.34373e-01
406 464 1.00000e+00 2.90729e+01 3.66487e-01
407 465 1.00000e+00 2.90728e+01 8.96300e-02
408 466 1.00000e+00 2.90727e+01 5.18939e-02
409 467 1.00000e+00 2.90727e+01 3.05101e-02
410 468 1.00000e+00 2.90727e+01 4.06064e-02
411 469 1.00000e+00 2.90727e+01 1.28930e-01
412 470 1.00000e+00 2.90726e+01 2.13296e-01
413 471 1.00000e+00 2.90726e+01 9.68276e-02
414 472 1.00000e+00 2.90724e+01 1.78460e-01
415 473 1.00000e+00 2.90721e+01 3.04569e-01
416 474 1.00000e+00 2.90715e+01 3.96365e-01
417 475 1.00000e+00 2.90706e+01 9.94284e-01
418 476 1.00000e+00 2.90687e+01 8.11553e-01
419 477 1.00000e+00 2.90638e+01 1.42913e+00
420 478 1.00000e+00 2.90433e+01 4.14509e+00
421 479 1.00000e+00 2.90082e+01 6.91957e+00
422 480 1.00000e+00 2.89322e+01 1.01340e+01
423 481 1.00000e+00 2.87979e+01 1.09846e+01
Cubic Backtracking
424 483 4.57692e-01 2.87151e+01 2.06038e+01
Cubic Backtracking
425 485 5.28357e-01 2.86564e+01 1.46396e+01
Cubic Backtracking
426 487 4.67009e-01 2.85846e+01 1.94174e+01
427 488 1.00000e+00 2.84744e+01 1.21767e+01
428 489 1.00000e+00 2.84041e+01 6.56336e+00
429 490 1.00000e+00 2.83504e+01 7.88567e+00
Cubic Backtracking
430 492 4.97337e-01 2.83242e+01 6.39655e+00
431 493 1.00000e+00 2.83147e+01 1.88986e+00
432 494 1.00000e+00 2.83063e+01 2.67096e+00
433 495 1.00000e+00 2.82998e+01 6.48956e-01
434 496 1.00000e+00 2.82968e+01 8.11180e-01
435 497 1.00000e+00 2.82928e+01 4.19737e-01
436 498 1.00000e+00 2.82911e+01 8.94852e-01
437 499 1.00000e+00 2.82895e+01 7.35128e-01
438 500 1.00000e+00 2.82884e+01 7.86002e-01
439 501 1.00000e+00 2.82882e+01 4.15919e-01
Function Evaluations exceeds maxIter
Test error of kernel multinomial logistic regression: 0.116000 (distance = 0.140000)
Test error of kernel ordinal logistic regression: 0.152000 (distance = 0.176000)
Graphical LASSO
Solve min_{W positive-definite} logdet(sigma+W), s.t. W_ij <= lambda
% Load data load 20news_w100.mat docs = full(documents)'; [nSamples,nVars] = size(docs); mu = mean(docs); centered = docs-repmat(mu,16242,1); sigma = (1/nSamples)*(centered'*centered); % Set up and solve problem lambda = .01; funObj = @(W)logdetFunction(W,sigma); LB = -lambda*ones(nVars); UB = lambda*ones(nVars); W = eye(nVars); W(:) = minConf_TMP(funObj,W(:),LB(:),UB(:)); K = inv(sigma+W); K(abs(W) < lambda) = 0; clf; drawGraph(K~=0,'labels',wordlist); pause
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 3.22718e-04 3.24233e+02 2.51681e+03
2 3 1.00000e+00 3.22501e+02 2.89437e+03
3 4 1.00000e+00 3.21744e+02 2.00445e+03
4 5 1.00000e+00 3.21430e+02 1.67003e+03
5 6 1.00000e+00 3.20927e+02 1.36135e+03
6 7 1.00000e+00 3.20824e+02 1.41744e+03
7 8 1.00000e+00 3.20591e+02 1.07992e+03
8 9 1.00000e+00 3.20494e+02 8.84046e+02
9 10 1.00000e+00 3.20371e+02 7.68824e+02
10 11 1.00000e+00 3.20265e+02 6.64347e+02
Cubic Backtracking
11 13 4.53484e-01 3.20210e+02 6.00605e+02
12 14 1.00000e+00 3.20159e+02 4.70352e+02
13 15 1.00000e+00 3.20136e+02 3.89314e+02
14 16 1.00000e+00 3.20107e+02 3.24526e+02
15 17 1.00000e+00 3.20102e+02 3.33957e+02
16 18 1.00000e+00 3.20084e+02 2.22999e+02
17 19 1.00000e+00 3.20079e+02 1.74439e+02
18 20 1.00000e+00 3.20073e+02 1.54548e+02
19 21 1.00000e+00 3.20068e+02 1.34781e+02
20 22 1.00000e+00 3.20064e+02 1.23284e+02
21 23 1.00000e+00 3.20063e+02 8.72360e+01
22 24 1.00000e+00 3.20061e+02 6.62358e+01
23 25 1.00000e+00 3.20060e+02 6.05389e+01
24 26 1.00000e+00 3.20060e+02 5.08497e+01
25 27 1.00000e+00 3.20059e+02 3.77442e+01
26 28 1.00000e+00 3.20059e+02 3.05205e+01
27 29 1.00000e+00 3.20059e+02 2.48642e+01
28 30 1.00000e+00 3.20059e+02 2.78130e+01
29 31 1.00000e+00 3.20059e+02 1.82057e+01
30 32 1.00000e+00 3.20059e+02 1.41699e+01
31 33 1.00000e+00 3.20059e+02 1.19657e+01
32 34 1.00000e+00 3.20059e+02 1.13690e+01
33 35 1.00000e+00 3.20059e+02 9.48376e+00
34 36 1.00000e+00 3.20059e+02 7.25901e+00
35 37 1.00000e+00 3.20059e+02 6.32951e+00
36 38 1.00000e+00 3.20059e+02 4.71239e+00
Cubic Backtracking
37 40 3.64475e-01 3.20059e+02 4.35473e+00
38 41 1.00000e+00 3.20059e+02 3.39223e+00
39 42 1.00000e+00 3.20059e+02 2.78117e+00
40 43 1.00000e+00 3.20059e+02 2.15063e+00
Cubic Backtracking
41 45 4.45570e-01 3.20059e+02 1.92475e+00
Function value changing by less than optTol
neato - Graphviz version 2.20.2 (Tue Mar 2 19:03:41 UTC 2010)
Associative Conditional Random Fields (trained with pseudo-likelihood)
Optimize pseudo-likelihood in CRF with Ising potentials, subject to constraint that edges are sub-modular (and hence the optimal MAP can be found using graph cuts)
% Load Data load X.mat y = int32(1+X); X = X + randn(size(X))/2; [nRows,nCols] = size(X); nNodes = nRows*nCols; nStates = 2; y = reshape(y,[1 1 nNodes]); X = reshape(X,1,1,nNodes); % Set up problem in UGM adj = latticeAdjMatrix(nRows,nCols); edgeStruct = UGM_makeEdgeStruct(adj,nStates); tied = 1; Xnode = [ones(1,1,nNodes) UGM_standardizeCols(X,tied)]; sharedFeatures = [1 0]; Xedge = UGM_makeEdgeFeaturesInvAbsDif(Xnode,edgeStruct.edgeEnds,sharedFeatures); ising = 1; [nodeMap,edgeMap,w] = UGM_makeCRFmaps(Xnode,Xedge,edgeStruct,ising,tied); nParams = length(w); funObj = @(w)UGM_CRF_PseudoNLL(w,Xnode,Xedge,y,nodeMap,edgeMap,edgeStruct); UB = inf(nParams,1); LB = [-inf;-inf;0;0]; w = minConf_TMP(funObj,w,LB,UB) [nodePot,edgePot] = UGM_CRF_makePotentials(w,Xnode,Xedge,nodeMap,edgeMap,edgeStruct,1); MAP = UGM_Decode_GraphCut(nodePot,edgePot,edgeStruct); imagesc(reshape(X,nRows,nCols));colormap gray figure imagesc(reshape(MAP,nRows,nCols));colormap gray
Iteration FunEvals Step Length Function Val Opt Cond
L-BFGS
1 2 3.09646e-04 1.47402e+02 4.44701e+02
2 3 1.00000e+00 1.27291e+02 3.25422e+02
3 4 1.00000e+00 9.54447e+01 1.55947e+02
4 5 1.00000e+00 7.87347e+01 8.84294e+01
5 6 1.00000e+00 6.43963e+01 4.44543e+01
6 7 1.00000e+00 5.64459e+01 2.15631e+01
7 8 1.00000e+00 5.30260e+01 8.65324e+00
8 9 1.00000e+00 5.21021e+01 3.18210e+00
9 10 1.00000e+00 5.19146e+01 1.13304e+00
10 11 1.00000e+00 5.18572e+01 9.32031e-01
11 12 1.00000e+00 5.18103e+01 1.02205e+00
12 13 1.00000e+00 5.17777e+01 8.63162e-01
13 14 1.00000e+00 5.17662e+01 2.54473e-01
14 15 1.00000e+00 5.17651e+01 1.68942e-02
15 16 1.00000e+00 5.17651e+01 3.46060e-03
Directional Derivative below optTol
w =
-0.7214
-2.1634
1.5427
0.9625
