Logistic Regression example

[iewaij]

A very primitive draft of Logistic Regression example using Optim.jl, future improvement includes:

• Use real dataset Street et al (1993) or MNIST
• Plot the data and the result
• Show differences between different methods
• Benchmarks against mini-batch gradient descent or reweighted least squares algorithm?

Any suggestions are welcome!

[pkofod]

Cool!

I think you should maybe focus on comparisons within Optim before considering going down other routes (mini-batch GD or IRLS), include run time as well, so highlight differences between methods that needs the hessian and those that do not (and even compared to heuristics as well).

Is this not a real data set btw? (re your initial point)

[iewaij]

The data is from the book ISLR, not sure if it's real, but it seems like the point was stuck at the local minimum. If reset initial value at [-5.0, -5.0, -5.0], the parameters become -4.99982, -4.5396, 0.00012985. I should have passed random initial values.

Okay I will add the benchmarks & traces between different methods, looks quite interesting and can illustrate how different method works.

[iewaij]

After selecting subgroup(non-student) of the data, the local minimum problem is gone and results seem convincing. I will put local minimum problem in method comparison notebook since such issues are common for gradient/hessian-required methods.