May, 2022 - François HU
Master of Science - EPITA
This lecture is available here: https://curiousml.github.io/
Unconstrained optimization in dimension $n\geq2$
Let $f:\mathbb{R}^n \to\mathbb{R}$ be a real valued function of $n$ variables
We know that a minimum $x^*$ verifies $\nabla f(x^*) = 0$. We can therefore try to to solve the equation $\nabla f(x) = 0$ by Newton method.
We can also approximate $f$ by $$ f(x+h) \approx f(x) + \nabla f(x)h + \dfrac{1}{2}h^TH_f(x)h $$ and minimise the quadratic approximation as a function of $h$
In both cases, we obtain the iteration, $$ x_{k+1} = x_k - H_f^{-1}(x_k)\nabla f(x_k) $$
We do not explicitly calculate the inverse of the Hessian. Instead, we solve the linear system $$ x_{k+1} = x_k + s_k $$
The convergence of Newton's method is quadratic provided you start the iteration close enough to the result.
Principle
Optimization, find: $x^*\in\arg\max_{x} S(x)$
for t = 0:
we sample $Y_1, \dots, Y_n \sim \mathcal{N}(\mu_t, \sigma_t^2)$
we choose the best $10\%$ $Y_i$ that maximize $S$
we estimate (by MLE) $\mu_{t+1}$ and $\sigma_{t+1}^2$