A set of interesting references

John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, Journal of Machine Learning Research, accepted pending minor revision. [ pdf ]

John Duchi and Yoram Singer. Efficient Online and Batch Learning using Forward Backward Splitting, Journal of Machine Learning Research (JMLR 2009) and Neural Information Processing Systems (NIPS 2009). [ pdf ]

Hu, Kwok, Pan. Accelerated Gradient Methods for. Stochastic Optimization and Online Learning. NIPS 09 [ pdf ].

Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. [ pdf ], [ html ].

Shalev-Shwartz, S., Singer, Y., and Srebro, N. 2007. Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. In Proceedings of the 24th international Conference on Machine Learning (Corvalis, Oregon, June 20 - 24, 2007). Z. Ghahramani, Ed. ICML '07, vol. 227. ACM, New York, NY, 807-814. [ pdf ].