An ICA page - papers, code, demos, links

Explanation: ICA is about factoring probability distributions, and doing blind source separation. It is related to lots of other things - entropy and information maximisation, maximum likelihood density estimation (MLE), EM (expectation maximisation, which is MLE with hidden variables)and projection pursuit. It is basically a way of finding special linear (non-orthogonal) co-ordinate systems in multivariate data, using higher-order statistics in various ways. If you don't understand, read the papers below!

Applications: anywhere you have ensembles of multivariate data, eg: anywhere you might use PCA (Principal Components Analysis). Examples include blind separation (eg: of mixed speech signals), biomedical data processing (eg: of EEG [brainwave] data), finding `features' in data (eg: learning edge-detectors for ensembles of natural images).

Algorithms: There are many different algorithms, but often they are not so different really. Read on.

-Tony Bell

Papers.

A selection of papers that I have access to right now. There is no pretence to completeness, so sorry if you're omitted! Email me if you have something you'd like to include. Many other papers are collected at Paris Smaragdis's site listed below.

Quick guide to the papers:
Our main work is in B1 and B3: infomax/ICA for source separation and image coding respectively. The related natural gradient approach (which we now use) is in A1, a special case of the algorithm proposed in CU. The relations between these algorithms and maximum likelihood (C1, PH) are gone through in C2, MK, PP and O1 (so we'd better believe it!). The infomax origins can be traced in NP. A nice review from the Finnish school appears in KA. D1 is a book containing much material on ICA. Clever extensions and analyses are appearing in G1, PN, T1, T2, and finally PP, where temporal context gives better separation. Perhaps the best work on source separation/deconvolution is in LA (a pun). Our work on EEGs and ERPs is in M1 and M2.

[NOTE: All papers are PostScript file. Compressed versions are X-compressed. They have ".ps.Z" on the end, and need UNIX `uncompress' to make them ".ps" files.]

[A1] Amari S. Cichocki A. and Yang H.H. 1996. A new learning algorithm for blind signal separation, Advances in Neural Information Processing Systems 8, MIT press. Paper

[A2] Amari S-I. 1997. Natural Gradient works efficiently in learning. submitted to Neural Computation Paper

[B1] Bell A.J. and Sejnowski T.J. 1995. An information maximisation approach to blind separation and blind deconvolution, Neural Computation, 7, 6, 1129-1159 Abstract , Paper (0.9MB), Compressed (0.3MB) (38 pages). [3 short conference papers on the same material: ICASSP 95 (1.4MB, 4 pages) , NIPS 94 (0.2MB, 8 pages), and NOLTA95 . ]

[B2] Bell A.J. and Sejnowski T.J. 1996a. Learning the higher-order structure of a natural sound, Network: Computation in Neural Systems, 7 Paper

[B3] Bell A.J. and Sejnowski T.J. 1996. The `Independent Components' of natural scenes are edge filters, to appear in Vision Research, [Please note that this is a draft] Paper (1.5MB) , Compressed (0.4MB) (27 pages).[*Compressed tar-file of figures in case they don't print properly when you print the paper].

[B4] Bell A.J. and Sejnowski T.J. 1996. Edges are the `independent components'of natural scenes,
Advances in Neural Information Processing Systems 9, MIT press. Paper

[C1] Cardoso J-F. and Laheld B. 1996. Equivariant adaptive source separation, IEEE Trans. on Signal Proc., to appear Paper

[C2] Cardoso J-F, 1997. Infomax and maximum likelihood for blind separation, to appear in IEEE Signal Processing Letters, Paper

[D1] Deco G. and Obradovic D. 1996. An information-theoretic approach to neural computing, Springer-verlag Book

[G1] Girolami M. and Fyfe C. 1996. Negentropy and kurtosois as projection pursuit indices provide generalised ICA algorithms. NIPS '96 working paper Paper

[KA] Karhunen J. 1996. Neural approaches to independent component analysis and source separation, Proc 4th European Symposium on Artificial Neural Networks (ESANN '96). Paper

[LA] Lambert R.H. 1996. Multi-channel blind deconvolution: FIR matrix algebra and separation of multipath mixtures, Ph.D. Thesis, Elec. Eng., Univ. of Southern California

[LB] Lee T-W., Bell A.J. and Lambert R. 1997. Blind separation of delayed and convolved sources
in Proceedings of NIPS*9 (1996) Paper

[L2] Lee T-W., Bell A.J. and Orglmeister R. 1997. Blind Separation of Real -World Signals,
in Proceedings of International Conference on Neural Networks (Houston), ICNN '97 Paper

[L3] Lee T-W. Girolami M., Bell A.J. and Sejnowski T.J. 1998. A unifying Information-theoretic framework for Independent Component Analysis. International Journal on Mathematical and Computer Modeling, in press. Paper

[MK] MacKay D. 1996. Maximum Likelihood and covariant algorithms for Independent components analysis, DRAFT 3.1 Paper

[M1] Makeig S., Bell A.J., Jung T-P. and Sejnowski T.J. 1995. Independent Component Analysis of Electroencephalographic Data, in Mozer M. et al (eds) Advances in Neural Information Processing Systems 8, MIT press Paper

[M2] Makeig S., Jung T-P., Bell A.J., Ghahremani D. and Sejnowski T.J. 1997. Blind Separation of Event-related Brain Responses into Independent Components, Proc. Natl. Acad. Sci. USA
to appear Paper

[NP] Nadal J-P. and Parga N. 1994. Non-linear neurons in the low-noise limit: a factorial code maximises information transfer, Network, 4:295-312, Paper

[O1] Olshausen B. 1996. Learning linear, sparse, factorial codes, MIT AI-memo No. 1580 , aper

[PN] Parga N and Nadal J-P. 1996. Blind source separation with time-dependent mixtures, Signal Processing, submitted. Paper

[PP] Pearlmutter B.A. and Parra L.C. 1996. A context-sensitive generalization of ICA, Proc. ICONIP '96, Japan Paper

[PF] Platt J.C. and Faggin F. 1992. Networks for the separation of sources that are superimposed
and delayed, NIPS 1992 Paper

[T1] Torkkola K. 1996a. Blind separation of delayed sources based on information maximisation, Proc. ICASSP, Atlanta, May 1996 Paper

[T2] Torkkola K. 1996b. Blind separation of convolved sources based on information maximisation, Paper

The following two papers are probably under-cited, containing, respectively, `early' theoretical and practical insights into algorithms which are part of the infomax/maximum-likelihood/natural-gradient family of ICA algorithms. Unfortunately we don't have them online. Does anyone else? Also, let us not forget the French originators: Herault/Jutten, Comon and others.

[PH] Pham D.T. Garrat P and Jutten C. 1992. Separation of a mixture of independent sources through a maximum likelihood approach, in Proc. EUSIPCO, p.771-774

[CU] Cichocki A., Unbehauen R., \& Rummert E. 1994. Robust learning algorithm for blind separation of signals, Electronics Letters, 30, 17, 1386-1387

Code.

Basic ICA code in MATLAB (as used in Bell and Sejnowski 1996) It's simple.

Scott Makeig et al's MATLAB 4.2 routines for applying ICA to psychophysiological data (tar-file)

Demos.

Demo of real-room blind separation/deconvolution of two sources

(by Te-Won Lee . Uses infomax/ICA techniques in freq. domain with FIR matrix methods,
pioneered by Russ Lambert at USC. See our NIPS*96 poster - paper [LB] above.)

Links.

Hastily assembled selection of Web pages with more ICA papers and details.
Paris Smaragdis' page has a huge amount of material. It is more-or-less a
superset of this page.

Paris Smaragdis' ICA & BSS page (MIT) GO TO THIS PAGE. IT'S GREAT.

Allan Barros's ICA page in Japan. Another huge collection of stuff.

Jean-François Cardoso

Andrzej Cichocki

Erkki Oja

Scott Makeig,

Shun-ichi Amari

J.-P. Nadal : Publications

Mark Girolami's Home Page

Homepage of Aapo Hyvärinen

NIPS*96 Signal Processing Workshop (several new papers).

The WWW homepage of Juha Karhunen

BSS Research (Experiments at RIKEN in Japan)