You are previewing An Introduction to Sparse Stochastic Processes.
O'Reilly logo
An Introduction to Sparse Stochastic Processes

Book Description

Providing a novel approach to sparse stochastic processes, this comprehensive book presents the theory of stochastic processes that are ruled by stochastic differential equations, and that admit a parsimonious representation in a matched wavelet-like basis. Two key themes are the statistical property of infinite divisibility, which leads to two distinct types of behaviour - Gaussian and sparse - and the structural link between linear stochastic processes and spline functions, which is exploited to simplify the mathematical analysis. The core of the book is devoted to investigating sparse processes, including a complete description of their transform-domain statistics. The final part develops practical signal-processing algorithms that are based on these models, with special emphasis on biomedical image reconstruction. This is an ideal reference for graduate students and researchers with an interest in signal/image processing, compressed sensing, approximation theory, machine learning, or statistics.

Table of Contents

  1. Coverpage
  2. Half title page
  3. Endorsement page
  4. Title page
  5. Copyright page
  6. Dedication
  7. Contents
  8. Preface
  9. Notation
  10. 1 Introduction
    1. 1.1 Sparsity: Occam’s razor of modern signal processing?
    2. 1.2 Sparse stochastic models: the step beyond Gaussianity
    3. 1.3 From splines to stochastic processes, or when Schoenberg meets Lévy
      1. 1.3.1 Splines and Legos revisited
      2. 1.3.2 Higher-degree polynomial splines
      3. 1.3.3 Random splines, innovations, and Lévy processes
      4. 1.3.4 Wavelet analysis of Lévy processes and M-term approximations
      5. 1.3.5 Lévy’s wavelet-based synthesis of Brownian motion
    4. 1.4 Historical notes: Paul Lévy and his legacy
  11. 2 Roadmap to the book
    1. 2.1 On the implications of the innovation model
      1. 2.1.1 Linear combination of sampled values
      2. 2.1.2 Wavelet analysis
    2. 2.2 Organization of the book
  12. 3 Mathematical context and background
    1. 3.1 Some classes of function spaces
      1. 3.1.1 About the notation: mathematics vs. engineering
      2. 3.1.2 Normed spaces
      3. 3.1.3 Nuclear spaces
    2. 3.2 Dual spaces and adjoint operators
      1. 3.2.1 The dual of L[sub(p)] spaces
      2. 3.2.2 The duals of D and S
      3. 3.2.3 Distinction between Hermitian and duality products
    3. 3.3 Generalized functions
      1. 3.3.1 Intuition and definition
      2. 3.3.2 Operations on generalized functions
      3. 3.3.3 The Fourier transform of generalized functions
      4. 3.3.4 The kernel theorem
      5. 3.3.5 Linear shift-invariant operators and convolutions
      6. 3.3.6 Convolution operators on L[sub(p)](R[sup(d)])
    4. 3.4 Probability theory
      1. 3.4.1 Probability measures
      2. 3.4.2 Joint probabilities and independence
      3. 3.4.3 Characteristic functions in finite dimensions
      4. 3.4.4 Characteristic functionals in infinite dimensions
    5. 3.5 Generalized random processes and fields
      1. 3.5.1 Generalized random processes as collections of random variables
      2. 3.5.2 Generalized random processes as random generalized functions
      3. 3.5.3 Determination of statistics from the characteristic functional
      4. 3.5.4 Operations on generalized stochastic processes
      5. 3.5.5 Innovation processes
      6. 3.5.6 Example: filtered white Gaussian noise
    6. 3.6 Bibliographical pointers and historical notes
  13. 4 Continuous-domain innovation models
    1. 4.1 Introduction: from Gaussian to sparse probability distributions
    2. 4.2 Lévy exponents and infinitely divisible distributions
      1. 4.2.1 Canonical Lévy–Khintchine representation
      2. 4.2.2 Deciphering the Lévy–Khintchine formula
      3. 4.2.3 Gaussian vs. sparse categorization
      4. 4.2.4 Proofs of Theorems 4.1 and 4.2
    3. 4.3 Finite-dimensional innovation model
    4. 4.4 White Lévy noises or innovations
      1. 4.4.1 Specification of white noise in Schwartz’ space S′
      2. 4.4.2 Impulsive Poisson noise
      3. 4.4.3 Properties of white noise
    5. 4.5 Generalized stochastic processes and linear models
      1. 4.5.1 Innovation models
      2. 4.5.2 Existence and characterization of the solution
    6. 4.6 Bibliographical notes
  14. 5 Operators and their inverses
    1. 5.1 Introductory example: first-order differential equation
    2. 5.2 Shift-invariant inverse operators
    3. 5.3 Stable differential systems in 1-D
      1. 5.3.1 First-order differential operators with stable inverses
      2. 5.3.2 Higher-order differential operators with stable inverses
    4. 5.4 Unstable Nth-order differential systems
      1. 5.4.1 First-order differential operators with unstable shift-invariant inverses
      2. 5.4.2 Higher-order differential operators with unstable shift-invariant inverses
      3. 5.4.3 Generalized boundary conditions
    5. 5.5 Fractional-order operators
      1. 5.5.1 Fractional derivatives in one dimension
      2. 5.5.2 Fractional Laplacians
      3. 5.5.3 L[sub(p)]-stable inverses
    6. 5.6 Discrete convolution operators
    7. 5.7 Bibliographical notes
  15. 6 Splines and wavelets
    1. 6.1 From Legos to wavelets
    2. 6.2 Basic concepts and definitions
      1. 6.2.1 Spline-admissible operators
      2. 6.2.2 Splines and operators
      3. 6.2.3 Riesz bases
      4. 6.2.4 Admissible wavelets
    3. 6.3 First-order exponential B-splines and wavelets
      1. 6.3.1 B-spline construction
      2. 6.3.2 Interpolator in augmented-order spline space
      3. 6.3.3 Differential wavelets
    4. 6.4 Generalized B-spline basis
      1. 6.4.1 B-spline properties
      2. 6.4.2 B-spline factorization
      3. 6.4.3 Polynomial B-splines
      4. 6.4.4 Exponential B-splines
      5. 6.4.5 Fractional B-splines
      6. 6.4.6 Additional brands of univariate B-splines
      7. 6.4.7 Multidimensional B-splines
    5. 6.5 Generalized operator-like wavelets
      1. 6.5.1 Multiresolution analysis of L[sub(2)](R[sup(d)])
      2. 6.5.2 Multiresolution B-splines and the two-scale relation
      3. 6.5.3 Construction of an operator-like wavelet basis
    6. 6.6 Bibliographical notes
  16. 7 Sparse stochastic processes
    1. 7.1 Introductory example: non-Gaussian AR(1) processes
    2. 7.2 General abstract characterization
    3. 7.3 Non-Gaussian stationary processes
      1. 7.3.1 Autocorrelation function and power spectrum
      2. 7.3.2 Generalized increment process
      3. 7.3.3 Generalized stationary Gaussian processes
      4. 7.3.4 CARMA processes
    4. 7.4 Lévy processes and their higher-order extensions
      1. 7.4.1 Lévy processes
      2. 7.4.2 Higher-order extensions of Lévy processes
      3. 7.4.3 Non-stationary Lévy correlations
      4. 7.4.4 Removal of long-range dependencies
      5. 7.4.5 Examples of sparse processes
      6. 7.4.6 Mixed processes
    5. 7.5 Self-similar processes
      1. 7.5.1 Stable fractal processes
      2. 7.5.2 Fractional Brownian motion through the looking-glass
      3. 7.5.3 Scale-invariant Poisson processes
    6. 7.6 Bibliographical notes
  17. 8 Sparse representations
    1. 8.1 Decoupling of Lévy processes: finite differences vs. wavelets
    2. 8.2 Extended theoretical framework
      1. 8.2.1 Discretization mechanism: sampling vs. projections
      2. 8.2.2 Analysis of white noise with non-smooth functions
    3. 8.3 Generalized increments for the decoupling of sample values
      1. 8.3.1 First-order statistical characterization
      2. 8.3.2 Higher-order statistical dependencies
      3. 8.3.3 Generalized increments and stochastic difference equations
      4. 8.3.4 Discrete whitening filter
      5. 8.3.5 Robust localization
    4. 8.4 Wavelet analysis
      1. 8.4.1 Wavelet-domain statistics
      2. 8.4.2 Higher-order wavelet dependencies and cumulants
    5. 8.5 Optimal representation of Lévy and AR(1) processes
      1. 8.5.1 Generalized increments and first-order linear prediction
      2. 8.5.2 Vector-matrix formulation
      3. 8.5.3 Transform-domain statistics
      4. 8.5.4 Comparison of orthogonal transforms
    6. 8.6 Bibliographical notes
  18. 9 Infinite divisibility and transform-domain statistics
    1. 9.1 Composition of id laws, spectral mixing, and analysis of white noise
    2. 9.2 Class C and unimodality
    3. 9.3 Self-decomposable distributions
    4. 9.4 Stable distributions
    5. 9.5 Rate of decay
    6. 9.6 Lévy exponents and cumulants
    7. 9.7 Semigroup property
      1. 9.7.1 Gaussian case
      2. 9.7.2 SαS case
      3. 9.7.3 Compound-Poisson case
      4. 9.7.4 General iterated-convolution interpretation
    8. 9.8 Multiscale analysis
      1. 9.8.1 Scale evolution of the pdf
      2. 9.8.2 Scale evolution of the moments
      3. 9.8.3 Asymptotic convergence to a Gaussian/stable distribution
    9. 9.9 Notes and pointers to the literature
  19. 10 Recovery of sparse signals
    1. 10.1 Discretization of linear inverse problems
      1. 10.1.1 Shift-invariant reconstruction subspace
      2. 10.1.2 Finite-dimensional formulation
    2. 10.2 MAP estimation and regularization
      1. 10.2.1 Potential function
      2. 10.2.2 LMMSE/Gaussian solution
      3. 10.2.3 Proximal operators
      4. 10.2.4 MAP estimation
    3. 10.3 MAP reconstruction of biomedical images
      1. 10.3.1 Scale-invariant image model and common numerical setup
      2. 10.3.2 Deconvolution of fluorescence micrographs
      3. 10.3.3 Magnetic resonance imaging
      4. 10.3.4 X-ray tomography
      5. 10.3.5 Discussion
    4. 10.4 The quest for the minimum-error solution
      1. 10.4.1 MMSE estimators for first-order processes
      2. 10.4.2 Direct solution by belief propagation
      3. 10.4.3 MMSE vs. MAP denoising of Lévy processes
    5. 10.5 Bibliographical notes
  20. 11 Wavelet-domain methods
    1. 11.1 Discretization of inverse problems in a wavelet basis
      1. 11.1.1 Specification of wavelet-domain MAP estimator
      2. 11.1.2 Evolution of the potential function across scales
    2. 11.2 Wavelet-based methods for solving linear inverse problems
      1. 11.2.1 Preliminaries
      2. 11.2.2 Iterative shrinkage/thresholding algorithm
      3. 11.2.3 Fast iterative shrinkage/thresholding algorithm
      4. 11.2.4 Discussion of wavelet-based image reconstruction
    3. 11.3 Study of wavelet-domain shrinkage estimators
      1. 11.3.1 Pointwise MAP estimators for AWGN
      2. 11.3.2 Pointwise MMSE estimators for AWGN
      3. 11.3.3 Comparison of shrinkage functions: MAP vs. MMSE
      4. 11.3.4 Conclusion on simple wavelet-domain shrinkage estimators
    4. 11.4 Improved denoising by consistent cycle spinning
      1. 11.4.1 First-order wavelets: design and implementation
      2. 11.4.2 From wavelet bases to tight wavelet frames
      3. 11.4.3 Iterative MAP denoising
      4. 11.4.4 Iterative MMSE denoising
    5. 11.5 Bibliographical notes
  21. 12 Conclusion
  22. Appendix A Singular integrals
    1. A.1 Regularization of singular integrals by analytic continuation
    2. A.2 Fourier transform of homogeneous distributions
    3. A.3 Hadamard’s finite part
    4. A.4 Some convolution integrals with singular kernels
  23. Appendix B Positive definiteness
    1. B.1 Positive definiteness and Bochner’s theorem
    2. B.2 Conditionally positive–definite functions
    3. B.3 Lévy–Khintchine formula from the point of view of generalized functions
  24. Appendix C Special functions
    1. C.1 Modified Bessel functions
    2. C.2 Gamma function
    3. C.3 Symmetric-alpha-stable distributions
  25. References
  26. Index