Discrete Stochastic Processes

Stochastic processes are found in probabilistic systems that evolve with time. Discrete stochastic processes change by only integer time steps (for some time scale), or are characterized by discrete occurrences at arbitrary times. Discrete Stochastic Processes helps the reader develop the understanding and intuition necessary to apply stochastic process theory in engineering, science and operations research. The book approaches the subject via many simple examples which build insight into the structure of stochastic processes and the general effect of these phenomena in real systems.
The book presents mathematical ideas without recourse to measure theory, using only minimal mathematical analysis. In the proofs and explanations, clarity is favored over formal rigor, and simplicity over generality. Numerous examples are given to show how results fail to hold when all the conditions are not satisfied.
Audience: An excellent textbook for a graduate level course in engineering and operations research. Also an invaluable reference for all those requiring a deeper understanding of the subject.

Springer Book Archives

1 Introduction and Probability Review.- 1.1 Introduction.- 1.2 Probability Review.- 1.3 Conditional Probabilities.- 1.4 Random Variables.- 1.5 Expectations.- 1.6 Transforms.- 1.7 Weak Law of Large Numbers.- 1.8 Strong Law of Large Numbers.- 1.9 Summary.- Table of Standard Random Variables.- Exercises.- Notes.- 2 Poisson Processes.- 2.1 Introduction.- 2.2 Definition and Properties of the Poisson Process.- 2.3 Combinations and Subdivisions of Independent Poisson Processes.- 2.4 Non-Homogeneous Poisson Processes.- 2.5 Order Statistics and Conditional Arrival Epochs.- 2.6 Summary.- Exercises.- Notes.- 3 Renewal Processes.- 3.1 Introduction.- 3.2 Strong Law of Large Numbers for Renewal Processes.- 3.3 Expected Number of Renewals.- 3.4 Renewal Reward Processes; Time Averages.- 3.5 Renewal Reward Processes; Ensemble Averages.- 3.6 Applications of Renewal Reward Theory.- 3.7 Delayed Renewal Processes.- 3.8 Summary.- Exercises.- Notes.- 4 Finite State Markov Chains.- 4.1 Introduction.- 4.2 Classification of States.- 4.3 The Matrix Representation.- 4.4 Perron-Frobenius Theory.- 4.5 Markov Chains with Rewards.- 4.6 Markov Decision Theory and Dynamic Programming.- 4.7 Summary.- Exercises.- Notes.- 5 Markov Chains with Countably Infinite State Spaces.- 5.1 Introduction and Classification of States.- 5.2 Branching Processes.- 5.3 Birth Death Markov Chains.- 5.4 Reversible Markov Chains.- 5.5 The M/M/1 Sampled Time Markov Chain.- 5.6 Round-Robin and Processor Sharing.- 5.7 Semi-Markov Processes.- 5.8 Example-M/G/1 Queue.- 5.9 Summary.- Exercises.- 6 Markov Processes with Countable State Spaces.- 6.1 Introduction.- 6.2 The Kolmogorov Differential Equations.- 6.3 Uniformization.- 6.4 Birth Death Processes.- 6.5 Reversibility for Markov Processes.- 6.6 Jackson Networks.- ClosedJackson Networks.- 6.7 Summary.- Exercises.- 7 Random Walks and Martingales.- 7.1 Introduction.- 7.2 The G/G/1 Queue.- 7.3 Detection, Decisions, and Hypothesis Testing.- 7.4 Threshold Crossing Probabilities.- 7.5 Wald's Identity and Walks with Two Thresholds.- 7.6 Martingales and Submartingales.- 7.7 Stopped Processes and Stopping Rules.- 7.8 The Kolmogorov Inequalities.- 7.9 Summary.- Exercises.- Notes.