Frequentist: 
is described as unknown and deterministic
Example: Throwing a coin for 100 times, there are 60 times that coin head appears, then %3D%5Ctheta%20%3D%5Cfrac%7B60%7D%7B100%7D%3D0.6)
. When sample data approaches infinity, this method from frequentist is able to give an accurate estimation of . However, if sample data is very scarce, then severe bias could occur. To conclude, more data, more accurate estimation of with frequentist method.
Bayesian: is described as random. There are two inputs: prior )
, likelihood (似然) )
. There is one output: Posterior )
Bayesian estimation is based on Bayesian rule:
%3D%5Cfrac%7BP(X%7C%5Ctheta)%20%5Ctimes%20P(%5Ctheta)%7D%7BP(X)%7D)
Therefore, we have 
. Because, 
as observation (or sample data) is given as a condition in , so %3Dconst)
.
Example: Considering this example of flipping a coin again. %3DP(%5Ctheta%7CX))
is a distribution, instead of a deterministic value of 0.6 in this example. With the increase of sample data, trusts measurement more than prior.
Note: if prior is uniform distribution, then Bayesian method is equal to frequentist method.
Maximum likelihood Estimation (MLE) - frequentist method
A given set of observations, random sample data )
, which is independent and identical distribution. The estimation of using MLE method can be expressed below:
%5C%5C%20%26%3D%20arg%20%5C%20max%20%5C%20P(x_1%3B%5Ctheta)P(x_2%3B%5Ctheta)...P(x_n%3B%5Ctheta)%5C%5C%20%26%3D%20arg%20%5C%20max%20%5Clog%20%5Cprod%20%5Climits_%7Bi%3D1%7D%5En%20P(x_i%3B%5Ctheta)%5C%5C%20%26%3D%20arg%20%5C%20max%20%5Csum_%7Bi%3D1%7D%5En%20%5Clog%20P(x_i%3B%5Ctheta)%5C%5C%20%26%3D%20arg%20%5C%20min%20-%5Csum_%7Bi%3D1%7D%5En%20%5Clog%20P(x_i%3B%5Ctheta)%20%5Cend%7Balign%7D)
The last line of above equation is called Negative Log likelihood (NLL)
Maximum A Posteriori (MAP) - Bayesian method
A given set of observations, random sample data , which is independent and identical distribution. The estimation of using MAP method can be expressed below:
%5C%5C%20%26%3D%20arg%20%5C%20min%20-%20%5Clog%20P(%5Ctheta%7CX)%5C%5C%20%26%3D%20arg%20%5C%20min%20-%20%5Clog%20P(X%7C%5Ctheta)%20-%20%5Clog%20P(%5Ctheta)%20%2B%20log%20P(X)%5C%5C%20%26%3D%20arg%20%5C%20min%20-%20%5Clog%20P(X%7C%5Ctheta)%20-%20%5Clog%20P(%5Ctheta)%20%5Cend%7Balign%7D)
Given that prior is a Gaussian distribution:
%20%3D%20const%20%5Ctimes%20e%5E%7B(-%5Cfrac%7B%5Ctheta%5E2%7D%7B2%5Csigma%5E2%7D)%7D)
Then
%20%3D%20const%20%2B%5Cfrac%7B%5Ctheta%5E2%7D%7B2%5Csigma%5E2%7D)
