Calculation of lateral earth pressure on retaining walls is one of the main issues in geotechnics. The upper and lower bound theorems of plasticity are used to analyze the stability of geotechnical structures include bearing capacity of foundations, lateral earth pressure on retaining walls and factor of safety of slopes. In this paper formulation of finite element limit analysis is introduced to determine plastic limit load in the perfect plastic materials. Elements with linear strain rates, which are used in the formulation, cause to eliminate the necessity of velocity discontinuities between the elements. Using non-linear programming based on second order cone programming (SOCP), which has good conformity with cone yield functions such as Mohr-Coulomb and Drucker-Prager, is another important advantage that remove the problem of using ordinary linear programming algorithms for yield functions such as divergent in the apexes. Finally, the optimization problem will be solved by mathematical method. The proposed method is used for calculating active earth pressure on retaining walls in cohesive-frictional soils. According to results of analysis, active earth force on retaining wall is decreased by increasing soil cohesion, wall inclination friction angle between backfill and wall and friction angle of soil wherein the force is increased by increasing surcharge on the backfill and the backfill slope. Mathematical method is used for obtaining accurate results in this research, however, heuristic methods are used when approximate solutions are sufficient.