A microstructure-based finite element analysis model was developed to predict the effective elastic property of cellulose nanowhisker reinforced all-cellulose composite. Analysis was based on the microstructure synthesized with assumption on volume fraction, size, and orientation distribution of cellulose nanowhiskers. Simulation results demonstrated some interesting discovery: With the increase of aspect ratio, the effective elastic modulus increases in isotropic microstructure. The elastic property anisotropy increases with the aspect ratio and anisotropy of nanowhisker orientation. Simulation results from microstructure-based finite element analysis agree well with experimental results, comparing with other homogenization methods: upper bound, lower bound, and self-consistent models. Capturing the anisotropic elastic property, the microstructure-based finite element analysis demonstrated the capability in guiding materials design to improve effective properties.