In this paper, we present a unified numerical framework for granular modelling. A constitutive model capable of describing both quasi-static and dynamic behaviours of granular material is developed. Two types of particle interactions controlling the mechanical responses, frictional contact and collision, are considered by a hypoplastic model and a Bagnold-type rheology relation, respectively. The model makes no use of concepts like yield stress or flow initiation criterion. A smooth transition between the solid-like and fluid-like behaviour is achieved. The Smoothed Particle Hydrodynamics method is employed as the unified numerical tool for both solid and fluid regimes. The numerical model is validated by simulating element tests under both quasi-static and flowing conditions. We further proceed to study three boundary value problems, i.e. collapse of a granular pile on a flat plane, and granular flows on an inclined plane and in a rotating drum.