This paper extends the deep material network (DMN) proposed by Liu et al. (2019) to tackle general 3-dimensional (3D) problems with arbitrary material and geometric nonlinearities. It discovers a new way of describing multiscale heterogeneous materials by a multi-layer network structure and mechanistic building blocks. The data-driven framework of DMN is discussed in detail about the offline training and online extrapolation stages. Analytical solutions of the 3D building block with a two-layer structure in both small- and finite-strain formulations are derived based on interfacial equilibrium conditions and kinematic constraints. With linear elastic data generated by direct numerical simulations on a representative volume element (RVE), the network can be effectively trained in the offline stage using stochastic gradient descent and advanced model compression algorithms. Efficiency and accuracy of DMN on addressing the long-standing 3D RVE challenges with complex morphologies and material laws are validated through numerical experiments, including 1) hyperelastic particle-reinforced rubber composite with Mullins effect; 2) polycrystalline materials with rate-dependent crystal plasticity; 3) carbon fiber reinforced polymer (CFRP) composites with fiber anisotropic elasticity and matrix plasticity. In particular, we demonstrate a three-scale homogenization procedure of CFRP system by concatenating the microscale and mesoscale material networks. The complete learning and extrapolation procedures of DMN establish a reliable data-driven framework for multiscale material modeling and design.