A class of data-sparse hierarchical matrices (ℋ-matrices) allows to approximate nonlocal (integral) operators with almost linear complexity [22 – 31]. In the present paper, a method is described for an explicitℋ-matrix approximation to the inverse of an elliptic differential operator with piecewise constant coefficients in ℝ d . Our approach is based on the additive splitting to the corresponding Green function, which leads to the sum of an ℋ-matrix and certain correction term including the product of data-sparse matrices of different hierarchical formats. In the case of jumping coefficients with respect to conformal domain decomposition, the approximate inverse operator is obtained as a direct sum of local inverses over subdomains and the Schur complement inverse on the interface. As a by-product, we obtain an explicit approximate inverse preconditioner with the data sparsity inherited from the ℋ-matrix format.