Abstract
In this paper we consider the following fractional system F(x,u(x),v(x),Fα(u(x)))=0,G(x,v(x),u(x),Gβ(v(x)))=0,$$\begin{array}{} \displaystyle \left\{ \begin{gathered} F(x,u(x),v(x),{\mathcal{F}_\alpha }(u(x))) = 0,\\ G(x,v(x),u(x),{\mathcal{G}_\beta }(v(x))) = 0, \\ \end{gathered} \right. \end{array}$$ where 0 < α , β < 2, 𝓕 α and 𝓖 β are the fully nonlinear fractional operators: Fα(u(x))=Cn,αPV∫Rnf(u(x)−u(y))x−yn+αdy,Gβ(v(x))=Cn,βPV∫Rng(v(x)−v(y))x−yn+βdy.$$\begin{array}{} \displaystyle {\mathcal{F}_\alpha }(u(x)) = {C_{n,\alpha }}PV\int_{{\mathbb{R}^n}} {\frac{{f(u(x) - u(y))}} {{{{\left| {x - y} \right|}^{n + \alpha }}}}dy} ,\\ \displaystyle{\mathcal{G}_\beta }(v(x)) = {C_{n,\beta }}PV\int_{{\mathbb{R}^n}} {\frac{{g(v(x) - v(y))}} {{{{\left| {x - y} \right|}^{n + \beta }}}}dy} . \end{array}$$ A decay at infinity principle and a narrow region principle for solutions to the system are established. Based on these principles, we prove the radial symmetry and monotonicity of positive solutions to the system in the whole space and a unit ball respectively, and the nonexistence in a half space by generalizing the direct method of moving planes to the nonlinear system.