Qualitative analysis for the nonlinear fractional Hartree type system with nonlocal interaction

: In the present paper we study the existence of nontrivial solutions of a class of static coupled nonlinear fractional Hartree type system. First, we use the direct moving plane methods to establish the maximum principle(Decay at infinity and Narrow region principle) and prove the symmetry and nonexistence of positive solution of this nonlocal system. Second, we make complete classification of positive solutions of the system in the critical case when some parameters are equal. Finally, we prove the existence of multiple nontrivial solutions in the critical case according to the different parameters ranges by using variational methods. To accomplish our results we establish the maximum principle for the fractional nonlocal system.

One of the motivation to study the system (1.1) is motivated by recent studies on the nonlinear fractional Schrödinger equation with nonlocal interaction If α = 2, N = 3 and p = 2, the system (1.5) is related to the condensate in the mean field regime. Indeed, concerning the movement of the identical and non-relativistic basic particles (such as bosons or electrons), under the influence of an external potential, the interaction between two particles is governed by the nonlinear nonlocal Hatree equation(see [17,18,23,24]). Here the function ψ is a radially symmetric two-body potential function defined and * denotes the convolution in R 3 . The most typical external potential is the Coulomb function C(x) = |x| −1 . On the other hand, the system (1.5) is also used in the description of the Bose-Einstein condensates, in which V is the trapping potential and the nonlocal interaction also describes the interaction between the bosons in the condensate [14,44,47]. When V = 0, (1.5) is also known as nonlinear Choquard equation [33,37,40], and the equation (1.5) with V = 0 also arises from the model of wave propagation in a media with a large response length [1,26]. Many papers considered the general interaction case.
That is, If α = 2, the paper [41] proved the existence and some properties of solution of (1.7). Recently, the paper [20] prove the existence of nodal solution of (1.7). For more general information one can refer to the papers [19,20,30,31,40,42,46] and references therein. For the fractional case 0 < α < 2, the paper [12] proved the regularity and classification of the solution of (1.7). Recently, by using the direct moving plane methods, the paper [11] make the classification of the positive solution of (1.7) when p = 2 and N − γ = 2α. The paper [28] make the classification of the positive solution of (1.7) for the general case.
The system (1.1) was studied in the sevral recent papers [51][52][53][54]56] for the case α = 2. This kind of systems are considered in the basic quantum chemistry model of small number of electrons interacting with static nucleii which can be approximated by Hartree or Hartree-Fock minimization problems (see [29,35,39]). In fact, the Euler-Lagrange equations corresponding to such Hartree problem are ( 1 . 8 ) where k ∈ N, V(x) describes the attractive interaction between the electrons and the nucleii, the integral term shows the repulsive Coulomb interaction between the electrons, and −ε i are the Lagrange multipliers.
Following the discussion in [39], we usually consider the case that some components are set to be equal in (1.8). For example, when k = 2 and u 1 = u 2 , then (1.8) is reduced to a scalar equation ( 1 . 9 ) The solutions of (1.9) were considered in, for example, [19,38,39]. We notice that in (1.8), the interaction between electrons is repulsive while the one in (1.5) is attractive. On the other hand, if k = 4, u 1 = u 2 and u 3 = u 4 , then we can also obtain (1.1) when V = 0, p = q = 2, N = 3 and γ = α = 2. Recent years, the following nonlocal system −ε 2 Δu + λ 1 (x)u = μ 1 ϕu|u| p−2 u + βϕv|u| p−2 u, x ∈ R N , (1.10) has been studied by many literatures, where ϕu(x) = R N C(x − y)|u(y)| p dy and C(x) is given in (1.6). Note that the system (1.10)(or (1.1)) has two semitrivial solutions (u, 0) and (0, v), where u, v are solutions of (1.7). In order to make clear statement one gives the following definitions. We say (u, v) is a nontrivial solution of (1.10)(or (1.1)) if u ≠ 0 and v ≠ 0. If u, v > 0, we say (u, v) is a positive solution of (1.10)(or (1.1)). A solution is called a nontrivial ground state solution(or positive ground state solution) if its energy is minimal among all the nontrivial solutions(or all the positive solutions) of (1.10)(or (1.1)). The paper [56] considered the semiclassical case. Under some conditions for the potential function λ i (x), i = 1, 2, the existence of a ground state solution of (1.10) for ε > 0 small and β > 0 sufficiently large was proved. Later, the paper [53] studied the case λ i (x) = λ i = constant, ε = 1, p = 2, N = 3 and γ = 2. The authors proved the existence and nonexistence of positive ground state solutions of (1.10). Moreover, various qualitative properties of ground state solutions are also obtained. Recently, the papers [51,54] studied the existence and properties of normolized solution of (1.10) with general interaction. Very recently, the paper [52] proved the existence of nontrivial solutions of the more general nonlocal interaction case of (1.10). Motivated by the previous works, in the present paper we shall study the general case of the system (1.10) in fractional situation. Precisely, the main purpose of this paper is the following three parts. First, we use the direct moving plane methods to establish the maximum principle(Decay at infinity and Narrow region principle) and prove the symmetry and nonexistence of positive solution of this nonlocal system. Second, we make complete classification of positive solutions of the system for the critical case when μ 1 = μ 2 . Finally, we prove the existence of multiple nontrivial solutions of the critical case for μ 1 ≠ μ 2 by using variational methods. To accomplish the first two conclusions, we shall use a variant (for nonlocal nonlinearity) of the direct method of moving planes for fractional Laplacians due to the paper [6,7] to obtain symmetry, monotonicity, nonexistence and classification of the positive solutions to (1.1). The methods of moving planes was initially invented by Alexanderoff in the early 1950s. Later, it was further developed by Serrin [48], Gidas, Ni and Nirenberg [21,22], Caffarelli, Gidas and Spruck [2], Chen and Li [4], Li and Zhu [32] and many others. For more literatures on this direction, one can see the papers [4,7,8,13,16,32,36,55] and the references therein. In this paper we establish the maximum principle(Decay at infinity and Narrow region principle, see Theorems 2.6-2.7 below) for the fractional nonlocal system. This is new and we believe that it will be useful to study the other nonlocal problems. To accomplish the third result, we should make careful study the uniqueness of the synchronous solutions of (1.1) and use the perturbation methods to obtain the nontrivial ground state solution of (1.1).
Then we first have the following main results for the subcritical and critical cases. Theorem 1.1. Assume that N ≥ 2, 0 < α < 2, 0 < γ < N and μ 1 , μ 2 , β > 0. Then the system (1.1) has no positive solution for 1 ≤ p < N+γ N−α . If p = N+γ N−α and μ 1 = μ 2 , then every positive solution (u, v) of (1.1) must has the following form u From the results of Theorem 1.1, we have the following corollary. Corollary 1.2. If u = v, μ 1 = μ 2 = 1 and p = N+γ N−α , then the following critical Hartree type equation has a unique positive solution of the form for some μ > 0 and x 0 ∈ R N . (1.12) Remark 1.3. The result of Corollary 1.2 has been obtained recently by [11,28].

Main conclusions from the direct methods of moving planes
Throughout the paper, we use the following notations: -Let c > 0 be an arbitrary constants. In this section we shall establish the main maximum principle theorems(Decay at infinity and Narrow region principle) for the fractional nonlocal system by using the direct methods of moving planes. We first recall the following classical Hardy-Littlewood-Sobolev inequality(see [34,Theorem 4.3]) and Cauchy-Schwarz inequality(see [34,Theorem 5.9] or [20,Inequality (3.3)]) for nonlocal problem. Lemma 2.1. (i) Assume that f ∈ L p1 (R N ) and g ∈ L q1 (R N ). Then one has where γ ∈ (0, N) and p ≥ 1.
For 0 < α < 2, we define which is endowed with the natural norm From [15, Theorem 6.5], we know that the embedding H α (R N ) → L q (R N )(∀q ∈ [2, 2 * α ]) and 2 * α = 2N N−α (N ≥ 3). From Hardy-Littlewood-Sobolev inequality Lemma 2.1 (i) and (iii), we know that if u ∈ H α (R N ), then the nonlinearity of the system (1.1) belongs to L p (R N ) for each 1 ≤ p ≤ N+γ N−α . Hence if 1 ≤ p < N+γ N−α , we call it a subcritical case in (1.1). If p = N+γ N−α , we say it a critical case in (1.1). Next we shall study the properties of the positive solution (u, v) of (1.1). To describe the asymptotic behavior of u and v at infinity, we introduce the Kelvin transformũ,ṽ of u, v center at 0 defined bỹ for each x ∈ R N \ {0}. It's obvious that the Kelvin transformũ,ṽ may have singularity at 0. Moreover we have lim |x|→∞ |x| N−αũ (x) = u(0) > 0 and lim |x|→∞ |x| N−αṽ (x) = u(0) > 0. We infer from the definition (2.3) and This conclusion holds similarly forṽ. Now we are ready to calculate the system for (ũ,ṽ). That is, one infers from (1.3) that Thus, we know that (ũ,ṽ) is the positive solution of where m := α + γ − p(N − α). Thus, we see that if m > 0, then 1 ≤ p < N+γ N−α (subcritical case). If m = 0, then p = N+γ N−α (critical case). Before going further, we need the following maximum principle for α 2 -superharmonic functions and Liouville theorem for α 2 -harmonic functions. For the details of the proof one can refer to the papers [7,49,57].
In addition, these conclusions hold for unbounded region Ω if we further assume that Then u ≡ C ≥ 0.
. If u is nonnegative solution of (1.1), then u satisfies the integral system (1.4), and vice versa. [27]). For each R > 0, we define (2.6) By using the properties of Green's function, we can derive that (w R (x), z R (x)) satisfies One deduces from the Maximum principle(Lemma 2.2) that foe any R > 0 For each fixed x ∈ R N , we have that Since μ i , β ≥ 0 and μ i + β > 0(i = 1, 2), we know that (2.14) This implies that C 0 = C 1 = 0. Thus, we arrive at (1.4). Conversely, we follow the idea of [5,8,57] is also a solution of (1.1).
The next lemma state the basic properties for the Kelvin transformũ,ṽ ∈ C 1,1 Proof. A direct computation shows that Similarly, we can prove the second equality in (2.15). Next we shall prove (2.16). As in [11,Ineqaulity (2.19)](or [28] Then we know that ψ(x) satisfies(see [5,8,57]) We infer from the properties of χ, μ 1 + β > 0 and (2.18) that where C 2 N,α > 0 is a positive constant depending on N and α. This implies that We infer from Lemma 2.2 that h R ≥ 0 in R N for each R > 0 large. Particularly, we have For arbitrarily fixed x, letting R → ∞ in (2.23), we get u ≥ ψ in R N . Hence we get that for |x| large enough. Thus, we deduce from (2.24) that Similarly, we can prove lim inf |x|→0ṽ (x) > 0. This finishes the proof.
Now we are ready to use the method of moving planes. For each λ ∈ R N , we define be the reflection of x about the plane T λ , and define We first need to prove that for λ sufficiently negative Then we can start moving plane T λ from near x 1 = −∞ to the right as long as (2.28) holds, until its limiting position and finally derive the symmetry. To accomplish this we set Then we have the following conclusion. Lemma 2.5. Assume that λ ≤ 0. Then we have the following conclusions.
Proof. We only give the proof of the case (3), other cases can be proved similarly. We infer from (2.5) that (2.34) We first give the estimate the L 1 . Since λ ≤ 0 and w λ , z λ ≤ 0, it follows that where we used the basic inequality Similarly we estimate the term L 2 as follows.
where we used the following basic inequality We can similarly get the estimates of L 3 , L 4 . Thus, we get the first inequality of (2.32). By using similar arguments one can obtain the second inequality of (2.32).
Then we have the following decay at infinity maximum principle for the nonlocal problem (1.1).
and the negative minimum of w λ or z λ is attained in the interior of Σ λ \ {0}. Then, there exists some R 0 > 0(depending on u, v, but is Then we know that (w λ , z λ ) satisfies (2.31). We infer from the definition of (1.3) and w λ (y λ ) = −w λ (y) that (2.40) where ω N denotes the volume of the unit ball in R N . Combining (2.39) and (2.40), we know that On the other hand, it follows from w λ (x 0 ) = min Σ − λ,w w λ (x) that where and
Next we prove the narrow region principle for the system (1.1).

Proof of Theorem 1.1
In this section we shall use the method of moving plane to prove the main results of Theorem 1.1. Precisely, we shall show the symmetry of u and v about T λ0 (λ 0 = 0) by moving plane T λ along x 1 direction from −∞ to the right. We first divide into the following three steps to prove the subcritical case 1 ≤ p < N+γ N−α .

Lemma 3.1. For each μ > 0 and y ∈ R N , we know that u
Proof. From the invariance of the system (1.1), we may assume μ = 1 and y = 0. Moreover, by using the Fourier transforms of the kernels of Riesz and Bessel potentials, we infer from Lemm [12,Lemma 4.2] that (
Proof. As in the subcritical case, we know that every positive solution of (1.1) is radially symmetric and monotone decreasing about some point x 0 ∈ R N . We claim that (u, v) has the desired asymptotic behavior at ∞. That is, it satisfies lim where u ∞ and v∞ are some positive constants. We use the contradiction argument. Assume that (3.29) does not hold. Suppose that x 1 and x 2 be any two different points in R N and let x 0 be the midpoint of the line segment x 1 x 2 . Consider the Kelvin type transform centered at x 0 Then z(x) and w(x) must have a singularity at x 0 . By using the same arguments as in the proof of the subcritical case, we can deduce that z, w must be radially symmetric and monotone decreasing about its singular point x 0 and hence u(x 1 ) = u(x 2 ). Since x 1 and x 2 are arbitrarily chosen in R N , u must be constant, thus u ≡ 0, which contradicts with u > 0. Hence the claim (3.29) holds. Now we borrow an idea of [5,8,43] to give the proof of this lemma. Assume that (u, v) is a nonnegative solution of (1.1). Then if x 0 = 0, we get that Without loss of generality we assume that a = 0 and μ 1 + β = 1. Since the proof is similar, we only give the proof for the first conclusion of (3.31). Let x 0 ∈ R N \ {0} be any fixed point and e = x 0 |x 0 | . We define the function (3.32) Then we know that w(0) = s α−N u∞ = u(0) = w(e) and u = v = s N−α 2 w is also a positive solution of (1.1). Thus w is radially symmetric with respect to some pointx that lies on the hyperplane e ⊥ + 1 2 e through 1 2 e which is perpendicular to e. Moreover, since u, v are radially symmetric about 0, it follows that for any 1 2 < r < 1 and x 1 , x 2 ∈ ∂Br(0) ∩ ∂Br(e), we can deduce from (3.32) that Hence we obtain that w(x) = w |x − 1 2 e| on e ⊥ + 1 2 e, andx = e 2 and w is actually radially symmetric about e 2 . It is clear that there exists σ ∈ − 1 2 , 1 2 such that |x 0 | = 1 2 +σ 1 2 −σ . We infer from (3.32) that This implies that For the case a ≠ 0, we can consider u(· + a) instead of u itself. As in [8,Lemma 3.2], one can prove that where (u, v) is a positive solution to (1.1) with symmetric center m. Finally, we definê This finishes the proof.
The next lemma states the relations between K and (1.1).
On the other hand, we infer from Hardy-Littlewood-Sobolev inequality(see Lemma 2.1) that Similarly, one can prove that Combining (4.18)-(4.20), we obtain that the claim (4.16) holds. Next we prove that for each β < 0, there exists positive (t R , s R ) such that (t R u R , s R v R ) ∈ N. Let (4.21) Then we infer from (4.16) that Since p > 1, we infer from s p R = (t 2−p R − t p R )A 1 /A 3 > 0 and A 3 < 0 that t > 1. Substituting this into (4.23), we obtain that We infer from p > 1 that G(1) = |A 3 | > 0 and lim t R →∞ G(t R ) = −∞. Hence we know that there exists t R , s R > 0 such that (t R u R , s R v R ) ∈ N. Furthermore, we claim that lim R→∞ (|t R − 1| + |s R − 1|) = 0. (4.25) To accomplish this we first prove that t R , s R are bounded. Assume that t R → ∞ as R → ∞. We infer from (4.23) that This implies that s R → ∞ as R → ∞. We deduce from p > 1 that Then we obtain Combining (4.28)-(4.29) we get This is a contradiction. Therefore s R , t R are bounded. We infer from (4.26) that (4.25) holds. Hence we infer from (1.17) that Then we infer from (1.20) that This is a contradiction. Then we finish the proof. where (4.47) In the next lemma we follow the idea of [10,50] to give the estimates for K ϵ.
Lemma 4.6. For each ϵ < p − 1, we have Proof. For any 0 < ϵ < p − 1, we know that the equation has a least energy solution u i (i = 1, 2). Hence we know that We divide into the following two cases to prove our results. If 1 + ϵ < p ≤ 2, the conclusion follows from [10, Lemma 2.7]. If p > 2, we definẽ (4.51) As in [50, Lemma 3.3], we know that Now we consider the function where (s, t) ∈ Λ = {(s, t) : s ≥ 0, t ≥ 0, (s, t) ≠ (0, 0)}. Then it is sufficient to show that f does not attain its minimum over Λ on the lines s = 0 or t = 0. We infer from p > 2 that Hence we know that f can not attain its minimum over Λ on the lines s = 0 or t = 0. This finishes the proof.
Then from Lemma 4.8 we know that Theorem 1.4 (iii) holds. Finally, we prove the Theorem 1.4 (iv).