The status of polycyclic group-based cryptography: A survey and open problems

Jonathan Gryak 1  and Delaram Kahrobaei 2
  • 1 CUNY Graduate Center, PhD Program in Computer Science, City University of New York, United States of America
  • 2 CUNY Graduate Center, PhD Program in Computer Science and NYCCT, Mathematics Department, City University of New York, United States of America
Jonathan Gryak and Delaram Kahrobaei


Polycyclic groups are natural generalizations of cyclic groups but with more complicated algorithmic properties. They are finitely presented and the word, conjugacy, and isomorphism decision problems are all solvable in these groups. Moreover, the non-virtually nilpotent ones exhibit an exponential growth rate. These properties make them suitable for use in group-based cryptography, which was proposed in 2004 by Eick and Kahrobaei []. Since then, many cryptosystems have been created that employ polycyclic groups. These include key exchanges such as non-commutative ElGamal, authentication schemes based on the twisted conjugacy problem, and secret sharing via the word problem. In response, heuristic and deterministic methods of cryptanalysis have been developed, including the length-based and linear decomposition attacks. Despite these efforts, there are classes of infinite polycyclic groups that remain suitable for cryptography. The analysis of algorithms for search and decision problems in polycyclic groups has also been developed. In addition to results for the aforementioned problems we present those concerning polycyclic representations, group morphisms, and orbit decidability. Though much progress has been made, many algorithmic and complexity problems remain unsolved; we conclude with a number of them. Of particular interest is to show that cryptosystems using infinite polycyclic groups are resistant to cryptanalysis on a quantum computer.

  • [1]

    Anshel I., Anshel M. and Goldfeld D., An algebraic method for public-key cryptography, Math. Res. Lett. 6 (1999), 287–291.

    • Crossref
    • Export Citation
  • [2]

    Assmann B. and Linton S., Using the Mal’cev correspondence for collection in polycyclic groups, J. Algebra 316 (2007), no. 2, 828–848.

    • Crossref
    • Export Citation
  • [3]

    Auslander L., The automorphism group of a polycyclic group, Ann. of Math. (2) 89 (1969), 314–322.

    • Crossref
    • Export Citation
  • [4]

    Batty M., Rees S., Braunstein S. and Duncan A., Quantum algorithms in group theory, Computational and Experimental Group Theory (Baltimore 2003), Contemp. Math. 349, American Mathematical Society, Providence (2004), 1–62.

  • [5]

    Bogopolski O., Martino A. and Ventura E., Orbit decidability and the conjugacy problem for some extensions of groups, Trans. Amer. Math. Soc. 362 (2010), no. 4, 2003–2036.

  • [6]

    Bonanome M., Quantum algorithms in combinatorial group theory, Ph.D. thesis, City University of New York, 2007.

  • [7]

    Dehn M., Über unendliche diskontinuierliche Gruppen, Math. Ann. 71 (1911), no. 1, 116–144.

    • Crossref
    • Export Citation
  • [8]

    du Sautoy M., Polycyclic groups, analytic groups and algebraic groups, Proc. Lond. Math. Soc. (3) 85 (2002), no. 1, 62–92.

    • Crossref
    • Export Citation
  • [9]

    Eick B., When is the automorphism group of a virtually polycyclic group virtually polycyclic?, Glasg. Math. J. 45 (2003), no. 3, 527–533.

    • Crossref
    • Export Citation
  • [10]

    Eick B. and Kahrobaei D., Polycyclic groups: A new platform for cryptography, preprint 2004,

  • [11]

    Eick B. and Ostheimer G., On the orbit-stabilizer problem for integral matrix actions of polycyclic groups, Math. Comp. 72 (2003), no. 243, 1511–1529.

    • Crossref
    • Export Citation
  • [12]

    Fesenko A., Vulnerability of cryptographic primitives based on the power conjugacy search problem in quantum computing, Cybernet. Systems Anal. 50 (2014), no. 5, 815–816.

    • Crossref
    • Export Citation
  • [13]

    Formanek E., Conjugate separability in polycyclic groups, J. Algebra 42 (1976), no. 1, 1–10.

    • Crossref
    • Export Citation
  • [14]

    Garber D., Kahrobaei D. and Lam H. T., Length-based attack for polycyclic groups, J. Math. Cryptol. 9 (2015), 33–44.

  • [15]

    Garber D., Kaplan S., Teicher M., Tsaban B. and Vishne U., Length-based conjugacy search in the braid group, Algebraic Methods in Cryptography (Bochum/Mainz 2005), Contemp. Math. 418, American Mathematical Society, Providence (2006), 75–87.

  • [16]

    Gebhardt V., Efficient collection in infinite polycyclic groups, J. Symbolic Comput. 34 (2002), no. 3, 213–228.

    • Crossref
    • Export Citation
  • [17]

    Grigoriev D. and Shpilrain V., Zero-knowledge authentication schemes from actions on graphs, groups, or rings, Ann. Pure Appl. Logic 162 (2010), 194–200.

    • Crossref
    • Export Citation
  • [18]

    Habeeb M., Kahrobaei D. and Shpilrain V., A secret sharing scheme based on group presentations and the word problem, Computational and Combinatorial Group Theory and Cryptography (Las Vegas/Ithaca 2011), Contemp. Math. 582, American Mathematical Society, Providence (2012), 143–150.

  • [19]

    Hall P., The Edmonton Notes on Nilpotent Groups, Queen Mary College Math. Notes, Queen Mary College, London, 1969.

  • [20]

    Holt D. F., Eick B. and O’Brien E. A., Handbook of Computational Group Theory, Discrete Math. Appl. (Boca Raton), Chapman & Hall/CRC, Boca Raton, 2005.

  • [21]

    Hughes J. and Tannenbaum A., Length-based attacks for certain group based encryption rewriting systems, preprint 2003,

  • [22]

    Ivanyos G., Sanselme L. and Santha M., An efficient quantum algorithm for the hidden subgroup problem in nil-2 groups, LATIN 2008 – Theoretical Informatics (Buzios 2008), Lecture Notes in Comput. Sci. 4957, Springer, Berlin (2008), 759–771.

  • [23]

    Kahrobaei D. and Khan B., Nis05-6: A non-commutative generalization of ElGamal key exchange using polycyclic groups, IEEE Global Telecommunications Conference (GLOBECOM ’06), IEEE Press, Piscataway (2006), 1–5.

  • [24]

    Kahrobaei D. and Koupparis C., Non-commutative digital signatures using non-commutative groups, Groups Complex. Cryptol. 4 (2012), 377–384.

  • [25]

    Ko K. H., Lee S. J., Cheon J. H., Han J. W., Kang J. and Park C., New public-key cryptosystem using braid groups, Advances in Cryptology (CRYPTO 2000), Lecture Notes in Comput. Sci. 1880, Springer, Berlin (2000), 166–183.

  • [26]

    Kotov M. and Ushakov A., Analysis of a certain polycyclic-group-based cryptosystem, J. Math. Cryptol. 9 (2015), no. 3, 161–167.

  • [27]

    Leedham-Green C. R. and Soicher L. H., Collection from the left and other strategies, J. Symbolic Comput. 9 (1990), no. 5–6, 665–675.

    • Crossref
    • Export Citation
  • [28]

    Lo E. and Ostheimer G., A practical algorithm for finding matrix representations for polycyclic groups, J. Symbolic Comput. 28 (1999), no. 3, 339–360.

    • Crossref
    • Export Citation
  • [29]

    Mal’cev A., On homomorphisms onto finite groups, Trans. Amer. Math. Soc. 119 (1983), 67–79.

  • [30]

    Milnor J., Growth of finitely generated solvable groups, J. Differential Geom. 2 (1968), no. 4, 447–449.

    • Crossref
    • Export Citation
  • [31]

    Myasnikov A. D. and Ushakov A., Length-based attack and braid groups: Cryptanalysis of Anshel–Anshel–Goldfeld key-exchange protocol, Public Key Cryptography – PKC 2007 (Beijing 2007), Lecture Notes in Comput. Sci. 4450, Springer, Berlin (2007), 76–88.

  • [32]

    Myasnikov A. G. and Roman’kov V., A linear decomposition attack, Groups Complex. Cryptol. 7 (2015), no. 1, 81–94.

  • [33]

    Myasnikov A. G., Shpilrain V., Ushakov A. and Mosina N., Non-Commutative Cryptography and Complexity of Group-Theoretic Problems, Math. Surveys Monogr. 177, American Mathematical Society, Providence, 2011.

  • [34]

    Myasnikov A. G. and Ushakov A., Random subgroups and analysis of the length-based and quotient attacks, J. Math. Cryptol. 2 (2008), no. 1, 29–61.

  • [35]

    Nickel W., Matrix representations for torsion-free nilpotent groups by Deep Thought, J. Algebra 300 (2006), no. 1, 376–383.

    • Crossref
    • Export Citation
  • [36]

    Remeslennikov V., Conjugacy in polycyclic groups, Algebra Logic 8 (1969), no. 6, 404–411.

    • Crossref
    • Export Citation
  • [37]

    Roman’kov V., The twisted conjugacy problem for endomorphisms of polycyclic groups, J. Group Theory 13 (2010), no. 3, 355–364.

  • [38]

    Segal D., Decidable properties of polycyclic groups, Proc. Lond. Math. Soc. (3) 61 (1990), no. 3, 61–497.

  • [39]

    Shor P., Algorithms for quantum computation: Discrete logarithms and factoring, 35th Annual Symposium on Foundations of Computer Science, IEEE Press, Piscataway (1994), 124–134.

  • [40]

    Shpilrain V., Search and witness problems in group theory, Groups Complex. Cryptol. 2 (2010), no. 2, 231–246.

  • [41]

    Shpilrain V. and Ushakov A., The conjugacy search problem in public key cryptography: Unnecessary and insufficient, Appl. Algebra Engrg. Comm. Comput. 17 (2006), no. 3–4, 285–289.

    • Crossref
    • Export Citation
  • [42]

    Shpilrain V. and Ushakov A., An authentication scheme based on the twisted conjugacy problem, Applied Cryptography and Network Security, Lecture Notes in Comput. Sci. 5037, Springer, Berlin (2008), 366–372.

  • [43]

    Shpilrain V. and Zapata G., Using the subgroup membership search problem in public key cryptography, Algebraic Methods in Cryptography (Bochum/Mainz 2005), Contemp. Math. 418, American Mathematical Society, Providence (2006), 169–178.

  • [44]

    Tsaban B., Polynomial-time solutions of computational problems in noncommutative-algebraic cryptography, J. Cryptology 28 (2015), 601–622.

    • Crossref
    • Export Citation
  • [45]

    Wehrfritz B., Two remarks on polycyclic groups, Bull. Lond. Math. Soc. 26 (1994), no. 6, 543–548.

    • Crossref
    • Export Citation
  • [46]

    Wolf J., Growth of finitely generated solvable groups and curvature of Riemannian manifolds, J. Differential Geom. 2 (1968), 421–446.

    • Crossref
    • Export Citation
Purchase article
Get instant unlimited access to the article.
Log in
Already have access? Please log in.

Log in with your institution

Journal + Issues

Groups – Complexity – Cryptology is a journal for speedy publication of articles in the areas of combinatorial and computational group theory, computer algebra, complexity theory, and cryptology. GCC primarily publishes research papers, but comprehensive and timely survey articles on a topic inside the scope of the journal are also welcome.