The Curious Case of Connectionism

Abstract Connectionist research first emerged in the 1940s. The first phase of connectionism attracted a certain amount of media attention, but scant philosophical interest. The phase came to an abrupt halt, due to the efforts of Minsky and Papert (1969), when they argued for the intrinsic limitations of the approach. In the mid-1980s connectionism saw a resurgence. This marked the beginning of the second phase of connectionist research. This phase did attract considerable philosophical attention. It was of philosophical interest, as it offered a way of counteracting the conceptual ties to the philosophical traditions of atomism, rationalism, logic, nativism, rule realism and a concern with the role symbols play in human cognitive functioning, which was prevalent as a consequence of artificial intelligence research. The surge in philosophical interest waned, possibly in part due to the efforts of some traditionalists and the so-called black box problem. Most recently, what may be thought of as a third phase of connectionist research, based on so-called deep learning methods, is beginning to show some signs of again exciting philosophical interest.


Introduction
When it comes to the relationship between computer models and philosophy, the class of models which broadly fall under the heading of 'connectionism' have arguably been amongst the most philosophically significant, or perplexing.1 What makes the case of connectionism 'curious' is the fact that it is one of the relatively rare case where innovations within sub-disciplines of cognitive science have had a significant influence in philosophy and generated a good deal of philosophical commentary. The undertaking here is to look at some of the philosophical claims and issues that connectionist research has given rise to. In doing this, the history of connectionism will be broken down into three broad phases, which will be treated chronologically. The first phase had little philosophical impact, but served to set the scene for the second phase, which had more significant philosophical effects. For this reason, it will only be briefly sketched here. Boden provides a much more detailed account.2 The main focus will instead be on the second phase of connectionism. The paper will conclude with a brief discussion of the third phase, which is beginning to show some philosophical promise. Walker provides a more detailed comparison between the first and second phases of connectionism, and the potential psychological implications.3 While working at the Cornell Aeronautical Laboratory, in 1958 Frank Rosenblatt published "The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain" in Psychological Review. In this paper Rosenblatt proposes to address questions of how information is stored and remembered and how this information influences recognition and behavior in 'higher organisms'. This paper is of interest because Rosenblatt explicitly describes the answers to the questions he intends to explore as being 'connectionist'.13 He is also far from shy about emphasizing the potential of perceptrons. In the concluding sentence of the paper, Rosenblatt states, By the study of systems such as the perceptron, it is hoped that those fundamental laws of organization which are common to all information handling systems, machines and men included, may eventually be understood.14 Rosenblatt followed McCulloch and Pitts in using processors which were all-or-none, in terms of their activation state. A focus on systems built up of components of this kind is one of the hallmarks of what is identified here as the first phase of connectionist research. However, Rosenblatt's main innovation beyond the work of McCulloch and Pitts was to design various systems that could undergo training. Later, in 1962, Rosenblatt extended this research in his monograph Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms.
Rosenblatt's Perceptrons consisted of banks of processing units arranged in a couple of layer, one of which acted as inputs to the system, the other acting as the outputs of the system, with modifiable weighted connections between the various components. Training took place by changes being made to these weighted connections. Of particular significance was the fact that Rosenblatt was able to prove the so-called 'Perceptron Convergence Theorem' for his systems. In layman's terms, this theorem showed that, if there was a solution to a particular problem, then within a finite number of steps, a perceptron would be able to find a solution to the problem, such that it gave correct responses for all the training patterns.15 Rosenblatt's work attracted quite a bit of media attention, which had both advantages and disadvantages, according to Boden.16 In particular, Hecht-Nielsen (quoted in Anderson and Rosenfield) noted that "[t]he wall-to-wall media coverage of Rosenblatt and his machine irked Minsky."17 A similar claim was made by Feldman in a talk at Buffalo, given on 26th July 1994. This is of significance, as the next phase of this story has Minsky and his colleague Papert front and center.

Perceptrons
Minsky was one of the early pioneers of artificial intelligence research. He, along with John McCarthy, co-founded what is now known as the Computer Science and Artificial Intelligence Laboratory, at MIT.18 Although Minsky initially studied connectionist style systems, he eventually focused upon exploring intelligence as based upon symbolic manipulation. This approach is often characterized as being a contrast class to connectionism.19 As Hecht-Nielson recounts the events surrounding the publication of Minsky and Papert's famous 1969 book Perceptrons, it was the culmination of a concerted campaign against Rosenblatt to ensure funding for their Artificial Intelligence research.20 Hecht-Nielson even compares Minsky to Darth Vader!21 The main point of Perceptrons was to draw attention to the limitations of perceptrons. They argued that, assuming some not unreasonable assumptions, such as limiting the conclusions to networks without massive interconnectivity between processing units, perceptrons could not handle connectedness, counting objects, or even learning the truth-table for XOR.
In many ways, the Hecht-Nielson account of the publication of Perceptrons paints Minsky and Papert as pantomime villains, straight out of central casting. A more nuanced and detailed account of events appears in Boden.22 Medler characterized the situation as follows: "[t]he publication of Perceptrons in 1969 by Minsky and Papert has taken on a mythical aura -it has been likened to the huntsman being sent out to bring back the heart of Snow White."23 The claims about funding for Artificial Intelligence (AI) appear to be false. According to Minsky, their research was "...very well funded, we had $1,000,000 per year from ARPA from 1963 to 1970, and more later. You can't prove theorems for political purposes, anyway."24 In an admirable turn of phrase, Minsky continues, "It would seem that Perceptrons has much the same role as The Necronomicon -that is, often cited but never read."25 Nonetheless, the publication of Perceptrons had a significantly chilling effect upon connectionist research. Most notably, according to Boden, it prevented funding of this line of research from major funding agencies.26 This effectively put the bulk of connectionist research (with a few notable exceptions) into a state of hibernation until the mid-1980s when the second phase of connectionism took off. It was this second phase of connectionism that held the greatest philosophical promise. The decline in interest in connectionism marked the end of the first phase of connectionist research.

Parallel Distributed Processing (PDP)
The key event which kicked off the second phase of connectionist research was the publication of a two volume work Parallel Distributed Processing: Explorations in the Miccrostructure of Cognition. The importance of these two volumes can be estimated by the fact that Davis refers to this work simply as the PDP 'Bible'.27 This work was attributed to Rumelhart, McClelland and The PDP Research Group. The members of this group are listed by Rumelhart,McClelland et al..28 Although this list contains people from various disciplines, such as psychology, computer science, cognitive science, and even linguistics, it is interesting to note in the current context that no philosophers are included in this list. Boden notes that the Rumelhart, McClelland, et al. text was designed to be accessible, in order to attract interest, especially from graduate students. Steps were also taken to bring down the cost to maximize accessibility. Boden also notes that, "[t]hey hoped to have well-thumbed, coffee-stained, volumes on student desks-not pristine pages sitting on library shelves."29 The systems discussed in this volume were fundamentally parallel, as they were composed of many comparatively simple processors. They were also distributed as, rather than there being a single processor, processing took place across multiple processing elements that worked together to determine the final output of the systems. In a sense, these systems were similar to perceptrons in these respects. However, the significant innovation of these systems was that, rather than just having input and output units, the systems they described also had intermediate processors, the so-called 'hidden layers', which were not directly connected to the inputs, or outputs.
The main technical innovation which launched the second phase of connectionist research was the development of processing elements which had activation functions which were more complex than the simple all-or-none processors of the Rosenblatt era. This in turn facilitated the development of more complex systems of processing units, with more layers of processors, namely the hidden units. A commonly used processor type had a sigmoid activation function. The sigmoid activation function had roughly an 'S' shape. So-called Radial Basis Function (RBF) units, which had a Gaussian activation were also introduced. The additional mathematical complexity of these activation functions made it possible to take derivatives, which in turn facilitated training these more complex networks. To a large degree, much of the work in Rumelhart and McClelland et al. volume built upon the work of Hopfield, who initially explored the mathematical properties of sigmoid activation functions.30 Another important milestone in the reemergence of connectionist research, which is often overlooked in standard accounts, was the publication in 1988 of McClelland and Rumelhart's, Explorations in Parallel Distributed Processing. The unique thing about this book was that it came with two floppy disks, which contained versions of many of the systems discussed in the 1986 Rumelhart, McClelland et al. book. This was a masterful stroke, as it made connectionist software available to anybody who had access to a computer. While it is one thing to read about the abilities of classes of systems in the abstract, it is something completely different to be able to play with the systems and let people 'get their hands dirty' with the software.
To appreciate why the Rumelhart, McClelland et al. publication from 1986 had such a significant impact, it is worth briefly sketching the philosophical context in which it arose. Authors such as Dreyfus and Dreyfus, Aizawa and Boden, to name a few, associated traditional AI research with certain philosophical inclinations.31 These include conceptual ties to the philosophical traditions of atomism, rationalism, logic, nativism, rule realism and a concern with the role symbols play in human cognitive functioning. This conceptual viewpoint is nicely typified in Haugeland's 1985 book Artificial Intelligence: The Very Idea. In this book, Haugeland christens this conceptual view point as 'GOFAI', standing for 'Good Old Fashioned Artificial Intelligence'. The problem was that at the time Haugeland's book came out GOFAI was in trouble. Early promising successes appeared to lead nowhere in particular. Thus, when Rumelhart, McClelland et al. published their two volume set, people were looking around for alternatives to GOFAI. In addition, philosophers who were opposed to the philosophical inclinations of GOFAI, were naturally drawn to potential alternatives. These then are likely the reasons that the second phase of connectionism began to have impacts in philosophy.
As a matter of fact, connectionist research had attracted philosophical attention, even prior to the publication of the PDP volumes. For instance, in 1985 Bechtel published an early philosophical paper on connectionism.32 A somewhat crude, but nonetheless quite effective means of gauging the impact of connectionism on philosophy can be gained by querying the Philosopher's Index and counting the number of hits it gives, by year. This was done looking for the terms 'connectionist' and 'connectionism' and the number of entries found were then tabulated and recorded. The results are recorded in Figure 1. The specific version of the Philosophers' Index used was the on-line version available through EBSCOHOST. For the sake of completeness, it is worth mentioning that the terms 'Parallel Distributed Processing' and 'PDP' were also checked, but produced too few results to be worth reporting.
A brief inspection of Figure 1 makes it abundantly clear that the period of peak interest in connectionism in the philosophical literature was the mid to late 1990s. The raw data upon which this plot was based is presented in Table 1.
While these results have heuristic value, they should not be construed as being canonical, for a number of reasons. First, during this period of time a number of new journals dealing mainly with connectionist themes, such as Connection Science and Neural Computation were founded. Both of these were founded in 1989. However, no papers in Neural Computation and just one paper in Connection Science are included in

Fodor and Pylyshyn (1988)
Although philosophical interest was piqued by the arrival of the second generation of connectionism, not all the interest was positive. Somewhat notoriously, Fodor and Pylyshyn published a long paper in Cognition arguing that connectionist models would fail to be able to handle systematicity and compositionality, unless they amounted to implementational variants of traditional 'classical' AI type systems. Systematicity is the property of a representational system which supposedly ensures that there are certain symmetries in the representational system such that the ability to represent something will ensure the ability to also represent other things with related semantic content. Compositionality is the idea that the meaning of a complex expression is a function of the constituents parts of the expression, along with the rules for combining those parts. So, to borrow one of Fodor and Pylyshyn's examples, if a system can handle the sentence 'John loves Mary', it should also be able to handle the sentence 'Mary loves John'. Their claim is that classical systems can do this with ease, while connectionist ones cannot, unless they are implementational variants of classical systems. Rhetorically, this was an inspired move. In 1984, Pylyshyn had argued that cognitive systems could be usefully discussed using three distinct levels of description.38 In this, Pylyshyn broadly followed Marr's three levels of analysis of cognitive systems.39 Marr's three levels were the computational level, the algorithmic level and the implementational level. However, it was axiomatic to Pylyshyn's position that the implementational level of description was non-cognitive. So, Fodor and Pylyshyn's conclusion, if accepted, would preclude connectionist systems from playing any fruitful, or useful, role in cognitive science.
For Fodor, connectionism represented a potential threat to his research program. Famously, Fodor had argued that there had to be a 'language of thought', with certain particular properties.40 His argument for this view was that this was 'the only game in town' (paraphrasing Richard Nixon). The properties that Fodor attributed to the language of thought were very similar to the kinds of properties that were found in both natural languages and many traditional approaches in Artificial Intelligence research. However, the rise of connectionism, at least potentially, suggested that there may be another 'game' possible. So, Fodor and Pylyshyn's publication represented a defense of the traditional philosophical view that derived from the AI research tradition.41 So, the publication from Fodor and Pylyshyn (1988) was one of the early salvos in a philosophical debate between connectionists and more traditional views.
Another reason that Fodor and Pylyshyn's paper is important is that it raised questions and topics which served to stimulate a good amount of both philosophical and technical commentary and debate. Of particular note was the lively philosophical debate that arose between Aizawa and Hadley concerning systematicity, productivity, and cognitive architecture. Aizawa provides a helpful summary of the various moves in this debate.42 Another philosophical response can be found in Van Gelder. He endorses a position that is neither unequivocally connectionist, nor traditional. Rather, he attempts to pursue something like a middle path.43 However, it was clear that the rise of connectionism succeeded in opening a proverbial philosophical 'can of worms'.
There was also push-back against Fodor and Pylyshyn, with more technical methods. Of particular note were the responses from Smolensky and Pollack.44 They both offered formal approaches that seemed to undermine the core claims advanced by Fodor and Pylyshyn. However, such challenges were not well received by Fodor, at least. Others, such as Chalmers (1993) and Dawson, Medler and Berkeley (1997) combined both philosophical and technical aspects.45 So, despite all the drama, the second generation of connectionist systems opened up the philosophical arena to a wider range of philosophical positions than had seemed plausible prior to these innovations.
Fodor warmed to his crusade against connectionism. In 1990, he followed his paper with Pylyshyn, with a paper with McLaughlin (1990).46 In this paper, Fodor and McLaughlan took specific issue with the proposals made by Smolensky in 1987, arguing that one proposal was too weak to be relevant and that the other failed to properly account for systematicity.47 Even after this, Fodor was not finished. In 1997 he published another paper with the provocative title, "Discussion: Connectionism and the Problem of Systematicity (continued): Why Smolensky's Solution Still does not Work." The philosophical debate that the original Fodor and Pylyshyn paper helped to generate explains, at least in part, how philosophers began to be interested in connectionist research.

The rise of connectionism in philosophy
As was noted earlier, connectionism provided an interesting alternative to the atomistic, rule based ideas that were prevalent in philosophy at the time. In 1989, two philosophical books were published which also helped to bring connectionism into the philosophical mainstream. The first of these was Lloyd's (1989) book, Simple Minds. This work at least took the potential contribution of connectionism to the philosophy of mind seriously. However, probably of greater significance was the publication in 1989 of Clark's Microcognition. The big advantage of Clark's book was that it offered a discussion of connectionism which was reasonably detailed, but did not get too bogged down with technicalities. This book was thus highly accessible to interested philosophers. Clark also made the inspired choice of introducing connectionism to philosophers mostly through a discussion of the Jets and Sharks model, originally published by McClelland, Rumelhart and Hinton.48 The Jets and Sharks model took the properties of the two gangs from the 1961 film West Side Story and mapped names onto gang affiliation, age, level of education, marital status and occupation. While the Jets and Sharks model did not learn and was in fact constructed by hand, it had two advantages. First, it was reasonably intuitive and easy to understand, even by novices. Second, an implementation of this model was included with the software supplied with the 1988 McClelland, and Rumelhart text. So, interested philosophers, or even skeptical philosophers, had a reasonably easy means to test the various claims that have been advanced.
In Microcognition, Clark was also able to avail himself of the relatively novel conceptual framework that connectionism seemed make appear plausible. He argued for a view of mind that was much less atomistic and decidedly more holistic, as compared to many of the alternatives. Other early advocates of the importance of connectionism to the philosophy of mind included Horgan and Tienson, Bechtel, Bechtel and Abrahamsen, Paul Churchland, and Patricia Churchland and Sejnowski, to name just a few.49 This era also had various meetings where connectionism was a major topic of discussion. For instance, a number of the papers that appear in Horgan and Tiensen's 1991 collection were originally presented at the Spindel Conference held in 1987 at Memphis State University.50 Similarly, the annual Vancouver Studies in Cognitive Science Conference held at Simon Fraser University in 1990 was focused on the topic of connectionism. Some of the papers from this conference subsequently appeared in edited collection by Stephen Davis.51 Interestingly, there is also a set of notes taken at this conference by graduate students.52 Contributions concerning connectionism also began to appear on the schedules of both national and regional philosophy conferences, during this period.
Suffice to say, by the early 1990s, connectionism was getting a great deal of interest, both in philosophy and in other disciplines. Indeed, as a movement, connectionism was taking on an almost revolutionary tone. In 1987, Schneider raised the question whether connectionism amounted to a fundamental paradigm shift in psychology.53 Also in 1987, Horgen and Tienson echoed this sentiment when they noted, "There is a Kuhnian crisis in GOFAI, brought on by a pattern of unfulfilled promises and disappointing results."54 In 1991, they were to explicitly argue the case for this Kuhnian interpretation.55 Another important, though often overlooked, event which helped propel connectionism was an unusual event which rejoiced in the name FISI-CS. This stood for The First International Summer Institute in Cognitive Science. It was held at SUNY Buffalo from July 5 th to 30 th , 1994. It was a remarkable event with around 400 participants from 32 countries and about 100 faculty and plenary speakers. Participants were, for the most part, graduate students and faculty, with a few undergrads drawn from universities and industries from all over the world.56 The institute offered a range of classes, in addition to a plenary speaker most weekday evenings. The formal presentations were followed by more informal so-called 'fire-side' chats. Over the weekends there were workshops and social events.57 The reason this is relevant here is because one of the courses was about connectionism. It was taught by Rumelhart and Smolensky. The course description said explicitly, "This course assumes no prior background in connectionism (or linguistics)." In other words, FISI-CS provided a perfect vehicle to proselytize the cause of connectionism. Needless to say, the Rumelhart and Smolensky class was very well attended. In addition, there was a two day workshop on "Connectionism and Neuroscience" and a two day workshop on just connectionism, which featured 56 For an incomplete list of participants, faculty and plenary speakers see http://ftp.cse.buffalo.edu/users/rapaport/fisiaddresses.html. 57 A detailed schedule of classes, talks and workshops appears in the FISI Handbook, which is available at https://cse.buffalo. edu/~rapaport/Papers/FISI-Handbook.pdf. a significant contingent of philosophers. Thus, connectionist ideas became easily accessible to anyone with an interest, including philosophers.
At this point it is worth turning attention more directly to looking at the specific philosophical claims which connectionism was (at least putatively) supposed to support. As the literature is pretty huge, rather than attempting to be encyclopedic, the approach that will be adopted here is to focus on particular philosophical claims, which arose with some regularity.

Nativism and innateness
Broadly speaking, nativism, or innateness is a view that important parts of the human epistemological systems, or cognitive architecture, must be naturally supplied to us, by some process (the exact answers differ here), rather than being simply learned from experience. Nativism, in one form or another, has been a philosophical position that has historically been through periods of popularity and decline.58 Traditionally, nativist positions have been advocated for by philosophers such as Plato, Descartes and Leibniz. As such, the view has became associated with rationalistic views of things. By contrast, more empirically inclined philosophers, such as Locke and Hume have battled against the nativist tradition.
In recent times, nativism has been advocated for in linguistics, by Chomsky (1965).59 In the philosophy of mind, nativism was famously championed by Fodor (1983).60 As many of the arguments concerning nativism have been focused upon human linguistic capacities, connectionist simulations that focused upon language were thus a potential source of anti-nativist arguments and evidence.
An important development concerning the relationship between connectionism and innateness occurred in 1996, when Elman, Bates et al. published a volume with the title Rethinking Innateness.61 This book amounted to a sustained attack upon many commonly held nativist views. Interestingly enough, this volume was followed in 1997 by Plunkett and Elman's work, which included the Tlearn suite of software, which provided a variety of connectionist simulations, which broadly supported the claims made in the book of the previous year.62 The text was accompanied by software for both Windows and Macintosh. In some senses, this strategy of offering up a theoretical work, which was then supported by software simulations, mirrored that pioneered by Rumelhart, McClelland et al. with McClelland and Rumelhart's Explorations text, a decade before. However, from a rhetorical point of view, making claims concerning nativism and innateness, supported by simulations, was more persuasive than merely offering arguments. So, with this, connectionism was again being used to attack the standard GOFAI inspired philosophical perspective. This was not the only front where connectionism acted as a disruptive influence.

The question of rules
One claim which commonly arises in the philosophical literature concerning connectionism is that in some sense networks provide an alternative to the systems of rules which often characterize work in GOFAI. For instance, Bechtel and Abrahamsen, while discussing a logic problem task, remark "...we raise the possibility that the rules can be eliminated entirely in the modeling medium, letting networks do all the work."63 Quite what the nature of the work the network was supposed to be doing, or how it is supposed to be doing it, is unclear. In fact, issues and questions concerning rules pretty rapidly gets really quite complicated. This is, in part, because of the large philosophical literature concerning rules and rule following.64 However, it is clear that this line of thinking is common in connectionist inspired philosophy. To cite a similar example, Clark remarks "The PDP model challenges this assumption that in-the-head mechanisms mirror structured, componential, rule-based linguistic theories."65 So, connectionism offered the promise of considering cognition from a perspective that was markedly dissimilar to that proposed by GOFAI.
Perhaps the best known of the rule skeptics are Horgan and Tienson.66 Their view was subject to philosophical doubts, objections and questions.67 There were also a further technical problems which arose from claims like these: there was simply a lack of any direct evidence. While in a program say running LISP, one could point to the program and say, "Look, there is the rule!" In the case of a connectionist network, there was nothing which could so easily be identified. Should one look at matrices of weights, activations of units, or something else? While there were many potential candidates, the correct one to look at was far from clear. Also, upon what basis could a choice of one potential candidate over another be justified? Some theorists took the fact that there were no obvious rules to count as evidence that there really were no rules. 68 Bechtel and Abrahamsen remark "Without a detailed analysis of the activities of the hidden units (which was not performed), we cannot determine exactly how the network solved this [logic] problem."69 A likely reason that this analysis was not performed will be discussed below. Before addressing this issue a few words about representations are in order.

Symbols and sub-symbols
One of the foundational tenets of GOFAI is the Physical Symbol System Hypothesis (PSSH), introduced in 1976 by Newell and Simon. This amounted to the hypothesis that a physical symbol system had the necessary and sufficient means to exhibit general intelligent action.70 Physical Symbol Systems are most naturally understood as the kinds of representations and operations found in traditional GOFAI research. The question of whether connectionist systems met the conditions of being physical symbol systems, in the relevant sense is a vexing one. Newell is clear he thought not, while Simon thought they were physical symbol systems. 71 Berkeley provides a more detailed discussion of these points. 72 As a general rule, connectionist theorists prefer to consider connectionist systems as operating at the sub-symbolic level, following the terminology introduced by Smolensky. As Smolensky uses the term, subsymbols are supposed to be the fine grained dynamical features that are below the conceptual level. He conceived the sub-symbolic domain as being between that of the symbolic level and the neural level.73 This, then, was another way in which connectionism represented an alternative to the GOFAI paradigm. Of course, just as the notion of symbolhood was not entirely conceptually clear, particularly across disciplines, so the idea of what actually constituted a sub-symbol was to some degree equally occult.74 However, this difference, whatever it may ultimately turn out to be, represented another way in which connectionism potentially provided a philosophical alternative to the received orthodoxy. Clark and Lutz put the matter succinctly when they claim that "Connectionist models...differ from those of conventional AI in (amongst other things) appearing to operate without traditional symbolic data structures over which computational operations may be defined." 75 There was, though, a significant problem which arose concerning many of the philosophical claims put forward on the basis of connectionist research. So, it is now time to address this issue.

The Black Box Problem
The Black Box Problem with connectionist systems is that once they have learned to perform a task with an acceptable level of proficiency, it is far from straightforward to determine how they are solving the task at hand. This is not a new problem. For instance, Mozer and Smolensky remark that, "...one thing that connectionist networks have in common with brains is that if you open them up and peer inside, all you can see is a big pile of goo."76 A similar, though more overtly pessimistic point is made by McCloskey when he remarks that "...connectionist networks should not be viewed as theories of human cognitive functions, or as simulations of theories, or even as demonstration of specific theoretical points."77 The point here is that if there is no detailed understanding of how trained networks solve problems, then they have very little that they can contribute to cognitive theorizing, or for that matter to philosophical theorizing. This, then, is the black box problem and it is a serious one.
Back in the early 1990s, there was some skepticism about solving the black box problem. Hecht-Nielson suggested it would require a profound revolution in information processing. 78 Robinson went further suggesting that, "We may have to accept the inexplicable nature of mature networks."79 Fortunately, such skepticism was unfounded.
A number of techniques have been proposed for analyzing trained networks. For example, a range of approaches are presented by Browne.80 Unfortunately, many of the methods proposed were subject to frustrating limitations. For instance, Bullinaria showed how the commonly deployed method of cluster analysis can mis-classify exceptional items and miss other crucial aspects of network performance. He also addresses the limitations that can arise from another commonly proposed technique, principle component analysis. Under certain conditions, Bullinaria argues that principle component analysis may fail to capture enough variance to recognize anything but the grossest features.81 Similarly, the Berkeley, Dawson et al. (1995) banding analysis82 method only apparently worked with relatively obscure so-called 'value units'. 83 Eventually, after some mathematical insights into the relationship between value units and more traditional sigmoid processing units, the banding analysis method was generalized to more traditional units, by Berkeley and Gunay.84 Berkeley and Raine were even able to show how banding analysis could be used to generate an entirely novel theory of how to solve a particular problem.85 However, these innovations had little philosophical impact, as philosophical interest in connectionism had passed, even if some interest remained in psychology and cognitive science. 86 The precise reasons for the sudden rapid rise in philosophical interest in connectionism, followed by a rapid decline is unclear. Perhaps it may be a topic of interest for future historians of ideas. However, the second generation of connectionist research did serve to open up the range of philosophical positions that could plausibly be defended, based upon simulation results.
Before attempting to draw some final conclusions about the impact of connectionism on philosophy, for the sake of completeness, it is worth briefly looking at what is happening with connectionist research these days, in what might be thought of as the third phase of connectionism, the era of so-called 'deep learning'.

Deep learning
Although the frenzy of interest in connectionism in philosophical circles has waned, it has not stopped completely. Philosophical papers on connectionism still appear from time to time.87 At the same time, technical research into connectionist systems has continued. Unlike the sharp transition between the first and second phases of connectionist research, the move to the third phase has been more gradual and incremental. The modern incarnation of connectionism takes the form of so-called 'Deep Learning'. 88 Deep learning is the third phase of connectionism, yet has garnered limited philosophical interest, so far. However, there are beginning to be signs that philosophical interest is on the upswing again. For instance, Rubio argues that deep learning architectures are compatible with a version of computational functionalism.89 If this conclusion is correct, then it would be reassuring. Also of interest is a recent paper by Buckner. In this paper Buckner argues that a class of systems called 'Deep Convolutional Neural Networks' (DCNNs) seem to perform surprisingly well in various domains. He also suggests that DCNNs can help address longstanding problems in empiricist philosophy of mind, by demonstrating 'transformational abstraction'. In particular, Buckner is able to propose a model based account of the philosophically puzzling conception of abstraction.90 If this is a trend which continues, then it seems that connectionism still has contributions to make to philosophy. It is also the case that novel ethical conundrums can arise through the use of these technologies. 91 Deep learning is certainly a useful technology. For instance, it can be used to do things like classify traffic signs.92 This is a handy ability, for instance, if one wishes to build self-driving cars. It also appears to be a technology that should continue to be of interest to philosophers, at least based upon the early signs. So, a conclusion of 'watch this space', seems to be in order.

Conclusion
Connectionism can be broadly analysed as having three phases, beginning in the 1940s. The first phase was technologically interesting and generated a certain amount of interest in the media, but had a minimal impact on philosophy. It came to a fairly abrupt end, at least according standard narratives, due to the activities of Minsky and Papert. The second phase of connectionism, due to a combination of factors, gave rise to a great deal of philosophical activity and excitement. It seemed to offer a means of challenging many of the received views concerning nativism, atomism, rules, and representations. Unfortunately, due to the black box problem and the intervention of Fodor, Pylysyn and McLaughlin, quite what the connectionist alternative to these views should be was both controversial and unclear. Eventually, philosophical interest began to wane, but still remains present. More modern developments in connectionist research in the third phase of connectionism is beginning to yield further philosophical insights.
It is certainly the case that the increase in computational power and the increasingly sophisticated architectures that became widely available between the 1940s and the present day has played a role in the rise of connectionism. We will just have to wait and see what interesting insights it will yet yield