neuroplausible

Path Model Q&A

Sun, 03 May 2020 00:00:00 +0000

Anna Henschel and Stephanie Allan kindly invited us to discuss our (Olivia Guest & Andrea Martin) work How computational modeling can force theory building in psychological science at their reading group Glasgow ReproducibiliTea last week. So we decided to write up the questions we received in this blog post. It was a very enjoyable experience. And we’re very grateful not least because it’s explicitly aimed towards and for students and more junior people in our field — an audience we believe is especially able to learn how to improve their scientific reasoning and theoretical construction skills.

Video from Glasgow ReproducibiliTea reading group.

We went into some background on why we created our path model, why computational modelling is a useful tool to help refine our own thinking as well as the theories scientists propose. You can watch a video of us talking and answering a subset of the questions, kindly edited and uploaded by Anna. Below are some of the same questions (with very minor edits for typos and clarity and reordered for ease of answering) that we received while we were chatting — some of the questions we answered in the video might not be answered here and vice versa. Also some questions are not answered below because we are working on follow-up work that addresses them and so to save time and space here we’ll just skip those for now. Super importantly before reading this, read the manuscript as our answers are long and don’t really make sense without that context.

General Questions

I might be confusing things here: but what is the difference between computational modeling and careful operationalization (in the empirical circle)?

This is the difference in our path model between specification and implementation. Captured by this tweet, which served as the inspiration for the section The pizza problem:

Why we need computational modelling: even if everybody agrees what the area of a circle is defined as there are apparently people unwilling to execute the formal model itself and not only that but the results are counterintuitive. 😂https://t.co/c1SKgRAZMh
— Olivia Guest is on the job market! | Ολίβια Γκεστ (@o_guest) October 21, 2019

One question I had from reading the paper was you say that most undergrads in psychology leave knowing a bit about statistical models. From my own experience (aware other degrees might be different) I do not think there was as much focus on theory building. Do you have any ideas how undergrad education could incorporate theory building?

In our experiences of working at various universities, there is indeed not much explicit focus during undergraduate degrees on general theory building, setting aside computational or formal modelling of theories. However, what happens very often is throughout each module the historical and current accounts for psychological theories (and indeed frameworks) are very much expounded on. Students learn about, for example, how Pavlov explored dogs’ behaviours and created the ideas behind classical conditioning. These historical and contemporary stories of how scientists develop their understanding of a series of phenomena are how theories are built. What might be happening is that students are not ready yet (due to the deluge of information) to also zoom out and notice that theory creation and development is being signposted to them. This is a normal byproduct of learning new things. Some statistics modules also contain very basic references to “theory building” as part of a falsification-focused approach.

One of many possible paths (in blue) that can be used to understand and describe how psychological research is carried out with examples of models at each step shown on the left (in green). Each research output within psychology can be described with respect to the levels in this path. The three levels superimposed on a red background (theory, specification, implementation) are those that are most often ignored or left out from research descriptions. From figure 2 in Guest and Martin (2020).

On the other hand, a specific module on “how to build theories” in undergraduate degrees is indeed typically lacking. It could be created but will be something incredibly difficult to get right and succeed pedagogically. We propose this might be for two related reasons: a) creating a novel theoretical account, or indeed building one from modifying existing theories, is extremely difficult for everybody. This is why in part we suspect that in the status quo certain people tend to avoid working in the red area of our path model’s depiction in figure 2, when that is where more focus is actually needed. The step in our path model called “theory” is the hardest one and — tangentially — certainly not something that only computational modellers should care about. Everyone, whether they realise it or admit it, if they are trying to figure something out, cares about theory. In addition, b) given how hard theory building/ the red area is in practice, imagine how much harder it will be to teach it. Pedagogy requires highly skilled individuals dedicating their lives to learning skills and how to teach others skills. How to build theories is what the whole of philosophy (and history) of science studies. So it is perhaps unsurprising given all this that psychology departments do not, or cannot, provide such a dedicated module. But that doesn’t mean it can be ignored or not taught.

This all being said, we do not take such a strong pessimistic outlook going forwards, hence why we wrote this manuscript. Realistic and helpful steps that can be taken to address theory building, in addition to reading our paper and thinking about how modelling can be applied to your work, is to actively engage with theoreticians in psychology, neuroscience, and cognitive science as broadly as possible. There are experts out there who make it their life goal to develop theories and, sometimes in lesser part, teach others how to do that also. We hope that any students/faculty reading this blog post might consider asking for/discussing a module that involves teaching and engaging with theoreticians’ and modellers’ works.

Probably related to the undergrad curriculum question but what do you have any advice on how someone who hasn’t been trained in comp modeling could “responsibly” start dabbling with it? Or should any serious work on the implementation step only be done in collaboration with a computational modeler?

We suggest people try working through toy examples like the one with the pizzas we work through in the paper (see section The pizza problem). And then try to make their own, in the simplest way possible. Then, perhaps by doing that for several explananda, your feeling about yourself in the process will change or settle. Start with a natural language sentence, then pizza model it! We go through the basic steps of very simple theory, to very simple specification, to very simple implementation, and you can too.

After all, this is the process, even more than the core methodological skills, that is required to make a simple model. The methods skills can come later and are a process of ongoing growth and change through one’s modelling life anyway. But the key is to grasp the endeavor, and start, rather than focus on the fancy bells and whistles or feel intimidated by them. The ability to begin with the statement in natural language and then even get to the first step of specification is thrilling!

For the second part: ideally, yes. But it’s not a requirement, and we think that is important to emphasize that. Collaborating with someone who already models will give you a chance to work with somebody who is more experienced and you will likely learn a lot. To sum up though, we want to emphasize that computational modelling is something everyone and anyone can effectively engage in, with little extensions from the skills and mindset they are asked to acquire in most undergrad psych degrees in 2020. For a list of modellers in cognitive science you might want to check out this list: compcog.science.

Pizza Problem Questions

Only by actually running the formal model of the pizza options can we know which option is more food.

Is “2 pizzas is more food” a theory in this context? And the pizzas are our data, right?

“2 pizzas is more food” is a hypothesis. It can be seen as a hypothesis based on our gut feeling (not an implementation). So we enter the path model at the hypothesis stage and then move to collect data, i.e., measure the amount of food in the pizzas.

So where would you fit the data that intuitively, people would prefer the two pizzas? Or is that entirely outside the metaphor you’re using?

The fact that people, we, intuitively think 2 pizzas is more food is the core of the pizza problem: an expectation violation (see section Model of psychological science). Rule two for constraining movement in our path model says that moving downwards is only possible if an expectation violation is resolved. So, in the pizza example, we can be seen as entering the path model at the hypothesis step. Our hypothesis is “two 12’’ pizzas are more food than one 18’’ pizza”, so we measure them (this is obviously not explicitly done because it’s a simple example, but imagine we order both options and measure the food in each). Now the data tells us our hypothesis is wrong. So we must move upwards to an appropriate level and figure out why we were wrong.

Path Model Questions

A pet raven.

So does the downward stream correspond to a hypothetico-deductive approach and the upward stream to an inductive approach? Or is that too simplistic?

This is something we discussed when we were writing this up. Mainly for reasons of article length (there was a 5,000 word limit) we, Andrea and Olivia, decided not to go into this potential interpretation of the path model. To get everybody on the same page, inductive reasoning is when our premises are taken to provide some evidence for the truth of our conclusions. For example, all our lives we see ravens that are black so believe that “all ravens are black”. This of course could be false if we encounter a non-black raven (recall Europeans discovering black swans). Deductive reasoning, on the other hand, is when our premises are used to reach logically certain conclusions. For example, all ravens are birds, my pet is a raven, therefore my pet is a bird. Science uses all these kinds of logical inference to various extents including abduction.

One thing to bear in mind as a limitation of the typical hypothetico-deductive model of science is that it does not explicitly in its typical formulation include modelling or theory development in a way that satisfies us (e.g., it ignores underdetermination). It merely mentions steps that take us from hypothesis to data collection, so within the context of our model it doesn’t really emphasise theory or any of the other steps in the red area of our figure 2: theory, specification, implementation. The red area is where we want to draw the most emphasis in this paper (see section Model of psychological science). This is a part of psychological science that we propose is often left out with serious repercussions in terms of scientific integrity, openness, reproducibility, and so on.

Can (or can’t) we think of examples where a field was operating rather model free, then models entered and shifted the focus away from what turned out to be actually more important later on, e.g. due to the necessary simplifications?

In our understanding of how science is carried out, there are always models at play. The issue is that they need to be made explicit.

Then why do you say “plz make comp models” if there is no model-free science?

Our serious request, presented in a tongue-in-cheek fashion.

The emphasis should be on the “make” as in make your models explicit through the use of computational instantiation. Also it’s a meme — well-known to be caricatures of the real world. It also touches on the message that making a computational model will force you to acknowledge that your science is not and cannot be model-free, even if you want to think it is for whatever reason.

Should open theory as you propose it in the paper undergo the same preregistration process as other parts of the experimental process?

Being pedantic and emphasising the word “should”: no. We don’t believe that such a tool makes sense for open theory because as we will discuss below there are other mechanisms that allow for constraining our science. We also do not believe that scientific prescriptivism facilitates useful work. This is a good chance to clarify that our path model is merely a description of the process of doing science. So we can take an existing literature and plot its path. It is able to account for any scientific act. If there are scientific acts (including malpractice, HARKing, etc.) that our account does not have the ability to describe (including ones that we can describe as skipping steps), we need to amend our model (see section What our path function model offers). In other words, we wish for our account to be a description of science as is currently understood. And to facilitate talking about scientific acts in useful, transparent ways.

A raven being pet.

Going back to the question, however and discussing the role of preregistration, we believe our path model depicts (the bidirectional arrows) how strong constraints from one level to another can percolate up/downwards and refine the other levels. What we mean by this is that following the path model itself allows for the types of constraints (that data modellers use preregistration for) to be applied at every step. In other words, preregistration and related tools are a way to diminish various forms of inadvertent or purposeful scientific malpractice at the hypothesis and data levels, and in some cases to promote openness and replication. Preregistration was adopted in psychology and neuroscience from clinical trials and it serves its purpose well. It plays the role that supervening theories or formal models could play, i.e., to constrain the space of hypotheses and data collected. However, in the clinical trials literature they do not typically derive hypotheses to test based on theory, they merely compare groups of patients when administered different or no drugs. So preregistration is extremely useful in such cases since no top-down control exists from a supervening theoretical account — there is only a hypothesis (see section Model of psychological science). It is completely unbounded and researches even if extremely careful are likely to fall into questionable research practises.

Final Comments

We both really enjoyed this and it has really helped us understand how junior researchers see and understand our work. Thank you again to Anna Henschel and Stephanie Allan!

Why Women in Psychology Can't Program

Mon, 26 Nov 2018 00:00:00 +0000

About two months ago my brother, who works in data science on social psychology data, asked me why his colleagues, who are women and have PhDs in psychology, cannot code and why they use SPSS. He was obviously just venting because when I replied he was surprised. I told him that it was because of sexism and because of lack of proper skills-training (both for statistics and coding) in many psychology departments. That got me thinking about this more…

"[That feeling when] your plant friends die cause you spend way too much time in virtual realities" by écharpe

Why can’t they code? Because they aren’t taught how to code. Why aren’t they taught? Because of two main, related, reasons:

Rhetoric and pressure from formal societies like the BPS as well as gatekeepers who worry about how the field might/could split as it is true that a undergrad degree (especially) is essentially zero sum. We just can’t fit it all in and coding is seen as not that relevant or worthy of being inserted into the curriculum.
Researchers and educators have internalised ideas about what undergrads, who are predominantly women, can and cannot learn. And they list off, based on their ideas, reasons about why it’s just not possible to teach students to code.

Me and my aforementioned brother. Note how I'm unable to share the computer (hand on mouse, remember white PCs?) and he's close to tears! Don't worry he grew up to be a data scientist, so I can't have traumatised him too much.

Every time I try to impress on others how important and useful (and fun!) coding is in my field (broadly psychology, cognitive science, neuroscience) there is pushback from very specific people. The good thing is that students in my experience (all levels, from undergrad to PhD) in psychology overwhelmingly agree with me, giving me hope that my messages on this issue get through. As do others of course too! The bad thing is this specific nay-saying group — they tend to be senior, e.g., profs, male academics (more senior and male are inherently correlated of course given the demographics of the field) who have the power to change things and introduce coding to students in a useful way: in a directly applied sense, for example, to create experiments (replacing, e.g., E-Prime, which also is not open source nor free) and to analyse data (replacing, e.g., SPSS, which again is not open source nor free). Replacing closed science tools with open science tools should be in and of itself a useful act too (recall my issues with Matlab).

*makes hissing noises at SPSS*

Before I jump in and address the arguments against teaching students, who are mostly young women of course, how to code, I’d like to explain a few things. Firstly, I want to underline that many universities do indeed teach coding to psych students very successfully. UNSW (Danielle Navarro), Glasgow (Dale Barr) and Edinburgh (Martin Corley), for example, all three of which teach undergrads R. This is something the nay-sayers all think is literally impossible and will cause undergraduates’ heads to implode or something, so much so they ignored Danielle’s book and work in this monster Twitter thread, which kick-started me into finally writing up this blog post.

Secondly, saying (indirectly of course) that women can’t code is an infamous trope related to the stereotype of the male geek. In other words, systemic sexism (undergrads “can’t be expected to learn how to code”, most undergrads in this field are women) plays a huge role. The male geek trope and related ideas contribute to driving women away from coding and generally STEM subjects in the West.

I am not from the West originally and did not have to suffer the effects of this pervasive trope growing up. I did however study computer science in the UK, which really hammered home to me the effects of gendering subjects. It had never occurred to me before that “women aren’t good at maths” or “women can’t code”. It was just not a concept I had come up against. My computer science undergraduate course was about 90 men and 6 women, 3 of whom were from Cyprus like me. That is even more surprising given there are only about a million Cypriots in the world. More stats on this whole, very depressing, situation here. Even more depressingly, coding used to be done predominantly by women. Removing them — yes, actively removing them — not only harmed and harms women to this day but also back then killed the UK tech industry.

*makes hissing noises at the BPS*

Thirdly, there is a specific issue with the BPS in the UK. It makes it hard for various reasons to make changes in the curriculum as it controls accreditation. It also biases undergraduate degrees towards training students to all be clinical psychologists and therapists. But can you really choose what job you want to do for the rest of your life at 18? Anyway, that’s another huge issue, but suffice it to say psychology and all undergraduate degrees would serve their students better if they were broad as possible and gave students the most widely applicable skills, like coding!

So why do these gatekeepers, people who have the power (even if just 1% more than myself) to change undergraduate and graduate teaching, say “no to coding”? Why do they think the students in psychology just can’t cope? In my opinion, there is an aspect of sexism at play. As I mentioned above the idea that women (recall undergrads in psych are basically all young women) can’t code is deeply embedded in culture and in people’s minds. But I would like to put aside whatever inherent biases they are holding onto and address the explicit reasons stated. I will paraphrase the various arguments I have heard, over the years as well as in the monster Twitter thread, below and provide reasons why they are not appropriate.

People choose psych as an easy undergrad, adding in coding makes it hard, ergo we will lose students.

"Back in the grimdark pre-Snapchat era of humanity (i.e. early 2011), I started teaching an introductory statistics class for psychology students offered at the University of Adelaide, using the R statistical package as the primary tool. I wrote my own lecture notes for the class, which have now expanded to the point of effectively being a book." — Danielle Navarro.

This kind of claim is a little strange because the same goes for stats. In fact I would argue stats is harder but that is my bias given I did an undergrad in compsci and had no stats training till I was pulled through stats against my will by my cognitive science MSc. I feel like “against my will” is true for a lot of things I eventually ended up liking, and that seems to be the case with people in general who responded to that thread, like the undergrads at Glasgow who demanded an even more advanced stats module or at UNSW where they have many options (R or Python). Also don’t forget, especially with things like coding and stats, those who are interested tend to begin teaching themselves more as they get more comfortable.

This kind of claim is also a bit off in a more general sense. I would argue somebody wanting to be a clinical psychologist or therapist (which is what the claim is: that most people seem to choose psychology for becoming a therapist or doing clinical work) is not necessarily choosing the easy life.

Students want or need GUIs, otherwise they don’t like the course and/or they can’t learn as well.

Assuming the reader is aware of what I think about GUIs, see my infamous Matlab post if not, you know already why I don’t think this is a good idea. Students might want GUIs but that is not a reason in of itself to teach them to use a GUI. Of course, GUIs are not all bad, they have their uses. RStudio, for example, is a very appropriate GUI/IDE to teach stats.

Often students tend to hate having to rote learn SPSS menus, after which anyway nothing seems to stick. I wonder why! Anyway, at the end of the day: students can learn well without a GUI. For example, at Glasgow they fell in love with markdown. I learned how to do loads of things without a GUI as did loads of people, I’m not special. Rote learning has its uses when the thing being memorised is generalisable, very specific stats GUI menus are not, multiplication tables, for example, are.

They just CANNOT learn to code. They just can’t. Not everybody can learn to code.

"Maddy Petrovich, 14, of Wellesley, Mass. first started learning how to use the programming language Scratch when she was 10 years old", see Computer Programming For Kids 8 And Up.

Coding is hard. Yes. Teaching people anything is hard. Teachers are undervalued. Why are the slightly more to much more powerful than me so against teaching them to code? Are they worried they will be held accountable for teaching? Do they not want to teach because they might be exposed as frauds in terms of their low teaching skills? Maybe…

English is hard. Not everybody is Maya Angelou or William Shakespeare. Not everybody is going to win a Nobel in Literature. We still learn English grammar at school, write essays, learn how to spell, etc.

Maths is hard. Not everybody is Grigori Perelman or Maryam Mirzakhani. Not everybody is going to win a Fields Medal. We still learn arithmetic and elementary algebra at school and some lucky people (I had no idea the US is this bad at teaching maths) learn calculus and linear algebra too.

Saying that some people can’t learn to code is a ridiculous, pessimistic, and elitist argument that only results in gatekeeping. If other subjects can do it, we can too.

Everybody already knows the very basic concepts of coding pretty much as they are derived from maths and linguistic structures that we learn at school and even before that. If-statements are pretty standard linguistic constructs, for example. Yes, there are more complex things, of course, but so what? There are more complex uses of language and maths too, but we still teach literacy and numeracy to kids. I can’t conjugate εκπλήσσομαι (to my shame) especially in the past tenses in Modern Greek and I’m a native Greek speaker.

Also, yeah, sure, some people can’t learn anything but the most rudimentary English or maths. So what? Does that mean nobody should be taught these things because some people can’t learn them? Of course not. The sooner we start teaching kids to code, the better. I think everybody should be taught to code in primary school and thankfully so do quite a few countries.

Teaching students to code is zero sum and that means removing other parts of the course.

Yes, and no — depends on what course! In my opinion, it’s a price we should be willing to pay to give students useful transferable skills for which to get a job with. Knowing how to code is so vital to be a productive member of the workforce. We owe it to our students, both in a moral and a “they pay us” way, to equip them for the job market properly. Being able to code is a useful skill both to get a postdoc or to get any other job pretty much. It’s also possible to teach them coding as a direct means to an end, as I mentioned: to design experiments, to run statistical analyses, to do computational modelling, etc. This by the way is exactly what the nay-sayers think is impossible/implausible even though both I and Danielle have taught and designed exactly these types of classes.

Teaching students to code is really hard and nobody in my department knows how to teach them probably.

Margaret Hamilton pretty much invented the modern concept of software as well as writing the code that got humans to the moon. "For Hamilton, programming meant punching holes in stacks of punch cards, which would be processed overnight in batches on a giant Honeywell mainframe computer that simulated the Apollo lander’s work" from Wired.

Yup, it’s hard. All teaching is hard. It takes time and is going to require sacrifices. It’s a massively undervalued and underpaid profession (again no coincidence it’s undervalued and it’s feminised generally). But you know what? We do it just fine at UCL. With a lecturer teaching them to code experiments in Python and Javascript (Christos Bechlivanidis, check out his teaching section). And they do it just fine at Glasgow, at Ediburgh, at UNSW, like I mentioned, where they teach them R for stats. Teaching will never be perfect, like anything in life — and always those doing the teaching will learn stuff themselves as they teach.

We can’t expect clinical people to learn how to code.

We can and it’s really useful for them, they say so often themselves to me and on Twitter (go look at the monster thread in the places they spoke up, e.g., here). Also see my replies to all the other points especially thinking about the fact we do teach them statistics even though clinical people will not need statistics to, e.g., perform their duties as a therapist.

Also there is no reason to assume all undergrads want to be clinical all else being equal — maybe they would fall in love with coding and computational modelling if they were exposed to those parts of psych? Just like they fell in love with markdown?

Who are you to say we should or could teach them to code?

Firstly, of course, I’m nobody and you don’t have to listen to me. But, also secondly: I’ve done it. I’ve taught psychology A-level students to code. I’ve taught undergraduates in their second year to code in Python to the point where they learned to code their own neural networks from scratch (that was the explicit goal). I’ve taught PhD students to code, mostly by begging them to teach themselves. I’ve also taught MSc and PhD students to code in a class as a TA (no begging there). It’s very possible and they actually enjoy it. Funnily enough and in my opinion all the hype around coding and machine learning and artificial intelligence makes them want to learn.

We can’t teach them to code because scoping [or any other programming concept] is really hard and time-consuming to learn.

When I taught a small class of undergraduates neural networks from scratch (no library, just for-loops) we didn’t do scoping, or indeed anything more than variables, conditionals, and loops pretty much, and they [s]coped just fine. They had zero programming experience in Python and only 2 of 15 had coded before. They all got a first or a 2:1 (two highest grades in UK system), and not because I’m too nice.

From a pedagogical perspective it’s not ideal to introduce scoping or other complex stuff straight away anyway. When I was in high school and during my separate computer science A-level, I recall scoping was not introduced until much later. This is a common pedagogical principle. You teach, for example, about 3 states of matter (solid, liquid, gas). You don’t go “oh, yeah, there’s also Bose-Einstein condensate”.

You can’t teach them how to code during a stats class because some students will have a “handicap” if they have not coded before.

Olivia from the before [pre-Dr] times at ICCM in 2012, photo by fellow coder and computational modeller, Burcu Arslan.

Somebody will always have a “handicap”. That is life, sadly. Whether it is maths in your native language versus in English or just the new jargon for a sub-area of science. One always learns new English words in every course as the technical terms and expressions are often totally new. I would not have known what, e.g., abscissa means in maths if I had not done maths in English before moving the UK. On the other hand, I do have an advantage over people who just speak English because I know all the Greek letters we use in maths and I have prima facie access to the meanings of scientific words made-up based on Greek, e.g., prosopagnosia.

It’s normal for the skills of students to be varied and a good teacher can and should cope. It’s also why a good teacher should shield students from biases like the male geek trope.

To end, I just want to say that I really appreciate the massive unwieldy thread even though I feel in places people have not read what I said and then repeated it back to me. Unfortunately, in some spots I do feel the thread went a bit too sexist. The women in the thread (me and Danielle, possibly others) also seemed to have direct experience on most of the issues brought up (she even wrote a book on it!), so it was in spots pretty egregious. I guess that is why this post needs to be written, the field has issues that need to be addressed (as does society in general).

Anyway, especially to the undergrads and other students who spoke up: thanks all for the feedback both before in the monster thread and after! This idea for a post has been in my mind for a few months now, so I am glad to have been given an extra boost of inspiration to write it all out.

A Mathematician’s Lament, Paul Lockhart
When Women Stopped Coding, NPR
A Brief History of Women in Computing, Faruk Ateş

Minority Cognitive Modellers

Sun, 22 Apr 2018 00:00:00 +0000

Christina Bergmann and I have wanted to create a list of women and non-binary people in computational cognitive modelling for probably around 3-4 years now (she has written more on this on CogTales, Building a network of women and nonbinary cognitive modelers). It was tough when we first tried to do it: because it is not always clear who is a modeller just from looking at a scientific publication, as often being/self-identifying as a modeller is not exactly the same as being on the author list of a modelling paper; because Twitter lists, which were our first idea, sadly live and die by the single person who created them on Twitter; and because at the time neither her nor I really knew how to reach out to get such lists from others, broadening our horizons.

The other day — thanks to Nora Newcombe for making me think of this by asking about a related topic — I decided my Twitter account was probably a good place to ask the scientific community for the names of women and non-binary people in cognitive modelling. To my pleasant surprise, the tweet (below and here) managed to generate a larger amount of names in the replies than I alone could ever:

Hey, science tweeps! 🧠

I want names of women & non-binary cognitive modellers.

👨🏿‍💻👩🏻‍💻👩🏼‍💻👨🏽‍💻👩🏾‍💻👩🏿‍💻👨🏼‍💻👩🏽‍💻👨🏾‍💻

Tag them here or if they aren't on twitter post a link to academic websites. Pls RT so we've a good chance of finding many! ☺️
— Olivia Guest (@o_guest) April 19, 2018

The dedicated website is: compcog.science — and all minorities in cognitive modelling are welcome! Finally, thank you everybody for all the names and support!

Block Practical: Connectionist Models and Cognitive Processes

Tue, 07 Nov 2017 00:00:00 +0000

This is less of a blog post and more of a materials dump from an elective practical I taught to second year undergraduate students in the Experimental Psychology Department at the University of Oxford. I thoughtlessly deleted the webpage that contained them, assuming no student after 2 years would need them. How wrong I was! I received an email the other day from a Ph.D. student at a university on the other side of the world pretty much asking where these materials had disappeared to. This made me question my assumption nobody was looking at these materials. So to save myself and others from looking for them again, here they are for everybody.

This elective practical taught second year undergraduates to program in Python at a basic level and to understand the basics of artificial neural networks. They proved highly suitable as my students had not done much/any programming before and had not really heard of neural networks (things might have changed now, hype, etc).

To clarify, I do not teach this course any more and I will not be updating or using these materials. If you want to use them for your own teaching, they are CC BY 4.0, and I would super appreciate an email or tweet me if you use them.

A really basic neural network diagram, by Wikipedia user Glosser.ca

Course Materials

Reading Materials

Programming

Exercises

Books

Inspiration

Libraries

Blogs

The Glowing Python: This blog has various examples of interesting code to play with and give you ideas for your own projects.
WildML: Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs: This blog also has other Machine Learning tutorials.

Video Lectures

Machine Learning, by The Royal Society
The Cognitive and Computational Neuroscience of Categorization, Novelty-Detection, and the Neural Representation of Similarity, by Mark Gluck

Online Courses

Machine Learning, by Andrew Ng
Neural Networks for Machine Learning, by Geoffrey Hinton
Introduction to Neural Networks, by Sebastian Seung

How to install Python

Windows Users

This is a little tricky:

Install Python: download from here
Install matplotlib, numpy, and scipy using pip. Specifically you need to download the following from here:
- matplotlib-1.4.3-cp27-none-win32.whl
- numpy-1.10.0b1+mkl-cp27-none-win32.whl
- scipy-0.16.0-cp27-none-win32.whl

This requires you to be in the Scripts folder of the Python27 installation. And to use the windows command prompt. For me this looks like:

C:\Python27\Scripts>pip install NAME_OF_WHEEL_FILE.whl

For all three of those you need to run a pip command like above.

Install PyGTK: download from here
To check that everything works, open network.py and see if it runs without any errors.

Mac Users

I finally managed to do this on my mac. Use Homebrew to install matplotlib, numpy, scipy, pygtk.

Linux Users

Use your favourite package manager to install matplotlib, numpy, scipy, pygtk.

Git (and GitHub) Cheat Sheet

Sun, 05 Nov 2017 00:00:00 +0000

I wrote a very simple, very basic introduction after I showed my lab some of the basics of version control (git — in the most simple terms, it is a system to keep track of your code) and a website providing such services (GitHub). It is meant to be a cheat sheet mnemonic with extra information to help remind them what each of the very basic commands I showed them does. Twitter showed extreme appreciation for it — perhaps because many tutorials go a bit like this — so maybe it is useful to a wider audience of newbies not just my lab. Two main assumptions I make are that the reader is interested in maintaining their code and knows what a terminal is, as all the following commands are meant to be typed into a *nix terminal. See this for a tutorial on how to use the command line.

Octocats are some cute GitHub creatures.

Create a New Repo

To create a new repository on GitHub go to: https://github.com/new

Get Unlimited Private Repositories

Sign up for the academic/student pack on GitHub as many lab projects (also known as repositories or repos) might need to be private (until publication, of course) and non-academics do not get unlimited free private ones. Sign up here — they only need your academic (.ac.uk or .edu, etc.) email address to associate it with your account.

Clone

Cloning means getting something for the first time. So to download a whole repository for the first time (e.g., my Gini repo):

git clone git@github.com:oliviaguest/gini.git

This command makes a new directory called in this case: gini. So you would need to change directory in order to see your newly downloaded files, i.e., cd, like so:

cd gini

Then you can ls and ls -a to see all your files. Feel free to actually clone my Gini repository as you cannot break it on GitHub since you do not have push rights.

Add Files

Adding a file means asking version control to start watching it, but it is not yet in the history of your repository. Just adding is not enough! All add does is say add this file to queue of files to be committed (the next section). You can be explicit and name the files you want to add (all other files will not be added):

git add filename_1 filename_2... filename_n

Alternatively, you can add everything (except the things in your .gitignore file):

git add -A

If you just add a file, it is not safe yet! It needs to be committed (next section) to be safely under version control!

Commit Files

Committing is adding all the files to version control, although not to the server (the next section).

To commit everything you have just added:

git commit -m "I've just made some very dramatic changes"

For your files to be 100% safe make you must also push them (the next section). Only committing makes files be under version control on your local machine, e.g., your laptop, but they will not be accessible from another computer.

Push Files

Pushing commits, and therefore files, makes your changes enter into the version control system on the server as well as your local machine, so on GitHub (or Overleaf, or GitLab, or whatever server you are using). So pushing is the superior form of backup and version control because it means that there are at least two copies of your work and its history: one local copy (the stuff you were just working on) and one server-side copy (what you just pushed).

Once everything you need is added and committed, it is time to push. Many adds may be in one commit, many commits may be in one push. But there is no reason to limit yourself to pushing once a day. Push as often as possible pretty much, is my advice.

Unsurprisingly, as you might guess, to push you type:

git push

Check the Status

To check what is going on, what changes have been made, compare your local repository’s status with that of the server, etc., type:

git status

This command will often tell you what you need to do given the current changes you have made, e.g., tell you you need to push your commits. If you are unsure of anything, running git status should give you a hint about what the next command you want to run is.

Importantly, if you have made server-side changes, i.e., you did some work on machine A and pushed all that work to the server and now you are on a completely different machine B and need to get back to working on your repo, you need to tell the repo on this different machine B to check the server for changes. This can be done by asking your local git, on B, to fetch the changes from the server:

git fetch

Bear in mind that fetch does not download any files, it merely updates what your local git knows about the changes you did on machine A which you then pushed to the server. After fetching, you may run git status as then the information on the differences between your local files and those on the server will be correct. Otherwise, if you just run git status you risk getting the wrong information about what is on the server versus your local repo.

Discard Changes

If you made some local changes and you do not want them around at all — you just want what is on the server, you can run:

git stash

This discards all your local changes that have not been added or committed.

Pull Files

Pulling means getting stuff from the server. If you have made changes at work and then go home and want to continue working where you left off, you run:

git pull

Unsurprisingly, pull does the opposite of what push does. It downloads all the files you previously pushed to the server from work on your home computer.

I Hate Matlab: How an IDE, a Language, and a Mentality Harm

Fri, 17 Mar 2017 00:00:00 +0000

This blog post is inspired by a few Matlab-related tweets of mine, which turned into days-long discussions with fellow science and non-science tweeps. Those tweets of mine in turn are motivated by two main things: my desire for programming in psychology, neuroscience, and science in general to be taught and taught well, and my desire for students to learn transferable skills more generally. This blog post is premised on a number of themes which came up on Twitter. The great need for scientists to be able to code. The fact that Matlab is akin to bad training wheels on a bicycle, which never aid with learning to ride, but are used over again because they are better than walking. And the idea that while there is a best tool for every job, not every tool is best for any job. The discussion on Twitter was motivating and so I promised everybody I would write up what I think. So this blog post is about how I think teaching Matlab, the whole ecosystem not just the language, within psychology harms students more than it helps them in many cases in my experience.

To clarify, Matlab used to be the best tool for many things. Before things like the NumPy/Matplotlib/Jupyter trilogy, it was probably the only tool that had “everything”. When Matlab first came out, the alternative was Fortran (which has goto statements, if you don’t know why this is scary, never mind, you’re lucky). But I believe it is now more a cause of brain-rot than mind-expanding awesomeness (please do not watch Arrival just to get this Sapir-Whorf reference). It is now more user- and science-jail than a freeing experience that allows us to make prototypes fast (it of course still does the latter).

The Matlab logo is a visually appealing render of an eigenfunction of the wave equation.

If you are a proficient coder and love Matlab, then this blog post is not really for you. Importantly, my intended audience are those who wish to see an improvement in the teaching of programming within psychology. I am talking from the perspective of my experiences within my field: psychology and cognitive science. I have designed from scratch: a course, that I taught when I was working as a postdoc at Oxford; and a workshop, while I was a PhD student; both with the aim of teaching the principles of coding before diving into Python specifically for psychology students. I also want people in science to have dependable transferrable skills, to be able to move to other languages, as well as having as much fun as possible while learning. Because of my training, I am privileged enough to be able to pick up a new language in a couple of hours. I want others to have such skill-related opportunities too, not only because it is useful for science as an endeavour to have skilled researchers, but for us as individuals: if one emerges from their degree a coder one will have more opportunities (both within and outside science).

To reiterate my titular claim: the way we teach Matlab in psychology appears to be more harmful than helpful. I would like us to move beyond Matlab because the ecosystem it provides is a dangerous attractor, which many of my peers and my students involuntarily get sucked into. In this post I will outline the main reasons why the Matlab ecosystem and language are as provocatively described above. I intend to use “Matlab” to mean the whole ecosystem: the IDE, the language, and the mentality it brings about because I think they are inseparable. In the same way “C programmers [allocate] their own damn memory, probably right after building their own computer out of rocks and twigs”, Matlab coders within psychology also have and create a culture around them aided by the IDE and the pre-existing community they have joined.

Limited Skill Transfer

Firstly, Matlab is not sufficient to provide us with a transferable programming skillset. Matlab provides a programming environment in which nothing, at least superficially, seems hard — and thus nothing meaningful about coding itself is learned. We do not need to worry about namespaces, nor even functions too much. And we do not need to learn anything too complex to get some OK-looking figures. This is great for prototyping — we can produce something that works well enough impressively quickly. But this comes at a huge cost to us as a newbie coder. We have not learned any of the important skills that would enable us to pick up another language. And we will undeniably need to pick up other languages because that is the state psychology is in — e.g., R is becoming the standard for statistical analyses. Yet we just learned a language that does not help us do that since it did not push us to learn the basics of what other languages have at their core.

IDEs are extremely useful if you are a proficient coder already. However, they can act more like bad training wheels on a bicycle, hindering deeper learning.

To put this another way, when one is learning to drive they do not tend to learn to drive using an automatic gearbox. They learn to drive with a manual gearbox and it is tough. Learning the harder of the two types, manual, allows us to then easily transfer to the easier of the two if need be. In the case of USAmericans, they mostly learn to drive an automatic gearbox and almost never learn manual (because their skills do not transfer easily). Although the metaphor is simplistic, it suffices to explain why Matlab is not the best language to learn, it is a car with an automatic gearbox. We cannot easily transfer what we have learned to driving stick and in fact licences for just automatic transmission exist in my home country and the UK: if you learn just automatic you cannot be expected to know stick, whereas if you learn manual transmission you know “everything”.

Furthermore, I posit that Matlab knowledge can make it harder than absolutely no programming knowledge for us to shift to another language. Matlab has an IDE that provides GUI functionality that allows us to edit variables dynamically like in Excel, which we know causes demonstrable problems. It causes some of our students to think that the Matlab IDE is what programming is, in much the same way some of our students think SPSS is what statistics is. Furthermore, high dependence on manually editing things is extremely bad because our workflow will not be reproducible nor replicable.

In addition, all the bells and whistles of the IDE and the GUI never force us to think about variables deeply (since we can always visualise them). This exercise in keeping a mental model of what the code is doing, writing down what the code should be doing, imagining the data structures, etc., is a skill one needs to be developing. More than once I have been asked to help people who were editing their variables in the GUI and hence did not properly understand their own code nor how to debug it. This is not their fault, but had they learned to code without this they would never have picked up such terrible habits. They had not learned exactly what a loop was and a lot of other helper scripts worked just fine, so they had no feedback that editing in the GUI is maladaptive per se.

In most other languages: there is no GUI and there is no IDE that has the language baked in. This results in many of us using Matlab by just pressing buttons and hoping something useful will come out the other end. And this observation, shocking though it may seem, that this is what we and our students do, has been backed up by so many of you over chat and Twitter. The GUI and IDE crutches will be snatched away from us as we will have to learn to code all over again — something we need never have to do if we had learned using a manual gearbox/not Matlab.

Matlab puts a ceiling on what kinds of projects we can do both in size and in scope. Optimising for hardware, needing to lower space and time complexity, wanting something very specific like web-scraping, etc., are all tougher within Matlab. This is because Matlab is more a domain-specific than a domain-general language, it is centrally controlled, and the GUI and IDE cannot cope with large projects easily (although there is a command line mode, which we will be predominantly uncomfortable with given we only know Matlab).

To further underline my point, Matlab explicitly teaches us some very unorthodox programming principles. Some “features” do not exist in (m)any other languages, and certainly not in any we will likely want to learn in the near future (Python, C/C++, R, Julia — even LaTeX). For example, we are not allowed to have more than a single externally accessible function per file, and that file must have the same filename as the function we wish to access. In essence this means we cannot have more than a function per file if we are, e.g., trying to code up a library in a clear way. Matlab does not permit us to store all our global variables in one file, e.g., if we need constant values. Due to all this, Matlab promotes spaghetti code. This adds to why many of us feel embarrassed to share our code online. We never learned to write neat code because Matlab allows us to be quick and dirty without any repercussions.

Perhaps most flagrantly, arrays in Matlab start at 1. One has no idea how maladaptive this is until they move outside Matlab. Computer science starts from zero for a reason. If we want to learn generalisable skills, learning that indexing starts at 1 will hinder us, perhaps even cause us to introduce very nasty hard-to-find bugs when we move outside the Matlab ecosystem. All these put together cause us to get more confused by new languages as the baggage we carry with us from learning Matlab needs to be actively unlearned and inhibited.

Closed Source Means Closed Science

Secondly, Matlab is closed source, proprietary, and prohibitively expensive if you have to buy it yourself. They obfuscate their source code in many cases, meaning bugs are much harder to spot and impossible to edit ourselves without risking court action. Moreover, using Matlab for science results in paywalling our code. We are by definition making our computational science closed.

The principles of open science, by Andreas E. Neuhold.

Many people in the mutually inclusive open science and open software movements hope to see Matlab surpassed sooner rather than later and some even think it is inevitable. By extension, people in these movements tend to think freely deciding to use Matlab (and indeed any closed source software) in science is at least questionable and at most unethical. I believe in free and open software and science, so I am in principle opposed to Matlab’s grip on science. This does not mean I believe the science done with Matlab is in any way worse in and of itself. By the same token, scientists who believe in open access do not think that science published in closed access journals is “bad science” — they think it is not the best publishing practice. Sadly, one can either be for open science or against it. So unless Matlab’s “core values and conviction to “Do the Right Thing”” start to also include open source and science, Matlab is incompatible with our aims.

Something that pains me immensely, and indirectly affects all Matlab coders is the incompatibility between Matlab versions. The main reason for this is, unlike Python or C++ or pretty much all languages out there, there is no Backus-Naur form for Matlab to my knowledge. This means that Matlab has no official and formally-specified grammar, it could, but it does not. This is incredibly bad if true, and explains the compatibility problems, making Matlab more like Microsoft Word (which is not backwards compatible and not a programming language). It also means Mathworks does not have to stick to any rules for the grammar of Matlab, they can change it on the fly. And by the same token, Octave compatibility is hard to maintain because the language is not defined.

Importantly, over and above the fact it is not open source, I propose Matlab (and thus similar languages like Octave and SciLab which are open) should not be our go-to languages for the reasons outlined herein. To re-iterate, Matlab is not the best language to teach our students and peers for pedagogical, skill transfer, and practical reasons over and above the ethical/openness reasons. These reasons in and of themselves serve to discredit Matlab and demote it from its place as the primary programming language for teaching in psychology.

Conclusion

In a nutshell, Matlab creates an environment where we learn how to code without ever doing anything too difficult, without ever developing skills that really transfer, and without ever understanding the core of what coding is about. I want us to be better equipping ourselves and our students for both life in science and giving them useful skills for life outside science. The default position in my part of science is you teach Matlab and then that is it. I do not levy these criticisms against those whose of us who use and teach a multitude of languages (including Matlab). I am focussing on the majority of us who teach and use for all intents and purposes only Matlab.

GUIs and IDEs are great — just like once we already know how to drive using a manual transmission we can easily switch to automatic — but they predominantly do not push us to develop our skills further. If we want to we can switch to a fancy IDE after we already know the tougher stuff. We learn multiplication tables off by heart before we switch to using our smartphone as a calculator. I am assuming we all want to develop our technical skills appropriately, so inevitably we will need to carry out much more complex tasks, like writing a bash script or compiling something from source — all these things are skills we need to be building up slowly over time. Matlab allows us to live in a lovely world where everything is easy but from which we cannot escape. Research will throw harder programming tasks at us than quickly making graphs or fast matrix multiplication. Thus we need to accept that sometimes learning new things can be hard (as well as fun).

Some will push for their own favourite language, e.g., Python. Nonetheless, as long as we move away from the paradigms the Matlab ecosystem enforces, we will have made serious gains pedagogically. I hope I have convinced my intended audience that even though Matlab has been the go-to language things should be and are rightfully changing. For examples, even within engineering where Matlab has a historically strong hold its widespread use is being eroded — the engineering department at Cambridge decided to teach Python instead.

Programming education in psychology can be better. Other languages provide more replicable and reproducible workflows, more opportunity to learn transferrable skills, and communities centered around open source and open science. If we can teach the Matlab ecosystem, then we can make a small step for great gains and teach a better more open ecosystem. We must teach the core concepts of programming and we must teach them well. We are in the midst of transition from closed source to open source, closed science to open science, black box workflows to reproducible and replicable workflows. Let’s make this transition happen by equipping our students and ourselves with the most appropriate skills.

Thanks

This blog post would not have been possible without discussions with my Software Sustainability Institute co-fellows and the institute’s staff, nor without the many many tweets from you all.

White Paper: MATLAB to Python, A Migration Guide
Learn Python on Codecademy
matlab rant 2 by Benjamin Vincent
MATLAB is a terrible programming language by Nikolaus Rath
Morlocks And Eloi At The Keyboard by Neal Stephenson

Artificial Neural Networks with Random Weights are Baseline Models

Mon, 06 Mar 2017 00:00:00 +0000

Where do the impressive performance gains of deep neural networks come from? Is their power due to the learning rules which adjust the connection weights or is it simply a function of the network architecture (i.e., many layers)? These two properties of networks are hard to disentangle. One way to tease apart the contributions of network architecture versus those of the learning regimen is to consider networks with randomised weights. To the extent that random networks show interesting behaviors, we can infer that the learning rule has not played a role in them. At the same time, examining these random networks allows us to evaluate what learning does add to the network’s abilities over and above minimising some loss function.

Figure 2A from Guest and Love (2017): "For the artificial neural network coding schemes, similarity to the prototype falls off with increasing distortion (i.e., noise). The models, numbered 1–11, are (1) vector space coding, (2) gain control coding, (3) matrix multiplication coding, (4), perceptron coding, (5) 2-layer network, (6) 3-layer network, (7) 4-layer network, (8) 5-layer network, (9) 6-layer network (10) 7-layer network, and (11), 8-layer network. The darker a model is, the simpler the model is and the more the model preserves similarity structure under fMRI."

In What the Success of Brain Imaging Implies about the Neural Code, we examined an artificial deep neural network, Inception-v3 GoogLeNet. This deep trained network, preserves the similarity of the input space and thus is functionally smooth. Importantly, however, we found that functional smoothness in this deep network breaks down at later layers. Is this because of the depth of the network, the many layers, or the specific learning regimen? We sought to explain why this happens by using a baseline, a model with random weights.

To answer this question, let us consider some much simpler plausible contenders for the neural code — a rudimentary set of models — the components of artificial neural networks: matrix multiplication and some kind of squashing (sigmoid, step, etc.) function (in our case, the hyperbolic tangent).

The first basic model, matrix multiplication, is how neural networks propagate activation from layer $\mathbf{m}$ to the next $\mathbf{n}$ via the weights $\mathbf{w}$ . For simplicity, our toy network contains layers $\mathbf{m}$ and $\mathbf{n}$ , which both contain three units. Thus to calculate the states for $\mathbf{n}$ , we take the matrix product of the previous layer $\mathbf{m}$ and the weights $\mathbf{w}$ :

$\mathbf{m} \times \mathbf{w} = \\ \begin{pmatrix} x_1 & x_2 & x_3 \\ \end{pmatrix} \times \begin{pmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \end{pmatrix} = \\ \begin{pmatrix} x_1 w_{11} + x_2 w_{21} + x_3 w_{31} \\ x_1 w_{12} + x_2 w_{22} + x_3 w_{32} \\ x_1 w_{13} + x_2 w_{23} + x_3 w_{33} \end{pmatrix}\ = \begin{pmatrix} y_1 & y_2 & y_3 \end{pmatrix} = \mathbf{n} \,$

where $x$ s represent the units in layer $\mathbf{m}$ , $w_{ij}$ represents a weight in $\mathbf{w}$ from unit $i$ in layer $\mathbf{m}$ to unit $j$ in $\mathbf{n}$ , and $y_j$ is a unit in $\mathbf{n}$ . For example, $w_{31}$ is the weight on the connection between the third unit of the shallower/earlier layer and the first unit of the deeper/later later (others use other notations).

Matrix multiplication calculates the states of a layer — easily done in Python using NumPy, specifically numpy.dot():

import numpy as np
m  = np.asarray([0.1, 0.2, 1.3]) # layer m with some dummy input
w = np.random.randn(3, 3) # random weights from m to n
n = np.dot(m, w) # pre-synaptic states in n
print(n)

To apply a squashing function, $\tanh$ , to n above, we may use numpy.tanh():

n = np.tanh(n) # post-synaptic states in n
print(n)

Non-linear transformations like hyperbolic tangent allow the network to have non-linear decision boundaries, e.g., between classes, making it able to capturing the statistics of the training set (more here and here).

In Guest and Love (2017) we presented the above as two separate models as well as a combined model, here I have cut to the part where they are combined to form a traditional two-layer network (also known as the perceptron model). As you might have guessed, from two layers we can generalise to many, by continuing to take the matrix product of the output (n in the code above) with some new weights, and so on.

Running an untrained neural network with random weights allows us to compare more complex (i.e., trained models) with their untrained selves. We can thus pick apart what aspects of the model are inherent to the architecture itself and which emerge as a function of training. Networks that have random weights can be given the same training and test sets, although importantly no training has happened yet, and we can examine their internal states and outputs This can serve as a guide to understand what the network “knows” a priori.

As we noted in Guest and Love (2017), networks naturally place items close together in their internal representational space that are similar/proximal in the input space. Hence why artificial neural networks are a plausible candidate for the neural code, i.e., they give rise to functionally smooth representations. The simple network above can be made deeper and deeper, and we can inspect every layer in it for smoothness for every pattern. Extending the above, we can do just that, and run the network on two very simple categories:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import numpy as np

prototypes  = np.random.randn(2, 100) # two toy categories
members = 10 # how many items per category
patterns = []

for p in prototypes:
    for i, m in enumerate(range(members)):
        # for each item, create a pattern that has noise as a function of the
        # number of items. First item in category has no noise, then 0.05 SD of
        # noise, then 0.1 SD, and so on.
        patterns.append(p + 0.01 * i * np.random.randn((len(p))))

layers = 20 # how many layers we want, i.e., how deep is the network
# random weights:
w = np.random.randn(layers, len(prototypes[0]), len(prototypes[0])) * 0.1

for pat in patterns:
    # for each pattern
    for i, l in enumerate(range(layers)):
        if i == 0:
            #if we are at the input layer, then set units to pattern
            n = pat
        # propagate through each layer
        n = np.dot(n, w[i]) # pre-synaptic states in n
        n = np.tanh(n) # post-synaptic states in n
        if i == layers-1:
            # print the layer, the first five features of the pattern applied at
            # input and the first five activations in the last layer
            print i, pat[0:5], n[0:5]

Even just by eye-balling the output in the terminal using the code above, we can see that indeed similar items (items within the same category) map to similar outputs, i.e., the network is functionally smooth without any training. We used a more complex version of the above to demonstrate this principle in Guest and Love (2017), where we calculate the correlations between the representations in the input space and in each layer. However, as we move deeper into the network, we see that functional smoothness has broken down and the network gives for all intents and purposes identical outputs for each items within a category, thus losing all structure within it. We cannot looking just at the output, predict which input generated it, only which category.

Using this result we can infer that the property of Inception-v3 GoogLeNet, and indeed any similar deep network, which causes it to both display (at early layers) and gradually lose functional smoothness (at deeper layers), is due to the nature of the architecture and not the learning rule. Because this property is present in simple untrained networks, it cannot be a byproduct of training.

Importantly, randomising weights can be done to any network with any topology, including to Inception-v3 GoogLeNet itself, to recurrent networks, and so on. We hope this idea proves to be a useful exercise to others too, as many connectionist and deep network accounts would benefit from an understanding of the inherent properties of the topological configuration versus the fully-trained model.

Using the Gini Coefficient to Evaluate Deep Neural Network Layer Representations

Sun, 26 Feb 2017 00:00:00 +0000

Sparsity is an issue in neural representation and we think it should be measured in artificial neural networks to understand how they are representing information at each layers. For example, are a few units doing the work or is there a distributed pattern across all units (i.e., overlapping units taking part in the representations of cat, car, etc.). So in What the Success of Brain Imaging Implies about the Neural Code we decided to use the Gini coefficient, inspired by its use in evaluating voxel activations, to uncover the degree of sparsity within each of the layers of Inception-v3 GoogLeNet.

The Gini coefficient is primarily used to give an idea of how wealth is distributed within a group of people, usually a whole nation. But it can also be used more generally on a vector of numbers, a distribution, to describe how distributed values are (more on this below).

Figure 2B from Guest and Love (2017): "A deep artificial neural network and the ventral stream can be seen as performing related computations. As in our simulation results, neural similarity should be more difficult to recover in the more advanced layers."

I looked around online for a dependable and fast Gini coefficient calculator in Python. Unfortunately, what I did find, while useful, were neither fast nor bug-free. So I decided to write one myself!

We were dealing with relatively big data, as Inception-v3 GoogLeNet has quite a few layers, so I needed something with relatively low space and time complexity. In terms of speed, my Gini calculator is quite a lot faster than (the current implementation of) PySAL’s Gini coefficient function (see the documentation) and outputs are indistinguishable before approximately 6 decimal places. And it is slightly faster than the Gini coefficient function by David on Ellipsix.

The Gini calculator function I wrote is based on the third equation here, which defines the Gini coefficient as:

$G = \dfrac{ \sum_{i=1}^{n} (2i - n - 1) x_i}{n \sum_{i=1}^{n} x_i},$

where $i$ is the index for each data point $x_i$ and $n$ is the total number of data points. For a very unequal sample, e.g., with 999 zeros and a single one, the Gini coefficient is very high (close to 1). For uniformly distributed random numbers, it will be low, around 0.33. While, for a homogeneous sample, the Gini coefficient is 0. In other words, the lower $G$ is the more equal the distribution of wealth/numbers is. Check out the readme file for examples of what can be passed to the gini() function.

The Gini calculation by definition requires non-zero positive (ascending-order) sorted values within a 1-dimensional vector. This is dealt with within the gini function. So these four assumptions can be violated, as they are controlled for:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import numpy as np

def gini(array):
    """Calculate the Gini coefficient of a numpy array."""
    # All values are treated equally, arrays must be 1d:
    array = array.flatten()
    if np.amin(array) < 0:
        # Values cannot be negative:
        array -= np.amin(array)
    # Values cannot be 0:
    array += 0.0000001
    # Values must be sorted:
    array = np.sort(array)
    # Index per array element:
    index = np.arange(1,array.shape[0]+1)
    # Number of array elements:
    n = array.shape[0]
    # Gini coefficient:
    return ((np.sum((2 * index - n  - 1) * array)) / (n * np.sum(array)))

And that is all there is to it! The only two inviolable assumptions it makes is that you have numpy installed and that you send it something like a numpy array (use np.asarray() to check if what you have is array-like).

But what does this have to do with artificial neural networks? Well, instead of people within a nation, we can consider the units within a layer. And instead of people’s wealth we can look at units’ activations after we have propagated input to the layer. So given an input to a layer, we can measure how sparse (unequal) the distribution of activations is. A single number can give us an idea of how localist or distributed the representation the layer has learned is. Averaging over the Gini coefficients for all the possible inputs to a layer, we can calculate how localist or distributed the representations within a layer are in general.

Inception-v3 GoogLeNet has output that is trained to be completely sparse/localist, since it uses one-hot coding for the classes. Representing the output classes using one-hot coding ensures that outputs are trained to be both orthogonal and localist (two properties which are not by definition mutually inclusive). In terms of the targets it learns per input image, the network’s output will have a Gini coefficient of approximately 1. And in general, we can expect the output’s Gini to be close to 1, except in the very rare cases where the network is completely unsure of what we have shown it.

On the other hand, on other/lower layers, we find that the Gini coefficient can be high or low. It decreases and increases non-monotonically as a function of layer depth. Although it does show a rough trend of becoming higher as we move deeper, it is by no means a given. What this implies is that the network is not representing things by definition in a more localist way as we move towards deeper/later layers. In the two layers we talked about in the aforementioned Guest and Love (2017), the network has a Gini coefficient of 0.579 for the penultimate layer and 0.947 for the shallower layer (on the specific stimuli we used). At the end the average Gini for the output is, as expected given the training regime, 0.941. These and other points with respect to the representational contents of each layer are discussed in depth in Guest and Love (2017).

See here for a translation of this article by Daniel Morales into Spanish: El coeficiente de Gini como herramienta para evaluar las representaciones de las capas en redes neuronales profundas.