An introduction to Data Analysis with Python

In this great 10.25 minutes you will be introduced to doing data analysis using Python. Very nice. A must see!

More on learning how to code Python - a Cognitive scientists journey to coding

In this post I will continue the discussion on programming in Python for cognitive scientists. I will go from a perspective of data collection, to analysis, and finally to writing your results up (yes, you can basically use python for all these tasks!)

Collecting data

There are several ways you may collect data as a cognitive scientist. All depends on your research question(s). I will in this post only discuss two; collecting online data using social media and/or questionnaires, and collecting data using laboratory experiments. In fact, I will barely mention the first one but you can scrape a lot of behavior:ish data off, for instance, Twitter and maybe throw in questionnaires in that.

Creating experiments to use in data collection

Programming, or building, experiments have for long been carried out with crappy and expensive tools (e.g., E-prime). Although I understand the attraction in simpler experiment building tools where you drag and drop objects. When you are finished building your experiment you generate a script by pressing a button. All fine. However, you may at times need to do more advanced stuff and then you will need add stuff like inline code (e.g., write some scripts and add to the "timeline" in the builder). Recently, it has appeared a couple of free and open-source Python tools for creating experiments. Two of them, PsychoPy and OpenSesame, offers builders and inline scripting (much like e-prime).

OpenSesame builder GUI

Some of the others just gives you an API to ease some of the coding of your experiment (PsychoPy can be used as a library, also). That is, you import it as it was any other Python library (after you have installed it, of course). For instance, if you use the Python library Expyriment you will import what you need from the library:

from expyriment import design, control, stimuli, io, misc

On Expyriments website you can find some beginner's tutorials.

If you are interested in using PsychoPy's builder mode you can watch the following youtube tutorial:

In this tutorial you will learn how to create a classical psychology experiment; the stroop task (of course, in its original form pen and paper were used...). For a psycho-linguistic researcher the following tutorial may be more adequate:

 

More resources on Psychopy can be found on the software's resources page.  You will find out that coding using the library of Psychopy (e.g., importing the stuff you need for your experiment from the PsychoPy library is much like the short Expyriment example above).
When you have learned how to create and code your own experiment in Python you will be able to collect a lot of data you probably want to analyze your data. Although MATLAB, and more recently, R have had the majority of the cognitive science crowd when it comes to analysis (you can also create experiment in MATLAB using psychotoolbox and such) you can OF COURSE do your analysis in Python.

Data analysis

Common statistical methods in Psychology, and related fields, are linear regression, t-test, and analysis of variance (ANOVA). Especially when it comes to experiments when doing more subjective survey studies other techniques such as factor analysis (FA) and structural equation modelling (SEM) are carried out. Of course, an experimental design may also need such multivariate analysis'. If you are interested in FA and SEM in Python I must disappoint you here, however. As far as I know you can only carry out principal component analysis (which is not 'real' factor analysis according to my old stats teacher!)

Enough of my rambling you say? What CAN I do in Python?! Well you CAN do t-tests, linear regression (non-linear also), ANOVA, etc. For instance, using the package Statsmodels we can carry out all of the methods (except for repeated measures ANOVA, however). Sci-kit learn, a machine learning library, can also do a lot of statistics. Of course, SciPy can do some basic parameteric tests and Pandas (and SciPy and NumPy) can carry out most descriptive statistics you'd want to have. Repeated measures ANOVA can be carried out using the package Pyvttbl which, sadly, seem to be un-maintained. No more updates of that...

That is it, most of the stuff I list here I found via this excellent site: Python and R as tools for data analysis and  creating Psychology experiments. If you follow the link you will find discussions on Python IDEs for Psychology researchers (or any other scientist), how to do ANOVA for repeated and dependent measures, and some descriptive statistics. All in Python.

That is it for me now. 

Please leave a comment if you have any suggestions!

Five excellent Python video lectures!

In this post you will find 5 great videos containing lectures on how to carry out data analysis using Python. These seem to be part of a course called "Programming for Psychology in Python". It seems pretty awesome.

The array data type

The first video covers numpy and how to use numpy to create and handle arrays.

Creating figures

I think the heading is quite self-explaning; the second video covers the creation of figures. It is using a library called vuesz which I have never heard of. Will test it later!

Descriptive statistics   

Third, we get to know how to do descriptive stats using Python. Also more figures... But very interestingly a bootstrapping method in Python for getting confidence intervals! Great!

Inferential statistics

In this post we get to know how to to t-tests but also a simulation based approach for understanding false positives and multiple comparisons. Cool! Correlations and scatterplots.

Power Analysis

Again, self-explaining title. We get to learn how to simulate data to calculate power. Yes! We want to determine sample size! 

All of the above videos have a homepage in which you can read some and see Python code. Also, there are some exercises. Greate of you wanna learn stuff!

Great resources for learning how to program in Python

In this post I am going to list a couple of great resources for learning how to write code in Python. I start with a couple of iPython notebooks. If you are not familliar with iPython (called jupyter these days):

"Notebook documents (or “notebooks”, all lower case) are documents produced by the Jupyter Notebook App which contain both computer code (e.g. python) and rich text elements (paragraph, equations, figures, links, etc...)."

• Intro to Python, iPython Notebook, Numpy, Matplotlib (Homework 0) (solution)

• Poll aggregation, web scraping, plotting, model evaluation, and forecasting (Homework 1) (solutions)

• Data prediction, manipulation, and evaluation (Homework 2) (solutions)

• Predictive modeling, model calibration, sentiment analysis (Homework 3) (solutions)

• Recommendation engines, Using mapreduce (Homework 4) (solutions)

• Network visualization and analysis (Homework 5) (solutions)

I follow up these with more notebooks and they are more on general Python learning;
I end this post with this 3+ hours long video on data analysis with Pandas. This is a tutorial that introduces you to manipulating and analyzing large and small structured data sets.  Hope these resources are enough for now! I may update the list when I find more!
The first jupyter/iPython notebooks come from a Harvard course.

It contains homeworks, and solutions to these homeworks. By doing the homeworks you will be guided through a number of data analysis, mining, scraping, manipulation problems with Python and iPython/Jupyter notebook!

• Introduction to Programming (using Python), an entire introductory Python course. This is super nice and helpful if you want to learn Python. 
• Python for Developers, a complete book on Python programming. Completely Free Python book?! Oh yes, the Python community is so great!
• Exploratory Computing with Python. Contains 15 notebooks (I know right!) that cover exploratory computing, data analysis, and visualization. No prior programming knowledge required. What is extra awesome is that they also contain a number of exercises (with solutions) that should take less than 4 hours to complete. 
• Understanding evolutionary strategies and covariance matrix adaptation