{
  "nbformat_minor": 0, 
  "nbformat": 4, 
  "cells": [
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "%matplotlib inline"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "\n\n# Export epochs to Pandas DataFrame\n\n\nIn this example the pandas exporter will be used to produce a DataFrame\nobject. After exploring some basic features a split-apply-combine\nwork flow will be conducted to examine the latencies of the response\nmaxima across epochs and conditions.\nNote. Equivalent methods are available for raw and evoked data objects.\n\nShort Pandas Primer\n-------------------\n\nPandas Data Frames\n~~~~~~~~~~~~~~~~~~\nA data frame can be thought of as a combination of matrix, list and dict:\nIt knows about linear algebra and element-wise operations but is size mutable\nand allows for labeled access to its data. In addition, the pandas data frame\nclass provides many useful methods for restructuring, reshaping and visualizing\ndata. As most methods return data frame instances, operations can be chained\nwith ease; this allows to write efficient one-liners. Technically a DataFrame\ncan be seen as a high-level container for numpy arrays and hence switching\nback and forth between numpy arrays and DataFrames is very easy.\nTaken together, these features qualify data frames for inter operation with\ndatabases and for interactive data exploration / analysis.\nAdditionally, pandas interfaces with the R statistical computing language that\ncovers a huge amount of statistical functionality.\n\nExport Options\n~~~~~~~~~~~~~~\nThe pandas exporter comes with a few options worth being commented.\n\nPandas DataFrame objects use a so called hierarchical index. This can be\nthought of as an array of unique tuples, in our case, representing the higher\ndimensional MEG data in a 2D data table. The column names are the channel names\nfrom the epoch object. The channels can be accessed like entries of a\ndictionary:\n\n    df['MEG 2333']\n\nEpochs and time slices can be accessed with the .ix method:\n\n    epochs_df.ix[(1, 2), 'MEG 2333']\n\nHowever, it is also possible to include this index as regular categorial data\ncolumns which yields a long table format typically used for repeated measure\ndesigns. To take control of this feature, on export, you can specify which\nof the three dimensions 'condition', 'epoch' and 'time' is passed to the Pandas\nindex using the index parameter. Note that this decision is revertible any\ntime, as demonstrated below.\n\nSimilarly, for convenience, it is possible to scale the times, e.g. from\nseconds to milliseconds.\n\nSome Instance Methods\n~~~~~~~~~~~~~~~~~~~~~\nMost numpy methods and many ufuncs can be found as instance methods, e.g.\nmean, median, var, std, mul, , max, argmax etc.\nBelow an incomplete listing of additional useful data frame instance methods:\n\napply : apply function to data.\n    Any kind of custom function can be applied to the data. In combination with\n    lambda this can be very useful.\ndescribe : quickly generate summary stats\n    Very useful for exploring data.\ngroupby : generate subgroups and initialize a 'split-apply-combine' operation.\n    Creates a group object. Subsequently, methods like apply, agg, or transform\n    can be used to manipulate the underlying data separately but\n    simultaneously. Finally, reset_index can be used to combine the results\n    back into a data frame.\nplot : wrapper around plt.plot\n    However it comes with some special options. For examples see below.\nshape : shape attribute\n    gets the dimensions of the data frame.\nvalues :\n    return underlying numpy array.\nto_records :\n    export data as numpy record array.\nto_dict :\n    export data as dict of arrays.\n\nReference\n~~~~~~~~~\nMore information and additional introductory materials can be found at the\npandas doc sites: http://pandas.pydata.org/pandas-docs/stable/\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "# Author: Denis Engemann <denis.engemann@gmail.com>\n#\n# License: BSD (3-clause)\n\nimport mne\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom mne.datasets import sample\n\nprint(__doc__)\n\ndata_path = sample.data_path()\nraw_fname = data_path + '/MEG/sample/sample_audvis_filt-0-40_raw.fif'\nevent_fname = data_path + '/MEG/sample/sample_audvis_filt-0-40_raw-eve.fif'\n\nraw = mne.io.read_raw_fif(raw_fname)\nraw.set_eeg_reference()  # set EEG average reference\n\n# For simplicity we will only consider the first 10 epochs\nevents = mne.read_events(event_fname)[:10]\n\n# Add a bad channel\nraw.info['bads'] += ['MEG 2443']\npicks = mne.pick_types(raw.info, meg='grad', eeg=False, eog=True,\n                       stim=False, exclude='bads')\n\ntmin, tmax = -0.2, 0.5\nbaseline = (None, 0)\nreject = dict(grad=4000e-13, eog=150e-6)\n\nevent_id = dict(auditory_l=1, auditory_r=2, visual_l=3, visual_r=4)\n\nepochs = mne.Epochs(raw, events, event_id, tmin, tmax, proj=True, picks=picks,\n                    baseline=baseline, preload=True, reject=reject)"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Export DataFrame\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "# The following parameters will scale the channels and times plotting\n# friendly. The info columns 'epoch' and 'time' will be used as hierarchical\n# index whereas the condition is treated as categorial data. Note that\n# this is optional. By passing None you could also print out all nesting\n# factors in a long table style commonly used for analyzing repeated measure\n# designs.\n\nindex, scale_time, scalings = ['epoch', 'time'], 1e3, dict(grad=1e13)\n\ndf = epochs.to_data_frame(picks=None, scalings=scalings, scale_time=scale_time,\n                          index=index)\n\n# Create MEG channel selector and drop EOG channel.\nmeg_chs = [c for c in df.columns if 'MEG' in c]\n\ndf.pop('EOG 061')  # this works just like with a list."
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }, 
    {
      "source": [
        "Explore Pandas MultiIndex\n\n"
      ], 
      "cell_type": "markdown", 
      "metadata": {}
    }, 
    {
      "execution_count": null, 
      "cell_type": "code", 
      "source": [
        "# Pandas is using a MultiIndex or hierarchical index to handle higher\n# dimensionality while at the same time representing data in a flat 2d manner.\n\nprint(df.index.names, df.index.levels)\n\n# Inspecting the index object unveils that 'epoch', 'time' are used\n# for subsetting data. We can take advantage of that by using the\n# .ix attribute, where in this case the first position indexes the MultiIndex\n# and the second the columns, that is, channels.\n\n# Plot some channels across the first three epochs\nxticks, sel = np.arange(3, 600, 120), meg_chs[:15]\ndf.ix[:3, sel].plot(xticks=xticks)\nmne.viz.tight_layout()\n\n# slice the time starting at t0 in epoch 2 and ending 500ms after\n# the base line in epoch 3. Note that the second part of the tuple\n# represents time in milliseconds from stimulus onset.\ndf.ix[(1, 0):(3, 500), sel].plot(xticks=xticks)\nmne.viz.tight_layout()\n\n# Note: For convenience the index was converted from floating point values\n# to integer values. To restore the original values you can e.g. say\n# df['times'] = np.tile(epoch.times, len(epochs_times)\n\n# We now reset the index of the DataFrame to expose some Pandas\n# pivoting functionality. To simplify the groupby operation we\n# we drop the indices to treat epoch and time as categroial factors.\n\ndf = df.reset_index()\n\n# The ensuing DataFrame then is split into subsets reflecting a crossing\n# between condition and trial number. The idea is that we can broadcast\n# operations into each cell simultaneously.\n\nfactors = ['condition', 'epoch']\nsel = factors + ['MEG 1332', 'MEG 1342']\ngrouped = df[sel].groupby(factors)\n\n# To make the plot labels more readable let's edit the values of 'condition'.\ndf.condition = df.condition.apply(lambda name: name + ' ')\n\n# Now we compare the mean of two channels response across conditions.\ngrouped.mean().plot(kind='bar', stacked=True, title='Mean MEG Response',\n                    color=['steelblue', 'orange'])\nmne.viz.tight_layout()\n\n# We can even accomplish more complicated tasks in a few lines calling\n# apply method and passing a function. Assume we wanted to know the time\n# slice of the maximum response for each condition.\n\nmax_latency = grouped[sel[2]].apply(lambda x: df.time[x.argmax()])\n\nprint(max_latency)\n\n# Then make the plot labels more readable let's edit the values of 'condition'.\ndf.condition = df.condition.apply(lambda name: name + ' ')\n\nplt.figure()\nmax_latency.plot(kind='barh', title='Latency of Maximum Response',\n                 color=['steelblue'])\nmne.viz.tight_layout()\n\n# Finally, we will again remove the index to create a proper data table that\n# can be used with statistical packages like statsmodels or R.\n\nfinal_df = max_latency.reset_index()\nfinal_df.rename(columns={0: sel[2]})  # as the index is oblivious of names.\n\n# The index is now written into regular columns so it can be used as factor.\nprint(final_df)\n\nplt.show()\n\n# To save as csv file, uncomment the next line.\n# final_df.to_csv('my_epochs.csv')\n\n# Note. Data Frames can be easily concatenated, e.g., across subjects.\n# E.g. say:\n#\n# import pandas as pd\n# group = pd.concat([df_1, df_2])\n# group['subject'] = np.r_[np.ones(len(df_1)), np.ones(len(df_2)) + 1]"
      ], 
      "outputs": [], 
      "metadata": {
        "collapsed": false
      }
    }
  ], 
  "metadata": {
    "kernelspec": {
      "display_name": "Python 2", 
      "name": "python2", 
      "language": "python"
    }, 
    "language_info": {
      "mimetype": "text/x-python", 
      "nbconvert_exporter": "python", 
      "name": "python", 
      "file_extension": ".py", 
      "version": "2.7.13", 
      "pygments_lexer": "ipython2", 
      "codemirror_mode": {
        "version": 2, 
        "name": "ipython"
      }
    }
  }
}