Data analysis tools

[Barny]

Say I'm doing some data analysis from scratch i.e. downloading the data from a website, and then doing some explorative stats on it.

What's the best way to go about this? For instance, do most people just load the dataset into something like mySQL and then do queries to get the numbers they need?

Or is it possible to load it straight into something like R, for instance?

Or maybe use python to get a string of numbers that you need?

Sorry for the wishy-washy question, any help much appreciated.

 

[Ken Abbott]

Spreadsheet.
Familiarize yourself with EDA techniques.
http://www.itl.nist.gov/div898/handbook/eda/eda.htm

 

[Barny]

Cheers Ken, I don't know why I overlooked spreadsheets. Here was me thinking that spending x hours fighting with python to suck some data off a website was perfectly normal if you wanted to do some data analysis.

Do you use third-party stats add-ins to do the analyses?

 

[Pawel]

hi,
i use WebHarvest to scrape pages + bash/wget to download some csv files,
i load all this data to mysql database so that it is easiliy queried+summarized,
i also have AmiBroker configured to use data from database,
and also R (RStudio) to analyse this data

This all works nice, but I miss one tool, which qould enable me to define and manage some measures over this data, anyone any ideas ?

 

[Pawel]

ahh and I forgot to mention that i use Hudson to run all defined jobs every day...

 

[Ken Abbott]

Cheers Ken, I don't know why I overlooked spreadsheets. Here was me thinking that spending x hours fighting with python to suck some data off a website was perfectly normal if you wanted to do some data analysis.

Do you use third-party stats add-ins to do the analyses?

Usually I just use what Excel makes available. Sometimes I have to tweak things a bit to do waterfall/walkthroughs or HLCO graphs.

 

[Barny]

Ken - have you seen this link? http://www.practicalstats.com/xlsstats/excelstats.html

The general consensus is that Excel is awful for stats stuff, unless they've fixed these problems for Excel 2011.

 

[alain]

I do most of my data analysis in R. It has almost all the tools I need and it can be hooked up to Excel if you want with the excellent RExcel plug-in.

 

[Barny]

Rexcel doesn't work on macs

 

[alain]

Rexcel doesn't work on macs

I only use Windows in extreme circumstances. My main working environment is Linux. I threw RExcel out there just in case the OP was afraid of using R.

 

[Barny]

Just started using R, it's really an amazing tool. Also if you combine it with pspp you can get survey data into R really easily.

 

[grcheng]

R/R studio is a easy tool to use and it's free. You can import data and library easily and get some plots as well if needed.
MATLAB is also good, but it's not free....

 

[geraud]

thanks!

 

[dogcopter]

I found an R app at the iTunes store called R Instructor. Anybody downloaded/heard of it?