python - Performance issue in computing multiple linear regression with huge data sets -



python - Performance issue in computing multiple linear regression with huge data sets -

i using np.linalg.lstsq calculating multiple linear regression. info set huge: has 20,000 independent variables(x) , 1 dependent variable (y). each independent variable has 10,000 datas. this:

x1 x2 x3.. x20,000 y data1 -> 10 1.8 1 1 3 data2 -> 20 2.3 200 206 5 .. .. .. .. .. data10,000-> 300 2398 878 989 998

it taking huge time (20-30 mins) compute regression coefficient using np.linalg.lstsq. can tell me improve solution according computation time?

the time spent seems follow n**2.8. can increment speed reducing number of info points.

if downsample info one thousand rows, can computations in couple of seconds. can repeat analysis different random sample.

in order combine results, have several options:

do, usual in cross correlation in statistics, , weight them inverse of norm of residuals (fast compute, in output). measure real residuals total dataset (that takes less 3 seconds) and: keep best one. weight them inverse of real distance.

the best alternative depends on how much accuracy need, , nature of data. if need gross estimation in medium noise, single downsampling should work. maintain in mind under determined already, solutions degenerated.

python numpy machine-learning linear-regression

Comments

Post a Comment

Popular posts from this blog

model view controller - MVC Rails Planning -

model view controller - MVC Rails Planning - i'm starting rails , i'm building app stores films info, user can watch film , give grade on web application. i have coded cinema model, controller , views. now, have 2 questions: should need create model , controller grading? in case, should create or should wait , create first users model , controller? yes, on model , controller grades. that'd feasible way maintain track of who's graded what. it's create first, i'd start users, since grades depend on both films , users. ruby-on-rails model-view-controller
Read more

ruby on rails - Devise Logout Error in RoR -

ruby on rails - Devise Logout Error in RoR - while using devise in ror, encountering unusual error when trying log out url different root url. displays me : actioncontroller::invalidauthenticitytoken in devise::sessionscontroller#destroy" for illustration if rooting path after log in '/structure', , seek log out '/general' error, while logging out '/structure' fine. any help highly appreciated. ruby-on-rails ruby devise
Read more

html - Submenu setup with jquery and effect 'fold' -

html - Submenu setup with jquery and effect 'fold' - i want open submenu jquery effect fold, problem if user "hover effect" fast menu remain open, how can avoid this, jquery code is: $('ul.mainmenu li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $(this).children('ul').hide('fold', 500); } ); my jsfiddle link is: http://jsfiddle.net/9wkbf/ updated fiddle the reason behind problem is, first event $(this).children('ul').show('fold', 570); queued , until completes, sec animation not start. the next snippet can workaround $('ul.mainmenu > li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $('ul:not(.mainmenu)').hide('fold', 500); } ); *important note : work current scenario. jquery html css
Read more