database - Why I cant write dataframe in DB? -

-

i have 32 gb ram , use jupyter , pandas. dataframe isn't big, when want write in arctic data base have "memoryerror":

df_q.shape (157293660, 10) def memory(df):     mem = df.memory_usage(index=true).sum() / (1024 ** 3)     print(mem) memory(df_q) 12.8912200034

and want write it:

from arctic import arctic import arctic arc store = arctic('.....') lib = store['mylib'] lib.write('quotes', df_q)

memoryerror traceback (most recent call last) in () 1 memory(df_q) ----> 2 lib.write('quotes', df_q)

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while true: 49 try: ---> 50 return f(*args, **kwargs) 51 except (duplicatekeyerror, serverselectiontimeouterror) e: 52 # re-raise errors won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/version_store.pyc in write(self, symbol, data, metadata, prune_previous_version, **kwargs) 561 562 handler = self._write_handler(version, symbol, data, **kwargs) --> 563 mongo_retry(handler.write)(self._arctic_lib, version, symbol, data, previous_version, **kwargs) 564 565 # insert new version version db

/usr/local/lib/python2.7/dist-packages/arctic/decorators.pyc in f_retry(*args, **kwargs) 48 while true: 49 try: ---> 50 return f(*args, **kwargs) 51 except (duplicatekeyerror, serverselectiontimeouterror) e: 52 # re-raise errors won't go away.

/usr/local/lib/python2.7/dist-packages/arctic/store/_pandas_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version) 301 def write(self, arctic_lib, version, symbol, item, previous_version): 302 item, md = self.to_records(item) --> 303 super(pandasdataframestore, self).write(arctic_lib, version, symbol, item, previous_version, dtype=md) 304 305 def append(self, arctic_lib, version, symbol, item, previous_version):

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in write(self, arctic_lib, version, symbol, item, previous_version, dtype) 385 version['type'] = self.type 386 version['up_to'] = len(item) --> 387 version['sha'] = self.checksum(item) 388 389 if previous_version:

/usr/local/lib/python2.7/dist-packages/arctic/store/_ndarray_store.pyc in checksum(self, item) 370 def checksum(self, item): 371 sha = hashlib.sha1() --> 372 sha.update(item.tostring()) 373 return binary(sha.digest()) 374

memoryerror:

wtf ? if use df_q.to_csv() wait years....

your issue not memory issue. if read errors, seems library having trouble accessing data...

1st error: says server has timed out. (serverselectiontimeouterror)

2nd error: trying update mongodb version.

3rd error: retries accessing server, fails.(serverselectiontimeouterror)

etc. problem lies in arctic package (see last error checksum error). can deduce fact df_q.to_csv() works, slow since not optimized artic. suggest trying reinstall arctic package


Comments

Post a Comment

Popular posts from this blog

sql - invalid in the select list because it is not contained in either an aggregate function -

-
i can't figure out rewrite sql statement avoid error below msg 8120, level 16, state 1, line 26 column 'staffmember.staffmemberid' invalid in select list because not contained in either aggregate function or group clause. sql statment select (select [dbo].[udf_readstaffmemberbasic](sm.staffmemberid) xml path ('staffmembersummary'), root ('items'), type ), count(sm.staffmemberid) totalresults, ceiling(cast(count(sm.staffmemberid) decimal)/ @pagesize) totalpages staffmember sm inner join staffmembertoadditionalrole ar on sm.staffmemberid = ar.staffmemberid (sm.firstname '%' + @name + '%' or sm.surname '%' + @name + '%') , (@floorid null or sm.floorid = @floorid) , (@directorateid null or sm.directorateid = @directorateid) , (@additionalroleid null or ar.additionalroleid = @additionalroleid) order sm.staffmemberid offset @page * @pagesize rows fe...
Read more

Angularjs unit testing - ng-disabled not working when adding text to textarea -

-
i trying test angularjs component using karma, mocha, power-assert. have textarea , button in component, button disabled based on length of text in textarea. when run component in browser works perfectly. can't figure out how test functionality. here's bit of code. inquiryform.js function inquiryform($scope) { // plan add logic here } angular.module('myapp').component('inquiryform', { controller: inquiryform, controlleras: 'inquiryform', templateurl: 'inquiryform.html' }); inquiryform.html <div class="contact"> <div>thanks contacting us.</div> <form> <textarea ng-model="inquiryform.inquirytext" name=""></textarea> <input type="submit" value="send" ng-disabled="!inquiryform.inquirytext.length"> </form> </div> inquiryformtest.js describe('inquiry form ', () => { let $r...
Read more

How to start daemon on android by adb -

-
can me please, head swollen... i trying run daemon on android emu/device command: adb -s <device_name> shell su -c /dir/daemon <port_number> but nothink happens, , no errors! if do: adb -s <device> shell and form shell cmdline: su -c /dir/daemon <port_number> than work fine. try use shell-script , run: adb -s <device_name> shell sh su -c /dir/script.sh <port_number> and try generate script qt code port number , use: adb -s <device_name> shell sh su -c /dir/script.sh but not helped... problem if enter adb shell previous run daemon - work. problem on different device/emu/iso_image different command format not work. example: adb -s <device_name> shell su -c /dir/daemon <port_number> // work @ emu, not @ iso and vice: adb -s <device_name> shell su -c "/dir/daemon <port_number>" // work @ iso etc. everybody can answer what's matter? sorry english adb shell su -c ...
Read more