sqlite - Android huge JSONObject toString goes OutOfMemoryError -

-

i need read lot of data sqlite database , create formatted json.

i'm achieving having object called requestpayload which contains arraylists in put data read sqlite.

the requestpayload class has parsejson() method returns jsonobject on call tostring() method obtain json string got written on file.

now, when i've got "small" quantities of data in sqlite goes fine. when i've got lot of data happens:

06-28 09:55:34.121 10857-6799/it.example.sampler e/androidruntime: fatal exception: thread-195240     process: it.example.sampler, pid: 10857     theme: themes:{com.cyanogenmod.trebuchet=overlay:system, com.android.settings=overlay:system, default=overlay:system, iconpack:system, fontpkg:system, com.android.systemui=overlay:system, com.android.systemui.navbar=overlay:system}     java.lang.outofmemoryerror: failed allocate 52962812 byte allocation 16764752 free bytes , 41mb until oom       @ java.lang.stringfactory.newstringfromchars(native method)       @ java.lang.abstractstringbuilder.tostring(abstractstringbuilder.java:629)       @ java.lang.stringbuilder.tostring(stringbuilder.java:663)       @ org.json.jsonstringer.tostring(jsonstringer.java:430)       @ org.json.jsonobject.tostring(jsonobject.java:690)       @ it.example.sampler.controllers.network.requestbodyencoder.serialise(requestbodyencoder.java:69)       @ it.example.sampler.controllers.network.requestbodyencoder.createpacket(requestbodyencoder.java:50)       @ it.example.sampler.services.filestorerunnable.run(filestorerunnable.java:57)       @ java.lang.thread.run(thread.java:818)  06-28 09:55:34.509 10857-10857/it.example.sampler e/windowmanager: android.view.windowleaked: activity it.example.sampler.ui.startsamplingactivity has leaked window com.android.internal.policy.phonewindow$decorview{74710d v.e...... r......d 0,0-1026,494} added here     @ android.view.viewrootimpl.<init>(viewrootimpl.java:372)     @ android.view.windowmanagerglobal.addview(windowmanagerglobal.java:299)     @ android.view.windowmanagerimpl.addview(windowmanagerimpl.java:86)     @ android.app.dialog.show(dialog.java:319)     @ it.example.sampler.ui.startsamplingactivity.executestop(startsamplingactivity.java:305)     @ it.example.sampler.ui.startsamplingactivity.onclickedbutton(startsamplingactivity.java:256)     @ it.example.sampler.ui.startsamplingactivity$3.onclick(startsamplingactivity.java:190)     @ android.view.view.performclick(view.java:5204)     @ android.view.view$performclick.run(view.java:21158)     @ android.os.handler.handlecallback(handler.java:739)     @ android.os.handler.dispatchmessage(handler.java:95)     @ android.os.looper.loop(looper.java:148)     @ android.app.activitythread.main(activitythread.java:5461)     @ java.lang.reflect.method.invoke(native method)     @ com.android.internal.os.zygoteinit$methodandargscaller.run(zygoteinit.java:726)     @ com.android.internal.os.zygoteinit.main(zygoteinit.java:616)

now code:

the error line method:

private string serialise() throws jsonexception {     // serialise     string samplestring = mpayload.parsejson().tostring(); // line      logger.get().d(log_tag, "serialised payload: -> " + samplestring);     return samplestring; }

and if got logcat correctly error during execution of tostring() method , appears json object big.

how can handle situation?

update: answer @yashjain comment asking size of jsonobject.

after sampling sampling lasted 2 hours had json containing:

  • 12 jsonarray containing 155.432 floats
  • 2 jsonarray containing 8.393 floats
  • 1 jsonarray containing 8.393 custom object containing 5 floats each (and string)
  • 1 2d array containing 155.432 ints

in terms of bytes since float made 4 bytes (i hope i've done calculus correctly) i've got circa:

(12 * 155432 * 4) + (2 * 8393 * 4) + (1 * 8393 * 5) + (1 * 155432 * 4) ~= 8.191.573 bytes

update: i've been asked used compression algorithm. use zlib via deflater , deflateroutputstream java classes.

thus flow is:

  • sample data -> store them in sqlite
  • read sqlite -> build requestpayload object in memory
  • convert requestpayload object jsonobject (with custom custom parser).
  • convert jsonobject string (using jsonobject tostring() method)
  • compress string bytes (using getbytes()) -> encode in base64
  • send final base64 string server

edit: respond "possibile duplicate flag" haven't seen question during researches way didn't me since makes use of largeheap directive (which don't use). there's no accepted answer , answer doesn't provide practical solution.

sorry not giving more feedbacks i've been busy other works , got last in priority queue.

anyhow worked me using google gson library.

my serialisation method has become simply:

private string serialise() {     // serialise     return mpayload.tostringfromgson(); }

and method tostringfromgson() in class payload is:

public string tostringfromgson() {     gsonbuilder gsonbuilder = new gsonbuilder();     gson gson = gsonbuilder.create();      return gson.tojson(this); }

anyhow think can considered temporary patch. in fact noticed memory used application increases dramatically during phase hence think problem postponed , occur heavier loads..


Comments

Post a Comment

Popular posts from this blog

sql - invalid in the select list because it is not contained in either an aggregate function -

-
i can't figure out rewrite sql statement avoid error below msg 8120, level 16, state 1, line 26 column 'staffmember.staffmemberid' invalid in select list because not contained in either aggregate function or group clause. sql statment select (select [dbo].[udf_readstaffmemberbasic](sm.staffmemberid) xml path ('staffmembersummary'), root ('items'), type ), count(sm.staffmemberid) totalresults, ceiling(cast(count(sm.staffmemberid) decimal)/ @pagesize) totalpages staffmember sm inner join staffmembertoadditionalrole ar on sm.staffmemberid = ar.staffmemberid (sm.firstname '%' + @name + '%' or sm.surname '%' + @name + '%') , (@floorid null or sm.floorid = @floorid) , (@directorateid null or sm.directorateid = @directorateid) , (@additionalroleid null or ar.additionalroleid = @additionalroleid) order sm.staffmemberid offset @page * @pagesize rows fe...
Read more

Angularjs unit testing - ng-disabled not working when adding text to textarea -

-
i trying test angularjs component using karma, mocha, power-assert. have textarea , button in component, button disabled based on length of text in textarea. when run component in browser works perfectly. can't figure out how test functionality. here's bit of code. inquiryform.js function inquiryform($scope) { // plan add logic here } angular.module('myapp').component('inquiryform', { controller: inquiryform, controlleras: 'inquiryform', templateurl: 'inquiryform.html' }); inquiryform.html <div class="contact"> <div>thanks contacting us.</div> <form> <textarea ng-model="inquiryform.inquirytext" name=""></textarea> <input type="submit" value="send" ng-disabled="!inquiryform.inquirytext.length"> </form> </div> inquiryformtest.js describe('inquiry form ', () => { let $r...
Read more

python - Error opening file in H5PY (File signature not found) -

-
i've been using following bit of code open hdf5 files, produced in matlab, in python using h5py: import h5py h5 data='dataset.mat' f=h5.file(data, 'r') however i'm getting following error: oserror: unable open file (file signature not found) i've checked files i'm trying open version 7.3 mat-files , hdf5 format. in fact i've used h5py open same files before. i've confirmed files exist , accessible i'm not sure error coming from. advice appreciated, in advance : ) usually message file signature not found indicates either: 1. file corrupted. ... think likely. said you've opened files before. maybe forgot closing file-handle can corrupt file. try checking file hdf5 utility h5debug (available on command line if you've installed hdf5 lib on os, check dpkg -s libhdf5-dev on linux). 2. file not in hdf5 format. this known cause error message. since said made sure, case , you've opened files before, i...
Read more