How to convert a big csv file into json array quickly in java -



How to convert a big csv file into json array quickly in java -

i want convert big csv file 20000 50000 record file json array takes 1 min convert in there way accomplish in less 5 sec.

resourcebundle rb = resourcebundle.getbundle("settings"); string path = rb.getstring("fileandfolder.path"); system.out.println(path + "ssdd"); string csvpath = request.getparameter("dp") != null ? request .getparameter("dp").tostring() : ""; string orname = path + csvpath; file file = new file(orname); filereader fin = new filereader(file); //read file 1 1 bufferedreader bi = new bufferedreader(fin); int res; string csv = ""; while ((res = fin.read()) != -1) { csv = csv + ((char) res); //converted int char , stored in csv } long start3 = system.nanotime(); jsonarray array = cdl.tojsonarray(csv); string csvs = array.tostring(); long time3 = system.nanotime() - start3; system.out .printf("took %.3f seconds convert %d mb file, rate: %.1f mb/s%n", time3 / 1e9, file.length() >> 20, file.length() * 1000.0 / time3);

there 2 glaring performance problems in code, both of them in snippet:

while ((res = fin.read()) != -1) { csv = csv + ((char) res); }

first problem: fin unbuffered filereader, each read() phone call doing scheme call. each syscall hundreds or thousands of instructions. , doing each , every character in input file.

remedy: read bi rather fin. (that's created ... presumably.)

second problem: each time execute csv = csv + ((char) res); creating new string 1 character longer previous one. if have n characters in input file, end copying n^2 characters build string.

remedy: instead of concatenating strings, utilize stringbuilder ... this:

stringbuilder sb = new stringbuilder(); .... sb.append((char) res); .... string csv = sb.tostring();

at point, not clear me if there performance problem in converting csv string json; i.e. in snippet.

jsonarray array = cdl.tojsonarray(csv); string csvs = array.tostring();

unfortunately, don't know jsonarray , cdl classes using. hence, hard why slow, or whether there faster way conversion. (but suspect, biggest performance problems in before snippet.)

java json

Comments

Post a Comment

Popular posts from this blog

ruby on rails - Devise Logout Error in RoR -

ruby on rails - Devise Logout Error in RoR - while using devise in ror, encountering unusual error when trying log out url different root url. displays me : actioncontroller::invalidauthenticitytoken in devise::sessionscontroller#destroy" for illustration if rooting path after log in '/structure', , seek log out '/general' error, while logging out '/structure' fine. any help highly appreciated. ruby-on-rails ruby devise
Read more

c# - Create a Notification Object (Email or Page) At Run Time -- Dependency Injection or Factory -

c# - Create a Notification Object (Email or Page) At Run Time -- Dependency Injection or Factory - i have code emails me notification when happens. trying add together page , also remove dependency on email. for reference; have name of notification want send stored string in database. @ runtime have list of people , string representation of want receive. i thought dependency injection ninject, kernel bindings got big , seemed hacking together. then thought simple factory, activator.createinstance(email/page) inotifcation, don't know how handle differing constructor parameters (unless utilize common, complex property, employee, has info need). so far have this: public interface inotification { void sendmessage(); } public class email: inotification { private readonly string _toemailaddress; private string body { { homecoming "email"; } } private static string smtphost { ...
Read more

model view controller - MVC Rails Planning -

model view controller - MVC Rails Planning - i'm starting rails , i'm building app stores films info, user can watch film , give grade on web application. i have coded cinema model, controller , views. now, have 2 questions: should need create model , controller grading? in case, should create or should wait , create first users model , controller? yes, on model , controller grades. that'd feasible way maintain track of who's graded what. it's create first, i'd start users, since grades depend on both films , users. ruby-on-rails model-view-controller
Read more