How to convert a big csv file into json array quickly in java -
How to convert a big csv file into json array quickly in java -
i want convert big csv file 20000 50000 record file json array takes 1 min convert in there way accomplish in less 5 sec.
resourcebundle rb = resourcebundle.getbundle("settings"); string path = rb.getstring("fileandfolder.path"); system.out.println(path + "ssdd"); string csvpath = request.getparameter("dp") != null ? request .getparameter("dp").tostring() : ""; string orname = path + csvpath; file file = new file(orname); filereader fin = new filereader(file); //read file 1 1 bufferedreader bi = new bufferedreader(fin); int res; string csv = ""; while ((res = fin.read()) != -1) { csv = csv + ((char) res); //converted int char , stored in csv } long start3 = system.nanotime(); jsonarray array = cdl.tojsonarray(csv); string csvs = array.tostring(); long time3 = system.nanotime() - start3; system.out .printf("took %.3f seconds convert %d mb file, rate: %.1f mb/s%n", time3 / 1e9, file.length() >> 20, file.length() * 1000.0 / time3);
there 2 glaring performance problems in code, both of them in snippet:
while ((res = fin.read()) != -1) { csv = csv + ((char) res); }
first problem: fin
unbuffered filereader
, each read()
phone call doing scheme call. each syscall hundreds or thousands of instructions. , doing each , every character in input file.
remedy: read bi
rather fin
. (that's created ... presumably.)
second problem: each time execute csv = csv + ((char) res);
creating new string 1 character longer previous one. if have n
characters in input file, end copying n^2
characters build string.
remedy: instead of concatenating strings, utilize stringbuilder ... this:
stringbuilder sb = new stringbuilder(); .... sb.append((char) res); .... string csv = sb.tostring();
at point, not clear me if there performance problem in converting csv
string json; i.e. in snippet.
jsonarray array = cdl.tojsonarray(csv); string csvs = array.tostring();
unfortunately, don't know jsonarray
, cdl
classes using. hence, hard why slow, or whether there faster way conversion. (but suspect, biggest performance problems in before snippet.)
java json
Comments
Post a Comment