Multiple threads can read and write the same file in several situations:
-
Multiple threads read the same file at the same time. In this case, there is no conflict
-
If multiple threads write the same file at the same time, write data will be lost
-
Multiple threads write and read the same file at the same time, which will cause dirty reading
If you want to deal with the data inconsistency caused by multithreading reading and writing files, the first thing you think of is locking. In java.concurrent.locks, ReadWriteLock defines optimistic lock read lock and pessimistic lock write lock respectively. Taking the above situations into account, it can well handle the situation of multiple threads reading and writing the same file. However, since locking will inevitably lead to low timeliness of multithreading in reading and writing files, it seems that there are better solutions in different situations:
-
Lock the file reading and writing process through ReadWriteLock to prevent data inconsistency with expectations and reduce the efficiency of multi-threaded processing
-
If multiple threads frequently write a small amount of data, you can create a class to cache the data to be written, and write data in batches on time, so as to reduce the problems caused by frequent file operation and locking operation.
-
Different locations can be planned through RandomAccessFile, and multiple threads can operate writes in different locations at the same time
For RandomAccessFile, previously in JAVA: Java IO (III) accessing files -- conversion stream and file stream I have a simple understanding of RandomAccessFile.
1. Practice a multithreaded writing file RandomAccessFile without locking
The optional modes of RandomAccessFile are r, rw, rws, or rwd. When writing, the original data will not be cleared, and the original content will be overwritten at the specified location to write new content.
So:
-
The method to empty the RandomAccessFile open file is rw.setLength(0)
-
In order to prevent multiple threads from overwriting each other, the write location needs to be planned, and the insertion will be more troublesome
-
There are two planned writing positions. a) if the number of words written in each line is the same, it is easy to calculate enough positions. b) reserve enough positions
-
The result of reserving enough space will actually have some defects. The results are as follows:
/* Multithreading writes files to the specified location */ class RWrite implements Runnable{ public int pos; public String text; public String fileName; public RWrite(int pos,String text,String fileName){ this.pos = pos; this.text = text; this.fileName = fileName; } @Override public void run(){ try { RandomAccessFile rw = new RandomAccessFile(this.fileName,"rw"); rw.seek(pos); rw.writeBytes(text); rw.close(); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } } public void test(){ ExecutorService pool = Executors.newFixedThreadPool(10); int linwsize = 20;//Enough space must be reserved for one line, otherwise it will lead to coverage. However, if the reserved space is too large, empty characters will also appear int lines = 50;//100 lines need to be written String fileName = "a.csv"; RandomAccessFile rw = null; try { rw = new RandomAccessFile(fileName,"rw"); rw.setLength(0);//Empty file /*Write title block*/ rw.writeBytes("index,text\n"); rw.close(); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } for(int i=1;i<=lines;i++){ pool.execute(new RWrite(i*linwsize,i+", text"+i+"\n",fileName)); } pool.shutdown(); }
2 practice 2 hand over the written file to a thread for processing
Similar to the producer consumer model, there is only one consumer.
Originally I just wanted to write it briefly, but it took a long time to debug because I wrote 0 as 9 somewhere.
/* Leave writing to a thread */ /* Task of processing data. producer */ class DataTask implements Runnable{ BlockingQueue<String> data_put; private int start; private int end; public DataTask(BlockingQueue<String> data_put,int start,int end){ this.data_put = data_put; this.start = start; this.end = end; } @Override public void run(){ /*System.out.println(String.format("%s-%s:%d-%d Start ", System.currentTimeMillis(), Thread.currentThread().getName(), this.start,this.end));*/ for(int i=start;i<=end;i++){ try { /* Processing data takes time and is random */ Thread.sleep(10+new Random().nextInt(20)); String s = String.format("%s-%s:%d-%d[%d]\n", System.currentTimeMillis(), Thread.currentThread().getName(), this.start,this.end,i); data_put.put(i+"\n"); } catch (InterruptedException e) { e.printStackTrace(); return; } } System.out.println(String.format("%s-%s:%d-%d end", System.currentTimeMillis(), Thread.currentThread().getName(), this.start,this.end)); } } /* Cache the submitted data and write */ class WriteTask implements Runnable{ private BlockingQueue<String> data_in = new ArrayBlockingQueue<>(10); private byte[] buffer = new byte[1024]; private int th = (int)(1024*0.8); int length=0; private String fileName; public WriteTask(String fileName){ this.fileName = fileName; try { /* Empty the file to write data to */ FileOutputStream fileOutputStream = new FileOutputStream(fileName); fileOutputStream.close(); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } public BlockingQueue<String> get_Queue(){ return data_in; } private void write(){ if(length==0) return; try { //System.out.println(length); //System.out.println(new String(buffer)); System.out.println("Start writing"); FileOutputStream fileOutputStream = new FileOutputStream(fileName,true); fileOutputStream.write(buffer,0,length); fileOutputStream.close(); System.out.println(length+"Write complete."); length = 0; } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } private void close(){ //System.out.println(new String(buffer)); this.write(); // System.out.println(length); //data_in = null; } @Override public void run(){ while (true){ try { byte[] tmp= data_in.take().getBytes(); System.arraycopy(tmp,0,buffer,length,tmp.length); length = length+tmp.length; if(length>=th){ this.write(); } } catch (InterruptedException e) { //e.printStackTrace(); break; } } } } public void test3(){ ThreadPoolExecutor pool = (ThreadPoolExecutor) Executors.newFixedThreadPool(10); String fileName = "b.csv"; WriteTask writeTask = new WriteTask(fileName); pool.execute(writeTask); int num = 20; int writenum = 100; for(int i=0;i<num;i++){ //System.out.println(i*writenum+"---"+((i+1)*writenum-1)); pool.execute(new DataTask(writeTask.get_Queue(),i*writenum,((i+1)*writenum-1))); } pool.shutdown(); while (true){ try { pool.awaitTermination(500,TimeUnit.MILLISECONDS); if(pool.getActiveCount()==1){ writeTask.close(); Thread.sleep(10); pool.shutdownNow(); } if(pool.getActiveCount()==0){ break; } } catch (InterruptedException e) { e.printStackTrace(); break; } } }
3 practice 3 multi thread copying file RandomAccessFile without lock
Using the same offset for reading and writing can also prevent conflicts during multi-threaded writing.
The copied file contains read and write operations. The advantage is that you don't have to plan the offset yourself.
/* Multi threaded copy file */ /* Multithreading writes files to the specified location */ class RCopy implements Runnable{ public int pos; public int len; public String readFile; public String writeFlie; public RCopy(String readFile,String writeFlie,int pos,int len){ this.pos = pos; this.len = len; this.readFile = readFile; this.writeFlie = writeFlie; } @Override public void run(){ byte[] bytes = new byte[len]; try { RandomAccessFile rr = new RandomAccessFile(this.readFile,"r"); RandomAccessFile rw = new RandomAccessFile(this.writeFlie,"rw"); rr.seek(pos); rw.seek(pos); rr.read(bytes); rw.write(bytes); rr.close(); rw.close(); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } } public void test2(){ ExecutorService pool = Executors.newFixedThreadPool(10); String readFile = "b.csv"; String writeFlie = "a.csv"; long totalLen = 0; int len = 1024; /* Write size per task */ try { //Read the size of the file to be copied RandomAccessFile file = new RandomAccessFile(readFile,"r"); totalLen = file.length(); System.out.println("length:"+totalLen); //Empty files to be written file = new RandomAccessFile(writeFlie,"rw"); file.setLength(0); } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } int tasknum = 11; for(int i=0;i<totalLen;i = i+len){ int alen = len; if(i+len>totalLen) alen = (int)totalLen-i; //System.out.println(i+":"+alen); pool.execute(new RCopy(readFile,writeFlie,i,alen)); } pool.shutdown(); }