abstract method throwing exception

Hi,

In InputStream class there is an abstract method:

public abstract int read() throws IOException

What is the functional impact "throws IOException" to this method?

Overriding this method in child class:

1) if I omit "public", I will get error
attempting to assign weaker access privileges; was public

2) If I change int to long, I will get error
[child class] is not abstract and does not override abstract method read() in InputStream

3) But I can omit "throws IOException" with no error.

Moreover, child method can throw some other exception. This code runs with no error:

import java.io.InputStream; import java.io.IOException; public class MyInput extends InputStream{     int cnt = 0;         public int read() throws ArithmeticException { if (cnt++ < 10){ return 7; } else{ return -1; }            }    public static void main(String[] args){        MyInput myin = new MyInput(); byte[] barr = new byte[3];        int b;        while( (b=myin.read()) > 0 ){          System.out.print(b);        }    } }

Is "throws IOException" purly informative with no impact on code compilation/execution?

Regards,
Grzegorz

Here is a very short but clear explanation: tutorialspoint

Grzegorz Wilanowski wrote:. . . there is an abstract method:

public abstract int read() throws IOException

What is the functional impact "throws IOException" to this method?

It means that method and its overriding methods might throw an IOException, and any code calling that method must be prepared to handle an IOException. Remember it is incorrect programming to cause an overriding method to throw an exception that is not thrown by its overridden (superclass) version.

Overriding this method in child class:

I know lots of people say, “child class,” but that term isn't accurate. Inheritance in programming terms is different from inheritance in biological terms or inheritance in money terms. Say, “subclass,” and say, “superclass” instead of “parent class”.

1) if I omit "public", I will get error . . .

An overriding method must be accessible to all code that can access the overridden method. You cannot make the method “disappear” from visibility by omitting its access modifier or making its access more restrictive.

2) If I change int to long, I will get error . . .

An overriding method must return a value that its calling code can handle. The calling code can handle an int, but it cannot handle a long. So you need the same return type (I think) as the overridden method, except that for reference types the return type may be covariant. The return type may be a subtype of the overridden method's return type. There is a concept of methods being return type substitutable, which should appear in the JLS (=Java® Language Specification) here.

3) But I can omit "throws IOException" with no error.

Yes, the overriding method can be an improvement and not throw any exceptions. It is incorrect to throw a new kind of exception. The calling method “expects” the method to run to completion by returning a value or throwing a particular exception, and can handle that. It is incorrect practice to throw another kind of exception because the calling code cannot handle that. Not even an unchecked exception like ArithmeticException. Remember that the compiler goes through the different kinds of exception and the clause throws SomeUncheckedExceptionType has no effect. The fact that the compiler ignores a particular exception declaration doesn't make it good programming.
* * * * * * * * * * * * * * * * * * * * * * * * *
I suggest you try a few programs using InputStream#read(), until you develop antibodies to it and never want to see it again See what happens if you inflict letters like Ł on read(). You will have to work out your own encoding It is a dreadful method to use and you should leave it to things like BufferedReader#readLine() to use read(). That sort of method doubtless uses read(), so you don't have to. But readLine() doesn't handle any IOExceptions, so you will still have to handle that exception.
* * * * * * * * * * * * * * * * * * * * * * * * *
Remember that the overriding method should do what the overridden method does, and mustn't do it worse, but may do it better. That also means the overriding method should comply with all of the specification (=documentation) for the overridden method.

@Campbell

See what happens if you inflict letters like Ł on read()

InputStream is for reading bytes, not characters. Low level eqivalent for reading characters is Reader.
Here is the example of reading Ł


import java.io.Reader; public class MyReader extends Reader{     public int read(char[] cbuf, int off, int len) { cbuf[0] = 'Ł';        return 1;    }    public void close(){    }    public static void main(String[] args){        MyReader myin = new MyReader();        char[] xArr = new char[1];        myin.read(xArr,0,1);        System.out.println(xArr[0]);    } }

Note: To make it work I need to compile like this
javac -encoding UTF-8  MyReader.java

Most of the time there's no good reason to read individual chars from a stream though, so most Reader implementations should not be used directly. The reason for this is that reading chars using a Reader may split up surrogate pairs.

To read textual data, you should typically use either:

BufferedReader, if you handle text one line at a time, or

Scanner, if you handle text in another way.

One scenario I can think of when you'd want to use a Reader to read a char[] or CharBuffer, is when you're reading a password. That way, you can zero out the password when you're done with it.

Grzegorz Wilanowski wrote:. . . InputStream is for reading bytes, not characters. . . .

Yes. Sorry. My mistake.

Grzegorz Wilanowski wrote:Note: To make it work I need to compile like this
javac -encoding UTF-8  MyReader.java

Starting with Java 18, the default encoding is UTF-8. (That implies that you should really use UTF-8 as your text editor's encoding; it looks like you do that already.)

Stephan van Hulst wrote:Most of the time there's no good reason to read individual chars from a stream though, so most Reader implementations should not be used directly..

With Reader, you can also read whole file content into array at once:

import java.io.FileReader; import java.io.IOException; public class MyFileReader{ public static void main(String[] args){ try{ FileReader reader = new FileReader("data.txt"); int maxLength = 1000; char[] fileContent = new char[maxLength]; reader.read(fileContent,0,maxLength); System.out.println(fileContent); } catch(IOException e){ System.out.println("Exception="+e); } } }

Well, what if the file is larger than 1000 UTF-16 code units?

And... if the file is less than 1,000 characters long that will leave the remainder of the array filled with zeros which println() will attempt to output.

Stephan wrote:what if the file is larger than 1000 UTF-16 code units?

Obviously, we need longer buffer or read in a loop.
But this is what BufferedReader is also doing.
In one of its constructors you can provide buffer size:
BufferedReader(Reader in, int sz)

Carey wrote: if the file is less than 1,000 characters long that will leave the remainder of the array filled with zeros

That's right.

Stephan, Carey: based on your remarks I fixed the program:

import java.io.FileReader; import java.io.IOException; public class MyFileReader2{ public static void main(String[] args){ try{ FileReader reader = new FileReader("data.txt"); int maxLength = 3; char[] fileContent = new char[maxLength]; int actuallyRead = 0; while ( (actuallyRead  =reader.read(fileContent,0,maxLength) ) >0 ){ if (actuallyRead == maxLength){ System.out.print(fileContent); } else{ for(int i = 0; i<actuallyRead; i++){ > System.out.print(fileContent[i]); } } } } catch(IOException e){ System.out.println("Exception="+e); } } }

data.txt contains two lines
aaabbbcccddde
eeff

Grzegorz Wilanowski wrote:But this is what BufferedReader is also doing.

Which is why I said:

most Reader implementations should not be used directly

and:

To read textual data, you should typically use [...] BufferedReader

There is no good reason to handle char arrays directly, unless you're working with passwords. Use BufferedReader to read complete lines as strings, or use Scanner to read token strings.

Here is the correct version:
public static void main(String... args) throws IOException {  var path = Path.of("data.txt");  try (    var reader = Files.newBufferedReader(path);  ) {    reader.lines().forEachOrdered(System.out::println);  } }

Grzegorz Wilanowski wrote:. . . Obviously, we need longer buffer or read in a loop.
But this is what BufferedReader is also doing.
In one of its constructors you can provide buffer size:
BufferedReader(Reader in, int sz). . .

I know we have strayed from the original question in this thread, but I am not happy about that. That looks like an old‑fashioned way to read a file. I am at a bit of a loss to know why somebody would want to read a text file into a char[] in the first place. But let's try a slightly more object‑oriented way to do it. Read the lines into an object and use that object and other objects to join the lines into one object.
You are using an old‑fashioned way to create your buffered reader, which has been regarded as legacy code since 2009. Use a Path instead. Also use the “new” (=only fourteen years old) way to close your resources.try (BufferedReader reader =      Paths.get("/home/campbell/Documents/myFile")           .newBufferedReader()) {   String line;   StringBuilder builder =         new StringBuilder(); // or declare as "var"   while ((line = reader.readLine()) != null)   {      builder.append(line).append(System.lineSeparator());   }   char[] /* or var */ chars =         builder.toString().toCharArray();   // etc } catch (IOException ex) {   // etc }I presume you are familiar with the idiom in line 8. You could use getBytes(), or serialise the resulting text with a ByteArrayOutputStream to get a byte[] instead.
I would prefer to use each line as it is read, or read them into a List<String>. Beware of a List if your file is millions of lines long in case you use too much memory.
You can probably use BufferedReader#lines() and maybe later use flatMap() to get your arrays into one array.

[edit]Add, “.newBufferedReader()” to lines 3. Have probably left some spelling errors somewhere.

Good idea, Carey

Or, since Java 11:
static void four() throws IOException {  System.out.println(Files.readString(Paths.get("data/problem.txt"))); }

Carey Brown wrote: static void three() throws IOException { String fileName = "data/problem.txt"; try( BufferedReader bufferedReader = new BufferedReader( new FileReader( fileName ) ) ) { String fileContent = bufferedReader .lines() .collect( Collectors.joining() ); System.out.println( fileContent ); } }

Hm, this has an issue - it eliminates all the line separators.  A corrected version:
 static void three() throws IOException {    String fileName = "data/problem.txt";    try (BufferedReader bufferedReader = new BufferedReader(new FileReader(fileName))) {      String fileContent = bufferedReader        .lines()        .collect(Collectors.joining(System.lineSeparator()));      System.out.println(fileContent);    }  }
Or, one more alternative:
 static void five() throws IOException {    try (var lines = Files.lines(Path.of("data/problem.txt"))) {      lines.forEachOrdered(System.out::println);    }  }

Hi,

I tried to read 16GB file with the programs from this thread.
I generated this file in Windows cmd with  command
fsutil file createnew data.txt 17179881809

This generated file seems to contain only spaces.

Below are the programs I tested.
I replaced System.out.print() with doing-nothing writer.print() function from class

class DummyWriter{ void print(Object o){} }


Grzegorz
import java.io.FileReader; import java.io.IOException; public class Grzegorz{    public static void main(String[] args){        try{            DummyWriter writer = new DummyWriter();            FileReader reader = new FileReader("data.txt");            int maxLength = 16384;            char[] fileContent = new char[maxLength];            int actuallyRead = 0;            while ( (actuallyRead  =reader.read(fileContent,0,maxLength) ) >0 ){                if (actuallyRead == maxLength){                    writer.print(fileContent);                }                else{                    for(int i = 0; i<actuallyRead; i++){                        writer.print(fileContent[i]);                    }                }            }        }        catch(IOException e){            System.out.println("Exception="+e);        }    } } class DummyWriter{ void print(Object o){} }

Stephan
import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; public class Stephan{    public static void main(String... args) throws IOException {        DummyWriter writer = new DummyWriter();        var path = Path.of("data.txt");        try (                var reader = Files.newBufferedReader(path);        ) {            reader.lines().forEachOrdered(writer::print);        }    } } class DummyWriter{ void print(Object o){} }

Carey1 (two)
import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; public class Carey1{    public static void main(String... args) throws IOException {        DummyWriter writer = new DummyWriter();        byte[]  ary = Files.readAllBytes( Paths.get( "data.txt" ) );        writer.print( new String( ary ) );    } } class DummyWriter{ void print(Object o){} }

Carey2 (three)
import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; import java.io.BufferedReader; import java.io.FileReader; import java.util.stream.Collectors; public class Carey2{    public static void main(String... args) throws IOException {        String fileName = "data.txt";        DummyWriter writer = new DummyWriter();        try( BufferedReader bufferedReader = new BufferedReader(                new FileReader( fileName ) ) )        {            String fileContent = bufferedReader                    .lines()                    .collect(Collectors.joining(System.lineSeparator()));            writer.print( fileContent );        }    } } class DummyWriter{ void print(Object o){} }

Mike1 (four)
import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; public class Mike1{    public static void main(String... args) throws IOException {        DummyWriter writer = new DummyWriter();        writer.print(Files.readString(Paths.get("data.txt")));    } } class DummyWriter{ void print(Object o){} }

Mike2 (five)
import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; public class Mike2{ public static void main(String... args) throws IOException { DummyWriter writer = new DummyWriter(); try (var lines = Files.lines(Path.of("data.txt"))) {     lines.forEachOrdered(writer::print);   } } } class DummyWriter{ void print(Object o){} }

Campbell
import java.io.IOException; import java.io.BufferedReader; import java.nio.file.Files; import java.nio.file.Paths; public class Campbell{    public static void main(String... args) throws IOException {        DummyWriter writer = new DummyWriter();        try (BufferedReader reader = Paths.get("data.txt").newBufferedReader())        {            String line;            StringBuilder builder =   new StringBuilder();            while ((line = reader.readLine()) != null)            {                builder.append(line).append(System.lineSeparator());            }            char[] chars = builder.toString().toCharArray();            writer.print(chars);        }        catch (IOException ex)        {            // etc        }    } } class DummyWriter{ void print(Object o){} }

@Campbell code does not compile
javac Campbell.java
Campbell.java:11: error: cannot find symbol
       try (BufferedReader reader = Paths.get("data.txt").newBufferedReader())
                                                                                             ^
 symbol:   method newBufferedReader()
 location: interface Path
1 error

Here are the results:

>java Grzegorz >java Stephan Exception in thread "main" java.lang.OutOfMemoryError: Java heap space        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)        at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)        at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:740)        at java.base/java.lang.StringBuilder.append(StringBuilder.java:229)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:376)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)        at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:571)        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)        at java.base/java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:772)        at Stephan.main(Stephan.java:14) >java Carey1 Exception in thread "main" java.lang.OutOfMemoryError: Required array size too large        at java.base/java.nio.file.Files.readAllBytes(Files.java:3294)        at Carey1.main(Carey1.java:9) >java Carey2 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)        at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)        at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:740)        at java.base/java.lang.StringBuilder.append(StringBuilder.java:229)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:376)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)        at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:571)        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)        at Carey2.main(Carey2.java:19) >java Mike1 Exception in thread "main" java.lang.OutOfMemoryError: Required array size too large        at java.base/java.nio.file.Files.readAllBytes(Files.java:3294)        at java.base/java.nio.file.Files.readString(Files.java:3366)        at java.base/java.nio.file.Files.readString(Files.java:3325)        at Mike1.main(Mike1.java:9) >java Mike2 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)        at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)        at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:740)        at java.base/java.lang.StringBuilder.append(StringBuilder.java:229)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:376)        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)        at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:571)        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)        at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)        at java.base/java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:772)        at Mike2.main(Mike2.java:10)

Only Grzegorz run successfully.
Then in PowerShell I found it took 28s for Grzegorz program to read this file
> Measure-Command { java Grzegorz }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 28
Milliseconds      : 629
Ticks             : 286296135
TotalDays         : 0,000331361267361111
TotalHours        : 0,00795267041666667
TotalMinutes      : 0,477160225
TotalSeconds      : 28,6296135
TotalMilliseconds : 28629,6135

Note: for testing I just copy pasted the code from your posts without analyzing it. Maybe there are small improvements to make them run successfuly

I think it's disingenuous to subject applications that are obviously intended to read multi-line text files to a file that contains 16GB of NUL characters. The only thing you've demonstrated is that your application is the only one that can deal with unrealistic input.

The only real situation I can think of where more than a gigabyte of text data would be on a single line, is when a large amount of data is serialized to a textual format like JSON or XML and minified to get rid of insignificant whitespace. In such cases, I wouldn't even use BufferedReader or Scanner, but rather a proper JSON or XML deserializer.

Your comparison would be more valuable if you downloaded a Lorem Ipsum generator and then generated a 16GB text file.

Also, if you're no longer actually doing anything with the data, performance will become meaningless.  The actual IO will be much slower than the other concerns.  And, is there any need to use readers and writers at all?  Does the encoding matter, or are you just trying to transfer bytes?  If the latter, there are a variety of other methods possible.
  static void six() throws IOException {    try (var reader = Files.newBufferedReader(Path.of("data/problem.txt"));         var writer = new BufferedWriter(new PrintWriter(System.out))) {      reader.transferTo(writer);    }  }  static void seven() throws IOException {    Files.copy(Path.of("data/problem.txt"), System.out);  }  static void eight() throws IOException {    try (var in = Files.newInputStream(Path.of("data/problem.txt"));         var out = System.out) {      in.transferTo(out);    }  }    static void nine() throws IOException {    var file = new File("data/problem.txt");    try (var in = FileChannel.open(file.toPath(), StandardOpenOption.READ);         var out = Channels.newChannel(System.out)) {      in.transferTo(0, file.length(), out);    }  }

Your original code stated that it was reading the whole file into memory, our examples followed that directive. Yep, you proved that doesn't work for huge files as we already knew. It helps to have a clear set of requirements before writing code.

The original code had numerous problems - it wasn't until he started capturing the return value of the read() statement using the actuallyRead variable, that it became more correct.  

static class NullStreamWriter extends OutputStreamWriter { public NullStreamWriter() { super( OutputStream.nullOutputStream() ); } } static void ten() throws IOException { try( var in = new InputStreamReader( new FileInputStream( new File( "data/problem.txt" ) ) ); var out = new NullStreamWriter(); ) { in.transferTo( out ); } }

Let's not forget the most efficient possible implementation, based on the latest test code:
static void eleven() {}

So it took you 30 seconds to read 16 megabytes of data from a disk file. That seems pretty reasonable to me. I wouldn't spend any more time looking for faster code, since the disk hardware limits the speed at which you can read data.

When my application started to get slow I went out and bought a new computer with much faster disk drives. It was a great success (my old computer was 12 years old) and no amount of rewriting code in the service of "efficiency" would have been anywhere near as good.

Grzegorz Wilanowski wrote:. . . @Campbell code does not compile
. . .

Sorry; I was trying to write it from memory. You hjave been given the right code.

. . . Seconds           : 28
Milliseconds      : 629
Ticks             : 286296135
. . .

I am very surprised that your clock cycles match seconds so precisely at 10MHz.

Paul Clapham wrote:So it took you 30 seconds to read 16 megabytes of data from a disk file. That seems pretty reasonable to me.

It would be poor performance for 16 megabytes.  But the file was actually 16 gigabytes, so yes, it's a reasonable time.

But, when testing actually copying the file to another file, we can do better.  In particular, Files.copy ends up being about six times faster on my system.  I suspect that's because it's delegated to the underlying hardware, with little intervention from the program.  Using FileChannel's transferTo method can be about three times faster, at least until we get to really big files.  An old-school InputStream-to-OutputStream copy that we would have written before Files or FileChannels classes were available, that is still about twice as fast.  Well, results vary.  But, considerable improvement is possible.
package ranch; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.FileReader; import java.io.FileWriter; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import java.io.PrintStream; import java.io.PrintWriter; import java.nio.channels.FileChannel; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.StandardOpenOption; import java.util.stream.Collectors; public class FileCopying {  private static Path dirPath = Path.of("./files");  private static String sourceName = "sample_file.txt";  static interface FileCopier {    void copy(Path source, Path dest) throws IOException;  }  public static void main(String[] args) throws IOException {    if (!Files.exists(dirPath)) {      Files.createDirectory(dirPath);    }    testSuite(10_000_000L);    testSuite(100_000_000L);    testSuite(500_000_000L);    testSuite(1_000_000_000L);    testSuite(16_000_000_000L);  }  static void testSuite(long sourceFileSize) throws IOException {    System.out.printf("%nSource file size: %,d%n", sourceFileSize);    makeSourceFile(sourceFileSize);    timeCopy("Files_readString", FileCopying::copy_Files_readString, 2_000_000_000L);    timeCopy("Files_readAllBytes", FileCopying::copy_Files_readAllBytes, 2_000_000_000L);    timeCopy("Reader_lines_joining", FileCopying::copy_Reader_lines_joining, 2_000_000_000L);    timeCopy("Files_lines_forEach", FileCopying::copy_Files_lines_forEach, 200_000_000L);    timeCopy("BufferedReader_transferTo", FileCopying::copy_BufferedReader_transferTo);    timeCopy("Files_copy", FileCopying::copy_Files_copy);    timeCopy("InputStream_transferTo", FileCopying::copy_InputStream_transferTo);    timeCopy("FileChannel_transferTo", FileCopying::copy_FileChannel_transferTo);    timeCopy("FileChannel_transferTo_fixed", FileCopying::copy_FileChannel_transferTo_fixed);    timeCopy("oldSchool", FileCopying::copy_oldSchool);    timeCopy("grzegorz", FileCopying::copy_grzegorz);    timeCopy("grzegorz_fixed", FileCopying::copy_grzegorz_fixed);  }  static void makeSourceFile(long size) throws IOException {    try (var writer = Files.newBufferedWriter(dirPath.resolve(sourceName))) {      long len = 0;      long count = 0;      while (len < size) {        String line = "Here is some text for you, line " + ++count + System.lineSeparator();        writer.write(line);        len += line.length();      }    }  }  static void timeCopy(String label, FileCopier copier) {    timeCopy(label, copier, Long.MAX_VALUE);  }  static void timeCopy(String label, FileCopier copier, long maxSize) {    Path source = dirPath.resolve(sourceName);    Path dest = dirPath.resolve("out_" + label + ".txt");    System.out.printf("%30s: ", label);    if (source.toFile().length() > maxSize) {      System.out.printf("%21s%n", "skipped");    } else {      try {        Files.deleteIfExists(dest);      }      catch (IOException e) {        throw new RuntimeException(e);      }      long start = System.currentTimeMillis();      try {        copier.copy(source, dest);      } catch (Throwable t) {        t.printStackTrace();      }      long stop = System.currentTimeMillis();      double elapsed = (stop - start) / 1000.0;      System.out.printf("%8.3f sec", elapsed);      if (source.toFile().length() != dest.toFile().length()) {        System.out.printf("  ERROR: incorrect file length %d, expected %d", dest.toFile().length(), source.toFile().length());      }      System.out.println();    }  }  // From Grzegorz  static void copy_grzegorz(Path source, Path dest) throws IOException {    var writer = new FileWriter(dest.toFile());    FileReader reader = new FileReader(source.toFile());    int maxLength = 16384;    char[] fileContent = new char[maxLength];    int actuallyRead;    while ((actuallyRead = reader.read(fileContent,0,maxLength)) > 0) {      if (actuallyRead == maxLength) {        writer.write(fileContent);      } else {        for (int i = 0; i<actuallyRead; i++) {          writer.write(fileContent[i]);        }      }    }  }  // From Grzegorz, but ensuring that the writer and reader are properly closed  static void copy_grzegorz_fixed(Path source, Path dest) throws IOException {    try (var writer = new FileWriter(dest.toFile());         var reader = new FileReader(source.toFile())) {      int maxLength = 16384;      char[] fileContent = new char[maxLength];      int actuallyRead;      while ((actuallyRead = reader.read(fileContent,0,maxLength)) > 0) {        if (actuallyRead == maxLength) {          writer.write(fileContent);        } else {          for (int i = 0; i<actuallyRead; i++) { >            writer.write(fileContent[i]);          }        }      }    }  }  // From Carey's two() method  static void copy_Files_readAllBytes(Path source, Path dest) throws IOException {    byte[]  ary = Files.readAllBytes(source);    try (var out = new PrintWriter(Files.newOutputStream(dest))) {      out.print(new String(ary));    }  }  // From Mike's three() method  static void copy_Files_readString(Path source, Path dest) throws IOException {    try (var out = new PrintWriter(Files.newOutputStream(dest))) {      out.print(Files.readString(source));    }  }  // From Carey's four() method  static void copy_Reader_lines_joining(Path source, Path dest) throws IOException {    try (var reader = Files.newBufferedReader(source); var out = new PrintWriter(Files.newOutputStream(dest))) {      String fileContent = reader.lines().collect(Collectors.joining(System.lineSeparator()));      out.println(fileContent);    }  }  // From Mike's five() method  static void copy_Files_lines_forEach(Path source, Path dest) throws IOException {    try (var out = new PrintStream(Files.newOutputStream(dest)); var lines = Files.lines(source)) {      lines.forEachOrdered(out::println);    }  }  // From Mike's six() method  static void copy_BufferedReader_transferTo(Path source, Path dest) throws IOException {    try (var reader = Files.newBufferedReader(source);         var writer = Files.newBufferedWriter(dest)) {      reader.transferTo(writer);    }  }  // From Mike's seven() method  static void copy_Files_copy(Path source, Path dest) throws IOException {    Files.copy(source, dest);  }  // From Mike's eight() method  static void copy_InputStream_transferTo(Path source, Path dest) throws IOException {    try (var in = Files.newInputStream(source);         var out = Files.newOutputStream(dest)) {      in.transferTo(out);    }  }  // From Mike's nine() method  static void copy_FileChannel_transferTo(Path source, Path dest) throws IOException {    try (var in = FileChannel.open(source, StandardOpenOption.READ);         var out = FileChannel.open(dest, StandardOpenOption.WRITE, StandardOpenOption.CREATE)) {      in.transferTo(0, source.toFile().length(), out);    }  }  // From Mike's nine() method, fixed because unlike other transferTo() methods, the one in FileChannel  // may transfer less than what is requested, for no particular reason.  static void copy_FileChannel_transferTo_fixed(Path source, Path dest) throws IOException {    try (var in = FileChannel.open(source, StandardOpenOption.READ);         var out = FileChannel.open(dest, StandardOpenOption.WRITE, StandardOpenOption.CREATE)) {      long size = in.size();      long bytesRead = 0;      while (bytesRead < size) {        bytesRead += in.transferTo(bytesRead, size - bytesRead, out);      }    }  }  // The way we would have written this prior to modern Streams or FileChannels or try-with-resources.  // It does use Path which is needed for compatibility with the FileCopier API  static void copy_oldSchool(Path source, Path dest) throws IOException {    byte[] buf = new byte[16 * 1024];    InputStream in = new FileInputStream(source.toFile());    try {      OutputStream out = new FileOutputStream(dest.toFile());      try {        for (int n; (n = in.read(buf)) > 0; ) {          out.write(buf, 0, n);        }      } finally {        out.close();      }    } finally {      in.close();    }  } }
Gives this output:
Source file size: 10,000,000              Files_readString:    0.049 sec            Files_readAllBytes:    0.042 sec          Reader_lines_joining:    0.138 sec           Files_lines_forEach:    0.882 sec     BufferedReader_transferTo:    0.032 sec                    Files_copy:    0.006 sec        InputStream_transferTo:    0.022 sec        FileChannel_transferTo:    0.012 sec  FileChannel_transferTo_fixed:    0.006 sec                     oldSchool:    0.013 sec                      grzegorz:    0.029 sec  ERROR: incorrect file length 9994240, expected 10000035                grzegorz_fixed:    0.027 sec Source file size: 100,000,000              Files_readString:    0.375 sec            Files_readAllBytes:    0.281 sec          Reader_lines_joining:    0.638 sec           Files_lines_forEach:    8.723 sec     BufferedReader_transferTo:    0.226 sec                    Files_copy:    0.031 sec        InputStream_transferTo:    0.188 sec        FileChannel_transferTo:    0.052 sec  FileChannel_transferTo_fixed:    0.054 sec                     oldSchool:    0.117 sec                      grzegorz:    0.259 sec  ERROR: incorrect file length 99999744, expected 100000016                grzegorz_fixed:    0.240 sec Source file size: 500,000,000              Files_readString:    1.549 sec            Files_readAllBytes:    1.272 sec          Reader_lines_joining:    2.976 sec           Files_lines_forEach:               skipped     BufferedReader_transferTo:    1.119 sec                    Files_copy:    0.164 sec        InputStream_transferTo:    0.851 sec        FileChannel_transferTo:    0.231 sec  FileChannel_transferTo_fixed:    0.235 sec                     oldSchool:    0.540 sec                      grzegorz:    1.157 sec  ERROR: incorrect file length 499998720, expected 500000022                grzegorz_fixed:    1.193 sec Source file size: 1,000,000,000              Files_readString:    2.846 sec            Files_readAllBytes:    2.325 sec          Reader_lines_joining:    5.005 sec           Files_lines_forEach:               skipped     BufferedReader_transferTo:    2.132 sec                    Files_copy:    0.369 sec        InputStream_transferTo:    1.488 sec        FileChannel_transferTo:    0.587 sec  FileChannel_transferTo_fixed:    0.616 sec                     oldSchool:    1.192 sec                      grzegorz:    2.221 sec  ERROR: incorrect file length 999997440, expected 1000000024                grzegorz_fixed:    2.210 sec Source file size: 16,000,000,000              Files_readString:               skipped            Files_readAllBytes:               skipped          Reader_lines_joining:               skipped           Files_lines_forEach:               skipped     BufferedReader_transferTo:   36.224 sec                    Files_copy:   12.816 sec        InputStream_transferTo:   25.255 sec        FileChannel_transferTo:   12.250 sec  ERROR: incorrect file length 2147483647, expected 16000000026  FileChannel_transferTo_fixed:   89.378 sec                     oldSchool:   27.415 sec                      grzegorz:   44.555 sec  ERROR: incorrect file length 16000000000, expected 16000000026                grzegorz_fixed:   45.611 sec

My stats. File_copy didn't do too well for me. Java 20.0.2, Windows 11.
FileCopyBenchmark Carey 2023-09-07 18-00 files directory on SSD 4TB SATA Source file size: 10,000,000              Files_readString:    0.032 sec            Files_readAllBytes:    0.015 sec          Reader_lines_joining:    0.047 sec           Files_lines_forEach:    0.330 sec     BufferedReader_transferTo:    0.015 sec                    Files_copy:    0.000 sec        InputStream_transferTo:    0.016 sec        FileChannel_transferTo:    0.000 sec  FileChannel_transferTo_fixed:    0.000 sec                     oldSchool:    0.000 sec                      grzegorz:    0.000 sec  ERROR: incorrect file length 9994240, expected 10000015                grzegorz_fixed:    0.015 sec Source file size: 100,000,000              Files_readString:    0.126 sec            Files_readAllBytes:    0.110 sec          Reader_lines_joining:    0.267 sec           Files_lines_forEach:    2.667 sec     BufferedReader_transferTo:    0.078 sec                    Files_copy:    0.016 sec        InputStream_transferTo:    0.031 sec        FileChannel_transferTo:    0.032 sec  FileChannel_transferTo_fixed:    0.031 sec                     oldSchool:    0.047 sec                      grzegorz:    0.063 sec  ERROR: incorrect file length 99999744, expected 100000021                grzegorz_fixed:    0.062 sec Source file size: 500,000,000              Files_readString:    0.627 sec            Files_readAllBytes:    0.534 sec          Reader_lines_joining:    1.193 sec           Files_lines_forEach:               skipped     BufferedReader_transferTo:    0.329 sec                    Files_copy:    3.700 sec        InputStream_transferTo:    2.008 sec        FileChannel_transferTo:    0.188 sec  FileChannel_transferTo_fixed:    0.188 sec                     oldSchool:    0.204 sec                      grzegorz:    0.330 sec  ERROR: incorrect file length 499998720, expected 500000001                grzegorz_fixed:    0.329 sec Source file size: 1,000,000,000              Files_readString:    1.286 sec            Files_readAllBytes:    1.875 sec          Reader_lines_joining:   12.282 sec           Files_lines_forEach:               skipped     BufferedReader_transferTo:    0.972 sec                    Files_copy:    8.185 sec        InputStream_transferTo:    2.557 sec        FileChannel_transferTo:    0.377 sec  FileChannel_transferTo_fixed:    4.312 sec                     oldSchool:    1.004 sec                      grzegorz:    1.552 sec  ERROR: incorrect file length 999997440, expected 1000000005                grzegorz_fixed:    3.043 sec Source file size: 16,000,000,000              Files_readString:               skipped            Files_readAllBytes:               skipped          Reader_lines_joining:               skipped           Files_lines_forEach:               skipped     BufferedReader_transferTo:   31.581 sec                    Files_copy:   78.420 sec        InputStream_transferTo:   72.008 sec        FileChannel_transferTo:   31.210 sec  FileChannel_transferTo_fixed:   31.472 sec                     oldSchool:   32.240 sec                      grzegorz:   31.346 sec  ERROR: incorrect file length 16000000000, expected 16000000027                grzegorz_fixed:   30.798 sec

@Mike, were you running under Windows or Linux?

No.

(MacBook Pro)

And you?

And, do the stats hold up (roughly) under repetition?  Mine generally do, though the high FileChannel.transferTo() on the last run may have been an outlier.

Windows 11. Yes, the numbers hold up under repetition. I shut down all extraneous programs and ran it from the command line instead of Eclipse.

I also tried copying the program to an SSD 2TB M.2 instead of the SATA drive. Overall the numbers were 8-10 times better but their relationship within the group of benchmarks stayed roughly the same. Still not as fast as your numbers.

Stephan van Hulst wrote:Your comparison would be more valuable if you downloaded a Lorem Ipsum generator and then generated a 16GB text file.

I repeated test for Moby-Dick ebook.
The first program (renamed to honor Stephan and Carey contribution    ) is still the fastest:

> Measure-Command { java GrzegorzStephanCarey }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        304,1668


> Measure-Command { java Stephan }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        327,9678


> Measure-Command { java Carey1 }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        401,4525


> Measure-Command { java CareyMike }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        407,3193


> Measure-Command { java Mike1 }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        326,9551


> Measure-Command { java Mike2 }|Select-Object "TotalMilliseconds"

TotalMilliseconds
-----------------
        490,6012

Carey Brown wrote:Your original code stated that it was reading the whole file into memory, our examples followed that directive. Yep, you proved that doesn't work for huge files as we already knew.

This is still true. When I concatenated Moby-Dick multiple times to get 1.3GB file, I was still observing OutOfMemory in programs others then GrzegorzStephanCarey.

On a side-note: while it doesn't really matter that much when you're testing I/O operations, when you're benchmarking an application you REALLY shouldn't be implementing your own timers, especially not using currentTimeMillis() or any of its ilk.

Instead, use a proper benchmarking framework like jmh.

Yeah, I did a quick-and-dirty implementation to make it very easy for anyone to run on their own machine with minimal setup.  But for greater accuracy and reliability, something like JMH is preferable.  My mini-framework is ignoring a host of issues.

Grzegorz Wilanowski wrote:When I concatenated Moby-Dick multiple times to get 1.3GB file, I was still observing OutOfMemory in programs others then GrzegorzStephanCarey.

That's fully expected for most of those methods, as they obviously read the whole file into memory, just like your original program.  However, that shouldn't happen for  Stephan's code, which only failed your previous attempt because there were no line breaks, and consequently reading one line was equivalent to reading the whole file.  Does that one really throw an exception for you using the moby dick text?  What stack trace do you see?  Stephen's version also has the advantage that it successfully writes the entire file.

I see I neglected to include Stephen's code in my previous comparison - sorry about that, Stephan! It looks like it behaves pretty much like the Files_lines_forEach that I did test.  They're slow compared to mot of the other methods tested, and it becomes tiring to wait for larger-size files to complete (which is why I was skipping them for later tests) - but if you give them time, they do complete successfully, even for the 16 GB file copy.
 // From Stephan's method  static void copy_BufferedReader_lines_forEach(Path source, Path dest) throws IOException {    try (var out = new PrintStream(Files.newOutputStream(dest)); var reader = Files.newBufferedReader(source)) {      reader.lines().forEachOrdered(out::println);    }  }