4dsdev
Views: 613,563 Main | Rules/FAQ | Memberlist | Active users | Last posts | Calendar | Stats | Online users | Search 11-20-17 06:55 AM
Guest:

0 users reading Copying a file efficiently (on Ninjhax / N3DS) | 1 bot

Main - Homebrew discussion - Copying a file efficiently (on Ninjhax / N3DS) New reply


d0k3
Posted on 06-04-15 08:54 AM Link | #170
I already asked something similar on GBAtemp, but haven't got much useful help there. What I'm trying to do is to copy a file efficiently on a small homebrew program I coded for Ninjhax (on N3DS, with CTRUlib and CTRcommon installed, if that matters). So far I've been using the C stdio.h functions (fopen / fread / fwrite), but it is awfully slow (around 100kB / sec, regardless of if directories are involved or not).

Here's my copy function:
bool fsPathCopy(const std::string path, const std::string dest, std::function<bool(u64 pos, u64 totalSize)> onProgress) {
if(fsIsDirectory(path)) {
if(mkdir(dest.c_str(), 0777) != 0) return false;
std::vector<FileInfo> contents = fsGetDirectoryContents(path);
for (std::vector<FileInfo>::iterator it = contents.begin(); it != contents.end(); it++)
if (!fsPathCopy((*it).path, dest + "/" + (*it).name)) return false;
return true;
} else {
bool ret = false;
u8 buffer[BUFSIZ];
u64 total = fsGetFileSize(path);
u64 pos = 0;
u64 size;
FILE* fp = fopen(path.c_str(), "rb");
FILE* fd = fopen(dest.c_str(), "wb");
if ((fp != NULL) && (fd != NULL)) {
while ((size = fread(buffer, 1, BUFSIZ, fp)) > 0) {
pos += fwrite(buffer, 1, size, fd);
if((onProgress != NULL) && !onProgress(pos, total)) break;
}
ret = (pos == total);
}
if(fp != NULL) fclose(fp);
if(fd != NULL) fclose(fd);
return ret;
}
}

I've seen that 4DsDev's own staplebutter prefers to use the file io function provided by CTRUlib, but their interfaces are somewhat clunky and I honestly wouldn't want to use them unless there is some kind of advantage in using them. So, is using the functions provided by CTRUlib instead of the standard C / CPP functions recommended?

StapleButter
Posted on 06-04-15 09:29 AM Link | #171
The only reason blargSNES uses them is because the fopen/etc API wasn't available back when I started working on blargSNES. I haven't bothered to change it because, well, it works.



Maybe you could try using them, but I don't think there's overhead from using the fopen/etc API. Instead, try increasing the buffer size. What is your current buffer size, btw?

If increasing the buffer size doesn't help, I don't really see what will.

____________________
blargSNES -- SNES emu for 3DS
More cool stuff

d0k3
Posted on 06-04-15 09:51 AM (rev. 3 of 06-04-15 09:53 AM) Link | #172
I already tried increasing the buffer size to 1024 * 1024 byte - didn't help. The current buffer size, I don't know exactly BUFSIZ is defined by stdio.h. I think it's 8096 byte. So, I don't think the buffer size is the problem here... Are there any estimates on how fast reading from / writing to the internal SD should be (it's a 64GB UHS-1, btw)?

Installing .CIAs via FBI (which is a kind of copy operation, too) seems to work much faster, and the corresponding function uses a combination of fopen / fread (for reading the CIA) and functionality provided by CTRUlib (for installing / writing the CIA). However, the faster speed could be due to other reasons than using CTRUlib functions.

Also: Thank you for your fast reply!

StapleButter
Posted on 06-04-15 10:05 AM Link | #173
Dunno what's pulled there, but maybe the issue is onProgress() taking too much time? Dunno.


It's really weird, though. blargSNES can load ROM files that are 512KB and more, and it's mostly instant. I fail to see why your code would be slow, unless there's some terrible overhead to using fread/fwrite, but I don't think there is.

____________________
blargSNES -- SNES emu for 3DS
More cool stuff

d0k3
Posted on 06-07-15 08:03 PM Link | #182
Alright, just an update. I fixed the slow speed issue (now it's comparable with the speed of other software out there), but how, I'm not exactly sure. I did increase the buffer size, and there where also some problems with my incorrect use of [&] lambda functions, so fixing up that my have solved that.

onProcess() itself had nothing to do with it, btw. While the problem still persisted, I disabed that with basically no effect on speed at all.

Anyways, thank you!


StapleButter
Posted on 06-07-15 08:07 PM Link | #183
Oh then, of course, if you use C++ shiz... just be sure to avoid things that can only be resolved at runtime.

____________________
blargSNES -- SNES emu for 3DS
More cool stuff

profi200
Posted on 06-08-15 12:29 PM Link | #191
Nintendos API is fucking slow. It DMAs the buffer 2 times before it is in your apps heap. I myself would not go below 2 MB of buffer size.

d0k3
Posted on 06-09-15 12:25 PM Link | #192
Posted by profi200
Nintendos API is fucking slow. It DMAs the buffer 2 times before it is in your apps heap. I myself would not go below 2 MB of buffer size.


A buffer that large could possibly slow down things for smaller file operations... Still, good point. I'll try even bigger buffers then.

StapleButter
Posted on 06-09-15 12:46 PM Link | #195
I don't see how it'd slow things down, smaller files would just fit entirely in the buffer and that's all. In those cases (file smaller than buffer), just make sure your code uses the size of the file rather than the size of the buffer.

____________________
blargSNES -- SNES emu for 3DS
More cool stuff


Main - Homebrew discussion - Copying a file efficiently (on Ninjhax / N3DS) New reply

Page rendered in 0.026 seconds. (2048KB of memory used)
MySQL - queries: 28, rows: 82/82, time: 0.019 seconds.
[powered by Acmlm] Acmlmboard 2.064 (2015-10-07)
© 2005-2008 Acmlm, Xkeeper, blackhole89 et al.