Views: 1,609,380 | Main | Rules/FAQ | Memberlist | Active users | Last posts | Calendar | Stats | Online users | Search | 11-22-24 03:27 AM |
Guest: |
0 users reading Copying a file efficiently (on Ninjhax / N3DS) | 1 bot |
Main - Homebrew discussion - Copying a file efficiently (on Ninjhax / N3DS) | Show post layouts | New reply |
d0k3 |
| ||
Member Normal user Level: 20 Posts: 1/75 EXP: 38198 Next: 4241 Since: 06-04-15 Last post: 3251 days ago Last view: 2998 days ago |
I already asked something similar on GBAtemp, but haven't got much useful help there. What I'm trying to do is to copy a file efficiently on a small homebrew program I coded for Ninjhax (on N3DS, with CTRUlib and CTRcommon installed, if that matters). So far I've been using the C stdio.h functions (fopen / fread / fwrite), but it is awfully slow (around 100kB / sec, regardless of if directories are involved or not).
Here's my copy function: bool fsPathCopy(const std::string path, const std::string dest, std::function<bool(u64 pos, u64 totalSize)> onProgress) {
if(fsIsDirectory(path)) { if(mkdir(dest.c_str(), 0777) != 0) return false; std::vector<FileInfo> contents = fsGetDirectoryContents(path); for (std::vector<FileInfo>::iterator it = contents.begin(); it != contents.end(); it++) if (!fsPathCopy((*it).path, dest + "/" + (*it).name)) return false; return true; } else { bool ret = false; u8 buffer[BUFSIZ]; u64 total = fsGetFileSize(path); u64 pos = 0; u64 size; FILE* fp = fopen(path.c_str(), "rb"); FILE* fd = fopen(dest.c_str(), "wb"); if ((fp != NULL) && (fd != NULL)) { while ((size = fread(buffer, 1, BUFSIZ, fp)) > 0) { pos += fwrite(buffer, 1, size, fd); if((onProgress != NULL) && !onProgress(pos, total)) break; } ret = (pos == total); } if(fp != NULL) fclose(fp); if(fd != NULL) fclose(fd); return ret; } } I've seen that 4DsDev's own staplebutter prefers to use the file io function provided by CTRUlib, but their interfaces are somewhat clunky and I honestly wouldn't want to use them unless there is some kind of advantage in using them. So, is using the functions provided by CTRUlib instead of the standard C / CPP functions recommended? |
StapleButter |
| ||
Member blarg Level: 30 Posts: 80/184 EXP: 151371 Next: 14498 Since: 10-27-14 From: France Last post: 2649 days ago Last view: 2559 days ago |
The only reason blargSNES uses them is because the fopen/etc API wasn't available back when I started working on blargSNES. I haven't bothered to change it because, well, it works.
Maybe you could try using them, but I don't think there's overhead from using the fopen/etc API. Instead, try increasing the buffer size. What is your current buffer size, btw? If increasing the buffer size doesn't help, I don't really see what will. |
d0k3 |
| ||
Member Normal user Level: 20 Posts: 2/75 EXP: 38198 Next: 4241 Since: 06-04-15 Last post: 3251 days ago Last view: 2998 days ago |
I already tried increasing the buffer size to 1024 * 1024 byte - didn't help. The current buffer size, I don't know exactly BUFSIZ is defined by stdio.h. I think it's 8096 byte. So, I don't think the buffer size is the problem here... Are there any estimates on how fast reading from / writing to the internal SD should be (it's a 64GB UHS-1, btw)?
Installing .CIAs via FBI (which is a kind of copy operation, too) seems to work much faster, and the corresponding function uses a combination of fopen / fread (for reading the CIA) and functionality provided by CTRUlib (for installing / writing the CIA). However, the faster speed could be due to other reasons than using CTRUlib functions. Also: Thank you for your fast reply! |
StapleButter |
| ||
Member blarg Level: 30 Posts: 81/184 EXP: 151371 Next: 14498 Since: 10-27-14 From: France Last post: 2649 days ago Last view: 2559 days ago |
Dunno what's pulled there, but maybe the issue is onProgress() taking too much time? Dunno.
It's really weird, though. blargSNES can load ROM files that are 512KB and more, and it's mostly instant. I fail to see why your code would be slow, unless there's some terrible overhead to using fread/fwrite, but I don't think there is. |
d0k3 |
| ||
Member Normal user Level: 20 Posts: 3/75 EXP: 38198 Next: 4241 Since: 06-04-15 Last post: 3251 days ago Last view: 2998 days ago |
Alright, just an update. I fixed the slow speed issue (now it's comparable with the speed of other software out there), but how, I'm not exactly sure. I did increase the buffer size, and there where also some problems with my incorrect use of [&] lambda functions, so fixing up that my have solved that.
onProcess() itself had nothing to do with it, btw. While the problem still persisted, I disabed that with basically no effect on speed at all. Anyways, thank you! |
StapleButter |
| ||
Member blarg Level: 30 Posts: 85/184 EXP: 151371 Next: 14498 Since: 10-27-14 From: France Last post: 2649 days ago Last view: 2559 days ago |
Oh then, of course, if you use C++ shiz... just be sure to avoid things that can only be resolved at runtime. |
profi200 |
| ||
Member Who knows? Level: 19 Posts: 7/70 EXP: 34510 Next: 1267 Since: 05-21-15 From: Germany Last post: 2993 days ago Last view: 2861 days ago |
Nintendos API is fucking slow. It DMAs the buffer 2 times before it is in your apps heap. I myself would not go below 2 MB of buffer size. |
d0k3 |
| ||
Member Normal user Level: 20 Posts: 7/75 EXP: 38198 Next: 4241 Since: 06-04-15 Last post: 3251 days ago Last view: 2998 days ago |
Posted by profi200 A buffer that large could possibly slow down things for smaller file operations... Still, good point. I'll try even bigger buffers then. |
StapleButter |
| ||
Member blarg Level: 30 Posts: 88/184 EXP: 151371 Next: 14498 Since: 10-27-14 From: France Last post: 2649 days ago Last view: 2559 days ago |
I don't see how it'd slow things down, smaller files would just fit entirely in the buffer and that's all. In those cases (file smaller than buffer), just make sure your code uses the size of the file rather than the size of the buffer. |
Main - Homebrew discussion - Copying a file efficiently (on Ninjhax / N3DS) | Show post layouts | New reply |
Page rendered in 0.016 seconds. (2048KB of memory used) MySQL - queries: 26, rows: 79/79, time: 0.006 seconds. Acmlmboard 2.064 (2018-07-20) © 2005-2008 Acmlm, Xkeeper, blackhole89 et al. |