XH 1.6.1: Improving Filebrowser Performance

Discussions and requests related to new CMSimple features, plugins, templates etc. and how to develop.
Please don't ask for support at this forums!
Post Reply
cmb
Posts: 14225
Joined: Tue Jun 21, 2011 11:04 am
Location: Bingen, RLP, DE
Contact:

XH 1.6.1: Improving Filebrowser Performance

Post by cmb » Fri Jan 03, 2014 11:35 pm

Hello Community,

Jerry reported issues regarding the performance of the default filebrowser again. These seem to be related to very many files and/or folders in the respective userfiles folder (Filebrowser_Controller::getFolders()) and/or to large content/many embedded images (Filebrowser_Controller::usedImages()).

I suggest we have a closer look at this, and optimize where sensible and feasible. IMO it's acceptable to have some limitations regarding the filebrowser (similar to the limitations regarding the size of content.htm) -- however, the fewer the better.

We may consider to backport eventual changes to XH 1.5.x.

Christoph
Christoph M. Becker – Plugins for CMSimple_XH

cmb
Posts: 14225
Joined: Tue Jun 21, 2011 11:04 am
Location: Bingen, RLP, DE
Contact:

Re: XH 1.6.1: Improving Filebrowser Performance

Post by cmb » Fri Sep 12, 2014 2:38 pm

Hello Community,

I've had a futher look at this issue. I set up a fresh CMSimple_XH 1.6.2 and put 1240 JPEG images (219 kB each) directly into userfiles/images/. Then I prepared a content.htm where each of these images is embedded once. The content had 155 pages which 8 images each, and a total size of 524 kB. Then I did some profiling; the following times are the server side processing only. All tests were done with PHP 5.4.19.

Requesting a page in view mode took roughly 250 msec. Requesting Files -> Images took about 4 sec. 1 sec was needed for finfo::file() (solely used in Filebrowser_View::isImageFile()). As fileinfo is most likely not available everywhere there is already a fallback checking the file extensions. Forcing this fallback reduced the overall processing time by nearly 1 sec -- a considerable improvement, which shouldn't cause any problems. We should change (and simplify) the code respectively.

Filebrowser_Controller::usedImages(), which I suspected to cause problems, seems not a real problem. In this scenario it took less than 100 msec, what is ignorable. Some further investigation showed that it only becomes a bottleneck, if many images are used repeately on many pages; that seems to be quite unlikely, so we can ignore this function.

Anyway, 4 sec for 1240 images is okay, and on a typical server it may be much faster. However, the measurements have been done multiple times, and I gave roughly an average stripping a few worst cases. And actually, these worst cases might have caused the timeout on Jerry's server. The particular problem seems to be getimagesize() (and finfo_file(), if would be used), which is called once for each image to get its size, so that it can be displayed (it is furthermore used to calculate the dimensions of the thumbnail, what is not necessary, as this could be handled by CSS, but as we're calling it anyway, that's not the problem). getimagesize() has to access not only the meta information of the directory entry, but it has to actually read (part of) the file, and that can be a problem when so many files have to be processed in a single request. For some requests I've measured ~100 msec per image, and some request even hit the max_execution_time (30 sec).

What makes that even worse, is that the file listing gets very long (the HTML content size is 1.7 MB), and that all the images have to be requested by the browser -- that makes for 1240 requests, which is way to much. Basically, I see two solutions to address these issues: Ajax or pagination. The Ajax solution might get pretty complex, and I don't know how it would feel in the end; maybe the file list could be loaded rather quickly, but the user would have to wait for individual detail information. A pagination is certainly more straight forward. A typical pagination (show only N images per page) is not really user friendly, though, and actually I don't think it is necessary. On one hand, hi_kcfinder already offers this possiblity, and on the other hand the user can do it in a more logical manner by partitioning the images across several subfolders.

So new setup: everything as before, but the 1240 images partitioned to 31 subfolders of userfiles/images/ (with 40 images each). The request times for Files -> Images or individual subfolders are now between 0.5 and 1 sec, the content size is about 70 kB, and the browser's load event is triggered after less than 1.5 sec. Considering that I'm testing on a moderate desktop computer which also runs the webserver, that is okay.

Summary

I suggest that we:
  • remove the finfo->file() call in Filebrowser_View::isImageFile(), using always the fallback on the file extension
  • document that users should not put too many files in a single folder (say less than 100), but rather split many images across several subfolders; or alternatively use hi_kcfinder
  • hope that somebody finds the time to improve the filebrowser for XH 1.7 or 2.0 (I see several issues, the most important being to get rid of the separate entry point of the editor browser, which requires several ugly workarounds)
Christoph
Christoph M. Becker – Plugins for CMSimple_XH

manu
Posts: 1086
Joined: Wed Jun 04, 2008 12:05 pm
Location: St. Gallen - Schweiz
Contact:

Re: XH 1.6.1: Improving Filebrowser Performance

Post by manu » Mon Sep 15, 2014 10:17 am

cmb wrote:...
A pagination is certainly more straight forward. A typical pagination (show only N images per page) is not really user friendly, though, and actually I don't think it is necessary. On one hand, hi_kcfinder already offers this possiblity, and on the other hand the user can do it in a more logical manner by partitioning the images across several subfolders.
what possibility? pagination? how? I let hi_kcfinder run over a gallery: 770files 202Mb. 1st run (with preview) crashed on execution time, 2nd run returns in seconds (probably Server Caching). It seems that hi_kcfinder does not crosscheck the usage of an image in content on deleting the image.
cmb wrote: Summary

I suggest that we:
  • remove the finfo->file() call in Filebrowser_View::isImageFile(), using always the fallback on the file extension
  • document that users should not put too many files in a single folder (say less than 100), but rather split many images across several subfolders; or alternatively use hi_kcfinder
  • hope that somebody finds the time to improve the filebrowser for XH 1.7 or 2.0 (I see several issues, the most important being to get rid of the separate entry point of the editor browser, which requires several ugly workarounds)
All your suggestions seem reasonable. IMO file browser runs pretty fast and solid, as does hi_kcfinder. Huge loads of images with linked in loads of pages are rather theoretical. If someone really needs this, one has to find the apropriate solution. CMSimple might be the wrong one. Therefore it might suffice to write a hint in the help file of the filebrowser(s) or in the certain plugin's overview.

cmb
Posts: 14225
Joined: Tue Jun 21, 2011 11:04 am
Location: Bingen, RLP, DE
Contact:

Re: XH 1.6.1: Improving Filebrowser Performance

Post by cmb » Mon Sep 15, 2014 1:33 pm

manu wrote:what possibility? pagination? how?
Indeed, hi_kcfinder doesn't have any pagination -- I must have confused that with Ajaxfilemanager_XH. :oops:
manu wrote:I let hi_kcfinder run over a gallery: 770files 202Mb. 1st run (with preview) crashed on execution time, 2nd run returns in seconds (probably Server Caching).
In the first run, kcfinder creates thumbnails of all the images. I assume that this takes quite a while.
manu wrote:It seems that hi_kcfinder does not crosscheck the usage of an image in content on deleting the image.
Indeed (AFAIK).
manu wrote:Therefore it might suffice to write a hint in the help file of the filebrowser(s) or in the certain plugin's overview.
I have added a "Limitations" section to the Wiki. I'm already working on the German translation.
Christoph M. Becker – Plugins for CMSimple_XH

cmb
Posts: 14225
Joined: Tue Jun 21, 2011 11:04 am
Location: Bingen, RLP, DE
Contact:

Re: XH 1.6.1: Improving Filebrowser Performance

Post by cmb » Sat Sep 20, 2014 11:10 am

cmb wrote:I have added a "Limitations" section to the Wiki. I'm already working on the German translation.
The German translation is finished.
cmb wrote:I suggest that we:
  • remove the finfo->file() call in Filebrowser_View::isImageFile(), using always the fallback on the file extension
Done (r1374).
Christoph M. Becker – Plugins for CMSimple_XH

Post Reply