Scaling issue with hundreds of pages

[4773] should help. I thought I checked this in long ago, but evidently I forgot to. Went ahead and put it in 2.1.

That'll buy us a little bit of breathing room. Good first step. Ideally, I'd like to see page rewrite rules be compressed into a couple generic rules, stuck at the bottom of the list.

http://lightpress.org/post/wp-pages-patch/

Patch is old but might have some bits worth gleaming.

I found a patch for WP 2.0.x and WP 2.1.x at: http://www.marshinsoftware.com/software/wordpress_page_management_patch/

I tested the WP 2.1.x version on WP 2.2 and both my users & I haven't found any problems.

I tried to test it thoroughly by going through the change logs from WP 2.1 to WP 2.2 and copying the changes over, but I was only able to get through the first quarter. There were no changes.

Solves the freezing, crashing or extreme sluggishness of Page Management view for WP 2.3 (Trunk) [Patch Version 5772]

Relates to #5303?

I have been dealing with this issue for months and have found no solution either.. I indeed emailed the developers. What I want to know it ... What are the guarentees that this issue will be addressed in the next Wordpress version (2.4) ? Because it has been 11 months since the initial start of this ticket... and many releases have been released since.

I would like to make some improvements in 2.4, but there are no guarantees it will happen given everything else that is scheduled for 2.4. Here are my thoughts on how to address the page rewrite rules.

We currently have 9 rewrite rules per page. This, obviously, sucks. The rules are there to avoid collisions for those who use category or tag bases of "/" or those using %category%, %tag%, or %postname% at the beginning of their permalink structure. Most people don't do this, making all of those page rules a huge waste. For those who don't use any of the above, let's add a catch-all set of page rules at the end of the rewrite rule list. Those who do use the above will get the old behavior. Losing hundreds of rewrite rules (thousands for some people) and the need to update the rewrite rules every time a page is changed would greatly improve performance.

Replying to ryan:

We currently have 9 rewrite rules per page.

The other thing i noticed about that, Is that they're all before the more commonly used pages(eg. blog posts, comments, categories/tags)

Maybe its possible to have them ordered somehow so that less commonly accessed URLs(such as trackback/attachment) are after the rest of the rules?

Some people of course, Will want pages first before the blog post rules if they're using it for CMS reasons, but the majority of page requests are going to be post related, not page related.

(I'm not too sure if this can be solved by changing priorities on some hooks, i've not looked into the rewrite details)

(In [6351]) Don't save page and attachemtn uris to page_uris and page_attachment_uris. This is not needed. Add an option to use wildcard page rewrite rules instead of per-page rules. see #3614

I did a little housekeeping. We were saving page and attachment rules in the options DB as page_uris and page_attachment_uris. This is not needed now that we cache the entire rewrite rule set to the options DB. page_uris and page_attachment_uris are now gone. I also added an option to force use of a generic set of wildcard page rules. Set WP_Rewrite::user_verbose_rewrite_rules to false to get the generic rules. With this set, you will get nine page rules total instead of nine per page. This code is still rough, so you have to change line 188 by hand. After changing true to false, visit your permalink options page and click the update button to force a rule refresh.

Replying to ryan:

Set WP_Rewrite::user_verbose_rewrite_rules to false to get the generic rules. With this set, you will get nine page rules total instead of nine per page.

The catch all page option breaks any rules which are added by plugins via the 'generate_rewrite_rules' action or 'rewrite_rules_array' filter, as they're added after the catch-all page rule.

Add the catch-all page rules after the filtering is done to ensure custom rules are before

Add the catch-all page rules after the filtering is done to ensure custom rules are before

Allthough on 2nd thoughts, The API of add_rewrite_rule() should probably be used instead by plugins, in which case, $this->extra_rules would go before $page_rules in the merge.

I've posted on this issue in this thread in the wp-forum http://wordpress.org/support/topic/99953?replies=27#post-657683. " Using the patch ... some pages don't show up in Manage->pages, e.g. no subpages below level 3, and almost no pages with two-digit pageID. Any ideas way?"

"Regarding the patch: I'm using the page category organiser-plugin, and it opens its "manage pages -page" without problems. It looks as if it's using a similar function as the patch above to get the pages to list, so that might be a way to proceed ...?

Hey, I added the new rewrite.php file. Replaced the line 188 (true to false). And created a page and it worked super fast! I resaved it and work fine. BUT HUGE PROBLEM. First of all the parent category I created the page in no longer works and also the page i created no longer works. I get a 404 error for both. So this fix definitely did not work for me.... I have to go back to my old rewrite.php file now....

Replying to kicksonfire:

Hey, I added the new rewrite.php file. Replaced the line 188 (true to false). And created a page and it worked super fast! I resaved it and work fine. BUT HUGE PROBLEM. First of all the parent category I created the page in no longer works and also the page i created no longer works. I get a 404 error for both. So this fix definitely did not work for me.... I have to go back to my old rewrite.php file now....

Also, when i reloaded my old rewrite file, the page I created started working and everything was fine. So basically the new file worked to create the page yet it didnt display it. The old rewrite file was able to show the page.

What are your permalink structure, category base, and tag base? Give examples of links that 404.

I am using /%postname% in my custom permalinks field... I actually deleted the pages already. But if you want to look into it. I will grant you access its my please. Please contact me on ceokhan@gmail.com or kicksonfire@hotmail.com

You can't turn off verbose page rules for permalink structures starting with %postname% or %category%. Those are wildcards that match everything. Since they come before the page rules, the page rules are never reached. Permalink structures like yours are the reason we have to do verbose rules for pages. We need those rules to avoid rewrite collisions.

The /%year%/%monthnum%/%day%/%postname%/ permalink structure avoids such collisions for all but one case. Pages that have names that look like years (2007) will 404 because they collide with the %year% rule. With verbose rules on, the page will win and the year will lose.

So what are my options? What am I to do? Am I suppose to live with this? Pleae advise me....

Query page attachments from page requests

Patch attached that allows querying both pages and page attachments from a pagename request. This is needed to support page attachments with the non-verbose rules.

kicksonfire, your permalink structure complicates things. We'd probably have to add a new query var that queries against post, page, and attachment post_types for a given post_name.

I only get the php error: Fatal error: Call to undefined method wpdb::insert() in /var/www/***/html/wp-includes/post.php on line 730 when i upload the new post.php. :(

Replying to skydrak:

I only get the php error: Fatal error: Call to undefined method wpdb::insert() in /var/www/***/html/wp-includes/post.php on line 730 when i upload the new post.php. :(

What version of WordPress are you using? AFAIK wpdb::insert() is only available on trunk(2.4), and nothing in this ticket uses the insert() function.

I use the latest version, 2.3.1.

Tested it again with removing the code by hand in the post.php, but same problem as before: Crashing the server.

Replying to skydrak:

I use the latest version, 2.3.1.

Yeah, You cant just use the latest post.php file, theres too many changes in it which is incompatible with the rest of the 2.3.1 source code.

You'll have to apply the changes in this changeset: [6351] to a stock 2.3.1 file if you want to use them, and even then, i'm not sure if that'll work due to oher file changes.

Well, i have tested it... Works fine, permalinks aren't stored in the database now. But same problem as before: i a will save a static page my server will crash, so i have to rename the index.php everytime if i want to save a static page. That sucks.

(In [6377]) Query page attachments from page requests so that they can both use the same set of rewrite rules. see #3614

Patch automatically uses generic page rules if a compatible permalink structure is being used. No need to change the value by hand.

Alright... So is this the final solution. is the problem solved? Now I am really confused as there are 4 different patches:

manage pages diff.patch (5.9 kB) - added by pirom on 06/28/07 15:11:01. 3614-page-after-filter.diff (1.3 kB) - added by DD32 on 12/04/07 02:31:45. page_attachment_query.diff (1.3 kB) - added by ryan on 12/11/07 08:43:10. enable_generic_page_rules.diff (0.9 kB) - added by ryan on 12/13/07 21:38:14.

Which ones do I need to use if I haven't edited any of the original files as of yet... Please advise.

(In [6398]) Use generic page rewrite rules for compatible permalink strucures. see #3614

You need [6351], [6377], and [63948]. These diffs are against svn trunk, however, not 2.3.1. I don't know if they will apply to 2.3.1. None of these changes will make a difference to permalink structures starting with %postname%. Supporting that takes a lot more work and may not be fixed for the next release.

ryan, DD32 you are the greatest. Added the modifications to my wp 2.3 blog and it works perfectly. If someone is interested in the modified files I can upload them.

Hey skydrak, I am running 2.3.1 and I am kind of a novice with this kind of stuff. I would greatly appreciate it if you can upload the modified files so I can just upload them onto my server. THank you in advance!

http://www.trendhure.com/modifiedfiles.rar

Have fun with it :)

Thank you for the files but big problem... AFTER i added them to wp-includes to replease the old ones, I created a page and got this error:

Fatal error: Call to undefined method: wp_rewrite->page_uri_index() in /home/kofcom/public_html/wp-includes/rewrite.php on line 290

Thanks for providing those files, Skydrak. I was experiencing the same issue and installed the modified files, which did seem to fix the initial problem.

An interesting side-effect is that I can no longer access pages that are children of my main blog page. I have Wordpress isntalled in a sub-folder, so most of my pages are peers with the blog page. The few that are sub-pages of the blog show up in the manage section of the admin, but when I click "view" or try to navigate to the URL directly, I get a 404.

Oops, I can't access paged archives on the blog, either. Looks like this patch won't work for me on 2.3.1.

Replying to daltonrooney:

Thanks for providing those files, Skydrak. I was experiencing the same issue and installed the modified files, which did seem to fix the initial problem. An interesting side-effect is that I can no longer access pages that are children of my main blog page. I have Wordpress isntalled in a sub-folder, so most of my pages are peers with the blog page. The few that are sub-pages of the blog show up in the manage section of the admin, but when I click "view" or try to navigate to the URL directly, I get a 404.

Can you provide example URLs as well as your permalink structure?

Nuts. My permalink structure is just %postname%, and reading through this thread again I realize that is the problem. Changing the permalink structure to include a year first seems to have taken care of that issue, sorry for the false alarm.

I am trying to implement this on http://www.kicksonfire.com but I will have to change my permalinks. The only thing I have a problem with not is that all my pages have links to eachother and are part of a link structure. If i change the permalinks, none of these links will work and I will have to edit over 2000+ pages... Any thoughts.

Wouldn't it be better to just write static rewrite rules to .htaccess if we can?

Got the pages issue fixed with all the help here and the patches. But one problem I run into now is that I have lost all my traffic due to my permalinks change.... I have tried several permanent redirect plugins but none of them work on my blog for some reason! Is it because I have changes these files??

Replying to kicksonfire:

Got the pages issue fixed with all the help here and the patches. But one problem I run into now is that I have lost all my traffic due to my permalinks change.... I have tried several permanent redirect plugins but none of them work on my blog for some reason! Is it because I have changes these files??

Unless we know the permalink structure, Its impossible to suggest help, You'd be better off asking in the support forum i think.

It'll be possible to write some .htaccess redirects for your permalink structure though, just dont think you'll find it in a pre-made plugin.

Well, I couldn't use these patches unless I changed my permalinks as Ryan stated earlier... I was using /%postname%/.... Now I have changed it to: /%year%/%monthnum%/%day%/%postname%/ in order for the patches to work properly. Everything works now... except that all my old slug pages crawled by google return a 404 error. I tried SEVERAL permalinks migration plugins (Advanced Permalink & Dean's Migration) but they didn't work for me as my files have been changed... Please help me because I completely get evaporated from all search engines.