Ticket #5678 (assigned enhancement)

Opened 1 year ago

Last modified 10 months ago

Respectfully strip newlines in some importers

Reported by: jdub Assigned to: hansengel (accepted)
Priority: normal Milestone: 2.9
Component: General Version: 2.5
Severity: normal Keywords: needs-patch has-docs
Cc:

Description

Filing this as an enhancement because it could do with some discussion and insight from wiser and more experienced heads before being labelled "defect". :-)

I noticed while helping some users import their blogs that importers of HTML content (such as the RSS importer) don't tidy up superfluous newlines in the import format, which results in unnecessary <br/> elements after wpautop() filtering for display. They turn up in the editor too, which reinforces the problem.

I've adapted one of the filter functions to strip superfluous newlines, and changed my RSS importer to use it. The results have been warmly welcomed by users, who no longer have to clean up their imported blog content. ;-)

strip_newlines() should probably go into wp-includes/formatting.php, if there isn't already a function that already serves this purpose. I couldn't find one, so I adapted this.

Given that similar HTML block/inline-savvy string-replacement code exists in other formatting functions, perhaps there's an opportunity for some refactoring here? I feel kind of silly proposing a function that is almost entirely duplicated from other code in the core.

I've used it immediately before the "Clean up content" section in wp-admin/import/rss.php's get_posts(), and in an Advogato importer that I've written (which also uses HTML as the content format).

function strip_newlines($text) {
	// Respectfully strip unnecessary newlines
	$textarr = preg_split("/(<[^>]+>)/Us", $text, -1, PREG_SPLIT_DELIM_CAPTURE);
	$stop = count($textarr); $skip = false; $output = ''; // loop stuff
	for ($ci = 0; $ci < $stop; $ci++) {
		$curl = $textarr[$ci];
		if (! $skip && isset($curl{0}) && '<' != $curl{0}) { // If it's not a tag
			$curl = preg_replace('/[\n\r]+/', ' ', $curl);
		} elseif (strpos($curl, '<code') !== false || strpos($curl, '<pre') !== false || strpos($curl, '<kbd') !== false || strpos($curl, '<style') !== false || strpos($curl, '<script') !== false) {
			$next = false;
		} else {
			$next = true;
		}
		$output .= $curl;
	}
	return $output;
}

Thoughts?

Attachments

5678.r6625.diff (1.5 kB) - added by hansengel on 01/16/08 14:01:08.
Fixes #5678 in r6625 (credit jdub if this is committed, please)
5678.r6631.diff (1.8 kB) - added by hansengel on 01/17/08 13:54:50.
Newer patch—adds PHPDoc and renames function to strip_html_newlines.
5678.r6633.diff (1.4 kB) - added by hansengel on 01/18/08 00:50:50.
An even-newer patch—simplifies strip_html_newlines()

Change History

01/16/08 13:45:41 changed by hansengel

  • owner changed from anonymous to hansengel.
  • status changed from new to assigned.
  • version set to 2.5.
  • milestone changed from 2.6 to 2.5.

+1. I'll get working on a patch.

01/16/08 14:01:08 changed by hansengel

  • attachment 5678.r6625.diff added.

Fixes #5678 in r6625 (credit jdub if this is committed, please)

01/16/08 14:06:23 changed by hansengel

  • keywords set to has-patch tested.

01/16/08 23:31:15 changed by lloydbudd

+1, I've encountered this issues a few times when importing customers blogs from Movable Type and TypePad?.

01/16/08 23:46:32 changed by mdawaffe

This seems like a good idea, but the function looks overly complicated. $skip and $next are never actually used.

I'd also appreciate a comment for the function saying that it's intended for fully marked up, valid HTML only. The function can't be trusted to strip lines from unencoded text, pre-wpautop half HTML/half text, or invalid HTML. That's fine, it should just be noted.

01/17/08 02:59:50 changed by jdub

Oh, I didn't think this would go from proposal to trunk-ready quite so quickly -- thanks.

LiveJournal? is another importer for which this will be useful.

mdawaffe is absolutely right that the function should only be used for fully marked up HTML, and needs to include a comment to that effect. Perhaps it should be renamed to 'strip_html_newlines' or something?

Luckily, for the proposed use case, that covers a large proportion of import formats. :-)

01/17/08 03:33:41 changed by lloydbudd

  • keywords changed from has-patch tested to needs-patch.

01/17/08 10:32:04 changed by westi

  • keywords changed from needs-patch to needs-patch needs-doc.

'strip_html_newlines' is a much better name - and don't forget to write some phpdoc for this new function please :-)

01/17/08 13:54:50 changed by hansengel

  • attachment 5678.r6631.diff added.

Newer patch—adds PHPDoc and renames function to strip_html_newlines.

(follow-up: ↓ 9 ) 01/17/08 23:45:26 changed by hansengel

  • keywords changed from needs-patch needs-doc to has-patch has-docs.

(in reply to: ↑ 8 ) 01/18/08 00:41:58 changed by lloydbudd

  • keywords changed from has-patch has-docs to needs-patch has-docs.

hansengel, the new patch suffers from the same problems as mdawffe identfied above "the function looks overly complicated. $skip and $next are never actually used."

01/18/08 00:50:50 changed by hansengel

  • attachment 5678.r6633.diff added.

An even-newer patch—simplifies strip_html_newlines()

01/18/08 02:32:32 changed by jdub

Unfortunately, you've just simplified it slightly too much. :-)

The element whitespace smarts was pretty crucial to why I wrote the function, given that the input can look like this:

<p>This is a paragraph in which the
wpautop() <a href="#elsewhere">wrapping will
happen in the middle</a> of an
anchor, which is an inline element.</p>

<pre>
But at the same time, we wouldn't want
the newlines to be removed from within
a block element such as a <pre>.
</pre>

With element whitespace smarts, the newlines in the middle of the <p> will be removed (so as not to be replaced with a <br/> during wpautop(), but those within the <pre> will not.

I'll write up another simplified patch that keeps this (and will also include using this function in other HTML-consuming importers).

Thanks. :-)

03/19/08 23:58:33 changed by ffemtcj

  • milestone changed from 2.5 to 2.6.