The wp_set_object_terms() function calls wp_insert_term() and then immediately assumes the return result is an array, around line 1121:
if ( !$id = is_term($term, $taxonomy) )
$id = wp_insert_term($term, $taxonomy);
$term_ids[] = $id['term_id'];
But wp_insert_term() doesn't always return an array, it can return a WP_Error object. This causes a fatal error:
PHP Fatal error: Cannot use object of type WP_Error as array in wordpress/wp-includes/taxonomy.php on line 1122
This turned up while I was writing unit tests. I'm not quite sure what the cause is yet because I can't backtrace a fatal error. It might not happen in normal operation, but I don't think a blind assumption like that is safe considering the consequences.