The orderby gallery shortcode has no effect on the order of the images in the gallery. The SQL it generates is incorrect. A gallery shortcode [gallery columns="4" orderby="post_name ASC"] generates the SQL query
SELECT DISTINCT * FROM wp_posts WHERE 1=1 AND post_type = 'attachment' AND wp_posts.post_parent = 8 AND (post_mime_type LIKE 'image/%') GROUP BY wp_posts.ID ORDER BY "post_name ASC" DESC
The ORDER BY clause is incorrect. In fact, even a default [gallery] shortcode generates an invalid ORDER BY:
...ORDER BY "menu_order ASC, ID ASC" DESC
It should be unquoted and without the final DESC.