Your gut feeling is somewhat correct. It's hard to get precise measurements because of the nature of how Apex executes on a shared resource, but generally speaking, if you anticipate less than about 50 values, using the if statement is very, very slimly faster (~1/100th of a second), and for about every ~200 values, the if statement version takes longer than the remove-null-at-end by another millisecond or so.
The if statement has a small advantage with a low amount of data, because hash codes take time to compute, but this is later dwarfed by the sheer number of if statements evaluated. Thus, from a performance optimization point of view, the answer is essentially "it doesn't matter which you use."
However, two important points to note, and this is why I always use the remove-null-at-end approach: cyclomatic complexity is reduced by 1, and the code is very slightly shorter, making it more legible. Using this pattern consistently would slightly reduce cognitive load, and make Apex PMD a little bit happier about one less branching statement.
If you're seriously hurting on CPU time, and you're looking for a quick fix, switching to remove-null-at-end won't help you; you'd need to be able to use it 20 times to even make a 1 second difference in CPU time, assuming 10,000 items in each pass, which is likely already going to blow up the CPU limits.