Thanks to Jack Bogle, most of us understand the impact of fees on long-term investment returns. He was the one who pushed the notion that while 1% does not sound like a lot, when returns revert to the mean, about 8%, 1% is 12.5% of the total return.

Personally, I think Wall Street is disingenuous about fees altogether. I mean we’re talking about an industry that has no problem charging loads as high as 5.25%, but at the same time, will sue partners and competitors up and down the Street over 20 basis points (i.e., 0.2%).

Despite this, I don’t believe fees represent the largest cost component of investing. Taxes do. Unnecessary taxes on your investments can take a much larger bite, especially right now as the market and your portfolio are rising double digits.

As unpleasant as it is to talk about taxes with April 15 still a fresh memory, at my firm, Gary Goldberg Financial Services, we start reviewing possible tax consequences for our clients just as summer starts. If you don’t know what your tax bite may be this year, or worse, your advisor doesn’t know, this may be the most important column you read all year.

Lets’ take a hypothetical portfolio that returned 15% in the past 12 months: 11% from capital appreciation and 4% from dividends. Let’s further assume that your federal and state income tax bracket is 30%. Under this scenario, your investment return could be reduced by as much as 4.5% as a result of taxes, or almost 30% of your total return. Yikes!

With some planning smart investors can reduce this cost by half or more.

First, if you are an income investor, select investments that pay a “qualified dividend.” The tax rate on these dividends is between 0% and 15% depending on your income tax bracket. Most dividends from U.S. corporations are qualified. However, these are not: real estate investment trusts (REITs), master limited partnerships (MLPs), dividends paid on employee stock options, dividends paid by tax-exempt companies, and dividends paid on savings or money market accounts and special (one-time) dividends are also non-qualified.

In addition, dividends from foreign corporations are not qualified unless the foreign corporation is qualified which, according to the Jobs and Growth Tax Relief and Reconciliation Act of 2003, means it is incorporated in a possession of the United States or eligible for benefits of a comprehensive income tax treaty with the United States. You can find out if a dividend from a foreign corporation is qualified or not by searching on its ticker at dividend.com.

Second, review your existing mutual fund investments and determine whether or not a tax-deferred variable annuity may not be more suitable. Remember, investments in variable annuities grow on a tax deferred basis. This means your capital can compound much more quickly, and the taxes ultimately paid are done so with deflated dollars.

Finally, make sure your advisor is offsetting your gains with portfolio losses. If you have some substantial gains, and given the last 12 months you should, the bite on taking them can be dramatically reduced by taking the losses on the dogs in your portfolio.

For some investors, taking losses is hard. There’s usually a lingering belief the stock will come back. Remember, that a stock or other investment that has fallen by 50% in value must now rise by 100% to get you back to even. The notion that a loss can be mitigated to some degree, by charging it against a gain, and thus paying not tax on the gain, just might give you the resolve you need to get rid of the dogs in your portfolio.

If your broker or advisor has not reviewed and discussed your tax return with you, you and your money are relegated to working harder than you have to. And given that making money is hard enough as it is, you might question the wisdom of working with someone who makes it even harder.

Managing tax consequences and minimizing their impact is critical, especially in a rising tax environment. Do you know how much you are paying in Alternative Minimum Taxes? Does your broker? If not, better sit down and have that discussion. It could save or cost you thousands.

Important Disclosure: Gary Goldberg Financial Services does not provide tax advice. The content of this column should not be viewed as tax, legal or investment advice and is meant to foster a conversation between investors and their advisors.